Commit b433f4c4 authored by Jingning Han's avatar Jingning Han
Browse files

Properly scale the distortion in lv-map optimize_b

Account for the bit-depth in calculating the distortion for lv-map
coefficient level optimization.

Change-Id: I47c60f15bfda5aeba8e32f1ac898015650a057db
parent ea636197
......@@ -2061,7 +2061,9 @@ int av1_optimize_txb(const AV1_COMMON *cm, MACROBLOCK *x, int plane,
const int shift = av1_get_tx_scale(tx_size);
const int64_t rdmult =
(x->rdmult * plane_rd_mult[is_inter][plane_type] + 2) >> 2;
((x->rdmult * plane_rd_mult[is_inter][plane_type] << (2 * (xd->bd - 8))) +
2) >>
2;
uint8_t levels_buf[TX_PAD_2D];
uint8_t *const levels = set_levels(levels_buf, width);
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment