Commit a8db0e8d authored by Sarah Parker's avatar Sarah Parker Committed by Debargha Mukherjee

Fix distortion for new_quant in optimize_b_greedy

This bug was causing a 14% drop in performance on the lowres set,
this fix reduces the drop to 0.2%.

Change-Id: I166efc9cc5735c3366a0989e924d2c9fae9e706b
parent e39525ba
......@@ -269,7 +269,7 @@ static int optimize_b_greedy(const AV1_COMMON *cm, MACROBLOCK *mb, int plane,
d2_a = ((int64_t)dx * dx + depth_round) >> depth_shift;
#else // CONFIG_DAALA_TX
#if CONFIG_NEW_QUANT
dx = av1_dequant_coeff_nuq(x, dqv, dequant_val[band_translate[i]]) -
dx = av1_dequant_coeff_nuq(x_a, dqv, dequant_val[band_translate[i]]) -
(coeff[rc] * (1 << shift));
dx >>= xd->bd - 8;
#else // CONFIG_NEW_QUANT
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment