Commit ed80c049 authored by Sarah Parker's avatar Sarah Parker Committed by Sebastien Alaiwan

Correct new-quant distortion computation with daala-tx

This mistake was fixed in a8db0e8d
and brought back in for
the daala-tx case in 4a05a58c.

Change-Id: I9cf36b63f2dacd9d01887bfaacd8b318245f882a
parent 5e982003
......@@ -261,7 +261,7 @@ static int optimize_b_greedy(const AV1_COMMON *cm, MACROBLOCK *mb, int plane,
if (x_a != 0) {
#if CONFIG_DAALA_TX
#if CONFIG_NEW_QUANT
dx = av1_dequant_coeff_nuq(x, dqv, dequant_val[band_translate[i]]) -
dx = av1_dequant_coeff_nuq(x_a, dqv, dequant_val[band_translate[i]]) -
coeff[rc];
#else // CONFIG_NEW_QUANT
dx -= (dqv + sz) ^ sz;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment