Commit d18c84cb authored by Jingning Han's avatar Jingning Han

Fix quantize_b_c function

Correctly compute the quantized coefficient index. This resolves
an enc/dec mismatch issue when the c function is in use.

Change-Id: I670b5a18310fa68c994f43b6a8bc9b0d8ff907ae
parent 0a6731f5
......@@ -83,7 +83,7 @@ static void quantize_b_helper_c(const tran_low_t *coeff_ptr, intptr_t n_coeffs,
quant_shift_ptr[rc != 0]) >>
(16 - log_scale)); // quantization
#endif // CONFIG_AOM_QM
qcoeff_ptr[rc] = (tmp ^ coeff_sign) - coeff_sign;
qcoeff_ptr[rc] = (tmp32 ^ coeff_sign) - coeff_sign;
#if CONFIG_AOM_QM
const int dequant =
(dequant_ptr[rc != 0] * iqm_ptr[rc] + (1 << (AOM_QM_BITS - 1))) >>
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment