Commit 568b7c7e authored by David Barker's avatar David Barker

aom-qm: Fix overflow in quantizer calculation

By using a combination of a small qmlevel and large qindex,
it is possible for the quantizer 'dqv' to end up > 2^15.
This currently causes dqv to overflow, so widen its type
to avoid that.

Note that this is certainly an extreme case, but not entirely
useless - if using a 32x32 transform, the dequantized value
can be as small as dqv/2, which may still fit in 16 bits.

Change-Id: I0451e70ba6b17620d94386a98d873f0a8c615635
parent bdfb4a6f
......@@ -130,7 +130,7 @@ static int decode_coefs(MACROBLOCKD *xd, PLANE_TYPE type, tran_low_t *dqcoeff,
const uint8_t *band_translate = get_band_translate(tx_size);
int dq_shift;
int v, token;
int16_t dqv = dq[0];
int32_t dqv = dq[0];
#if CONFIG_NEW_QUANT
const tran_low_t *dqv_val = &dq_val[0][0];
#endif // CONFIG_NEW_QUANT
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment