Commit 4e6a8f30 authored by Timothy B. Terriberry's avatar Timothy B. Terriberry Committed by Tim Terriberry

Fix signed left-shifts added in e93acb2d.

Since we now require C99, this is undefined behavior.
Thanks to Luc Trudeau for the report and Alex Converse for the
suggestion on how to make the macro safe for all integer sizes.

Change-Id: I99a1342dfedb3e17a6869269be317c2ed26bfe9b
parent cea11f2a
......@@ -35,6 +35,11 @@ extern "C" {
#define IS_POWER_OF_TWO(x) (((x) & ((x)-1)) == 0)
/* Left shifting a negative value became undefined behavior in C99 (downgraded
from merely implementation-defined in C89). This should still compile to the
correct thing on any two's-complement machine, but avoid ubsan warnings.*/
#define AOM_SIGNED_SHL(x, shift) ((x) * (((x)*0 + 1) << (shift)))
// These can be used to give a hint about branch outcomes.
// This can have an effect, even if your target processor has a
// good branch predictor, as these hints can affect basic block
......
......@@ -386,7 +386,8 @@ static int av1_pvq_decode_helper(od_dec_ctx *dec, tran_low_t *ref_coeff,
// copy int16 inputs to int32
for (i = 0; i < blk_size * blk_size; i++) {
ref_int32[i] = ref_coeff_pvq[i] << (OD_COEFF_SHIFT - coeff_shift);
ref_int32[i] =
AOM_SIGNED_SHL(ref_coeff_pvq[i], OD_COEFF_SHIFT - coeff_shift);
}
od_pvq_decode(dec, ref_int32, out_int32, quant[1] << (OD_COEFF_SHIFT - 3),
......
......@@ -1212,8 +1212,9 @@ PVQ_SKIP_TYPE av1_pvq_encode_helper(
// copy int16 inputs to int32
for (i = 0; i < tx_blk_size * tx_blk_size; i++) {
ref_int32[i] = ref_coeff_pvq[i] << (OD_COEFF_SHIFT - coeff_shift);
in_int32[i] = coeff_pvq[i] << (OD_COEFF_SHIFT - coeff_shift);
ref_int32[i] =
AOM_SIGNED_SHL(ref_coeff_pvq[i], OD_COEFF_SHIFT - coeff_shift);
in_int32[i] = AOM_SIGNED_SHL(coeff_pvq[i], OD_COEFF_SHIFT - coeff_shift);
}
if (abs(in_int32[0] - ref_int32[0]) < pvq_dc_quant * 141 / 256) { /* 0.55 */
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment