• David Barker's avatar
    aom-qm: Fix overflow in quantizer calculation · 568b7c7e
    David Barker authored
    By using a combination of a small qmlevel and large qindex,
    it is possible for the quantizer 'dqv' to end up > 2^15.
    This currently causes dqv to overflow, so widen its type
    to avoid that.
    Note that this is certainly an extreme case, but not entirely
    useless - if using a 32x32 transform, the dequantized value
    can be as small as dqv/2, which may still fit in 16 bits.
    Change-Id: I0451e70ba6b17620d94386a98d873f0a8c615635
detokenize.c 12.8 KB