• Yi Luo's avatar
    Fix rectangle transform computation overflow · 63bd6dc9
    Yi Luo authored
    - Add 16-bit saturation in fdct_round_shift().
    - Add extreme value tests and round trip error tests.
    - Fix inv 4x8 txfm calculation accuracy.
    - Fix 4x8, 8x4, 8x16, 16x8, 16x32, 32x16 extreme value tests.
    - BDRate: lowres: -0.034
              midres: -0.036
              hdres:  -0.013
    BUG=webm:1340
    
    Change-Id: I48365c1e50a03a7b1aa69b8856b732b483299fb5
    63bd6dc9
dct.c 63.3 KB