• Yi Luo's avatar
    Fix the overflow of av1_fht32x32() in 2D DCT_DCT · 157e45a4
    Yi Luo authored
    - Use range check function to avoid DCT_DCT overflow.
      We need to re-develop the column txfm side scaling/rounding. Now,
      we prefer to maintain the current BDRate level.
    - Encoder user level time reduction <1% owing to av1_fht32x32_avx2.
    - Add MemCheck unit test and fdct32() unit test.
    
    Change-Id: I1e67030f67bc637859798ebe2f6698afffb8531c
    157e45a4
hybrid_fwd_txfm.c 15.6 KB