• Yaowu Xu's avatar
    optimize forward 16x16 DCT for accuracy · 499fe05d
    Yaowu Xu authored
    This commit added pre/post scaling for first half of fDCT16x16 to
    reduce error, by simulation of 100,000 blocks for random inputs,
    the average sse reduced from 2.1/block to 0.0498/block.
    also enabled tests for 16x16 fDCT and iDCT
    Change-Id: Id2a95f0464c6dd4118797d456237ae90274c0f02
vp9_dct.c 72.4 KB