• Yaowu Xu's avatar
    added a last stage rounding for 8x8 inverse dct · a66c945c
    Yaowu Xu authored
    Prior to the added rounding, tests on randomly generated data showed
    that forward-inverse transform round trip errors are about 3.02/block
    for input range [-10,10] and 2.68/block for input range [-256, 255].
    The added rounding reduced the errors to 0.031/block for input range
    [-10,10] and 0.037/block for input range [-256, 255].
    Maximum round trip error on for any pixel position is 1.
    The average errors are calculated based on 100,000 blocks of randomly
    with the specified ranges.
    Paul mentioned in discussion that the change was not clear on why we
    need change the rounding, so Patch 2 intends to make the rationale
    obvious in code, it merged the two separate shifts into one, and the
    two separate rounding factors into one. Patch 1 and 2 have same
    numerical test results.
    Change-Id: Ic5e2f5463de17253084d8b2398c4a210194b20de
idctllm.c 12.4 KB