Skip to content
  • Yaowu Xu's avatar
    added code to clear 2nd order block when appropriate · 88e24f07
    Yaowu Xu authored
    It is discovered that in rare situations the 2nd order block may
    produce a few small magnitude coefficients that has no effect on
    reconstruction. The situations are a combination of low quantizer
    values (high quality) and low energy in residual signals (content
    dependent). This commit added code to detect such cases and reset
    the 2nd order block to all 0.
    
    Patch 1 to 4 used code to do all-zero-check on idct result buffer,
    and tests on derf set showed a consistent gain of .12%-.14% on all
    metrics.But due to a recent change Ie31d90b, the idct result buffer
    is not longer populated. So patch 5&6 use an alternative method to
    detect the situations. Tests on derf set now shows a consistent
    quality gain of .16%-.20%.
    
    As suggested by Jim, Patch 7&8 removed the condition of all first
    order block not having any coefficient, instead we reset 2nd order
    coefficients to all 0 if sum of absolute value of the coefficients
    is small. So it does slightly more than just detecting the oddity
    as discussed above, but tests on derf set now show a consistent
    gain of .20%-.23% on all metrics.
    
    It is worth noting here that this change does not have any effect
    on mid/high quantizer range, it only affects the quantizer value
    18 or blow. Within this range, the change helps compression by up
    to 2.5% on clips in the derf set.
    
    Change-Id: I718e19cf59a4fc2462cb7070832759beb9f7e7dd
    88e24f07