Skip to content
  • Dominic Symes's avatar
    [NORMATIVE-DECODING, txmg]: Adjust IADST4 · 6f81f128
    Dominic Symes authored
    Coefficients used for the VP9 version of IADST4 satisfy the
    mathematical relation sinpi[1]+sinpi[2]==sinpi[4] which is
    a trigonometric property useful for optimizing the four point
    transform. Unfortunately the change in shift used in the latest
    code means that rounding errors are introduced so that this identity
    no longer holds. We think the identity should be restored by
    rounding 4964.563 to 4964 rather than 4965. Similarly the identity
    has been fixed for other bit shifts in the table where it does not
    hold (even though these are not currently used I think).
    
    This change also fixes a bug with the range checking for IADST4.
    
    We see little change on the statistics from a local ACWY run
    on objective-1-fast I-frame only:
    
             PSNR  PSNR-HVS SSIM  PSNR-Cr APSNR APSNR-Cr MS-SSIM PSRNR-YUV
    Average  +0.01 +0.01    -0.02 -0.15   +0.01 -0.15    +0.01   +0.01
    
    BUG=aomedia:1360
    
    Change-Id: Icc09a1c59929e58bbc922d4b8b73de4a14104a8e
    6f81f128