• Jingning Han's avatar
    Use saturated addition in SSSE3 of 32x32 quant · 458c2833
    Jingning Han authored
    The 32x32 forward transform can potentially reach peak coefficient
    value close to 32700, while the rounding factor can go upto 610.
    This could cause overflow issue in the SSSE3 implementation of 32x32
    quantization process.
    
    This commit resolves this issue by replacing the addition operations
    with saturated addition operations in 32x32 block quantization.
    
    Change-Id: Id6b98996458e16c5b6241338ca113c332bef6e70
    458c2833
vp9_rtcd_defs.sh 43.3 KB