Skip to content
  • Steinar Midtskogen's avatar
    Retune the CLPF kernel · 4f0b3ed8
    Steinar Midtskogen authored
    CLPF performance had degraded by about 0.5% over the past six months,
    which isn't totally surprising since the codec is a moving target.
    About half of that degradation comes from the improved 7 bit filter
    coefficients.  Therefore, CLPF needs to be retuned for the current
    codec.
    
    This patch makes two (normative) changes to the CLPF kernel:
    
    * The clipping function was changed from clamp(x, -s, s) to
          sign(x) * max(0, abs(x) - max(0, abs(x) - s +
                 (abs(x) >> (bitdepth - 3 - log2(s)))))
      This adds a rampdown to 0 at -32 and 32 (for 8 bit, -128 & 128
      for 10 bit, etc), so large differences are ignored.
    
    * 8 taps instead of 6 taps:
                   1
        4          3
      13 31  ->  13 31
        4          3
                   1
    
    AWCY results: low delay  high delay
    PSNR:           -0.40%     -0.47%
    PSNR HVS:        0.00%     -0.11%
    SSIM:           -0.31%     -0.39%
    CIEDE 2000:     -0.22%     -0.31%
    APSNR:          -0.40%     -0.48%
    MS SSIM:         0.01%     -0.12%
    
    About 3/4 of the gains come from the new clipping function.
    
    Change-Id: Idad9dc4004e71a9c7ec81ba62ebd12fb76fb044a
    4f0b3ed8