• Luc Trudeau's avatar
    [CFL] Fix rare overflow in distortion computation · 4c5df105
    Luc Trudeau authored
    Worst case SSE for a 12-bit 64x64 block requires 48 bits
    (2*(12+log(64)+log(64))). As such, the dist variable must
    be int64.
    
    Results on Subset1 (compared to 19b5c8fa with CfL enabled)
    
      PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
    0.0030 |  0.0001 |  0.0100 |   0.0026 | 0.0024 | -0.0008 |     0.0028
    
    Change-Id: I1364c089c223b96daed942175a915fed0f6f1023
    4c5df105
rdopt.c 451 KB