[CFL] Fix negative rounding issue in alpha dist
get_scaled_luma_q0(-alpha_q3, pred_buf_q3[i]) is NOT equivalent to -get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]). When the product alpha_q3*pred_buf_q3[i] is an exact multiple of 32 (0.5 in Q6), then the right shift will round both positive and negative values towards infinity, creating a bias. So, e.g., get_scaled_luma_q0(-4, 8) will yield 0, but -get_scaled_luma_q0(4, 8) will yield -1. Results on Subset1 (compared to parent With CfL enabled) PSNR | PSNR Cb | PSNR Cr | PSNR HVS | SSIM | MS SSIM | CIEDE 2000 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 https://arewecompressedyet.com/?job=cfl-no-round-fix%402017-10-07T11%3A50%3A47.711Z&job=cfl-round-fix%402017-10-07T02%3A15%3A51.359Z Change-Id: I8a7900c32fbd7213f1ed4e09c3626c063800e186
parent
ef1b74c7
No related branches found
No related tags found
Loading
Please register or sign in to comment