Skip to content
Snippets Groups Projects
Commit a45b104a authored by Luc Trudeau's avatar Luc Trudeau
Browse files

[CFL] Fix negative rounding issue in alpha dist

get_scaled_luma_q0(-alpha_q3, pred_buf_q3[i]) is NOT equivalent to
-get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]). When the product
alpha_q3*pred_buf_q3[i] is an exact multiple of 32 (0.5 in Q6), then the right
shift will round both positive and negative values towards infinity, creating a
bias. So, e.g., get_scaled_luma_q0(-4, 8) will yield 0, but
-get_scaled_luma_q0(4, 8) will yield -1.

Results on Subset1 (compared to parent With CfL enabled)
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0000 |  0.0000 |  0.0000 |   0.0000 | 0.0000 |  0.0000 |     0.0000

https://arewecompressedyet.com/?job=cfl-no-round-fix%402017-10-07T11%3A50%3A47.711Z&job=cfl-round-fix%402017-10-07T02%3A15%3A51.359Z

Change-Id: I8a7900c32fbd7213f1ed4e09c3626c063800e186
parent ef1b74c7
No related branches found
No related tags found
2 merge requests!6Rav1e 11 yushin 1,!3Rav1e 10 yushin
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment