Commit a45b104a authored by Luc Trudeau's avatar Luc Trudeau

[CFL] Fix negative rounding issue in alpha dist

get_scaled_luma_q0(-alpha_q3, pred_buf_q3[i]) is NOT equivalent to
-get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]). When the product
alpha_q3*pred_buf_q3[i] is an exact multiple of 32 (0.5 in Q6), then the right
shift will round both positive and negative values towards infinity, creating a
bias. So, e.g., get_scaled_luma_q0(-4, 8) will yield 0, but
-get_scaled_luma_q0(4, 8) will yield -1.

Results on Subset1 (compared to parent With CfL enabled)
  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0000 |  0.0000 |  0.0000 |   0.0000 | 0.0000 |  0.0000 |     0.0000

https://arewecompressedyet.com/?job=cfl-no-round-fix%402017-10-07T11%3A50%3A47.711Z&job=cfl-round-fix%402017-10-07T02%3A15%3A51.359Z

Change-Id: I8a7900c32fbd7213f1ed4e09c3626c063800e186
parent ef1b74c7
......@@ -6221,12 +6221,13 @@ static int64_t cfl_alpha_dist_lbd(const int16_t *pred_buf_q3,
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
const int uv = src[i];
const int scaled_luma = get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]);
diff = uv - clip_pixel(scaled_luma + dc_pred);
diff = uv -
clip_pixel(get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]) + dc_pred);
dist += diff * diff;
diff = uv - clip_pixel(-scaled_luma + dc_pred);
diff = uv - clip_pixel(get_scaled_luma_q0(-alpha_q3, pred_buf_q3[i]) +
dc_pred);
dist_neg += diff * diff;
}
pred_buf_q3 += MAX_SB_SIZE;
......@@ -6267,12 +6268,15 @@ static int64_t cfl_alpha_dist_hbd(const int16_t *pred_buf_q3,
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
const int uv = src[i];
const int scaled_luma = get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]);
diff = uv - clip_pixel_highbd(scaled_luma + dc_pred, bit_depth);
diff = uv - clip_pixel_highbd(
get_scaled_luma_q0(alpha_q3, pred_buf_q3[i]) + dc_pred,
bit_depth);
dist += diff * diff;
diff = uv - clip_pixel_highbd(-scaled_luma + dc_pred, bit_depth);
diff = uv - clip_pixel_highbd(
get_scaled_luma_q0(-alpha_q3, pred_buf_q3[i]) + dc_pred,
bit_depth);
dist_neg += diff * diff;
}
pred_buf_q3 += MAX_SB_SIZE;
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment