Commit 5c453db2 authored by Luc Trudeau's avatar Luc Trudeau
Browse files

[CFL] clip CFL prediction to avoid overflow

The value predicted using CfL is clipped to avoid going out of the
scope of the uint8. Both overflow and underflow was detected over
Subtset1.

Results on Subset1 (compared to 7e55571e with CfL enabled)

  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0019 |  0.0001 |  0.0009 |   0.0047 | 0.0020 |  0.0023 |     0.0012

Change-Id: Ie1190e2286aa90542eaa68b814cc5cfa031acb73
parent 4aee17d7
......@@ -239,7 +239,8 @@ void cfl_predict_block(MACROBLOCKD *const xd, uint8_t *dst, int dst_stride,
cfl_load(cfl, row, col, width, height);
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
dst[i] = (uint8_t)(alpha * (y_pix[i] - avg) + dc_pred + 0.5);
// TODO(ltrudeau) call clip_pixel_highbd when HBD is enabled.
dst[i] = clip_pixel((int)(alpha * (y_pix[i] - avg) + dc_pred + 0.5));
}
dst += dst_stride;
y_pix += MAX_SB_SIZE;
......
......@@ -1469,9 +1469,13 @@ static int cfl_alpha_dist(const uint8_t *y_pix, int y_stride,
for (int t_i = b_i; t_i < w; t_i++) {
const double scaled_luma = alpha * (t_y_pix[t_i] - y_average);
const int uv = t_src[t_i];
diff = uv - (int)(scaled_luma + dc_pred_bias);
// TODO(ltrudeau) add support for HBD.
diff = uv - clip_pixel((int)(scaled_luma + dc_pred_bias));
dist += diff * diff;
diff = uv + (int)(scaled_luma - dc_pred_bias);
// TODO(ltrudeau) add support for HBD.
diff = uv - clip_pixel((int)(-scaled_luma + dc_pred_bias));
dist_neg += diff * diff;
}
t_y_pix += y_stride;
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment