Commit b844ee1b authored by David Barker's avatar David Barker Committed by Urvang Joshi

[intra, bugfix] Prevent overflow in DC_PRED

Commit https://aomedia-review.googlesource.com/c/aom/+/40541 replaced
a division in the DC intra predictor by an approximate
multiply+shift sequence.

Unfortunately, this approximation is able to produce out-of-range
values. For example, consider 4x8 DC_PRED, with bit depth = 10.
If all of the context pixels are 0x3FF (the max value), then we get:

sum = 12 * 0x3FF
expected_dc = (sum * 0xAB) >> 11 = 1024 = 0x400

This means that we need to insert a clip_pixel(_highbd) operation
at the end of the DC prediction, to bring this value back in range.

BUG=aomedia:1272

Change-Id: I9beb9ac8a4b39803865f7e23932402ecd1d6f672
parent 8152737f
......@@ -385,6 +385,7 @@ static INLINE void dc_predictor_rect(uint8_t *dst, ptrdiff_t stride, int bw,
}
expected_dc = (int)(((uint64_t)sum * multiplier) >> shift);
expected_dc = clip_pixel(expected_dc);
for (r = 0; r < bh; r++) {
memset(dst, expected_dc, bw);
......@@ -1020,7 +1021,6 @@ static INLINE void highbd_dc_predictor_rect(uint16_t *dst, ptrdiff_t stride,
const uint16_t *left, int bd,
uint32_t multiplier, int shift) {
int i, r, expected_dc, sum = 0;
(void)bd;
for (i = 0; i < bw; i++) {
sum += above[i];
......@@ -1030,6 +1030,7 @@ static INLINE void highbd_dc_predictor_rect(uint16_t *dst, ptrdiff_t stride,
}
expected_dc = (int)(((uint64_t)sum * multiplier) >> shift);
expected_dc = clip_pixel_highbd(expected_dc, bd);
for (r = 0; r < bh; r++) {
aom_memset16(dst, expected_dc, bw);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment