• David Barker's avatar
    [intra, bugfix] Prevent overflow in DC_PRED · b844ee1b
    David Barker authored
    Commit https://aomedia-review.googlesource.com/c/aom/+/40541 replaced
    a division in the DC intra predictor by an approximate
    multiply+shift sequence.
    
    Unfortunately, this approximation is able to produce out-of-range
    values. For example, consider 4x8 DC_PRED, with bit depth = 10.
    If all of the context pixels are 0x3FF (the max value), then we get:
    
    sum = 12 * 0x3FF
    expected_dc = (sum * 0xAB) >> 11 = 1024 = 0x400
    
    This means that we need to insert a clip_pixel(_highbd) operation
    at the end of the DC prediction, to bring this value back in range.
    
    BUG=aomedia:1272
    
    Change-Id: I9beb9ac8a4b39803865f7e23932402ecd1d6f672
    b844ee1b
intrapred.c 43.5 KB