• David Barker's avatar
    loop-retoration: Fix overflow in self-guided filter · 9c1f92ba
    David Barker authored
    A while ago, I calculated some bounds on the intermediate values inside
    the self-guided filter. These bounds turned out to be not quite correct
    in one particular instance (when we have a large region of max-value
    pixels).
    
    This caused a variable to overflow a uint32_t when decoding 12-bit
    streams in the reference decoder, and would force 8/10-bit-only
    hardware to use wider buffers than intended in order to match the
    reference code.
    
    Fortunately, this can be fixed quite easily, with minimal changes
    to the filter output. See comments within the patch for the exact
    details.
    
    Also re-instate a Wikipedia link which seems to have gone missing
    but which provided useful context for the derivation of the bounds.
    
    Change-Id: I83d4a277a37eff048af9989cccf19202fafb17b5
    9c1f92ba
restoration.c 83.4 KB