Skip to content
  • James Zern's avatar
    Fix variance (signed integer) overflow · 98473443
    James Zern authored
    In the variance calculations the difference is summed and later squared.
    When the sum exceeds sqrt(2^31) the value is treated as a negative when
    it is shifted which gives incorrect results.
    
    To fix this we force the multiplication to be unsigned.
    
    The alternative fix is to shift sum down by 4 before multiplying.
    However that will reduce precision.
    
    For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
    change).
    
    This change is based on:
    16982342 Missed some variance casts
    fea3556e Fix variance overflow
    
    Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
    98473443