Skip to content
  • Johann Koenig's avatar
    Fix variance overflow · fea3556e
    Johann Koenig authored
    In the variance calculations the difference is summed and later squared.
    When the sum exceeds sqrt(2^31) the value is treated as a negative when
    it is shifted which gives incorrect results.
    
    To fix this we cast the result of the multiplication as unsigned.
    
    The alternative fix is to shift sum down by 4 before multiplying.
    However that will reduce precision.
    
    For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
    change).
    
    PPC change is untested.
    
    Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
    fea3556e