• Johann's avatar
    Fix variance overflow · fea3556e
    Johann authored
    In the variance calculations the difference is summed and later squared.
    When the sum exceeds sqrt(2^31) the value is treated as a negative when
    it is shifted which gives incorrect results.
    To fix this we cast the result of the multiplication as unsigned.
    The alternative fix is to shift sum down by 4 before multiplying.
    However that will reduce precision.
    For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
    PPC change is untested.
    Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
variance_c.c 12.7 KB