1. 07 Nov, 2012 1 commit
    • James Zern's avatar
      Fix variance (signed integer) overflow · 98473443
      James Zern authored
      In the variance calculations the difference is summed and later squared.
      When the sum exceeds sqrt(2^31) the value is treated as a negative when
      it is shifted which gives incorrect results.
      
      To fix this we force the multiplication to be unsigned.
      
      The alternative fix is to shift sum down by 4 before multiplying.
      However that will reduce precision.
      
      For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
      change).
      
      This change is based on:
      16982342 Missed some variance casts
      fea3556e Fix variance overflow
      
      Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
      98473443
  2. 06 Nov, 2012 12 commits
  3. 05 Nov, 2012 4 commits
  4. 03 Nov, 2012 1 commit
  5. 02 Nov, 2012 5 commits
  6. 01 Nov, 2012 3 commits
  7. 31 Oct, 2012 10 commits
  8. 30 Oct, 2012 4 commits