1. 07 Nov, 2012 1 commit
    • James Zern's avatar
      Fix variance (signed integer) overflow · 98473443
      James Zern authored
      In the variance calculations the difference is summed and later squared.
      When the sum exceeds sqrt(2^31) the value is treated as a negative when
      it is shifted which gives incorrect results.
      
      To fix this we force the multiplication to be unsigned.
      
      The alternative fix is to shift sum down by 4 before multiplying.
      However that will reduce precision.
      
      For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
      change).
      
      This change is based on:
      16982342 Missed some variance casts
      fea3556e Fix variance overflow
      
      Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
      98473443
  2. 11 Oct, 2012 1 commit
  3. 03 Aug, 2012 1 commit
    • Daniel Kang's avatar
      16x16 DCT blocks. · fed8a183
      Daniel Kang authored
      Set on all 16x16 intra/inter modes
      
      Features:
      - Butterfly fDCT/iDCT
      - Loop filter does not filter internal edges with 16x16
      - Optimize coefficient function
      - Update coefficient probability function
      - RD
      - Entropy stats
      - 16x16 is a config option
      
      Have not tested with experiments.
      
      hd:     2.60%
      std-hd: 2.43%
      yt:     1.32%
      derf:   0.60%
      
      Change-Id: I96fb090517c30c5da84bad4fae602c3ec0c58b1c
      fed8a183
  4. 29 Jun, 2012 1 commit
    • Daniel Kang's avatar
      Add 8x8 fDCT unit test. · 26641c74
      Daniel Kang authored
      Also factorize ACMRandom to acm_random.h.
      
      Change-Id: I3b6eeb36fcbf7ae6dd3d2892bc40348f5c17982b
      26641c74
  5. 28 Jun, 2012 1 commit
  6. 26 Jun, 2012 1 commit