Fix variance overflow
In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we cast the result of the multiplication as unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). PPC change is untested. Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
Showing
- vp8/encoder/arm/armv6/vp8_variance16x16_armv6.asm 1 addition, 1 deletionvp8/encoder/arm/armv6/vp8_variance16x16_armv6.asm
- vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_h_armv6.asm 1 addition, 1 deletion...ncoder/arm/armv6/vp8_variance_halfpixvar16x16_h_armv6.asm
- vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_hv_armv6.asm 1 addition, 1 deletion...coder/arm/armv6/vp8_variance_halfpixvar16x16_hv_armv6.asm
- vp8/encoder/arm/armv6/vp8_variance_halfpixvar16x16_v_armv6.asm 1 addition, 1 deletion...ncoder/arm/armv6/vp8_variance_halfpixvar16x16_v_armv6.asm
- vp8/encoder/arm/neon/variance_neon.asm 11 additions, 11 deletionsvp8/encoder/arm/neon/variance_neon.asm
- vp8/encoder/arm/neon/vp8_subpixelvariance16x16_neon.asm 2 additions, 2 deletionsvp8/encoder/arm/neon/vp8_subpixelvariance16x16_neon.asm
- vp8/encoder/arm/neon/vp8_subpixelvariance16x16s_neon.asm 8 additions, 8 deletionsvp8/encoder/arm/neon/vp8_subpixelvariance16x16s_neon.asm
- vp8/encoder/arm/neon/vp8_subpixelvariance8x8_neon.asm 2 additions, 2 deletionsvp8/encoder/arm/neon/vp8_subpixelvariance8x8_neon.asm
- vp8/encoder/ppc/variance_altivec.asm 3 additions, 3 deletionsvp8/encoder/ppc/variance_altivec.asm
- vp8/encoder/ppc/variance_subpixel_altivec.asm 1 addition, 1 deletionvp8/encoder/ppc/variance_subpixel_altivec.asm
- vp8/encoder/variance_c.c 5 additions, 5 deletionsvp8/encoder/variance_c.c
- vp8/encoder/x86/variance_mmx.c 5 additions, 5 deletionsvp8/encoder/x86/variance_mmx.c
- vp8/encoder/x86/variance_sse2.c 4 additions, 4 deletionsvp8/encoder/x86/variance_sse2.c
- vp8/encoder/x86/variance_ssse3.c 2 additions, 2 deletionsvp8/encoder/x86/variance_ssse3.c
Loading
Please register or sign in to comment