Commit 05a62c0a authored by Brennan Shacklett's avatar Brennan Shacklett

Fix eobs buffer overflow caused by SSSE3 quantize_fp

The SSSE3 version of av1_quantize_fp stored the new value of eob
with a 32bit mov instruction, but the eob values are stored in an
array of uint16_t. This caused the last eob store to write 2 extra
bytes off the end of the eob array.

This doesn't seem to currently be a problem on master, but it
causes errors for my RDO refactoring under certain conditions.

Change-Id: I9f7e9c4e40b37b8a2c232b87d170d24593c1066c
parent de2ba4f6
......@@ -174,7 +174,7 @@ cglobal quantize_%1, 0, %2, 15, coeff, ncoeff, skip, zbin, round, quant, \
pshuflw m7, m8, 0x1
pmaxsw m8, m7
pextrw r6, m8, 0
mov [r2], r6
mov [r2], r6w
; skip-block, i.e. just write all zeroes
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment