• Brennan Shacklett's avatar
    Fix eobs buffer overflow caused by SSSE3 quantize_fp · 05a62c0a
    Brennan Shacklett authored
    The SSSE3 version of av1_quantize_fp stored the new value of eob
    with a 32bit mov instruction, but the eob values are stored in an
    array of uint16_t. This caused the last eob store to write 2 extra
    bytes off the end of the eob array.
    
    This doesn't seem to currently be a problem on master, but it
    causes errors for my RDO refactoring under certain conditions.
    
    Change-Id: I9f7e9c4e40b37b8a2c232b87d170d24593c1066c
    05a62c0a
quantize_ssse3_x86_64.asm 8.41 KB