• Yaowu Xu's avatar
    fixed an issue related to 2nd order size due to merge artifacts. · 9ded6e37
    Yaowu Xu authored
    For 8x8 transformed macroblock, the 2nd order transform is a 2x2 haar
    transform, here there is only 4 coefficients total. A previous merge
    changed these to 64, causing crashes when encoding with 8x8 transform
    enabled. (i.e. when input video image size > 640x360 ) This commit
    reverts them back to 4 and fixes the crashes.
    Change-Id: I3290b81f8c0d32c7efec03093a61ea57736c0550
quantize.c 38.5 KB