• Yaowu Xu's avatar
    moved scaling from dequantization to inverse transform for T8x8 · 454c7abc
    Yaowu Xu authored
    Previously, the scaling related to extended quantize range happens in
    dequantization stage, which implies the coefficients form forward
    transform are in different scale(4x) from dequantization coefficients
    This worked fine when there was not distortion computation done based
    on 8x8 transform, but it completely wracked the distortion estimation
    based on transform coefficients and dequantized transform coefficients
    introduced in commit f64725a0 for macroblocks using 8x8 transform.
    This commit fixed the issue by moving the scaling into the stage of
    inverse 8x8 transform.
    TODO: Test&Verify the transform/quantization pipeline accuracy.
    Change-Id: Iff77b36a965c2a6b247e59b9c59df93eba5d60e2