Skip to content
  • John Koleszar's avatar
    quantizer: fix assertion in fast quantizer path · 0a49747b
    John Koleszar authored
    The fast quantizer assembly code has not been updated to match the new
    exact quantizer, which was made the default in commit 6adbe090.
    Specifically, they are not aware of the potential for the coefficient
    to be scaled, which results in the quantized result exceeding the range
    of the DCT. This patch restores the previous behavior of using the
    non-shifted coefficients when in the fast quantizer code path, but
    unfortunately requires rebuilding the tables when switching between the
    two.
    
    Change-Id: I0a33f5b3850335011a06906f49fafed54dda9546
    0a49747b