• David Barker's avatar
    [non-normative] Clarify clamping code for inverse transforms · 92b49a6d
    David Barker authored
    Although the clamping for the inverse transforms is still being
    discussed, it is useful to have clear code.
    The current code is not ideal - both av1_inv_txfm1d.c and av1_inv_txfm2d.c
    have functions called clamp_buf(), but they do different things - one
    clamps to the exact number of bits passed in, the other optionally
    clamps to 16 bits.
    This commit refactors things so that:
    * clamp_* always clamps to the exact number of bits passed in
      To compensate, we push the "if bit <= 16: clamp to 16 bits"
      logic into the callers of the clamp_* functions
    * We deduplicate some of the code
    Change-Id: I1b9d85d6db5d291814c055a8113b7b054690b04c
av1_inv_txfm2d.c 20.7 KB