• David Barker's avatar
    Unify high-precision convolve filters: convolve-round · 726a953c
    David Barker authored
    * Reduce bit widths of intermediate values where possible
    * Change ROUND_POWER_OF_TWO_SIGNED to ROUND_POWER_OF_TWO
      in av1(_highbd)_convolve_2d
    * Apply offsetting and bounds checking, to match the intended
      hardware implementation
    * Separate the implementations of av1(_highbd)_convolve_2d
      into compound-round and non-compound-round cases. This is because
      there are now a significant number of differences between the
      functions.
    
    Overall, this is expected to affect the bitstream and encoder output
    when convolve-round alone is enabled, but *not* when compound-round
    is enabled.
    
    Change-Id: I8c21e0645fd11f64c59552885f87f4a5dd40ccf7
    726a953c
convolve.c 39.4 KB