Skip to content
  • Johann Koenig's avatar
    improve integer version of filter · 20b855c3
    Johann Koenig authored
    the lookup table is based on floating point calculations (see source)
    
    by moving the *3 before the downshift and adding the rounding bit, the
    delta (LUT - integer) goes from:
    ______________________________________
    __ 1__ 1______________________________
    __ 1__ 1______________________________
    ____ 1______ 1________________________
    ____ 1 2__ 2 1________________________
    ______ 1 1 2__ 2__ 2__ 2 1 1__________
    ________ 1 1 2 2__ 1 2 3 1 2__ 2__ 2__
    to:
    __-1__-1______________________________
    ______________________________________
    ____-1______-1________________________
    ______________________________________
    ________-1______________-1____________
    ______________________________________
    
    it's important to be able to use the integer version because the LUT
    more or less precludes SIMD optimizations
    
    Change-Id: I45a81127dc7b72a06fba951649135d9d918386c0
    20b855c3