• Debargha Mukherjee's avatar
    Fix loopfilter for rectangular transforms · 3eb713e2
    Debargha Mukherjee authored
    Properly determine and use horizontal and vertical masks
    for loop filtering when rectangular transforms are used.
    Fixes an intermittent mismatch issue and improves coding
    BDRATE results for ext-tx + rect-tx:
    lowres: -3.739% (up from -3.443%)
    midres: -3.366% (up from -3.006%)
    Change-Id: If26fa14261f3893662eb1245f0b876d68513247c
loopfilter.c 78.2 KB