• Timothy B. Terriberry's avatar
    var_tx: Fix distortion calc. in av1_tx_block_rd_b · ab141115
    Timothy B. Terriberry authored
    This was hard-coding the assumption that the block size for the
    smallest TX size was also the smallest block size. This is no
    longer true since fe67ed6a landed.
    
    As a result, for TX blocks that overlapped the frame edge, it was
    only measuring distortion on the upper-left 2x2 part of each 4x4
    sub-block, causing the encoder to prefer larger transforms which
    cause such overlap and avoid transforms which do not, causing a
    regression.
    
    This patch uses the appropriate conversion table, which fixes the
    regression.
    
    BUG=aomedia:593
    
    Change-Id: Id253cf0f3a5252378e3f340b8350120639ff5c88
    ab141115
rdopt.c 398 KB