• David Barker's avatar
    Fix bug in av1_optimize_b when tx64x64 is enabled · c747a78e
    David Barker authored
    In both versions of av1_optimize_b, the code to calculate the
    dequantized coefficient from a modified quantized coefficient
    was not compatible with tx64x64.
    
    Specifically, the new dequantized coefficient was calculated as an
    offset applied to the original coefficient. This had a fixup block
    for 32x32 transforms, but not for 64x64 transforms.
    
    For simplicity, instead of extending the fixup block, we now
    recalculate the dequantized coefficient directly.
    
    BUG=aomedia:537
    
    Change-Id: I86c1c8820f324ad3f6afd6a720ff59f754384a19
    c747a78e
encodemb.c 74 KB