• Rupert Swarbrick's avatar
    Fix calculation of tx_type cost for rectangular transforms · ffbff57d
    Rupert Swarbrick authored
    This patch fixes up av1_tx_type_cost to match the code in
    av1_write_tx_type. Beforehand, we wrongly assumed a 32x16 block needed
    to signal its transform size (with rect-tx-ext & rect-tx-ext-intra)
    because we were passing 16x16 to get_ext_tx_types.
    
    I've also changed av1_write_tx_type to use get_min_tx_size rather than
    inlining its body. No functional change, but it's probably better to
    use the same helper function both times.
    
    Change-Id: Iff6ee0bff2d332d5270fe0219db88c95e0b051d0
    ffbff57d
bitstream.c 178 KB