• Rupert Swarbrick's avatar
    Don't work in TX64X64 blocks on a subsampled chroma plane · 4e7b7d6f
    Rupert Swarbrick authored
    With ext-partition and tx64x64 enabled, the encoder could choose
    TX_64X64 to encode the transform for a subsampled plane of a
    BLOCK_128X128 block. This broke an assumption in the nested loop in
    write_tokens_b and also caused bug 827 (with a rather cryptic
    failure).
    
    This patch changes get_vartx_max_txsize to ensure that the encoder and
    decoder don't think they can use TX_64X64 in this situation. It also
    adds a couple of assertions to the loop mentioned above so that if
    something comes unstuck it'll be much more obvious what went wrong.
    
    BUG=aomedia:827
    
    Change-Id: Ie093f2f20f6242949d68e950c8f95b100867ee17
    4e7b7d6f
bitstream.c 207 KB