• Sarah Parker's avatar
    Use tx_size 1 level down for transform type search · 90024e44
    Sarah Parker authored
    This addresses an inconsistency between the set used
    to decode the tx_type in the bitstream and the set used
    for the tx_type search. Previously, the set used to
    read/write the tx_type was based on the smallest tx_size
    in the vartx partitioning, but the search uses a set
    based on the largest possible tx_size. This patch
    changes the tx_type search to use the transform type
    set associated with the tx_size 1 recursive level down from
    the max square tx_size to make the search more consistent
    with the bitstream syntax. If a tx_size is selected for an
    invalid tx_type, DCT_DCT is used for that partition instead.
    
    This patch also adds assertions to all exposed transform
    functions to ensure that no illegal transform type/size
    combinations occur.
    
    This currently gets a 0.1% drop in performance on lowres.
    The drop is due to the reduction of the tx_types available
    for 32x16 and 16x32 transform sizes. Before this patch,
    32x16 and 16x32 transforms were getting assigned a
    set of 12 tx_types, some of which we did not intend to
    support for these sizes.
    
    Change-Id: I44aca4876b261c345623cd04ad6235bca4532701
    90024e44
decodeframe.c 148 KB