-
Sarah Parker authored
This addresses an inconsistency between the set used to decode the tx_type in the bitstream and the set used for the tx_type search. Previously, the set used to read/write the tx_type was based on the smallest tx_size in the vartx partitioning, but the search uses a set based on the largest possible tx_size. This patch changes the tx_type search to use the transform type set associated with the tx_size 1 recursive level down from the max square tx_size to make the search more consistent with the bitstream syntax. If a tx_size is selected for an invalid tx_type, DCT_DCT is used for that partition instead. This patch also adds assertions to all exposed transform functions to ensure that no illegal transform type/size combinations occur. This currently gets a 0.1% drop in performance on lowres. The drop is due to the reduction of the tx_types available for 32x16 and 16x32 transform sizes. Before this patch, 32x16 and 16x32 transforms were getting assigned a set of 12 tx_types, some of which we did not intend to support for these sizes. Change-Id: I44aca4876b261c345623cd04ad6235bca4532701
90024e44