Commit b85a5a7e authored by Peter de Rivaz's avatar Peter de Rivaz
Browse files

Correction to costing rect_tx

When built with var_tx and ext_tx, select_tx_size_fix_type is used
to compute the cost for using a particular tx_type.
The code indexes the array inter_tx_type_costs at the wrong location
resulting in a zero cost for signalling tx_type for rect_tx blocks.

Change-Id: Iba38be3a0d822109f778f0600b242dfb40359766
parent abf6fb99
......@@ -3309,8 +3309,8 @@ static int64_t select_tx_size_fix_type(const AV1_COMP *cpi, MACROBLOCK *x,
int ext_tx_set = get_ext_tx_set(mbmi->tx_size, bsize, is_inter);
if (is_inter) {
if (ext_tx_set > 0)
*rate += cpi->inter_tx_type_costs[ext_tx_set]
[txsize_sqr_up_map[mbmi->tx_size]]
*rate +=
cpi->inter_tx_type_costs[ext_tx_set][txsize_sqr_map[mbmi->tx_size]]
[mbmi->tx_type];
} else {
if (ext_tx_set > 0 && ALLOW_INTRA_EXT_TX)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment