Commit ab141115 authored by Timothy B. Terriberry's avatar Timothy B. Terriberry
Browse files

var_tx: Fix distortion calc. in av1_tx_block_rd_b

This was hard-coding the assumption that the block size for the
smallest TX size was also the smallest block size. This is no
longer true since fe67ed6a landed.

As a result, for TX blocks that overlapped the frame edge, it was
only measuring distortion on the upper-left 2x2 part of each 4x4
sub-block, causing the encoder to prefer larger transforms which
cause such overlap and avoid transforms which do not, causing a

This patch uses the appropriate conversion table, which fixes the


Change-Id: Id253cf0f3a5252378e3f340b8350120639ff5c88
parent 1ce569d7
...@@ -3946,7 +3946,8 @@ void av1_tx_block_rd_b(const AV1_COMP *cpi, MACROBLOCK *x, TX_SIZE tx_size, ...@@ -3946,7 +3946,8 @@ void av1_tx_block_rd_b(const AV1_COMP *cpi, MACROBLOCK *x, TX_SIZE tx_size,
src + ((idy * src_stride + idx) << tx_size_wide_log2[0]); src + ((idy * src_stride + idx) << tx_size_wide_log2[0]);
uint8_t *const r = uint8_t *const r =
rec_buffer + ((idy * MAX_TX_SIZE + idx) << tx_size_wide_log2[0]); rec_buffer + ((idy * MAX_TX_SIZE + idx) << tx_size_wide_log2[0]);
cpi->fn_ptr[0].vf(s, src_stride, r, MAX_TX_SIZE, &this_dist); cpi->fn_ptr[txsize_to_bsize[0]].vf(s, src_stride, r, MAX_TX_SIZE,
tmp += this_dist; tmp += this_dist;
} }
} }
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment