Commit 008ea46a authored by Jingning Han's avatar Jingning Han

Account for 2x2 tx_size in extrabits_size

Offset the transform block size with respect to TX_4X4 when
counting the cat6 extra bit size. This accounts for the potential
use of 2x2 transform block size in cb4x4.


Change-Id: Ia15ac123e4f31dc66455c3aa6925ae5e754de493
parent e9f77424
......@@ -119,7 +119,12 @@ static INLINE int av1_get_cat6_extrabits_size(TX_SIZE tx_size,
// TODO(debargha): Does TX_64X64 require an additional extrabit?
if (tx_size > TX_32X32) tx_size = TX_32X32;
int bits = (int)bit_depth + 3 + (int)tx_size;
int tx_offset = (tx_size < TX_4X4) ? 0 : (int)(tx_size - TX_4X4);
int tx_offset = (int)(tx_size - TX_4X4);
int bits = (int)bit_depth + 3 + tx_offset;
assert(bits <= (int)sizeof(av1_cat6_prob));
return bits;
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment