Commit 2615d6ea authored by Yunqing Wang's avatar Yunqing Wang

Fix the transform size/type search condition

While encoding a key frame with quantizer = 0 and aq-mode = 1,
for some segment_ids, the quantizer got modified and could be
> 0, and lossless[segment_id] might be 0 or 1 depending on the
segment_id. Namely, blocks with lossless[segment_id] = 0 were
allowed to choose transform sizes other than 4x4. This conflicted
with tx_mode which was a frame-level decision. In this patch,
the transform search condition was modified so that the transform
choice was consistent with tx_mode of that frame.

BUG=aomedia:104

Change-Id: Ia39127b5dee129283a133cf5e4000da62d9e0f1c
parent feb925fe
......@@ -1849,7 +1849,7 @@ static void super_block_yrd(const AV1_COMP *const cpi, MACROBLOCK *x,
assert(bs == xd->mi[0]->mbmi.sb_type);
if (xd->lossless[xd->mi[0]->mbmi.segment_id]) {
if (xd->lossless[0]) {
choose_smallest_tx_size(cpi, x, rd_stats, ref_best_rd, bs);
} else if (cpi->sf.tx_size_search_method == USE_LARGESTALL) {
choose_largest_tx_size(cpi, x, rd_stats, ref_best_rd, bs);
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment