Commit 04eb9594 authored by Yushin Cho's avatar Yushin Cho
Browse files

Fix daala-dist, rd tx search

Previously, for block >=8x8, and tx < 8x8,
we skipped setting the early-exit flag in block_rd_txfm() because
distortion for sub8x8 tx block is from MSE but reference (best)
is from daala-dist.
However, not setting early-exit flag turned out to be the reason
for a regression in MSE probe mode of daala-dist because
it loses the chance to set rd_stats properly.

On the other hand, there is still a small regression, say 0.05% psnr bd-rate,
which seems to occur in the case that a tx block in a partition has chosen
the skipped rd_cost since it is smaller than non-skip rd_cost and
set the early-exit flag to 0 (so, not exit), but the daala-dist applied
to the whole partition cannot access the same info but can choose from
two kinds of rd_costs:
1) all tx blocks are skipped (even if a tx block has non-zero coeff) and 0 bits
2) sum of final distortion of all tx blocks (i.e. non-zero coeff decoded)
and bits to encode coeffs.

Change-Id: I2ec69972aa1f22d465293cb9e8d5e18ef2c6f7f3
parent a0cc9aa8
......@@ -1757,6 +1757,8 @@ static void daala_dist_sub8x8_txfm_rd(MACROBLOCK *x, BLOCK_SIZE bsize,
args->rd_stats.rdcost = rd;
args->this_rd = rd;
if (args->this_rd > args->best_rd) args->exit_early = 1;
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment