• Yushin Cho's avatar
    Fix daala-dist, rd tx search · 04eb9594
    Yushin Cho authored
    Previously, for block >=8x8, and tx < 8x8,
    we skipped setting the early-exit flag in block_rd_txfm() because
    distortion for sub8x8 tx block is from MSE but reference (best)
    is from daala-dist.
    However, not setting early-exit flag turned out to be the reason
    for a regression in MSE probe mode of daala-dist because
    it loses the chance to set rd_stats properly.
    On the other hand, there is still a small regression, say 0.05% psnr bd-rate,
    which seems to occur in the case that a tx block in a partition has chosen
    the skipped rd_cost since it is smaller than non-skip rd_cost and
    set the early-exit flag to 0 (so, not exit), but the daala-dist applied
    to the whole partition cannot access the same info but can choose from
    two kinds of rd_costs:
    1) all tx blocks are skipped (even if a tx block has non-zero coeff) and 0 bits
    2) sum of final distortion of all tx blocks (i.e. non-zero coeff decoded)
    and bits to encode coeffs.
    Change-Id: I2ec69972aa1f22d465293cb9e8d5e18ef2c6f7f3