• Yushin Cho's avatar
    New experiment DIST_8x8 · b7b60c57
    Yushin Cho authored
    A framework for computing a distortion at 8x8 luma block level
    during RDO-based mode decision search. New 8x8 distortion metric can
    be plugged in by way of this tool.
    Existing daala_dist now uses this experiment as well.
    Other possible applications that can make use of this experiment would be
    a distortion meric, which should apply at 8x8 pixels such as PSNR-HVS, SSIM, or etc.
    A rd_cost for final coding mode decision for a super block is
    computed for a partition size 8x8 or larger. For a block larger than 8x8,
    a distortion of each 8x8 block is independently computed then summed up.
    The rd_cost for 8x8 block with new 8x8 distortion metric is computed
    only when the mode decision of its sub8x8 blocks are completed.
    However, MSE distortion metric is used with sub8x8 mode decision. Thus,
    early termination is also determined with the MSE based rd_cost.
    Because the best rd_cost (i.e. the reference rd_cost) during sub8x8 prediction
    or sub8x8 tx is based on new 8x8 distortion while each sub8x8 uses MSE,
    the existing early termination cannot be used (And this can be the one of possible reason
    for the BD-Rate change with this revision).
    For a sub8x8 prediction, prediction mode for each sub8x8 block of a 8x8 block is
    decided with existing MSE and then av1_dist_8x8() is applied to the 8x8 pixels.
    (There is also av1_dist_8x8_diff, which can input diff signal directly)
    For a sub8x8 tx in a block larger than 8x8, instead of computing MSE distortion for
    each sub8x8 tx block, we wait until all sub8x8 tx blocks are encoded before av1_dist_8x8()
    is applied to 8x8 pixels.
    Sub8x8 prediction and transformas were most of tricky parts in this change.
    Two kind of distortions, for a) predicted pixels and b) decoded pixels
    (i.e. predicted + possible reconstructed residue), are always computed during RDO.
    In order to access those two signals a) and b) for a 8x8 block after
    its sub8x8 mode decision is finished, a) and b) need be properly stored for later retrieval.
    The CB4X4 makes the task of accessing a) and b) signals for sub8x8 block further difficult,
    since the intermediate data (i.e. a and/or b) for sub8x8 block
    are not easily accessible outside of current partition unless reconstruced
    with decided coding modes.
    Change-Id: If60301a890c0674a3de1d8206965bbd6a6495bb7