• Steinar Midtskogen's avatar
    Make the CDEF RDO handle 4:2:2 properly · ef1b74c7
    Steinar Midtskogen authored
    This fixes an assert:
    av1/common/cdef_block.c:561: cdef_filter_fb: Assertion `bsize ==
    BLOCK_8X8 || bsize == BLOCK_4X4' failed.
    The RDO simply assigned a strength of 0 in the 4:2:2 case and called
    cdef_filter_fb(), but cdef_filter_fb() will complain about 4:2:2 even
    if the strength is 0.
    The fix assigns a chroma mse of 0 when the the subsampling is
    different for x and y rather than to call the filter.  This is faster
    also.  The mse isn't really 0, but calculating the actual chroma mse
    doesn't change result.
    Change-Id: I6154e21ddcca30e51baf805684dace10459c3350
pickcdef.c 19.3 KB