Skip to content
Snippets Groups Projects
Commit ef1b74c7 authored by Steinar Midtskogen's avatar Steinar Midtskogen
Browse files

Make the CDEF RDO handle 4:2:2 properly

This fixes an assert:

av1/common/cdef_block.c:561: cdef_filter_fb: Assertion `bsize ==
BLOCK_8X8 || bsize == BLOCK_4X4' failed.

The RDO simply assigned a strength of 0 in the 4:2:2 case and called
cdef_filter_fb(), but cdef_filter_fb() will complain about 4:2:2 even
if the strength is 0.

The fix assigns a chroma mse of 0 when the the subsampling is
different for x and y rather than to call the filter.  This is faster
also.  The mse isn't really 0, but calculating the actual chroma mse
doesn't change result.

BUG=aomedia:881

Change-Id: I6154e21ddcca30e51baf805684dace10459c3350
parent 146a060a
No related branches found
No related tags found
2 merge requests!6Rav1e 11 yushin 1,!3Rav1e 10 yushin
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment