• Yushin Cho's avatar
    Fix daala-dist for var-tx · 0474912c
    Yushin Cho authored
    The var-tx has its own suite of tx size/type RD search functions,
    which recursively split the partition into square tx blocks.
    
    The Daala-dist requires access to 8x8 pixels (both decoded and predicted)
    since it measures the distortion for multiple of a 8x8 pixels.
    Thus, if tx block is smaller than 8x8, it waits until all of sub8x8 blocks
    are RD searched (with MSE) then replaces the MSE of 8x8 pixels with
    daala-dist's calculated distortion for 8x8 pixels.
    
    It is also applied to luma pixels only.
    
    Change-Id: Ic4891e89b4ef05cf880aa26781d2d06ccf3142de
    0474912c
rdopt.c 434 KB