• Guillaume Martres's avatar
    PVQ: Fix incorrect calculation of rd_stats · 930118c5
    Guillaume Martres authored
    When PVQ is on, we reencode at the end of choose_tx_size_type_from_rd to
    get the entropy contexts right, previously this was done using
    txfm_rd_in_plane but this is different from the encodes done in the loop
    which use txfm_yrd, the result is that rd_stats is set incorrectly at
    the end of choose_tx_size_type_from_rd when PVQ is on.
    
    Results on objective-1-fast with --limit=5:
    
       PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
    -0.5803 | -1.0598 | -1.4565 |  -0.3377 | -0.8153 | -0.5934 |    -0.9943
    
    See https://goo.gl/Hvv0E2
    
    Change-Id: Iccc7b0afaff849f959a0084eb48dbb838bc3cb1a
    930118c5
rdopt.c 416 KB