Skip to content
  • Monty Montgomery's avatar
    Move all of LBD Daala TX to up-4, down-1 shift · 5500ce76
    Monty Montgomery authored
    Now that tran_low_t is assumed to be 32 bit when Daala TX is active,
    there's no reason for multi-stage shifting to fit coefficients into 16
    bits for the inter-tranform transpose matrix. Go to a consistent up by
    four, down by one shifting scheme for all TX block sizes.
    
    (Note this is for the current AV1 coefficient scaling scheme with
    av1_get_tx_scale and deeper coefficients for higher bitdepth input.
    Daala TX is moving to the long-intended constant-coefficient-depth in
    upcoming patches).
    
    subset 1:
    monty-4-1-baseline-s1@2017-11-11T05:57:15.857Z ->
     monty-4-1-test-s1@2017-11-11T05:57:52.983Z
    
       PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
    -0.0117 | -0.0246 |  0.0530 |   0.0238 | 0.0254 |  0.0447 |    -0.0442
    
    Change-Id: I2214e94ac822542c504d472276723277ed350abf
    5500ce76