• Nathan E. Egge's avatar
    daala_tx: Unify the asym and ortho DST designs. · b2f82ebd
    Nathan E. Egge authored
    This patch refactors the DST transforms so that the orthonormal and
     asymmetric transforms are now nearly identical (up to multiplicaiton
     constants and an extra set of shifts).
    This means that the DST designs are now embeddable for every level
     and should address hardware concerns about gate area.
    In addition, minor changes were made to improve transform accuracy:
     - all of the transforms now have perfect reconstruction for those
        computations outside the rotations, i.e., all +/- butterfly steps
        are exactly invertible
     - two multiplication constants were reduced below < 1.0 (better for
        SIMD and gives slightly improved accuracy)
     - the averaging bias is removed which saves an extra addition for each
        of the averaging steps
    Additional averaging steps can be removed from the 8-point Type-IV DST
     giving a 68% reduction in MSE for the 32-point DCT, but has not been
     done in the event we use it in place of the 8-point Type-VII DST.
    master-daala_tx@2017-12-10T22:38:19.651Z ->
      PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
    0.0057 | -0.0210 | -0.1821 |   0.0085 | -0.0002 |  0.0147 |    -0.0674
    Change-Id: Ib124eebf6f2e4b3c51c078d4e8f229fc5ec26171
daala_tx_kernels.h 61.6 KB