Commit b2f82ebd authored by Committed by Nathan EggeBrowse files
daala_tx: Unify the asym and ortho DST designs.
This patch refactors the DST transforms so that the orthonormal and asymmetric transforms are now nearly identical (up to multiplicaiton constants and an extra set of shifts). This means that the DST designs are now embeddable for every level and should address hardware concerns about gate area. In addition, minor changes were made to improve transform accuracy: - all of the transforms now have perfect reconstruction for those computations outside the rotations, i.e., all +/- butterfly steps are exactly invertible - two multiplication constants were reduced below < 1.0 (better for SIMD and gives slightly improved accuracy) - the averaging bias is removed which saves an extra addition for each of the averaging steps Additional averaging steps can be removed from the 8-point Type-IV DST giving a 68% reduction in MSE for the 32-point DCT, but has not been done in the event we use it in place of the 8-point Type-VII DST. subset-1: master-daala_tx@2017-12-10T22:38:19.651Z -> new-daala_tx@2017-12-10T22:37:50.844Z PSNR | PSNR Cb | PSNR Cr | PSNR HVS | SSIM | MS SSIM | CIEDE 2000 0.0057 | -0.0210 | -0.1821 | 0.0085 | -0.0002 | 0.0147 | -0.0674 Change-Id: Ib124eebf6f2e4b3c51c078d4e8f229fc5ec26171