Commit 5b69b199 authored by Nathan E. Egge's avatar Nathan E. Egge Committed by Tim Terriberry

daala_tx: New flattened 2-point Type-IV asym DST.

This change improves the accuracy of the 4-point Type-II DCT:

Old MSE: 6.2711279572488185887270981198199e-08
New MSE: 6.0281623825882593130347914239103e-08

It also reduces a multiplication constant so it is less than 1.

subset-1:

daala_tx@2017-12-04T01:58:11.321Z -> new_dst2a@2017-12-04T01:59:12.884Z

  PSNR | PSNR Cb | PSNR Cr | PSNR HVS |   SSIM | MS SSIM | CIEDE 2000
0.0274 |  0.0255 |  0.0969 |  -0.0024 | 0.0274 |  0.0027 |     0.0110

Change-Id: I7c9d389af8e98cb39f3bc5923134b5dfe174ba0a
parent 695ebacf
......@@ -161,15 +161,15 @@
do { \
int t_; \
int u_; \
t_ = p0h - p1; \
/* 3135/8192 ~= (Cos[3*Pi/8] - Sin[3*Pi/8])/Sqrt[2] = 0.38268343236509 */ \
u_ = (p0*3135 + 4096) >> 13; \
/* 60547/32768 ~= (Cos[3*Pi/8] + Sin[3*Pi/8])*Sqrt[2] = 1.847759065023 */ \
p0 = (p1*60547 + 16384) >> 15; \
t_ = p0h + p1; \
/* 15137/16384 ~= (Cos[3*Pi/8] + Sin[3*Pi/8])/Sqrt[2] = 0.9238795325112867 */ \
u_ = (p0*15137 + 8192) >> 14; \
/* 3135/4096 ~= (Cos[3*Pi/8] - Sin[3*Pi/8])*Sqrt[2] = 0.7653668647301795 */ \
p0 = (p1*3135 + 2048) >> 12; \
/* 8867/16384 ~= Cos[3*Pi/8]*Sqrt[2] = 0.5411961001461971 */ \
t_ = (t_*8867 + 8192) >> 14; \
p0 += t_; \
p1 = u_ + t_; \
p1 = u_ - t_; \
} \
while (0)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment