Commit 6f81f128 authored by Dominic Symes's avatar Dominic Symes Committed by Angie Chiang

[NORMATIVE-DECODING, txmg]: Adjust IADST4

Coefficients used for the VP9 version of IADST4 satisfy the
mathematical relation sinpi[1]+sinpi[2]==sinpi[4] which is
a trigonometric property useful for optimizing the four point
transform. Unfortunately the change in shift used in the latest
code means that rounding errors are introduced so that this identity
no longer holds. We think the identity should be restored by
rounding 4964.563 to 4964 rather than 4965. Similarly the identity
has been fixed for other bit shifts in the table where it does not
hold (even though these are not currently used I think).

This change also fixes a bug with the range checking for IADST4.

We see little change on the statistics from a local ACWY run
on objective-1-fast I-frame only:

         PSNR  PSNR-HVS SSIM  PSNR-Cr APSNR APSNR-Cr MS-SSIM PSRNR-YUV
Average  +0.01 +0.01    -0.02 -0.15   +0.01 -0.15    +0.01   +0.01

BUG=aomedia:1360

Change-Id: Icc09a1c59929e58bbc922d4b8b73de4a14104a8e
parent 7d8b13ed
......@@ -766,14 +766,16 @@ void av1_iadst4_new(const int32_t *input, int32_t *output, int8_t cos_bit,
return;
}
assert(sinpi[1] + sinpi[2] == sinpi[4]);
// stage 1
s0 = range_check_value(sinpi[1] * x0, stage_range[1]);
s1 = range_check_value(sinpi[2] * x0, stage_range[1]);
s2 = range_check_value(sinpi[3] * x1, stage_range[1]);
s3 = range_check_value(sinpi[4] * x2, stage_range[1]);
s4 = range_check_value(sinpi[1] * x2, stage_range[1]);
s5 = range_check_value(sinpi[2] * x3, stage_range[1]);
s6 = range_check_value(sinpi[4] * x3, stage_range[1]);
s0 = range_check_value(sinpi[1] * x0, stage_range[1] + bit);
s1 = range_check_value(sinpi[2] * x0, stage_range[1] + bit);
s2 = range_check_value(sinpi[3] * x1, stage_range[1] + bit);
s3 = range_check_value(sinpi[4] * x2, stage_range[1] + bit);
s4 = range_check_value(sinpi[1] * x2, stage_range[1] + bit);
s5 = range_check_value(sinpi[2] * x3, stage_range[1] + bit);
s6 = range_check_value(sinpi[4] * x3, stage_range[1] + bit);
s7 = clamp_value(x0 - x2, stage_range[1]);
// stage 2
......
......@@ -82,13 +82,13 @@ static const int32_t cospi_arr_data[7][64] = {
14359, 12785, 11204, 9616, 8022, 6424, 4821, 3216, 1608 }
};
// sinpi_arr_data[i][j] = (int)round((sqrt(2) * sin(kPi/9) * 2 / 3) * (1 <<
// (cos_bit_min + i)))
// sinpi_arr_data[i][j] = (int)round((sqrt(2) * sin(j*Pi/9) * 2 / 3) * (1 <<
// (cos_bit_min + i))) modified so that elements j=1,2 sum to element j=4.
static const int32_t sinpi_arr_data[7][5] = {
{ 0, 330, 621, 836, 951 }, { 0, 660, 1241, 1672, 1902 },
{ 0, 1321, 2482, 3344, 3803 }, { 0, 2642, 4965, 6689, 7606 },
{ 0, 5283, 9929, 13377, 15212 }, { 0, 10566, 19858, 26755, 30425 },
{ 0, 21133, 39717, 53510, 60849 }
{ 0, 330, 621, 836, 951 }, { 0, 660, 1241, 1672, 1901 },
{ 0, 1321, 2482, 3344, 3803 }, { 0, 2642, 4964, 6689, 7606 },
{ 0, 5283, 9929, 13377, 15212 }, { 0, 10566, 19858, 26755, 30424 },
{ 0, 21133, 39716, 53510, 60849 }
};
static INLINE const int32_t *cospi_arr(int n) {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment