Commit 358abfb7 authored by Monty Montgomery's avatar Monty Montgomery Committed by Christopher Montgomery

Move Daala TX to fixed coeff depth of 12 (Q4)

This patch activates all the preceeding work, moving Daala TX to a
greater, fixed coefficient depth (12).  This reclaims the regression
caused by going to Q3.

subset-1:
monty-rest-of-stack-rmscale-s1@2017-11-13T14:40:20.646Z ->
 monty-rest-of-stack-Q4-s1@2017-11-13T14:40:44.807Z

   PSNR | PSNR Cb | PSNR Cr | PSNR HVS |    SSIM | MS SSIM | CIEDE 2000
-0.0745 |     N/A | -0.1040 |  -0.1017 | -0.0660 | -0.0522 |    -0.0806

Change-Id: If2a0853b320d57c2fa3a66f919ceb2dc526d017f
parent 27d1b373
......@@ -30,7 +30,7 @@ static INLINE tran_high_t dct_const_round_shift(tran_high_t input) {
static INLINE tran_high_t check_range(tran_high_t input, int bd) {
#if CONFIG_DAALA_TX
// Daala TX coeffs cover a different range from AV1 TX
// all depths: 18 bit integer
// all depths: 19 bit integer
const int32_t int_max = (1 << (TX_COEFF_DEPTH + 6)) - 1;
const int32_t int_min = -int_max - 1;
(void)bd;
......
......@@ -69,7 +69,7 @@ extern "C" {
such that the real coefficient depth precision is always TX_CEOFF_DEPTH
regardless of bitdepth or transform size.
*/
# define TX_COEFF_DEPTH (11)
# define TX_COEFF_DEPTH (12)
# define OD_DISABLE_CFL (1)
# define OD_DISABLE_FILTER (1)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment