Commit c11c1c99 authored by Urvang Joshi's avatar Urvang Joshi

TX_64X64: Reduce cos bits further.

This is to keep the high range within 32 bits.

AV1FwdTxfm2d.CfgTest passes after this fix.

Change-Id: I2df463c4ec9260c544d68aad445b60cabe2b531b
parent 763ccd8c
......@@ -117,7 +117,7 @@ static const int8_t fwd_stage_range_idx_64[1] = { 0 };
// cos bit
static const int8_t fwd_cos_bit_col_dct_64[12] = { 13, 13, 13, 13, 13, 13,
13, 13, 13, 13, 13, 13 };
static const int8_t fwd_cos_bit_row_dct_64[12] = { 13, 13, 13, 12, 11, 10,
static const int8_t fwd_cos_bit_row_dct_64[12] = { 13, 13, 12, 11, 10, 10,
10, 10, 10, 10, 10, 10 };
// ---------------- row config fwd_dct_4 ----------------
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment