Commit 417f58a6 authored by Timothy B. Terriberry's avatar Timothy B. Terriberry Committed by Thomas Davies

new_multisymbol: Fix some illegal CDF values.

The probability of several of the symbols used for extra bits for
category 6 tokens, which as far as I can tell are legal to code,
was set to zero, and in one case even negative.

Change-Id: If6bd539b53889d75f974d444d666ed031faf18e0
parent b2f32d81
......@@ -71,13 +71,13 @@ const aom_cdf_prob av1_cat6_cdf1[CDF_SIZE(16)] = { 9314, 15584, 19741, 22540,
32310, 32527, 32671, 32768 };
const aom_cdf_prob av1_cat6_cdf2[CDF_SIZE(16)] = { 29548, 31129, 31960, 32004,
32473, 32498, 32511, 32512,
32745, 32757, 32764, 32764,
32768, 32768, 32768, 32768 };
32745, 32757, 32763, 32764,
32765, 32766, 32767, 32768 };
const aom_cdf_prob av1_cat6_cdf3[CDF_SIZE(16)] = { 32006, 32258, 32510, 32512,
32638, 32639, 32640, 32640,
32766, 32767, 32768, 32768,
32768, 32768, 32768, 32768 };
const aom_cdf_prob av1_cat6_cdf4[CDF_SIZE(4)] = { 32513, 32641, 32769, 32768 };
32761, 32762, 32763, 32764,
32765, 32766, 32767, 32768 };
const aom_cdf_prob av1_cat6_cdf4[CDF_SIZE(4)] = { 32513, 32641, 32767, 32768 };
const aom_cdf_prob *av1_cat6_cdf[] = {
av1_cat6_cdf0, av1_cat6_cdf1, av1_cat6_cdf2, av1_cat6_cdf3, av1_cat6_cdf4
};
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment