Commit 01b01c0d authored by Yue Chen's avatar Yue Chen
Browse files

Make the first cdf prob < CDF_PROB_TOP to avoid infinite loop

The probability training tool generates all CDF_PROB_TOP table if
the counts are all zero. Although it means such cases are never
used in actual encoding, in rd cost pre-calculation, we still
sometimes run into such cases and the token cost calcualtion
function cannot handle all-CDF_PROB_TOP cdfs (Will trigger assertion
when debug mode is on).

Change-Id: I259620e210fea0f6e1f7848724fa006fa06da226
parent b433f4c4
......@@ -212,6 +212,7 @@ static int counts_to_cdf(const aom_count_type *counts, aom_cdf_prob *cdf,
else
cdf[i] = (csum[i] * CDF_PROB_TOP + round_shift) / sum;
}
if (sum <= 0) cdf[0] = CDF_PROB_TOP - 1;
return 0;
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment