Commit 8fdcf6e3 authored by Nathan E. Egge's avatar Nathan E. Egge Committed by Nathan Egge

Fix NaN failure in CfL unit test on x86.

BUG=aomedia:993

Change-Id: I555f430541413a42e9e14310bfde93304dc15cfa
parent 03a32926
......@@ -230,8 +230,7 @@ static void cfl_subtract_averages(CFL_CTX *cfl, TX_SIZE tx_size) {
}
int avg_q3 = (sum_q3 + (1 << (num_pel_log2 - 1))) >> num_pel_log2;
// Loss is never more than 1/2 (in Q3)
assert(fabs((double)avg_q3 - (sum_q3 / ((double)(1 << num_pel_log2)))) <=
0.5);
assert(abs((avg_q3 << num_pel_log2) - sum_q3) <= 1 << num_pel_log2 >> 1);
tx_pred_buf_q3 = pred_buf_q3;
for (int t_j = 0; t_j < tx_height; t_j++) {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment