Commit 6eb35eb4 authored by Urvang Joshi's avatar Urvang Joshi Committed by Sebastien Alaiwan
Browse files

optimize_b_greedy: Avoid left shift of negative value.

accu_error was always <= 0:
- It was initialized to zero, and
- Became more and more negative through the iterations.

This meant that RDOPT() macro usage on accu_error was causing left shift
of negative value.

- accu_error is initialized to a large positive value.
- It becomes smaller on every iteration but remains >= always.

This is functionally equivalent: verified that there's no change in
output due to this patch.


Change-Id: I654572a72234c588c8f332d5b66a5587fc6610b4
parent 6c20c78f
......@@ -218,7 +218,9 @@ static int optimize_b_greedy(const AV1_COMMON *cm, MACROBLOCK *mb, int plane,
const int ctx0 = ctx;
/* Record the r-d cost */
int64_t accu_rate = 0;
int64_t accu_error = 0;
// Initialized to the worst possible error for the largest transform size.
// This ensures that it never goes negative.
int64_t accu_error = ((int64_t)1) << 50;
rate0 = get_token_bit_costs(*(token_costs_ptr + band_translate[0]), 0, ctx0,
......@@ -413,6 +415,7 @@ static int optimize_b_greedy(const AV1_COMMON *cm, MACROBLOCK *mb, int plane,
token_cache[rc] = av1_pt_energy_class[t0];
assert(accu_error >= 0);
x_prev = qcoeff[rc];
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment