Commit e47d9f1d authored by James Zern's avatar James Zern

rd_pick_inter_mode: prevent signed integer overflow

calculate the txfm_cache difference first as both values may be
INT64_MAX with the intent that they cancel each other out.

Change-Id: I214d072458e1b24f60289974e6302af1aff7b66c
parent f2541f8a
......@@ -4090,7 +4090,9 @@ static void rd_pick_inter_mode(VP9_COMP *cpi, MACROBLOCK *x,
for (i = 0; i < NB_TXFM_MODES; i++) {
int64_t adj_rd;
if (this_mode != B_PRED) {
adj_rd = this_rd + txfm_cache[i] - txfm_cache[cm->txfm_mode];
const int64_t txfm_mode_diff =
txfm_cache[i] - txfm_cache[cm->txfm_mode];
adj_rd = this_rd + txfm_mode_diff;
} else {
adj_rd = this_rd;
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment