Commit acaa384f authored by Yunqing Wang's avatar Yunqing Wang
Browse files

Port an integer overflow fix in mcomp.c

Ported a bug fix from VP9.
(https://chromium-review.googlesource.com/c/473566/)

Change-Id: I5669c3878b34a5fda4ce1aabb8c653ea823d5e42
parent 89912f9a
......@@ -81,10 +81,8 @@ static int mv_err_cost(const MV *mv, const MV *ref, const int *mvjcost,
int *mvcost[2], int error_per_bit) {
if (mvcost) {
const MV diff = { mv->row - ref->row, mv->col - ref->col };
// This product sits at a 32-bit ceiling right now and any additional
// accuracy in either bit cost or error cost will cause it to overflow.
return ROUND_POWER_OF_TWO(
(unsigned)mv_cost(&diff, mvjcost, mvcost) * error_per_bit,
return (int)ROUND_POWER_OF_TWO_64(
(int64_t)mv_cost(&diff, mvjcost, mvcost) * error_per_bit,
RDDIV_BITS + AV1_PROB_COST_SHIFT - RD_EPB_SHIFT +
PIXEL_TRANSFORM_ERROR_SCALE);
}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment