Commit 2c7b94f6 authored by Deb Mukherjee's avatar Deb Mukherjee Committed by Gerrit Code Review

Merge "Prevent negative cost for highbitdepth"

parents 00a4b20f 431cdc33
......@@ -491,7 +491,9 @@ static void block_rd_txfm(int plane, int block, BLOCK_SIZE plane_bsize,
if (x->plane[plane].eobs[block]) {
int64_t dc_correct = coeff[0] * coeff[0] -
(coeff[0] - dqcoeff[0]) * (coeff[0] - dqcoeff[0]);
#if CONFIG_VP9_HIGHBITDEPTH
dc_correct >>= ((xd->bd - 8) * 2);
#endif
if (tx_size != TX_32X32)
dc_correct >>= 2;
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment