Commit 5fb95f7d authored by Urvang Joshi's avatar Urvang Joshi

Palette: Handle case of too few unique centroids.

Even though number of unique colors in a block may be at least
PALETTE_MIN_SIZE, sometimes the K-means can generate fewer than
PALETTE_MIN_SIZE unique centroids due to integer rounding.

This was resulting in "k - PALETTE_MIN_SIZE" being negative and using
that as an index into palette_y_size_cost array was causing a UBSan
error.

BUG=aomedia:515

Change-Id: Ia00b4bb2efebbe43d1fec7b619ebb837b1ae961c
parent 9395b34e
......@@ -2501,6 +2501,11 @@ static int rd_pick_palette_intra_sby(const AV1_COMP *const cpi, MACROBLOCK *x,
centroids[i] = lb + (2 * i + 1) * (ub - lb) / n / 2;
av1_k_means(data, centroids, color_map, rows * cols, n, 1, max_itr);
k = av1_remove_duplicates(centroids, n);
if (k < PALETTE_MIN_SIZE) {
// Too few unique colors to create a palette. And DC_PRED will work well
// for that case anyway. So skip.
continue;
}
#if CONFIG_HIGHBITDEPTH
if (cpi->common.use_highbitdepth)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment