This change makes the conversions similar to those in av1_quantize.c,
and fix ubsan warnings shown in nightly tests.

Change-Id: I90851a80dcb9f052a32bf22199fd9ef8ff927725