Commit 3ebfe2a4 authored by Yushin Cho's avatar Yushin Cho

Fix fixed-pt PVQ compand/expand outputs zero gain

For fixed-point version of PVQ, which is current default,
added MAXI(1, ) to limit the minimum companded or expanded gain to be one.
Previously, gain compand/expand function, which is invoked when
activity masking is enabled, sometimes outputs zero
then triggered the assert(gain != 0).

Metric change from floating-pt to fixed-pt PVQ is:
PSNR  PSNR-HVS  SSIM  CIEDE-2000  PSNR Cb PSNR Cr MS-SSIM VMAF
0.02  0.10      0.08  0.11        0.01    0.02    0.13    -0.30

Change-Id: I64a60d1970d35a26af227841e4a5e50a89ddc44c
parent fc1598ad
......@@ -698,7 +698,8 @@ od_val32 od_gain_expand(od_val32 cg0, int q0, od_val16 beta) {
tmp = cg0*q0*(int64_t)irt;
/*Expanded gain must be in Q(OD_COMPAND_SHIFT), thus OD_COMPAND_SHIFT is
not included here.*/
return OD_VSHR_ROUND(tmp, OD_CGAIN_SHIFT + sqrt_outshift + sqrt_inshift);
return OD_MAXI(1,
OD_VSHR_ROUND(tmp, OD_CGAIN_SHIFT + sqrt_outshift + sqrt_inshift));
#endif
}
else {
......@@ -715,7 +716,7 @@ od_val32 od_gain_expand(od_val32 cg0, int q0, od_val16 beta) {
expr = od_pow(cg, beta);
/*Expanded gain must be in Q(OD_COMPAND_SHIFT), hence the subtraction by
OD_COMPAND_SHIFT.*/
return OD_SHR_ROUND(expr, OD_EXP2_OUTSHIFT - OD_COMPAND_SHIFT);
return OD_MAXI(1, OD_SHR_ROUND(expr, OD_EXP2_OUTSHIFT - OD_COMPAND_SHIFT));
#endif
}
}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment