Fixes stack overflow for some custom modes

This only affects custom modes (builds with --enable-custom-modes) with frame
sizes 2.5, 5, 10, and 20ms and sampling rates below 40 kHz. The problem does
not affect normal use of Opus (using OpusEncoder/OpusDecoder) even when built
with custom modes enabled, but only special applications that use
OpusCustomEncoder/OpusCustomDecoder.