Do not shrink decoding buffer when parameters change

As files with a changing number of channels or bit depth are rare
and difficult the handle anyway, improve fuzzer performance with
them by limiting realloction of the decoding buffer