Add paranoid checking for the validity of the encoder and the decoder

state before using it. Handle malloc failures for the encoder and
decoder setup gracefully and without leaks.