Not all the failure paths cleaned up after themselves properly.
Thanks to tangobravo for complaining about similar issues in
 libvorbisfile, which prompted me to audit this code.