libtheora: theora_clear crashes after encoding certain image sizes
Hi,
the attached test-program (which encodes one frame into an ogg stream) crashes.
Versions: libtheora-1.0 (tarball from xiph.org), libogg-1.1.3-2 (and 1.1.3-4) of Debian Stable
Using gdb, the crash is in the cleanup at the end, in:
EClearFrameInfo (enc/encoder_toplevel.c), line: _ogg_free(cpi->yuv0ptr);
The function was called by theora_encode_clear (enc/encoder_toplevel.c), which in turn was called by theora_clear (dec/apiwrapper.c), line: (*((oc_state_dispatch_vtbl *)_th->internal_encode)->clear)(_th); (theora_clear was called for the first time in the program.)
When the pointer pointer is attempted to be freed, it has the same value as after allocation in EInitFrameInfo (enc/encoder_toplevel.c):
cpi->yuv0ptr after malloc: (gdb) print cpi->yuv0ptr $1 = (YUV_BUFFER_ENTRY *) 0xb5fdc008 ""
cpi->yuv0ptr before attempt to free: (gdb) print cpi->yuv0ptr $2 = (YUV_BUFFER_ENTRY *) 0xb5fdc008 '\200' <repeats 200 times>... (gdb) c Continuing. *** glibc detected *** free(): invalid pointer: 0xb5fdc008 ***
(The same crash happens with stepping a single code-line further, I used "c" for continuing only in this case.)
Setting image width and height to 256 instead of 1024, the crash is at the same point in the program, and the value of the pointer is also unchanged with respect to after allocation, only now there is just a segfault, without a message from glibc. Setting image width and height to 64, 16, or 4, the pointer (non-null) is freed in EClearFrameInfo without a crash.
I did not find where this pointer had been freed before in libtheora. The memory must have been freed over a pointer not named "yuv0ptr", possibly resulting from some (wrong) pointer arithmetic...
BTW if "theora_clear" is not called, apparently normal video files (image width and height 1024) are produced.