Uncontrolled alloca() in oggenc which may lead to a remote code execution in 32-bit environment

During a fuzz testing, I found a program-crashing bug in the latest version of oggenc. When a malicious AIFF audio file is provided as an input, segmentation fault or remote code execution may occur.

I downloaded http://downloads.xiph.org/releases/vorbis/vorbis-tools-1.4.0.tar.gz file, and compiled it with clang 3.8.

In aiff_open() function of oggenc/audio.c file, size argument of alloca() call is not checked tightly, and therefore a large size of memory can be requested.

    if(!find_aiff_chunk(in, "COMM", &len))
    {   
        fprintf(stderr, _("Warning: No common chunk found in AIFF file\n"));
        return 0; /* EOF before COMM chunk */
    }

    if(len < 18)
    {   
        fprintf(stderr, _("Warning: Truncated common chunk in AIFF header\n"));
        return 0; /* Weird common chunk */
    }

    buffer = alloca(len);

    if(fread(buffer,1,len,in) < len)
    {
        fprintf(stderr, _("Warning: Unexpected EOF in reading AIFF header\n"));
        return 0;
    }

In 64-bit environment, this will simply make the program to crash, but in 32-bit environment this bug can lead to a remote code execution. If a malicious attacker requests a large size of memory (e.g. alloca(0xffffff00)), this will lift up the stack pointer (%esp register) instead of correctly allocating a stack buffer. Then, the subsequent fread() call will overwrite the stack and corrupt the saved return address.

I attach the PoC input file to reproduce this bug.

poc

jason@debian-stretch:~/ground/vorbis-tools-1.4.0$ gdb oggenc/oggenc -q
Reading symbols from oggenc/oggenc...done.
(gdb) run ~/poc
Starting program: /home/jason/ground/vorbis-tools-1.4.0/oggenc/oggenc ~/poc
Warning: Unexpected EOF in reading AIFF header

Program received signal SIGSEGV, Segmentation fault.
0x41414141 in ?? ()
(gdb) info reg $eip
eip            0x41414141       0x41414141