I found four heap buffer overflow vuls in function mapping0_forward() of libvorbis-1.3.6 by fuzzing libtheora, one of the crash sample behaves as follows, others behave similar:

Program received signal SIGABRT, Aborted.
0x00007ffff6fdfb55 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff6fdfb55 in raise () from /lib64/libc.so.6
#1  0x00007ffff6fe1131 in abort () from /lib64/libc.so.6
#2  0x0000000000520a0b in __sanitizer::Abort () at /home/jiangxin/hunter-tool/llvm5/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:146
#3  0x000000000051eb3a in __sanitizer::Die () at /home/jiangxin/hunter-tool/llvm5/projects/compiler-rt/lib/sanitizer_common/sanitizer_termination.cc:59
#4  0x00000000005051a5 in ~ScopedInErrorReport (this=<optimized out>, __in_chrg=<optimized out>) at /home/jiangxin/hunter-tool/llvm5/projects/compiler-rt/lib/asan/asan_report.cc:225
#5  __asan::ReportGenericError (pc=<optimized out>, bp=bp@entry=140737488324304, sp=sp@entry=140737488324296, addr=<optimized out>, is_write=is_write@entry=false, access_size=access_size@entry=4, exp=<optimized out>, exp@entry=0, fatal=<optimized out>, fatal@entry=true) at /home/jiangxin/hunter-tool/llvm5/projects/compiler-rt/lib/asan/asan_report.cc:420
#6  0x0000000000505c03 in __asan::__asan_report_load4 (addr=<optimized out>) at /home/jiangxin/hunter-tool/llvm5/projects/compiler-rt/lib/asan/asan_rtl.cc:133
#7  0x000000000069fac3 in mapping0_forward (vb=0x7fffffffd330) at mapping0.c:370
#8  0x00000000006244af in vorbis_analysis (vb=0x7fffffffd330, op=0x0) at analysis.c:46
#9  0x000000000052f14e in fetch_and_process_audio (audio=0x616000000380, audiopage=0x7fffffffd5e0, vo=0x7fffffffceb0, vd=0x7fffffffd260, vb=0x7fffffffd330, audioflag=0) at encoder_example.c:996
#10 0x0000000000536546 in main (argc=5, argv=0x7fffffffdfe8) at encoder_example.c:1754
(gdb) 

These vuls are because of lacking of var “channels” check”, and here is the details of these four vuls in mapping0.c: vul 1: line 370 of mapping0.c

    366     for(i=0;i<vi->channels;i++){
    367       /* the encoder setup assumes that all the modes used by any
    368          specific bitrate tweaking use the same floor */
    369 
    370       int submap=info->chmuxlist[i];//int array[256] oob read, vi->channels need check

vul 2: line 614 of mapping0.c

    612       /* encode floor, compute masking curve, sep out residue */
    613       for(i=0;i<vi->channels;i++){
    614         int submap=info->chmuxlist[i];//int array[256] oob read, vi->channels need check
    615         int *ilogmask=iwork[i];

vul 3 and 4: line 666 and 678 of mapping0.c

    665         for(j=0;j<vi->channels;j++){
    666           if(info->chmuxlist[j]==i){//int array[256] oob write, vi->channels need check
    667             zerobundle[ch_in_bundle]=0;
    668             if(nonzero[j])zerobundle[ch_in_bundle]=1;
    669             couple_bundle[ch_in_bundle++]=iwork[j];
    670           }
    671         }
    672 
    673         classifications=_residue_P[ci->residue_type[resnum]]->
    674           class(vb,b->residue[resnum],couple_bundle,zerobundle,ch_in_bundle);
    675 
    676         ch_in_bundle=0;
    677         for(j=0;j<vi->channels;j++)
    678           if(info->chmuxlist[j]==i)//int array[256] oob write, vi->channels need check
    679             couple_bundle[ch_in_bundle++]=iwork[j];
    680 
    681         _residue_P[ci->residue_type[resnum]]->
    682           forward(opb,vb,b->residue[resnum],
    683                   couple_bundle,zerobundle,ch_in_bundle,classifications,i);

Note: I need compile libtheora and libvorbis by clang asan and use encoder_example of libtheora to reproduce this vul.

The cmdline to reproduce this vul is like this : ./encoder_example crash_sample xxx.y4m -o out.ogv

The binary encoder_example belongs to libtheora.

recommanded patch : adding on line 238 of mapping0.c

    230 static int mapping0_forward(vorbis_block *vb){
    231   vorbis_dsp_state      *vd=vb->vd;
    232   vorbis_info           *vi=vd->vi;
    233   codec_setup_info      *ci=vi->codec_setup;
    234   private_state         *b=vb->vd->backend_state;
    235   vorbis_block_internal *vbi=(vorbis_block_internal *)vb->internal;
    236   int                    n=vb->pcmend;
    237   int i,j,k;
    238   if(vi->channels > MAX_CHANNEL || vi->channels < 0) return -1;//recommanded patch for these vuls