CVE-2017-14633: Don't allow for more than 256 channels

Otherwise

 for(i=0;i<vi->channels;i++){
      /* the encoder setup assumes that all the modes used by any
         specific bitrate tweaking use the same floor */
      int submap=info->chmuxlist[i];

overreads later in mapping0_forward since chmuxlist is a fixed array of
256 elements max.