• Erik de Castro Lopo's avatar
    stream_encoder.c : Fix an arithmetic overflow in the RICE2 partitioner. · 6f7ec60c
    Erik de Castro Lopo authored
    For a specific 24 bit WAV file provided by Leigh Dyer
    
        http://lists.xiph.org/pipermail/flac-dev/2013-July/004284.html
    
    encoding with compression level 7 was generating a file a couple of
    orders of magintude larger than the original.
    
    Debugging showed that variable abs_residual_partition_sum (a FLAC__uint32)
    in function precompute_partition_info_sums_() was suffering from an
    arithmetic overflowing on some 24 bit input files although this value
    overflowing did not always cause larger output files.
    
    Since the value abs_residual_partition_sum is eventually stored in an
    array of FLAC__uint64, it makes sense to make abs_residual_partition_sum
    a FLAC__uint64 anyway.
    
    Debugging this problem was made easier by use of the Clang compiler's
    -fsanitize=integer option.
    6f7ec60c
stream_encoder.c 170 KB