assert(enc->streams == NULL) would fail because pad_samples was missing
a fractional number of samples due to the resampler output latency not
being an integer.