Infinite loop in 2-pass theora encoding
Downstream bug report w/ sample file & ffmpeg2theora command line: https://phabricator.wikimedia.org/T115190
Per derf on irc this looked like a bug with the calculation of frames_needed in oc_enc_rc_2pass_in leading to broken assumptions in that loop... I tracked it down to a bug in the clamping, where some values were unsigned and some signed. When the window size was slightly larger than the buf_delay setting (which should be due to dupe frames or such) this ended up throwing a monkey wrench in by picking the remaining frame count as the max clamped value instead of the expected negative value.
Fix that seems to work is to cast the frames_left vals to (int) so we don't have mixed signedness: https://github.com/brion/theora/commit/5d05cde8e4859d1056ab10312574843b9e6b1d8e
Attaching also as a patch here.