Variable frame size fixes (still not exposed in the API)

This fixes an actual error in the downmix (using the float version even
for the int API), as well as a bunch of conversion warnings.