Fixes some cases where MIN/MAX macros result in duplicated function calls

Also enforces an upper bound of 510 kb/s even for frames that are
smaller than 20 ms. This reduces waste for high bitrate VBR.