The Opus RFC doesn't really say what to do beyond rejecting
a particular packet, but having the reference reinforces that
we're trying to leverage the same constraints in the specific
context of ogg encapsulation, and this isn't a new rule.