Replace an assertion with a proper error on bad bitstream

The example in bug 712 is a bitstream that signals a global motion
type of ROTZOOM, but its second frame has shear parameters that fail
the is_affine_shear_allowed check at warped_motion.c:754. This is
quite possible (and it's not obvious how to change the bitstream
format so that you can't signal something like this).

This patch replaces the failing assertion with a proper "no you
can't!" error.

BUG=aomedia:712

Change-Id: I6a32632d17031b777acd2f78a887491a40177785