As part of 9cf0c9cd the buffering was made
to better handle 4:2:2, but that causes regressions in the tests, so we're
backing out part of it for now.

Change-Id: I9ca4cfeb159aa65514613989e3dcbc30f86ec5b2