Transforms with one of the dimensions being 64 use an approximate 2D DCT
transform, which makes round trips errors large for randomized input.

So, for such transform sizes, we use a modified test that compares (a)
reference forward transform + inverse transform to (b) forward transform
+ inverse transform.

BUG=aomedia:1114

Change-Id: Ic47b0037a0883add4c9f84ac16e03d75b258d5ae