With ext-partition and tx64x64 enabled, the encoder could choose
TX_64X64 to encode the transform for a subsampled plane of a
BLOCK_128X128 block. This broke an assumption in the nested loop in
write_tokens_b and also caused bug 827 (with a rather cryptic
failure).

This patch changes get_vartx_max_txsize to ensure that the encoder and
decoder don't think they can use TX_64X64 in this situation. It also
adds a couple of assertions to the loop mentioned above so that if
something comes unstuck it'll be much more obvious what went wrong.

BUG=aomedia:827

Change-Id: Ie093f2f20f6242949d68e950c8f95b100867ee17