Return invalid rate (previously only invalid rdcost) if the
mode combination to check is < 8x8 tx_size + filter_intra mode.

BUG=aomedia:1006

Change-Id: If90f431c7692473c88ac7a644bfa969a1acb3573