Simplify implementation of rd_test_partition3

The rd_test_partition3 function basically calls rd_pick_sb_modes on
each of three sub-blocks of a partition (this is one of the mixed size
partitions like PARTITION_HORZ_A) and adds up the cost. There's also
some extra book-keeping to do with stopping early if we know the
partition is more expensive than what we've already seen.

Before this patch, each call and its supporting code was written out
in rd_test_partition3 with lots of repetition (made even more
unreadable because of the #if/#endif blocks). This patch moves the
"per-subblock" logic into a new function, rd_try_subblock, and
replaces the code with calls to that function.

The patch also rewrites the nested conditionals

  if (A) { if (B) { if (C) { X; } } } return;

as

  if (! A) return; if (! B) return; if (! C) return; X;

which means the code for X doesn't need to be so indented.

The patch also uses the new function to replace the bodies of the
PARTITION_HORZ_4 and PARTITION_VERT_4 partition types.

Note that lots of the remaining ugliness is to do with supporting
CONFIG_SUPERTX with ext-partition types. These explicitly aren't
supported together at the moment, so another option would be to rip
out that code entirely.

Change-Id: I9af40d96aa1384f24a088a73a711311638490250