Use small transform size in non-rd real-time mode

In non-rd real-time mode, choosing smaller transform size in
encoding gives better video quality and good speed gain than
choosing larger transform size. This patch set tx size search
method to ALLOW_8X8, which is better than using 4x4 or other
larger sizes.

Borg tests on rtc set at speed 6 showed significant gain on quality.
PSNR gain: 11.034% and SSIM gain: 15.466%.

The speed gain is 5% - 12% for <720p clips, and 2% - 7% for
720p clips.

Change-Id: If4dc74ed2df359346b059f47fb73b4a0193ec548