With this patch, --enable-var-tx only enables recursive transform
partitioning without using rectangular transforms.
To enable use of rectangular transforms in addition, use:
--enable-var-tx --enable-rect-tx

The RD selection process is not fully tested under the var-tx flag
only. We might expect certain performance loss there.

Change-Id: Ie6aa17f1bbc3e8563b9990bc9ff79cc860d9a361