Resolve conflict between var-tx and super-tx

This commit aligns the rate-distortion metric for the recursive
transform block partitioning and the super transform. It resolves
the conflicts between these two experiments. The coding performance
gains of the combined experiments (var-tx + super-tx) has been
improved:

derf   0.89%  ->  1.9%
hevcmr 1.06%  ->  1.8%
stdhd  0.29%  ->  1.4%
hevchr 0.80%  ->  2.3%

Change-Id: I7e33994ad70c1b2751435620815f867d82172f41