This commit makes the rate distortion modeling run in the unit of
maximum transform block size. No compression/speed change observed.
It is for the use of later fast forward transform purpose.

Change-Id: Ibaaedb69c765e8d0c5d5012f0ec07f36fd9f68fd