Properly determine and use horizontal and vertical masks
for loop filtering when rectangular transforms are used.

Fixes an intermittent mismatch issue and improves coding
efficiency.

BDRATE results for ext-tx + rect-tx:
lowres: -3.739% (up from -3.443%)
midres: -3.366% (up from -3.006%)

Change-Id: If26fa14261f3893662eb1245f0b876d68513247c