The txk-sel allows each transform block to select its own
transform kernel. Such locality enables one to store the selected
RD cost including tx_type selection per transform block size.
It reduces the needed hash map size to 1 / 16 of what is needed
without txk-sel.

This commit re-works the hash map RD cost fetch for txk-sel. Tested
on red_kayak_480p in speed 1, enabling txk-sel makes the encoding
speed 12% faster than the baseline without txk-sel on. Further
enabling reduced hash map size gains speed 1 another 10%.

Change-Id: I4a5d99d27e2a76b10e76c00a8178f692c95fdf13