In handle_inter_mode, the reference frames are set in refs buffer.
One can use refs buffer directly to avoid redundant fetch.

Change-Id: I811d408cae52dcd5e053dd4bfe69550eb6a2ff56