When testing frame sse to choose a loop filter value and
when checking ambient error in kf Q selection, use 64 bit
values for accumulating the sse, to avoid risk of overflow
for large image formats.

Change-Id: I03765d16c843d0ade61a45b0cd46312472697e57