For 8-bit the subtrahend is small enough to fit into uint32_t.

cherry-picked from libvpx:
47b9a0912 Resolve -Wshorten-64-to-32 in highbd variance.

This is the same that was done for:
c0241664a Resolve -Wshorten-64-to-32 in variance.

For 10/12-bit apply:
63a37d16f Prevent negative variance

BUG=aomedia:445

Change-Id: Iab35e3f3f269035e17c711bd6cc01272c3137e1d