promote the unsigned int calculation to uint64_t rather than int64_t for
type consistency

cherry-picked from libvpx:
47d6f16a0 get_prob(): rationalize int types

Change-Id: Ic34dee1dc707d9faf6a3ae250bfe39b60bef3438