Fix ioc in trellis optimization with hbd

Use int64_t type for distortion. This avoids integer overflow
issues in the trellis optimization function in high bit-depth
settings.

Change-Id: I550c3ca9f11a3191ef8638a152887018cd476141