fdct4x4_test: fix unsigned overflow

The difference between src and dst will be signed, the error will be
unsigned. The change quiets -fsanitize=integer:
    unsigned integer overflow: 4294967295 * 4294967295

Change-Id: Ide662f26fae682309a3474243f8a5b5ccea2049e