Use `add` for pointer arithmetic to avoid casting

`add` takes a `usize` parameter, so this method can be used to avoid an
unnecessary cast to `isize`.