sharedbook: fix signed overflow and division by 0

Fixes ubsan (signed integer multiplication overflow) and address sanitizer (division by 0) issues found by the same testcase from: Issue 19619: tremor:decode_fuzzer: Floating-point-exception in _book_unquantize

See: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=19619

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/tremor