Skip to content
Snippets Groups Projects
Commit 173774bb authored by Timothy B. Terriberry's avatar Timothy B. Terriberry Committed by Jean-Marc Valin
Browse files

Replace log2_frac in the delta offset calculation.

Adds a new bitexact_log2tan() function which is much simpler, and
 more accurate.
The new approximation has an RMS error of 0.0038 bits from the
 correctly rounded result over the range of inputs we use, compared
 to an RMS error of 0.013 for the old log2_frac() method.
The actual computation of delta is also changed to use FRAC_MUL16,
 since this allows us to keep the full accuracy of the new method
 while avoiding 16-bit overflow.
The old delta computation actually could overflow 16 bits: it needed
 8 for the log2_frac() result, 1 for the sign of the difference, and
 8 more for N.
parent 2b13401f
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment