Skip to content
  • Timothy B. Terriberry's avatar
    Replace log2_frac in the delta offset calculation. · 173774bb
    Timothy B. Terriberry authored and Jean-Marc Valin's avatar Jean-Marc Valin committed
    Adds a new bitexact_log2tan() function which is much simpler, and
     more accurate.
    The new approximation has an RMS error of 0.0038 bits from the
     correctly rounded result over the range of inputs we use, compared
     to an RMS error of 0.013 for the old log2_frac() method.
    The actual computation of delta is also changed to use FRAC_MUL16,
     since this allows us to keep the full accuracy of the new method
     while avoiding 16-bit overflow.
    The old delta computation actually could overflow 16 bits: it needed
     8 for the log2_frac() result, 1 for the sign of the difference, and
     8 more for N.
    173774bb