Skip to content
  • Timothy B. Terriberry's avatar
    Enhancements the fixed-point approximations of non-linear functions. · a9ffc14a
    Timothy B. Terriberry authored and Jean-Marc Valin's avatar Jean-Marc Valin committed
    Accuracy for rsqrt, rcp, cos, and log2 is now at the level of truncation error
     for the current output resolution of these functions.
    sqrt and exp2 still have non-trivial algebraic error, but this cannot be
     reduced much further using the current method without additional computation.
    Also updates the fast float approximations for log2 and exp2 with coefficients
     that give slightly lower maximum relative error.
    
    Patch modified by Jean-Marc Valin to leave the cos approximation as is and
    leave the check for x<-15 in exp2 as is.
    a9ffc14a