Precision-Rounding error

Issue #1040 invalid
Former user created an issue

Hi to whom it may concern:

Just wanted to let you know that your program is excellent.

Please however correct the rounding issue, see examples below:

Automatic precision (4.38 * 10^6) * (7.75 * 10^3) = 3.3945e10

2 Digit Precision: (4.38 * 10^6) * (7.75 * 10^3) = 3.39e10

Thanks,

@ri

Comments (4)

  1. Tey'

    Why do you believe there is a rounding issue here? 3.3945e10 is closer to 3.3900e10 than to 3.4000e10.

  2. Log in to comment