- changed status to invalid
Power of two operation gives incorrect result
Issue #764
invalid
Windows 7 64 bit I came across into incorrect result when trying calculate power of two with exponent 64 that's get me to examine with excel to see if it gave the same result. With surprise I found out it's completely incorrect starting with exponent 50 so on. The same result I get with windows 7 64 bit calculator.
Comments (1)
-
- Log in to comment
Speedcrunch is right, and Excel is wrong.
Excel uses double precision floating point, which cannot represent these numbers exactly. It basically rounds any results to ~16 significant digits.