Codea's Lua number type
'Number' is one of Lua's eight basic types, the other seven being nil, boolean, string, function, userdata, thread, and table. Number represents real numbers.
By default, Lua's number type represents double-precision floating-point numbers. However, Codea's Lua interpreter uses another internal representation for numbers: single-precision float.
The IEEE 754 standard specifies a single-precision float as having: sign bit - 1 bit, exponent width - 8 bits, significand precision - 24 bits (23 bits are stored). That is, 32 bits (4 bytes) are stored in total.
This gives a precision of 6 to 9 significant decimal digits and a range for positive values of between about 1.4e−45 to about 3.4e+38.
Range of precise integers
After 16,777,216, integers may not be recorded precisely in a single precision float. For example:
function setup() i = 16777215 print(string.format("%d", i)) i = i + 1 print(string.format("%d", i)) -- Output: 16777216 i = i + 1 print(string.format("%d", i)) -- Output: 16777216 end
Some numbers that can be represented precisely in decimal cannot be represented precisely in binary. Examples are 0.1 and 0.01.
This can result in rounding errors. For example:
print(0.1 * 0.1 - 0.01) -- Outputs: 9.31323e-10 (not 0) print(0.1 * 0.1 == 0.01) -- Outputs: false
The logic of code may need to take into account the lack of precision. For example:
epsilon = 0.0001 -- Set tolerance for error print(math.abs(0.1 * 0.1 - 0.01) < epsilon) -- Outputs: true