0.12: rounding errors in simple hex subtraction

[Former user]

0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFC2F-0x1ae168fea63dc339a3c58419466ceaeef7f632653266d0e1236431a950cfe52a = 0xE51E970159C23CC65C3A7BE6B99315110809CD9ACD992F1EDC9BCE55AF301704.FFFAACABE24B18A

when it should be

0xe51e970159c23cc65c3a7be6b99315110809cd9acd992f1edc9bce55af301705

I don't see why speedcrunch makes this rounding error - especially when it could perform a simple symbolic subtraction on the HEX data.

Arch Linux (x86 64 bit)

[Pol Welter]

SpeedCrunch internally casts all numbers to a floating point representation with 255 significant bits. Hence that is also the limit up to which integers may be represented exactly. One hex digit represents 4 bits, so any hex operation with 64 figures or more will be subject to rounding errors.

Still, the issue persists when I remove a few digits. Something we need to look into.

[Lammen]

Some technical explanations first: The engine in Speedcrunch is decimal based, which means it can represent 0.1 exactly, something you cannot do in binary encoding. Its internal capacity is 78 decimal digits, which covers both unsigned and signed 256 bit integers. So internally, the calculation is done without error. You can see this, if you apply frac() or mask() to the result: frac(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFC2F-0x1ae168fea63dc339a3c58419466ceaeef7f632653266d0e1236431a950cfe52a) = 0

mask(0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFC2F-0x1ae168fea63dc339a3c58419466ceaeef7f632653266d0e1236431a950cfe52a;16) = 0x1705

The mask operation above yields the least significant 16 bits - in perfect accordance with the correct result. So the culprit is the routine converting the internal precise result for output.

Wolf