Formatting functions fail randomly for a same expression

[Tey']

1. Enter hex(1e10000)
2. The result is 0x9,B84EA28556BF2697F(+0d8304)
3. Enter hex(1e10000) again
4. The result is NaN
5. Enter hex(1e10000) again
6. The result is 0x9,B84EA28556BF2697F(+0d8304) again

It also happens with bin and oct. It does work correctly with other numbers though (hex(1e100) for instance), but I remember I've seen this behavior before with other numbers as well.

Also:

#!

hex(1e1000)
= 0x3,CE36C7E774F6B015E(+0d830)

hex(2)
= NaN

hex(2)
= 0x2

hex(2)
= 0x2

This seems to indicate some state is not reset after processing the first expression, which makes the second expression fail.

[Pol Welter]

Weirdly enough the value in the tooltip is ok.

[Tey']

Yes, and ans is not set to NaN but to the correct value.

The problem is related to the formatting method (float_out() called from _doFormat()) which sets an error which is never checked so never cleared. The error is then checked later when formatting the next result which produces the NaN.

[Tey']

Return the correct error code when non-decimal fixed point formatting fails (and clear the global float error code as well). Fixes issue #628

→ <<cset 763b8c99bd53>>