SGL,DBL,EXT to String.vi adaptive formatting is a problem for passing arrays as lists to Python node

[X]

The LabVIEW Python node currently supports a limited number of types, which makes dictionaries formatted as JSON strings a preferred approach to pass generic data structures in and out.

[json.loads(input) passed from LabVIEW or json_string = json.dumps(dict) returned to LabVIEW]

The JSONtext library is super useful for these tasks (encoding and decoding the respective strings) but there is a hidden catch in the encoding VIs, designed to be economical in terms of final string size, but in practice detrimental for the preservation of SGL, DBL and EXT precision.

The question is whether an option “full precision” (preferably turned on by default, so that this eliminates those gotchas) could be added to the conversion VIs.

[James Powell]

The “economical in terms of final string size” isn’t actually more that a minor consideration. JSONtext has the design principle of preserving decimal values, which is the opposite of the inbuilt JSON library, which is intended to preserve serialized floating-point values.

An illustration:

Imagine that an Operator sets the Power Level to 0.28; the corresponding Setting JSON with JSONtext will contain “Power Level”:0.28. But The inbuilt JSON function will have Power Level”:0.28000000000000002665. Now, the extra characters are uneconomical (and hard to read), but that is a minor issue. The real problem is that the value is not preserved. Any code that triggers off of a change in JSON settings will register 0.28-->0.28000000000000002665 as a change of setting.

Now, in your use case, you actually care about exact preservation of floating-point machine values (no human in sight). You have a value like 0.28000000000000008216, which JSONtext converts via “Power Level”:0.28 and then back to the value 0.28000000000000002665. To your code, the floating-point value changed, but to JSONtext, those are two equivalent floating-point approximations of 0.28.

So anyway, chance of me doing a “full precision” by default is zero. Chance of me calling it “full precision” is also zero, as the full precision of 0.28 is 0.28000000000000000000000…, not 0.28000000000000002665. But, some kind of “full floating-point serialization” option might be a good idea, to support these use cases.

[James Powell]

One issue: the techniques I use to format the numbers into strings don’t actually work to better than DBL precision, so I can’t easily support EXT. Though given that the inbuilt function has the never-to-be-fixed bug that it flattens EXT to only 6 significant figures(!?!), then JSONtext is not too shabby.

[X]

Not sure about the ‘decimal representation’ preservation bit. If the operator uses an instrument that communicates via a serial port AND the underlying code treats the user input as a string or casts the numeric to a 2 digit string, sure that is important, although I argue that the data should probably be first precast to a formatted string in that case, before being fed to JSON conversion.

But that is not what I was referring to in my illustration.

My goal is to have equivalent data passed to the Python node by JSON string OR list OR named tuple OR marshalled numpy array. Right now, I have to use the native LabVIEW to JSON string for numeric, or I will lose precision (however imperfect the floating point numeric representation is).

I threw in the EXt type for good measure. LabVIEW has been flaky in its support in many other places.