Inject a LVObject Serialization Strategy

[logmanoriginal]

Right now I need to add boilerplate code to parse private object data from JSON, which is annoying if you want to parse large nested object trees (i.e. for RESTful APIs). So I went ahead and forked your repository to add native support for that.

Since merging is tricky in LV, I open this as an issue rather than a pull request. You can find my branch here: https://bitbucket.org/logmanoriginal/jsontext/src/lvobjectserialization/

I made sure not to change existing code as much as possible, so LVDiff should give you a pretty good idea of what changed. Also, this branch is based on version 1.2.10 and does not include your recent changes.

The implementation is based on the flatten string representation of LabVIEW objects, as described here: https://labviewwiki.org/wiki/LabVIEW_Object#Flattened string representation of LabVIEW objects

Examples are included in the Test folder to show that it can actually parse object trees (including parent and aggregate objects). One of the benefits is, that you can build and extend complex object hierarchies over time, without the need for a common ancestor. To be fair, LV2020 interfaces will probably make this much easier. Nevertheless, I think this is something worth sharing.

The current implementation breaks backwards compatibility with objects that were stored with previous versions of JSONtext. This can probably be fixed by looking at the JSON data and calling a fallback method. Then again, some data types might fail as I didn’t test all of them.

The code is efficient enough for my needs and I made sure to cache the object lookup routine to greatly improve performance. It also works in executables. Let me know if this is something worth further investigation.

[James Powell]

Copied from commit in forked project:

Serialize LabVIEW Objects from/to JSON

LabVIEW Objects should behave similar to Clusters. They should be
serialized as JSON Objects or JSON Arrays depending on the contained
data (named or unnamed).

Examples:
Empty Object: `{}`
Object with data: `{"String":""}`

Ancestors should be consolidated into the same object.

Example:
`{"String":"","Parent String":""}`

Composite objects (objects inside objects) should appear as nested
objects in JSON.

Example:
`{"Nested":{"String":""},"String":""}`

Elements with <JSON> tag should be handled as raw JSON text.

Example:
`{"payload":{}}`

This commit adds the necessary code to serialize LabVIEW objects
from/to JSON in a generic way by utilizing the Flatten To String
function (as before) in combination with information about the
private class data control to name the data.

Note that this depends on the memory model of LabVIEW Objects, as
did the previous implementation that stored the output of Flatten
To String as a text value in JSON.

This change will break objects stored with previous versions.

Tested in LV2017 & LV2019, both in IDE and RTE.

[James Powell]

I’m trying to think of some way that your object code can be injected into JSONtext. Too many possibilities for a clear format for Objects.

[logmanoriginal]

Just as a heads-up, the code in my branch is outdated and buggy. I made a separate library to provide functions that can extract and inject data from objects: https://github.com/LogMANOriginal/LabVIEW-Composition

Objects in LabVIEW can be broken down into two parts:

• Private data clusters
• Library versions

Library versions are only needed for mutation history (as in flattening to/from XML). I’m not aware of any other language that has this feature. Considering that JSONtext is already quite flexible with how it interprets values, support for mutation history is probably not a requirement.

Private data clusters can be stored in different ways:

1. Array of objects
2. Object of objects
3. Common object

Each of these has different pros and cons.

• Array of objects

  • Pros

    • Inheritance hierarchy is maintained
    • Supports duplicate member names (for example, when a child class defines a member with the same name as its parent)
    • Lossless

  • Cons

    • Objects are stored by relationship
    • More complex object structure
    • Each element in the JSON array has different key-value-pairs
    • Same structure as an array of clusters

• Object of objects

  • Pros

    • Inheritance hierarchy is maintained
    • Supports duplicate member names (for example, when a child class defines a member with the same name as its parent)
    • Lossless

  • Cons

    • Objects are stored by relationship
    • More complex object structure
    • Same structure as a cluster of clusters

• Common object

  • Pros

    • Simple object structure
    • Deserialization between different class hierarchies and clusters possible.

  • Cons

    • Inheritance hierarchy is hidden
    • Same structure as regular clusters
    • Lossy (duplicate member names are lost)

I believe that a common object is the most pragmatic implementation, which allows smooth transition between regular clusters and objects of any complexity.

Here is a simple example to illustrate the different types. Notice that all classes define their own “Title” field:

Array of objects

[
    {
        "Title": "Base Class"
    },
    {
        "Title": "Parent Class"
    },
    {
        "Title": "Child Class"
    }
]

Object of objects

{
    "base.lvclass": {
        "Title": "Base Class"
    },
    "parent.lvclass": {
        "Title": "Parent Class"
    },
    "child.lvclass": {
        "Title": "Child Class"
    }
}

Common object

{
    "Title": "Child Class"
}

‌

[James Powell]

One option that is simple for a way to “hook” in your chosen flattening code is for JSONtext to have an abstract Parent class that defines “To JSON” and “From JSON” dynamic methods. Then you can make a child that overrides those methods with your implementation, and inherit all your concrete classes off of that.

Another option is to provide an input to the JSONtext functions for an “LVObject Serializer” that provided the flattening functionality. But you would have to wire a constant of that Serializer to each function that needs it.

A modification would be a globally overridable serializer, but that has all the problems of a global.

[logmanoriginal]

Your first option is very similar to AQ Character Lineator. It requires a common base class, which is not always possible. I wish this was possible with interfaces, but AQ already explained why this would be a bad idea: https://lavag.org/topic/21894-openg-and-object-compatibility/?do=findComment&comment=135030

A separate input for the desired serialization strategy sounds good to me (this idea could even be extended to allowing custom strategies for any type, including unknown types for later versions of LabVIEW - but that is a different discussion). The constants I’d simply wrap in another VIM and put them in a custom package that extends the JSONtext palette.

With such an input users could even build their own globally overridable serializer if they really wanted to. Not my cup of tea, though.

[James Powell]

The constants I’d simply wrap in another VIM and put them in a custom package that extends the JSONtext palette.

Good point. I am leaning toward doing both the first two options: an overridable parent “To JSON” class and a “Serialization strategy” input, as they serve different use cases (the first for letting the class decide it’s JSON form, and the second providing formatting of all classes).

[James Powell]

Made a #56_Objects branch, with a LVClass Serializer input. Also a Demo child class that (easiest thing to do) just flattens to XML in a JSON string.

a0daf95b0868f4d6cc7f86acaeff9a753d16cc8c

Can you try and make an extension to JSON text that injects your Object-flattening stuff? It should be an independant extension, not part of JSONtext itself.

[logmanoriginal]

Sounds good to me.

I have done a few tests with your branch and found one issue, the serializer needs to run reentrant to work with nested objects.

‌

[logmanoriginal]

Found another issue.

From JSON Text does not call the serializer for arrays of objects. It looks like Scalar JSON text to Variant does not use the serializer.

‌

[James Powell]

the serializer needs to run reentrant to work with nested objects.

Done.

[James Powell]

From JSON Text does not call the serializer for arrays of objects. It looks like Scalar JSON text to Variant does not use the serializer.

Objects are now not “Scalers”, meaning they are potentially recursive (the “Scaler” version is inlined and can’t do recursive types). The bug was a test on type where Clusters/Arrays/Waveforms were identified as non-saler, but LVClass was not included. Fixed. Also removed the now-confusing code for LVObjects in Scalar JSON text to Variant.

a70adc35bb96aaff6737a85e233c19246fd7ae28

‌

[logmanoriginal]

Awesome! Everything works now. Here is my repository, including tests and a use-case example: https://github.com/LogMANOriginal/JSONtext-Object-Serialization
I have added JSONtext as a subtree. My intention is to remove it when this is released as a package.

There are still some limitations, but those are unrelated to JSONtext. For example, deserialization only works on parent classes (since JSON doesn’t have type information). I’ll probably solve this with a syntax similar to JSON.NET.

Anyway, I believe this feature is complete. Thanks for your awesome work!

[James Powell]

I would suggest automatically adding a item “LVClass” with the class name. You should also consider the use case of dynamically loading classes from a provided subdirectory.

I’ll make a Beta build soon, so you can have a VIPM-installed package to link to.

[James Powell]

@logmanoriginal I’ve built and posted a Beta VIPM here:

https://forums.ni.com/t5/JDP-Science-Tools/BETA-version-of-JSONtext-1-5-0/m-p/4136112#M32

[logmanoriginal]

That was quick, thanks a lot!

I would suggest automatically adding a item “LVClass” with the class name. You should also consider the use case of dynamically loading classes from a provided subdirectory.

Good idea, I’ll add those to the backlog.

[logmanoriginal]

There is a bug in the beta 1.5.0, which produces wrong output for the Merge into Object function. Here is the test case:

‌

[logmanoriginal]

I got it fixed by selecting the correct “<JSON>” element in Set Multiple Object Items

It probably makes sense to give each element a unique name.

[James Powell]

Thanks, I found that yesterday (my Unit Testing was lacking). I’ve made quite a few changes yesterday, so I will build a 1.5.1 version this weekend.

[James Powell]

New 1.5.1 version posted: https://forums.ni.com/t5/JDP-Science-Tools/BETA-version-of-JSONtext-1-5-0/m-p/4137184#M34

[logmanoriginal]

Great, all tests are passing on my end

[James Powell]

1.5.2 version, with Serializer input added to all methods going to/from LVtype. https://forums.ni.com/t5/JDP-Science-Tools/BETA-version-of-JSONtext-1-5-0/m-p/4137184/highlight/false#M34

[logmanoriginal]

I made the necessary changes for version 1.5.2 (the class input terminal on the JSON text to Variant function moved). Everything still works as planned.

One more thing, please add the “enable LabVIEW extensions?” and “Pretty Print?” terminals to the “LVObject to JSON” method. That way all serialization functions will have the same pattern.

[James Powell]

please add the “enable LabVIEW extensions?” and “Pretty Print?” terminals to the “LVObject to JSON” method

Doing it now.

[James Powell]

Marking resolved, as this is submitted to teh Tools Network

[James Powell]

One option that is simple for a way to “hook” in your chosen flattening code is for JSONtext to have an abstract Parent class that defines “To JSON” and “From JSON” dynamic methods. Then you can make a child that overrides those methods with your implementation, and inherit all your concrete classes off of that.

See #88 for this.