Adding data structure name to the XMLtoJSON-converter

[Torbjørn Høivik]

Hey, we've been using the itoolkit for a little while and noticed the XMLtoJSON-converter cuts away the datastructure name from the XML.

Example:

XML response:

<?xml version='1.0'?><myscript><pgm name='SKSFOR03' error='fast'>
<parm io='META'>
<ds len='1'>
<data type='256A'></data>
<data type='50A'>FTJ01</data>
<data type='5A'>00000</data>
<data type='15A'>INFO</data>
<data type='100A'>Processing ok (00000)</data>
<data type='10A'>2017-09-14</data>
<data type='8A'>09.50.15</data>
<data type='12A'>000000000000</data></ds>
</parm><parm io='FORINN'>
<ds len='1'><data type='11A'>2346</data></ds>
</parm><parm io='FORUTMETA'>
<ds len='1'>
<data type='5A'>7</data>
<data type='11A'>400509401</data></ds></parm>
<parm io='FORUT'>
<ds dim='140' len='1'>
<data type='11A'>15365409</data>
<data type='5A'>4</data>
<data type='3A'>M07</data>
<data type='30A'>Bilforsikring (ansatt)</data>
<data type='75A'>HONDA , 2016, ZA17792, 75% bonus 1.Ã¥r, -> 75% bonus 2.Ã¥r</data><data type='1A'>A</data>
<data type='8A'>20170525</data>
<data type='8A'>20180525</data>
<data type='10A'>NICE_SP1G</data>
<data type='5A'>AUTO</data>
<data type='1A'>Y</data>
<data type='11A'>70</data>
<data type='25A'>SpareBank 1 Skadeforsikri</data><data type='50A'>HONDA</data><data type='10A'>10</data>
<data type='10A'>ZA17792</data>
<data type='1A'>N</data><data type='1A'>Y</data>
<data type='1A'>Y</data><data type='1A'>N</data>
<data type='1A'>N</data><data type='1A'>J</data>
<data type='1A'></data></ds>
<ds dim='140' len='1'><data type='11A'>10700169</data><data type='5A'>13</data><data type= ...
...
...
...
..

After we've converted it to JSON:

{
    "type": "pgm",
    "success": true,
    "data": [
        {
            "type": "256A",
            "value": ""
        },
        {
            "type": "50A",
            "value": "FTJ01"
        },
        {
            "type": "5A",
            "value": "00000"
        },
        {
            "type": "15A",
            "value": "INFO"
        },
        {
            "type": "100A",
            "value": "Processing ok (00000)"
        },
        {
            "type": "10A",
            "value": "2017-09-27"
        },
        {
            "type": "8A",
            "value": "10.26.18"
        },
        {
            "type": "12A",
            "value": "000000000000"
        },
        {
            "type": "11A",
            "value": "2346"
        },
        {
            "type": "5A",
            "value": "7"
        },
        {
            "type": "11A",
            "value": "400509401"
        },
        {
            "type": "11A",
            "value": "15365409"
        },
        {
            "type": "5A",
            "value": "4"
        },
        {
            "type": "3A",
            "value": "M07"
        },
        {
            "type": "30A",
            "value": "Bilforsikring (ansatt)"
        },
        {
            "type": "75A",
            "value": "HONDA , 2016, ZA17792, 75% bonus 1.år, -> 75% bonus 2.år"
        },
        {
            "type": "1A",
            "value": "A"
        },
        {
            "type": "8A",
            "value": "20170525"
        },
        {
            "type": "8A",
            "value": "20180525"
        },
        {
            "type": "10A",
            "value": "NICE_SP1G"
        },
        {
            "type": "5A",
            "value": "AUTO"
        },
        {
            "type": "1A",
            "value": "Y"
        },
        {
            "type": "11A",
            "value": "70"
        },
        {
            "type": "25A",
            "value": "SpareBank 1 Skadeforsikri"
        },
        {
            "type": "50A",
            "value": "HONDA"
        },
        {
            "type": "10A",
            "value": "10"
        },
        {
            "type": "10A",
            "value": "ZA17792"
        },
        {
            "type": "1A",
            "value": "N"
        },
        {
            "type": "1A",
            "value": "Y"
        },
        {
            "type": "1A",
            "value": "Y"
        },
        {
            "type": "1A",
            "value": "N"
        },
        {
            "type": "1A",
            "value": "N"
        },
        {
            "type": "1A",
            "value": "J"
        },
        {
            "type": "1A",
            "value": ""
        },
        {
            "type": "11A",
            "value": "10700169"
        },
        {
            "type": "5A",
            "value": "13"
        },
        {
            "type": "3A",
            "value": "M07"
        },
        {
            "type": "30A",
            "value": "Bilforsikring (ansatt)"
        },
        ...
       ...
...
}

We would very much like to keep the <parm io='META'> etc. tags in the json-response so we can see where the data belongs. This becomes especially true with the 'FORUT'-tag as this can have several items in it's response and makes it difficult to seperate out which data belongs to which item.

[Aaron Bartell]

Hi @torban91,

Agree on the need to retain more meta data.

I am hesitant to change the current functionality because it (XMLSERVICE) might be completely replaced in this Node.js iToolkit repo with xjservice. See this comment.

@rangercairns can you speak to the status of xjservice?

[Former user]

Hey, we've been using the itoolkit for a little while and noticed the XMLtoJSON-converter cuts away the datastructure name from the XML

Correct. This converter XMLtoJSON is faulty. That is, XMLSERVICE will send back complete data structures and records (assuming input correct), but XMLtoJSON will 'data reduce' many things out of existence. This is one of the Achilles heels of this itoolkit.

What to do? Well, XMLtoJSON needs to be completely restructured. I really don't have the time, so volunteers would be welcome.

can you speak to the status of xjservice?

In progress. Much faster. All JSON (or any other format xml, cvs, etc.). Not done. When? Well, truthfully probably takes until end of year to complete all the work.

[Former user]

BTW -- xjservice is not the name ... actual project is litmis/db2sock.

[Brian Jerome]

@rangercairns @aaronbartell I'm looking into db2sock as a replacement for this iToolkit XMLtoJSON in NodeJS. Is there is a roadmap or estimated release date for the new project that includes NodeJS support?

[Former user]

litmis/db2sock ... roadmap or estimated release date for the new project that includes NodeJS support?

No 'end date'. That is, Open Source evolves based on user requirements. There is a growing list of demands for basic changes to PASE old driver libdb400.a (long over due).

All binary db2sock toolkit suffix releases 'db2sock-n.n.n-sg(n)' are considered test releases. Available on Yips at SuperDriver. You may download, install, and run today.

... db2sock as a replacement for this iToolkit XMLtoJSON in NodeJS.

The db2sock json toolkit is already available (libtkit400.a), many json directory examples. That is, you can already build a better, faster, node toolkit today. Of course, design of your new node toolkit can drastically affect usability and performance.

What to do ...

I suggest open a litmis/db2sock issue 'replacement node toolkit'. This will allow quick 'documented' work through 'public design' build a new node toolkit. I suspect you will find you are up and running in a few days. Followed by your own iterations for usability, performance, etc. Also, should you need additions to db2sock, we can respond quickly using Open Source db2sock, usually a few hours to a few days (not a typical 'big company' project).

Obviously, I will expect you to do some/most of work for your new node toolkit. You will need to set aside some time. However, db2sock should do all heavy lifting. That is, you should never find anything 'difficult' to author with your new toolkit. This 'cooperative' effort ultimately produces the best possible outcomes. Aka, you get exactly what you want, because you spend your time making it so.

Ok, open an issue on litmis/db2sock if you want to proceed.

[Former user]

... db2sock as a replacement for this iToolkit XMLtoJSON in NodeJS.

Brian, you did not hop right on volunteer train to help build a db2sock json based nodejs toolkit. Why not?

... a taste of "new node toolkit"

First, "new toolkit" we have to accept some basic truths (that hurt).

Truth 1) Node wants all I/O operations to be conducted 'asynchronously' to avoid stalling the main thread JavaScript interpreter. Essentially for 'long running' relative operations like calling RPG, this means using threads get out of the way of the main thread interpreter.

Easy peasy, db2sock offers all CLI interfaces in 'async' and 'non-async' formats (+/- devil in node extension writing c programs).

pthread_t SQL400JsonAsync (
SQLHDBC  hdbc, SQLCHAR * injson, SQLINTEGER  inlen, 
SQLCHAR * outjson, SQLINTEGER  outlen, 
void * callback );

Truth 2) Nobody writes threaded RPG programs. N-O-B-O-D-Y. Millions of RPG programs to be called, maybe 3-4 are built for threaded 'calls'.

For 'new toolkit', RPG no threads please, means 'fastest' performing in-memory call option is off the table. Aka, node SQL400JsonAsync in 'in-memory' mode would kill any existing RPG program (trust me).

So, 'new toolkit', we acknowledge the best 'call RPG' solution is using DB2 server mode (or REST HTTP), wherein each RPG program can 'muddle about' in it's own non-threaded environment job/process (just like 5250 sign-on).

Fortunately, PASE DB2 drivers allow server mode used by most script db2 extensions (SQL_ATTR_SERVERMODE_SUBSYSTEM), which forces a QSQSRVR proxy job for any given db2 connection (db2sock or/and original libdb400.a). Therein, SQL400JsonAsync will work like a charm, getting out of the way of the node interpreter, and, using a QSQSRVR job for the RPG program non-threaded execution of the toolkit call.

Of course, SQL400JsonAsync means the 'new toolkit' will have to get 'connection pools' correct this time around. A problem for both current node db2 driver and toolkit, both extensions do nothing to help with connection pools, therefore node scripts fail often under stressed async operations (usually during prime time rush users).

This was just a taste of the real issues dealing with a asynchronous callback language like node. As you can see, having help from db2sock can go a long, long, way to making this 'new toolkit' a reasonable effort.

Penny in the air ... (penny drops).

Well, are you up to the volunteer task?

[Brian Jerome]

@rangercairns Thanks for the comments. I have been pretty busy on other projects so I haven't had time to write a formal response. I started looking into the source code to see what the input/output JSON looks like and how it differs from my current iToolkit implementation. I am currently writing up an issue to post in the litmus/db2sock repo that will be up soon (not sure how soon).

[Former user]

No problem. I am out of office until next week anyway. As suggestion, you could use db2sock Json toolkit as REST via either fast cgi Apache or nginx. This would allow your new toolkit to stay completely in nice JavaScript (node). The node rest interface is already completely asynchronous therefore you new toolkit would have the correct look and feel. Later, I can help you get the fast version working across database connection. Please look at litmis/db2sock source toolkit fast cgi or nginx directories for setup. The db2sock toolkit parser-json directory readme has added comments dealing with db2 connections. Also db2sock tests/json many examples. Good luck, should you find time. I will help when return to office.

[Former user]

I forgot to mention ... I already have example rest use db2sock json toolkit in directory tests/php. See php basic auth tests. The principle is same in JavaScript (node), and php is fairly trivial to understand.

Btw -- you can develop your entire new JavaScript toolkit on laptop with rest interface to IBM i. Adding the faster database interface layer is a snap because toolkit is really all json in, json out. (I am 400 geek, so I always go native in a chroot, but worth mentioning rest laptop development option).

Important to note, with rest interface, you should have a web authorization scheme like at least basic auth to IBM i. The db2sock toolkit is very powerful (call anything), so web auth front door is in order. Of course if you are already behind trusted company firewall, you can take more liberties with web auth (aka, none).

FYI -- My php db2sock examples are basic auth because my security guys in my lab do not allow open ports even behind firewall.

[Brian Jerome]

Since the db2sock project is pretty much dead has there been more considerations for adding/keeping the DS names in the converted data?

<ds name="MYDATA">...</ds>

-->
"data" : [{
  "MYDATA": [
    ...
  ]
}]

[Brian Jerome]

@aaronbartell https://bitbucket.org/litmis/nodejs-itoolkit/pull-requests/3/refactoring-ixmlnodedsopen-to-use-one/diff