PGM Complex JSON String Output Distorted

[Brian Jerome]

I have a simple pgm returning a JSON string:

H main(RETURNJSON)                                                         
 **************************************************************************
 **  Web Service PCML Parameter Setup:                                   **
 **                                                                      **
 **    Parameter    Usage        Counter     Description                 **
 **    =========    ==========   ==========  ================            **
 **    outone       output                   JSON One                    **
 **************************************************************************
d*---------------------------------------------------------------------    
d RETURNJSON      pr                  extpgm('RETURNJSON')                 
d  outone                    64000                                         
d*info|use:output|jType:S|                                                 
d*cmnt|comment:This field is the first JSON Output.|                       
 **************************************************************************
 ** Procedure (Local) Definitions                                        **
 **************************************************************************
p RETURNJSON      b                                                        
d                 pi                                                       
d  outone                    64000                                         
 **************************************************************************
 **************************************************************************
 **                          Main Routine Code                           **
 **************************************************************************
 /free                                                                     
  outone = '{"title":"Hello World","content":"<span' +                     
  'class=\"subheading\">Subhead</span>","footer":' +                       
  '{"links":["www.google.com","www.mysite.co"],"height":50}}';             
 /end-free                                                                 
 **************************************************************************
p RETURNJSON      e                                                        

And I'm calling the PGM with the following with the sg10 build:

CALL
DB2JSON.DB2PROCJR('{"pgm":[
    {"name":"RETURNJSON",  "lib":"BJEROME"},
    {"s": {"name":"outone", "type":"64000a", "by":"out"}}
]}');

I'm expecting it to return something like:

{"script":[{"pgm":["RETURNJSON","BJEROME",{"outone":"{\"title\":\"Hello World\",\"content\":\"<span class=\\\"subheading\\\">Subhead</span>\",\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],\"height\":50}}"}]}]}

What I get is the output distorted in a weird way:

{"script":[{"pgm":["RETURNJSON","BJEROME",{"outone":"{\"title\"\"\"HellloWorld\",\"\"oontent:\"<spancccass=\\\"\"bheaddinn\\\">>ubbad</sspaa>\"\",fooeer:{{\"llnkks:[[wwwgooogge..cm\"\"www.mmysstee.o\"\"\"heighttt50}}}"}]}]}

Might this be related to the change in sg8 to escape quotes in strings? I ran this code with sg6 and the text was fine but not escaped.

[Former user]

DB2JSON.DB2PROCJR

First, DB2PROCJR uses an ILE port of PASE libtkit400.a function. So, well, ccsid hell, not recommended interface for anything production beyond remote access via ODBC/CLI LUW laptops.

Just a warning ... mmm ... 'worked before' escape sequences maybe indicates not using a client side json parser/verifier on return output. Aka, if building an intermediate 'middleware chunk' (language toolkit), you may be passing 'invalid json' buck-stops-here to your callers/users.

What I get is the output distorted in a weird way

However, generally, we should be able to wade through json escape sequences correctly. In this case you have a whole lot of them (Uf Da).

I will take a look.

[Former user]

user error ... i think

Looks like your RPG program has a 'half-escaped' return.

outone = '{"title":"Hello World","content":"<span' +                     
  'class=\"subheading\">Subhead</span>","footer":' +                       
  '{"links":["www.google.com","www.mysite.co"],"height":50}}

-- should be ---

outone = '{"title":"Hello World","content":"<span' +                     
  'class="subheading">Subhead</span>","footer":' +                       
  '{"links":["www.google.com","www.mysite.co"],"height":50}}

Note: class=\"subheading\" should be class="subheading" (no escape)

I corrected your RPG in my program ... and works.

     H AlwNull(*UsrCtl)

       dcl-pr Main extpgm;
         outone char(64000);
       end-pr;

       dcl-pi Main;
         outone char(64000);
       end-pi;
       outone = '{"title":"Hello World","content":"<span ' +                     
                'class="subheading">Subhead</span>","footer":' +                       
                '{"links":["www.google.com","www.mysite.co"],"height":50}}';
       return;  

Note (minor): You also missed space between '<span class'.

The run ...

$ ./test1000_sql400json32 ../json/j0911_pgm_danny03_escape
input(5000000):
{"pgm":[
    {"name":"DANNY03",  "lib":"DB2JSON"},
    {"s": {"name":"outone", "type":"64000a", "by":"out"}}
]}


output(217):
{"script":[{"pgm":["DANNY03","DB2JSON",
{"outone":"{\"title\":\"Hello World\",
\"content\":\"<span class=\"subheading\">Subhead</span>\",
\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],
\"height\":50}}"}]}]}

result:
success (0)

Suggestion ... arrogant not intended ... assume toolkit is correct and error is test case (first). Aka, close look at your test. I am only working part time Mon-Wed, so things could sit unanswered every week.

Note: I have extra line feeds added to above output to fit within small box of this issues output text. You need to remove extra line feeds to validate json cut/paste from here (obvious???).

[Former user]

BTW -- Please close issues when satisfied with answer/result. No need to clutter issues database with finished items. I always forget and have to read through the last chat each week. Thanks

[Brian Jerome]

If it has class="subheading" then it's not a valid JSON string.

I took that out completely to avoid that being an "issue" but it is still distorted in sg10.

outone = '{"title":"Hello World","content":"<span ' +       
'>Subhead</span>","footer":' +                              
'{"links":["www.google.com","www.mysite.co"],"height":50}}';

{"script":[{"pgm":["RETURNJSON","BJEROME",{"hello":"{\"title\"\"\"HellloWorld\",\"\"oontent:\"<span   Subhea</span>\"\"\"\"fooot\":{\"\"inkkkk\"[[\"ww.ooole....o\"\",\"\"wmyyiteeeec\"\",\"\"eght\":55}}}"}]}]}

[Former user]

Mmm ... same test works for me (posted above). Maybe try moving up a few releases to test driver - 1.1.2-sg4? If still an issue, i will have to figure out why mine works and you does not.

[Former user]

I have few ideas for you.

First, try calling 1.1.2-sg4, danny03. Let's try to see if escape works right taking your RPG out of the equation.

Second, Let's go back to you original statement use to work before escape sequences added to db2sock. Possible test your RPG guy (not you), escaped the json data for another use ( Java , etc). Therein we are not getting the full story here (already escaped data).

If this is the case, we need to add flag like "s" like "no escape":"on".

Again, I do not say to offend. I am merely trying to help. Your other closed issue your RPG guy did not give you the real data layout. So, well, we maybe should assume RPG guy is messing with you again. Not to worry, we will get it working even if your RPG guys make it difficult.

Btw - sorry about the typos, my bloody iPad keeps substituting wrong words while I am typing (big help it be .... Not).

[Former user]

Continued ... Just realized I am not handling escape json input correctly.

Please look into previous suggested tests. We need to understand if RPG already escaped your output (we do it twice then).

Monday I will look into possible ways of automatically detecting already escaped json both input and output.

Thanks for your help.

[Brian Jerome]

I'm thinking the data might have been escaped for Java, but I'll have to discuss that at another time when they are available. I will try getting the latest driver when I have some time. Probably won't be able to until Monday unfortunately (same goes for the other inputDS issue)

[Former user]

Cool. i need to do some more work anyway (below).

Note to self (geek reminder - myself) ... i should move json specific formating out of base toolkit. Dealing with json escape should be contained in toolkit/parser-json, not in base toolkit ILE converters. We have not yet added other parser formats like xml, cvs, but probably will some date.

[Former user]

Delay. Working on unrelated PASE lab issue. Will let you know when avail here.

[Brian Jerome]

Updated to sg5..

I tried running danny03 test but that has the same weird result as mine. Something else must be wrong here.

CALL
DB2JSON.DB2PROCJR('{"pgm":[
    {"name":"DANNY03",  "lib":"DB2JSON"},
    {"s": {"name":"outone", "type":"64000a", "by":"out"}}
]}');

{"script":[{"pgm":["DANNY03","DB2JSON",{"outone":"{\"title\"\"\"HellloWorld\",\"\"oontent:\"<span   lass=\"ubheadinnn\">Subbhd</spannn\",\"ffoor\"::\"liiikk\"\":[\"\"wggoggll..oom\",,w..yssiieecco\"\"\"hightt\"\"0}}}}"}]}]}

[Former user]

Updated to sg5..

I have not fixed this yet (other Rochester lab work). In db2sock json-parser, we should attempt to 'discover' if json data is already escaped to avoid double slash-quote error (distorted). I suspect we will be able to sort this out when i find time to work on this problem.

[Former user]

CALL DB2JSON.DB2PROCJR

I repeat. DB2PROCJR is not a good interface to build a toolkit, only used for remote use laptops, etc.. You should use direct call SQL400Json and/or SQL400JsonAsync.

SQLRETURN SQL400Json( SQLHDBC  hdbc, 
SQLCHAR * injson, SQLINTEGER  inlen, 
SQLCHAR * outjson, SQLINTEGER  outlen );

pthread_t SQL400JsonAsync ( SQLHDBC  hdbc, 
SQLCHAR * injson, SQLINTEGER  inlen, 
SQLCHAR * outjson, SQLINTEGER  outlen, 
void * callback );

Expanding to be clear ...

Assuming your new toolkit is targeting node. If your node toolkit is using REST (fastcgi), then you will get natural async behaviour (good node scripts). However, using CALL DB2JSON.DB2PROCJR is not a good idea for a db2 interface for your node toolkit (aka, not async at all). You need to modify the node db2 driver to support SQL400Json/SQL400JsonAsync to build a good db2 interface (aka, change db2a c code for node/npm). If this is beyond your skill level, you should ask for help.

[Former user]

I have not fixed this yet (other Rochester lab work). In db2sock json-parser, we should attempt to 'discover' if json data is already escaped ...

Ok. I think this fix may work for none, full or partial escaped json "value" (aka, your RPG guys pre-escaping some/all json ... argh).

• Yips Super Driver - 1.1.2-sg6 - test driver - toolkit test json escape values from RPG (partial, full, none escaped) ... i hope

=============================
test1000_sql400json64 ../json/j0911_pgm_danny03_escape
=============================
input(5000000):
{"pgm":[
    {"name":"DANNY03",  "lib":"DB2JSON"},
    {"s": [
     {"name":"outone", "type":"64000a", "by":"out"},
     {"name":"flag", "type":"10i0", "value":0, "by":"in"}
    ]}
]}


output(217):
{"script":[{"pgm":["DANNY03","DB2JSON",{"outone":"{\"title\":\"Hello World\",\"content\":\"<span class=\"subheading\">Subhead</span>\",\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],\"height\":50}}"}]}]}

result:
success (0)
=============================
test1000_sql400json64 ../json/j0912_pgm_danny03_escape_partial
=============================
input(5000000):
{"pgm":[
    {"name":"DANNY03",  "lib":"DB2JSON"},
    {"s": [
     {"name":"outone", "type":"64000a", "by":"out"},
     {"name":"flag", "type":"10i0", "value":1, "by":"in"}
    ]}
]}


output(217):
{"script":[{"pgm":["DANNY03","DB2JSON",{"outone":"{\"title\":\"Hello World\",\"content\":\"<span class=\"subheading\">Subhead</span>\",\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],\"height\":50}}"}]}]}

result:
success (0)

RPG danny03

       dcl-pi Main;
         outone char(64000);
         flag int(10);
       end-pi;
       // partial escaped json ... argh (subheading)
       if flag = 1;
         outone = '{"title":"Hello World","content":"<span ' +                     
                'class=\"subheading\">Subhead</span>","footer":' +                       
                '{"links":["www.google.com","www.mysite.co"],"height":50}}';
       // not escaped json (yes)
       else;
         outone = '{"title":"Hello World","content":"<span ' +                     
                'class="subheading">Subhead</span>","footer":' +                       
                '{"links":["www.google.com","www.mysite.co"],"height":50}}';
       endif;

[Brian Jerome]

I'm not understanding what you mean by partial escape. If the JSON value (which is already a string) contains a JSON string then shouldn't the \ also be escaped to \\ in the output? I figured otherwise there would be an error from invalid JSON. Am I overthinking this?

...\"<span class=\"subheading\">Subhead</span>\"...

...\"<span class=\\\"subheading\\\">Subhead</span>\"...

[Former user]

Am I overthinking this?

Argh ... escape logic same effect looking into a mirror with a mirror behind you, and you, and you , and you, one you, two you, three you, big you, little you, mini you, ... argh!!!

JSON validator liked both of following escape sequences.

• https://jsonlint.com/

loves it ... \"<span class=\\\"subheading\\\">Subhead</span>\", ...
== and ==
loves it ... \"<span class=\"subheading\">Subhead</span>\", ...

full json ...

to escape ...

{
    "script": [{
        "pgm": ["DANNY03", "DB2JSON", {
            "outone": "{\"title\":\"Hello World\",\"content\":\"<span class=\\\"subheading\\\">Subhead</span>\",\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],\"height\":50}}"
        }]
    }]
}

or not to escape ...

{
    "script": [{
        "pgm": ["DANNY03", "DB2JSON", {
            "outone": "{\"title\":\"Hello World\",\"content\":\"<span class=\"subheading\">Subhead</span>\",\"footer\":{\"links\":[\"www.google.com\",\"www.mysite.co\"],\"height\":50}}"
        }]
    }]
}

I don't know. Maybe just best to assume that all output is never 'escaped'. Aka, you go argue with your RPG guys about not pre-escaping json before the toolkit????

[Brian Jerome]

I'll have a discussion about that. Running JSON.parse() (in JS) on the content of outone gets an error so we need that extra escape in there. I'll mark this resolved for now as it looks good. Thanks.

[Former user]

update db2procjr (laptop)

You may have been using db2procj or db2projr. Escapes same issues as Jess G. Re-try your test with 1.1.2-sg9.

[Brian Jerome]

Will do. Thanks!

Edit This is looking better with 1.1.3-sg3.