Experimental Driver - Segmentation Fault with Large Result Set

[Brian Jerome]

This is an issue with the db2sock toolkit found at the bottom of the SuperDriver page.

I have test PGM that returns a large result set (DS with 15000 records). I've tested this with the db2sock toolkit (toolkitasync.js) and calling the DB2PROCJR (sql) stored procedure. The latter works fine. With the former I am getting a Segmentation fault (core dumped) when I try to return around >11000 records.

RPG

H AlwNull(*UsrCtl)

       dcl-ds inputDS qualified;
         in1 char(100);
         in2 char(100);
       end-ds;

       dcl-ds oDS qualified;
         o1 char(500);
         o2 char(500);
       end-ds;

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         outputA likeds(oDS) dim(15000);
         last char(20);
       end-pr;

       dcl-pi Main;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         outputA likeds(oDS) dim(15000);
         last char(20);
       end-pi;

       dcl-s i int(10);
       outCount = inCount;
       for i = 1 to inCount;
         outputA(i).o1 ='Lorem ipsum dolor sit amet, consectetur ' +
          'adipiscing elit, sed do eiusmod tempor incididunt ut ' +
          'labore et dolore magna aliqua. Ut enim ad minim veniam, ' +
          'quis nostrud exercitation ullamco laboris nisi ut ' +
          'aliquip ex ea commodo consequat. Duis aute irure dolor ' +
          'in reprehenderit in voluptate velit esse cillum dolore ' +
          'eu fugiat nulla pariatur. Excepteur sint occaecat ' +
          'cupidatat non proident, sunt in culpa qui officia ' +
          'deserunt mollit anim id est laborum.';
         outputA(i).o2 = 'value';
       endfor;
       last = '"quoted" text';
       outCount = i - 1;
       return;

Input JSON

{"pgm":[
    {"name":"DRTEST04", "lib":"BJEROME"},
    {"s": {"name":"inCount", "type":"10i0", "value":15000, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"100a", "value":"i1"},
            {"name":"in2", "type":"100a", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0"}},
    {"ds": [{"name":"outputA","dim":15000, "by": "out", "dou": "outCount"},
        {"s":[
            {"name":"o1", "type":"500a", "value":"i1"},
            {"name":"o2", "type":"500a", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

Changing the value of inCount changes the outCount value.

[Former user]

db2sock no limit SQL400JsonAsync(hdbc, jin, ilen, jout, olen)

First, for clarity, you are using json toolkit of db2sock. Which, also means that db2ai is simply a transport of toolkit json via SQL400Json(Async). Further there is no limit on SQL400Json(Async) API. Everything in this discussion is about the experimental node db2ai driver.

experimental node db2ia driver ... size limit

This not a production driver (see next topic ). Currently size limit around 5 MB in experimental node driver (below). This is an arbitrary size to test concept of using SQL400Json a heap variable tmpJsonOut (below).

• src/db2ia/dbconn.h

#include "PaseCliAsync.h" /* @adc */
#define DB2_I5_DB2SOCK_OUT_SIZE 5000000 /* @adc */


void DbConn::ToolkitCall(const ARGUMENTS& args) {
  Isolate* isolate = args.GetIsolate();
  HandleScope scope(isolate);
  DbConn* obj = ObjectWrap::Unwrap<DbConn>(args.Holder());

  DEBUG("ToolkitCall().\n");
  CHECK(obj->connAllocated == false, CONN_NOT_READY, "The Connection handler is not initialized.", isolate)
  CHECK(args.Length() != 1, INVALID_PARAM_NUM, "The toolkitSync() method accept only one parameter.", isolate)

  String::Utf8Value arg0(args[0]);
  SQLCHAR* tmpJsonIn = *arg0;
  SQLINTEGER tmpJsonInLen = strlen(tmpJsonIn);
  SQLCHAR * tmpJsonOut = (SQLCHAR *) malloc(DB2_I5_DB2SOCK_OUT_SIZE);
  SQLINTEGER tmpJsonOutLen = DB2_I5_DB2SOCK_OUT_SIZE;

production node db2ia driver ... limitless

Probably a 'limit/size' input parameter will be needed in production driver.

toolkit(jsonIn, limit, callback);

dbconn.toolkit(json, 150000000,
  function (result) {
    console.log(result);
  }
);

confusion on direction node toolkit design (your issue#10)

However, seemingly endless design options in node toolkit makes pinning down a design of a node db2ia a bit difficult. From other db2sock toolkit issue #10 ... In fact, you froze me to inaction. “Nobody move! I dropped me brain!” – Jack Sparrow (Pirates of Caribbean)

[Brian Jerome]

I don't want to have to pass in an output limit. That'd be more code I have to maintain on my end. I was hoping to get away from managing the PLUG size of the input/output the old toolkit used.

[Former user]

I was hoping to get away from managing the PLUG size of the input/output the old toolkit used.

There will always be some limit to output size specified by the 'user code'. In this case the user code is the db2ia driver. The db2sock interface below is traditional CLI-style interface found every db2 driver. In this case the node db2ia 'user code' driver is using the API, so it 'picks' the size (plug size is store procedure thinking).

SQLRETURN SQL400Json(SQLHDBC hdbc,
 SQLCHAR * injson,
 SQLINTEGER inlen, 
 SQLCHAR * outjson,
 SQLINTEGER outlen) 

However, seemingly endless design options in node toolkit makes pinning down a design of a node db2ia 'toolkit' interface a bit difficult.

What do you want me to do with my experimental node db2ia driver?

[Former user]

DB2PROCJR ... not limitless

Before we start thinking that DB2PROCJR call is the bees knees (great beyond mom's fresh baked cookies). I think we should point out that DB2PROCJR is not limitless either. Thats is it will return a result set, but total size of the result set is 'limited' by the RPG program returning that data.

       dcl-s sLast151 int(10) inz(0);
       dcl-s Occurs151 packed(5:0) inz(5001);
       dcl-ds Output151 occurs(5001);
         Out151 char(3000);
       end-ds;

       // result set of row for concat
       exec sql          
         Set Result Sets Array :Output151 For :Occurs151 Rows;

total size: 5000 * 3000 = 15,000,000 (15 MB roughly)

Of course, you can set these limits higher, and higher, and higher ... until PASE runs out of memory ... but hey man.

There is always some kind of 'sanity' limit. So we will never be able to satisfy people that want to ftp an entire data base to the client over a toolkit ... in json. In fact, I am berry much trying not to say 'crazy users', but, well you get the idea. There is always some type of limit to a toolkit interface.

[Former user]

good topic for public debate ... limits

Question: That is, when does a toolkit call returning MBs of json become 'too much to expect for a toolkit'?

Basically two extremes exist.

1) Never any limit. We should be able to wrap every data base file on the system into a single json representation. Gigabytes, upon gigabytes, upon gigabytes worth of json going across web to your browser. MU-AH HA ... its alive (Frankenstein parody).

2) A limit. Approximately 15MB should be enough for any rational json based call.

Why did you pick 15MB Ranger (me)?

1) The worst case default 32 bit threaded language (language built in web server)

For example, when you compile a language (node, etc.), 32 bit mode 'small model' default. Total memeory for use, you get one 256 mb segment to share bss (program data), stack, and heap (malloc). When language is threaded (aka, async interfaces), each thread could make a toolkit call at the 'same time. Thereby, a language 'web server' could have 17 concurrent thread callers (256MB/15MB = 17 - each 15 MB buffer).

2) The better case 'huge model' 32 bit threaded language (language built in web server)

'Huge' model 32 bit case we could have roughly 9*256MB memory (each 256MB, a segment). Or approximately 17 calls (15MB each) * 9 (segments). Thereby, a language 'web server' could have 153 concurrent thread callers.

2) Best case 64 bit ...

Default 64 bit heap is 256MB * 256 (segments). Or approximately 17 calls (15MB each) * 256 (segments). Thereby, a language 'web server' could have 4352 concurrent thread callers.

Note: Math is not exact here, but you get 'pase memory' idea. Always a limit on a 'toolkit call' in any given language environment. Aka, 'crazy expectation' is quantifiable to some extent (in my opinion).

(Reminder, db2sock is not limited per say. This is a toolkit topic).

[Danny Roessner]

16,773,104 bytes? https://www.ibm.com/support/knowledgecenter/SSAE4W_9.5.1/com.ibm.etools.iseries.ilerpg.doc/RNF0376.htm

[Former user]

... 16,773,104 bytes (16MB) ... 15,000,000 bytes (15MB) ... who cares???

In fact, may be best to limit slightly under to allow for 'crazy big' to slip a bit here and there.

BTW -- Truth in lending (fine print) ... fairness we could argue for additional toolkit design enabled multiple calls to collect paginated parts of a single PGM, SRVPGM call result.Aka, we could boost ILE memory far beyond PASE range, until the RPG program called was standing on tip, tip toes with memory limits (ILE limits).

Hey man ... this is exactly a discussion about 'too much'. You are getting the picture of the work that would cross into the scripting language toolkit driver ... no free lunch with 'crazy', every man needs to hunt the white whale, or not (Moby Dick).

BTW -- I choose not. But then again, not sure I would have joined Capitan Ahab on the white whale hunt to begin with .... 15MB seems ok.

[Former user]

Are we done Ishmael/Danny (Moby-Dick)? Or are we going to hunt down the 'limitless memory' white whale of toolkit calls???

[Danny Roessner]

Limitless seems overkill. I figured the existing RPGLE limit of 16MB would be the way to go. 15MB should be plenty, but don't see a reason not to match. Any chance you can push out a new build of the experimental driver with a higher limit? 15MB if you prefer.

[Former user]

Limitless seems overkill ... 15MB should be plenty,

Ok, updated my test version of node db2ia on Yips.

• Yips Super Driver - Experimental test node db2ia driver (bottom page) - 2018–03–05 14:04 - Added Danny 15MB size (up from 5MB test version)

bash-4.3$ cd /QOpenSys/QIBM/ProdData/OPS/Node6/lib/node_modules/npm/bin/node-gyp-bin/src/db2ia

bash-4.3$ grep DB2_I5_DB2SOCK_OUT_SIZE dbconn.h                                                                             
#define DB2_I5_DB2SOCK_OUT_SIZE 15000000 /* @adc (Danny) */

[Brian Jerome]

No longer getting a segmentation fault, but now getting a memory fault. I'll create a new issue for that later.