gmake: virtual memory exhausted

[Aaron Bartell]

I am attempting to compile db2util and am getting the following error when run within chroot:

% git clone [e-mail]:litmis/db2util.git
% cd db2util
% /opt/freeware/bin/make
make: *** virtual memory exhausted.  Stop.

I do not get this error outside of chroot. So that's why I am posting the issue here in the ibmichroot project vs. the db2util project.

Some AIX forums report this error with gmake v3.80 and an upgrade to v3.81 fixes it. I am running gmake v4.1 from perzl both inside and outside chroot.

The Linux solution is to increase swap file size, and many point at this tutorial on how to do that. I am not sold it is a swap file issue because it works outside of chroot and the ulimit command delivers the same values both inside and outside chroot.

Others in the IBM i community have had this error when running gmake from QP2TERM and it resolved itself by running from an SSH session. I am already in an SSH session.

My next plan is go deeper with dbx. Here's my first attempt...

% dbx /opt/freeware/bin/make
Type 'help' for help.
reading symbolic information ...warning: no source compiled with -g

(dbx) run
make: *** virtual memory exhausted.  Stop.

execution completed (exit code 2)

I am following the line of thinking from our original deep journey into dbx. In that scenario we were blind debugging and knew the offending function; mmap64. In this case we don't know the offending function.

I am armed with the dbxme.py program you provided but am lost as to what I should be looking for. That's where I am hoping you can guide me concerning what to look for. Please.

[Former user]

I am compiling inside a chroot and do not have any error.

[Aaron Bartell]

I am probably missing a dependency and the current error is a red herring. What chroot config (aka .lst) file did you use? Or do you have a custom one?

[Former user]

Here is what i am using in my chroot ...

[adc@oc7083008330 ~]$ ssh -X monoroot@ut28p63
monoroot@ut28p63's password: 
$ bash
bash-4.3$ cd db2util/
bash-4.3$ ls /
QOpenSys  bin       dev       etc       home      lib       opt       sbin      tmp       usr       var
bash-4.3$ export PATH=/opt/freeware/bin:$PATH
bash-4.3$ export LIBPATH=/opt/freeware/lib:/usr/lib
bash-4.3$ which make
/opt/freeware/bin/make
bash-4.3$ ls -l /opt/freeware/bin/make
-rwxr-xr-x    1 monoroot 0            211998 Oct  6 2014  /opt/freeware/bin/make
bash-4.3$ make --version
GNU Make 4.1
Built for powerpc-ibm-aix5.3.0.0
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
bash-4.3$ make
gcc -g -I. -I/usr/include -I.. -c db2util.c
gcc -g db2util.o -L. -lpthreads -liconv -ldl -lpthread -ldb400 -o db2util
bash-4.3$ db2util "select * from QIWS/QCUSTCDT where LSTNAM=? or LSTNAM=?" -p Jones Vine
"839283","Jones   ","B D","21B NW 135 St","Clay  ","NY","13041","400","1","100.00",".00"
"392859","Vine    ","S S","PO Box 79    ","Broton","VT","5046","700","1","439.00",".00"

bash-4.3$ 

[Former user]

What chroot config

I am using ... pkg_perzl_gcc-4.8.3.lst (bitbucket ibmichroot)

[Former user]

BTW -- i can run xmlservice with 1.0.3 beta ... binary found at ... YIPS download. Also, note that i can run system command again, even in the chroot, because we are using server mode in the db2util command line (QSQSRVR jobs used).

bash-4.3$ db2util "call xmlservice.iplugr512k(?,?,?)" -p "*na" "*here" "<?xml version='1.0'?><xmlservice><sh>system -i 'dsplibl'</sh></xmlservice>"
"<?xml version='1.0'?><xmlservice><sh>
 5770SS1 V7R1M0  100423                    Library List                                          7/25/16 15:30:52        Page    1
                          ASP
   Library     Type       Device      Text Description
   QSYS        SYS                    System Library
   QSYS2       SYS                    System Library for CPI's
   QHLPSYS     SYS
   QUSRSYS     SYS                    System Library for Users
   QGPL        USR                    General Purpose Library
   QTEMP       USR
   QDEVELOP    USR
   QBLDSYS     USR
   QBLDSYSR    USR
                          * * * * *  E N D  O F  L I S T I N G  * * * * *
</sh>
</xmlservice>"

bash-4.3$ 

[Former user]

Hey i am noticing ...

% git clone [e-mail]:litmis/db2util.git
% cd db2util
% /opt/freeware/bin/make

You have a percent sign for the input marker ... just checking ... you are NOT trying to compile in a green screen??? We already know this does not work (virtual memory exhausted).

Also, i tend to run with X enabled, but i have no problem without (below) ...

[adc@oc7083008330 ~]$ ssh monoroot@ut28p63
monoroot@ut28p63's password: 
$ bash
bash-4.3$ make --version
GNU Make 4.1
Built for powerpc-ibm-aix5.3.0.0
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
bash-4.3$ gcc --version
gcc (GCC) 4.8.3
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
bash-4.3$ cd db2util/
bash-4.3$ ls
LICENSE            contributors.txt   db2util.c
Makefile           cpysqlincludes.sh  db2util.o
README.md          db2util            sqlcli1.h
bash-4.3$ touch db2util.c
bash-4.3$ export PATH=/opt/freeware/bin:$PATH
bash-4.3$ export LIBPATH=/opt/freeware/lib:/usr/lib
bash-4.3$ make
gcc -g -I. -I/usr/include -I.. -c db2util.c
gcc -g db2util.o -L. -lpthreads -liconv -ldl -lpthread -ldb400 -o db2util
bash-4.3$ 

[Aaron Bartell]

Here is what i am using in my chroot ...

That's what I have also. I am going to incrementally build the chroot and find the issue. I have an article due (<cough>it's late</cough>) so I suppose building a chroot environment to compile db2util is the article topic :-)

i can run system command again, even in the chroot, because we are using server mode in the db2util command line (QSQSRVR jobs used).

Very cool! But I am not sure I follow how you accomplished that. Where is system being called in the db2util code base or examples?

[Aaron Bartell]

You have a percent sign for the input marker ... just checking ... you are NOT trying to compile in a green screen???

I am compiling from a shell session. The % is the default prompt for zsh.

[Former user]

Try bash ...

[Former user]

Very cool! But I am not sure I follow how you accomplished that. Where is system being called in the db2util code base or examples?

I am inside my chroot using new db2util calling xmlservice with <sh>system </sh> ... no tricks. Again, db2util is using server mode (SQL_ATTR_SERVER_MODE), therefore my db2util inside my chroot is attaching to pre-started QSQ jobs (started outside chroot), thereby we walk outside the chroot via Db2 stored procedure call ... bingo ... we can run qsh stuff again.

db2util (in chroot) -> call xmlservice.iplugr(?,?,?) -> |(attach)| -> QSQSRVR (run xmlservice outside chroot, therefore 'system' works)

rc = SQLSetEnvAttr((SQLHENV)henv, SQL_ATTR_SERVER_MODE, (SQLPOINTER)&attr, SQL_IS_INTEGER);

[Aaron Bartell]

Try bash ...

Yep, tried that before creating this issue.

Btw, my simple chroot environment is working with gmake, so there's probably something hosed in my other more complicated one. For others visiting this thread some day, here's the steps to create a fully functioning chroot environment to compile db2util.

$ cd /QOpenSys
$ git clone [e-mail]:litmis/ibmichroot.git
$ cd ibmichroot
$ chroot_setup.sh chroot_minimal.lst /QOpenSys/ibmichroot_spaces/db2util
$ cp -R /QOpenSys/ibmichroot /QOpenSys/ibmichroot_spaces/db2util/QOpenSys/.
$ chroot /QOpenSys/ibmichroot_spaces/db2util /usr/bin/sh
$ cd /QOpenSys/ibmichroot
$ ./pkg_setup.sh pkg_perzl_gcc-4.8.3.lst
$ gmake -v
GNU Make 4.1
Built for powerpc-ibm-aix5.3.0.0
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[Former user]

Btw, my simple chroot environment is working with gmake

So, would appear we have partial answer, aka, something in your 'more complicated' set-up is leading to an error in the tooling. Well, i assume i don't have access to said 'more complicated' (proprietary litmis i suspect), so i cannot even hazard a guess.

[Aaron Bartell]

Well, i assume i don't have access to said 'more complicated' (proprietary litmis i suspect), so i cannot even hazard a guess.

It actually is just using a vanilla Litmis Space environment that I manually installed pkg_perzl_gcc-4.8.3.lst into. I will figure out the differences between the ibmichroot .lst file and mine.

More of what I was hoping you could comment on is what steps would you take to debug the error given we have no idea what C method it is bombing on. Doing it in the open, like we did last time, will allow many other people to visit this in the future.

Do I just start doing dbx stepi and register commands and blinding look for things that seem out of place?

[Aaron Bartell]

I am inside my chroot using new db2util calling xmlservice with <sh>system </sh> ... no tricks.

I follow now.

Last week I needed a short term solution so I did something similar by putting a "Poor Mans Compile RPG from PASE" project together that uses the Node.js iToolKit. I will change out my poor man's version for the new db2util version once I get everything working as expected.

[Former user]

The science answer ... peek gnumake source code outputs message "virtual memory exhausted" misc.c . A few places misc.c related malloc (multiple places, example below). You could set a object code break in the code and watch them all. YIKES!!!

The speculation answer ... Any event, not likely malloc subsystem is in error, as 5 billion malloc hamburgers served every day (AIX and PASE). So, if malloc really ran out memory, implies something was looping like crazy or size was really, really big request. Anyway ... if loop ... which ... you may see in wrkactjob stack (depending on how fast all running).

void *
xmalloc (unsigned int size)
{
  /* Make sure we don't allocate 0, for pre-ISO implementations.  */
  void *result = malloc (size ? size : 1);
  if (result == 0)
    fatal (NILF, _("virtual memory exhausted"));
  return result;
}

[Former user]

Well, if the above did not put you off the debug in public idea ... here is some more ...

There was also a place in read.c. We can see read.c parse_file_seq is loaded with ifdefs (compile in/out), so not sure acutally compiled into gnumake module, but anything that says 'list of' filenames, etc, environment plays a roll.

      else
        switch (glob (name, GLOB_NOSORT|GLOB_ALTDIRFUNC, NULL, &gl))
          {
          case GLOB_NOSPACE:
            fatal (NILF, _("virtual memory exhausted"));

          case 0:

in this function ...

/* Parse a string into a sequence of filenames represented as a chain of
   struct nameseq's and return that chain.  Optionally expand the strings via
   glob().
   The string is passed as STRINGP, the address of a string pointer.
   The string pointer is updated to point at the first character
   not parsed, which either is a null char or equals STOPCHAR.
   SIZE is how big to construct chain elements.
   This is useful if we want them actually to be other structures
   that have room for additional info.
   PREFIX, if non-null, is added to the beginning of each filename.
   FLAGS allows one or more of the following bitflags to be set:
        PARSEFS_NOSTRIP - Do no strip './'s off the beginning
        PARSEFS_NOAR    - Do not check filenames for archive references
        PARSEFS_NOGLOB  - Do not expand globbing characters
        PARSEFS_EXISTS  - Only return globbed files that actually exist
                          (cannot also set NOGLOB)
        PARSEFS_NOCACHE - Do not add filenames to the strcache (caller frees)
  */

void *
parse_file_seq (char **stringp, unsigned int size, int stopchar,
                const char *prefix, int flags)

[Former user]

I don not have time to look into this now (vacation), so here is a debug trick for you.

You can script dbx like this ...

#!/opt/freeware/bin/python
# syntax: pdbxwtach gmake
import sys, os, time, fcntl
import subprocess
proc = subprocess.Popen(['dbx','-d 100','-I .','-I ..',sys.argv[1]], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
proc.stdin.write('stop in malloc\n')
proc.stdin.write('cont\n')
while True:
  # get output from process "Something to print"
  nextline = proc.stdout.readline()
  if 'internal error' in nextline:
    continue
  sys.stdout.write(nextline)
  sys.stdout.flush()
  if 'execution completed' in nextline:
    exit(0)
  # write 'a line\n' to the process
  if 'virtual' in nextline:
    print "****"
    proc.stdin.write('quit\n')
  if 'stopped' in nextline:
    print "****"
    proc.stdin.write('print $r3\n')
    proc.stdin.write('cont\n')

However when i tried this on the green screen, gmake worked ok, aka, does not put up the virtual memory message.

                             /QOpenSys/usr/bin/-sh

 > cd /tmp
   $
 > pdbxwatch /opt/freeware/bin/gmake
   Type 'help' for help.
   reading symbolic information ...
   [1] stop in malloc
   [1] stopped in malloc at 0xd011c7e0
   ****
   0xd011c7e0 (malloc)    9421ffb0        stwu   r1,-80(r1)
   0x000002a0
   [1] stopped in malloc at 0xd011c7e0 ($t1)
   ****
   0xd011c7e0 (malloc)    9421ffb0        stwu   r1,-80(r1)
   0x00001400
   [1] stopped in malloc at 0xd011c7e0 ($t1)
   ****
   0xd011c7e0 (malloc)    9421ffb0        stwu   r1,-80(r1)
   0x0000000c
   [1] stopped in malloc at 0xd011c7e0 ($t1)
   ****
   0xd011c7e0 (malloc)    9421ffb0        stwu   r1,-80(r1)
   0x00000188
   [1] stopped in malloc at 0xd011c7e0 ($t1)
   ****

[Former user]

I am not going to chase. To wit, I am using gmake for many projects in a chroot and works fine. Aka, 1 billion hamburgers served by gmake, so i am not motivated to look for edge effects most likely user environment or usage errors.

Also not sure this belongs in ibmichroot. This project loads stuff, where we check stuff works at all (my chroots work fine), then user is on their own really.

I am closing the issue.

[Aaron Bartell]

For the archives...

Turns out gmake -version works fine in QP2TERM with gmake v4.0 and v4.2.1 but ~not~ with gmake v4.1. Not taking the time to learn why, we just know v4.1 doesn't work in all circumstance.

I am going to update gmake in our xxxx.lst configs to use 4.2.1 instead of 4.1.