npm crashes with assertion in DelayedTaskScheduler

[Thomas Raddatz]

Command "npm -v" crashes with the following assertion when executed from a PuTTY or Git Bash session:

bash-4.4$ node -v
v10.16.3
bash-4.4$ npm -v
[29172]: ../src/node_platform.cc:58:std::unique_ptr<unsigned int> node::BackgroundTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
IOT/Abort trap (core dumped)

The commands (node and npm) worked fine in the past.

IBM support is out of ideas and directed me to Bitbucket to create a ticket:

“From SWMA support we are at the end of idea's so we suggest you to take the next step and that is creating a bitbucket Issues Request:

https://www.ibm.com/support/pages/node/1102461

Then provide those devs with all the data we previously created and then include this link:

https://github.com/nodejs/help/issues/1754

It seems to derive from the same issue. This is something on the PC with threading. Possibly the SSH client has limited threads etc. We are unsure but the devs will need to review and invest this.”

Thank you for helping me with this issue.

Regards,

Thomas.

‌

[Kevin Adler]

What’s the output of rpm -qi nodejs8 nodejs10 nodejs12?

[Thomas Raddatz]

The output of rpm -qi nodejs8 nodejs10 nodejs12 is:

‌

login as: raddatz
[e-mail]'s password:
$ rpm -qi nodejs8 nodejs10 nodejs12
package nodejs8 is not installed
package nodejs10 is not installed
package nodejs12 is not installed
$ PATH=/QOpenSys/pkgs/bin:$PATH
$ export PATH
$ rpm -qi nodejs8 nodejs10 nodejs12
package nodejs8 is not installed
Name        : nodejs10
Version     : 10.16.3
Release     : 1
Architecture: ppc64
Install Date: Tue Oct 29 12:56:49 2019
Group       : Unspecified
Size        : 82601299
License     : MIT
Signature   : (none)
Source RPM  : nodejs10-10.16.3-1.src.rpm
Build Date  : Tue Aug 27 04:43:12 2019
Build Host  : UT25BP3
Relocations : (not relocatable)
URL         : https://www.nodejs.org
Summary     : Node.js JavaScript Runtime
Description :
Node.js is a JavaScript runtime built on Chrome V8 JavaScript engine. Node.js
uses an event-driven, non-blocking I/O model that makes it lightweight and
efficient.
Name        : nodejs12
Version     : 12.8.1
Release     : 2
Architecture: ppc64
Install Date: Mon Nov  4 12:04:36 2019
Group       : Unspecified
Size        : 98828752
License     : MIT
Signature   : (none)
Source RPM  : nodejs12-12.8.1-2.src.rpm
Build Date  : Thu Oct 31 17:00:33 2019
Build Host  : UT25BP3
Relocations : (not relocatable)
URL         : https://www.nodejs.org
Summary     : Node.js JavaScript Runtime
Description :
Node.js is a JavaScript runtime built on Chrome V8 JavaScript engine. Node.js
uses an event-driven, non-blocking I/O model that makes it lightweight and
efficient.
$ npm -v
[41170]: ../src/node_platform.cc:63:std::unique_ptr<unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
Abort - core dumped

‌

[Kevin Adler]

This is very strange, as we would normally only see this in QSH when QIBM_MULTI_THREADED is not set to Y.

Xu Meng, can you please investigate why Node.js is failing to create a thread in SSH?

[mengxumx]

Currently I still can not recreate this issue on my local Windows system. But it seems to be a resource limit issue. Would you try to call `ulimit -a` in both putty and 5250 terminal? And collect the system limit info?

-bash-4.4$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2000
pipe size            (512 bytes, -p) 64
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited

And you could also try to install the previous versions of Node.js to check if this issue start to occur in a specific version.

yum install -y nodejs8
PATH=/QOpenSys/pkgs/lib/nodejs12/bin:$PATH
node -v && npm -v

‌

[Thomas Raddatz]

I created various screenshots to provide the requested information. Their names should describe the content. I also captured the output of qp2term when installing node 8. Apparently command ‘ulimit -a' is not supported on the IBM i. Therefore I executed 'ulimit’ instead. Files uploaded:

ulimit-GitBash.png
ulimit-PuTTY.png
ulimit-PuTTY-node-8.png
ulimit-qp2term.png
ulimit-strqsh.png
qp2term-install-node-8.txt

[Thomas Raddatz]

See comment 2019-11-12, 08:20.

[Kevin Adler]

What version of IBM i are you running on?

[Thomas Raddatz]

We are on 7.3.

[mengxumx]

I still can not recreate the issue. But recently I encounterred a similar issue when using idb-connector – Error: SQLSTATE=57011 SQLCODE=-904 Resource limit exceeded. When I manually IPLed my system and cleared all job queues, then the issue was resolved.

It seems that some jobs allocated too many thread resources (the SSH daemon service SSHD?). I have no idea how to identify that. Maybe we need to consult some experts to check some system health monitor data.

‌

[Thomas Raddatz]

Our IBM i did an IPL on last Monday (11.11.2019). As far as I know there are no processes using SSH. The developers who want to learn and check Node.js stopped their self-education or moved to qp2term, because of the current issue.

I am a developer and not responsible for maintaining our IBM i system. So I asked our ‘Infrastructure Services’ team for permission to restart the SSH server. I wait for their response.

Thank you for your hard work so far.

[Geert Braem]

Xu, Kevin, not sure why you seem to consider the threads, or resource limit issue to be limited on the IBMi? As for what i know there is no prove of that. The issues are on the Putty using ssh or bash session on the Windows PC. Also the qshell and qp2term does not fail when running on the 5250 session directly. Just thinking along here.

[mengxumx]

@Thomas Raddatz You may also want to check the PTF status of product 5733SC1(OpenSSH/OpenSSL) – DSPPTF LICPGM(5733SC1)

It is recommended to apply the latest SC1 PTF SI71076.

[Thomas Raddatz]

This PTF is missing on our IBM i system. I let our Infrastructure Services team know about that to get that PTF installed.

[Kevin Adler]

Thomas, what is the shell being used when logging in via SSH? Can you give the output of grep ibmpaseforishell /QOpenSys/QIBM/UserData/SC1/OpenSSH/etc/sshd_config please. Also, the output of env when run from SSH would be good too.

[Thomas Raddatz]

Kevin, the grep command does not return anything, because ‘ibmpaseforishell' could not be found in ‘sshd_config’. As far as I can see, there are only three active properties in 'sshd_config’:

• Protocol 2
• UsePrivilegeSeparation no
• Subsystem sftp /QOpenSys/QIBM/ProdData/SC1/OpenSSH/libexec/sftp-server

For details, please refer to the screenshots and sshd_config.

[Thomas Raddatz]

Output of 'grep' command.

[Thomas Raddatz]

Hi Kevin an Xu,

Good news for this PMR: SSH works fine again.

[Thomas Raddatz]

But it seems that the actual problem was not the missing PTF but a misconfigured job description of our SSH subsystem. Our ‘Infrastructure Services’ team applied the PTF and, at the same time, changed parameter ALWMLTTHD of the job description of the SSH subsytsem from *NO to *YES. After that SSH worked fine. Then they changed that parameter back to *NO and SSH failed working again. Sadly we could not find an answer for the question why SSH worked fine in the past with parameter ALWMLTTHD set to *NO. The ‘Infrastructure Services’ team swears that parameter ALWMLTTHD was set *NO from the beginning of our SSH subsystem.

Another thing worth to mention is that the IBM technical note that describes how to setup a SSH subsystem does not mention parameter ALWMLTTHD at all. The default of parameter ALWMLTTHD of command CRTJOBD is *NO as stated in document https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_73/cl/crtjobd.htm.

The technical note that describes how to create a SSH subsystem is located here: https://www.ibm.com/support/pages/starting-ssh-daemon-dedicated-subsystem-environment

Maybe the technical note should be updated.

Thank you very much to all who helped solving the problem.

[Geert Braem]

Thomas, I made a correction on that document:

This correction will be seen in a few days, have added to check the ALWMLTTHD , thanks.

[Thomas Raddatz]

Great, and thanks again for your help. It is greatly appreciated.