Nginx and PHP-FPM error child exited on signal 11 (SIGSEGV)

[Chris Sears]

I have spent a few days scouring the internet and still am not able to get this issue resolved. Hoping the IBMi experts can chime in with some direction. I am on IBMi V7R3.

I can get a basic PHP page with phpinfo() to display successfully with Nginx and PHP-FPM, however when I try a database connection using the newly installed ODBC driver I get a “502 Bad Gateway” response with the error logs below:

Nginx error:

upstream prematurely closed connection while reading response header from upstream

PHP-FPM error:

WARNING: [pool www] child #### exited on signal 11 (SIGSEGV) after 24.209474 seconds from start

‌

One suggestion online was to enable a Core dump, but I am not sure what other configurations on the IBMi are needed to get this enabled. In the php-fpm.conf file I set rlimit_core = unlimited but the PHP-FPM logs are still not showing the Core dump info. The log should update to “… child #### exited on signal 11 (SIGSEGV - core dumped)“, but I cannot seem to get the Core dump enabled. Any suggestions? Is this possible on the IBMi?

‌

The other online suggestions for this issue/error were the session save_path permissions, or certain PHP extensions causing issues, but I have eliminated these suggestions as possible solutions thru troubleshooting and enabling/disabling, etc…

Let me know what other information that would be helpful in troubleshooting.

‌

[Chris Sears]

I did find a file named “core” in the same directory with my php script. So even though the log did not indicate “core dumped” I do see the core file.

I installed ‘gdb’ package via yum, and got the backtrace below:

Core was generated by `php-fpm'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0900000006867524 in std::locale::operator=(std::locale const&) () from /QOpenSys/pkgs/lib/libstdc++.so.6(shr_64.o)
(gdb) bt
#0 0x0900000006867524 in std::locale::operator=(std::locale const&) () from /QOpenSys/pkgs/lib/libstdc++.so.6(shr_64.o)
#1 0x090000000686d134 in std::ios_base::_M_init() () from /QOpenSys/pkgs/lib/libstdc++.so.6(shr_64.o)
#2 0x09000000067d3128 in std::basic_ios<char, std::char_traits<char> >::init () from /QOpenSys/pkgs/lib/libstdc++.so.6(shr_64.o)
#3 0x0900000007936d30 in global constructors keyed to 65535_0__QOpenSys_jenkins_workspace_odbc_core_source_picococo.cpp_AE91A18C_0xe63b5225263dd1
() from /QOpenSys/pkgs/lib/libcwbcore.so
#4 0x0900000007987724 in _GLOBAL__FI_libcwbcore_so () from /QOpenSys/pkgs/lib/libcwbcore.so
#5 0x0900000007987864 in _GLOBAL__AIXI_libcwbcore_so () from /QOpenSys/pkgs/lib/libcwbcore.so
#6 0x0900000007812234 in _GLOBAL__FI_libcwbodbc_so () from /QOpenSys/pkgs/lib/libcwbodbc.so
#7 0x09fffffff0002668 in mod_init1 () from /QOpenSys/sbin/usla
#8 0x09fffffff0003848 in usl_init_mods () from /QOpenSys/sbin/usla
#9 0x09fffffff0002204 in uload () from /QOpenSys/sbin/usla
#10 0x09000000002bc198 in load1 () from /QOpenSys/usr/lib/libc.a(shr_64.o)
#11 0x09000000002bd514 in load () from /QOpenSys/usr/lib/libc.a(shr_64.o)
#12 0x0900000000315a70 in loadAndInit () from /QOpenSys/usr/lib/libc.a(shr_64.o)
#13 0x090000000035260c in dlopen () from /QOpenSys/usr/lib/libc.a(shr_64.o)
#14 0x090000000622089c in vm_open () from /QOpenSys/pkgs/lib/libltdl.so.7(shr_64.o)
#15 0x090000000621a324 in tryall_dlopen () from /QOpenSys/pkgs/lib/libltdl.so.7(shr_64.o)
#16 0x090000000621cbfc in try_dlopen () from /QOpenSys/pkgs/lib/libltdl.so.7(shr_64.o)
#17 0x090000000621d368 in lt_dlopenadvise () from /QOpenSys/pkgs/lib/libltdl.so.7(shr_64.o)
#18 0x090000000621d530 in lt_dlopen () from /QOpenSys/pkgs/lib/libltdl.so.7(shr_64.o)
#19 0x09000000061bf968 in odbc_dlopen () from /QOpenSys/pkgs/lib/libodbc.so.2(shr_64.o)
#20 0x09000000061c0278 in __connect_part_one () from /QOpenSys/pkgs/lib/libodbc.so.2(shr_64.o)
#21 0x09000000061c7350 in SQLDriverConnect () from /QOpenSys/pkgs/lib/libodbc.so.2(shr_64.o)
#22 0x0900000006420e50 in pdo_odbc_handle_factory () from /QOpenSys/pkgs/lib/php-7.3/extensions/pdo_odbc.so
#23 0x090000000625697c in zim_PDO_dbh_constructor () from /QOpenSys/pkgs/lib/php-7.3/extensions/pdo.so
#24 0x00000001000d12b8 in execute_ex (ex=0x9001000a0ac8188 <_ZN17PiSvRuntimeConfig4cfg_E+456>)
at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/Zend/zend_execute.c:52397
#25 0x00000001000d3fc8 in zend_execute (op_array=0x9001000a0ac8188 <_ZN17PiSvRuntimeConfig4cfg_E+456>, return_value=0xfffffffffff48f0)
at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/Zend/zend_execute.c:60939
#26 0x00000001000370b0 in zend_execute_scripts (type=-1599307384, retval=0xfffffffffff48f0, file_count=0)
at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/Zend/zend.c:1568
#27 0x0000000100150c7c in php_execute_script (primary_file=0xffffffffffff650) at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/main/main.c:2639
#28 0x0000000100001480 in main (argc=-2144116105, argv=0x180336277) at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/sapi/fpm/fpm/fpm_main.c:1950

‌

I then entered the commands below, but didn’t get and details:

(gdb) print (char *)(executor_globals.function_state_ptr->function)->common.function_name
There is no member named function_state_ptr.
(gdb) print (char *)executor_globals.active_op_array->function_name
There is no member named active_op_array.
(gdb) print (char *)executor_globals.active_op_array->filename
There is no member named active_op_array.

I am exploring around at the frames in the backtrace currently. Let me know if there are any suggestions or indicates of what the problem may be.

[Korinne Adler]

What version of the ODBC driver are you running? rpm -qi ibm-iaccess

[Chris Sears]

I am using ibm-iaccess 1.1.0.14-0, from a recently downloaded and reinstalled file ibm-iaccess-1.1.0.14-0.ibmi7.2.ppc64

[Korinne Adler]

Can you ensure that you have the latest libstdcplusplus6 package installed? There was a bug a while ago because we shipped the non-threaded versions and you can’t mix and match.

‌

If that is up to date, can you also try the 1.1.0.13 driver version and see if it has the same issue?

[Chris Sears]

I am currently on libstdcplusplus6 version 6.3.0-15. There is a newer version available at version 6.3.0-25. I tried the yum update command and got the error messages below.

Downloading Packages:

ftp://public.dhe.ibm.com/software/ibmi/products/pase/rpms/repo/ppc64/libstdcplusplus6-6.3.0-25.ibmi7.2.ppc64.rpm: [Errno 14] FTP Error 451 - server did not report OK, got 451

Trying other mirror.

Error Downloading Packages:

libstdcplusplus6-6.3.0-25.ppc64: failure: ppc64/libstdcplusplus6-6.3.0-25.ibmi7.2.ppc64.rpm from ibm: [Errno 256] No more mirrors to try.

‌

Would this version of libstdcplusplus6 (6.3.0-15) have that bug you mentioned, with non-threaded versions?

Can you let me know where to download the older version for the ibm-iaccess ODBC driver (1.1.0.13)? I can give that a try next.

Thanks for all your help, I appreciate it!

[Chris Sears]

The full backtrace shows the information on frame 28, with a couple messages about “Cannot access memory…”.

#28 0x0000000100001480 in main (argc=-2144116105, argv=0x180336277) at /HOME/QSECOFR/rpmbuild/BUILD/php-7.3.14/sapi/fpm/fpm/fpm_main.c:1950
        primary_script = 0x700000000003380 <error: Cannot access memory at address 0x700000000003380>
        __bailout = {-1, -1, -1, -1, 4294971288, 1152921504606841472, 6443195896, 6443388832, -1, 0, 0, 0, 0, 0, 0, 0, -1, 0, 1, 0, 6442454672,
          6442454960, 1, 1152921504606845184, 1152921504606844768, 6, 0 <repeats 19 times>, 2594077940715686000, 8, 1, 6443356696,
          648535941211936184, 648535941213278296, 24, 1152921504606843552, 648535941213295936, 1152921504606843520, 648535941213828360,
          1152921504606843536, 648535941212342384, 24, 0, 648535941212270576, 0, 1152921504606843584, 272756745, 1152921504606843600,
          648535941213295936, 648535941213278284, 648535941213828360, 0, 648535941211803072, 1152921504606843696, 648535941211888872,
          648518346344007296, 648535941213296888, 1152921504606843696, 128, 648535941212271392, 648535941211803064, 1152921504606843696,
          648535941211888864, 648518346344638976, 648535941212342384, 648518346345256128, 648535941213299784, 8}
        exit_status = 0
        use_extended_info = -2144115867
        file_handle = {handle = {fd = 117440512, fp = 0x700000000077000, stream = {handle = 0x700000000077000, isatty = 0, mmap = {len = 2584,
                pos = 0, map = 0x0, buf = 0x700000000233000 <error: Cannot access memory at address 0x700000000233000>, old_handle = 0x0,
                old_closer = 0x0}, reader = @0x1800a0798: 0x100163bf0 <_php_stream_read>,
              fsizer = @0x1800a0780: 0x10014de90 <php_zend_stream_fsizer>, closer = @0x1800a07c8: 0x10014de10 <php_zend_stream_mmap_closer>}},
          filename = 0x700000000003000 <error: Cannot access memory at address 0x700000000003000>, opened_path = 0x0, type = ZEND_HANDLE_MAPPED,
          free_filename = 0 '\000'}
        orig_optind = 0
        orig_optarg = 0x0
        max_requests = 0
        requests = 0
        request = 0x18033b670
        fpm_config = 0x0
        fpm_prefix = 0x30 <error: Cannot access memory at address 0x30>
        fpm_pid = 0x0
        test_conf = -2144115994
        force_daemon = -2144116010
        force_stderr = 0
        php_information = 12288
        php_allow_to_run_as_root = 0
        __func__ = 0x180000850 <_fpmmain.rw_> ""
        len = 134225921

‌

[Korinne Adler]

To fix the yum issue, you’ll need to follow the steps here: https://ibmi-oss-docs.readthedocs.io/en/latest/yum/README.html?highlight=http#switching-from-ftp-to-http-s

‌

That should definitely fix your problem. I believe 6.3.0-17 fixed the issue.

‌

[Chris Sears]

It’s working now!!!

I successfully updated to the new ibm.repo, which in turn allowed me to successfully update libstdcplusplus6 to version 6.3.0-25.

I restarted Nginx and PHP-FPM (to be safe), and I am no longer getting “child #### exited on signal 11 (SIGSEGV)” errors.

I can now continue with the ODBC connection setup and testing.

Thank you for pointing me to the fix!

[Chris Sears]

Update was needed to package libstdcplusplus6