testResetAndSolve (test_ts_py.TestTSPython) aborted with core dump

[sagitter]

Hi all.

PETSc-3.12.4, petsc4py-3.12.0 compiled on Fedora 32+ with Python-3.8 and MPICH-3.3.2.

testResetAndSolve fails on PPC64 Little Endian architecture only with this error:

‌

testResetAndSolve (test_ts_py.TestTSPython) ... [0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
python3: src/petsc4py.PETSc.c:319281: __Pyx_PyCFunction_FastCall: Assertion `!PyErr_Occurred()' failed.
Aborted (core dumped)

If individually ran:

‌

PYTHONPATH=../build/lib.linux-ppc64le-3.8 python3 test_ts_py.py -v
testFDColor (__main__.TestTSPython) ... ok
testResetAndSolve (__main__.TestTSPython) ... ok
testSolve (__main__.TestTSPython) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.009s

OK

‌

[Lisandro Dalcin]

There is something odd with these tests. I’m aware something is not wright, but I could never manage to track-down the issue. My guess is that it is somehow related to Python’s garbage collector and the double PETSc+Python refcount handling in petsc4py.

[sagitter]

I obtained a more verbose output with OpenMPI-4.0.3:

[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
python3: src/petsc4py.PETSc.c:319281: __Pyx_PyCFunction_FastCall: Assertion `!PyErr_Occurred()' failed.
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] *** Process received signal ***
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] Signal: Aborted (6)
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] Signal code:  (-6)
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x7fff966d04d8]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 1] /lib64/libc.so.6(gsignal+0x104)[0x7fff964f93d4]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 2] /lib64/libc.so.6(abort+0x180)[0x7fff964d7de8]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 3] /lib64/libc.so.6(+0x3dc90)[0x7fff964edc90]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 4] /lib64/libc.so.6(__assert_fail+0x64)[0x7fff964edd34]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 5] build/lib.linux-ppc64le-3.8/petsc4py/lib/PETSc.cpython-38-ppc64le-linux-gnu.so(+0x25ce60)[0x7fff8458ce60]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 6] /usr/lib64/openmpi/lib/libpetsc.so.3.12(PetscError+0xe4)[0x7fff82f31f34]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 7] /usr/lib64/openmpi/lib/libpetsc.so.3.12(PetscSignalHandlerDefault+0x23c)[0x7fff82f36f0c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 8] /usr/lib64/openmpi/lib/libpetsc.so.3.12(+0x127008)[0x7fff82f37008]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [ 9] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x7fff966d04d8]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [10] /lib64/libpython3.8.so.1.0(PyCode_NewWithPosOnlyArgs+0xe4)[0x7fff961aaf14]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [11] /lib64/libpython3.8.so.1.0(PyCode_New+0x84)[0x7fff9618165c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [12] build/lib.linux-ppc64le-3.8/petsc4py/lib/PETSc.cpython-38-ppc64le-linux-gnu.so(+0x39427c)[0x7fff846c427c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [13] build/lib.linux-ppc64le-3.8/petsc4py/lib/PETSc.cpython-38-ppc64le-linux-gnu.so(+0x3a598c)[0x7fff846d598c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [14] build/lib.linux-ppc64le-3.8/petsc4py/lib/PETSc.cpython-38-ppc64le-linux-gnu.so(+0x3cdc28)[0x7fff846fdc28]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [15] /usr/lib64/openmpi/lib/libpetsc.so.3.12(TSSetFromOptions+0x1950)[0x7fff83d4f250]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [16] build/lib.linux-ppc64le-3.8/petsc4py/lib/PETSc.cpython-38-ppc64le-linux-gnu.so(+0x135988)[0x7fff84465988]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [17] /lib64/libpython3.8.so.1.0(+0xf788c)[0x7fff961c788c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [18] /lib64/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xbd0)[0x7fff961aea30]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [19] /lib64/libpython3.8.so.1.0(+0xf5b1c)[0x7fff961c5b1c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [20] /lib64/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xbd0)[0x7fff961aea30]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [21] /lib64/libpython3.8.so.1.0(+0xf5b1c)[0x7fff961c5b1c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [22] /lib64/libpython3.8.so.1.0(+0x104654)[0x7fff961d4654]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [23] /lib64/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0x75c)[0x7fff961ae5bc]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [24] /lib64/libpython3.8.so.1.0(+0xf5b1c)[0x7fff961c5b1c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [25] /lib64/libpython3.8.so.1.0(_PyEval_EvalFrameDefault+0xbd0)[0x7fff961aea30]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [26] /lib64/libpython3.8.so.1.0(_PyEval_EvalCodeWithName+0x3a4)[0x7fff961ac164]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [27] /lib64/libpython3.8.so.1.0(_PyFunction_Vectorcall+0xf8)[0x7fff961c5608]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [28] /lib64/libpython3.8.so.1.0(+0x10471c)[0x7fff961d471c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] [29] /lib64/libpython3.8.so.1.0(PyVectorcall_Call+0x8c)[0x7fff961d4b8c]
[1fe59000cbb04b6e9c262b70cb8c73c2:00031] *** End of error message ***
Aborted (core dumped)

‌

[sagitter]

This error comes up with Python-3.8 only, just successfully tested with Python-3.9