cffi-1.9.1: segmentation fault during self test on NetBSD

Issue #321 resolved
_wiz_
created an issue

When running the self tests on NetBSD 8.99.1/amd64, there are segfaults:

cd /scratch/devel/py-cffi/work/cffi-1.10.0 && py.test-3.6 -v
============================================================================ test session starts =============================================================================
platform netbsd8 -- Python 3.6.1, pytest-3.1.2, py-1.4.33, pluggy-0.4.0 -- /usr/pkg/bin/python3.6
cachedir: .cache
rootdir: /scratch/devel/py-cffi/work/cffi-1.10.0, inifile:
collected 1927 items 

c/test_c.py::test_load_library PASSED
c/test_c.py::test_all_rtld_symbols PASSED
c/test_c.py::test_new_primitive_type PASSED
c/test_c.py::test_inspect_primitive_type PASSED
c/test_c.py::test_cast_to_signed_char PASSED
c/test_c.py::test_sizeof_type PASSED
c/test_c.py::test_integer_types PASSED
c/test_c.py::test_no_float_on_int_types PASSED
c/test_c.py::test_float_types PASSED
c/test_c.py::test_complex_types SKIPPED
c/test_c.py::test_character_type PASSED
c/test_c.py::test_pointer_type PASSED
c/test_c.py::test_inspect_pointer_type PASSED
c/test_c.py::test_pointer_to_int PASSED
c/test_c.py::test_pointer_bool PASSED
c/test_c.py::test_pointer_to_pointer PASSED
c/test_c.py::test_reading_pointer_to_int PASSED
c/test_c.py::test_reading_pointer_to_float PASSED
c/test_c.py::test_cast_float_to_int PASSED
c/test_c.py::test_newp_integer_types PASSED
c/test_c.py::test_reading_pointer_to_char PASSED
c/test_c.py::test_reading_pointer_to_pointer PASSED
c/test_c.py::test_load_standard_library PASSED
c/test_c.py::test_no_len_on_nonarray PASSED
c/test_c.py::test_cmp_none PASSED
c/test_c.py::test_invalid_indexing PASSED
c/test_c.py::test_default_str PASSED
c/test_c.py::test_default_unicode PASSED
c/test_c.py::test_cast_from_cdataint PASSED
c/test_c.py::test_void_type PASSED
c/test_c.py::test_array_type PASSED
c/test_c.py::test_inspect_array_type PASSED
c/test_c.py::test_array_instance PASSED
c/test_c.py::test_array_of_unknown_length_instance PASSED
c/test_c.py::test_array_of_unknown_length_instance_with_initializer PASSED
c/test_c.py::test_array_initializer PASSED
c/test_c.py::test_array_add PASSED
c/test_c.py::test_array_sub PASSED
c/test_c.py::test_ptr_sub_unaligned PASSED
c/test_c.py::test_cast_primitive_from_cdata PASSED
c/test_c.py::test_new_primitive_from_cdata PASSED
c/test_c.py::test_cast_between_pointers PASSED
c/test_c.py::test_alignof PASSED
c/test_c.py::test_new_struct_type PASSED
c/test_c.py::test_new_union_type PASSED
c/test_c.py::test_complete_struct PASSED
c/test_c.py::test_complete_union PASSED
c/test_c.py::test_struct_instance PASSED
c/test_c.py::test_union_instance PASSED
c/test_c.py::test_struct_pointer PASSED
c/test_c.py::test_struct_init_list PASSED
c/test_c.py::test_array_in_struct PASSED
c/test_c.py::test_offsetof PASSED
c/test_c.py::test_function_type PASSED
c/test_c.py::test_inspect_function_type PASSED
c/test_c.py::test_function_type_taking_struct PASSED
c/test_c.py::test_function_void_result PASSED
c/test_c.py::test_function_void_arg PASSED
c/test_c.py::test_call_function_0 PASSED
c/test_c.py::test_call_function_0_pretend_bool_result PASSED
c/test_c.py::test_call_function_1 PASSED
c/test_c.py::test_call_function_2 PASSED
c/test_c.py::test_call_function_3 PASSED
c/test_c.py::test_call_function_4 PASSED
c/test_c.py::test_call_function_5 PASSED
c/test_c.py::test_call_function_6 PASSED
c/test_c.py::test_call_function_7 PASSED
c/test_c.py::test_call_function_20 PASSED
c/test_c.py::test_call_function_21 PASSED
c/test_c.py::test_call_function_22 PASSED
c/test_c.py::test_call_function_23 PASSED
c/test_c.py::test_call_function_23_bis PASSED
c/test_c.py::test_call_function_23_bool_array PASSED
c/test_c.py::test_cannot_pass_struct_with_array_of_length_0 PASSED
c/test_c.py::test_call_function_9 PASSED
c/test_c.py::test_cannot_call_with_a_autocompleted_struct PASSED
c/test_c.py::test_new_charp PASSED
c/test_c.py::test_load_and_call_function PASSED
c/test_c.py::test_read_variable SKIPPED
c/test_c.py::test_read_variable_as_unknown_length_array SKIPPED
c/test_c.py::test_write_variable SKIPPED
c/test_c.py::test_callback [1]   Segmentation fault (core dumped) py.test-3.6 -v
*** Error code 139

In total, I have to disable the following tests so that I see now core dumps:

py.test-3.6 -v -k "not test_callback and not test_a_lot_of_callbacks
and not test_wchar and not test_errno_callback and not
test_functionptr_simple and not test_functionptr_voidptr_return and
not test_functionptr_intptr_return and not
test_functionptr_void_return and not test_cast_functionptr_and_int and
not test_structptr_argument and not test_array_argument_as_list and
not test_array_of_func_ptr and not test_function_pointer and not
test_access_callback and not test_win32_calling_convention_0 and not
test_ffi_callback and not test_array_length_from_constant and not
test_some_integer_type_for_issue73"

This gives me:

<<<moved 3.6MB of output to dump.txt>>>

This is not the same as #303; due to changes in NetBSD and libffi in pkgsrc, all self tests for libffi now pass successfully.

Let me know which of the other self tests failures need their own bug reports. Thanks!

Comments (12)

  1. Armin Rigo

    Unsure what to do with this. Tests about callbacks still segfault, plus some others that are not about callbacks. A large number of tests fail, some of them e.g. because the C++ compiler invoked during some tests is not working like expected, I think. I cannot do anything without logging to a NetBSD machine and trying to track down the segfaults. I'm sorry to say but this is still more likely to be some combination of remaining libffi bugs (cffi's tests pass on a variety of other machines) and some environment issues (like the C++ one). I fear my own interest in tracking down such issues on *BSD is low, sorry. Thanks anyway for reporting the problems here; I could point to this issue from the cffi documentation, at least.

  2. _wiz_ reporter

    If I provide a C++ compiler, I need to deselect one more test, and I end up with:

    ======================================== 6 failed, 1727 passed, 92 skipped, 99 deselected, 3 xfailed, 194 warnings in 294.52 seconds =========================================
    

    The command line is:

    py.test-3.6 -v -k "not test_callback and not test_a_lot_of_callbacks and not test_wchar and not test_errno_callback and not test_functionptr_simple and not test_functionptr_voidptr_return and not test_functionptr_intptr_return and not test_functionptr_void_return and not test_cast_functionptr_and_int and not test_structptr_argument and not test_array_argument_as_list and not test_array_of_func_ptr and not test_function_pointer and not test_access_callback and not test_win32_calling_convention_0 and not test_ffi_callback and not test_array_length_from_constant and not test_some_integer_type_for_issue73 and not test_macro_var_callback"
    

    I'm happy to give you access to a NetBSD machine if you want it.

  3. Armin Rigo

    The next step would be for someone to track down a callback segfault. It is still my guess that this person could reduce the crash to a small, pure-C example that crashes libffi. If it is indeed possible, then this issue is about libffi and not really cffi.

  4. jsonn

    As discussed earlier on IRC, the problem here is the custom closure allocator. The one in libffi itself has the necessary fixes.

    https://raw.githubusercontent.com/NetBSD/pkgsrc/d194723e1a8d41951356aa4170c4822ec41236c5/devel/py-cffi/patches/patch-c_cffibackend.c

    is a forward port of the old branch. From the discussion, it might be that the custom malloc implementation is preferable for SELinux, but I think it has the same fork issues as the upstream implementation in libffi. The patch requires very little changes to support both implementations, i.e. only the calls to ffi_closure_free and ffi_closure_alloc would have to be changed for that.

  5. _wiz_ reporter

    With jsonn's patch applied, I see

    ================================================================== 1824 passed, 99 skipped, 4 xfailed, 200 warnings in 473.53 seconds ==================================================================
    

    Thanks, jsonn!

  6. Armin Rigo

    If you'd like this patch to be applied into CFFI, then it needs some #ifdef NetBSD or equivalent.

    @jsonn I don't understand your comment "I think it has the same fork issues as the upstream implementation in libffi". There is no fork issue because there is no mmap(MAP_SHARED) in the current malloc_closure.h.

  7. Log in to comment