Handling of SystemExit if different pid (after os.fork)

Issue #658 on hold
Torsten Landschoff
created an issue

Hi *,

I am trying to use our self written ForkServer (something like https://hg.python.org/cpython/file/c917ba25c007/Lib/multiprocessing/forkserver.py but for python2) in unit tests with py.test.

Background

It turned out that using os._exit in each child process forked by the ForkServer is not a good idea, because each child process assumes to be a real python process with atexit.register working and correct cleanup of global resources. This causes a deviation between Windows and Linux as well, as we use multiprocessing to start child processes on Windows where atexit and friends work fine of course.

Now the ForkServer is basically a zygote process and is started really early in our application setup code. Therefore we can switch to using sys.exit() just fine, nothing will catch the SystemExit exception from unwinding.

Problem

Now this change kills a lot of unit tests because the SystemExit exception propagates to py.test when raised from a fork server created inside a unit test. Obviously using sys.exit here is probably not a good idea, because this will run cleanups from a lot of stuff (other tests, pytest itself etc.).

So what I would like to do is to wrap each test function and catch SystemExit there. If SystemExit comes from a different process id that the original pid, this should be mapped to os._exit(), otherwise the SystemExit exception should be propagated as usual.

I think this would even be the correct course of action in pytest proper.

Unfortunately I can not for the life of me figure out how to do this. I tried to implement the hook pytest_runtest_call in my conftest.py but this only causes the an additional execution of item.runtest() instead of replacing the original implementation.

Example Code

The attached test in test_fork.py illustrates the problem. Running it I get the following output:

(env)torsten@horatio:~/pytest-fork/bug-fork$ py.test -vrsx
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /home/torsten/pytest-fork/env/bin/python
collected 1 items 

test_fork.py::test_can_test_fork PASSED

=========================== 1 passed in 0.01 seconds ===========================


=================================== FAILURES ===================================
______________________________ test_can_test_fork ______________________________

    @pytest.mark.skipif(not hasattr(os, "fork"),
            reason="os.fork is missing, I need a real operating system")
    def test_can_test_fork():
        pid = os.fork()
        if not pid:
>           sys.exit()
E           SystemExit

test_fork.py:9: SystemExit
=========================== 1 failed in 0.02 seconds ===========================
(env)torsten@horatio:~/pytest-fork/bug-fork$ 

What I would have expected is successful termination.

Attempted patch

The attachment catch_system_exit.diff contains a patch that fixes the problem for me. Your mileage may vary, applying this everywhere might not be such a good idea.

Unfortunately I can not easily replace pytest on all our build machines with a custom build. I'd really like to stay with a released version and add this behaviour via conftest. Hints greatly appreciated.

Comments (3)

  1. Torsten Landschoff reporter

    What I just noticed: another change must be made in our case so that os.fork() in fixture functions works as well. We create the ForkServer there which does os.fork

    Event after patching call_fixture_func to accept SystemExit, our tests still kill my machine with an avalanche of py.test processes.

  2. Log in to comment