Fixture finalizer failure causes other finalizers to not be run

Issue #287 on hold
Andreas Pelme
created an issue

When tearing down fixtures and one finalizer (registered via request.addfinalizer) fails (with an uncaught exception or with pytest.fail) - no other finalizers will be run.

Of course - the fixture finalizer could be fixed to not throw exceptions and the problem would go away.

However, I find it useful to be able to make assertions in the fixture teardown. I have a stubbing fixture (similar to pytest's monkeypatch). It checks that the stubbed methods actually was called during the tests, and triggers assertions otherwise. When this happens in one test case, it causes database fixture and all other kinds of failures for all other tests.

I am not sure this is a valid use case for fixtures, if not, feel free to close it. :)

Comments (6)

  1. Holger Krekel

    Independently of whether it's a good idea to do post-condition checking in teardowns (not really IMO), i wonder if we should guard each call of a finalizer. Not running finalizers due to an early failing one can bork the whole test run more than neccessary.

  2. Andreas Pelme reporter

    (I am not sure it is a good idea either, just playing around with different approaches at the moment.)

    It can be a bit surprising that the outcome of the test run is different because of the implicit fixture finalizer order.

  3. Peter Feiner

    I was also surprised to see the behavior that Andreas reported. The interaction with the tmpdir fixture is particularly painful. If tmpdir isn't "finalized" because an earlier finalizer failed, then the next test will use the previous test's tmpdir!

    I came across this issue when a finalizer threw an exception due to a bug in the test (the finalizer tried to wait() on a pid that the test had erroneously waited on). The bug in the test was non-deterministic and the probability of it appearing was only non-infinitesimal when 10+ tests were running in parallel using xdist. Hence, occasionally during heavy test runs I'd see a bunch of tests fail because they got very confused by files in their tmpdirs that had been left around by other tests. If a fresh tmpdir had been provided every time, then I would have just seen the test with the buggy wait() fail and presumably fixed it much sooner ;-)

    Here's a small test to demonstrate the problem with tmpdir:

    import pytest
    
    @pytest.fixture(scope='function')
    def fixture(request, tmpdir):
            print 'setup'
            def death():
                    print 'teardown'
                    raise Exception('death cometh')
            request.addfinalizer(death)
            return tmpdir
    
    def test_a(fixture):
            print 'test_a running with', fixture
    
    def test_b(fixture):
            print 'test_b running with', fixture
    

    And the output:

    $ py.test simple_test.py  -v --capture=no
    === test session starts ===
    platform linux2 -- Python 2.6.6 -- pytest-2.3.5 -- /usr/bin/python
    plugins: xdist
    collected 2 items
    
    simple_test.py:12: test_a setup
    test_a running with /tmp/pytest-221/test_a0
    PASSEDteardown
    
    simple_test.py:12: test_a ERROR
    simple_test.py:16: test_b test_b running with /tmp/pytest-221/test_a0
    PASSED
    
    === ERRORS ===
    ___ ERROR at teardown of test_a ___
    
        def death():
                print 'teardown'
    >           raise Exception('death cometh')
    E           Exception: death cometh
    
    simple_test.py:8: Exception
    === 2 passed, 1 error in 0.12 seconds ===
    
  4. Log in to comment