Fixture finalizer failure causes other finalizers to not be run

Issue #287 on hold
Andreas Pelme
created an issue

When tearing down fixtures and one finalizer (registered via request.addfinalizer) fails (with an uncaught exception or with - no other finalizers will be run.

Of course - the fixture finalizer could be fixed to not throw exceptions and the problem would go away.

However, I find it useful to be able to make assertions in the fixture teardown. I have a stubbing fixture (similar to pytest's monkeypatch). It checks that the stubbed methods actually was called during the tests, and triggers assertions otherwise. When this happens in one test case, it causes database fixture and all other kinds of failures for all other tests.

I am not sure this is a valid use case for fixtures, if not, feel free to close it. :)

Comments (6)

  1. Holger Krekel

    Independently of whether it's a good idea to do post-condition checking in teardowns (not really IMO), i wonder if we should guard each call of a finalizer. Not running finalizers due to an early failing one can bork the whole test run more than neccessary.

  2. Andreas Pelme reporter

    (I am not sure it is a good idea either, just playing around with different approaches at the moment.)

    It can be a bit surprising that the outcome of the test run is different because of the implicit fixture finalizer order.

  3. Peter Feiner

    I was also surprised to see the behavior that Andreas reported. The interaction with the tmpdir fixture is particularly painful. If tmpdir isn't "finalized" because an earlier finalizer failed, then the next test will use the previous test's tmpdir!

    I came across this issue when a finalizer threw an exception due to a bug in the test (the finalizer tried to wait() on a pid that the test had erroneously waited on). The bug in the test was non-deterministic and the probability of it appearing was only non-infinitesimal when 10+ tests were running in parallel using xdist. Hence, occasionally during heavy test runs I'd see a bunch of tests fail because they got very confused by files in their tmpdirs that had been left around by other tests. If a fresh tmpdir had been provided every time, then I would have just seen the test with the buggy wait() fail and presumably fixed it much sooner ;-)

    Here's a small test to demonstrate the problem with tmpdir:

    import pytest
    def fixture(request, tmpdir):
            print 'setup'
            def death():
                    print 'teardown'
                    raise Exception('death cometh')
            return tmpdir
    def test_a(fixture):
            print 'test_a running with', fixture
    def test_b(fixture):
            print 'test_b running with', fixture

    And the output:

    $ py.test  -v --capture=no
    === test session starts ===
    platform linux2 -- Python 2.6.6 -- pytest-2.3.5 -- /usr/bin/python
    plugins: xdist
    collected 2 items test_a setup
    test_a running with /tmp/pytest-221/test_a0
    PASSEDteardown test_a ERROR test_b test_b running with /tmp/pytest-221/test_a0
    === ERRORS ===
    ___ ERROR at teardown of test_a ___
        def death():
                print 'teardown'
    >           raise Exception('death cometh')
    E           Exception: death cometh Exception
    === 2 passed, 1 error in 0.12 seconds ===
  4. Log in to comment