py.test fails to call teardown_module under some conditions

Issue #78 resolved
Jeff Stearns
created an issue

Under some conditions, py.test fails to call teardown_module. This can break subsequent test runs.

Consider this contrived example:

def setup_module(mod): launch_foo_daemon() x = 0/0 # Provoke an exception

def teardown_module(mod): stop_foo_daemon()

When py.test runs this code, it will call setup_module, but will fail to call teardown_module. This is an error. It leaves the foo daemon running in the background, which will interfere with subsequent test runs.

py.test should call teardown_module even if setup_module fails.

Comments (9)

  1. Holger Krekel repo owner

    In this example:

    def setup_module(mod):
        from xyz import startdaemon 
        mod.daemon = startdaemon() 
    
    def teardown_module(mod):
        mod.daemon.stop()
    

    if import 'xyz' fails - do you want this to cause two errors? I've had people arguing exactly the other way round than how you do and say that teardown should not run.

    I personally believe that there is no one "true" solution and that it is better to use funcargs for functional-testing setups. The mechanisms there are much more explicit about setup and teardown (see e.g. "cached_setup" in the funcarg docs). And they can be used to do setup across a whole set of test modules uniformly.

    cheers, holger

  2. Jeff Stearns reporter

    Holger, I propose this:

    • teardown_module is called if setup_module is called.
    • In cases where setup_module raises an exception, py.test should catch the exception, call teardown_module, and then re-raise the original exception.
    • In cases of a double-fault (failure in both setup_module and teardown_module), the first exception is the most relevant, so it's the one that py.test should report.

    Thanks for the pointer to cached_setup. It looks like it might be helpful, but the documentation doesn't describe how the teardown argument is handled when setup fails. If py.test is designed so that a failure in setup_module prevents teardown_module from being called, then I imagine that a failure in cached_setup.setup() would prevent cached_setup.teardown() from being called.

  3. Holger Krekel repo owner
    • changed status to open

    regarding cached_setup and funcargs: you are right that teardown will not be called if setup fails. However, this is done per-function-argument so it's very specific. Also you have the "request.addfinalizer(function)" lower-level method - and this will be called even if the funcarg-factory call fails as a whole. Semantics are thus much more controlable with funcargs.

    In the usual setup_module (or setup_class/method) often several things are setup and a partial failure and not calling to teardown_module will leave resources un-finalized, indeed. As described funcargs have finer-grained possibilities to prevent irritations.

    I just checked: nosetests and unittest (Python2.6) both do not call teardown in case of a setup failure. With py.test i just experimented and it seems it's a one-line change to call teardown in any case - it would nicely report both tracebacks. see http://paste.pocoo.org/show/168612/ Surpressing the teardown one i'd rather not like to do in any case.

    I just posted to the py-dev mailing list (are you subscribed, btw?) and the TIP mailing list to get feedback on your suggestion.

    cheers, holger

  4. Jeff Stearns reporter

    The page at http://codespeak.net/py/dist/test/customize.html#project-specific-local-or-named-global-plugins says:

    Each test item is usually executed by calling the following three hooks:

    • pytest_runtest_setup(item)
    • pytest_runtest_call(item)
    • pytest_runtest_teardown(item)

    If pytest_runtest_setup fails then pytest_runtest_teardown will be called but not pytest_runtest_call.

    This seems odd. A test teardown function is always called, but a module teardown function is not. The difference is surprising, and it seems rather arbitrary.

    If nothing else, the different rules probably deserve some documentation. (I'll bet that many folks don't realize that module teardown and test teardown follow different rules.)

  5. Holger Krekel repo owner
    • changed status to new

    Not true, teardown_function/method is not called currently if setup fails.

    FYI there are two levels here: one is general test item setup and test items are not neccessarily python test functions - you can have text files (e.g. ReST syntax checks) etc. The other is how python test function setup and teardown is handled. This is originally motivated from unittest.py semantics. So it's true that the internal test run protocol always calls teardown - it's another place in the code that decides how to use those hooks to implement the python test semantics. Which is the reason it's only a one-line change basically :)

    that being said, I meanwhile i got feedback from people also encouraging your suggested change. So i am going to implement it in a second, need to write a test first ...

    many thanks for your efforts in pushing and discussing this!

    cheers, holger

  6. Holger Krekel repo owner

    fix issue78 - now python-level teardown functions are now called even if the setup failed. Important detail: if the setup raises a Skipped exception, teardown will not be called. This helps to avoid breaking setup_module/class that performs a skip - it would otherwise internally be considered as a "successful" setup in order to have teardown called later. I guess it also makes sense to treat Skip specially because it is unlikely a teardown should be called if a Skip was raised on setup.

    In any case, failing setups and teardowns will be reported separately.

    f0035a36f714

  7. Holger Krekel repo owner

    again addresses issue78 : we now call teardown also if setup raised a Skipped exception. I also made sure, setup_module/class will only be called once - before they'd be call again and again if they raise an error or a skip - for each test in their scope.

    b27a62c65746

  8. Log in to comment