Adding external tests to a collection

Issue #421 on hold
Trevor Bekolay
created an issue

It would be nice if it were possible to run tests from outside of the current package when running py.test on the current package. A possible use case for this functionality would be to have a common set of tests for many packages implementing a certain protocol or standard. Another use case is for testing pluggable features (this was the the origin of the idea; see this SO question).

Two things are necessary for this to happen. First, you must get a set of tests contained in another package. Second, you must add that set of tests to the current package.

For getting access to external tests, you can do something similar to what occurs when doing py.test --collect-only and providing the path to the external package. However, it should additionally be possible to disable all reporting, and to get programmatic access to the tests collected by that run.

For adding external tests to the current collection, one possibility is pytest_collection_modifyitems. If the tests can be collected programmatically they can just be added to the items list. However, a more explicit hook like pytest_collection_addexternalitems or something might be preferable.

Since I'd like this for one of my projects, I'm motivated to implement this, but it'd be best to discuss a nice API for doing this before starting an implementation.

Comments (5)

  1. Holger Krekel

    Thanks for the initiative. One design issue is how to "override" or "substitute" fixtures. Supose you want to run tests found in somepath/test_base.py and tests there need a fix1 fixture which is defined in a somepath/conftest.py file. We then want to be able to have our "inheriting" project define fix1 and have it used for those tests (that's kind of the whole point). We can not simply ignore the other conftest file because it may define other fixtures which somepath/test_base.py needs. I guess we would need to have something in the API that allows to say use fix1 from here for these tests.

  2. Trevor Bekolay reporter

    Right, that's a good point, it's very likely (certain?) that there will be some conflicting fixtures. How is this handled right now within a package? I just tried defining a funcarg in a test file and in conftest.py: the one in the test file took precedence over the one in conftest.py, and it worked as expected. Could there be a similar precedence order in which the fixtures in the current package (the one that first invoked pytest) take precedence over those in the 'remote' package? So, however we collect tests from the remote package, it could start off with the configuration of the local package and only add fixtures, not replace existing ones.

  3. Holger Krekel

    Right now, fixtures are looked up according to "source order", meaning first in the class of the test, then the module, then the next conftest, (the next conftest) and finally global plugins. The internal data structures are tied to this source/file order. For the inheritance/base scheme we would need to modify/extend the lookup logic i guess. It is implemented in the FixtureManager class in _pytest/python.py, see the doc string.

  4. Log in to comment