1. Wieland Hoffmann
  2. pytest


pytest / ISSUES.txt

improve / add to dependency/test resource injection
tags: wish feature  docs

write up better examples showing the connection between
the two.

refine parametrize API
tags: critical feature 

extend metafunc.parametrize to directly support indirection, example:

    def setupdb(request, config):
        # setup "resource" based on test request and the values passed 
        # in to parametrize.  setupfunc is called for each such value.
        # you may use request.addfinalizer() or request.cached_setup ...
        return dynamic_setup_database(val)

    @pytest.mark.parametrize("db", ["pg", "mysql"], setupfunc=setupdb)
    def test_heavy_functional_test(db):

There would be no need to write or explain funcarg factories and
their special __ syntax.

The examples and improvements should also show how to put the parametrize
decorator to a class, to a module or even to a directory.  For the directory
part a conftest.py content like this::

    pytestmark = [
        @pytest.mark.parametrize_setup("db", ...),

probably makes sense in order to keep the declarative nature.   This mirrors
the marker-mechanism with respect to a test module but puts it to a directory

When doing larger scoped parametrization it probably becomes neccessary 
to allow parametrization to be ignored if the according parameter is not
used (currently any parametrized argument that is not present in a function will cause a ValueError). Example:

        @pytest.mark.parametrize("db", ..., mustmatch=False)

means to not raise an error but simply ignore the parametrization
if the signature of a decorated function does not match. XXX is it
not sufficient to always allow non-matches?

allow parametrized attributes on classes

tags: wish 2.4


    @pytest.mark.parametrize_attr("db", setupfunc, [1,2,3], scope="class")
    @pytest.mark.parametrize_attr("tmp", setupfunc, scope="...")
    class TestMe:
        def test_hello(self):
            access self.db ...

this would run the test_hello() function three times with three
different values for self.db. This could also work with unittest/nose
style tests, i.e. it leverages existing test suites without needing
to rewrite them. Together with the previously mentioned setup_test()
maybe the setupfunc could be ommitted?

checks / deprecations for next release
tags: bug 2.4 core xdist

* check oejskit plugin compatibility
* move pytest_nose out of pylib because it implicitely extends
  the protocol now - setup/teardown is called at module level.
  consider making calling of setup/teardown configurable

tags: 2.4 core

- look at ihook optimization such that all lookups for
  hooks relating to the same fspath are cached.

fix start/finish partial finailization problem
tags: bug core

if a configure/runtest_setup/sessionstart/... hook invocation partially
fails the sessionfinishes is not called.  Each hook implementation
should better be repsonsible for registering a cleanup/finalizer
appropriately to avoid this issue.  Moreover/Alternatively, we could 
record which implementations of a hook succeeded and only call their

consider and document __init__ file usage in test directories
tags: bug  core

Currently, a test module is imported with its fully qualified
package path, determined by checking __init__ files upwards.
This has the side effect that a source package at the root
of the test dir could be imported as well.  This is somewhat
convenient but complicates the picture for running tests against
different versions of a package.  Also, implicit sys.path
manipulations are problematic per-se.  Maybe factorting out
a pytest_addsyspath hook which can be disabled from the command line
makes sense.  In any case documentation/recommendations for
certain scenarios makes sense.

relax requirement to have tests/testing contain an __init__
tags: feature 
bb: http://bitbucket.org/hpk42/py-trunk/issue/64

A local test run of a "tests" directory may work
but a remote one fail because the tests directory
does not contain an "__init__.py". Either give
an error or make it work without the __init__.py
i.e. port the nose-logic of unloading a test module.

customize test function collection
tags: feature 

- introduce py.test.mark.nocollect for not considering a function for
  test collection at all.  maybe also introduce a py.test.mark.test to
  explicitely mark a function to become a tested one.  Lookup JUnit ways
  of tagging tests.

introduce pytest.mark.importorskip
tags: feature 

in addition to the imperative pytest.importorskip also introduce
a pytest.mark.importorskip so that the test count is more correct.

introduce py.test.mark.platform
tags: feature 

Introduce nice-to-spell platform-skipping, examples:

    @py.test.mark.platform("not python3")
    @py.test.mark.platform("win32 and not python3")
    @py.test.mark.platform("not (jython and win32)")
    @py.test.mark.platform("not (jython and win32)", xfail=True)

etc. Idea is to allow Python expressions which can operate
on common spellings for operating systems and python
interpreter versions.

pytest.mark.xfail signature change
tags: feature 

change to pytest.mark.xfail(reason, (optional)condition)
to better implement the word meaning.  It also signals
better that we always have some kind of an implementation
reason that can be formualated.
Compatibility? how to introduce a new name/keep compat?

allow to non-intrusively apply skipfs/xfail/marks
tags: feature 

use case: mark a module or directory structures
to be skipped on certain platforms (i.e. no import
attempt will be made).

consider introducing a hook/mechanism that allows to apply marks
from conftests or plugins. (See extended parametrization)

explicit referencing of conftest.py files
tags: feature 

allow to name conftest.py files (in sub directories) that should
be imported early, as to include command line options.

improve central py.test ini file
tags: feature 

introduce more declarative configuration options:
- (to-be-collected test directories)
- required plugins
- test func/class/file matching patterns
- skip/xfail (non-intrusive)
- pytest.ini and tox.ini and setup.cfg configuration in the same file

new documentation
tags: feature 

- logo py.test
- examples for unittest or functional testing
- resource management for functional testing
- patterns: page object

have imported module mismatch honour relative paths
tags: bug 

With 1.1.1 py.test fails at least on windows if an import
is relative and compared against an absolute conftest.py
path. Normalize.

consider globals: py.test.ensuretemp and config
tags: experimental-wish 

consider deprecating py.test.ensuretemp and py.test.config
to further reduce py.test globality.  Also consider
having py.test.config and ensuretemp coming from
a plugin rather than being there from the start.

consider allowing funcargs for setup methods
tags: experimental-wish 

Users have expressed the wish to have funcargs available to setup
functions.  Experiment with allowing funcargs there - it might
also help to make the py.test.ensuretemp and config deprecation.
For filling funcargs for setup methods, we could call funcarg
factories with a request object that not have a cls/function
attributes.  However, how to handle parametrized test functions
and funcargs?

maybe introduce a setup method like:

    setup_invocation(self, request)

which has full access to the test invocation through "request"
through which you can get funcargvalues, use cached_setup etc.
Therefore, the access to funcargs would be indirect but it
could be consistently implemented.  setup_invocation() would
be a "glue" function for bringing together the xUnit and funcargs

consider pytest_addsyspath hook

py.test could call a new pytest_addsyspath() in order to systematically
allow manipulation of sys.path and to inhibit it via --no-addsyspath
in order to more easily run against installed packages.

Alternatively it could also be done via the config object
and pytest_configure.

show plugin information in test header
tags: feature 

Now that external plugins are becoming more numerous
it would be useful to have external plugins along with
their versions displayed as a header line.

deprecate global py.test.config usage
tags: feature 

py.test.ensuretemp and py.test.config are probably the last
objects containing global state.  Often using them is not
neccessary.  This is about trying to get rid of them, i.e.
deprecating them and checking with PyPy's usages as well
as others.

remove deprecated bits in collect.py
tags: feature 

In an effort to further simplify code, review and remove deprecated bits
in collect.py.  Probably good:
- inline consider_file/dir methods, no need to have them
  subclass-overridable because of hooks

implement fslayout decorator
tags: feature 

Improve the way how tests can work with pre-made examples,
keeping the layout close to the test function:

        #  empty
        test_%(NAME)s:  # becomes test_run1.py
            def test_function(self):
def test_run(pytester, fslayout):
    p = fslayout.findone("test_*.py")
    result = pytester.runpytest(p)
    assert result.ret == 0
    assert result.passed == 1

Another idea is to allow to define a full scenario including the run
in one content string::

            import pytest
            def test_that_fails():
                assert 0

            def test_hello():

            import pytest
            def pytest_runsetup_setup(item):

        runpytest -rsxX
        *1 skipped* 

This could be run with at least three different ways to invoke pytest:
through the shell, through "python -m pytest" and inlined. As inlined
would be the fastest it could be run first (or "--fast" mode).