Exclude from coverage based on Python version.

Issue #172 resolved
Simon Sapin created an issue


Projects that run on Python 2 and 3 with the same code base typically have some parts with different code paths depending on the python version. For example:

{{{ if sys.version_info[0] < 3: from urllib2 import urlopen else: from urllib.request import urlopen }}}

This example is simple, but there could be whole functions or classes in such an if statement. Of course, for in a given run, only one of the clauses will be executed. The other will be marked as "missing" in the coverage report.

Currently I use "pragma: no cover" on both clauses but this is not ideal, especially if functions or classes are defined in there.

Having different .coveragerc files would work, but this only shift the burden on the test runner to have some switching logic.

I would like to have something like "[report] exclude_lines =..." configuration that would depend on the python version. exclude_lines_py2 and exclude_lines_py3 configs would be a good start, but I can imagine that some projects would also have different code paths for eg. 2.5 and 2.6.

Comments (5)

  1. Ned Batchelder repo owner

    Hmmm, it's hard for me to envision having such specific version sensitivity in coverage.py. I would suggest separate .coveragerc files, but you've already considered it. Can you explain why it's difficult to invoke different .coveragerc files for different versions.

  2. Simon Sapin reporter

    It seems this is a case of "teddy bear problem solving". I got the idea of separate .coveragerc files while in the middle of writing/explaining the problem. You’re right, it is probably the way to go.

    To give a more concrete example: I run tests constantly when writing code for WeasyPrint (sometimes with coverage), but only on a single Python version. Much less frequently, I run tox which is configured to run the tests in four Python versions. A Jenkins server is also configured to do the same and extract coverage report.

    tox does support different configuration for each environment (python version) so it could be carefully configured with separate .coveragerc files. As for the "writing code" scenario, I can decide to always do it, say, in Python 3, and have that be the default.

    The only problem is the every similar project will have to duplicate the work of setting this up correctly. There is also a little bit of duplication inside each project in .coveragerc and tox.ini files.

    TL;DR: I can understand if this feature is too specific to be included. There is a work-around, but it’s not as obvious to setup and induces a bit of logic/config duplication.

  3. Ned Batchelder repo owner

    It seems to me that one of the reasons you're having this problem when perhaps other don't is because you are trying to achieve complete coverage separately on different Python versions. A more common approach is to run tests on separate versions, but to them combine the coverage results to get one complete measurement. Under this scenario, you don't need to exclude version-specific code, because if it is covered by at least one of your test runs, it will be counted as covered.

    Could you instead create one combined coverage result, and not concern yourself with which version covered which code?

  4. Simon Sapin reporter

    I didn’t know about combining results, it seems to be the answer indeed. I’ll look into it.

  5. Log in to comment