Add support for conditional coverage

Issue #660 closed
Thomas Pansino
created an issue

Coming from the Perl world with its excellent Devel::Cover module, I'm quite surprised that the most popular equivalent coverage package in Python doesn't have any support for conditional coverage. This can be a useful metric similiar to branch coverage to ensure all combinations of boolean conditionals are tested properly.

The best tool I've found so far to measure this is @Matthew Desmarais's instrumental utility, which is in need of some updates for Python 3 compatibility and some general polishing and love.

I'm thinking there may be an opportunity here to incorporate some of the work done in instrumental into coverage.py. I don't know that I have the Python expertise to do the work myself, but I'm happy to learn and contribute in whatever way I can.

First thing's first - where does this idea fall on the coverage.py roadmap?

Comments (6)

  1. Ned Batchelder repo owner

    @Thomas Pansino I haven't thought about how hard it would be to implement something like this. Generally, behavior within a line is not visible to coverage.py, since the Python trace function works on a line-by-line basis.

    Can you show some examples of how the results would be reported? That's often another challenge with information-rich features like this.

  2. Thomas Pansino reporter

    Generally, behavior within a line is not visible to coverage.py, since the Python trace function works on a line-by-line basis.

    That's the problem where I think instrumental could help, it looks like the dev there has solved this.

    Can you show some examples of how the results would be reported? That's often another challenge with information-rich features like this.

    Agreed. I have some initial thoughts on this, but since a picture is worth a thousand words, I'll work on a screenshot of how I think the text report and HTML report could be modified to display this efficiently.

  3. Matthew Desmarais

    There are condition / decision coverage report examples in the instrumental docs here. I think it would be straightforward to incorporate this in textual coverage reports. I'll be very interested to see the ideas that @Thomas Pansino has for displaying instrumental's coverage information.

  4. Thomas Pansino reporter

    Ok, I've played around with coverage.py's reporting, and I think I have an initial idea I can show.

    Here is the test script run.py that I ran coverage on to report the screenshots further down:

    #!/usr/bin/env python3
    class MyClass():
        def run():
            a = True
            b = True
    
            if (a):
                pass
    
            if (a and b):
                pass
    
    MyClass.run()
    

    So I have 2 branches that aren't covered, and 2/3 conditions uncovered.

    With branching and conditional coverage enabled, the text report might look like: python_coverage_report.png

    And the HTML report: python_coverage_html.png

    And the detailed HTML report for run.py (with hover text visible): python_coverage_file.png

    For some context, here is the coverage report using Devel::Cover and cover on the equivalent code in Perl. First the text: perl_coverage_report.png

    And in HTML: perl_coverage_html.png

    And the run.pl detail HTML page: perl_coverage_file.png

    And the additional Conditional Coverage page which links from line 18, conditional column: perl_coverage_condition.png

    I like the color-coding visual truth table that the Perl report offers quite a bit, and would love to do that in lieu of all the sentences in the hover box, but the implementation of the coverage.py file detail page right now uses <span>, which doesn't allow inserting a table into it (or anything else). So if we wanted to do something like that, we would need to use a different element there in the coverage.py HTML report (I'm pretty new to front-end web dev, so I'm not sure what the alternatives are)

    Overall I think implementing the visual reporting changes would be much less difficult than the underlying coverage tracking changes. For that, we have a few options.

    The first of course is trying to directly leverage the techniques @Matthew Desmarais (Matt) is using. This seems like it would be a significant departure from what coverage.py is doing today, however, and essentially wraps each underlying function with tracking code, thus modifying the program on the fly. For most developers, this probably isn't much of an issue, but there may be some for which it is, so that's a consideration here. Matt can elaborate more than I can since he wrote the code.

    Second, Matt has alternatively suggested that we take the approach of using instrumental as a sort of alternative driver to the normal coverage.py coverage collector. In essence, this would mean changes to the output format of instrumental to make it coverage.py compatible, and then of course the visual changes to the coverage.py reports to interpret the results. This would almost definitely be faster and easier to implement, even if not as well-integrated with coverage.py.

    The third option here as I see it would be to go the other direction and port some of the reporting code in coverage.py over to instrumental, to improve the polish, and just keep the two tools divorced. The trouble is that coverage.py has the brand recognition, so an effort like that may not end up with much visibility, or get much use from people, and will have less community and therefore support around it for long-term maintenance.

    I'm curious what @Ned Batchelder and maybe others in the community think of these approaches.

  5. Thomas Pansino reporter

    Any feedback here? I'd like to work on this in my spare time as a contribution to the community, but I need input on which direction we think we'd like to go.

  6. Log in to comment