no-op plugin produces different, nonsensical reports

Issue #637 resolved
Barry Warsaw created an issue

I'm experimenting with the new configuration plugin support in 4.5, but the results I'm seeing don't make sense. I'm not sure if I'm doing something wrong, or if this is a bug in 4.5.

Using this importlib_resources branch, when I comment out the [run]plugins lines in coverage.ini and then run tox -e py36-cov, I get the following results:

Name                          Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------
importlib_resources/_py3.py     178      1     52      1    99%   23, 20->23

This makes sense, as I would expect these lines to be missed under Python 3.6. However, if I uncomment the lines, I get these different results:

Name                          Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------
importlib_resources/_py3.py     178     30     52      0    86%   1-26, 41, 50, 62, 94, 130, 138, 153-154, 196, 250

This makes no sense. It's saying that all the lines from the top of the file to the first def are missed, as well as every def (or decorated-def) line in the rest of the file.

My plugin is essentially no-op:

from __future__ import absolute_import
# This file is used in both Python 2 and 3.

from coverage import CoveragePlugin


class MyConfigPlugin(CoveragePlugin):
    def configure(self, config):
        opt_name = 'report:exclude_lines'
        print(config.get_option(opt_name))
        ## exclude_lines = config.get_option(opt_name)
        ## exclude_lines.append(r"pragma: custom")
        ## exclude_lines.append(r"pragma: or whatever")
        ## config.set_option(opt_name, exclude_lines)


def coverage_init(reg, options):
    reg.add_configurer(MyConfigPlugin())

and I do see the expected lines printed to stdout:

['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']

Is this a bug, or am I doing something wrong?

Comments (18)

  1. Ned Batchelder repo owner

    I'd like to reproduce this, but when I try I get "No data to report."

    $ git clone git@gitlab.com:python-devs/importlib_resources.git
    Cloning into 'importlib_resources'...
    remote: Counting objects: 573, done.
    remote: Compressing objects: 100% (330/330), done.
    remote: Total 573 (delta 293), reused 460 (delta 229)
    Receiving objects: 100% (573/573), 130.50 KiB | 2.17 MiB/s, done.
    Resolving deltas: 100% (293/293), done.
    
    $ cd importlib_resources/
    
    $ git checkout 137e05226cb4a8b5b5d26885925800794251d901
    Note: checking out '137e05226cb4a8b5b5d26885925800794251d901'.
    
    You are in 'detached HEAD' state. You can look around, make experimental
    changes and commit them, and you can discard any commits you make in this
    state without impacting any branches by performing another checkout.
    
    If you want to create a new branch to retain commits you create, you may
    do so (now or later) by using -b with the checkout command again. Example:
    
      git checkout -b <new-branch-name>
    
    HEAD is now at 137e052... Checkpointing
    
    $ git log --oneline | head -3
    137e052 Checkpointing
    2e1e34f Merge branch 'release02' into 'master'
    1e56c62 Updates for the 0.2 release
    
    $ tox -e py36-cov
    py36-cov create: /private/tmp/importlib_resources/.tox/py36-cov
    py36-cov installdeps: coverage
    py36-cov develop-inst: /private/tmp/importlib_resources
    py36-cov installed: coverage==4.5,-e git+git@gitlab.com:python-devs/importlib_resources.git@137e05226cb4a8b5b5d26885925800794251d901#egg=importlib_resources
    py36-cov runtests: PYTHONHASHSEED='3417395414'
    py36-cov runtests: commands[0] | python -m coverage run --rcfile=/private/tmp/importlib_resources/coverage.ini -m unittest discover
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    ..................................................................................................
    ----------------------------------------------------------------------
    Ran 98 tests in 0.039s
    
    OK
    Coverage.py warning: No data was collected. (no-data-collected)
    py36-cov runtests: commands[1] | python -m coverage combine --rcfile=/private/tmp/importlib_resources/coverage.ini
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    py36-cov runtests: commands[2] | python -m coverage html --rcfile=/private/tmp/importlib_resources/coverage.ini
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    No data to report.
    ERROR: InvocationError: '/private/tmp/importlib_resources/.tox/py36-cov/bin/python -m coverage html --rcfile=/private/tmp/importlib_resources/coverage.ini'
    _____________________________________________________________________________________________ summary ______________________________________________________________________________________________
    ERROR:   py36-cov: commands failed
    
    $ sed -i '' -e 15,16s/^/#/ coverage.ini
    
    $ cat coverage.ini
    [run]
    branch = true
    parallel = true
    omit =
         setup*
        .tox/*/lib/python*/site-packages/*
        */tests/*.py
        /tmp/*
        /private/var/folders/*
        */testing/*.py
        importlib_resources/_py${OMIT}.py
        importlib_resources/__init__.py
        importlib_resources/_compat.py
        importlib_resources/abc.py
    #plugins =
    #     importlib_resources.tests.coverage
    
    [report]
    exclude_lines =
        pragma: nocover
        raise NotImplementedError
        raise AssertionError
        assert\s
    
    [paths]
    source =
        importlib_resources
        .tox/*/lib/python*/site-packages/importlib_resources
    
    $ tox -e py36-cov
    py36-cov develop-inst-nodeps: /private/tmp/importlib_resources
    py36-cov installed: coverage==4.5,-e git+git@gitlab.com:python-devs/importlib_resources.git@137e05226cb4a8b5b5d26885925800794251d901#egg=importlib_resources
    py36-cov runtests: PYTHONHASHSEED='2705296418'
    py36-cov runtests: commands[0] | python -m coverage run --rcfile=/private/tmp/importlib_resources/coverage.ini -m unittest discover
    ..................................................................................................
    ----------------------------------------------------------------------
    Ran 98 tests in 0.035s
    
    OK
    Coverage.py warning: No data was collected. (no-data-collected)
    py36-cov runtests: commands[1] | python -m coverage combine --rcfile=/private/tmp/importlib_resources/coverage.ini
    py36-cov runtests: commands[2] | python -m coverage html --rcfile=/private/tmp/importlib_resources/coverage.ini
    No data to report.
    ERROR: InvocationError: '/private/tmp/importlib_resources/.tox/py36-cov/bin/python -m coverage html --rcfile=/private/tmp/importlib_resources/coverage.ini'
    _____________________________________________________________________________________________ summary ______________________________________________________________________________________________
    ERROR:   py36-cov: commands failed
    
    $
    
  2. Barry Warsaw reporter

    That's very odd! I just verified my results by doing a fresh clone on Debian unstable, for which system python3 is 3.6.4:

    % git clone https://gitlab.com/python-devs/importlib_resources.git
    ...
    % cd importlib_resources
    % git checkout coverage
    ...
    % tox -e py36-cov
    py36-cov create: /home/barry/projects/importlib_resources/.tox/py36-cov
    py36-cov installdeps: coverage
    py36-cov develop-inst: /home/barry/projects/importlib_resources
    py36-cov installed: coverage==4.5,-e git+https://gitlab.com/python-devs/importlib_resources.git@137e05226cb4a8b5b5d26885925800794251d901#egg=importlib_resources,pkg-resources==0.0.0
    py36-cov runtests: PYTHONHASHSEED='653536378'
    py36-cov runtests: commands[0] | python -m coverage run --rcfile=/home/barry/projects/importlib_resources/coverage.ini -m unittest discover
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    ..................................................................................................
    ----------------------------------------------------------------------
    Ran 98 tests in 0.041s
    
    OK
    py36-cov runtests: commands[1] | python -m coverage combine --rcfile=/home/barry/projects/importlib_resources/coverage.ini
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    py36-cov runtests: commands[2] | python -m coverage html --rcfile=/home/barry/projects/importlib_resources/coverage.ini
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    py36-cov runtests: commands[3] | python -m coverage xml --rcfile=/home/barry/projects/importlib_resources/coverage.ini
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    py36-cov runtests: commands[4] | python -m coverage report -m --rcfile=/home/barry/projects/importlib_resources/coverage.ini --fail-under=100
    ['pragma: nocover', 'raise NotImplementedError', 'raise AssertionError', 'assert\\s']
    Name                          Stmts   Miss Branch BrPart  Cover   Missing
    -------------------------------------------------------------------------
    importlib_resources/_py3.py     178     30     52      0    86%   1-26, 41, 50, 62, 94, 130, 138, 153-154, 196, 250
    ERROR: InvocationError: '/home/barry/projects/importlib_resources/.tox/py36-cov/bin/python -m coverage report -m --rcfile=/home/barry/projects/importlib_resources/coverage.ini --fail-under=100'
    ___________________________________ summary ____________________________________
    ERROR:   py36-cov: commands failed
    % cat coverage.ini
    [run]
    branch = true
    parallel = true
    omit =
         setup*
        .tox/*/lib/python*/site-packages/*
        */tests/*.py
        /tmp/*
        /private/var/folders/*
        */testing/*.py
        importlib_resources/_py${OMIT}.py
        importlib_resources/__init__.py
        importlib_resources/_compat.py
        importlib_resources/abc.py
    plugins =
         importlib_resources.tests.coverage
    
    [report]
    exclude_lines =
        pragma: nocover
        raise NotImplementedError
        raise AssertionError
        assert\s
    
    [paths]
    source =
        importlib_resources
        .tox/*/lib/python*/site-packages/importlib_resources
    
  3. Ned Batchelder repo owner

    OK, mine was on Mac. Is there something obvious (to you) in the code that would make a difference there? I can spin up other machines, but it takes more work :)

  4. Barry Warsaw reporter

    I do most of my development on macOS 10.13, so I can't think of anything that would be platform specific. My initial report was from observed behavior on Mac. I do have a bunch of things installed from Homebrew, though I don't know why or if that would make a difference. Except for tox and python3.6, everything should be installed fresh into the venvs.

    I'm getting these results very consistently on both platforms.

    I should try to dig into coverage itself, but I just haven't had the time yet.

  5. Barry Warsaw reporter

    A few more quick data points. I still get the broken reporting if my coverage_init() just does a pass and nothing more. So it seems like it just takes the presence of a plugin to affect the reporting.

    Next, I commented out the combine, html, xml, and report lines in my tox.ini. I ran tox -e py36-cov both with the plugin enabled and disabled. This gives me two .coverage files, and they look like they have different contents. I'll see if I can attach them, and if not, I'll paste their contents.

  6. Barry Warsaw reporter

    Looks like you can't. So here first is the .coverage file produced with the no-op coverage_init() enabled:

    !coverage.py: This is a private format, don't read it directly!{"arcs":{"/Users/barry/projects/implib/trunk/importlib_resources/_py3.py":[[-62,64],[-41,42],[42,43],[43,44],[44,45],[45,-41],[64,-62],[44,47],[47,-41],[64,65],[-26,27],[27,34],[34,35],[35,38],[38,-26],[65,66],[-50,57],[57,59],[59,-50],[66,67],[67,71],[71,72],[72,73],[73,74],[74,75],[75,-62],[35,36],[36,-26],[65,-62],[27,28],[28,29],[29,30],[30,-26],[28,32],[32,-26],[57,58],[58,-50],[67,68],[68,-62],[-94,99],[99,-94],[99,100],[100,101],[101,102],[102,106],[106,107],[107,108],[108,109],[109,110],[110,111],[111,-94],[100,-94],[102,103],[103,-94],[75,76],[76,80],[80,81],[81,82],[82,83],[83,84],[84,85],[85,86],[86,87],[87,88],[88,89],[89,-62],[111,112],[112,116],[116,117],[117,118],[118,119],[119,120],[120,121],[121,122],[122,123],[123,124],[124,125],[125,-94],[85,91],[91,-62],[121,127],[127,-94],[-153,163],[163,-153],[163,164],[164,165],[165,166],[166,174],[174,175],[175,176],[176,177],[177,-153],[164,-153],[166,167],[167,168],[168,170],[170,171],[171,174],[176,179],[179,180],[180,184],[184,185],[185,186],[186,187],[187,188],[188,190],[190,191],[191,-153],[168,169],[169,-153],[179,-153],[191,192],[192,193],[193,-153],[-130,132],[132,-130],[132,133],[133,134],[134,135],[135,-130],[133,-130],[134,-130],[-138,147],[147,-138],[147,148],[148,149],[149,150],[150,-138],[148,-138],[149,-138],[-250,257],[257,258],[258,259],[259,264],[264,265],[265,266],[266,-250],[-196,201],[201,202],[202,203],[203,204],[204,206],[206,207],[207,210],[210,211],[211,-196],[118,121],[82,85],[264,267],[267,268],[268,269],[269,270],[270,272],[272,273],[273,274],[274,-250],[207,208],[208,209],[209,-196],[269,-250],[210,215],[215,216],[216,217],[217,-196],[216,218],[218,219],[219,-196],[273,275],[275,276],[276,277],[277,278],[278,279],[279,280],[280,293],[293,294],[294,279],[293,295],[295,296],[296,297],[297,279],[279,-250],[259,260],[260,-250],[260,261],[261,-250],[204,205],[205,-196],[296,298],[298,299],[299,300],[300,301],[301,302],[302,279],[300,279],[218,224],[224,225],[225,226],[226,227],[227,228],[228,229],[229,230],[230,231],[231,232],[232,233],[233,235],[235,230],[232,242],[242,-196]]}}
    

    This is the .coverage file with the plugin disabled:

    !coverage.py: This is a private format, don't read it directly!{"arcs":{"/Users/barry/projects/implib/trunk/importlib_resources/_py3.py":[[-1,1],[1,2],[2,3],[3,5],[5,6],[6,7],[7,8],[8,9],[9,10],[10,11],[11,12],[12,13],[13,14],[14,15],[15,16],[16,19],[19,20],[20,21],[21,26],[26,41],[41,51],[51,62],[62,97],[97,130],[130,141],[141,153],[153,154],[154,196],[196,250],[250,-1],[-62,64],[-41,42],[42,43],[43,44],[44,45],[45,-41],[64,-62],[44,47],[47,-41],[64,65],[-26,27],[27,34],[34,35],[35,38],[38,-26],[65,66],[-50,57],[57,59],[59,-50],[66,67],[67,71],[71,72],[72,73],[73,74],[74,75],[75,-62],[35,36],[36,-26],[65,-62],[27,28],[28,29],[29,30],[30,-26],[28,32],[32,-26],[57,58],[58,-50],[67,68],[68,-62],[-94,99],[99,-94],[99,100],[100,101],[101,102],[102,106],[106,107],[107,108],[108,109],[109,110],[110,111],[111,-94],[100,-94],[102,103],[103,-94],[75,76],[76,80],[80,81],[81,82],[82,83],[83,84],[84,85],[85,86],[86,87],[87,88],[88,89],[89,-62],[111,112],[112,116],[116,117],[117,118],[118,119],[119,120],[120,121],[121,122],[122,123],[123,124],[124,125],[125,-94],[85,91],[91,-62],[121,127],[127,-94],[-153,163],[163,-153],[163,164],[164,165],[165,166],[166,174],[174,175],[175,176],[176,177],[177,-153],[164,-153],[166,167],[167,168],[168,170],[170,171],[171,174],[176,179],[179,180],[180,184],[184,185],[185,186],[186,187],[187,188],[188,190],[190,191],[191,-153],[168,169],[169,-153],[179,-153],[191,192],[192,193],[193,-153],[-130,132],[132,-130],[132,133],[133,134],[134,135],[135,-130],[133,-130],[134,-130],[-138,147],[147,-138],[147,148],[148,149],[149,150],[150,-138],[148,-138],[149,-138],[-250,257],[257,258],[258,259],[259,264],[264,265],[265,266],[266,-250],[-196,201],[201,202],[202,203],[203,204],[204,206],[206,207],[207,210],[210,211],[211,-196],[118,121],[82,85],[264,267],[267,268],[268,269],[269,270],[270,272],[272,273],[273,274],[274,-250],[207,208],[208,209],[209,-196],[269,-250],[210,215],[215,216],[216,217],[217,-196],[216,218],[218,219],[219,-196],[273,275],[275,276],[276,277],[277,278],[278,279],[279,280],[280,293],[293,294],[294,279],[293,295],[295,296],[296,297],[297,279],[279,-250],[259,260],[260,-250],[260,261],[261,-250],[204,205],[205,-196],[296,298],[298,299],[299,300],[300,301],[301,302],[302,279],[300,279],[218,224],[224,225],[225,226],[226,227],[227,228],[228,229],[229,230],[230,231],[231,232],[232,233],[233,235],[235,230],[232,242],[242,-196]]}}
    

    Pretty sure I got that ordered correctly :)

  7. Ned Batchelder repo owner

    First mystery solved: I was working in /tmp, and you omit anything in /tmp, so nothing was measured! Now to debug the actual bug report....

  8. Barry Warsaw reporter

    Same!

    % tox --version
    2.9.1 imported from /usr/local/lib/python3.6/site-packages/tox/__init__.py
    % .tox/py36-cov/bin/python -V
    Python 3.6.4
    % .tox/py36-cov/bin/python -c "import coverage; print(coverage.__version__)"
    4.5
    
  9. Barry Warsaw reporter

    Ah, cool. (I probably don't need to omit /tmp for this project, but y'know, cargo culting is fun!)

  10. Ned Batchelder repo owner

    OK, second mystery solved: the reason this plugin affects the measurement of _py3.py is because coverage has to import the plugin file, which imports importlib_resources/__init__.py, which imports _py3.py. This all happens before coverage measurement begins, so all of those lines in _py3.py are executed before coverage has started measuring, so they are now missed.

    I'm not sure what the best approach is here. If you move the plugin to a top-level file in your working tree, so that it doesn't force importlib_resources imports, it will work.

  11. Barry Warsaw reporter

    OMG, that totally makes sense, and you're exactly right. Moving the file does fix the problem. Man, I'm sorry for the extra work. Does the documentation describe this caveat?

    Again, thanks and I'll close this issue.

  12. Ned Batchelder repo owner

    I added a warning to the docs in 0de4ed9562de.

    I'm not sure if there's anything else I can do here? I guess in theory I could look at the plugins, and see if any of them are inside the source= or include= ? But the problem can be larger than that, if the plugin is outside, but imports files that are inside anyway?

  13. Barry Warsaw reporter

    I think you're right that coverage does need to do anything to detect this situation, and it might be difficult, problematic, or error prone to do so. The documentation warning should be enough, although I might suggest something additional like, "We recommend that you put your plugin next to your coverage.ini file."

  14. Ned Batchelder repo owner

    This got me thinking, and I hacked something together. In Coverage.start(), I loop over all the modules in sys.modules, and check if their __file__ would be included in the coverage criteria. If they would, spit out a warning. It works well enough that with a slight modification to your coverage.ini, the bad scenario in this ticket now prints:

    Coverage.py warning: Already imported a file that will be measured: /src/importlib_resources/importlib_resources/_py3.py
    

    The tweak I had to make to your coverage.ini is that it only has omits, it doesn't have include or source. Once I added "source = importlib_resources", the warning kicks in.

    I haven't looked into the test failures I'm getting, or the other possible downsides to running this warning, but it's got a good chance of being in a future release. :)

  15. Log in to comment