Coverage drop on Travis-CI (can't reproduce locally)

Issue #654 closed
jab
created an issue

I am seeing a strange coverage drop on Travis-CI that I can't reproduce locally, and I was wondering if you might have any idea what's going on.

Background: I store various metadata associated with my bidict library – e.g. __version__ etc. – in a module metadata.py. My setup.py gets the metadata out of metadata.py so it can use it to describe my package. Since setup.py needs to be able to read metadata.py even before the package is installed, it must resort to either modifying sys.path so that metadata.py can be imported, or using exec. I'd prefer to modify sys.path, but when I do this, I see the coverage drop on Travis that I can't reproduce locally, so I'm left having to resort to exec for now. I'd prefer to modify sys.path, but can't until I can get parity with Coverage on Travis. Any idea why this could be happening?

Here is the relevant excerpt from my setup.py:

# Get bidict's package metadata from bidict/metadata.py.
# For some reason, importing it directly with
#
#     from sys import path
#     path.insert(0, join(CWD, 'bidict'))
#     from bidict import metadata
#
# causes coverage on Travis to drop from 100 to 60-70% - cannot reproduce locally.
#
# So instead read and exec the file into a `metadata` dict:
metadata = dict(__name__=__name__)  # pylint: disable=invalid-name
with open(join(CWD, 'bidict', 'metadata.py'), encoding='utf-8') as f:
    exec(f.read().encode('utf8'), metadata)  # pylint: disable=exec-used

And an example coverage report generated on Travis when the sys.path modification is used instead of exec can be found in https://travis-ci.org/jab/bidict/jobs/367517091:

----------- coverage: platform linux, python 3.6.3-final-0 -----------
Name                       Stmts   Miss Branch BrPart    Cover
--------------------------------------------------------------
bidict/__init__.py            14     14      0      0     0.0%
bidict/_abc.py                15      7      2      0    58.8%
bidict/_base.py              229     59     90      0    80.3%
bidict/_bidict.py              4      4      0      0     0.0%
bidict/_dup.py                 9      9      0      0     0.0%
bidict/_exc.py                 6      6      0      0     0.0%
bidict/_frozen.py              9      6      2      0    45.5%
bidict/_frozenordered.py       9      9      2      0     0.0%
bidict/_marker.py              5      5      0      0     0.0%
bidict/_miss.py                3      3      0      0     0.0%
bidict/_mut.py                44     19      8      0    63.5%
bidict/_named.py              33      8      4      0    78.4%
bidict/_noop.py                3      3      0      0     0.0%
bidict/_ordered.py            31      9      4      0    74.3%
bidict/_orderedbase.py       160     25     28      0    86.7%
bidict/_util.py               24      7     12      0    80.6%
bidict/_version.py             1      0      0      0   100.0%
bidict/compat.py              30     29      2      1     6.2%
bidict/metadata.py            14     14      2      0     0.0%
--------------------------------------------------------------
TOTAL                        643    236    156      1    69.3%

You can see there are spurious "miss" lines spread throughout the codebase. When I generate a coverage report locally, trying to reproduce the Travis environment as closely as possible, I get a coverage percentage of like 98%. (It only varies by a percent or so due to using hypothesis to generate test cases.)

I am using latest pytest (3.5.0) with latest pytest-cov (2.5.1) and coverage (4.5.1).

Please let me know if there is any other info I can provide, and thanks in advance for any clues!

Comments (3)

  1. Ned Batchelder repo owner

    This looks kind of intricate. Modifying sys.path makes me think that the problem is the same module being found two different ways in the modified path. Do your entries overlap? That is, could a source file be found through two different sys.path entries, with different relative paths?

    FWIW, I use exec in my setup.py to get data from the source tree, it seems like a fine option to me :)

  2. jab reporter

    Thanks for your quick reply!

    could a source file be found through two different sys.path entries, with different relative paths?

    Sounds very possible. But would that cause the observed discrepancy in the coverage reports?

    When I run locally, I get the ~same number of statements, but the "Miss" count is way lower, as expected (e.g. only the statements in Python2-specific branches should be missed when running with Python3):

    ---------- coverage: platform darwin, python 3.6.5-final-0 -----------
    Name                       Stmts   Miss Branch BrPart    Cover
    --------------------------------------------------------------
    bidict/__init__.py            14      0      0      0   100.0%
    bidict/_abc.py                15      0      2      0   100.0%
    bidict/_base.py              229     10     90      2    96.2%
    bidict/_bidict.py              4      0      0      0   100.0%
    bidict/_dup.py                 9      0      0      0   100.0%
    bidict/_exc.py                 6      0      0      0   100.0%
    bidict/_frozen.py              9      0      2      0   100.0%
    bidict/_frozenordered.py       9      1      2      1    81.8%
    bidict/_marker.py              5      0      0      0   100.0%
    bidict/_miss.py                3      0      0      0   100.0%
    bidict/_mut.py                44      0      8      0   100.0%
    bidict/_named.py              33      0      4      0   100.0%
    bidict/_noop.py                3      0      0      0   100.0%
    bidict/_ordered.py            31      1      4      1    94.3%
    bidict/_orderedbase.py       160      6     28      1    96.3%
    bidict/_util.py               24      0     12      0   100.0%
    bidict/_version.py             1      0      0      0   100.0%
    bidict/compat.py              30      9      2      0    71.9%
    bidict/metadata.py            16      0      4      0   100.0%
    --------------------------------------------------------------
    TOTAL                        645     27    158      5    96.0%
    

    Is there any way coverage could detect when sys.path is responsible for such miscounting and compensate, or at least warn about it?

    FWIW, I use exec in my setup.py to get data from the source tree, it seems like a fine option to me :)

    Every time I use another code analysis tool it inevitably complains about use of exec, so I figured if there's a different way to achieve the same result, I might as well explore it. Good to know you think exec is fine though. FWIW, when I try the third alternative of importing by path (e.g. via importlib) I can both avoid exec and the coverage discrepancy: https://github.com/jab/bidict/blob/cb1d80c/setup.py#L23-L37

    What do you think of that approach? Do you prefer exec because it's more consistent across Python versions?

  3. Log in to comment