Combining coverage from different versions of coverage silently "fails"

Issue #562 invalid
Aleks Bunin
created an issue

Consider following example: I use tox to run tests and see coverage in three different version of Python. For some odd reason, I've ended up with 4.2.x in python 2.7 and 3.5, and 4.3.4 in 3.6 tests and combine step.

In .coveragerc I have instruct which paths should be combined in the paths section

[paths]
source =
    src
    .tox/py27/lib/python2.7/site-packages
    .tox/py35/lib/python3.5/site-packages
    .tox/py36/lib/python3.6/site-packages

When I run combine, I get output which includes all versions, and result is not combined.

I've spent hours debugging this, but accidentally spotted that I use two different version of coverage. If results of coverage are not compatible between versions, combine step should report this.

This is clearly my mistake somewhere, but nevertheless it is happened to be for second time in relatively short period of time and it would be nice if warning, or better yet, error be displayed.

Comments (6)

  1. Ned Batchelder repo owner

    I don't know why this slight version skew would prevent combining data. Is your repo public so that I can try to reproduce the problem?

    You say, "I get output which includes all versions, and result is not combined." I don't understand what that means? It sounds like a success and a failure at the same time.

  2. Aleks Bunin reporter

    You say, "I get output which includes all versions, and result is not combined." I don't understand what that means? It sounds like a success and a failure at the same time.

    I expect output from the combined report to have single mention of the source file in the report, not multiple record like shown bellow:

    Name                                                          Stmts   Miss  Cover   Missing
    -------------------------------------------------------------------------------------------
    .tox/py27/lib/python2.7/site-packages/docker_ci/__init__.py      15      4    73%   17, 28-30, 42
    .tox/py35/lib/python3.5/site-packages/docker_ci/__init__.py      15      4    73%   17, 34-37, 42
    .tox/py36/lib/python3.6/site-packages/docker_ci/__init__.py      15      4    73%   17, 34-37, 42
    .tox/py36/lib/python3.6/site-packages/docker_ci/__main__.py       5      5     0%   1-7
    

    In this case, py27 and py35 used coverage=4.2.0, while py36 used coverage=4.3.4.

    As soon as I ensured that all coverages were generated using same version (4.3.4), I've got report as expected to show single record per file, actually combining them:

    Name                        Stmts   Miss  Cover   Missing
    ---------------------------------------------------------
    src/docker_ci/__init__.py      15      2    87%   17, 42
    src/docker_ci/__main__.py       5      5     0%   1-7
    

    No, project isn't public.

  3. Aleks Bunin reporter
    [tox]
    envlist=py{27,35,36}
    
    [testenv]
    deps=
        wheel==0.29
        pytest
        coverage>=4.3
        pytest-cov>=2.4
    setenv =
        COVERAGE_FILE=.coverage.{envname}
        PIP_DISABLE_PIP_VERSION_CHECK=1
    commands=
        py.test -r a --pyargs docker_ci --cov docker_ci --cov-report=term-missing {posargs}
    
    [testenv:codestyle]
    deps = pycodestyle 
    commands = pycodestyle {posargs}
    
    [codestyle]
    exclude = .git,.tox,dist,doc,*egg,adm,bootstrap
    max-line-length = 120
    
    
    [testenv:coverage]
    skip_install = true
    deps =
        coverage>=4.3
    setenv=
        COVERAGE_FILE=.coverage
    commands =
        coverage combine
        coverage html
        coverage xml
        coverage report --show-missing
    
  4. Aleks Bunin reporter

    It's impossible to reproduce -- same code, which was failing before, now works successfully. I have no clue what was causing -- I checked out commit which failed, and tried to run it again and it succeeds now.

  5. Log in to comment