Subprocess coverage - strange detection of lines losing coverage

Issue #492 resolved
Michał Bultrowicz
created an issue

I wanted to test out how well subprocess coverage works for WSGI apps and I came across a strange behavior. After running the tests a few times I deliberately commented out a test to see if the coverage dropped - it didn't.

I experimented with that a bit and it turns out, that the loss of coverage will be detected if I run "coverage erase" twice before running the tests. It also seems that it works when I run "erase" from shell and then run my "run_coverage" script with "coverage erase" lines removed. Adding "sleep" in "run_coverage" before a single "coverage erase" doesn't work.

Reproduction steps:

  • clone https://github.com/butla/experiments
  • checkout commit ID c4d441de7c
  • go to subprocess_coverage dir
  • create a Python 3.4 virtualenv and install requirements.txt (I have coverage_pth there)
  • delete one of "coverage erase" lines in "run_coverage.sh"
  • run "run_coverage.sh"
  • comment out one of the tests in tests/test_wsgi_app.py
  • run "run_coverage.sh" again - coverage still at 100%

Comments (12)

  1. Dan Riti

    After some more investigation, it seems this is occurring because of the inclusion of the coverage-pth package:

    https://github.com/butla/experiments/blob/master/subprocess_coverage/requirements.txt#L4

    This seems to be causing coverage to generate coverage data files when you run any coverage command. For instance:

    (.env)[driti@ubuntu subprocess_coverage]$ coverage run -m py.test tests/ 
    ================================== test session starts ==================================
    platform linux2 -- Python 2.7.3, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
    rootdir: /home/driti/dev/experiments/subprocess_coverage/tests, inifile: 
    collected 5 items 
    
    tests/test_bla.py .
    tests/test_bottle_service.py .
    tests/test_wsgi_app.py ...
    
    =============================== 5 passed in 4.46 seconds ================================
    (.env)[driti@ubuntu subprocess_coverage]$ ls -al
    total 56
    drwxrwxr-x 4 driti driti 4096 Jun  2 19:51 .
    drwxr-xr-x 9 driti driti 4096 Jun  2 18:15 ..
    -rw-rw-r-- 1 driti driti   43 Jun  2 16:55 .coveragerc
    -rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76695.615877
    -rw-rw-r-- 1 driti driti  414 Jun  2 19:51 .coverage.ubuntu.76695.989394
    -rw-rw-r-- 1 driti driti  406 Jun  2 19:51 .coverage.ubuntu.76702.583650
    -rw-rw-r-- 1 driti driti  409 Jun  2 19:51 .coverage.ubuntu.76703.517175
    -rw-rw-r-- 1 driti driti  426 Jun  2 19:51 .coverage.ubuntu.76704.344998
    -rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76713.186275
    -rw-rw-r-- 1 driti driti  434 Jun  2 19:51 .coverage.ubuntu.76718.515879
    -rw-rw-r-- 1 driti driti  264 Jun  2 18:15 requirements.txt
    -rwxrwxr-x 1 driti driti  224 Jun  2 18:15 run_coverage.sh
    drwxrwxr-x 3 driti driti 4096 Jun  2 18:20 some_code
    drwxrwxr-x 3 driti driti 4096 Jun  2 18:38 tests
    

    You can see in the output above, that we get the expected .coverage.ubuntu.* output files. However, notice what happens when I run coverage combine below:

    (.env)[driti@ubuntu subprocess_coverage]$ coverage combine
    (.env)[driti@ubuntu subprocess_coverage]$ ls -al
    total 36
    drwxrwxr-x 4 driti driti 4096 Jun  2 19:51 .
    drwxr-xr-x 9 driti driti 4096 Jun  2 18:15 ..
    -rw-rw-r-- 1 driti driti  485 Jun  2 19:51 .coverage
    -rw-rw-r-- 1 driti driti   43 Jun  2 16:55 .coveragerc
    -rw-rw-r-- 1 driti driti  389 Jun  2 19:51 .coverage.ubuntu.76783.398020
    -rw-rw-r-- 1 driti driti  264 Jun  2 18:15 requirements.txt
    -rwxrwxr-x 1 driti driti  224 Jun  2 18:15 run_coverage.sh
    drwxrwxr-x 3 driti driti 4096 Jun  2 18:20 some_code
    drwxrwxr-x 3 driti driti 4096 Jun  2 18:38 tests
    

    We get out expected .coverage "combined" data file, but we've also created a new .coverage.ubuntu.76783.398020 file, which is unexpected.

    With this file lingering around, it will cause follow on runs to have skewed results, which the original issue creator was able to observe. Thus, a workaround would be to manually delete the coverage data files (using rm) instead of using coverage erase.

    Continuing investigation for a code fix.

  2. Log in to comment