1. Ned Batchelder
  2. coverage.py
  3. Issues
Issue #301 invalid

Combine from same python process only considers first coverage run when using API

Olivier Trempe
created an issue

My use case:

I have a package with multiple modules. For each modules, I have a corresponding test module. I want to measure the coverage for each module explicitly targeted by its corresponding test module without side effects from other test modules and output the results in a single report.

Here's how I achieve this with command line:

coverage erase

coverage run -p --branch --include=package/a_module.py nosetests test_package/test_a_module
coverage run -p --branch --include=package/another_module.py nosetests test_package/test_another_module
etc..

coverage combine
coverage report -m

I wanted to run all this from within a python script:

import nose
from coverage import coverage

cov = coverage()
cov.erase()

for module in my_modules:
    module_cov = coverage(branch=True, include=path_to_module, data_suffix=True)
    module_cov.start()
    nose.run(argv=['', path_to_test_module])
    module_cov.stop()
    module_cov.save()

cov.combine()
cov.save()
cov.report(show_missing=True)

With this script, when looking at the results, the coverage is fine for the first module, but all other modules have no more coverage than the coverage coming from importing the module.

I realized that it comes from parallel coverage files.

With command line, I get:

.coverage.machine_name.first_coverage_run_PID.first_nose_run_PID
.coverage.machine_name.second_coverage_run_PID.second_nose_run_PID
etc...

But, when using the python script, I obviously get:

.coverage.machine_name.python_script_run_PID.first_nose_run_PID
.coverage.machine_name.python_script_run_PID.second_nose_run_PID
etc...

And it seems the bug comes from the parallel coverage files having the same python_script_run_PID.

My workaround currently is to use multiprocessing.Process to get different python_script_run_PID for each module:

def run_module_coverage(module_path, test_module_path):
    cov = coverage(branch=True, include=module_path, data_suffix=True)
    cov.start()
    nose.run(argv=['', test_module_path])
    cov.stop()
    cov.save()

cov = coverage()
cov.erase()

for module in my_modules:
    process = Process(target=run_module_coverage, args=(module_path, test_module_path))
    process.start()
    process.join()

cov.combine()
cov.save()
cov.report(show_missing=True)

And it works as expected.

Comments (9)

  1. Ned Batchelder repo owner

    What version of coverage are you using? Since 3.3, the parallel data file name has had a random number in it which should be enough to separate them. You described the data file as ".coverage.machine_name.python_script_run_PID.second_nose_run_PID", but the last component is actually the random number. If they are different, then it shouldn't matter that "python_script_run_PID" is the same for all of them.

  2. Olivier Trempe reporter

    I am using 3.7.1

    I didn't pay attention if it was actually a PID or a random number, but if the .coverage files come from the same process, they get ".coverage.machine_name.same_number.different_number".

    Do you want me to run coverage with some debug info and post the output?

  3. Ned Batchelder repo owner

    Right, so from the same process, you get a number of different files. So why don't they combine properly? The filenames don't actually matter, so long as they are distinct, which they are.

    Can you provide a reproducible test case for me? For example as a .zip with instructions?

  4. Olivier Trempe reporter

    I couldn't reproduce it with a simple little project... I still got the probleme with my actual project though. As soon as I find a way to reproduce the bug, I'll attach the files to the ticket.

    Carefully checking the results with the html report, I realized the coverage was ok for tested function body, but all statements at module scope (imports, function definitions, class definition, etc...) are marked as not covered... weird.

  5. Olivier Trempe reporter

    I found something interesting.

    My tested code is a nose plugin. If I remove the entry point for my plugin so that nose doesn't see it as a plugin anymore, coverage now works fine.

  6. Ned Batchelder repo owner

    It sounds like you are using nose to test a nose plugin. When you start nose, it will find and import all of its plugins. This happens before you have started coverage. Later, after coverage has started, you import the modules again, but they've already been imported, so they are not re-run. This is why the top-level statements in your modules are marked as not covered.

    If you can avoid the early import, you are all set. Another option is to explicitly remove the modules from sys.modules so that when they are imported after coverage has started, they will execute again.

    Let me know when you've found a solution, and if I can close this issue, thanks.

  7. Log in to comment