Enable coverage measurement of code run by `multiprocessing.Process`

Issue #117 resolved
Ram Rachum created an issue

If you have a multiprocessing.Process that runs on some lines of code, Coverage currently doesn't detect that and doesn't mark these lines as executed.

A sample module that uses multiprocessing.Process is attached; when run with coverage, all the 1 + 1 lines are marked as not executed, when in fact they have been executed by the auxiliary process.

Comments (22)

  1. amcnabb8

    I am also experiencing this problem. I also tried adding the following lines to the run() method in the minimal working example posted by Ram:

    import coverage coverage.process_startup()

    Unfortunately, this did not help.

  2. amcnabb8

    Hmm. I don't have a very complete picture for how coveragepy works, but it looks like when multiple processes share a coverage file, each process clobbers the file when it writes. I had expected to see something like file locking to prevent race conditions, but it looks like this doesn't happen.

    One possible solution would be to have a .coverage directory (instead of a .coverage file), and to have each process write to its own file within the directory. The directory could be cleared as the main process starts, and reporting could combine the data from all of the files in the directory.

  3. memedough


    Multiprocessing uses fork, the child process has same coverage obj as parent so clobber.

    pytest-cov, nose-cov, nose2-cov use multiprocessing hook after fork to child start coverage with unique data file, also use multiprocessing finalizer to stop then write file. Use them to get multiprocessing coverage out of box.

    If coverage start doing multiprocessing then please let know so I make plugins works with changed coverage. Look at cov-core near top the file if you like.


  4. Anton Löfgren

    I'm experiencing some weirdness when running coverage.py on a test suite which uses multiprocessing.Process. Namely, it gets confused with regards to the tracer functions resulting in:

    Coverage.py warning: Trace function changed, measurement is likely wrong: <bound method PyTracer._trace of <coverage.collector.PyTracer object at 0x2b98b0626390>>

    Removing the call to threading.settrace() (in coverage.collector.Collector) seems to resolve the issue, though I'm not sure about the reliability of the coverage measurements reported.

  5. Ned Batchelder repo owner

    @schettino72 I couldn't get your monkey-patching code to work. Do you have a complete example?

  6. Ned Batchelder repo owner

    @schettino72 Thanks, that works for me also, if I don't forget to use "coverage combine"! :)

  7. Former user Account Deleted

    I found a hole in the patch while I was working with it. It worked quite well, but processes started under coverage typically didn't take a coveragerc file with them, especially a custom one without that name.

    This is the version I used;

    def coverage_multiprocessing_process(): # pragma: no cover
            import coverage as _coverage
        from coverage.collector import Collector
        from coverage import coverage
        # detect if coverage was running in forked process
        if Collector._collectors:
            original = multiprocessing.Process._bootstrap
            class Process_WithCoverage(multiprocessing.Process):
                def _bootstrap(self):
                    cov = coverage(data_suffix=True,config_file=os.environ['COVERAGE_PROCESS_START'])
                        return original(self)
            return Process_WithCoverage
    ProcessCoverage = coverage_multiprocessing_process()
    if ProcessCoverage:
        multiprocessing.Process = ProcessCoverage

    Just a tiny improvement I thought I'd share with the community. Hope it helps.

  8. Ned Batchelder repo owner

    @mezarim thanks, can you explain it a little more? You are using an unusually-named configuration file?

  9. Ned Batchelder repo owner

    This monkeypatching is implemented in coverage.py as of f84263e515c2. I'll wait to close the issue until I can implement the config_file improvement also.

  10. Pieter Rautenbach

    I've experimented with this branch, and it works great. It would be great to see this incorporated into a stable (non-alpha) release, with a .coveragerc configuration option.

  11. Pieter Rautenbach

    While experimenting with this patch, I was wondering whether the multiprocessing.dummy wrapper could be used as an alternative. I'm suggesting this without having investigated its feasibility, but it could be possible to replace e.g. import multiprocessing with import multiprocessing.dummy as multiprocessing. The disadvantage is that the code under testing must be modified, but the advantage is that the standard coverage.py tool would do the job. Just a thought...

  12. hugovk

    I'd like to start using nose's parallel testing to speed up Pillow's tests on Travis CI.

    It looks like the --include=diris ignored with --concurrency=multiprocessing.

    Currently it does something like this (using Coverage 3) (before some later processing to include C coverage, but let's ignore that for now).

    coverage erase
    coverage run --append --include=PIL/* selftest.py
    coverage run --append --include=PIL/* -m nose -vx Tests/test_*.py

    coverage report only lists files under PIL/ as expected, coverage is 79% (1903 missed of 9119). For example: https://travis-ci.org/hugovk/Pillow/jobs/61138890#L2408

    To use nose's parallel testing, I added --processes=-1 --process-timeout=600 to nose, and then (using Coverage 4.0a5) add --concurrency=multiprocessing to coverage, and run coverage combine at the end:

    coverage erase
    coverage run --append --include=PIL/* selftest.py
    coverage run --append --concurrency=multiprocessing --include=PIL/* -m nose -vx Tests/test_*.py --processes=-1 --process-timeout=600
    coverage combine

    Nose runs the tests in parallel, they take under 100 seconds instead of over 300 seconds.

    I'm sure those appends aren't quite right, but anyway, it seems --include=PIL/* is being ignored as coverage report gives 43% coverage, 22502 of 39533 lines missed, and also lists files under Tests/ and /home/travis/virtualenv/python2.7.9/ which isn't desired. For example: https://travis-ci.org/hugovk/Pillow/jobs/61133954#L2621

  13. Ned Batchelder repo owner

    I'm going to mark this issue as fixed, now that 4.0 has --concurrency=multiprocessing. @hugovk If your situation is still a problem, could you open a new issue?

  14. Ned Batchelder repo owner

    @andreycizov Would you mind opening a new issue, with details of the problem? A reproducible test case would be great, and also information about whether this was always a problem, or is it a regression? Thanks.

  15. Log in to comment