coverage.py cannot cover multiprocessing.Pool

Issue #338 duplicate
Anonymous created an issue

Related to #117, I have a program that uses multiprocessing.Pool, but when I use coverage run or any other method of calling coverage, the multiprocessing function doesn't get listed in coverage.

I've done some looking around, and most solutions relate to multiprocessing.Process instead. For the record, I understand that a pool is a set of processes, but I'm not getting anything out of fixes to multiprocessing.Process.

I've included my test program. This program uses solutions that use fork() instead of multiprocessing.Pool, inspired by #56, but those have been commented out for purposes of this issue.

Comments (6)

  1. Ned Batchelder repo owner

    @mezarim The attached file, which is your program, combined with #117's monkeypatching, works for me. Can you try it with your program? I had forgotten earlier to use "coverage combine" before reporting.

    $ coverage run multitest.py
    (75853, 0)
    (75853, 1)
    (75853, 9)
    (75853, 16)
    (75853, 25)
    (75853, 36)
    (75854, 4)
    (75853, 49)
    (75853, 81)
    (75854, 64)
    (75853, 100)
    (75853, 121)
    (75853, 169)
    (75854, 144)
    (75853, 196)
    (75853, 256)
    (75854, 225)
    (75853, 289)
    (75854, 324)
    (75853, 361)
    Coverage.py warning: No data was collected.
    $ coverage combine
    $ coverage report -m
    Name           Stmts   Miss  Cover   Missing
    --------------------------------------------
    multitest.py      15      0   100%
    

    multitest.py:

    #!/usr/bin/env python
    
    import multiprocessing
    import os
    
    # monkey patch multiprocessing to enable  code coverage
    def coverage_multiprocessing_process(): # pragma: no cover
        try:
            import coverage as _coverage
            _coverage
        except:
            return
    
        from coverage.collector import Collector
        from coverage import coverage
        # detect if coverage was running in forked process
        if Collector._collectors:
            original = multiprocessing.Process._bootstrap
            class Process_WithCoverage(multiprocessing.Process):
                def _bootstrap(self):
                    cov = coverage(data_suffix=True)
                    cov.start()
                    try:
                        return original(self)
                    finally:
                        cov.stop()
                        cov.save()
            return Process_WithCoverage
    
    ProcessCoverage = coverage_multiprocessing_process()
    if ProcessCoverage:
        multiprocessing.Process = ProcessCoverage
    
    
    
    def func(x):
        pid = os.getpid()
        return pid, x*x
    
    pool = multiprocessing.Pool(3)
    inputs = range(20)
    outputs = pool.imap_unordered(func, inputs)
    for i in outputs:
        print i
    pool.close()
    pool.join()
    
  2. Anonymous

    I've just tested the modified monkey patch and it works! Thanks to you and schettino72 for your help.

  3. Log in to comment