pytest and multiprocessing: CoverageException: Can't combine line data with arc dat

I'm getting CoverageException: Can't combine line data with arc dat when using coverage with the pytest-cov pytest extension. I think this is similar to #399, but that issue is closed.

Example failure on travis: https://travis-ci.org/samuelcolvin/arq/builds/148541711

The test causing problems is using click's CliRunner, code here. The CLI command that's testing starts another process (using multiprocessing.Process) to run the worker which is what's causing the problem, if you prevent the worker process being started the exception doesn't occur.

I tried changing the concurrency mode but it made no difference.

To reproduce: clone, pip install -e . && pip install -r tests/requirements.txt, py.test --cov=arq.

relevant bits of pip freeze:

click==6.6
coverage==4.2
pytest==2.9.2
pytest-cov==2.3.0

Let me know if you need anymore information.

Any update on this? I'm currently having to choose between coverage and running some tests.

@Samuel Colvin Sorry, I haven't started digging into this. It does sounds a lot like the fix I put in 4.2, so I will have to look at it soon. Have you tried it with the tip of coverage.py?

Same error with hg clone ... setup.py install I'm afraid:

...
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/pytest_cov/engine.py", line 150, in finish
INTERNALERROR>     self.cov.combine()
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/control.py", line 822, in combine
INTERNALERROR>     self.data_files.combine_parallel_data(self.data, aliases=aliases, data_paths=data_paths)
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/data.py", line 716, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/data.py", line 481, in update
INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data

I tried running your scenario, and did not get the error you describe. The full output is here: https://gist.github.com/nedbat/a482ea69fa509de5e0a7c3da8f378e4b

The test run ends with:

 tests/test_worker.py ✓✓✓✓✓✓✓✓✓✓✓                                                                                                                100% ██████████

---------- coverage: platform darwin, python 3.5.2-final-0 -----------
Name              Stmts   Miss Branch BrPart  Cover
---------------------------------------------------
arq/__init__.py       4      0      0      0   100%
arq/cli.py           15      0      0      0   100%
arq/logs.py          18      1      2      1    90%
arq/main.py          85      1     22      1    98%
arq/testing.py      164      6     34      0    97%
arq/utils.py         53      2     12      1    95%
arq/version.py        2      0      0      0   100%
arq/worker.py       245     31     54      4    88%
---------------------------------------------------
TOTAL               586     41    124      7    93%


Results (4.23s):
      29 passed
      11 failed
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /Users/ned/foo/arq/tests/test_main.py:137: assert None
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)

I'm not sure what caused my test failures, but I wouldn't think they are significant in this case.

looks like you don't have redis installed.

@Samuel Colvin I'm willing to try reproducing this, but I need explicit instructions, or I'll wander in the redis-wilderness for a while. What should I do on my Mac to get things to pass? Do you think the test failures are preventing the coverage failure?

Sorry about that, on ubuntu redis runs as a service so is always there. I've dusted off my mac and tried to run the code however brew seems to be broken so I can't install python3.5.

As far as I know it should be as simple:

• brew install redis
• redis-server, leave the terminal with redis running
• in another terminal in the arq directory git fetch && git checkout coverage-broken (I've added a test on that branch which breaks coverage), see here.
• run py.test all tests should pass
• run py.test --cov=arq you should get CoverageException as on travis.

I've taken a look at this as well, my opinion is that a combination of configuration, subprocess use and cwd changes cause the problem. That's why you get the data file saved without arcs (branch=False).

There are thee ways to deal with this:

• Specify --cov-config=setup.cfg (pytest-cov will absolutize it before tmpworkdir is used). Alternatively, you could use a .coveragerc (pytest-cov absolutize that if it exists).
• Fix the test to not use subprocesses. Afaik the test spawns a suprocess that sigterms the parent. This makes no sense to me - you can do that with no subprocess at all. Eg: signal.kill(os.getpid(), signal.SIGTERM) or even a thread.
• Stop changing current working directory (tmpworkdir fixture).

There is no bug in either coverage or pytest-cov here.

Also, note that concurrency = multiprocessing is completely un-necessary if you use pytest-cov (it completely manages coverage measurements in subprocesses for you).

I've come to some different conclusions that Ionel did, though he was very helpful in getting me going with debugging with his aspectlib.

The problem here is the Process spawned by test_repeat_worker_close. It doesn't find the same configuration file that the rest of the tests do. Specifying --cov-config=setup.cfg doesn't help, because coverage.py only reads that file properly when it is read implicitly, not explicitly (that could be the subject of another bug report).

I think this is a combination of the direct use of Process, which perhaps coverage.py hasn't patched enough, and having settings in the implicitly-read setup.cfg file.

One simple solution is to move your coverage.py settings into a .coveragerc file. If you do that, things work.

Ned, thanks so much for the help. I've switched to using .coveragerc and that seems to have fixed it.

Perhaps a comment in the docs saying that using setup.cfg is not always the same as using .coveragerc would be useful?

pytest-cov could have an improvement to use setup.cfg if .coveragerc don't exist (same as coverage, but with pytest-cov's correct handling of relatiev paths). Open a bug report if you want it.

@Ionel Cristian Mărieș it isn't enough to tell coverage.py to use setup.cfg. It won't be read properly unless it is discovered implicitly.

What do you mean, "discovered implicitly"?

The sections are named differently ("[run]" vs "[coverage:run]"). If you specify setup.cfg as the file, coverage.py will try to read the "[run]" section and won't find it. It only looks for "[coverage:run]" when it is reading setup.cfg because it couldn't find .coveragerc.

Of course, this behavior could change, but that's the way the code is now.

I don't see how it's relevant. pytest-cov doesn't read the file, it will only pass the absolute path (if it exists).

The following patch retries with the coverage: prefix if a non default file was specified and nothing was read from it. There could be a cleaner way to do the same and I'm willing to work on it if you think it is worth implementing. To be fully backward compatible it should only retry with the coverage: prefix in the case where a file was provided and nothing was read from it because it serves no useful purpose. Unless there is a valid use case for --rcfile=/dev/null ?

diff -r 24aff3d7bfd5 coverage/config.py
--- a/coverage/config.py    Mon Dec 12 08:26:18 2016 +0100
+++ b/coverage/config.py    Thu Dec 15 10:35:14 2016 +0100
@@ -151,6 +151,7 @@
         # Metadata about the config.
         self.attempted_config_files = []
         self.config_files = []
+        self.any_set = False

         # Defaults for [run]
         self.branch = False
@@ -223,12 +224,11 @@

         self.config_files.extend(files_read)

-        any_set = False
         try:
             for option_spec in self.CONFIG_FILE_OPTIONS:
                 was_set = self._set_attr_from_config_option(cp, *option_spec)
                 if was_set:
-                    any_set = True
+                    self.any_set = True
         except ValueError as err:
             raise CoverageException("Couldn't read config file %s: %s" % (filename, err))

@@ -253,18 +253,18 @@
         if cp.has_section('paths'):
             for option in cp.options('paths'):
                 self.paths[option] = cp.getlist('paths', option)
-                any_set = True
+                self.any_set = True

         # plugins can have options
         for plugin in self.plugins:
             if cp.has_section(plugin):
                 self.plugin_options[plugin] = cp.get_section(plugin)
-                any_set = True
+                self.any_set = True

         # Was this file used as a config file? If no prefix, then it was used.
         # If a prefix, then it was only used if we found some settings in it.
         if section_prefix:
-            return any_set
+            return self.any_set
         else:
             return True

@@ -422,9 +422,12 @@
             config_read = config.from_file(fname, section_prefix=prefix)
             is_config_file = fname == config_file

-            if not config_read and is_config_file and specified_file:
-                raise CoverageException("Couldn't read '%s' as a config file" % fname)
-
+            if is_config_file and specified_file:
+                if not config_read:
+                    raise CoverageException("Couldn't read '%s' as a config file" % fname)
+                if not config.any_set:
+                    config_read = config.from_file(fname, section_prefix="coverage:")
+                    
             if config_read:
                 break

I think I may have hit this issue too.

Builds used to pass without any problems, but when moving the configuration from .coveragerc to setup.cfg (https://github.com/Peque/osbrain/commit/f9eb8c65c5f0e16618b6602786e74eaff9e46a4a), it fails (https://travis-ci.org/Peque/osbrain/builds/237146025?utm_source=github_status&utm_medium=notification).

Maybe you need to tell pytest-cov what is the coverage config file (eg --cov-config=setup.cfg).

@Ionel Cristian Mărieș Thanks. That helped and it currently finishes the report. However, the source and omit parameters are ignored.

Have not figured out why. Here is the repo/branch: https://github.com/Peque/osbrain/tree/coverage

Development has moved to GitHub. This issue has been moved to https://github.com/nedbat/coveragepy/issues/512