Mocking os.path.exists causes failures

Issue #416 resolved
Anonymous created an issue

When coverage is run on the OpenStack Octavia project with version 4.0 of coverage multiple unit tests fail. However, if those unit tests are run without coverage or with version 3.7.1 of coverage the tests pass.

You can reproduce these errors with the following steps: 1. Checkout Octavia: git fetch https://review.openstack.org/openstack/octavia refs/changes/48/225548/3 && git checkout FETCH_HEAD 2. Change into the octavia directory: cd octavia 3. Run the example tests:

The file path issue: Passing test: tox -e py27 -- octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_listener Failing test: tox -e cover -- octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_listener

The StopIteration issue: Passing: tox -e py27 -- octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_cert Fails: tox -e cover -- octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_cert

To test with version 3.7.1 of coverage edit the test-requirements.txt file: replace: coverage>=3.6 with: coverage!=4.0,>=3.6

Run the above tox tests with the "r" flag to have it reload the python modules. Example: tox -re cover -- octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_listener

Note, the tests pass with version 3.7.1

Below is an example error when running with version 4.0:

======================================================================
FAIL: octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_delete_listener
tags: worker-1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "octavia/tests/functional/amphorae/backend/agent/api_server/test_server.py", line 196, in test_delete_listener
    mock_exists.assert_called_with('/var/lib/octavia/123/haproxy.cfg')
  File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/six.py", line 692, in raise_from
    raise value
AssertionError: Expected call: exists('/var/lib/octavia/123/haproxy.cfg')
Actual call: exists('/usr/lib/python2.7/stringprep.py')

Comments (10)

  1. Michael Johnson

    Yes, we are 100% sure this is not related to the ignore-errors typo as we have fixed that issue in a separate patch. These are two completely different issues. With the typo the tests don't run. Here they run but return incorrect results.

  2. Michael Johnson

    I wanted to add a stack trace for the other "StopIteration" issue that we are seeing with coverage 4.0. It seems to be related.

    Basically any test that we wrap the method with "@mock.patch('os.path.exists') and then set a side effect against the mock or assert_called the mock will fail with coverage 4.0 but not 3.7.1.

    ======================================================================
    FAIL: octavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_get_certificate_md5
    tags: worker-3
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File "octavia/tests/functional/amphorae/backend/agent/api_server/test_server.py", line 384, in test_get_certificate_md5
        '/listeners/123/certificates/test.pem')
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/test.py", line 774, in get
        return self.open(*args, **kw)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/flask/testing.py", line 108, in open
        follow_redirects=follow_redirects)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/test.py", line 742, in open
        response = self.run_wsgi_app(environ, buffered=buffered)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/test.py", line 661, in run_wsgi_app
        self.cookie_jar.extract_wsgi(environ, rv[2])
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/test.py", line 187, in extract_wsgi
        U2Request(get_current_url(environ)),
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/wsgi.py", line 91, in get_current_url
        return uri_to_iri(''.join(tmp))
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/urls.py", line 607, in uri_to_iri
        return url_unparse((uri.scheme, uri.decode_netloc(),
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/urls.py", line 150, in decode_netloc
        rv = _decode_idna(self.host or '')
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/werkzeug/_internal.py", line 324, in _decode_idna
        parts[idx] = part.decode('idna')
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/lib/python2.7/encodings/__init__.py", line 100, in search_function
        level=0)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/coverage/control.py", line 580, in _should_trace
        disp = self._should_trace_internal(filename, frame)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/coverage/control.py", line 455, in _should_trace_internal
        filename = self._source_for_file(dunder_file)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/coverage/control.py", line 375, in _source_for_file
        if os.path.exists(py_filename):
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
        return _mock_self._mock_call(*args, **kwargs)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 1121, in _mock_call
        result = next(effect)
      File "/mnt/projects/coveragefix/octavia.cover/.tox/cover/local/lib/python2.7/site-packages/mock/mock.py", line 127, in next
        return _next(obj)
    StopIteration
    
  3. Log in to comment