coverage==4.0 hangs indefinitely on python2.7 + windows

Issue #420 resolved
Anthony Sottile created an issue

The reproduction here is a bit complicated because I haven't quite narrowed it down yet but here goes.

I noticed this initially when my build was running indefinitely (and eventually failing) here : https://ci.appveyor.com/project/asottile/pre-commit/branch/master/job/r57jhonel6dlgd9d

My "minimal" reproduction is as follows:

  1. git clone git://github.com/pre-commit/pre-commit
  2. cd pre-commit
  3. virtualenv venv
  4. . venv/Scripts/activate
  5. pip install -r requirements-dev.txt
  6. pip install 'coverage<4'
  7. coverage run -m pytest tests -k test_parse_merge # Runs and succeeds
  8. pip install 'coverage>=4'
  9. coverage run -m pytest tests -k test_parse_merge # Hangs forever

I've attached a screenshot (the contrast sucks because the window froze on my VM)

Comments (13)

  1. Ionel Cristian Mărieș

    It would appear this is the minimal reproduce instructions (for now):

    Put this in a conftest.py:

    import subprocess
    
    def cmd_output(*cmd, **kwargs):
        proc = subprocess.Popen(
            cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = proc.communicate()
        returncode = proc.poll()
    
        if returncode is not None and returncode:
            raise CalledProcessError(
                returncode, cmd, retcode, output=(stdout, stderr),
            )
        return returncode, stdout, stderr
    
    print(cmd_output('python', '-c', 'print(123)'))
    

    And then run:

    virtualenv ve
    ve\scripts\pip install pytest coverage
    ve\scripts\coverage run -m pytest --help #gets stuck
    

    It would appear that the conftest gets run somewhere in the middle of pytest initializing all the plugins (including the capture plugin).

  2. Anthony Sottile reporter

    So I went to bisect this and I can't seem to reproduce when checking out the tagged coverage-4.0.

    I'm using the git mirror because I understand git (for bisect) way better than I do mercurial.

    # C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> cat .\asottile\asottile.py
    import subprocess
    import traceback
    import threading
    
    SUCCESS = 0
    TIMEOUT = 1
    FAILED = 2
    
    
    def call_with_timeout(*cmd, **kwargs):
        timeout = kwargs.pop('timeout', 10)
    
        class out:
            pass
    
        def execute_command():
            out.proc = subprocess.Popen(cmd, **kwargs)
            out.proc.communicate()
            if out.proc.returncode:
                out.reason = FAILED
            else:
                out.reason = SUCCESS
    
        thread = threading.Thread(target=execute_command)
        thread.start()
    
        thread.join(timeout)
    
        if thread.is_alive():
            out.proc.terminate()
            thread.join()
            out.reason = TIMEOUT
    
        return out.reason
    
    
    def main():
        try:
            subprocess.check_call(('rm', '-rf', 'venv'))
            subprocess.check_call(('virtualenv', 'venv'))
            subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
        except Exception:
            print('*' * 79)
            print('Assuming not installable')
            print('*' * 79)
            traceback.print_exc()
            return 0
    
        return call_with_timeout('venv/Scripts/coverage.exe', 'run', '-m', 'pytest', '--help', timeout=2)
    
    if __name__ == '__main__':
        exit(main())
    

    With this (checked out at https://github.com/nedbat/coveragepy/releases/tag/coverage-4.0), I get the following:

    New python executable in venv\Scripts\python.exe
    Installing setuptools, pip...done.
    Processing c:\users\anthony\desktop\git\coveragepy
    Collecting pytest
      Using cached pytest-2.8.1-py2.py3-none-any.whl
    Collecting py>=1.4.29 (from pytest)
      Using cached py-1.4.30-py2.py3-none-any.whl
    Collecting colorama (from pytest)
      Using cached colorama-0.3.3.tar.gz
    Installing collected packages: py, colorama, pytest, coverage
      Running setup.py install for colorama
      Running setup.py install for coverage
    Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
    usage: pytest.py [options] [file_or_dir] [file_or_dir] [...]
    
    # The rest of pytests's help
    
    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> echo $LASTEXITCODE
    0
    

    However if I modify my script to just be:

    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> C:\\Users\\Anthony\\AppData\\Local\\GitHub\\PortableGit_c2ba306e5
    36fdf878271f7fe636a147ff37326ad\\bin\diff.exe -u .\asottile\asottile.py .\asottile\asottile2.py
    --- .\asottile\asottile.py      Mon Oct  5 19:19:09 2015
    +++ .\asottile\asottile2.py     Mon Oct  5 19:19:28 2015
    @@ -38,7 +38,7 @@
         try:
             subprocess.check_call(('rm', '-rf', 'venv'))
             subprocess.check_call(('virtualenv', 'venv'))
    -        subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
    +        subprocess.check_call(('venv/Scripts/pip.exe', 'install', 'coverage', 'pytest'))
         except Exception:
             print('*' * 79)
             print('Assuming not installable')
    

    I get:

    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> python .\asottile\asottile2.py
    New python executable in venv\Scripts\python.exe
    Installing setuptools, pip...done.
    ←[33mYou are using pip version 6.1.1, however version 7.1.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.←[0m
    Collecting coverage
      Using cached coverage-4.0-cp27-none-win32.whl
    Collecting pytest
      Using cached pytest-2.8.1-py2.py3-none-any.whl
    Collecting py>=1.4.29 (from pytest)
      Using cached py-1.4.30-py2.py3-none-any.whl
    Collecting colorama (from pytest)
      Using cached colorama-0.3.3.tar.gz
    Installing collected packages: coverage, py, colorama, pytest
      Running setup.py install for colorama
    Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> echo $LASTEXITCODE
    1
    

    Maybe a bad wheel?

  3. Anthony Sottile reporter

    In fact, changing it to pip install --no-use-wheel coverage pytest, my little test also succeeds.

  4. Anthony Sottile reporter

    I'm getting somewhere, turns out the wheels / sdists that were "working" weren't building the C bits, I can reliably get both sdist and wheel to fail now -> to git bisect!

  5. Anthony Sottile reporter

    Bisect points at this commit:

    https://github.com/nedbat/coveragepy/commit/81f56

       70  git bisect start
       71  git bisect bad `git rev-parse HEAD`
       72  git bisect good `git rev-parse coverage-3.7.1`
       73  git bisect run python asottile/asottile.py
    
    ...
    
    81f5697e7cb2f5a064fd891617c0c8caf5ab21d9 is the first bad commit
    commit 81f5697e7cb2f5a064fd891617c0c8caf5ab21d9
    Author: Ned Batchelder <ned@nedbatchelder.com>
    Date:   Sat Nov 8 18:42:32 2014 -0500
    
        Use a WeakKeyDictionary to track coroutine objects to prevent leaks. Fixes #330.
    
    :040000 040000 4d1c3da10c0447aca06560a908d48a5f726d1803 517dc4f3cc1ce573673e1937b7250613cd1f02b3 M      coverage
    :040000 040000 34a2a5d7944a390cada7f2b71f16fee9e1ca68fc 02892b0265447b6e85f0c2485ae73eb3978ce03c M      tests
    bisect run success
    
  6. Ned Batchelder repo owner

    I've confirmed that @ionelmc's conftest.py demonstrates the problem, and that the commit @asottile found is the first showing it. Its parent does not have the problem. Any theories about what's wrong with that commit?

  7. Anthony Sottile reporter

    For what it's worth here's a simpler reproduction. Removing any line here causes it to pass.

    # test.py
    import test2
    
    # test2.py
    import subprocess
    subprocess.Popen(
        ('echo',),
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
    ).communicate()
    

    And coverage run -m test

  8. Anthony Sottile reporter

    Yay, seems fixed:

    (ignore the sha, that's from the git repo I mercurial cloned in a subdir (sanity is not always my forte))

    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ pip wheel .
    Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy
    Building wheels for collected packages: coverage
      Running setup.py bdist_wheel for coverage
      Stored in directory: c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse
    Successfully built coverage
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ ls wheelhouse/
    coverage-4.0.3-cp27-none-win32.whl
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ pip install wheelhouse/coverage-4.0.3-cp27-none-win32.whl
    Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse\coverage-4.0.3-cp27-none-win32.whl
    Installing collected packages: coverage
    Successfully installed coverage-4.0.3
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test
    test.py    test2.py   test2.pyc  tests/
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test
    test.py    test2.py   test2.pyc  tests/
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test*.py ./
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ coverage run -m test
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ echo $?
    0
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $
    
  9. Log in to comment