coverage==4.0 hangs indefinitely on python2.7 + windows

Issue #420 resolved
Anthony Sottile
created an issue

The reproduction here is a bit complicated because I haven't quite narrowed it down yet but here goes.

I noticed this initially when my build was running indefinitely (and eventually failing) here : https://ci.appveyor.com/project/asottile/pre-commit/branch/master/job/r57jhonel6dlgd9d

My "minimal" reproduction is as follows:

  1. git clone git://github.com/pre-commit/pre-commit
  2. cd pre-commit
  3. virtualenv venv
  4. . venv/Scripts/activate
  5. pip install -r requirements-dev.txt
  6. pip install 'coverage<4'
  7. coverage run -m pytest tests -k test_parse_merge # Runs and succeeds
  8. pip install 'coverage>=4'
  9. coverage run -m pytest tests -k test_parse_merge # Hangs forever

I've attached a screenshot (the contrast sucks because the window froze on my VM)

Comments (13)

  1. Ionel Cristian Mărieș

    It would appear this is the minimal reproduce instructions (for now):

    Put this in a conftest.py:

    import subprocess
    
    def cmd_output(*cmd, **kwargs):
        proc = subprocess.Popen(
            cmd,
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        stdout, stderr = proc.communicate()
        returncode = proc.poll()
    
        if returncode is not None and returncode:
            raise CalledProcessError(
                returncode, cmd, retcode, output=(stdout, stderr),
            )
        return returncode, stdout, stderr
    
    print(cmd_output('python', '-c', 'print(123)'))
    

    And then run:

    virtualenv ve
    ve\scripts\pip install pytest coverage
    ve\scripts\coverage run -m pytest --help #gets stuck
    

    It would appear that the conftest gets run somewhere in the middle of pytest initializing all the plugins (including the capture plugin).

  2. Anthony Sottile reporter

    So I went to bisect this and I can't seem to reproduce when checking out the tagged coverage-4.0.

    I'm using the git mirror because I understand git (for bisect) way better than I do mercurial.

    # C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> cat .\asottile\asottile.py
    import subprocess
    import traceback
    import threading
    
    SUCCESS = 0
    TIMEOUT = 1
    FAILED = 2
    
    
    def call_with_timeout(*cmd, **kwargs):
        timeout = kwargs.pop('timeout', 10)
    
        class out:
            pass
    
        def execute_command():
            out.proc = subprocess.Popen(cmd, **kwargs)
            out.proc.communicate()
            if out.proc.returncode:
                out.reason = FAILED
            else:
                out.reason = SUCCESS
    
        thread = threading.Thread(target=execute_command)
        thread.start()
    
        thread.join(timeout)
    
        if thread.is_alive():
            out.proc.terminate()
            thread.join()
            out.reason = TIMEOUT
    
        return out.reason
    
    
    def main():
        try:
            subprocess.check_call(('rm', '-rf', 'venv'))
            subprocess.check_call(('virtualenv', 'venv'))
            subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
        except Exception:
            print('*' * 79)
            print('Assuming not installable')
            print('*' * 79)
            traceback.print_exc()
            return 0
    
        return call_with_timeout('venv/Scripts/coverage.exe', 'run', '-m', 'pytest', '--help', timeout=2)
    
    if __name__ == '__main__':
        exit(main())
    

    With this (checked out at https://github.com/nedbat/coveragepy/releases/tag/coverage-4.0), I get the following:

    New python executable in venv\Scripts\python.exe
    Installing setuptools, pip...done.
    Processing c:\users\anthony\desktop\git\coveragepy
    Collecting pytest
      Using cached pytest-2.8.1-py2.py3-none-any.whl
    Collecting py>=1.4.29 (from pytest)
      Using cached py-1.4.30-py2.py3-none-any.whl
    Collecting colorama (from pytest)
      Using cached colorama-0.3.3.tar.gz
    Installing collected packages: py, colorama, pytest, coverage
      Running setup.py install for colorama
      Running setup.py install for coverage
    Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
    usage: pytest.py [options] [file_or_dir] [file_or_dir] [...]
    
    # The rest of pytests's help
    
    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> echo $LASTEXITCODE
    0
    

    However if I modify my script to just be:

    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> C:\\Users\\Anthony\\AppData\\Local\\GitHub\\PortableGit_c2ba306e5
    36fdf878271f7fe636a147ff37326ad\\bin\diff.exe -u .\asottile\asottile.py .\asottile\asottile2.py
    --- .\asottile\asottile.py      Mon Oct  5 19:19:09 2015
    +++ .\asottile\asottile2.py     Mon Oct  5 19:19:28 2015
    @@ -38,7 +38,7 @@
         try:
             subprocess.check_call(('rm', '-rf', 'venv'))
             subprocess.check_call(('virtualenv', 'venv'))
    -        subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
    +        subprocess.check_call(('venv/Scripts/pip.exe', 'install', 'coverage', 'pytest'))
         except Exception:
             print('*' * 79)
             print('Assuming not installable')
    

    I get:

    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> python .\asottile\asottile2.py
    New python executable in venv\Scripts\python.exe
    Installing setuptools, pip...done.
    ←[33mYou are using pip version 6.1.1, however version 7.1.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.←[0m
    Collecting coverage
      Using cached coverage-4.0-cp27-none-win32.whl
    Collecting pytest
      Using cached pytest-2.8.1-py2.py3-none-any.whl
    Collecting py>=1.4.29 (from pytest)
      Using cached py-1.4.30-py2.py3-none-any.whl
    Collecting colorama (from pytest)
      Using cached colorama-0.3.3.tar.gz
    Installing collected packages: coverage, py, colorama, pytest
      Running setup.py install for colorama
    Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
    C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> echo $LASTEXITCODE
    1
    

    Maybe a bad wheel?

  3. Anthony Sottile reporter

    I'm getting somewhere, turns out the wheels / sdists that were "working" weren't building the C bits, I can reliably get both sdist and wheel to fail now -> to git bisect!

  4. Anthony Sottile reporter

    Bisect points at this commit:

    https://github.com/nedbat/coveragepy/commit/81f56

       70  git bisect start
       71  git bisect bad `git rev-parse HEAD`
       72  git bisect good `git rev-parse coverage-3.7.1`
       73  git bisect run python asottile/asottile.py
    
    ...
    
    81f5697e7cb2f5a064fd891617c0c8caf5ab21d9 is the first bad commit
    commit 81f5697e7cb2f5a064fd891617c0c8caf5ab21d9
    Author: Ned Batchelder <ned@nedbatchelder.com>
    Date:   Sat Nov 8 18:42:32 2014 -0500
    
        Use a WeakKeyDictionary to track coroutine objects to prevent leaks. Fixes #330.
    
    :040000 040000 4d1c3da10c0447aca06560a908d48a5f726d1803 517dc4f3cc1ce573673e1937b7250613cd1f02b3 M      coverage
    :040000 040000 34a2a5d7944a390cada7f2b71f16fee9e1ca68fc 02892b0265447b6e85f0c2485ae73eb3978ce03c M      tests
    bisect run success
    
  5. Anthony Sottile reporter

    For what it's worth here's a simpler reproduction. Removing any line here causes it to pass.

    # test.py
    import test2
    
    # test2.py
    import subprocess
    subprocess.Popen(
        ('echo',),
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
    ).communicate()
    

    And coverage run -m test

  6. Anthony Sottile reporter

    Yay, seems fixed:

    (ignore the sha, that's from the git repo I mercurial cloned in a subdir (sanity is not always my forte))

    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ pip wheel .
    Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy
    Building wheels for collected packages: coverage
      Running setup.py bdist_wheel for coverage
      Stored in directory: c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse
    Successfully built coverage
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ ls wheelhouse/
    coverage-4.0.3-cp27-none-win32.whl
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ pip install wheelhouse/coverage-4.0.3-cp27-none-win32.whl
    Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse\coverage-4.0.3-cp27-none-win32.whl
    Installing collected packages: coverage
    Successfully installed coverage-4.0.3
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test
    test.py    test2.py   test2.pyc  tests/
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test
    test.py    test2.py   test2.pyc  tests/
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ cp ../../test*.py ./
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ coverage run -m test
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $ echo $?
    0
    (venv)
    IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
    $
    
  7. Log in to comment