TestPackageIndex Frequently fails on Python 3

Issue #315 resolved
Lennart Regebro
created an issue

setuptools.tests.test_packageindex.TestPackageIndex will intermittently fail on Python 3.2 and 3.3 (I haven't seen it fail on 3.1 yet).

Traceback (most recent call last): File "/home/projects/distribute/distribute/build/src/setuptools/tests/test_packageindex.py", line 128, in test_links_priority server.stop() File "/home/projects/distribute/distribute/build/src/setuptools/tests/server.py", line 44, in stop urllib.request.urlopen(url, timeout=5) File "/opt/python33/lib/python3.3/urllib/request.py", line 160, in urlopen return opener.open(url, data, timeout) File "/opt/python33/lib/python3.3/urllib/request.py", line 473, in open response = self._open(req, data) File "/opt/python33/lib/python3.3/urllib/request.py", line 491, in _open '_open', req) File "/opt/python33/lib/python3.3/urllib/request.py", line 451, in _call_chain result = func(*args) File "/opt/python33/lib/python3.3/urllib/request.py", line 1272, in http_open return self.do_open(http.client.HTTPConnection, req) File "/opt/python33/lib/python3.3/urllib/request.py", line 1257, in do_open r = h.getresponse() File "/opt/python33/lib/python3.3/http/client.py", line 1131, in getresponse response.begin() File "/opt/python33/lib/python3.3/http/client.py", line 354, in begin version, status, reason = self._read_status() File "/opt/python33/lib/python3.3/http/client.py", line 316, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/opt/python33/lib/python3.3/socket.py", line 297, in readinto return self._sock.recv_into(b) socket.timeout: timed out

Since it works sometimes, especially of you run it quickly after the failure, this is likely an bug in the tests, and not in the actual code.

Comments (4)

  1. Jason R. Coombs

    I've observed this as well. It looks like a socket timeout on read. Could it just be that the timeouts are close to the threshold for the time it takes PyPI to respond to that query? I see the timeout is set to 5, so if it takes PyPI more than 5 seconds to load the page, that would trigger the error.

    I found an example of this error occurring on Python 2.7 as well. I believe it's just an intermittent bug that's grown worse over time due to the growth of PyPI (and the resulting degraded performance). Perhaps the solution is to just bump the timeout to 8 seconds?

  2. Lennart Regebro reporter

    It is in fact a local server, and the problem comes when stopping it. And if it was just a question of something being to slow, this error would appear as frequently on all Python versions. This seems to be some change that happened in Python 3.2.

    Quick investigations seem to indicate that the problem in fact is that the server exits before handling the upcoming request, which is done to make sure the shutdown request is handled.

  3. Lennart Regebro reporter

    I *was* going to just note this problem and then ignore it, but Jasons reply had me looking deeper into it, and what happens is that the stop() method is called before the server is done dealing with the last request, setting the _run flag to False. Then then next request (the one that is supposed to make the server notice it should shutdown) is called. At this point the server finishes the last request, notices the _run flag is False, and quits. This leaves the new request hanging until it times out.

    Adding a time.sleep(0.1) before setting _run = False solves the problem.

    The only change I can find in 3.2's What's New that could cause this is the reworking of the GIL which in theory could mean that the main thread continues to run until it ends up trying to open the socket to the server thread. But that's based entirely on me not coming up with any better explanations.

  4. Log in to comment