This hopefully resolves issue #1068. All I did was ensure that SSL_fileobject.recv() so that it never returns more than the number of bytes it was asked for, which is what the assert statement on line 1055 of wsgiserver2.py is checking on - that's where the failure occurs.
I'm not really sure if this is the best or most efficient way to do it, but this definitely appears to resolve it on my end, and it does not cause additional unit test failures.
I did not add unit tests for this because no unit tests are run in HTTPS mode, however I think someone with the required expertise should consider having the unit tests run in that mode.
Why are we buffering a call with a fixed size as an argument?
I have make a couple of test with some 500MB files and is working fine, the call to self._sock.pending()
is not required for the purpose of that method.
BUT after thinking a little more I see that probably is a good idea to buffer the method and ask for pending plus
verifying the size as you suggest. Probably the examples that I'm making are not a real use case because of the
low latency of the localhost. Anyway the real issue is indeed in this method not in the caller in wsgiserver2.py,
the thing that's bothers me is that on python3 is working fine and the almost exactly the same implementation
(except for some extra locking ) in CP3.1 is also working fine.
Anyway, I'm glad you think this is the right way to go.
Per the OpenSSLConnection documentation (emphasis mine):
Receive data from the Connection. The return value is a string representing the data received. The maximum amount of data to be received at once, is specified by bufsize.
And that's what appears to be the problem - at least to my inexperienced eyes - more than the maximum amount of data might be retrieved for some reason. As you've seen, this will check for that and stop appending to the buffer if it's going to exceed the maximum.
Finally everything has fall into place, I do have a reason why it is working on CP3.1 and Python3 (this is pretty simple).
This use to work on CP3.1 because the body of the request was read with readline which does not verify the
length of the data that recv returns, also it always call recv with a fixed argument, the default read size of socket self._rbufsize
which in my platform is 8192 in CP3.1 recv does return more than gets asked, almost always 2 * 8192 bytes, because of a
coincidence of the record size of openssl -> http://repo.or.cz/w/mirror-openssl.git/blob/HEAD:/ssl/ssl2.h#l141
This works in Python3 because the wsgiserver3.py does not use pyopenssl, and therefore does not usessl_pyopenssl.py
:|, simple as that.
The reason to not include any call to self._sock.pending to the fix is because that method only verifies that the underlying ssl connection does not have any buffered bytes of previously fetched records (e.g. records of 16383 bytes),
and most important the buffering between records is already handled by SSL_read
which is wrapped by pyOpenSSL at the SSLConnection.recv. In particular that recv method does not support the -1 and
the existence of it is just to be thread safe, not to add more buffering magic.