Cloning Git repos is very unreliable over HTTPS for large repos

Issue #13812 resolved
Richard Bresnan
created an issue

Often Bitbucket is unable to successfully clone repositories using the HTTPS protocol. A typical error response is:

Cloning Repo: git@bitbucket.org:team/repo.git
error: RPC failed; curl 52 Empty reply from server

Failed workarounds:

  • We tried setting this config variable as a workaround, but it does not work: git config --global http.postBuffer 524288000

  • We implemented retries, but this does not work either.

A reliable workaround exists, which is to use SSH cloning instead, however this is not always possible.

It appears as though once the repository get into a certain state the error occurs deterministically. This appears to be at least somewhat related to the repository size - repositories over 800 MB are more likely to be affected.

In the past, we've been told that affected repositories needed git gc to be run on the server or that "repository/Git/network configurations need to be tweaked", however this is not something that we can do as a Bitbucket user.

Official response

  • Jonathan Robson staff

    This is most likely caused by a VPN or a proxy sitting between you and Bitbucket. I would recommend approaching the problem from that perspective. You might start by trying to reproduce the problem from a different location (e.g. from home instead of your office). If you can consistently reproduce the problem from multiple locations and can provide steps for us to consistently reproduce it as well, feel free to reopen this issue.

Comments (13)

  1. Jonathan Robson staff

    @Richard Bresnan (or anyone else experiencing this issue) I was wondering if you could answer some questions to help me figure this out for you...

    1. What version of Git are you using?
    2. Are you cloning over a VPN?
    3. Do you have anything set for http.proxy in your Git config?
    4. Is that error message you shared the full error message? If not, can you provide it?
  2. Zack Moore

    I ran into this issue also using Hg on a < 500MB repo. Clones fine on my desktop but on build server it is timing out. I was able to resolve the issue by switching to ssh but 500 MB is not that large of a repo. Switching to ssh was not a great solution as TeamCity's ssh support for Hg is not great and required a lot of manual config edits and limits using a lot advanced features such as agent checkout.

  3. Jonathan Robson staff

    I've been working on this full time for over a week now, with some help from other Bitbucket developers and members of our ops team, and we've been unable to reproduce any reliability issues with cloning large Git repos over HTTPS. The original error message from the description of this ticket happens under very specific conditions (according to the Git client), and it's most likely caused by a VPN or a proxy sitting between you and Bitbucket.

    I'll explain...

    When you clone a Git repo over HTTPS (or just HTTP), the first step is the Git client makes a GET request to the Git server asking for info about the repo's "refs" (like branches and tags). The server responds with a list of all refs in the repo and commits they point to. Then the client figures out what commits it wants and which ones it already has. In the last step, the client sends a POST request to the server with a list of "want" and "have" commits, and the server generates a "packfile" containing all of the objects the client is requesting and returns it in the response. That second POST request is an RPC (remote procedure call) request because it's basically just a small wrapper around the underlying Git command that it runs.

    The error message happens when the client makes the RPC request and the server appears to close the connection without sending any sort of response back, no headers or anything. Hence the error message:

    error: RPC failed; curl 52 Empty reply from server
    

    The second part of that error message is from the "libcurl" C library that the Git client uses to make HTTP requests. Even outside the context of Git, that error message is a very common symptom of a VPN or proxy issue. Typically, when any web server encounters an error, it returns some sort of HTTP response for that error, even if it's an unknown error. It's unlikely for some sort of error to happen on the server that would cause it to close the connection without ever sending any sort of response. Of course, it's still possible. But you described the problem as "very unreliable" and that in some states "the error occurs deterministically," implying that it's not unlikely. If an error like that was was as predictable or widespread as your suggesting, it would have been hard for us not to notice. We have external services that continuously monitor our reliability and availability over the internet, and we haven't noticed anything like this. Also, as I mentioned above, I've tried very hard over the past week or so to reproduce this in any way I can think of, and I haven't been able to.

    You also mentioned that it might have to do with whether or not garbage collection needed to run. Regardless of whether garbage collection needs to run, the actual HTTP requests are the same because it generates a packfile with the needed objects, even if the Git objects on the server are not packed. That error message is either a very serious but unlikely problem with the web server or a networking issue, not a problem with the underlying application that sits behind the web server.

    As I said above, this is most likely caused by a VPN or a proxy sitting between you and Bitbucket. I would recommend approaching the problem from that perspective. You might start by trying to reproduce the problem from a different location (e.g. from home instead of your office). If you can consistently reproduce the problem from multiple locations and can provide steps for us to consistently reproduce it as well, feel free to reopen this issue.

  4. Jonathan Robson staff

    @Zack Moore This issue was mainly for cloning Git repos over HTTPS. Although, if you're getting the same error message mentioned above, you might be experiencing a similar problem to the one I described in my last comment. If it's a different problem, we should probably create a separate issue to keep track of it.

  5. Jonathan Robson staff

    This is most likely caused by a VPN or a proxy sitting between you and Bitbucket. I would recommend approaching the problem from that perspective. You might start by trying to reproduce the problem from a different location (e.g. from home instead of your office). If you can consistently reproduce the problem from multiple locations and can provide steps for us to consistently reproduce it as well, feel free to reopen this issue.

  6. Log in to comment