Issue #6383 resolved

Unstable connection to GIT repositories

Peter Meyer
created an issue

Hi there,

We are having trouble accessing our GIT repositories. This shows as longer waiting time than normal, and sometimes timeouts, when pushing and pulling code via SSH connection using SSH keys.

For instance, a colleague got the following message while trying to pull code: Stopping at 'sites/all/themes/custom/alleradmin'; script returned non-zero status.

We usually don't have any problems accessing our GIT repositories via SSH with SSH keys, so this is strange behavior.

Are there any stability issues at the moment? Can you help us with this?

Regards, Peter Meyer

Comments (7)

  1. Brian Nguyen staff

    Hi Peter,

    We haven't seen this issue before. It sounds like the commit hook failed for some reason, but unfortunately its hard to tell from here.

    Are you still getting these errors? If so, would you be able to tell us what repository this problem occurs on? We can then investigate it from our end.

    Cheers, Brian

  2. Peter Meyer reporter

    Hi Brian,

    We are still having problems, and it seems to be with all our repositories.

    A colleague did a verbose debug by following this support article: https://confluence.atlassian.com/display/BITBUCKET/Troubleshooting+SSH+Issues#TroubleshootingSSHIssues-ssh-Tconnectiontest%28GitBash/MacOSX/Linux%29

    Then he gets the following info, where (STALL) is the point it stalls, then it just continues after a while without any error, but sometimes it stop completely.

    HOME:~ ricken$ ssh -Tv git@bitbucket.org OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
    debug1: Reading configuration data /etc/ssh_config
    debug1: Connecting to bitbucket.org [207.223.240.182] port 22.
    debug1: connect to address 207.223.240.182 port 22: Operation timed out
    debug1: Connecting to bitbucket.org [207.223.240.181] port 22. (STALL)
    debug1: Connection established.
    debug1: identity file /Users/ricken/.ssh/identity type -1
    debug1: identity file /Users/ricken/.ssh/id_rsa type 1
    debug1: identity file /Users/ricken/.ssh/id_dsa type -1
    debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
    debug1: match: OpenSSH_5.3 pat OpenSSH*
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_5.2
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: server->client aes128-ctr hmac-md5 none
    debug1: kex: client->server aes128-ctr hmac-md5 none
    debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
    debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
    debug1: Host 'bitbucket.org' is known and matches the RSA host key.
    debug1: Found key in /Users/ricken/.ssh/known_hosts:50
    debug1: ssh_rsa_verify: signature correct
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey
    debug1: Next authentication method: publickey
    debug1: Trying private key: /Users/ricken/.ssh/identity
    debug1: Offering public key: /Users/ricken/.ssh/id_rsa
    debug1: Remote: Forced command: conq username:lorique
    debug1: Remote: Port forwarding disabled.
    debug1: Remote: X11 forwarding disabled.
    debug1: Remote: Agent forwarding disabled.
    debug1: Remote: Pty allocation disabled.
    debug1: Server accepts key: pkalg ssh-rsa blen 277
    debug1: read PEM private key done: type RSA
    debug1: Remote: Forced command: conq username:lorique
    debug1: Remote: Port forwarding disabled.
    debug1: Remote: X11 forwarding disabled.
    debug1: Remote: Agent forwarding disabled.
    debug1: Remote: Pty allocation disabled.
    debug1: Authentication succeeded (publickey).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    conq: logged in as lorique.
    
    You can use git or hg to connect to Bitbucket. Shell access is disabled.
    debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
    debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
    debug1: channel 0: free: client-session, nchannels 1
    Transferred: sent 2176, received 2888 bytes, in 0.4 seconds Bytes per second: sent 5610.4, received 7446.1
    debug1: Exit status 0
    

    This is a very big problem for us, because we have setup continues deploy with Jenkins, which we now don't know is getting all code succesfully. It is set up to get code from several repositories and merge the output of those. So we now have big trouble getting code to production sites.

    Thanks in advance for your help.

    Regards, Peter Meyer

  3. Brian Nguyen staff

    Hi Peter,

    Looking at that log, the ssh connection fails the first time with

    debug1: Connecting to bitbucket.org [207.223.240.182] port 22.
    debug1: connect to address 207.223.240.182 port 22: Operation timed out
    

    However the connection is established the second time. After that point, the key has been sent and the session authenticated as the user lorique.

    We shouldn't have any intermittent issues establishing a connection like that, so is there anything on your side that may be closing a connection to bitbucket?

    Also how large are your repositories that Jenkins is cloning. If the repositories are excessively large (around 2GB or more), the clone may be taking so long that the connection is closed before the clone is complete.

    Cheers, Brian

  4. Erik van Zijst staff

    Looking at that output, the problem is not with the push's payload, but with actually establishing a TCP connection, which is odd and not something we have seen before.

    Can you do a ping to our servers and see if there is any packet loss?

  5. Log in to comment