Issue #7574 resolved

Slow Repo Clone speeds in AWS EU-WEST (BB-8781)

MetaBroadcast
created an issue

Hi,

We've been intermittently experiencing extremely slow git clone speeds for our BitBucket repository 'mbst/infra' in our AWS Infrastructure in Region eu-west. This has been exhibited on multiple Availability Zones, and on different instance types. While normally clone speeds are >1MB/s, we see sustained periods of time where the clone speed is ~25-50KB/s. I've tested with other services like GitHub and the clone speeds there are unaffected.

I have included a tracepath for an instance which is actively experiencing issues retrieving a repo from bitbucket.org:

traceroute bitbucket.org
traceroute to bitbucket.org (207.223.240.182), 30 hops max, 60 byte packets
 1  ip-10-56-64-2.eu-west-1.compute.internal (10.56.64.2)  0.436 ms  0.450 ms  0.557 ms
 2  ip-10-1-72-0.eu-west-1.compute.internal (10.1.72.0)  0.377 ms  0.520 ms ip-10-1-73-0.eu-west-1.compute.internal (10.1.73.0)  0.753 ms
 3  100.64.12.198 (100.64.12.198)  0.531 ms 100.64.12.196 (100.64.12.196)  0.462 ms 100.64.12.6 (100.64.12.6)  0.446 ms
 4  100.64.12.137 (100.64.12.137)  0.466 ms 100.64.12.9 (100.64.12.9)  0.548 ms 100.64.12.201 (100.64.12.201)  0.636 ms
 5  ec2-79-125-1-100.eu-west-1.compute.amazonaws.com (79.125.1.100)  0.815 ms ec2-79-125-1-96.eu-west-1.compute.amazonaws.com (79.125.1.96)  0.684 ms ec2-79-125-1-100.eu-west-1.compute.amazonaws.com (79.125.1.100)  0.780 ms
 6  178.236.0.138 (178.236.0.138)  1.397 ms  1.122 ms  1.348 ms
 7  178.236.0.121 (178.236.0.121)  1.388 ms 178.236.0.119 (178.236.0.119)  10.688 ms 178.236.0.121 (178.236.0.121)  1.583 ms
 8  213.242.106.85 (213.242.106.85)  1.686 ms 178.236.0.121 (178.236.0.121)  1.532 ms 213.242.106.85 (213.242.106.85)  1.636 ms
 9  213.242.106.85 (213.242.106.85)  1.619 ms ae-1-4.bar1.Dublin3.Level3.net (4.69.153.238)  2.321 ms ae-12-12.ebr2.Paris1.Level3.net (4.69.153.242)  118.822 ms
10  ae-12-12.ebr2.Paris1.Level3.net (4.69.153.242)  118.799 ms ae-1-4.bar1.Dublin3.Level3.net (4.69.153.238)  2.238 ms ae-42-42.ebr2.Washington1.Level3.net (4.69.137.54)  117.943 ms
11  ae-12-12.ebr2.Paris1.Level3.net (4.69.153.242)  118.726 ms  118.886 ms ae-5-5.ebr2.Washington12.Level3.net (4.69.143.222)  117.384 ms
12  ae-5-5.ebr2.Washington12.Level3.net (4.69.143.222)  117.372 ms  117.537 ms ae-6-6.ebr2.Chicago2.Level3.net (4.69.148.146)  117.145 ms
13  ae-6-6.ebr2.Chicago2.Level3.net (4.69.148.146)  119.911 ms  118.903 ms  118.618 ms
14  ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113)  117.702 ms  117.191 ms ae-6-6.ebr2.Chicago2.Level3.net (4.69.148.146)  118.831 ms
15  ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113)  116.817 ms  116.095 ms 4.69.201.2 (4.69.201.2)  236.039 ms
16  ae-11-11.car1.StLouis1.Level3.net (4.69.132.185)  162.234 ms XIOLINK-LLC.car1.StLouis1.Level3.net (4.53.160.186)  118.854 ms 4.69.201.2 (4.69.201.2)  228.078 ms
17  XIOLINK-LLC.car1.StLouis1.Level3.net (4.53.160.186)  118.945 ms *  117.270 ms
18  XIOLINK-LLC.car1.StLouis1.Level3.net (4.53.160.186)  117.303 ms b1-e21.c1-f00.stl1.rackmy.com (63.246.15.2)  206.049 ms  205.715 ms
19  * b1-e21.c1-f00.stl1.rackmy.com (63.246.15.2)  205.112 ms *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

When I clone the repository from our London based office, we see much more acceptable speeds. The traceroute is as follows:

traceroute bitbucket.org
traceroute: Warning: bitbucket.org has multiple addresses; using 207.223.240.182
traceroute to bitbucket.org (207.223.240.182), 64 hops max, 52 byte packets
 1  ip-192-168-0-1 (192.168.0.1)  0.861 ms  0.736 ms  0.737 ms
 2  ip-192-168-1-254 (192.168.1.254)  2.324 ms  1.030 ms  1.073 ms
 3  * * *
 4  ip-10-17-8-242 (10.17.8.242)  16.015 ms
    o2-uk-8-1-0-grtlontc2.6.16.84.in-addr.arpa (84.16.6.234)  16.913 ms  17.530 ms
 5  ip-10-17-8-241 (10.17.8.241)  22.375 ms
    xe2-0-0-0-grtlontc2.red.telefonica-wholesale.net (84.16.7.193)  21.959 ms
    ip-10-17-8-241 (10.17.8.241)  21.602 ms
 6  xe2-0-0-0-grtlontc2.red.telefonica-wholesale.net (84.16.7.193)  17.832 ms
    xe-3-1-0-0-grtnycpt2.red.telefonica-wholesale.net (94.142.126.65)  89.503 ms
    xe3-2-0-0-grtnycpt2.red.telefonica-wholesale.net (84.16.12.22)  89.205 ms
 7  xe3-2-0-0-grtnycpt2.red.telefonica-wholesale.net (84.16.12.22)  149.796 ms
    xe-3-1-0-0-grtnycpt2.red.telefonica-wholesale.net (94.142.126.65)  90.085 ms *
 8  xe5-2-0-0-grtwaseq5.red.telefonica-wholesale.net (84.16.14.217)  98.278 ms
    xe0-1-0-0-grtwaseq2.red.telefonica-wholesale.net (94.142.122.206)  105.504 ms
    xe5-1-5-0-grtwaseq1.red.telefonica-wholesale.net (94.142.116.201)  108.013 ms
 9  213.140.55.78 (213.140.55.78)  110.536 ms  88.676 ms
    xe4-1-1-0-grtwaseq5.red.telefonica-wholesale.net (94.142.119.2)  108.548 ms
10  213.140.55.78 (213.140.55.78)  100.454 ms
    207.88.14.162.ptr.us.xo.net (207.88.14.162)  121.391 ms
    213.140.55.2 (213.140.55.2)  94.979 ms
11  vb6.rar3.chicago-il.us.xo.net (207.88.12.33)  129.309 ms  120.043 ms
    207.88.14.162.ptr.us.xo.net (207.88.14.162)  116.345 ms
12  vb6.rar3.chicago-il.us.xo.net (207.88.12.33)  122.240 ms
    ae0d0.mcr1.marylandheights-mo.us.xo.net (216.156.0.178)  115.553 ms  123.828 ms
13  ae0d0.mcr1.marylandheights-mo.us.xo.net (216.156.0.178)  130.351 ms  124.275 ms
    206.181.23.22 (206.181.23.22)  120.053 ms
14  63-246-15-30.xiolink.com (63.246.15.30)  119.776 ms
    206.181.23.22 (206.181.23.22)  124.865 ms  124.670 ms
15  * * 63-246-15-30.xiolink.com (63.246.15.30)  118.596 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *

If you are accelerating BitBucket via a CDN for AWS sites, are there any known performance issues or bottlenecks you're aware of?

Regards,

Adam Horwich

Comments (3)

  1. Log in to comment