Issues

Issue #8263 resolved

http 400: bad request error when pulling from mercurial repo

Nathan Koenig
created an issue

The repository in question is http://bitbucket.org/osrf/gazebo

I can hg clone, diff, out, but I can't hg pull. Every time I get a HTTP 400 Error.

If I switch to ssh instead of https, the problem goes away.

Comments (30)

  1. Brian Nguyen

    Hi Nathan,

    I tried to reproduce this myself and haven't been able to. I suspect that the issue it specific to your local repository. Can you try running:

    hg pull --debug -v
    

    This may give us more useful information. Also do you get the same issue if you clone a separate copy of the repository?

    Cheers, Brian

  2. Nathan Koenig reporter

    Here is the output of hg pull --debug -v

    using https://bitbucket.org/osrf/gazebo
    http auth: user nkoenig, password not set
    sending capabilities command
    bitbucket.org certificate successfully verified
    pulling from https://nkoenig@bitbucket.org/osrf/gazebo
    query 1; heads
    sending batch command
    searching for changes
    all local heads known remotely
    sending getbundle command
    abort: HTTP Error 400: Bad request
    
  3. Brian Nguyen

    Hi Nathan,

    I've been searching through our access logs and unfortunately I have been unable to find the request that is returning a 400 http code. We've also checked the repository on our servers and haven't found anything untoward.

    Could you try this again and tell us exactly what time you sent the request? It may help us find the error.

    Also since some users can pull and others can't, we suspect that the error is something in your local repository. Could you try running hg verify on your local clone of the repository?

    Cheers, Brian

  4. Aris Synodinos

    Nathan I have been able to overcome the problem by doing a hg rollback and then pulling again. It seems that something specific in the commit we were on was causing the http 400 error.

  5. Nathan Koenig reporter

    This problem occurs for my co-workers as well. I've tried the rollback trick, and it seems to re-download the whole repository.

    hg verify returns

    checking changesets
    checking manifests
    crosschecking files in changesets and manifests
    checking files
    8820 files, 10593 changesets, 55260 total revision
    

    The only solution that I've found is switching to ssh. Once I switch to ssh, and do an hg up, I can switch back to https.

  6. Jon Mooring staff

    Hi Nathan,

    Sorry for the delay in responding to this request. It got lost in my queue. I have a few questions for you that will help us debug this issue further:

    1. What version of Mercurial are you running?
    2. Which Mercurial extensions are you using?
    3. How many topological heads does this repository have? (hg heads -t)

    Thanks for your patience while we work through this.

    Jon

  7. Steven Peters

    I work with Nathan Koenig and see the same errors, so I'll share my answers to these questions:

    1. Mercurial 2.2.2

    2. I use extdiff to view diffs in a gui program (meld). I'm not sure if that's what you're asking.

    3. 806 topological heads

    $ hg heads -t | grep '^changeset:' | wc -l
    806
    
  8. Jon Mooring staff

    Steven Peters thanks for the quick response. Would it be possible for you to submit a tarball of your local copy of the repository in question to support@bitbucket.org? We'd like to run some tests on our end to see if we can't find the source of the issue.

    Thanks,
    Jon

  9. Steven Peters

    It's 350 MB; it'll take me a few minutes to upload it somewhere. In the meantime, here's the top of the log if that helps:

    $ hg log | head -22
    changeset:   10926:151d3a2ca334
    branch:      fix_header_style
    tag:         tip
    parent:      10906:3a2106c34462
    user:        "Nate Koenig <natekoenig@gmail.com>"
    date:        Mon Oct 28 08:01:28 2013 -0700
    summary:     Fix header style
    
    changeset:   10925:e8582915ea2f
    branch:      gzmodify
    parent:      10453:c4379e8ed34a
    parent:      10906:3a2106c34462
    user:        "Nate Koenig <natekoenig@gmail.com>"
    date:        Mon Oct 28 07:57:08 2013 -0700
    summary:     Merged from default
    
    changeset:   10924:9da75255d291
    branch:      dem_support
    user:        Carlos Agüero <caguero@osrfoundation.org>
    date:        Mon Oct 28 00:29:53 2013 -0700
    summary:     Minor style fix
    
  10. Jon Mooring staff

    Hi Steven Peters, Nathan Koenig,

    We've done some additional research on this issue and it does indeed seem to be a problem with the number of topological heads causing the maximum header data length to be exceeded. While we certainly could increase the maximum length, this is not a viable solution because as the number of heads grows, you'll likely just hit the limit again. Here is a bug report against Mercurial about the issue.

    My first suggestion would be to use SSH, which does not suffer from this issue. We provide SSH access both via port 22 and port 443. My next, maybe less appealing suggestion would be to merge all of the heads back into default as you wrap up work on named branches. This would help prevent the number of topological heads from becoming so bloated over time.

    I hope this is helpful for you. Please let me know if you have any other questions.

    Thanks,
    Jon

  11. Steven Peters

    Can you elaborate on the second suggestion? Please correct me if I am wrong; after reading the hg bug report, it sounds like closing a branch after merging is not the right way to do things. But this is how bitbucket pull requests work! You merge a branch and then check the box to close that branch after merging. Am I missing something?

  12. Jon Mooring staff

    Sorry for the slow response, this issue fell out of my queue after I prematurely resolved it. You're absolutely right that pull requests close the branch after merging, which is incorrect (or at least ill-advised).

    Thanks for contributing to #5665. I've raised that issue with the team and we'll re-evaluate our process around merging Hg pull requests.

  13. Bernardo Costa

    I have this same issue using any mercurial version beyond 1.9. If I use version 1.8.4 I don't have this issue. The strange thing is that it seems to affect just a copy of this repo because even using any 2.x version, I can still clone and after it do a normal pull. But this up until somebody pushes into the remote server. After it, the issue appears again.

    with 2.x version

    $ hg pull --debug -v using https://myserver:8443/scm/hg/myrepo/ sending capabilities command myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd http authorization required realm: myrealm user: myuser password: http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd pulling from https://myserver:8443/scm/hg/myrepo/ preparing listkeys for "bookmarks" sending listkeys command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd query 1; heads sending batch command http auth: user myuser, password ** myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd searching for changes taking initial sample searching: 2 queries query 2; still undecided: 1948, sample size is: 200 sending known command abort: HTTP Error 400: Bad Request

    up to 1.8.4 version

    $ hg pull --debug -v using https://myserver:8443/scm/hg/myrepo/ sending between command myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd http authorization required realm: petrobras.biz user: myuser password: http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd pulling from https://myserver:8443/scm/hg/myrepo/ sending heads command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd searching for changes sending branches command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd examining ec36b4ae971b:192362ffb8e3 found incomplete branch ec36b4ae971b:192362ffb8e3 searching: 1 queries sending between command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd narrowing 1:5 95b7eca9eda6 narrowing 2:5 8ff79356bb50 narrowing 4:5 4a60c31f3572 narrowed branch search to 8ff79356bb50:4a60c31f3572 searching: 2 queries sending between command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd narrowing 1:2 17f1b19b3106 found new branch changeset 8ff79356bb50 found new changesets starting at 8ff79356bb50 2 total queries sending capabilities command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd capabilities: stream changegroupsubset unbundlehash batch httpheader=1024 lookup pushkey known unbundle=HG10GZ,HG10BZ,HG10UN branchmap getbundle sending changegroupsubset command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd adding changesets changesets: 1 chunks add changeset 8ff79356bb50 changesets: 2 chunks add changeset 95b7eca9eda6 changesets: 3 chunks add changeset ec36b4ae971b adding manifests manifests: 1/3 chunks (33.33%) manifests: 2/3 chunks (66.67%) manifests: 3/3 chunks (100.00%) adding file changes adding grails-app/conf/UrlMappings.groovy revisions files: 1/5 chunks (20.00%) adding grails-app/controllers/base/TabelaPocosController.groovy revisions files: 2/5 chunks (40.00%) adding grails-app/controllers/pi/DadosPIController.groovy revisions files: 3/5 chunks (60.00%) adding test/unit/base/TabelaPocosControllerTests.groovy revisions files: 4/5 chunks (80.00%) adding test/unit/pi/DadosPIControllerTests.groovy revisions files: 5/5 chunks (100.00%) added 3 changesets with 5 changes to 5 files (+1 heads) updating the branch cache checking for updated bookmarks sending listkeys command http auth: user myuser, password * myserver certificate matched fingerprint 2f:d6:10:6f:c7:1c:bb:06:a2:7d:94:98:e0:92:76:24:28:ba:08:bd (run 'hg heads' to see heads, 'hg merge' to merge)

  14. Bernardo Costa

    jonmorring, can this ticked be reopened ? I have a repo with just few heads (six) and still get this problem. No proxy configuration and mercurial client is tortoisehg version 2.4.2. If I import this copy to linux and run hg pull with version 1.8.4 I don't get the problem.

  15. npetchimuthu

    Jon Mooring

    I have the same issue.

    Mercurial version : 2.7 topological heads : 3237

    FYI: I can't able to merge these heads into default. Because each head is used for different set of peoples.

    But the same repository is working fine when using SSH mode.

    Thanks, Muthu.

  16. Steven Peters

    npetchimuthu You don't have to merge all the heads into default. Presumably some of these topological heads are closed branches that were already merged into other branches. If so, then you can create a new branch called dangling_heads and merge all the closed heads to that branch.

    I've done this for one of our repositories, so I can advise on the specifics. To get a list of closed topological heads, you can use the following:

    for b in `hg heads -tc | grep '^branch:' | sed -e 's@^branch: *@@'`
    do
        hg log -r "branch($b) and closed()"
    done
    

    We had a lot of these.

    UPDATE: this script is extremely slow.

  17. Steven Peters

    I just did another round of fixing this for osrf/gazebo. I used the following script to identify closed topological heads, and it's faster than the script I posted before:

    hg log -r "heads(0:tip) and closed()"
    

    Note that "closed topological heads" are closed branches that haven't been merged into anything. On osrf/gazebo, I made a branch called dangling_heads and merged all the closed topological heads into that branch, which helps with this mercurial issue.

  18. Steven Peters

    This is recurring again on osrf/gazebo even though we have less than half as many heads as when this first started happening (about 355 now, compared to over 800 before).

  19. Log in to comment