Issue #11 resolved

keyring has stopped working with subrepos

Jeff Schenck
created an issue

With mercurial 1.9.x, the keyring extension no longer works with repos containing subrepos. Normal operation (i.e, a single repo) is fine. When using suprepos, I'm prompted for my password once for each subrepo.

1.8.3 works well. I'm sticking there for now. I'm using the latest version of mercurial keyring (0.4.6), on a Linux (CentOS) box. Thanks for your help.

Comments (23)

  1. Marcin Kasperski repo owner

    Does it happen just once, or repeats on every pull/push? Or, in other words, in case you enter this pasword for subrepo when prompted, are you re-asked for it for the same subrepo later on?

  2. Jeff Schenck reporter

    If I do a pull on a freshly cloned repo, it doesn't prompt me for a password. I see the problem when I initially clone. I don't know about pulling on a repo that needs updating. I will test.

  3. Marcin Kasperski repo owner

    Hmm, it is likely related to the way "base URL" for authentication is decided upon - I suspect mercurial_keyring treats every your clone as separate different URL and saves pasword for everyone separately.

    mercurial and mercurial_keyring rely here on auth prefixes to avoid duplication - for example to avoid being re-asked passwords for my group of repos I tend to have in my /.hgrc sth like

    [auth]
    myserver.prefix = somewhere.com/basepath
    myserver.username = mynickthere
    

    and alias repositories as myserver – but to have alias you need to have the directory. I never really digged into how this behaves on clone so I find it unexpected that it works from scratch on any hg version (do subrepos introduce some standard alias?)

  4. Marcin Kasperski repo owner

    I may try taking a deeper look, but could you help me preparing test scenario? Creating some repository with subrepos here on the bitbucket (or anywhere in the net I could clone from) and sharing the commands I am to issue to reproduce the problem would do…

  5. Jeff Schenck reporter

    I've created two repos on bitbucket for testing, keyring_test and keyring_sub, where keyring_sub is (no surprise) a subrepo of keyring_test and given you write permission. The sub contains one file, foo.txt. When I clone I get the following:

    $ hg clone https://bitbucket.org/jschenck/keyring_test test1
    requesting all changes
    adding changesets
    adding manifests
    adding file changes
    added 2 changesets with 3 changes to 2 files
    updating to branch default
    cloning subrepo keyring_sub from https://bitbucket.org/jschenck/keyring_sub
    requesting all changes
    adding changesets
    adding manifests
    adding file changes
    added 1 changesets with 1 changes to 1 files
    http authorization required
    realm: Bitbucket.org HTTP
    user: jschenck (fixed in .hg/hgrc)
    password: 
    2 files updated, 0 files merged, 0 files removed, 0 files unresolved
    

    (Notice the password prompt.) Every time I clone a new copy, I get prompted. If there were two subrepos I'd get prompted twice. The relevant parts of my .hgrc file are

    [ui]
    username = Jeff Schenck <jeff.schenck@dts.com>
    
    [auth]
    bitbucket.prefix = bitbucket.org/jschenck
    bitbucket.username = jschenck
    bitbucket.schemes = https
    
    [extensions]
    mercurial_keyring =
    

    Thanks again for your help.

  6. Jeff Schenck reporter

    Could this have something to do with the new HTTP client library that was added to mercurial in 1.9? 1.9.1 was when I first noticed this issue.

  7. Marcin Kasperski repo owner

    The problem turns out to be non-trivial and seems triggered by some mercurial tighter integration of subrepos code.

    Some background: mercurial_keyring purpose is to save passwords for future reuse (in keyring database) and cache them for immediate reuse (so one is not prompted a few times during clone/pull/push etc). But not all passwords are good. If the entered password is mispelled, we need to ask the user again, not to resend his bad password ad infinitum.

    Unfortunately at the moment mercurial_keyring plugs into mercurial code, it is not yet known whether the password is good or bad. My code just sneaks into the default prompt for password logic and returns the password to mercurial where it is used.

    I used to assume that if my code is not re-called, then the password is good, and if – within one mercurial run – I am called again for the password to the same username+url, then the password must have been bad, so should be cleared from caches and user should be prompted again. Until recently this logic worked properly (mercurial kept obtained credentials in its http client objects so re-prompted only in case it obtained 401).

    Looks like not anymore – I guess that subrepos handling logic is integrated well enough to have them being cloned/pulled/pushed within the same process as master (= during single mercurial run), but not enough to reuse the same http client. So I am re-prompted for password for the same url (note that url here is "prefix") and my code interprets it as a proof that previous password was wrong.

  8. Jeff Schenck reporter

    Marcin, Thanks for looking into the problem. It sounds as if my guess about the new http client library may have been right. It also sounds like this would be hard to fix on your end. Do you think that this is something that the mercurial people would consider looking at, since the caching behavior of their http client seems to have changed -- or would they say "not a bug, not our problem"? I guess I could find out...

    Fortunately, since it only happens when cloning and not with pushing and pulling (I believe), it is not too painful.

  9. Marcin Kasperski repo owner

    I am to think about the problem and look for the solution, just said that it won't be that easy.

    Regarding Mercurial code: if I read the case correctly, there may be inefficiency on their side – in particular, user cloning subrepo (without mercurial_keyring, using just bare-bones mercurial) will be prompted for his password twice even if it is totally unnecessary (and proper prefix is set in /.hgrc).

  10. Mo Cassidy

    I'm not sure if things have changed within Mercurial, but now on version 2.0.2, if I disable mercurial_keyring, and attempt to clone, if I ever input an incorrect password, whether on master or sub repo, I get "abort: authorization failed", and it immediately puts me back to the command prompt. If I enable mercurial_keyring, it allows me to try an incorrect password six times (again, on master or sub repo) before kicking me back to the command prompt.

    I'm not sure why there's a difference, but if the retry logic is part of your mercurial_keyring, then it looks like plain mercurial won't ever re-ask for password on failed attempt.

    Does this mean you can go back to relying on that logic to determine if you need to automatically resubmit the same password without bothering the user?

  11. Eric Blood

    I recently encountered this issue and while digging into it I think that the real problem has to do with change http://selenic.com/repo/hg/rev/ce99d887585f in Mercurial itself. That change moved Http command parameters from the query string to Http headers. Since that change if Mercurial issues the same command twice in a row but with different parameters, mercurial_keyring thinks there was an authentication failure.

    That change was made last May so I'm not sure why the problem has only really shown up more recently but maybe there have been other changes that have made this situation happen more often.

    I've attempted to make a fix (which seems to work for the test cases I came up with) for the problem: https://bitbucket.org/eblood66/mercurial_keyring/overview

    The change monkeypatches the authentication handlers to get access to the Http request object. It only prompts for a new password if there is a password request for the same realm, url and request object twice in a row. I think this may also provide a fix for issues #10 and #13 and possibly #14.

  12. Eric Blood

    I was using the latest (2.0.2) on Linux. I've also got 2.0.2 on my Windows machine but I haven't tested it there yet.

    That's a good point though. I'm not sure the patch points will work right in older versions. My guess is that they probably will since they are actually Python methods rather than Mercurial specific but I didn't test that. If I can find the time I'll see if I can check that out.

    How far back are you interested in maintaining backwards compatibility?

  13. Marcin Kasperski repo owner

    A the very least I'd like to know which versions I support ;-)

    I am very busy those weeks but I hope to review your patch within few days, looks like good approach.

  14. Eric Blood

    I was able to build and test Mercurial versions 1.8, 1.9, 1.9.3, 2.0 and 2.0.2 and it passed a very basic test for each one. I couldn't build a working copy of Mercurial for versions previous to that. But the Ubuntu Lucid repository had Mercurial 1.4.3 and the patch worked there as well. So I think it should be compatible at least that far back.

    For Mercurial versions 1.8 and 1.4.3 there was no difference in behavior between the current keyring version and my patch. Starting in 1.9 the current version prompts for a password when cloning a subrepo but does not with the patch.

    I also changed the password to make sure it would prompt when there is a true authentication failure. That worked for each tested version as well.

  15. Marcin Kasperski repo owner

    I accepted your patch, Eric, and just released mercurial_keyring 0.5.0 based on it. Thank you very much, your approach seems better and more stable than ideas I considered (related to monkeypatching mercurial deeper).

    Jeff's testcase seems to be handled properly, so I close the issue. Let me know in case there are any problems with the patch.

  16. Log in to comment