AttributeError during process finish with keyring >= 16.0.1

Issue #63 resolved
Matt Harbison created an issue

Originally reported to the hg bug tracker:

https://bz.mercurial-scm.org/show_bug.cgi?id=6043

I wonder if the change in hg 8e89c2bec1f7 unmasked an existing problem. OTOH, it doesn't appear to be a problem with TortoiseHg 4.6.1, which shipped 1.1.8.

Official response

Comments (19)

  1. Marcin Kasperski repo owner

    I do some brief testing (using hg 4.9.1). I indeed get the AttributeError mentioned, but looks like it happens during cleanup, after everything was done:

    $ hg  out -T'{node}\n'   bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    No handlers could be found for logger "keyring.backend"
    searching for changes
    eb32d82badad8e8bff48927aca2f4d98c05b7eaf
    fd7d29049015a10e52a767796e8dd14f50f90aba
    Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method trdescrepo.__del__ of <hgext3rd.evolve.trdescrepo object at 0x7fc184ee2550>> ignored
    

    Not yet sure what this is about, but still, I logged into, compared the state, and only then got this error. So mayhaps people in bug you linked misinterpret this error. IIRC there were serious problems with keyring support on Mac recently, due to keyring packaging changes. I wrote more on this in #61.

  2. Matt Harbison reporter

    Thanks for looking into this. I agree that it happens after the command does what it’s supposed to, so it isn’t breaking any functionality. I’m baffled that the usual base hg stacktrace with dispatch functions aren’t present. So I’m not sure how to debug this either.

    But this doesn’t seem related to the Mac issue. It definitely happens on FreeBSD, and I’ve personally hit it. I switch between Windows, Linux and OS X, but I don’t think it was on a Mac because I still have 4.6.1 installed there. So I suspect it was Linux, which we build from source and the latest pypi packages for keyring.

  3. Marcin Kasperski repo owner

    Cleaner picture after disabling evolve and hg-git:

     hg --config extensions.hgext.git=! --config extensions.evolve=! out bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    No handlers could be found for logger "keyring.backend"
    searching for changes
    no changes found
    Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method httppeer.__del__ of <mercurial.httppeer.httppeer object at 0x7f1abab4cd10>> ignored
    
  4. Marcin Kasperski repo owner

    After deeper look: the error happens when httppeer iterates over self._urlopener.handlers calling close on each of them. There are 12 handlers most work OK, but calling close on mercurial.url.httpshandler triggers the problem (still, this object has close, something deeper is problematic). Compatibility layers introduced for py3 make analysing what happens more difficult.

    mercurial_keyring is indeed responsible somehow, no problem when I disable it (and enter password manually).

  5. Marcin Kasperski repo owner

    It's some strange lifecycle case. I went up to editing mercurial sources a bit and calling the same close methods sequence from hg.py (before return from the command) and there they work properly. But left to global destruction they fail, like if some modules were unloaded, but then reused (via demandimport?). To make things more irritating, under debugger the problem doesn't appear.

    Hard.

  6. Matt Harbison reporter

    I'm having problems reproducing this by checking outgoing to a bitbucket repo. But if you think there's a demand import issue, have you tried setting HGDEMANDIMPORT=disable in the environment?

    I'm also wondering if the way the extension is registered (i.e. not using reposetup(), uisetup(), etc), is having an effect on something here. I'm not sure how far back you want to support versions of hg, but I upstreamed the mercurial.exthelper class in 4.9 to make it easier to register commands and functions in an extension. Maybe you can just grab that and put it in mercurial-extension-utils for older versions.

  7. Marcin Kasperski repo owner

    To reproduce with outgoing, you probably need to use private repo.
    Setting HGDEMANDIMPORT to disable doesn't seem to change anything.

    Regarding lack of reposetup, uisetup etc - I am monkepatching mercurial (sad necessity as there is no way to plug password management), and monkeypatching is usually better to be done ASAP (I don't really remember details as I hacked most of it loong time ago, but IIRC I had some problems with „what happens first”). My „normal” extensions use *setup APIs.

    I tried injecting some cleanups with atexit but no success so far. Mayhaps I will try to bisect hg to make sure which change introduced sth.

  8. Marcin Kasperski repo owner

    Funny thing. It looks like behaviour is dependant on ……… keyring (the library) version.

    I found that with newest (18.0.0) keyring even older mercurials (like 4.5.3 which I daily use without problems) trigger our error.

    Once I downgraded keyring:

      pip install keyring==16.0.0
    

    there is no error, even with mercurial 4.9.1

  9. Marcin Kasperski repo owner

    More exactly, keyring 16.0.1 - no problem, keyring 16.0.2 - problem exists.

    $ pip install keyring==16.0.1
    ……
    
    $ hg --config extensions.hgext.git=! --config extensions.evolve=! out bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    searching for changes
    no changes found
    
    $ pip install keyring==16.0.2
    …
    
    $ hg --config extensions.hgext.git=! --config extensions.evolve=! out bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    searching for changes
    no changes found
    Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method httppeer.__del__ of <mercurial.httppeer.httppeer object at 0x7f13e20f1e90>> ignored
    
  10. Marcin Kasperski repo owner

    No clue but nn current (18.0.0) keyring removing Windows.py helps:

    $ pip install  -U keyring
    …
    Successfully installed keyring-18.0.0
    
    $ hg --config extensions.hgext.git=! --config extensions.evolve=! out bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    searching for changes
    no changes found
    Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method httppeer.__del__ of <mercurial.httppeer.httppeer object at 0x7f62ac315e90>> ignored
    
    $ rm ~/.virtualenvs/hg_dev/lib/python2.7/site-packages/keyring/backends/Windows.py*
    
    
    $ hg --config extensions.hgext.git=! --config extensions.evolve=! out bitbucket
    comparing with https://bitbucket.org/Mekk/cfg-mercurial
    No handlers could be found for logger "keyring.backend"
    searching for changes
    no changes found
    

    I am on Linux ;-)

  11. Marcin Kasperski repo owner

    I wrote some further observations in https://bz.mercurial-scm.org/show_bug.cgi?id=6043
    Summary (for the sake of history):

    I can't find a reasonable way to resolve the problem without touching keyring (the library). Fixing it there is fairly trivial.

    Raised https://github.com/jaraco/keyring/issues/386

    To summarize:

    • since keyring 16.0.1 keyring/backends/Windows.py in case of failed imports (which is always a case on Linux or Mac) sets global variable (keyring.backends.Windows.missing_import) to the object which has - on one of it's attributes - traceback from the moment the exception was thrown. This traceback impacts lifecycles of the objects and modules and somehow triggers the problem we discuss

    • this is the only place in keyring code where such a code is present

    • the problem can be trivially fixed in keyring code, by adding in aforementioned Windows.py line like
      if missing_deps:
      missing_deps = True
      or
      if missing_deps:
      missing_deps.traceback = None
      or (preferably) removing .traceback attribute from keyring.errors.ExceptionInfo object altogether.

    In terms of packaged distributions for non-windows platforms (like TortoiseHg), the issue can be also mitigated by simply omitting keyring/backends/Windows.py file from the package).

  12. Matt Harbison reporter

    Thanks a lot for looking into this and resolving it! I never would have figured this out.

  13. Log in to comment