prechangegroup.pre_push hook raised an exception: Missing key 'username' in os.environ {}

Issue #834 resolved
Jakob Malm created an issue

This is almost identical to Adam Dawidziuk's issue #510, but the problem appears at some other point in the code.

With RhodeCode 1.6.0-rc1 (pip install, 2013-05-04) I get the following logs when doing a pull in the repo admin page:

2013-05-06 08:14:39.445 INFO  [rhodecode.lib.auth] user jakob.malm is authenticated and granted access to ReposController:__before__ using RegularAuth
Traceback (most recent call last):
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/rhodecode/lib/utils2.py", line 592, in _extract_extras
    rc_extras = json.loads(env['RC_SCM_DATA'])
  File "/opt/rhodecode/venv/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'RC_SCM_DATA'

pulling from http://sehvasrv060/hg/chameleon/
searching for changes
{'LANG': 'en_US.UTF-8', 'TERM': 'cygwin', 'SHLVL': '0', 'CELERY_LOADER': 'rhodecode.lib.celerypylons.loader.PylonsLoader', 'PWD': '/', 'HOME': '/home/rhodecode', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'GIT_CONFIG_NOGLOBAL': '1'}
error: prechangegroup.pre_push hook raised an exception: Missing key 'username' in os.environ {}
2013-05-06 08:14:39.643 ERROR [rhodecode.model.scm] Traceback (most recent call last):
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/rhodecode/model/scm.py", line 474, in pull_changes
    repo.pull(clone_uri)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/rhodecode/lib/vcs/backends/hg/repository.py", line 496, in pull
    pull(self.baseui, self._repo, url)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/commands.py", line 4743, in pull
    modheads = repo.pull(other, heads=revs, force=opts.get('force'))
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/localrepo.py", line 1699, in pull
    result = self.addchangegroup(cg, 'pull', remote.url())
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/localrepo.py", line 58, in wrapper
    return orig(repo.unfiltered(), *args, **kwargs)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/localrepo.py", line 2246, in addchangegroup
    self.hook('prechangegroup', throw=True, source=srctype, url=url)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/localrepo.py", line 420, in hook
    return hook.hook(self.ui, self, name, throw, **args)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/hook.py", line 185, in hook
    r = _pythonhook(ui, repo, name, hname, hookfn, args, throw) or r
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/mercurial/hook.py", line 79, in _pythonhook
    r = obj(ui=ui, repo=repo, hooktype=name, **args)
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/rhodecode/lib/hooks.py", line 95, in pre_push
    ex = _extract_extras()
  File "/opt/rhodecode/venv/local/lib/python2.7/site-packages/rhodecode/lib/utils2.py", line 603, in _extract_extras
    raise Exception('Missing key %s in os.environ %s' % (e, rc_extras))
Exception: Missing key 'username' in os.environ {}

Comments (26)

  1. Marcin Kuzminski repo owner

    hmm that's interesting, is it hg repo or git ?

    How do you serve RhodeCode via apache ? nginx ?

  2. Jakob Malm reporter

    It's a hg repo.

    Did a quick try with a git repo just now, and didn't have any problems (except an ignored exception: AttributeError("'_DummyThread' object has no attribute '_Thread__block'",)...). There was nothing to update, though...

    Proxying is done via Apache:

    <VirtualHost *:80>
            ServerName swrepos.cd.corp
            ServerAlias swrepos.cd.corp
    
            <Proxy *>
                    Order allow,deny
                    Allow from all
            </Proxy>
    
            #important!
            #Directive to properly generate url (clone url) for pylons
            ProxyPreserveHost On
    
            #rhodecode instance
            ProxyPass / http://127.0.0.1:5000/
            ProxyPassReverse / http://127.0.0.1:5000/
    
            #to enable https use line below
            #SetEnvIf X-Url-Scheme https HTTPS=1
    </VirtualHost>
    
  3. Marcin Kuzminski repo owner

    Can you check if the error happens also when you bypass apache ? Ie. would change the IP in the ini file to 0.0.0.0 and try cloning from the server ip on port 5000

  4. Jakob Malm reporter

    Same error when accessing rhodecode directly on port 5000.

    RC_SCM_DATA doesn't seem to be declared in the environment. Should it be?

  5. Marcin Kuzminski repo owner

    yes it should, generally it works fine since we all use 1.6 or 1.7beta (same code).

    • does it crash for all repos ?
    • is there's something uncommon in your setup (os/python) ?
  6. Jakob Malm reporter

    System Info and Packages

    Python - 2.7.3

    System - Linux-3.5.0-23-generic-x86_64-with-Ubuntu-12.04-precise

    amqplib 1.0.2
    anyjson 0.3.3
    argparse    1.2.1
    Babel   0.9.6
    Beaker  1.6.4
    celery  2.2.10
    decorator   3.4.0
    distribute  0.6.24
    docutils    0.8.1
    dulwich 0.8.7
    FormEncode  1.2.4
    git 1.7.9
    kombu   1.5.1
    Mako    0.7.3
    Markdown    2.2.1
    MarkupSafe  0.15
    mercurial   2.5.4
    mock    1.0.1
    nose    1.3.0
    Paste   1.7.5.1
    PasteDeploy 1.5.0
    PasteScript 1.7.5
    pip 1.1
    psycopg2    2.5
    py-bcrypt   0.3
    Pygments    1.6
    Pylons  1.0
    pyparsing   1.5.7
    Python  2.7
    python-dateutil 1.5
    repoze.lru  0.6
    RhodeCode   1.6.0rc1
    Routes  1.13
    simplejson  2.5.2
    SQLAlchemy  0.7.10
    Tempita 0.5.1
    waitress    0.8.2
    WebError    0.10.3
    WebHelpers  1.3
    WebOb   1.0.8
    WebTest 1.4.3
    Whoosh  2.4.1
    wsgiref 0.1.2
    
  7. Jakob Malm reporter

    It seems to crash for any repo, yes. I just set up a simple repo and served it from my work station with hg serve. Same result.

    Nothing special in the setup. Quite similar to https://gist.github.com/jennings/2866413. I use postgresql, directory structure as follows:

    • /opt/rhodecode/production.ini
    • /opt/rhodecode/venv
    • /opt/rhodecode/repos
  8. Jakob Malm reporter

    CORRECTION: Cloning your https://secure.rhodecode.org/pastebin doesn't give me the error:

    2013-05-06 16:47:06.036 INFO  [rhodecode.lib.base] IP: 192.168.230.114 User: <AuthUser('id:2:jakob.malm|True')> accessed /_admin/repo_pull/pastebin
    2013-05-06 16:47:06.039 INFO  [rhodecode.lib.auth] user jakob.malm is authenticated and granted access to ReposController:__before__ using RegularAuth
    pulling from https://secure.rhodecode.org/pastebin
    searching for changes
    no changes found
    warning: secure.rhodecode.org certificate with fingerprint 59:f3:6a:b7:25:10:12:2c:cb:3e:11:5e:6b:70:82:ed:57:bf:6e:b7 not verified (check hostfingerprints or web.cacerts config setting)
    2013-05-06 16:47:07.066 INFO  [rhodecode.RequestWrapper] IP: 192.168.230.114 Request to /_admin/repo_pull/pastebin time: 1.046s
    2013-05-06 16:47:07.104 INFO  [rhodecode.lib.base] IP: 192.168.230.114 User: <AuthUser('id:2:jakob.malm|True')> accessed /pastebin/settings
    2013-05-06 16:47:07.104 INFO  [rhodecode.lib.auth] user jakob.malm is authenticated and granted access to ReposController:__before__ using RegularAuth
    2013-05-06 16:47:07.258 INFO  [rhodecode.RequestWrapper] IP: 192.168.230.114 Request to /pastebin/settings time: 0.170s
    

    The repos I've cloned previously were served either directly with hg serve or via Apache WSGI and mercurial.hgweb, no SSL.

  9. Jakob Malm reporter

    Ok, another test. Tried pulling again from a local repo, served with hg serve, that had not changed -- no errors.

  10. Marcin Kuzminski repo owner

    So it's only when pulling from the same server on certain repos ? Hmm very odd, i tried to reproduce that and i couldn't.

    Could you try to do it on demo.rhodecode.org ?

  11. Jakob Malm reporter

    Can't reproduce on demo.rhodecode.org. Did the following:

    1. Create a repo (demo.rhodecode.org/pull-test, clone of abc-test)
    2. Pull changes (none) from remote location -- ok
    3. Create a repo pull-test2 as a clone (not 'fork') of pull-test
    4. Edit and commit a file on RhodeCode
    5. Pull changes from remote location -- ok

    I think this should be quite visible also in the pull-test2 repository.

  12. Marcin Kuzminski repo owner

    then we should chec what makes your server special to have this error, where do you host it ?

  13. Jakob Malm reporter

    It's an internal server, behind a firewall. I will most likely not be able to open it up to the outside world...

    Would you like to try to do something interactively, maybe via some instant messaging?

  14. Jakob Malm reporter

    Sure. Would you have time tomorrow? I'm in Sweden (UTC+01:00, I think) and will be at the office 8-17.

  15. Marcin Kuzminski repo owner

    Just jump into there, and ping me, i'm most of the time available. Please stick around if i don't respond :)

  16. Marcin Kuzminski repo owner

    Those env variables weren't set properly. The reason it worked sometimes was kind of side-effect of other actions. This fix was backported to stable branch, and considered fixed now.

  17. Marcin Kuzminski repo owner

    I'm planning 1.6.1 release this week. You can always do

    pip install https://secure.rhodecode.org/rhodecode/archive/default.zip

  18. Egidijus Lukauskas

    Same issue here on v2.2.1 stable.

    Traceback (most recent call last):
      File "/home/dev/rhodecode-venv/local/lib/python2.7/site-packages/rhodecode/lib/utils2.py", line 619, in _extract_extras
        rc_extras = json.loads(env['RC_SCM_DATA'])
      File "/home/dev/rhodecode-venv/lib/python2.7/UserDict.py", line 23, in __getitem__
        raise KeyError(key)
    KeyError: 'RC_SCM_DATA'
    
    error: prechangegroup.pre_push hook raised an exception: Missing key 'username' in os.environ {}
    

    Also using Ubuntu server 12.04 LTS with 3.2.0-55-generic kernel.

    Environment variable has no RC_SCM_DATA and looks like this:

    {'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'PYTHON_EGG_CACHE': '/tmp', 'CELERY_LOADER': 'rhodecode.lib.celerypylons.loader.PylonsLoader', 'PWD': '/', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'GIT_CONFIG_NOGLOBAL': '1'}
    {'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'PYTHON_EGG_CACHE': '/tmp', 'CELERY_LOADER': 'rhodecode.lib.celerypylons.loader.PylonsLoader', 'PWD': '/', 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'GIT_CONFIG_NOGLOBAL': '1'}
    
  19. Marcin Kuzminski repo owner

    When does this error happens, after what operations ? Do you use multi-process environment ?

    We got an error report when using multi-process gunicorn setup and doing remote clones, this is fixed in latest code, and will be a part of release 2.2.2

  20. Egidijus Lukauskas

    Exception was thrown while cloning new repository from remote Mercurial server.

    Good to know. Just updated source to 2.2.2.dev0 and it works. Thank you for a really quick response :)

  21. Log in to comment