Error while push large changes.

Issue #800 resolved
Xpol Wan created an issue

I'm running rhodecode 1.5.2 and via apache2 proxy server.

When i try to push large changes to rhodecode(250M bundle with changes to 10000+ file)

The tortoisehg faild with 502 Bad Gateway.

The apache2 log says:

[Wed Mar 20 17:28:52 2013] [error] (103)Software caused connection abort: proxy: pass request body failed to 127.0.0.1:5000 (127.0.0.1)
[Wed Mar 20 17:28:52 2013] [error] proxy: pass request body failed to 127.0.0.1:5000 (127.0.0.1) from x.x.x.84 ()

The paster log does not reoprt error.

BTW: I googled find the talk saying bypass the apache, to see what happen. But how to enable paster server access by the LAN clients?

Comments (12)

  1. Marcin Kuzminski repo owner

    Put ip 0.0.0.0 in ini file then just access the paster via ip and port. By default it's 5000

  2. Xpol Wan reporter

    I update the system packages, and update rhodecode to 1.5.4. And pushed to rhodecode without apache2. The issue still exists.

    Hg says:

    URLError: [Errno 10053] 
    

    And the paster log have nothing about the error.

    Any ideas?

    PS.

    The only error message, which is not cause by the push and even not form the same clinet ip nor same user account. It caused by web access. it says:

    Exception AttributeError: AttributeError("'_DummyThread' object has no attribute '_Thread__block'",) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
    
  3. Xpol Wan reporter

    The package info:

    Python - 2.7.3
    
    System - Linux-3.2.0-38-generic-x86_64-with-Ubuntu-12.04-precise
    
    amqplib 1.0.2
    anyjson 0.3.3
    apt-xapian-index    0.44
    argparse    1.2.1
    Babel   0.9.6
    Beaker  1.6.4
    celery  2.2.10
    chardet 2.0.1
    command-not-found   0.2.44
    decorator   3.4.0
    distribute  0.6.34
    docutils    0.8.1
    dulwich 0.8.7
    FormEncode  1.2.4
    git 1.7.9
    GnuPGInterface  0.3.2
    httplib2    0.7.2
    keyring 0.9.2
    kombu   1.5.1
    Landscape-Client    12.05
    language-selector   0.1
    launchpadlib    1.9.12
    lazr.restfulclient  0.12.0
    lazr.uri    1.0.3
    Mako    0.7.3
    Markdown    2.2.1
    MarkupSafe  0.15
    mercurial   2.4.2
    mock    1.0.1
    MySQL-python    1.2.4
    nose    1.2.1
    oauth   1.0.1
    PAM 0.4.2
    Paste   1.7.5.1
    PasteDeploy 1.5.0
    paster  0.8
    PasteScript 1.7.5
    pip 1.2.1
    py-bcrypt   0.2
    pycrypto    2.4.1
    pycurl  7.19.0
    Pygments    1.6
    Pylons  1.0
    pyOpenSSL   0.12
    pyparsing   1.5.6
    pyserial    2.5
    Python  2.7
    python-apt  0.8.3ubuntu7
    python-dateutil 1.5
    python-debian   0.1.21ubuntu1
    python-ldap 2.4.10
    repoze.lru  0.6
    RhodeCode   1.5.4
    Routes  1.13
    setuptools  0.6c11
    simplejson  2.5.2
    SQLAlchemy  0.7.9
    Tempita 0.5.1
    Twisted-Core    11.1.0
    ufw 0.31.1-1
    unattended-upgrades 0.1
    virtualenv  1.8.4
    wadllib 1.3.0
    waitress    0.8.1
    WebError    0.10.3
    WebHelpers  1.3
    WebOb   1.0.8
    WebTest 1.4.3
    Whoosh  2.4.1
    wsgiref 0.1.2
    zope.interface  3.6.1
    
  4. Marcin Kuzminski repo owner

    Maybe you need to increasy tje body size on waitress server in the ini file.

    I think it's max_body_size attribute you need to put next to the ip config

  5. Xpol Wan reporter

    After add max_body_size like:

    #WAITRESS
    threads = 5
    use = egg:waitress#main
    
    host = 0.0.0.0
    port = 5000
    max_body_size=1073741824
    

    I got :

    Traceback (most recent call last):
      File "/usr/local/bin/paster", line 9, in <module>
        load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
      File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 104, in run
        invoke(command, command_name, options, args[1:])
      File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 143, in invoke
        exit_code = runner.run(args)
      File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 238, in run
        result = self.command()
      File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 311, in command
        serve()
      File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 295, in serve
        server(app)
      File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 189, in server_wrapper
        **context.local_conf)
      File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 56, in fix_call
        val = callable(*args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/waitress/__init__.py", line 21, in serve_paste
        serve(app, **kw)
      File "/usr/local/lib/python2.7/dist-packages/waitress/__init__.py", line 11, in serve
        server = _server(app, **kw)
      File "/usr/local/lib/python2.7/dist-packages/waitress/server.py", line 47, in __init__
        self.adj = Adjustments(**kw)
      File "/usr/local/lib/python2.7/dist-packages/waitress/adjustments.py", line 136, in __init__
        raise ValueError('Unknown adjustment %r' % k)
    ValueError: Unknown adjustment 'max_body_size'
    
  6. Ton Plomp

    I had an issue where the push made my server run out of memory. Pushing in parts was possible though.

  7. Xpol Wan reporter

    After add max_request_body_size access without apache the HG error code become:

    URLError: [Errno 10054]
    

    The paster log still have nothing logged.

  8. Marcin Kuzminski repo owner

    Did you try to run the rhodecode server without daemonizing it and checking live stdout output, and IF maybe it doesn't die during the large push ? Also i had similar issues posted by someone in which it turned out some corporate proxy was killing the connection

  9. Artien

    Setting the max_request_body_size = 2073741824 for waitress in the production.ini solved this issue for me.

    My total repo size appeared to be 1.4 GB thus the default limit was the issue.

  10. Log in to comment