Cannot clone repository with Kallithea in Python 3 with Gunicorn

Issue #359 resolved
vyom created an issue

After moving to new Kallithea version with Python 3, unable to clone repository, it gives worker TIMEOUT error and operations are rolled back, the same clone works with Python 2 and older version. In logs there is no error.

http authorization required for https://hg.xxxxxxxx.com/test/test-repository
realm: Kallithea
user: kallithea
password:
files [====>                                                                                        ]  358/6509 7m18sdestination directory: test-repository
requesting all changes
adding changesets
adding manifests
adding file changes
transaction abort!
rollback completed
abort: HTTP request error (incomplete response)
(this may be an intermittent network failure; if the error persists, consider contacting the network or server operator)

The Gunicorn service log is:

Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Starting gunicorn 20.0.4
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Listening at: http://0.0.0.0:5000 (7755)
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7755] [INFO] Using worker: sync
Feb 13 22:37:23 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:37:23 +0800] [7758] [INFO] Booting worker with pid: 7758
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [7755] [CRITICAL] WORKER TIMEOUT (pid:7758)
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [7758] [INFO] Worker exiting (pid: 7758)
Feb 13 22:42:05 kallithea-5-2 gunicorn[7755]: [2020-02-13 22:42:05 +0800] [8876] [INFO] Booting worker with pid: 8876

Comments (7)

  1. vyom reporter

    Resolved the issue by changing Gunicorn settings. Gunicorn version 20 has many breaking changes and one of them is it does not pick up configuration from paste ini.

    So after changing few more settings in Gunicorn it works now.

    [Unit]
    Description=kallithea-scm
    After=network.target
    
    [Service]
    PIDFile=/run/gunicorn/pid
    User=www-data
    Group=www-data
    RuntimeDirectory=gunicorn
    WorkingDirectory=/home/kallithea/projects/kallithea
    Environment=VIRTUAL_ENV=/home/kallithea/projects/kallithea/venv3
    Environment=PATH=$VIRTUAL_ENV/bin:$PATH
    ExecStart=/home/kallithea/projects/kallithea/venv3/bin/gunicorn --pid /run/gunicorn/pid --paste new.ini -b 0.0.0.0:5000 -t
    600 --threads 3
    ExecReload=/bin/kill -s HUP $MAINPID
    ExecStop=/bin/kill -s TERM $MAINPID
    PrivateTmp=true
    
    [Install]
    WantedBy=multi-user.target
    

  2. Mads Kiilerich

    Thanks for the thorough testing.

    What gunicorn version were you using before, and which setting made the crucial difference?

    Is it something we should change in the documentation or kallithea-cli config-create new.ini http_server=gunicorn ?

  3. vyom reporter

    I was using version 19 which worked with Python 2. In version 20 they had breaking changes and one of them was paste deploy. So all the settings related to gunicorn in new.ini like host, port, threads did not work and need specify them in my systemd file in command line.

    [server:main]
    ## WAITRESS ##
    use = egg:gunicorn#main
    ## number of worker threads
    threads = 2
    ## MAX BODY SIZE 100GB
    max_request_body_size = 107374182400
    ## use poll instead of select, fixes fd limits, may not work on old
    ## windows systems.
    #asyncore_use_poll = True
    
    ## COMMON ##
    host = 0.0.0.0
    port = 5000
    

    Based on gunicorn documentation I think the issue is:

    Breaking Change
        Removed gaiohttp worker
        Drop support for Python 2.x
        Drop support for EOL Python 3.2 and 3.3
        Drop support for Paste Deploy server blocks
    

    So Paste Deploy server blocks are dropped so may be all the settings need to define in separate configuration file or on command line.

  4. Thomas De Schampheleire

    Out of curiosity, for which reasons have you selected gunicorn? Which other alternatives did you consider and why did you not select them?

  5. vyom reporter

    In case of Kallithea I used gunicorn to see if there is a performance increase and remove apache2, also support of gunicorn to load paste deployment ini was straight forward. Didn’t choose uwsgi, cherrypy I need to write my own config files. Before gunicorn I was using kallithea with mod_wsgi.

    Following is the old configuration with name based virtual hosting:

    <VirtualHost *:443>
            ServerAdmin support@test.com
            ServerName kallithea.test.com
            DocumentRoot /home/kallithea/projects/kallithea
            ErrorLog /var/log/apache2/error.kallithea.log
            CustomLog /var/log/apache2/access.kallithea.log combined
        SSLEngine On
        SSLCertificateFile /etc/apache2/ssl/kallithea.test.com.crt
        SSLCertificateKeyFile /etc/apache2/ssl/kallithea.test.com.key
    
        WSGIDaemonProcess kallithea processes=1 threads=4
        WSGIScriptAlias / /home/kallithea/projects/kallithea/dispatch.wsgi
        WSGIPassAuthorization On
        <Directory /home/kallithea/projects/kallithea/dispatch.wsgi>
         Require all granted
        </Directory>
           <Location />
         Require all granted
            </Location>
    
        # Error handlers
        ErrorDocument 500 /errordocs/500.html
    
    </VirtualHost>
    

    This is dispatch.wsgi (now fileConfig has been deprecated in new paste).

    activate_this = '/home/kallithea/projects/kallithea/venv/bin/activate_this.py'
    execfile(activate_this,dict(__file__=activate_this))
    import os
    os.environ['HOME'] = '/home/kallithea/projects/kallithea'
    ini = '/home/kallithea/projects/kallithea/kallithea.ini'
    from paste.script.util.logging_config import fileConfig
    fileConfig(ini)
    from paste.deploy import loadapp
    application = loadapp('config:' + ini)
    

    I did a test on following for my other project with WSGI apps using Greenlets (via Eventlet and Gevent):

    1. uwsgi
    2. cherrypy (now its cheroot)
    3. gunicorn
    4. mod_wsgi

    I use all of them in my different projects. In benchmark tests with WRK benchmark tool and Locust - A modern load testing framework.

    1. Cherrypy was the most reliable one but under heavy load was bit slow. It will timeout requests but will not return a wrong response.
    2. Gunicorn and uwsgi were neck to neck but the error response under heavy load was not very good for both. The only thing I did not like about both of them was under heavy stress they will return wrong responses.
    3. mod_wsgi I use when I am on apache, it’s easier to setup and work with. It’s performance wise was slower than gunicorn and uwsgi (I did that a while ago so might have change by now).

    In our team for developers we use cherrypy (now cheroot) as it’s production quality pure python wsgi server works well across linux, mac and windows. In production use either uwsgi or gunicorn.

  6. Log in to comment