Issue #229 resolved
codingtony
created an issue

Happens from time to time and it shuts down kallithea

Using kallithea 0.3.2 on Ubuntu 16.04 LTS

2016-07-07 02:27:01.500 INFO  [kallithea.RequestWrapper] IP: 192.168.20.1 Request to /XXXX/YYYYYYY/ time: 0.213s
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 9, in <module>
    load_entry_point('PasteScript', 'console_scripts', 'paster')()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 236, in run
    result = self.command()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 319, in command
    serve()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 303, in serve
    server(app)
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 189, in server_wrapper
    **context.local_conf)
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 21, in serve_paste
    serve(app, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 18, in serve
    server.run()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/server.py", line 154, in run
    use_poll=self.adj.asyncore_use_poll,
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 145, in poll
    r, w, e = select.select(r, w, e, timeout)
select.error: (9, 'Bad file descriptor')

Comments (28)

  1. almkuzmin

    Dear Thomas,

    Upgrading to waitress 0.9.0 didn't help.

    2016-08-19 10:16:03.816 INFO  [kallithea.lib.middleware.simplegit] push action on Git repo "rancid/net-cvs/xxx" by "rancid" from 10.x.x.x
    2016-08-19 10:16:06.512 INFO  [kallithea.RequestWrapper] IP: 10.x.x.x Request to /rancid/net-cvs/xxx/git-receive-pack time: 2.989s
    2016-08-19 10:16:06.685 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected 127.0.0.1:40082 at 0x7f9fe5ab1790>
    Traceback (most recent call last):
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/task.py", line 78, in handler_thread
        task.service()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/channel.py", line 369, in service
        request.close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/parser.py", line 249, in close
        body_rcv.getbuf().close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 298, in close
        buf.close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 108, in close
        self.file.close()
    IOError: [Errno 9] Bad file descriptor
    
  2. almkuzmin

    As far as I can see this happens on huge updates. And it's easy reproducible. What I'm afraid most of is that repo could get broken by this unsuccessful update.

  3. Mads Kiilerich

    I hope there will. I just doubt it is something that can be fixed in Kallithea - not because we won't but because that isn't where the problem is. But it is possible that some Kallithea user can reproduce the problem and track down what is going on and get it fixed.

    @Andrej Shadura - you are using git and paster serve ... with waitress? Have you seen this problem?

    I guess the simplest workaround is to just not use waitress - see http://kallithea.readthedocs.io/en/latest/overview.html#web-server for the big picture.

  4. Andrej Shadura

    @Mads Kiilerich, I’m not exactly using Kallithea with Git, I’m mostly working on improving Git support but I’m not using it in production unfortunately, so my observations won’t be enough to see that if it’s not always reproducible.

  5. almkuzmin

    @domruf I started creating 300 files 50 kb long each. Then pushed them with no problem.

      for i in `seq 1 300`; do dd if=/dev/urandom of=file.$i bs=50000 count=1; done
      git add .
      git commit -am test
      git push
    

    Next step was increasing file size up to 500Kb, and it made web server crash.

  6. codingtony reporter

    @almkuzmin I've encountered the bug not always when there are big pushes. However when I restart Kallithea, I re-push and it goes through without error. We have git and hg repos. We have more hg repo than git ones and the problem can occur on both (hg or git) pushes.

  7. almkuzmin

    Well. Just an observation. While receiving commit, Kallithea (waitress?) uses /tmp, which is mounted as tmpfs in my case. At some point some file(s) apparently get removed while they are still opened by the server process. I can see no disk usage, but there's a lot of occupied space (shown by df) that is released after service restart.

  8. almkuzmin

    I luckily managed to find a document on setting up WSGI. Now I'm in front of a new obstacle: the server returns 411 on my push requests (error: RPC failed; HTTP 411 curl 22 The requested URL returned error: 411 Length Required). As far as I understand I have to open new issue.

  9. almkuzmin

    @Mads Kiilerich I think it perfectly fits into "Troubleshooting" section. Q: Git push fails with HTTP 411 code (Length Required)? A: Set variable http.postBuffer to a reasonably high value (e.g. 500M): git config --global http.postBuffer 524288000

  10. Log in to comment