Bad File Descriptor

Issue #229 resolved
codingtony created an issue

Happens from time to time and it shuts down kallithea

Using kallithea 0.3.2 on Ubuntu 16.04 LTS

2016-07-07 02:27:01.500 INFO  [kallithea.RequestWrapper] IP: 192.168.20.1 Request to /XXXX/YYYYYYY/ time: 0.213s
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 9, in <module>
    load_entry_point('PasteScript', 'console_scripts', 'paster')()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/command.py", line 236, in run
    result = self.command()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 319, in command
    serve()
  File "/Kallithea-0.3.2/.eggs/PasteScript-2.0.2-py2.7.egg/paste/script/serve.py", line 303, in serve
    server(app)
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/loadwsgi.py", line 189, in server_wrapper
    **context.local_conf)
  File "/Kallithea-0.3.2/.eggs/PasteDeploy-1.5.2-py2.7.egg/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 21, in serve_paste
    serve(app, **kw)
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/__init__.py", line 18, in serve
    server.run()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.8-py2.7.egg/waitress/server.py", line 154, in run
    use_poll=self.adj.asyncore_use_poll,
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 145, in poll
    r, w, e = select.select(r, w, e, timeout)
select.error: (9, 'Bad file descriptor')

Comments (28)

  1. almkuzmin

    Dear Thomas,

    Upgrading to waitress 0.9.0 didn't help.

    2016-08-19 10:16:03.816 INFO  [kallithea.lib.middleware.simplegit] push action on Git repo "rancid/net-cvs/xxx" by "rancid" from 10.x.x.x
    2016-08-19 10:16:06.512 INFO  [kallithea.RequestWrapper] IP: 10.x.x.x Request to /rancid/net-cvs/xxx/git-receive-pack time: 2.989s
    2016-08-19 10:16:06.685 ERROR [waitress] Exception when servicing <waitress.channel.HTTPChannel connected 127.0.0.1:40082 at 0x7f9fe5ab1790>
    Traceback (most recent call last):
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/task.py", line 78, in handler_thread
        task.service()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/channel.py", line 369, in service
        request.close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/parser.py", line 249, in close
        body_rcv.getbuf().close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 298, in close
        buf.close()
      File "/opt/data/kallithea/lib/python2.7/site-packages/waitress/buffers.py", line 108, in close
        self.file.close()
    IOError: [Errno 9] Bad file descriptor
    
  2. Mads Kiilerich

    There is no Kallithea code in the stack trace. It could seem like you are hitting a waitress bug.

  3. almkuzmin

    As far as I can see this happens on huge updates. And it's easy reproducible. What I'm afraid most of is that repo could get broken by this unsuccessful update.

  4. Mads Kiilerich

    I hope there will. I just doubt it is something that can be fixed in Kallithea - not because we won't but because that isn't where the problem is. But it is possible that some Kallithea user can reproduce the problem and track down what is going on and get it fixed.

    @andrew_shadura - you are using git and paster serve ... with waitress? Have you seen this problem?

    I guess the simplest workaround is to just not use waitress - see http://kallithea.readthedocs.io/en/latest/overview.html#web-server for the big picture.

  5. Mads Kiilerich

    Clarification: it is not really "Kallithea shuts down". It is the web server you use for serving Kallithea that crashes.

  6. Andrej Shadura Account Deactivated

    @kiilerix, I’m not exactly using Kallithea with Git, I’m mostly working on improving Git support but I’m not using it in production unfortunately, so my observations won’t be enough to see that if it’s not always reproducible.

  7. Andrej Shadura Account Deactivated

    Meanwhile, I think it would be good to ditch our own custom Git SmartHTTP protocol implementation and just use one provided by Dulwich.

  8. Mads Kiilerich

    Fine with me. It would be weird if it solved this problem ... but who knows it it would realign the stars ...

  9. almkuzmin

    @domruf I started creating 300 files 50 kb long each. Then pushed them with no problem.

      for i in `seq 1 300`; do dd if=/dev/urandom of=file.$i bs=50000 count=1; done
      git add .
      git commit -am test
      git push
    

    Next step was increasing file size up to 500Kb, and it made web server crash.

  10. almkuzmin

    @kiilerix I'm not familiar with running wsgi applications. Can you please provide start script for use with nginx/apache?

  11. codingtony reporter

    @almkuzmin I've encountered the bug not always when there are big pushes. However when I restart Kallithea, I re-push and it goes through without error. We have git and hg repos. We have more hg repo than git ones and the problem can occur on both (hg or git) pushes.

  12. domruf

    @almkuzmin does this happen every time on your machine? I just tried it several times (300 files @ 500kb each) and it only happend once. I even tried it with 1MB for each file.

  13. almkuzmin

    Well. Just an observation. While receiving commit, Kallithea (waitress?) uses /tmp, which is mounted as tmpfs in my case. At some point some file(s) apparently get removed while they are still opened by the server process. I can see no disk usage, but there's a lot of occupied space (shown by df) that is released after service restart.

  14. almkuzmin

    I luckily managed to find a document on setting up WSGI. Now I'm in front of a new obstacle: the server returns 411 on my push requests (error: RPC failed; HTTP 411 curl 22 The requested URL returned error: 411 Length Required). As far as I understand I have to open new issue.

  15. Mads Kiilerich

    Please consider contributing a documentation patch ( http://kallithea.readthedocs.io/en/latest/contributing.html ) if you can find a place where local git client configuration could fit in the Kallithea documentation. Or perhaps first discuss what you could have in mind. I guess the git documentation would be a better place.

    Anyway, it still sounds like a waitress bug if a client interrupting a POST (before Kallithea gets invoked) can make it crash.

  16. almkuzmin

    @kiilerix I think it perfectly fits into "Troubleshooting" section. Q: Git push fails with HTTP 411 code (Length Required)? A: Set variable http.postBuffer to a reasonably high value (e.g. 500M): git config --global http.postBuffer 524288000

  17. Mads Kiilerich

    Right. I totally agree.

    But that will only cover a part of the problem - not the waitress crashing?

  18. Log in to comment