IOError: [Errno 24] Too many open files

Issue #32 resolved
Sven R. Kunze created an issue

We are using kallithea via Apache WSGI.

What can we do about it?

mod_wsgi (pid=8239): Exception occurred processing WSGI script '/localhome/kallithea/apache/'.
Traceback (most recent call last):
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/", line 38, in __call__
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/", line 163, in __call__, new_environ, catch_exc_info=True)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/", line 48, in call_wsgi_application
    app_iter = application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/", line 43, in __call__
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/", line 277, in __call__
    return self._handle_request(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/", line 68, in _handle_request
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/", line 277, in __call__
    return self._handle_request(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/", line 73, in _handle_request
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/weberror/", line 156, in __call__
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 155, in __call__
    return self.wrap_app(environ, session_start_response)    
  File "/localhome/kallithea/lib/python2.7/site-packages/routes/", line 131, in __call__
    response =, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/", line 107, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/", line 312, in dispatch
    return controller(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/", line 383, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/controllers/", line 266, in __call__
    return response(environ, self.start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/webob/", line 917, in __call__
    start_response(self.status, headerlist)
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 149, in session_start_response
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 717, in persist
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 423, in save
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 236, in release_write_lock
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 259, in close
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/", line 672, in do_close
    fh = open(self.file, 'wb')
IOError: [Errno 24] Too many open files: '/home/kallithea/data/sessions/container_file/c/cc/cc773b66d9ed479cb92c0920d4e83331.cache'

Comments (21)

  1. Mads Kiilerich

    This looks very much like a configuration issue, not a bug in Kallithea.

    Please reach out to the community on the mailig list - and make sure you include all relevant information about your setup.

  2. domruf

    I had this once. I'm not sure if it was with rhodecode or already with kallithea.

    But the problem was fixed after changing the beaker configuration from memory to db.

  3. Sven R. Kunze reporter

    @domruf Thanks for that idea. I will look into it and come back soon.

    @kiilerix It would be great if new users could recognize this as a configuration error as easily as you can do.

    Btw. fixing it by setting beaker configuration from memory to db does not look like a configuration issue. From my point of view, that would imply that memory should be a disallowed option.

  4. Sven R. Kunze reporter

    @domruf It seems to work. Thank you.

    Indeed. I am not sure why db is not the default. Shall I keep that that issue open until it is the default?

  5. Sven R. Kunze reporter

    For all who use host-based authentication:

    ## db session ##
    beaker.session.type = ext:database
    ## host-based authentication = postgresql://@/kallithea
    beaker.session.table_name = db_session
  6. Mads Kiilerich

    We use the default beaker settings in production and do not see any problem.

    It do however sound like there in some situations is a leak. I think we should try very hard to work around bugs by recommending a special configuration. We should try to find the root cause.

    I guess this also is with git repositories? We only use Mercurial. I don't know if that could explain the difference.

    Could you try to find out out more about what the problem is? Is it leaking a fixed number of FDs per request? Do you also see it if you try with a Mercurial repo? Or if you run a test instance with 'paster serve'?

  7. Sven R. Kunze reporter

    @kiilerix I agree finding the root cause is the better way.

    To answer your questions: Yes, git only; never tried with Mercurial. We used paster only to test if the server starts up properly but not for long-lasting usage.

    And we indeed do many git clones.

  8. Mads Kiilerich

    Please try

    --- a/kallithea/lib/vcs/
    +++ b/kallithea/lib/vcs/
    @@ -386,6 +386,7 @@ class SubprocessIOChunker(object):
             self.process = _p
             self.output = bg_out
             self.error = bg_err
    +        self.inputstream = inputstream
         def __iter__(self):
             return self
    @@ -413,6 +414,10 @@ class SubprocessIOChunker(object):
    +        try:
    +            os.close(self.inputstream)
    +        except:
    +            pass
         def __del__(self):

    (Assuming you are on linux, the fd leak can be seen in the right /proc/$PID/fd/ )

  9. Sven R. Kunze reporter

    Is there a testcase that we can use to make sure the fd is closed? We would not want to jeopardize our production system.

  10. Mads Kiilerich

    You could keep an eye on the number of open fds, both in the initial setup and with my patch.

    I do however not see (and cannot reproduce) that beaker should make a difference.

  11. Sven R. Kunze reporter

    Well, I meant something like: py.test -k test_all_fds_are_closed. I also think that would be a good addition in order to indicate such problems in the future.

    I think I can check this for a short time without causing too much trouble in our systems. I come back to you soon.

  12. Sven R. Kunze reporter

    Alright, it works fine now. It can be merged. Thank you for your time and consideration.

  13. Mads Kiilerich

    No - I will push the fix soon. But I'm sure there will be other opportunities for contributing ;-)

  14. Thomas De Schampheleire

    So the remaining part to be applied to Kallithea would be:

    diff --git a/kallithea/lib/vcs/ b/kallithea/lib/vcs/
    --- a/kallithea/lib/vcs/
    +++ b/kallithea/lib/vcs/
    @@ -57,15 +57,17 @@ class StreamFeeder(threading.Thread):
         def run(self):
             t = self.writeiface
    -        if self.bytes:
    -            os.write(t, self.bytes)
    -        else:
    -            s = self.source
    -            b =
    -            while b:
    -                os.write(t, b)
    +        try:
    +            if self.bytes:
    +                os.write(t, self.bytes)
    +            else:
    +                s = self.source
                     b =
    -        os.close(t)
    +                while b:
    +                    os.write(t, b)
    +                    b =
    +        finally:
    +            os.close(t)
         def output(self):

    Is that correct?

    What is the correct attribution for this change? Did you write it or Uma?

    It seems the upstream at which you opened the issue is not active anymore, last change years ago.

  15. Log in to comment