1. Software Freedom Conservancy
  2. Kallithea
  3. kallithea
  4. Issues

Issues

Issue #32 resolved

IOError: [Errno 24] Too many open files

Sven R. Kunze
created an issue

We are using kallithea via Apache WSGI.

What can we do about it?

mod_wsgi (pid=8239): Exception occurred processing WSGI script '/localhome/kallithea/apache/wsgi.py'.
Traceback (most recent call last):
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/gzipper.py", line 38, in __call__
    response.gzip_start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/paste/registry.py", line 379, in __call__
    app_iter = self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/middleware.py", line 163, in __call__
    self.app, new_environ, catch_exc_info=True)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/util.py", line 48, in call_wsgi_application
    app_iter = application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/wrapper.py", line 43, in __call__
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/base.py", line 277, in __call__
    return self._handle_request(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/simplegit.py", line 68, in _handle_request
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/base.py", line 277, in __call__
    return self._handle_request(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/middleware/simplehg.py", line 73, in _handle_request
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/weberror/errormiddleware.py", line 156, in __call__
    return self.application(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/middleware.py", line 155, in __call__
    return self.wrap_app(environ, session_start_response)    
  File "/localhome/kallithea/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/wsgiapp.py", line 107, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/wsgiapp.py", line 312, in dispatch
    return controller(environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/kallithea/lib/base.py", line 383, in __call__
    return WSGIController.__call__(self, environ, start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/pylons/controllers/core.py", line 266, in __call__
    return response(environ, self.start_response)
  File "/localhome/kallithea/lib/python2.7/site-packages/webob/response.py", line 917, in __call__
    start_response(self.status, headerlist)
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/middleware.py", line 149, in session_start_response
    session.persist()
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/session.py", line 717, in persist
    self._session().save()
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/session.py", line 423, in save
    self.namespace.release_write_lock()
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/container.py", line 236, in release_write_lock
    self.close(checkcount=True)
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/container.py", line 259, in close
    self.do_close()
  File "/localhome/kallithea/lib/python2.7/site-packages/beaker/container.py", line 672, in do_close
    fh = open(self.file, 'wb')
IOError: [Errno 24] Too many open files: '/home/kallithea/data/sessions/container_file/c/cc/cc773b66d9ed479cb92c0920d4e83331.cache'

Comments (19)

  1. Mads Kiilerich

    This looks very much like a configuration issue, not a bug in Kallithea.

    Please reach out to the community on the mailig list - and make sure you include all relevant information about your setup.

  2. domruf

    I had this once. I'm not sure if it was with rhodecode or already with kallithea.

    But the problem was fixed after changing the beaker configuration from memory to db.

  3. Sven R. Kunze reporter

    domruf Thanks for that idea. I will look into it and come back soon.

    Mads Kiilerich It would be great if new users could recognize this as a configuration error as easily as you can do.

    Btw. fixing it by setting beaker configuration from memory to db does not look like a configuration issue. From my point of view, that would imply that memory should be a disallowed option.

  4. Sven R. Kunze reporter

    For all who use host-based authentication:

    ## db session ##
    beaker.session.type = ext:database
    ## host-based authentication
    beaker.session.sa.url = postgresql://@/kallithea
    beaker.session.table_name = db_session
    
  5. Mads Kiilerich

    We use the default beaker settings in production and do not see any problem.

    It do however sound like there in some situations is a leak. I think we should try very hard to work around bugs by recommending a special configuration. We should try to find the root cause.

    I guess this also is with git repositories? We only use Mercurial. I don't know if that could explain the difference.

    Could you try to find out out more about what the problem is? Is it leaking a fixed number of FDs per request? Do you also see it if you try with a Mercurial repo? Or if you run a test instance with 'paster serve'?

  6. Sven R. Kunze reporter

    Mads Kiilerich I agree finding the root cause is the better way.

    To answer your questions: Yes, git only; never tried with Mercurial. We used paster only to test if the server starts up properly but not for long-lasting usage.

    And we indeed do many git clones.

  7. Mads Kiilerich

    Please try

    --- a/kallithea/lib/vcs/subprocessio.py
    +++ b/kallithea/lib/vcs/subprocessio.py
    @@ -386,6 +386,7 @@ class SubprocessIOChunker(object):
             self.process = _p
             self.output = bg_out
             self.error = bg_err
    +        self.inputstream = inputstream
    
         def __iter__(self):
             return self
    @@ -413,6 +414,10 @@ class SubprocessIOChunker(object):
                 self.error.close()
             except:
                 pass
    +        try:
    +            os.close(self.inputstream)
    +        except:
    +            pass
    
         def __del__(self):
             self.close()
    

    (Assuming you are on linux, the fd leak can be seen in the right /proc/$PID/fd/ )

  8. Mads Kiilerich

    You could keep an eye on the number of open fds, both in the initial setup and with my patch.

    I do however not see (and cannot reproduce) that beaker should make a difference.

  9. Sven R. Kunze reporter

    Well, I meant something like: py.test -k test_all_fds_are_closed. I also think that would be a good addition in order to indicate such problems in the future.

    I think I can check this for a short time without causing too much trouble in our systems. I come back to you soon.

  10. Log in to comment