Cloning large git repository fails in Kallithea with MemoryError

Issue #301 closed
Geoff Bache created an issue

I just converted a ~2GB repository to Git and uploaded it to Kallithea. It turns out it cannot be cloned via Kallithea. Cloning it outside Kallithea works fine. Any attempts write the following to the Kallithea log:

2017-12-06 14:17:23.789 ERROR [waitress] Exception when serving /Documentation/doc/git-upload-pack
Traceback (most recent call last):
  File "c:\kallithea\env\lib\site-packages\waitress\channel.py", line 337, in service
    task.service()
  File "c:\kallithea\env\lib\site-packages\waitress\task.py", line 173, in service
    self.execute()
  File "c:\kallithea\env\lib\site-packages\waitress\task.py", line 392, in execute
    app_iter = self.channel.server.application(env, start_response)
  File "c:\python27\lib\site-packages\paste\gzipper.py", line 38, in __call__
    return response.write()
  File "c:\python27\lib\site-packages\paste\gzipper.py", line 68, in write
    out.seek(0)
  File "c:\python27\lib\StringIO.py", line 106, in seek
    self.buf += ''.join(self.buflist)
MemoryError

The machine isn't short of physical memory. We can see that the memory used by the Kallithea process increases by about 1GB before it throws this exception.

Comments (9)

  1. Ron Winacott

    I have not seen this type of problem since the pre-kallithea fork days, but I did see this when the /tmp or /temp (depending on OS of the server running Kallithea was full or not large enough of the large repository. Once the cloning was complete, or the error was thrown, the space in /tmp was freed again.

  2. Geoff Bache reporter

    Server is running Windows Server 2012. My temp (C: disk) has 8GB free and the repository is slightly less than 2GB.

  3. Mads Kiilerich

    I guess that is caused by Kallithea entering a mode where it will zip all content before serving it ... and thus buffers it all up. That has been fixed on the development branch.

    If using a "real" web server that can serve static content, try using

    [app:main]
    static_files = true
    

    It seems like you are using waitress from paster serve. In that case, perhaps try editing kallithea/config/middleware.py and delete the app = make_gzip_middleware(app, global_conf, compress_level=1) line.

  4. Mads Kiilerich

    (It would still be interesting to get confirmation if it works with 32-bit on the "default" (development) branch ...)

  5. Log in to comment