pypy3.5 5.8.0 memory leak?

Issue #2579 resolved
Dan Stromberg
created an issue

I'm running http://stromberg.dnsalias.org/svn/backshift/trunk/ (that's an SVN URL) on a self-compiled pypy3.5 5.8.0. This results in a very slow run, sluggishness in other applications on the same machine, and eventually the message "Killed" and a terminated pypy3.5.

I suspect this is a memory leak which is leading the Linux OOM Killer to kill the process.

The code runs fine on CPython 2.x, CPython 3.x, Jython and Pypy2. It used to run fine on Pypy3, but does no longer.

It's possible that the new lzma module is where the memory leak is. The new lzma module is also the main reason I want to run the code on Pypy3.5. Backshift uses lzma/xz heavily.

Sure enough, if I hack backshift to use ctypes and liblzma instead of the lzma module, I don't get a memory leak.

Any suggestions?

Thanks!

Comments (7)

  1. Armin Rigo

    Our _lzma module is written in pure Python with CFFI, which should make it debuggable. You could also try running your code on PyPy2 but with the same _lzma module, if that is supported by your code base; if it leaks too, then we know where to look.

    Otherwise, please give a detailled step-by-step decription about how to reproduce the problem: something like

    svn checkout http://stromberg.dnsalias.org/svn/backshift/trunk/
    pip install xyz
    python something.py
    do this or that and see memory usage increase a lot
    
  2. Dan Stromberg reporter

    I pip'd lzmaffi into an old pypy2 5.3.1, renamed it from lzmaffi to lzma using mv, and then checked what kind of compression backshift is using; it said it's using "lzma module". So I ran a backshift backup of my /usr. It ran for a while, and before long it said:

    perc100 /usr/local/backshift-test/bin/backshift: line 8: 29848 Killed "/usr/local/pypy-5.3.1/bin/pypy" "/usr/local/backshift-test"/lib/backshift/backshift "$@"

    So I suppose this means there's a memory leak in the lzmaffi module.

    Thanks!

  3. Log in to comment