Possible memory leak on show fille annotation

Issue #652 resolved
Alexey Larikov created an issue

I'm having memory issues with RhodeCode. It runs out of memory, so I have to restart it. I was able to track it down to file annotation.

Steps to reproduce. Get memory monitor (I'm using top) Open some file, and press 'Show annotation'. Watch memory usage. It increases and never released.

With big file, like 200 KiB it eats all 4GBs (Yes, I'm on 32 bit version)

From that moment I'm receiving constant memory errors for almost any operation.

Comments (9)

  1. Alexey Larikov reporter

    If you need more info just let me know.

    Maybe it makes sense to crash app on MemoryError? Anyway it's not usable in this condition.

  2. Marcin Kuzminski repo owner

    Nice info would be the general info about the repository, like disk size, number of changesets, and binary files. I see the crash is on reading the metadata from mercurial. The amount of memory needed to read it is probably fine, the issue is why it's not released.

  3. Alexey Larikov reporter

    Repo size (.hg dir) is 880Mb it has 2000 changesets. Repo was converted from SVN recently. It has about 30000 files with 240 files larger than 500k and 23 files larger 10M. There are binary files among them.

    I've did a test with the same repo on clean stand-alone RC installation and was able to confirm the issue. I've attached one more error log for you

  4. Alexey Larikov reporter

    I was able to trace down and fix this issue. https://bitbucket.org/zinzila/rhodecode-pam-authentication/changeset/b6f98a06d79f78cbe8ff82c58963d17434c3ab72

    The reason was when RC generates annotate it creates a list of tuples for each line of file, and that tuple has MercurialChangeset for each line. For big repo with long history this list becomes huge, eating up all memory. I've introduced get_file_annotate_gen() method returning generator instead of list. Also done some fixes for AnnotateHtmlFormater() class to use this generator.

    There seems to be some issue with lazy properties. If using lazy properties for annotate memory is never freed.

  5. Marcin Kuzminski repo owner

    Awsome ! thanks for your time spent on this issue. As far i understand your fix is just partially as it uses just less memory but it's not released after ? The LazyProperty gives a huge performance boost, but i guess it's more tricky when it comes to GC those. I need to spent some time re-thinking this and playing with the code a little to try optimize that.

  6. Alexey Larikov reporter

    This fixes annotation issue completely, as I'm avoiding using properties for generating annotation. So, no memory wasted.

    I think issue with lazy property could be caching. If RC caches objects and object has a huge property that property would never be GCed, as there is a reference to this object.

  7. Marcin Kuzminski repo owner

    Thanks for your patch. Based on it I switched the annotation fully to generator now for both mercurial and git. I think we can close this one.

  8. Log in to comment