running out of application memory when generating report for Python's stdlib

Issue #253 invalid
Brett Cannon created an issue

While preparing for the upcoming sprints at PyCon CA I was generating an HTML report and as usual it was taking hours to generate the HTML report, so I left it running overnight. But when I checked on it this morning I had a message from OS X reporting that there was no more application memory left and I needed to quit some programs even though I only had Chrome, Sublime Text 2, Terminal, and Finder running.

I was able to generate per-module reports by generating them by letter (e.g. Lib/a*.py), but that doesn't give an HTML overview of coverage. Plus I can't do it for a text-based report either as it gets hung up when reaching 'locale'.

I'm wondering what coverage is keeping around in memory that could be triggering this. Are the reports not tossed after being created?

I have attached the coverage report in case that is in any way helpful.

Comments (4)

  1. Ned Batchelder repo owner
    • changed component to html

    Hmm, I'm not aware of anything being kept, but it could be something simple....

  2. Brett Cannon reporter

    I stared at the code for a while and nothing leapt out at me as being stored anywhere or would somehow suggest why the memory is blowing out. It also doesn't help that if I run the html statement with PyPy (after hacking around PyPy being Python 2 and thus defaulting to ASCII while Python 3 is UTF-8) it finishes in seconds.

    Anyway, I ran it again by printing out the file lines and it worked fine. I'm going to bet that I forgot to run under a released version of Python 3 and that led to some bug (that got fixed) causing the memory loss.

    Regardless, this led to a pull request to stop using locals() as Alex Gaynor says it speeds things up for PyPy.

  3. Log in to comment