test_for_leaks failures with PyPy 1.8 when running test.test_oddball by itself

Issue #186 wontfix
Marc Abramowitz created an issue

This is from an Ubuntu 11 vm:

{{{ marca@ubuntu:~/dev/git-repos/coveragepy$ while true; do PATH=$PATH:~/src/pypy-1.8/bin .tox/pypy_py/bin/nosetests test.test_oddball; sleep 5; done ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1073152


Ran 11 tests in 10.175s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1069056


Ran 11 tests in 9.156s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 8.884s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 532480


Ran 11 tests in 9.005s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 8.638s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 11.813s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 11.329s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 798720


Ran 11 tests in 8.788s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1069056


Ran 11 tests in 12.260s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1073152


Ran 11 tests in 14.271s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1339392


Ran 11 tests in 8.791s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 8.391s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 1073152


Ran 11 tests in 8.842s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 802816


Ran 11 tests in 13.602s

FAILED (failures=1) ....F...... ====================================================================== FAIL: test_for_leaks (test.test_oddball.MemoryLeakTest)


Traceback (most recent call last): File "/home/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 161, in test_for_leaks self.assertTrue(ram_growth < 100000, "RAM grew by %d" % (ram_growth)) AssertionError: False is not true : RAM grew by 798720


Ran 11 tests in 8.870s

FAILED (failures=1) }}}

Comments (4)

  1. Ned Batchelder repo owner

    Hmm, looks like running the file by itself is a bad idea. Testing for memory leaks is kind of a crazy thing for me to attempt anyway. PyPy has a very different memory footprint than other implementations.

    I've added a note of warning in d12daaa05f8c , and am marking this as wontfix.

  2. Marc Abramowitz reporter

    Should test_oddball or test_oddball.ThreadingTest be skipped entirely on PyPy?

    [last: 0] marca@SCML-MarcA:~/dev/git-repos/coveragepy$ tox -e pypy_py
    GLOB sdist-make: /Users/marca/dev/git-repos/coveragepy/setup.py
    pypy_py create: /Users/marca/dev/git-repos/coveragepy/.tox/pypy_py
    pypy_py installdeps: nose
    pypy_py sdist-inst: /Users/marca/dev/git-repos/coveragepy/.tox/dist/coverage-3.5.3b1.zip
    pypy_py runtests: commands[0]
    pypy_py runtests: commands[1]
    ........................................................................................................................................................................................................................................FF.................................................................
    ======================================================================
    FAIL: test_thread_run (test.test_oddball.ThreadingTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/Users/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 50, in test_thread_run
        [1,3,4,5,6,7,9,10,12,13,14], "")
      File "/Users/marca/dev/git-repos/coveragepy/test/coveragetest.py", line 328, in check_coverage
        self.assertEqual(analysis.missing_formatted(), missing)
    AssertionError: '5-7, 10' != ''
    
    ======================================================================
    FAIL: test_threading (test.test_oddball.ThreadingTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/Users/marca/dev/git-repos/coveragepy/test/test_oddball.py", line 31, in test_threading
        [1,3,4,6,7,9,10,12,13,14,15], "10")
      File "/Users/marca/dev/git-repos/coveragepy/test/coveragetest.py", line 328, in check_coverage
        self.assertEqual(analysis.missing_formatted(), missing)
    AssertionError: '7, 10' != '10'
    
    ----------------------------------------------------------------------
    Ran 299 tests in 41.348s
    
    FAILED (failures=2)
    ERROR: InvocationError: '/Users/marca/dev/git-repos/coveragepy/.tox/pypy_py/bin/nosetests -w /Users/marca/dev/git-repos/coveragepy -e test_farm'
    _________________________________________________________________________ summary __________________________________________________________________________
    ERROR:   pypy_py: commands failed
    

    Something along the lines of:

    diff --git a/test/test_oddball.py b/test/test_oddball.py
    index 859648f..2be813e 100644
    --- a/test/test_oddball.py
    +++ b/test/test_oddball.py
    @@ -7,47 +7,48 @@ sys.path.insert(0, os.path.split(__file__)[0]) # Force relative import for Py3k
     from coveragetest import CoverageTest
     import osinfo
     
    -class ThreadingTest(CoverageTest):
    -    """Tests of the threading support."""
    -
    -    def test_threading(self):
    -        self.check_coverage("""\
    -            import threading
    -
    -            def fromMainThread():
    -                return "called from main thread"
    -
    -            def fromOtherThread():
    -                return "called from other thread"
    -
    -            def neverCalled():
    -                return "no one calls me"
    -
    -            other = threading.Thread(target=fromOtherThread)
    -            other.start()
    -            fromMainThread()
    -            other.join()
    -            """,
    -            [1,3,4,6,7,9,10,12,13,14,15], "10")
    -
    -    def test_thread_run(self):
    -        self.check_coverage("""\
    -            import threading
    -
    -            class TestThread(threading.Thread):
    -                def run(self):
    -                    self.a = 5
    -                    self.do_work()
    -                    self.a = 7
    -
    -                def do_work(self):
    -                    self.a = 10
    -
    -            thd = TestThread()
    -            thd.start()
    -            thd.join()
    -            """,
    -            [1,3,4,5,6,7,9,10,12,13,14], "")
    +if not hasattr(sys, 'pypy_version_info'):
    +    class ThreadingTest(CoverageTest):
    +        """Tests of the threading support."""
    +    
    +        def test_threading(self):
    +            self.check_coverage("""\
    +                import threading
    +    
    +                def fromMainThread():
    +                    return "called from main thread"
    +    
    +                def fromOtherThread():
    +                    return "called from other thread"
    +    
    +                def neverCalled():
    +                    return "no one calls me"
    +    
    +                other = threading.Thread(target=fromOtherThread)
    +                other.start()
    +                fromMainThread()
    +                other.join()
    +                """,
    +                [1,3,4,6,7,9,10,12,13,14,15], "10")
    +    
    +        def test_thread_run(self):
    +            self.check_coverage("""\
    +                import threading
    +    
    +                class TestThread(threading.Thread):
    +                    def run(self):
    +                        self.a = 5
    +                        self.do_work()
    +                        self.a = 7
    +    
    +                    def do_work(self):
    +                        self.a = 10
    +    
    +                thd = TestThread()
    +                thd.start()
    +                thd.join()
    +                """,
    +                [1,3,4,5,6,7,9,10,12,13,14], "")
     
     
     class RecursionTest(CoverageTest):
    

    which yields:

    [last: 0] marca@SCML-MarcA:~/dev/git-repos/coveragepy$ .tox/pypy_py/bin/nosetests test.test_oddball -v
    test_doctest (test.test_oddball.DoctestTest) ... ok
    test_exception (test.test_oddball.ExceptionTest) ... ok
    test_multi_layers (test.test_oddball.GettraceTest) ... ok
    test_round_trip (test.test_oddball.GettraceTest) ... ok
    test_for_leaks (test.test_oddball.MemoryLeakTest) ... ok
    test_pyexpat (test.test_oddball.PyexpatTest) ... ok
    test_long_recursion (test.test_oddball.RecursionTest) ... ok
    test_long_recursion_recovery (test.test_oddball.RecursionTest) ... ok
    test_short_recursion (test.test_oddball.RecursionTest) ... ok
    
    ----------------------------------------------------------------------
    Ran 9 tests in 1.118s
    
    OK
    [last: 2] marca@SCML-MarcA:~/dev/git-repos/coveragepy$ .tox/py27_py/bin/nosetests test.test_oddball -v
    test_doctest (test.test_oddball.DoctestTest) ... ok
    test_exception (test.test_oddball.ExceptionTest) ... ok
    test_multi_layers (test.test_oddball.GettraceTest) ... ok
    test_round_trip (test.test_oddball.GettraceTest) ... ok
    test_for_leaks (test.test_oddball.MemoryLeakTest) ... ok
    test_pyexpat (test.test_oddball.PyexpatTest) ... ok
    test_long_recursion (test.test_oddball.RecursionTest) ... ok
    test_long_recursion_recovery (test.test_oddball.RecursionTest) ... ok
    test_short_recursion (test.test_oddball.RecursionTest) ... ok
    test_thread_run (test.test_oddball.ThreadingTest) ... ok
    test_threading (test.test_oddball.ThreadingTest) ... ok
    
    ----------------------------------------------------------------------
    Ran 11 tests in 0.116s
    
    OK
    

    ?

  3. Ned Batchelder repo owner

    I'm talking to PyPy people about why the threading test fails. We can skip it in PyPy as a last resort, but there's no reason this test should fail on PyPy, so I'd like to understand more before taking that drastic course of action.

  4. Ned Batchelder repo owner

    As of a567885df652, the memory test is less prone to false negatives. Now running oddball only on PyPy fails in one of the recursion tests? Weird.

  5. Log in to comment