1. Ned Batchelder
  2. coverage.py
Issue #84 new

nondeterministic results having to do with .pyc files

Zooko O'Whielacronx
created an issue

Dear Ned:

I have a test of trialcoverage which occasionally fails (maybe 1% of the time or so). The trial test runner's "--until-failure" feature is useful for this. :-) If you untar the attached trialcoverage-0.3.12-r46.tar.bz2 and run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs" then you should see a test failure eventually. If instead you run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp" then it can run indefinitely without failing. The difference between these two tests is that the one called "_with_pycs" allows CPython (which it is executing in a subprocess) to generate .pyc files, and the other one passes PYTHONDONTWRITEBYTECODE=1 to prevent that. In other words, the generation of .pyc files seems to trigger an irreproducible bug.

If you could have a look at this I would appreciate it. The bug is that the wrong source code is being used to produce a summary. I wonder if coverage.py could internally in itself implement a policy of ignoring .pyc files (icky things are more trouble than they are worth), and if it did if it would eliminate this bug.

Regards,

Zooko

Oh, this was with today's tip of coverage.py.

Comments (2)

  1. Zooko O'Whielacronx reporter

    Steps to reproduce as a transcript from my shell:

     Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ tar xf trialcoverage-0.3.12-r46.tar.bz2.tar
     Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ cd trialcoverage-0.3.12-r46
     Zooko-Ofsimplegeos-MacBook-Pro:~/tmp/trialcoverage-0.3.12-r46$ trial --until-failure  trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs
    Test Pass 1
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.948s
    
    PASSED (successes=1)
    Test Pass 2
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.466s
    
    PASSED (successes=1)
    Test Pass 3
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.296s
    
    PASSED (successes=1)
    Test Pass 4
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.344s
    
    PASSED (successes=1)
    Test Pass 5
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.439s
    
    PASSED (successes=1)
    Test Pass 6
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 3.582s
    
    PASSED (successes=1)
    Test Pass 7
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.437s
    
    PASSED (successes=1)
    Test Pass 8
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.740s
    
    PASSED (successes=1)
    Test Pass 9
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.454s
    
    PASSED (successes=1)
    Test Pass 10
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.400s
    
    PASSED (successes=1)
    Test Pass 11
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                    [OK]
    
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.393s
    
    PASSED (successes=1)
    Test Pass 12
    trialcoverage.test.test_tests
      Subprocess
        test_clean_start_subp_with_pycs ...                                  [FAIL]
    
    ===============================================================================
    [FAIL]
    Traceback (most recent call last):
      File "/Users/zooko/tmp/trialcoverage-0.3.12-r46/trialcoverage/test/test_tests.py", line 434, in _check3
        self.failUnlessIn("Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).", result)
    twisted.trial.unittest.FailTest: 'Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).' not in "running trial\nrunning egg_info\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nUnable to execute darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files.  If you are not building a package then you can ignore this warning.\nreading manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nwriting manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nrunning build_ext\nfakepackage7.test.test_fakemodule7.T.test_thing ... [OK]\nCoverage results written to /Users/zooko/tmp/trialcoverage-0.3.12-r46/_trial_temp/tmpX58Foq/.coverage\n\n-------------------------------------------------------------------------------\ncode coverage summary\nWARNING code coverage regression\nPrevious best coverage left 0 total lines untested (0 lines uncovered and 0 lines partially covered).\nCurrent coverage left 3 total lines untested (3 lines uncovered and 0 lines partially covered).\n\n-------------------------------------------------------------------------------\nRan 1 tests in 0.009s\n\nFAILED (successes=1)\n"
    
    trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs
    -------------------------------------------------------------------------------
    Ran 1 tests in 2.410s
    
    FAILED (failures=1)
    
  2. Log in to comment