nondeterministic results having to do with .pyc files

Dear Ned:

I have a test of trialcoverage which occasionally fails (maybe 1% of the time or so). The trial test runner's "--until-failure" feature is useful for this. :-) If you untar the attached trialcoverage-0.3.12-r46.tar.bz2 and run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs" then you should see a test failure eventually. If instead you run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp" then it can run indefinitely without failing. The difference between these two tests is that the one called "_with_pycs" allows CPython (which it is executing in a subprocess) to generate .pyc files, and the other one passes PYTHONDONTWRITEBYTECODE=1 to prevent that. In other words, the generation of .pyc files seems to trigger an irreproducible bug.

If you could have a look at this I would appreciate it. The bug is that the wrong source code is being used to produce a summary. I wonder if coverage.py could internally in itself implement a policy of ignoring .pyc files (icky things are more trouble than they are worth), and if it did if it would eliminate this bug.

Regards,

Zooko

Oh, this was with today's tip of coverage.py.

P.S.:

"I reported a bug in test test tests and included my test test test tests to demonstrate the bug. Ned understands me." — http://twitter.com/zooko/status/21044048130

Steps to reproduce as a transcript from my shell:

 Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ tar xf trialcoverage-0.3.12-r46.tar.bz2.tar
 Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ cd trialcoverage-0.3.12-r46
 Zooko-Ofsimplegeos-MacBook-Pro:~/tmp/trialcoverage-0.3.12-r46$ trial --until-failure  trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs
Test Pass 1
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.948s

PASSED (successes=1)
Test Pass 2
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.466s

PASSED (successes=1)
Test Pass 3
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.296s

PASSED (successes=1)
Test Pass 4
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.344s

PASSED (successes=1)
Test Pass 5
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.439s

PASSED (successes=1)
Test Pass 6
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 3.582s

PASSED (successes=1)
Test Pass 7
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.437s

PASSED (successes=1)
Test Pass 8
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.740s

PASSED (successes=1)
Test Pass 9
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.454s

PASSED (successes=1)
Test Pass 10
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.400s

PASSED (successes=1)
Test Pass 11
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                    [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 2.393s

PASSED (successes=1)
Test Pass 12
trialcoverage.test.test_tests
  Subprocess
    test_clean_start_subp_with_pycs ...                                  [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/Users/zooko/tmp/trialcoverage-0.3.12-r46/trialcoverage/test/test_tests.py", line 434, in _check3
    self.failUnlessIn("Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).", result)
twisted.trial.unittest.FailTest: 'Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).' not in "running trial\nrunning egg_info\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nUnable to execute darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files.  If you are not building a package then you can ignore this warning.\nreading manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nwriting manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nrunning build_ext\nfakepackage7.test.test_fakemodule7.T.test_thing ... [OK]\nCoverage results written to /Users/zooko/tmp/trialcoverage-0.3.12-r46/_trial_temp/tmpX58Foq/.coverage\n\n-------------------------------------------------------------------------------\ncode coverage summary\nWARNING code coverage regression\nPrevious best coverage left 0 total lines untested (0 lines uncovered and 0 lines partially covered).\nCurrent coverage left 3 total lines untested (3 lines uncovered and 0 lines partially covered).\n\n-------------------------------------------------------------------------------\nRan 1 tests in 0.009s\n\nFAILED (successes=1)\n"

trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs
-------------------------------------------------------------------------------
Ran 1 tests in 2.410s

FAILED (failures=1)

Sorry that I left this issue for so long. I don't suppose there's any chance you are still able to reproduce this?

Well... I haven't been maintaining trialcoverage for years now, but I think someone else may have picked it up.

In any case, if I really made the irreproducible bug as reproducible as I claimed in the comments, then a new maintainer or you yourself have a chance if reproducing it.

Btw, I still think that pyc/pyo files should be eliminated and everyone everywhere should set PYTHONDONTWRITEBYTECODE=1 and should refuse to load or acknowledge the existence of pyc/pyo files. :-)

You make a good point about the reproducibility of the bug.

And I'll stay agnostic about the .pyc philosophy :)