nondeterministic results having to do with .pyc files
Dear Ned:
I have a test of trialcoverage which occasionally fails (maybe 1% of the time or so). The trial test runner's "--until-failure" feature is useful for this. :-) If you untar the attached trialcoverage-0.3.12-r46.tar.bz2 and run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs" then you should see a test failure eventually. If instead you run "trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp" then it can run indefinitely without failing. The difference between these two tests is that the one called "_with_pycs" allows CPython (which it is executing in a subprocess) to generate .pyc files, and the other one passes PYTHONDONTWRITEBYTECODE=1 to prevent that. In other words, the generation of .pyc files seems to trigger an irreproducible bug.
If you could have a look at this I would appreciate it. The bug is that the wrong source code is being used to produce a summary. I wonder if coverage.py could internally in itself implement a policy of ignoring .pyc files (icky things are more trouble than they are worth), and if it did if it would eliminate this bug.
Regards,
Zooko
Oh, this was with today's tip of coverage.py.
Comments (7)
-
-
Steps to reproduce as a transcript from my shell:
Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ tar xf trialcoverage-0.3.12-r46.tar.bz2.tar Zooko-Ofsimplegeos-MacBook-Pro:~/tmp$ cd trialcoverage-0.3.12-r46 Zooko-Ofsimplegeos-MacBook-Pro:~/tmp/trialcoverage-0.3.12-r46$ trial --until-failure trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs Test Pass 1 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.948s PASSED (successes=1) Test Pass 2 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.466s PASSED (successes=1) Test Pass 3 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.296s PASSED (successes=1) Test Pass 4 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.344s PASSED (successes=1) Test Pass 5 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.439s PASSED (successes=1) Test Pass 6 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 3.582s PASSED (successes=1) Test Pass 7 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 2.437s PASSED (successes=1) Test Pass 8 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 2.740s PASSED (successes=1) Test Pass 9 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 2.454s PASSED (successes=1) Test Pass 10 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 2.400s PASSED (successes=1) Test Pass 11 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [OK] ------------------------------------------------------------------------------- Ran 1 tests in 2.393s PASSED (successes=1) Test Pass 12 trialcoverage.test.test_tests Subprocess test_clean_start_subp_with_pycs ... [FAIL] =============================================================================== [FAIL] Traceback (most recent call last): File "/Users/zooko/tmp/trialcoverage-0.3.12-r46/trialcoverage/test/test_tests.py", line 434, in _check3 self.failUnlessIn("Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).", result) twisted.trial.unittest.FailTest: 'Current coverage left 2 total lines untested (2 lines uncovered and 0 lines partially covered).' not in "running trial\nrunning egg_info\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nwriting fakedistribution7.egg-info/PKG-INFO\nwriting top-level names to fakedistribution7.egg-info/top_level.txt\nwriting dependency_links to fakedistribution7.egg-info/dependency_links.txt\nUnable to execute darcs -- if you are building a package with 'setup.py sdist', 'setup.py bdist_egg', or other package-building commands, then the resulting package might be missing some files. If you are not building a package then you can ignore this warning.\nreading manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nwriting manifest file 'fakedistribution7.egg-info/SOURCES.txt'\nrunning build_ext\nfakepackage7.test.test_fakemodule7.T.test_thing ... [OK]\nCoverage results written to /Users/zooko/tmp/trialcoverage-0.3.12-r46/_trial_temp/tmpX58Foq/.coverage\n\n-------------------------------------------------------------------------------\ncode coverage summary\nWARNING code coverage regression\nPrevious best coverage left 0 total lines untested (0 lines uncovered and 0 lines partially covered).\nCurrent coverage left 3 total lines untested (3 lines uncovered and 0 lines partially covered).\n\n-------------------------------------------------------------------------------\nRan 1 tests in 0.009s\n\nFAILED (successes=1)\n" trialcoverage.test.test_tests.Subprocess.test_clean_start_subp_with_pycs ------------------------------------------------------------------------------- Ran 1 tests in 2.410s FAILED (failures=1)
-
Sorry that I left this issue for so long. I don't suppose there's any chance you are still able to reproduce this?
-
- changed status to on hold
-
Well... I haven't been maintaining trialcoverage for years now, but I think someone else may have picked it up.
In any case, if I really made the irreproducible bug as reproducible as I claimed in the comments, then a new maintainer or you yourself have a chance if reproducing it.
Btw, I still think that pyc/pyo files should be eliminated and everyone everywhere should set PYTHONDONTWRITEBYTECODE=1 and should refuse to load or acknowledge the existence of pyc/pyo files. :-)
-
You make a good point about the reproducibility of the bug.
And I'll stay agnostic about the .pyc philosophy :)
-
- changed status to closed
- Log in to comment
P.S.:
"I reported a bug in test test tests and included my test test test tests to demonstrate the bug. Ned understands me." — http://twitter.com/zooko/status/21044048130