Performance degradation from 3.7.1 to 4.0

Issue #387 resolved
Ned Batchelder repo owner created an issue

(From DRMacIver) Here's a reproduction for the problem we discussed on IRC.

I've attached a file that shows significant performance loss running under coverage and modest performance loss under 4.0a5 vs 3.7.1

These are run against 32a214a141bbc81bd624890216899261745f8588 of git@github.com:DRMacIver/hypothesis.git. Note that if you run against 1.2.1 this test has significantly different performance characteristics (it will be much faster).

Command I used to run, from the root of a hypothesis checkout:

With coverage: time PYTHONPATH=src python -m coverage run --branch --rcfile=/dev/null test_coverage_perf.py

Without coverage: time PYTHONPATH=src python test_coverage_perf.py

Numbers are about as variable as you would expect from this extremely shonky benchmarking method, but roughly what I'm seeing is:

Without coverage: ~14s
With coverage 3.7.1: ~35s
With coverage 4.0a5: ~50s

This is running on CPython 3.3.6 on an oldish laptop running Linux Mint. Coverage was using CTracer in both instances.

Let me know if there's anything I can do to help with debugging this or if you need more information.

Comments (4)

  1. Ned Batchelder reporter

    Thanks for the test case. I've reproduced your results. Normalizing the non-coverage time to 1.0, running under coverage 3.7.1 is 2.28, and under coverage 4.0a6 is 2.99, for a 31% slowdown between coverage versions.

    I also ran under 2.7.10, which had a similar-ish slowdown between coverage versions. The surprising thing was the non-coverage run time for your code was 4.3x slower under Python 3.3.6 than under 2.7.10!?

  2. Log in to comment