Running doctest prevents complete coverage collection

Issue #575 resolved
Ned Batchelder
repo owner created an issue

sample.py:

import doctest

TEST = """
>>> x = 1
>>> x
1
"""

doctest.run_docstring_examples(TEST, {})

q = 11
r = 12

Running coverage leaves the last two lines unmeasured:

$ coverage run sample.py && coverage report -m
Name        Stmts   Miss  Cover   Missing
-----------------------------------------
sample.py       5      2    60%   11-12

Switching from the C tracer to the Python tracer fixes it:

$ coverage run --timid sample.py && coverage report -m
Name        Stmts   Miss  Cover   Missing
-----------------------------------------
sample.py       5      0   100%

The problem only appears in Python 3. It's due to the sys.settrace in the finally clause in this commit: https://github.com/python/cpython/commit/31f5929c1e28adcaa1fdb302da366f3c7a92a98a, which was to fix this bug: http://bugs.python.org/issue10990

There seems to be something wrong with getting and restoring the C trace function. (Odd though, since there are tests that check this...)

Comments (7)

  1. Ned Batchelder reporter

    A simpler reproducer, with no doctest:

    sample.py:

    from swap import swap_it
    
    print(3)
    swap_it()
    print(5)
    

    swap.py:

    import sys
    
    def swap_it():
        no_really()
    
    def no_really():
        save_trace = sys.gettrace()
        sys.settrace(save_trace)
    

    Three runs:

    $ coverage run sample.py && coverage report -m
    3
    5
    Name        Stmts   Miss  Cover   Missing
    -----------------------------------------
    sample.py       4      0   100%
    swap.py         6      0   100%
    -----------------------------------------
    TOTAL          10      0   100%
    $ coverage run --source=sample sample.py && coverage report -m
    3
    5
    Name        Stmts   Miss  Cover   Missing
    -----------------------------------------
    sample.py       4      1    75%   5
    $ coverage run --source=sample --timid sample.py && coverage report -m
    3
    5
    Name        Stmts   Miss  Cover   Missing
    -----------------------------------------
    sample.py       4      0   100%
    

    I'm not sure why --source changes the behavior. Also, if the settrace(gettrace) happens in sample.py, then the problem doesn't happen. All of this is the same on Python 2.7 and Python 3.6 (the py3 claim above is because of changes in doctest).

  2. Ned Batchelder reporter

    More details as they dribble in: this also happens with coverage.py 3.7.1.

    Seems that if the file that does the settrace(gettrace) isn't being traced itself, the problem happens. This is why it happens with doctest, and with my own file without --source.

  3. Log in to comment