Not picking up doctest tests

Larry Gilbert avatarLarry Gilbert created an issue

In modules that incorporate doctest tests, lines that are run by doctest are missed (and show up as "missed") by coverage.py.

(This could be a usage error, but I have not found any other references to this problem.)

Comments (9)

  1. Ned Batchelder

    If I understand what you mean, coverage.py won't indentify the lines in the doctests as executable. This is because they aren't executable: it's the doctest module that pre-processes the docstring and executes it.

    Or do you mean that your product code executed by doctests is not identified as executed?

    If the first, I wouldn't have thought this would be a problem, since doctests don't typically include conditionals that would make coverage testing important.

    If the latter, please provide a small code sample.

  2. Larry Gilbert

    You're right, it wouldn't make sense to expect the doctest lines themselves to show up as covered.

    The problem I encountered was that lines of code tested with doctest were not being highlighted correctly. I will try to work up an example for you.

    Incidentally, I've been trying out nose for tests, which as you probably know uses coverage.py for its coverage reports. nosetests has an option to include doctest test runs when calculating covered code. This did what I expected in the first place. So maybe I am expecting it to be coverage.py's job when it really is up to something like nose. I'm still a rookie at all of this. :-)

  3. Anonymous

    Hello Dears,

    First of all, thanks Ned for your coverage tool.

    I have made a simple example for you that is using doctests. The results shown by coverage are not ok unfortunately:

    check_doctest_coverage.py:

    def return_arg_or_void(arg):
      """If <arg> is None, return "Void"; otherwise return <arg>
      
      >>> return_arg_or_void(None)
      'Void'
      >>> return_arg_or_void("arg")
      'arg'
      >>> return_arg_or_void("None")
      'None'
      """
      if arg is None:
        return "Void"
      else:
        return arg
      
    if __name__ == "__main__":
      import doctest
      doctest.testmod()
    

    When you run python check_doctest_coverage.py -v

    Trying:
        return_arg_or_void(None)
    Expecting:
        'Void'
    ok
    Trying:
        return_arg_or_void("arg")
    Expecting:
        'arg'
    ok
    Trying:
        return_arg_or_void("None")
    Expecting:
        'None'
    ok
    1 items had no tests:
        __main__
    1 items passed all tests:
       3 tests in __main__.return_arg_or_void
    3 tests in 2 items.
    3 passed and 0 failed.
    Test passed.
    

    When you annotate the file with coverage annotate -d res check_doctest_coverage.py

    > def return_arg_or_void(arg):
    >   """If <arg> is None, return "Void"; otherwise return <arg>
        
    >   >>> return_arg_or_void(None)
    >   'Void'
    >   >>> return_arg_or_void("arg")
    >   'arg'
    >   >>> return_arg_or_void("None")
    >   'None'
    >   """
    >   if arg is None:
    >     return "Void"
    !   else:
    !     return arg
        
    > if __name__ == "__main__":
    >   import doctest
    >   doctest.testmod()
    

    So the else part is reported as not executed, although it is (the run of doctest reported 3 successful tests)

    Kind regards,

    -- Frederic F. M

  4. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.