can't get coverage to run doctests

Issue #152 invalid
Mario Frasca
created an issue

I think I followed the steps described in <<issue 19>>, but the result I get here is even worse than what is reported there. I'm using coverage 3.4 and 3.5.1, but in both cases I get an "empty" annotation, no line seems to be covered at all.

Comments (11)

  1. Mario Frasca reporter

    I'm on ubuntu 11.04, python 2.7...

    this is what I'm doing in a bash shell.

    mario@lt92:~$ uname -srvi
    Linux 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64
    mario@lt92:~$ which python
    /usr/bin/python
    mario@lt92:~$ python --version
    Python 2.7.1+
    mario@lt92:~$ coverage --version
    Coverage.py, version 3.5.1.  http://nedbatchelder.com/code/coverage
    mario@lt92:~$ cat check_doctest_coverage.py
    #!/usr/bin/python
    
    def return_arg_or_void(arg):
      """If <arg> is None, return "Void"; otherwise return <arg>
      
      >>> return_arg_or_void(None)
      'Void'
      >>> return_arg_or_void("arg")
      'arg'
      >>> return_arg_or_void("None")
      'None'
      """
      if arg is None:
        return "Void"
      else:
        return arg
      
    if __name__ == "__main__":
      import doctest
      doctest.testmod()
    mario@lt92:~$ python check_doctest_coverage.py -v
    Trying:
        return_arg_or_void(None)
    Expecting:
        'Void'
    ok
    Trying:
        return_arg_or_void("arg")
    Expecting:
        'arg'
    ok
    Trying:
        return_arg_or_void("None")
    Expecting:
        'None'
    ok
    1 items had no tests:
        __main__
    1 items passed all tests:
       3 tests in __main__.return_arg_or_void
    3 tests in 2 items.
    3 passed and 0 failed.
    Test passed.
    mario@lt92:~$ rm -fr res
    mario@lt92:~$ coverage annotate -d res check_doctest_coverage.py
    mario@lt92:~$ cat res/check_doctest_coverage.py,cover 
      #!/usr/bin/python
      
    ! def return_arg_or_void(arg):
    !   """If <arg> is None, return "Void"; otherwise return <arg>
        
    !   >>> return_arg_or_void(None)
    !   'Void'
    !   >>> return_arg_or_void("arg")
    !   'arg'
    !   >>> return_arg_or_void("None")
    !   'None'
    !   """
    !   if arg is None:
    !     return "Void"
    !   else:
    !     return arg
        
    ! if __name__ == "__main__":
    !   import doctest
    !   doctest.testmod()
    
    
  2. Ned Batchelder repo owner

    You've only invoked coverage.py once: to annotate. You first need to use it to run the code. Replace "python check_doctest_coverage.py -v" with "coverage run check_doctest_coverage.py -v" and all will be fine.

  3. Mario Frasca reporter

    well, the small sample here works fine and in fact I was missing RUNNING the tests before using the (uncollected) data. but now this is driving me crazy... I have a not-so-small library, extensively tested. a couple of tests are doc tests, most are unit tests, but I do not manage to have coverage see the doc tests!

    I'm making use of buildout, it takes care of defining a bin/test script that runs all tests in the library.

    does it already sound as a well known issue, or shall I provide a small sample library showing the problem? or ask the question for example on stackoverflow? -- but then I need to explain the situation from scratch --

  4. Ned Batchelder repo owner

    If you mean, the code executed by the doctests isn't covered, that sounds like a real issue. If you mean, the code *in* my doctests (that is, in the docstrings) isn't marked as covered, that's just the way it works. Those "lines of code" aren't really code as far as the Python interpreter is concerned.

  5. Mario Frasca reporter

    no, no, I understand that doctest are just constant character objects within the code.

    the only way I managed to have the doctest cover code lines is by adding the three lines

    if __name__ == "__main__":
      import doctest
      doctest.testmod()
    

    at the end of the code and invoking the tests with ./bin/coverage.

    if I use ./bin/test, doctests are not considered.

  6. Mario Frasca reporter

    it seems to be a messed up buildout.cfg... so I'm stripping it to the bones until it works and then reconstruct it leaving out the part that prevent it from working. thanks for tool and patience.

  7. Mario Frasca reporter

    no, it's not a problem in the buildout configuration (exactly the same configuration works in a small example library but does not in more complex one). I do not have the time to examine it in depth and am going to run tests with an ad-hoc script. if you want, I can send you the full sources of the library that is not being tested properly.

  8. Mario Frasca reporter

    seems that it's the executable bit of the file that stands in the way.

    mario@lt92:~/example$ ls -l foo_bar/
    total 36
    -rw-r--r-- 1 mario mario 2780 2011-10-14 12:30 check.py
    -rw-r--r-- 1 mario mario 1581 2011-10-14 12:24 check.py~
    -rw-r--r-- 1 mario mario 1820 2011-10-14 12:30 check.pyc
    -rw-r--r-- 1 mario mario  184 2011-10-14 12:24 check_tests.py
    -rw-r--r-- 1 mario mario  210 2011-10-14 09:36 check_tests.py~
    -rw-r--r-- 1 mario mario  639 2011-10-14 12:25 check_tests.pyc
    -rw-r--r-- 1 mario mario   23 2011-10-14 12:23 __init__.py
    -rw-r--r-- 1 mario mario   32 2011-10-14 09:34 __init__.py~
    -rw-r--r-- 1 mario mario  157 2011-10-14 12:23 __init__.pyc
    mario@lt92:~/example$ bin/test 
    ...
    Name            Stmts   Miss  Cover   Missing
    ---------------------------------------------
    foo_bar             1      0   100%   
    foo_bar.check      15      0   100%   
    ---------------------------------------------
    TOTAL              16      0   100%   
    ----------------------------------------------------------------------
    Ran 3 tests in 0.003s
    
    OK
    mario@lt92:~/example$ chmod +x foo_bar/check.py
    mario@lt92:~/example$ bin/test 
    .
    Name            Stmts   Miss  Cover   Missing
    ---------------------------------------------
    foo_bar             1      0   100%   
    foo_bar.check      15     10    33%   46, 71-82
    ---------------------------------------------
    TOTAL              16     10    38%   
    ----------------------------------------------------------------------
    Ran 1 test in 0.001s
    
    OK
    mario@lt92:~/example$ 
    

    is this "as designed"?

  9. Ned Batchelder repo owner

    This may be a consequence of your test runner. Test runners try to discover and run tests automatically. But they are wary of executing files that have the +x bit, because that could indicate that the file is meant to be used as a script, not a module. That is, if the "script" were executed, it would do something real, like delete files, and so the test runner doesn't want to execute it directly. So test runners use the +x bit as a sign that the file should be skipped.

  10. Log in to comment