1. Ned Batchelder
  2. coverage.py
  3. Issues
Issue #243 resolved

SymPy script for documentation coverage: are you interested?

Aaron Meurer
created an issue

We have a script that we've been using for SymPy to test the "coverage" of our documentation. It lives at https://github.com/sympy/sympy/blob/master/bin/coverage_doctest.py. I was wondering if you would be interested in including it in coverage. I figure this sort of thing could be useful to everyone, not just SymPy.

The script supports checking for the following:

  • Checks all public functions, classes, and methods (name doesn't start with _) for docstrings. It skips test files.
  • If the function has a docstring, checks if it has a doctest.
  • If the function has a doctest, checks if the doctest actually mentions the function (this can be overridden by adding "# indirect doctest" to the docstring).
  • Checks if the function is imported into the Sphinx docs.

Example output:

$./bin/coverage_doctest.py sympy/matrices/sparse.py -v
DOCTEST COVERAGE for sympy/matrices/sparse.py
======================================================================

----------------------------------------------------------------------
sympy.matrices.sparse
----------------------------------------------------------------------

CLASSES

Missing docstrings
  * LINE 1047: MutableSparseMatrix

FUNCTIONS

Missing docstrings
  * LINE 109: copy(self)
  * LINE 497: submatrix(self, keys)
  * LINE 1052: as_mutable(self)
  * LINE 1338: copyin_list(self, key, value)
  * LINE 1343: copyin_matrix(self, key, value)

Missing doctests
  * LINE 347: scalar_multiply(self, scalar)
  * LINE 923: solve(self, rhs, method=LDL)
  * LINE 1016: as_immutable(self)
  * LINE 1021: zeros(cls, r, c=None)
  * LINE 1037: eye(cls, n)

Indirect doctests
  * LINE 190: row_list(self)
  * LINE 220: col_list(self)

    Use "# indirect doctest" in the docstring to supress this warning

Not imported into Sphinx
  * LINE 1343: copyin_matrix(self, key, value)
  * LINE 1052: as_mutable(self)
  * LINE 1338: copyin_list(self, key, value)
  * LINE 497: submatrix(self, keys)
  * LINE 109: copy(self)

----------------------------------------------------------------------
Doctests: 66% (26 of 39)
Sphinx: 87% (34 of 39)
----------------------------------------------------------------------

======================================================================
TOTAL DOCTEST SCORE for sympy.matrices.sparse: 66% (26 of 39)
TOTAL SPHINX SCORE for sympy.matrices.sparse: 87% (34 of 39)

If you are interested, I should note that the code is very ugly. It may need to be completely rewritten. It works by importing each module, though this is probably unnecessary, as the whole thing could be done using ast instead. It's also very sympy specific (there are no configuration variables or anything).

The Sphinx docs checking requires that the Sphinx docs be built and that they use the viewcode extension (this makes it easy to find what functions are included, see for example .

All of SymPy is BSD licensed, which as far as I can tell coverage is too, so there are no issues there.

Comments (2)

  1. Ned Batchelder repo owner

    Aaron, thanks, but I think this is different enough from coverage.py's mission that it should be its own tool. In many ways, this is more like pylint than like coverage, since it doesn't involve measuring actual execution. It's doing static analysis to conformance to rules about how to write doctests.

  2. Log in to comment