1. Ned Batchelder
  2. coverage.py
Issue #160 new

Coverage of assert statements.

npinto
created an issue

Hello,

I'm trying to use coverage.py to get the coverage (duh!) of both outcomes in assert statements [1]. Am I missing something or is this not (yet) possible?

Let me know how can I contribute.

Nicolas

[1] In our code base we use many (possibly very slow) asserts during development, this is really useful in production for performance reasons since we can use python -OO to remove the execution of asserts.

Comments (10)

  1. Ned Batchelder repo owner

    Coverage.py can't influence what code paths you execute, it merely monitors what code paths are executed. It's up to you to design a test suite that exercises all of the test cases you need tested. Maybe I don't understand your problem?

  2. npinto reporter

    But coverage.py won't tell me if my test suite elicited both path of an assert statement (True or False). Any thoughts on how to expose this in coverage.py ?

  3. Ned Batchelder repo owner

    I see what you mean. There are a few things that come to mind:

    1) I didn't realize this, but an assert statement is actually compiled as if it were: if condition: raise AssertionError, so I suppose there's enough information in the bytecode to do what you want. Actually, I wonder why it doesn't already do what you want.

    2) It's not clear to me that everyone would want to require that all their asserts go both ways to consider their code fully covered.

  4. Anonymous

    Thanks for the fast answer.

    Re: 1)

    Is there an easy way to expose this, and maybe enable this behavior with a command line option?

    Re: 2)

    This is a valid point, but I like using asserts since they are very concise *and* easily removed with python -OO for production.

    I'll be happy to take a stab at the code but since I'm not "in" the codebase right now, I would appreciate pointers / strategies on how to do this optimally (i.e. that will fit your requirements to get merged).

    Thanks again.

  5. npinto reporter

    Ned,

    I'll be happy to contribute a pull request if you let me know how you would like this to be exposed, of course only if you think the feature is interesting.

    Thanks.

    Nicolas

  6. Ned Batchelder repo owner

    Earlier I said "I don't know why it doesn't already do this." It's because the compiled code is all associated with the same source line. For example, coverage also can't do branch coverage for a one-line if like this:

    if a: print b

    So something significant would have to change in coverage.py for it to be able to detect whether asserts went both ways. Possibilities include:

    1. Preprocess the source to change assert statements
    2. Preprocess the bytecode to fiddle with the line numbers
    3. Change the trace function to record exceptions raised

    None of these are simple, and represent a major new feature in coverage.py.

  7. npinto reporter

    Thanks Ned for the quick reply.

    I'd like to try "1." (i.e. code transformation before coverage), at least get the ball rolling. What would be the best entry point to begin this?

    Thanks again.

  8. Ned Batchelder repo owner

    Hmm, I'm reluctant to start transforming source code, especially since it would throw off the source code reporting. I'm not sure any of these approaches is a good idea, but #3 fits the current code base best. I'm honestly not sure what information is available in the trace function for tracking exceptions, but you can give it a try. Look in collector.py, line 96 to get started.

  9. felipeochoa

    Not at all familiar with the coverage.py code, but given that the 3 proposed alternatives seemed so difficult let me chime in with a 4th:

    Add a # pragma: raises Exception_1 Exception_2 ... pragma that lets the user hint what exceptions are potentially raised on a given line. Then, in parallel/after the existing coverage.py analysis run a simple scan to parse out try/except clauses using ast and tokenize to pull out the comments (like in your SO answer here). You could then add a branch from the pragma line to the error handler.You could start out only supporting intra-file raises pragmas for simplicity.

  10. Log in to comment