Coverage of assert statements.

Issue #160 closed
npinto created an issue


I'm trying to use to get the coverage (duh!) of both outcomes in assert statements [1]. Am I missing something or is this not (yet) possible?

Let me know how can I contribute.


[1] In our code base we use many (possibly very slow) asserts during development, this is really useful in production for performance reasons since we can use python -OO to remove the execution of asserts.

Comments (11)

  1. Ned Batchelder repo owner can't influence what code paths you execute, it merely monitors what code paths are executed. It's up to you to design a test suite that exercises all of the test cases you need tested. Maybe I don't understand your problem?

  2. npinto reporter

    But won't tell me if my test suite elicited both path of an assert statement (True or False). Any thoughts on how to expose this in ?

  3. Ned Batchelder repo owner

    I see what you mean. There are a few things that come to mind:

    1) I didn't realize this, but an assert statement is actually compiled as if it were: if condition: raise AssertionError, so I suppose there's enough information in the bytecode to do what you want. Actually, I wonder why it doesn't already do what you want.

    2) It's not clear to me that everyone would want to require that all their asserts go both ways to consider their code fully covered.

  4. Former user Account Deleted

    Thanks for the fast answer.

    Re: 1)

    Is there an easy way to expose this, and maybe enable this behavior with a command line option?

    Re: 2)

    This is a valid point, but I like using asserts since they are very concise *and* easily removed with python -OO for production.

    I'll be happy to take a stab at the code but since I'm not "in" the codebase right now, I would appreciate pointers / strategies on how to do this optimally (i.e. that will fit your requirements to get merged).

    Thanks again.

  5. npinto reporter


    I'll be happy to contribute a pull request if you let me know how you would like this to be exposed, of course only if you think the feature is interesting.



  6. Ned Batchelder repo owner

    Earlier I said "I don't know why it doesn't already do this." It's because the compiled code is all associated with the same source line. For example, coverage also can't do branch coverage for a one-line if like this:

    if a: print b

    So something significant would have to change in for it to be able to detect whether asserts went both ways. Possibilities include:

    1. Preprocess the source to change assert statements
    2. Preprocess the bytecode to fiddle with the line numbers
    3. Change the trace function to record exceptions raised

    None of these are simple, and represent a major new feature in

  7. npinto reporter

    Thanks Ned for the quick reply.

    I'd like to try "1." (i.e. code transformation before coverage), at least get the ball rolling. What would be the best entry point to begin this?

    Thanks again.

  8. Ned Batchelder repo owner

    Hmm, I'm reluctant to start transforming source code, especially since it would throw off the source code reporting. I'm not sure any of these approaches is a good idea, but #3 fits the current code base best. I'm honestly not sure what information is available in the trace function for tracking exceptions, but you can give it a try. Look in, line 96 to get started.

  9. Felipe

    Not at all familiar with the code, but given that the 3 proposed alternatives seemed so difficult let me chime in with a 4th:

    Add a # pragma: raises Exception_1 Exception_2 ... pragma that lets the user hint what exceptions are potentially raised on a given line. Then, in parallel/after the existing analysis run a simple scan to parse out try/except clauses using ast and tokenize to pull out the comments (like in your SO answer here). You could then add a branch from the pragma line to the error handler.You could start out only supporting intra-file raises pragmas for simplicity.

  10. Log in to comment