Issue #135 invalid

allow to disable the asserts

Anonymous avatarAnonymous created an issue

Hey - we use py.test as our product test platform - and we find it very useful.

One feature though we uses and might be nice if will be shared with the word is disable asserts during test development. When we write integration test and system test - that take long time to run - we asserts a lot of thing (the single assert per test methodology is nice but in practice it reduce the time take the tests to run) - and most of the time (again in development) we fail on asserts. Moreover - we use mocks for speed the development process of this kind of tests (find name errors and other thing that we don't want to waste half a hour to discover) - but the mocks tend to fail on the first assert the meet.

Therefore we used your great frame work and write some plugin that monkey patch the assert rewrite module with the following class instead of AssertionRewriter class.

class AssertionRemover(AssertionRewriter):
    def visit_Assert(self, assert_):
        return ast.copy_location(ast.Pass(), assert_)

and patches PYC_EXT - to '.pycNoAsserts'. That works nice and helped us a lot. but because of the entire monkey patching is kind of fragile (the code might of rewrite module might change someday and we will never know) it would be nice if this option will be supported naturally by py.test. (we know it sounds a strange feature to ask - the ability to disables asserts in test - but it works and allow us to reduce our development time)

Comments (6)

  1. Ronny Pfannschmidt

    that use-case does indeed sound rather strange

    before accepting we'd like to get a better understanding about the reasons you do things the way you do them

    your general description leaves me with a "something is not right/missing" feeling

  2. Ronny Pfannschmidt

    lt me recap my basic understanding of the problem

    1. in real tests you use system objects, stuff is fine
    2. in quick tests you use mocks, asserts fail

    is it correct that you want to skip over assertions if a certain mock object is used?

  3. Anonymous

    Thanks for your quick response. I will try to give more details about how we works in order to clarify the issue.

    We develop a complex project not in python - but we use py.test and python in order to write our integration tests. Most of our tests look like this :

    def test_scenario(a_remote_object):
        res1 = do_step1(a_remote_object)
        check_results(res1)
    
        res2 = do_step2(a_remote_object)
        check_results(res2)
    
        res3 = do_step3(a_remote_object)
        check_results(res3)
    
        ...
    

    As you can see our tests are just series of actions against a remote object. What we trying to test is the remote object behavior (not the API for working with it). The problem is that the work against the remote object is very very very slow (as i said complex project..) . So, when we develop an integration test - we run it against mock objects. We don't want to run the tests against mock object after we finished develop them because as we mentioned before - we test the remote object. We use mocks to discover python NameErrors\TypeErrors and mistakes in the tests flows (most of the test are more complex than 3 steps scenarios). The problem is that our mock object don't mock entirely the remote objects so they tend to fail by the asserts in the test in early steps. its frustrated to work like this. The solution found by one of our programmers is to use the AST rewrite that pytest contains to add --cancel-asserts command line option that removes the asserts for the test and replace them with pass or printing.

    I guess the feature that we really asks is the ability customize the use of py.test AST rewrite code. Now we just hack around it - but we think it would be great if we could implement a hook that return our implementation of AssertionRewriter class. For example - wouldn't it be nice if instead of failed assert - you could run your tests in a mode that open a pdb shell in the assert place and allow you to continue the test after your see what failed? All it's required is to replace:

    assert (expr) (msg) 
    

    with a nodes that does:

    if not (expr) : 
        print "Assert Failed : (msg)"
        import pdb
        pdb.set_trace()
    

    And the options are endless.

  4. holger krekel

    Anonymous, the example you are giving does not directly contain asserts. Are they contained in the check_function?

    I am also reluctant to expose AssertionRewriter and i don't think you could implement this pdb behaviour easily or did you try that?

    Did you try to a route your checks through a function which could be instrumented via e.g. --cancel-asserts to ignore assertion errors?

    Maybe in the end a general --cancel-asserts like option might make some sense. Probably spelled --assert=remove. I am not convinced yet.

  5. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.