1. Devin Jeanpierre
  2. doctest2
  3. Issues
Issue #37 open

Add support for isolation between examples

Thomas Lotze
created an issue

One of the big problems of doctests is that all examples run inside the same namespace which means that later examples may depend on the state of the namespace and the environment in general that was left behind by earlier examples. It would be nice to be able to tear down and set up again the environment including the execution namespace at a similar granularity that is usual with programmatic unit tests. I don't have a particular opinion on a good approach or good syntax just yet, but I'd be very much interested in discussing the subject.

Comments (2)

  1. Devin Jeanpierre repo owner
    • changed status to open

    Agree. Syntax might be an issue, especially while leaving it backwards compatible -- whatever is used to indicate a new section must not ever naturally appear inside docstrings.

    Most likely it'd be some kind of label/title for the new doctest sequence. For example, maybe something like

            >>> foo
            >>> bar
        === barfoo permutation ===
            >>> bar
            >>> foo

    Of course, "=== <title> ===" is terrible notation because it wouldn't look good at all in Sphinx. OTOH Sphinx has its own way of solving this problem. So I don't think that matters. It's also probably better if it's required to be on a separate indentation level, so that it can't be confused with doctest output.

    If there's some other way to separate doctests arleady in use, that'd be preferable. Also I know that some other doctest-like things have had section labels before.

    It is totally acceptable (to me) to require the section label to be something that can only appear immediately before a new doctest, and not in between two, such as this modification of the syntax stolen from http://code.google.com/p/shell-doctest/ :

        [New Label]
        >>> # ^ this is the label/group for the new doctest
        [New Label]
        >>> # ^ that was expected output from the previous doctest test-case

    I think I want to make doctest sections/labeling identical to Sphinx's "group" (see http://sphinx.pocoo.org/ext/doctest.html ), so that if doctests have the same section title they go in the same doctest sequence, and if they have the title "*" they go in all of them. This can be used to implement common setup/teardown stuff without copy-paste and without too much craziness.

    Anyway. I'll think on it. Thanks for the report. Please let me know what you think of the above. :)

  2. Thomas Lotze reporter

    You're right about looking at how Sphinx solves this problem, as that is an existing and well-known solution and I think whatever the solution may turn out to be, it should be compatible with both Sphinx and plain restructured-text rendering. That leaves basically ReST comments to denote things like grouping or resetting the environment between doctest snippets. It would be OK with me if a Sphinx extension was necessary to make Sphinx run doctests marked up in such a way.

    OTOH, I would prefer a solution that is very easy to use (as in, requires very little thinking and typing) in the simple and rather common case that one has a doctest file with a number of test snippets that should all be isolated from each other but otherwise share a common set-up and tear-down. Using groups for this would require coming up with a new group name for every single test snippet, so a simpler solution (in addition to grouping maybe) should be available. Maybe something like

    .. testreset

    that would run tearDown followed by setUp.

  3. Log in to comment