1. Brianna Laugher
  2. pytest

Source

pytest / doc / en / resources.txt

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599

.. _resources:

test resource management and xUnit setup (on steroids)
=======================================================

.. _`Dependency injection`: http://en.wikipedia.org/wiki/Dependency_injection

.. versionadded: 2.3

pytest offers advanced resource parametrization and injection mechanisms
including a fully integrated generalization of the popular xUnit
setup-style methods.  A resource is created by a ``@pytest.mark.factory`` 
marked function and its name is the name of the function.  A resource
is injected into test or setup functions if they use the name 
in their signature.  Therefore and also for historic reasons, resources are
sometimes called "funcargs" because they ultimately appear as 
function arguments.

The pytest resource management and setup features are exposed through
three decorators:

* a `@pytest.mark.factory`_ marker to define resource factories,
  their scoping and parametrization.

* a `@pytest.mark.setup`_ marker to define setup functions and their 
  scoping.

* a `@pytest.mark.parametrize`_ marker for executing test functions
  multiple times with different parameter sets

Generally, resource factories and setup functions: 

- can be defined in test modules, test classes, conftest.py files or
  in plugins.

- can themselves receive resources through their function arguments, 
  simplifying the setup and use of interdependent resources.

- can use the special `testcontext`_ object for access to the 
  context in which the factory/setup is called and for registering 
  finalizers.

This document showcases these features through some basic examples.

Note that pytest also comes with some :ref:`builtinresources` which
you can use without defining them yourself.

Background and terms
---------------------------

The pytest resource management mechanism is an example of `Dependency
Injection`_ which helps to de-couple test code from resource
instantiation code required for them to execute.  At test writing time
you typically do not need to care for the details of how your required
resources are constructed, if they live through a function, class,
module or session scope or if the test will be called multiple times
with different resource instances.

To create a value with which to call a test function a resource factory 
function is called which gets full access to the test context and can
register finalizers which are to be run after the last test in that context
finished. Resource factories can be implemented in same test class or 
test module, in a per-directory ``conftest.py`` file or in an external plugin.  This allows total de-coupling of test and setup code.

A test function may be invoked multiple times in which case we
speak of :ref:`parametrized testing <parametrizing-tests>`. This can be
very useful if you want to test e.g. against different database backends
or with multiple numerical arguments sets and want to reuse the same set
of test functions.


.. _`@pytest.mark.factory`:

``@pytest.mark.factory``: Creating parametrized, scoped resources
-----------------------------------------------------------------

.. regendoc:wipe

.. versionadded:: 2.3

The `@pytest.mark.factory`_ marker allows to

* mark a function as a factory for resources used by test and setup functions 
* define parametrization to run tests multiple times with different
  resource instances
* set a scope which determines the level of caching. valid scopes
  are ``session``, ``module``, ``class`` and ``function``.

Here is a simple example of a factory creating a shared ``smtplib.SMTP``
connection resource which test functions then may use across the whole
test session::

    # content of conftest.py
    import pytest
    import smtplib

    @pytest.mark.factory(scope="session")
    def smtp(testcontext):
        smtp = smtplib.SMTP("merlinux.eu")
        testcontext.addfinalizer(smtp.close)
        return smtp

The name of the resource is ``smtp`` (the factory function name) 
and you can now access the ``smtp`` resource by listing it as
an input parameter in any test function below the directory where
``conftest.py`` is located::

    # content of test_module.py
    def test_ehlo(smtp):
        response = smtp.ehlo()
        assert response[0] == 250 
        assert "merlinux" in response[1]
        assert 0  # for demo purposes

    def test_noop(smtp):
        response = smtp.noop()
        assert response[0] == 250
        assert 0  # for demo purposes

If you run the tests::

    $ py.test -q
    collecting ... collected 2 items
    FF
    ================================= FAILURES =================================
    ________________________________ test_ehlo _________________________________
    
    smtp = <smtplib.SMTP instance at 0x1c7c638>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
            assert "merlinux" in response[1]
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:5: AssertionError
    ________________________________ test_noop _________________________________
    
    smtp = <smtplib.SMTP instance at 0x1c7c638>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    2 failed in 0.18 seconds

you will see the two ``assert 0`` failing and can see that
the same (session-scoped) object was passed into the two test functions.

If you now want to test multiple servers you can simply parametrize
the ``smtp`` factory::

    # content of conftest.py
    import pytest
    import smtplib

    @pytest.mark.factory(scope="session", 
                         params=["merlinux.eu", "mail.python.org"])
    def smtp(testcontext):
        smtp = smtplib.SMTP(testcontext.param)
        def fin():
            smtp.close()
        testcontext.addfinalizer(fin)
        return smtp

The main change is the definition of a ``params`` list in the
``factory``-marker and the ``testcontext.param`` access within the
factory function.  No test code needs to change.  So let's just do another
run::

    $ py.test -q
    collecting ... collected 4 items
    FFFF
    ================================= FAILURES =================================
    __________________________ test_ehlo[merlinux.eu] __________________________
    
    smtp = <smtplib.SMTP instance at 0x1d162d8>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
            assert "merlinux" in response[1]
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:5: AssertionError
    __________________________ test_noop[merlinux.eu] __________________________
    
    smtp = <smtplib.SMTP instance at 0x1d162d8>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    ________________________ test_ehlo[mail.python.org] ________________________
    
    smtp = <smtplib.SMTP instance at 0x1d1f098>
    
        def test_ehlo(smtp):
            response = smtp.ehlo()
            assert response[0] == 250
    >       assert "merlinux" in response[1]
    E       assert 'merlinux' in 'mail.python.org\nSIZE 10240000\nETRN\nSTARTTLS\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
    
    test_module.py:4: AssertionError
    ________________________ test_noop[mail.python.org] ________________________
    
    smtp = <smtplib.SMTP instance at 0x1d1f098>
    
        def test_noop(smtp):
            response = smtp.noop()
            assert response[0] == 250
    >       assert 0  # for demo purposes
    E       assert 0
    
    test_module.py:10: AssertionError
    4 failed in 6.42 seconds

We get four failures because we are running the two tests twice with
different ``smtp`` instantiations as defined on the factory.
Note that with the ``mail.python.org`` connection the second test
fails in ``test_ehlo`` because it expects a specific server string.

You can also look at what tests pytest collects without running them::

    $ py.test --collectonly
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7
    plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
    collecting ... collected 4 items
    <Module 'test_module.py'>
      <Function 'test_ehlo[merlinux.eu]'>
      <Function 'test_noop[merlinux.eu]'>
      <Function 'test_ehlo[mail.python.org]'>
      <Function 'test_noop[mail.python.org]'>
    
    =============================  in 0.02 seconds =============================

Note that pytest orders your test run by resource usage, minimizing
the number of active resources at any given time.


Accessing resources from a factory function
----------------------------------------------------------

You can directly use resources as funcargs in resource factories.  
Extending the previous example we can instantiate an application
object and stick the live ``smtp`` resource into it::

    # content of test_appsetup.py
   
    import pytest

    class App:
        def __init__(self, smtp):
            self.smtp = smtp

    @pytest.mark.factory(scope="module")
    def app(smtp):
        return App(smtp)

    def test_exists(app):
        assert app.smtp

Let's run this::

    $ py.test -v test_appsetup.py
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
    cachedir: /home/hpk/tmp/doc-exec-398/.cache
    plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
    collecting ... collected 2 items
    
    test_appsetup.py:12: test_exists[merlinux.eu] PASSED
    test_appsetup.py:12: test_exists[mail.python.org] PASSED
    
    ========================= 2 passed in 5.96 seconds =========================

Due to the parametrization of ``smtp`` the test will
run twice with two different ``App`` instances and respective smtp servers.
There is no need for the ``app`` factory to be aware of the parametrization.


.. _`new_setup`:
.. _`@pytest.mark.setup`:

``@pytest.mark.setup``: xUnit setup methods on steroids
-----------------------------------------------------------------

.. regendoc:wipe

.. versionadded:: 2.3

The ``@pytest.mark.setup`` marker allows 

* to define setup-functions close to test code or in conftest.py files
  or plugins.
* to mark a function as a setup method; the function can itself
  receive funcargs and will execute multiple times if the funcargs
  are parametrized
* to set a scope which influences when the setup function going to be
  called.  valid scopes are ``session``, ``module``, ``class`` and ``function``.

Here is a simple example.  First we define a global ``globdir`` resource::

    # content of conftest.py
    import pytest

    @pytest.mark.factory(scope="module")
    def globdir(testcontext, tmpdir):
        def fin():
            print "finalize", tmpdir
        testcontext.addfinalizer(fin)
        print "created resource", tmpdir
        return tmpdir

And then we write a test file containing a setup-marked function 
taking this resource and setting it as a module global::

    # content of test_module.py
    import pytest

    @pytest.mark.setup(scope="module")
    def setresource(testcontext, globdir):
        print "setupresource", globdir
        testcontext.module.myresource = globdir

    def test_1():
        assert myresource
        print "using myresource", myresource

    def test_2():
        assert myresource
        print "using myresource", myresource

Let's run this module::

    $ py.test -qs
    collecting ... collected 2 items
    ..
    2 passed in 0.26 seconds
    created resource /home/hpk/tmp/pytest-4427/test_10
    setupresource /home/hpk/tmp/pytest-4427/test_10
    using myresource /home/hpk/tmp/pytest-4427/test_10
    using myresource /home/hpk/tmp/pytest-4427/test_10
    finalize /home/hpk/tmp/pytest-4427/test_10

The two test functions in the module use the same global ``myresource``
object because the ``setresource`` set it as a module attribute.

The ``globdir`` factory can now become parametrized without any test
or setup code needing to change::

    # content of conftest.py
    import pytest

    @pytest.mark.factory(scope="module", params=["aaa", "bbb"])
    def globdir(testcontext, tmpdir):
        newtmp = tmpdir.join(testcontext.param)
        def fin():
            print "finalize", newtmp
        testcontext.addfinalizer(fin)
        print "created resource", newtmp
        return newtmp

Running the unchanged previous test files now runs four tests::

    $ py.test -qs
    collecting ... collected 4 items
    ....
    4 passed in 0.26 seconds
    created resource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
    setupresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
    using myresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
    using myresource /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
    finalize /home/hpk/tmp/pytest-4428/test_1_aaa_0/aaa
    created resource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
    setupresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
    using myresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
    using myresource /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb
    finalize /home/hpk/tmp/pytest-4428/test_1_bbb_0/bbb

Each parameter causes the creation of a respective resource and the
unchanged test module uses it in its ``@setup`` decorated method.

.. note::

   Tests using a particular parametrized resource instance will
   executed next to each other.  Any finalizers will be run before the 
   next parametrized resource instance is being setup and tests
   are rerun.

Grouping tests by resource parameters
----------------------------------------------------------

.. regendoc: wipe

pytest minimizes the number of active resources during test runs.
If you have a parametrized resource, then all the tests using one
resource instance will execute one after another.  Then any finalizers
are called for that resource instance and then the next parametrized
resource instance is created and its tests are run.  Among other things,
this eases testing of applications which create and use global state.

The following example uses two parametrized funcargs, one of which is 
scoped on a per-module basis::

    # content of test_module.py
    import pytest

    @pytest.mark.factory(scope="module", params=["mod1", "mod2"])
    def modarg(testcontext):
        param = testcontext.param
        print "create", param
        def fin():
            print "fin", param
        testcontext.addfinalizer(fin)
        return param

    @pytest.mark.factory(scope="function", params=[1,2])
    def otherarg(testcontext):
        return testcontext.param

    def test_0(otherarg):
        print "  test0", otherarg
    def test_1(modarg):
        print "  test1", modarg
    def test_2(otherarg, modarg):
        print "  test2", otherarg, modarg

Let's run the tests in verbose mode and with looking at the print-output::

    $ py.test -v -s
    =========================== test session starts ============================
    platform linux2 -- Python 2.7.3 -- pytest-2.3.0.dev7 -- /home/hpk/venv/1/bin/python
    cachedir: /home/hpk/tmp/doc-exec-398/.cache
    plugins: xdist, bugzilla, cache, oejskit, cli, pep8, cov
    collecting ... collected 8 items
    
    test_module.py:16: test_0[1] PASSED
    test_module.py:16: test_0[2] PASSED
    test_module.py:18: test_1[mod1] PASSED
    test_module.py:20: test_2[1-mod1] PASSED
    test_module.py:20: test_2[2-mod1] PASSED
    test_module.py:18: test_1[mod2] PASSED
    test_module.py:20: test_2[1-mod2] PASSED
    test_module.py:20: test_2[2-mod2] PASSED
    
    ========================= 8 passed in 0.03 seconds =========================
      test0 1
      test0 2
    create mod1
      test1 mod1
      test2 1 mod1
      test2 2 mod1
    fin mod1
    create mod2
      test1 mod2
      test2 1 mod2
      test2 2 mod2
    fin mod2

You can see that the parametrized module-scoped ``modarg`` resource caused
a re-ordering of test execution. The finalizer for the ``mod1`` parametrized 
resource was executed before the ``mod2`` resource was setup.

.. currentmodule:: _pytest.python
.. _`testcontext`:

``testcontext``: interacting with test context
---------------------------------------------------

The ``testcontext`` object may be received by `@pytest.mark.factory`_ or
`@pytest.mark.setup`_ marked functions.  It contains information relating
to the test context within which the function executes. Moreover, you
can call ``testcontext.addfinalizer(myfinalizer)`` in order to trigger
a call to ``myfinalizer`` after the last test in the test context has executed.
If passed to a parametrized factory ``testcontext.param`` will contain
a parameter (one value out of the ``params`` list specified with the 
`@pytest.mark.factory`_ marker).

.. autoclass:: _pytest.python.TestContext()
    :members:

.. _`@pytest.mark.parametrize`:

``@pytest.mark.parametrize``: directly parametrizing  test functions
----------------------------------------------------------------------------

.. versionadded:: 2.2

The builtin ``pytest.mark.parametrize`` decorator enables
parametrization of arguments for a test function.  Here is an example
of a test function that wants check for expected output given a certain input::

    # content of test_expectation.py
    import pytest
    @pytest.mark.parametrize(("input", "expected"), [
        ("3+5", 8),
        ("2+4", 6),
        ("6*9", 42),
    ])
    def test_eval(input, expected):
        assert eval(input) == expected

we parametrize two arguments of the test function so that the test
function is called three times.  Let's run it::

    $ py.test -q 
    collecting ... collected 11 items
    ..F........
    ================================= FAILURES =================================
    ____________________________ test_eval[6*9-42] _____________________________
    
    input = '6*9', expected = 42
    
        @pytest.mark.parametrize(("input", "expected"), [
            ("3+5", 8),
            ("2+4", 6),
            ("6*9", 42),
        ])
        def test_eval(input, expected):
    >       assert eval(input) == expected
    E       assert 54 == 42
    E        +  where 54 = eval('6*9')
    
    test_expectation.py:8: AssertionError
    1 failed, 10 passed in 0.04 seconds

As expected only one pair of input/output values fails the simple test function.

Note that there are various ways how you can mark groups of functions,
see :ref:`mark`.

Generating parameters combinations, depending on command line
----------------------------------------------------------------------------

.. regendoc:wipe

Let's say we want to execute a test with different computation
parameters and the parameter range shall be determined by a command
line argument.  Let's first write a simple (do-nothing) computation test::

    # content of test_compute.py

    def test_compute(param1):
        assert param1 < 4

Now we add a test configuration like this::

    # content of conftest.py

    def pytest_addoption(parser):
        parser.addoption("--all", action="store_true",
            help="run all combinations")

    def pytest_generate_tests(metafunc):
        if 'param1' in metafunc.funcargnames:
            if metafunc.config.option.all:
                end = 5
            else:
                end = 2
            metafunc.parametrize("param1", range(end))

This means that we only run 2 tests if we do not pass ``--all``::

    $ py.test -q test_compute.py
    collecting ... collected 2 items
    ..
    2 passed in 0.03 seconds

We run only two computations, so we see two dots.
let's run the full monty::

    $ py.test -q --all
    collecting ... collected 5 items
    ....F
    ================================= FAILURES =================================
    _____________________________ test_compute[4] ______________________________
    
    param1 = 4
    
        def test_compute(param1):
    >       assert param1 < 4
    E       assert 4 < 4
    
    test_compute.py:3: AssertionError
    1 failed, 4 passed in 0.03 seconds

As expected when running the full range of ``param1`` values
we'll get an error on the last one.