How to use the Integration Test Framework
How to open the runner
You open the Integration Tests Runner from the menu bar Unity Test Tools/Integration Tests Runner or by Shift+Ctrl+Alt+T combination.
How does the Integration Test Runner work?
The Integration Tests are designed to run on a separate scene. You can consider the scene where you place your tests as a test suite. One test scene can contain multiple tests. You should not put your tests on your production scenes. Instead, create a separate scene for them.
A Test Object is a GameObject on the scene that has TestComponent attached to it. Everything under the Test Object in the hierarchy is considered to belong to this test. Any object that is not under a Test Object will be common for every test on the scene (it's usually environment like floor, walls etc.). You shouldn't care about creating the Test Object manually. Everything is done through the Test Runner. Tests can have hierarchical structure. One Test Object can be a direct child of another Test Object. A Test Object with any Test Object under it becomes a group and it's not executed as a test.
Only one test can be active at any time. When you select a test, all other tests will become disabled, so you can work only with one test at a time.
This scene is an exemplary scene shipped with the framework. If you look at the hierarchy view you will notice 7 Test Objects and 2 groups. First group has all the tests that are passing in this example. The second group consists of tests that are failing. The remaining, non grouped test, is set to be ignored and not executed. Each of them has an icon showing the result from last time the test was run. In this example the Test3 - Failure is the active selection, so all other tests are disabled. This test contains only one GameObject, the Sphere. If you take a look at the scene now you will notice two additional cubes: red and green. Those cubes are not places under any Test Object, so they will be shared by any test in the scene. In the hierarchy window you will find them as_CubeTriggerSuccess_ and CubeTriggerFailure. Additionally, you can also see a TestRunner object. This object is responsible for driving the test run once execution starts. It will be added automatically when you add the first test on a scene.
When you run the tests the following steps are performed by the runner:
- Play mode is enabled
- The first test gets enabled (becomes active)
- Wait until the test has finished (or a timeout has occurred)
- The current active test gets disabled
- If there are more tests in the queue, enable next test and go to step 3
- Report results and finish test run
Creating Integration Tests from code (Dynamic Integration Tests)
It's possible to mark a MonoBehaviour with an attribute that make it loaded by the test runner. On your class that derives from MonoBehaviour, put the [IntegrationTest.DynamicTest (string)] attribute and pass the name of the test scene as the argument. The runner will show the test on the test list and you will be able to run it. The test will contain the script that was marked with the attribute. Additionally, you can mark the test with attribute change test's parameters:
If the currently opened scene matches the string provided in the attribute constructor, the test runner will try to create the test on scene. It will be, however, marked with HideFlags.DontSave so you dynamic tests won't be saved on the scene. If you change any of the attributes related to the tests, remember to refresh the object on the scene (in the inspector) or simply delete it from the scene to get it rebuilt.
How to control a test flow (How to start and finish a test)
A test starts once the Test Object gets enabled. The Test can finish its run in multiple ways:
- Function Testing.Pass() is called. This will successfully finish the test.
- Function Testing.Fail() is called. This will fail the test.
- Execution times out. This can happen when none of the above functions is called within a specified period of time (you can set the timeout value per test).
- An unhandled exception is thrown.
- An expected exception is thrown (Expect exception must be checked)
- Every Assertion Component on objects under tests is checked at least once ( the "Succeed after all assertions are executed" option needs to be selected)
Note that you can use a set of pre-made assets for controlling test flow. They are placed in the IntegrationTestsFramework / TestingAssets folder.
Integration Test Runner
The Integration Test Runner window functionality:
- Run all tests in the scene (excluding ignored tests)
- Run selected test(s).
- Create a new test - creates new test object on the scene
Options - options for working with Integration Tests
- Add GameObjects under selected test - when selected, when you add a new object to the scene it will be automatically placed under the test GameObject instead of the hierarchy root
- Block UI when running - when selected, a dialog will appear during test execution
Test Filter - will filter out tests where name does not not contain the string
- Show succeeded - show tests that succeeded
- Show failed - show tests that failed
- Show ignored - show tests that are ignored
- Show not runed - show tests that hasn't been run
Test list - list of all tests available in the scene
- Test log and exception messages
- Test name - name of the test
- Included platform - on what platform the test should included
- Timeout - number of second after the test will timeout
- Ignored - ignore the test when running all tests
- Succeed after all assertions are executed - select if the test should finish after all assertions from Game Object in the test got checked at least once.
- Expect exception - the test will not fail if an exception if thrown.
- Expected exception list - a list of exception that will not fail the test when thrown. Separate the exceptions with comma (","). Derived types from types on the list will also be considered as expected. If the list is empty, any exception type will be accepted.
- Succeed when exception is thrown - the test will succeed when one of the excepted exceptions is thrown.
- Selected test
- Result icon
- Test Runner object
- Common objects - objects that are not under a test node will be active in every test
Creating simple tests
These steps will walk you through writing two simple tests:
- Create a new scene that will contain the tests
- Open the Integration Tests Runner Window. (in menu bar, Unity Test Tools/Integration Tests Runner, or ctrl+alt+shift+t)
- Click the "plus" button to add new test and rename it to Test Pass. Notice that a TestRunner object should be automatically added.
- Select the test
- Add a Cube to the scene. If the Add new GameObjects under selected test option is checked the Cube will automatically be placed under the test node. Otherwise move it there manually.
The hierarchy should look like this:
- Add a Rigidbody component to the cube. You can run the test and verify the cube is falling down. To run the test right click on it and select Run (you can also use ctrl+t combination if the test is selected). The test will timeout after 5 seconds (by default) because Testing.Pass() was never called.
- Add another Cube below the first one that will call Testing.Pass() on collision. You can use the test assets provided with the framework. Find CubeTriggerSuccess _prefab under _Assets\UnityTestFramework\IntegrationTestsFramework and put it on the scene below the first Cube so it will fall on it. If you look at the prefab you will see that it has a script attached. The script is responsible for calling Testing.Pass() _in OnCollisionEnter function._
- Rerun the test. The test should now pass.
- Now, add another test, and an empty GameObject. Attach CallTesting script from the Testing Assets and make it fail after 1 second.
- Try running both tests now. The result should be similar to this one:
A few testing assets are provided with the framework:
- CallTesting.cs script - allows you to call Testing.Pass() or Testing.Fail() automatically from selected methods or after a desired amount of time or number of frames.
This configuration will make the test succeed if the object enters a collision (OnCollisionEnter) or gets destroyed (OnDestroy). Otherwise, the test will fail after 3 seconds.
- CubeTriggerSuccess, CubeTriggerFailure - cube prefabs that will succeed/fail on OnTriggerEvent.
Running tests on platforms
To automatically run build and run a test scene on a target platform use the Platform Runner (Unity Test Tools/Integration Tests/Platform Runner/Run on platform, Ctrl+Shift+R).
- List of scenes to build - Multiselectable list of scene to build. Selected scenes need to be test scenes in order to run properly
- Build tests for - A list of target platform player will be built for. The list correspond to BuildTarget.
- Make default target platform - will make currently selected platform the default one
- Result target directory - a target directory where the results should be saved. The path should be valid and exist. It's possible to specify network location, i.e. \network-drive\results
- Send results to editor - the runner will communicate with the editor via TCP protocol in order to send the resuts. The target device needs to be connected to the same network.
- Network interface - the interface the runner should use in order to connect with the editor. Useful when the target device is connected to a different subnetwork.
- Network port - which network port should be used
- Build and run tests - will do what it says
Additionally, Unity Test Tools/Platform Runner/Run current scene (Ctrl+Alt+Shift+R) will run currently open scene in the default platform.
When running tests from the editor, the results are available in the Test Runner.
You build and run test using the Player Runner, after the tests are executed you will see a result report on the screen of the targeted device.
The player runner will now send the result back to the editor via TCP if possible. The results can be saved under specified path. This will only work on platforms that support TCP communication.
If you run test in batch mode, a file with results will be generated and the application will close itself. The result file is an XML file and it's located in project's root folder, unless a parameter is specified. The results are in nUnit result format. The schema for the file can be found here: http://www.nunit.org/docs/2.6.2/files/Results.xsd.
For platform that don't support file system, you can parse platform's log output for results.
Headless running (batch mode)
It is possible to execute integration tests from command line. In order to do that, run Unity in batch mode and execute UnityTest.Batch.RunIntegrationTests method on start.
- testscenes - Comma-separated list of scene names to be run. If the parameter is not provided, the runner will automatically pick all scene matching following pattern: "*Test[s].unity"
- targetPlatform - Platform you want to run the tests on. Value has to correspond BuildTarget enum. If not present, the tests will be run in the editor.
- resultsFileDirectory - A path to a folder where the results should be placed. If not specified, the results will be placed in project's root folder.
Unity.exe -batchmode -projectPath PATH_TO_YOUR_PROJECT -executeMethod UnityTest.Batch.RunIntegrationTests -testscenes=TestScene1,TestScene2 -targetPlatform=StandaloneWindows -resultsFileDirectory=C:\temp\
This will run tests from scenes TestScene1 and TestScene2 in Standalone player for Windows and generate results in C:\temp\ folder.
The editor will exit with a return code according to the result of the run:
0 - Run succeeded, no failures occurred
2 - Run succeeded, some tests failed
3 - Run failure (other failure)
#####TIP: On Windows, for batch mode, run Unity with followind command in order to receive the return code properly:
start /WAIT Unity.exe ARGUMENT_LIST
It will allow you to get the return code from ERRORLEVEL variable.