Commits

Matt Oswald committed 6b34ed4

full wiki created

Comments (0)

Files changed (7)

+== Assert and Check ==
+
+xUnit++ has a robust suite of test assert methods. While most C++ testing frameworks use preprocessor macros to implement only a few checks, xUnit++ provides two class instances with many built-in methods. And since these tests are not implemented with macros, you get much nicer feedback while editing (assuming your editor can provide such detail) and compiling.
+
+The two objects are {{{Assert}}} and {{{Check}}}. With one exception, they both offer the same check methods. The main difference between the two is that failed check methods called from the {{{Assert}}} object will immediately halt the test, while failed {{{Check}}} check methods will be reported after the test runs to completion.
+
+{{{
+#!c++
+Check.Fail();   // failure is logged, test continues
+Assert.Fail();  // stops the test here
+Check.Fail();   // test never executes this line
+}}}
+
+==== The Methods ====
+
+As stated before, {{{Assert}}} and {{{Check}}} share the same check methods, with one exception: {{{Assert}}} offers {{{Throws}}} while {{{Check}}} does not.
+
+* {{{Contains}}} asserts that a container contains some value. Overloads exist for raw strings and {{{std::string}}}.
+* {{{ContainsPred}}} is an alternative form of {{{Contains}}} which takes a predicate instead of a value.
+* {{{DoesNotContain}}}} asserts that a container does not contain some value. Overloads exist for raw strings and {{{std::string}}}.
+* {{{DoesNotContainPred}}} is an alternative form of {{{DoesNotContain}}} which takes a predicate instead of a value.
+* {{{DoesNotThrow}}} asserts that the given code does not throw any exceptions. Anything that resolves to the equivalent of {{{void (*)()}}} will work.
+* {{{Equal}}} tests object values for equality. There are overloads for raw strings, {{{std::string}}}, floating point types (with //precision//, instead of //tolerance//), and iterator ranges.
+* {{{Empty}}} will assert that the container object is empty. {{{TContainer::empty()}}} will be used if it exists, otherwise the container will be converted to a range using {{{std::begin}}} and {{{std::end}}}.
+* {{{Fail}}} will automatically fail the test.
+* {{{False}}} will fail the test if the supplied parameter resolves to {{{true}}}.
+* {{{InRange}}} checks that the given value fits within the range [{{{min}}}, {{{max}}}).
+* {{{NotEmpty}}} is the opposite of {{{Empty}}}.
+* {{{NotEqual}}} is the opposite of {{{Equal}}}.
+* {{{NotInRange}}} checks that the given value does not fit within the range [{{{min}}}, {{{max}}}).
+* {{{NotNull}}} checks that the value is not equal to {{{nullptr}}}.
+* {{{NotSame}}} verifies that the supplied objects are not the same //object instance//.
+* {{{Null}}} checks that the value is equal to {{{nullptr}}}.
+* {{{Same}}} checks that the supplied objects are the same //object instance//.
+* {{{Throws}}} is an {{{Assert}}}-only member that checks that the supplied code throws a specific exception. If it succeeds, it will return the exception to your test.
+{{{
+#!c++
+auto ex = Assert.Throws<std::exception>([]() { throws std::runtime_error(""); });
+}}}
+
+==== Printing Values ====
+
+Some methods may try to print the values of the objects using {{{to_string}}} with argument-dependent lookup (Koenig lookup). To take advantage of this, implement a {{{to_string}}} function within your object's namespace.
+
+{{{
+#!c++
+namespace NS
+{
+    class A {};
+    std::string to_string(const A &a) { return "A"; }
+}
+}}}
+
+==== Custom Messages ====
+
+If you want to add a custom message to failing tests, use the overloaded {{{operator <<}}}.
+
+{{{
+#!c++
+Assert.Fail() << "This is an example message " << some_value;
+}}}
+
+==== File and Line Info ====
+
+Normally, when a test fails the test runner will report the file and line number for the test itself. This is typically sufficient for most tests, as tests should really only assert one thing at a time. However, if you want to be specific about which check failed, each check method accepts an optional {{{xUnitpp::LineInfo}}} object. The easiest way to do this is to pass the {{{LI}}} macro as the final parameter to the test.
+
+{{{
+#!c++
+Assert.Equal(0, 1, LI) << "0 is never equal to 1!";
+}}}
+== How does xUnit++ compare to ...? ==
+
+xUnit++ is far from the only unit testing framework for C++, and is certainly not perfect. However, it stands up quite well when compared to other systems. Most systems utilize preprocessor macros to implement a small subset of checks, and most also require you to provide your own test runner.
+
+|=|= xUnit++ |= Boost UTF |= Google Test |= MSTest |= UnitTest++ |
+| | | | | | |
+|= Implementation | Static Library | Any! (See Note 1.) | User-provided (See Note 2.) | Requires Visual Studio 2012+ | Static Library |
+|= Test Implementation | Member functions | Preprocessor macros | Preprocessor macros | Static member functions | Preprocessor macros |
+|= Test Runner | Provided | User-supplied | User-supplied | Visual Studio | User-supplied |
+|= Fatal Failures | {{{Assert.Equal}}} | {{{BOOST_REQUIRE_EQUAL}}} | {{{ASSERT_EQ}}} | {{{Assert::AreEqual}}} | {{{CHECK_EQUAL}}} |
+|= Failures | {{{Check.Equal}}} | {{{BOOST_CHECK_EQUAL}}} | {{{EXPECT_EQ}}} | //n/a// | //n/a// |
+|= Warnings | //n/a// | {{{BOOST_WARN_EQUAL}}} | //n/a// | //n/a// | //n/a// |
+|= Per-Check Messages | Use an overloaded stream operator.\\{{{Assert.Equal(a, b) << "message"}}} | Append {{{_MESSAGE}}} to any check macro.\\{{{BOOST_REQUIRES_EQUAL_MESSAGE(a, b, "message")}}} | Use an overloaded stream operator.\\{{{ASSERT_EQ(a, b) << "message"}}} | Additional parameter\\{{{Assert::AreEqual(a, b, "message")}}} | //na// |
+|= Attributes | Up to 8 per test. | //n/a// | //n/a// | Unlimited per module, fixture, or method. | //n/a// |
+|= Timed Tests | Halt long-running tests. | //n/a// | //n/a// | //n/a// | Fail tests that went over time limit. |
+| | | | | | |
+
+**Note 1:** Boost has just about every implementation you could ask for. If your test library is small and simple, you can just include a single header file, or you can link it as a static library, or you can reference it as a dynamic library.
+
+**Note 2:** Google Test requires most users to create a project to build the test executable. Some projects are provided, but several are out of date.
+
+**A note on Test Runners:** With the exception of MSTest, the other test frameworks require implementing (in some fashion) {{{int main()}}}. xUnit++ prefers to instead offer standard test runners that know how to run your tests for you natively.
 [[InstallAndSetup.wiki|//Installation and Setup//]]\\
 [[Tests.wiki|//Tests//]]\\
 [[SuitesAndAttributes.wiki|//Suites and Attributes//]]\\
-[[Assertions.wiki|//The Assert methods//]]\\
+[[Assertions.wiki|//Assert and Check//]]\\
 [[RunningTests.wiki|//Running tests//]]\\
 [[Compare.wiki|//How does xUnit++ compare to ...?//]]
 
 }
 }}}
 
-Now you've got a couple of tests that are going to expose the error in the function {{{Double}}}, but boy is that annoying to have to write all those different tests that are pretty much the same thing repeated over and over. There's got to be a better way, and with xUnit++, there is:
+Now you've got a couple of tests that are going to expose the error in the function {{{Double}}}, but it's a bit annoying to have to write all those different tests that are each five lines long, four of which are boilerplate.. There's got to be a better way, and with xUnit++, there is:
 
 {{{
 #!c++
 
 A //Theory// creates a parameterized test. Each set of parameters is bound to an instance of the test and run individually. In this example, the {{{THEORY}}} macro creates a test named {{{DoubleTest}}} which takes parameters {{{(int expected, int value)}}}. The rest of the macro's parameters are {{{std::tuples}}} whos members resolve to the same type as the test parameters.
 
-If your //Theory// data requires detailed setup (for instance, if the details of the test parameters are stored in a local database or CSV file), another macro {{{DATA_THEORY}}} creates the same sort of test, only instead of individual {{{std::tuples}}}, it takes any function object (raw function, {{{std::function}}}, functor object) that returns {{{std::vector<std::tuple>>}}}:
+If your //Theory// data requires detailed setup (for instance, if the details of the test parameters are stored in a local database or CSV file), another macro {{{DATA_THEORY}}} creates the same sort of test, only instead of individual {{{std::tuples}}}, it takes any function object (raw function pointer, {{{std::function}}}, or functor object) that returns {{{std::vector<std::tuple>>}}}:
 
 {{{
 #!c++
 
 // functor
-struct
+struct SqlProvider
 {
-   std::vector<std::tuple<int, int>> operator()() const
+   SqlProvider()
    {
-      return std::vector<std::tuple<int, int>>();
+      // connect to database
+      // fill data
    }
+
+   const std::vector<std::tuple<int, int>> &operator()() const
+   {
+      return data;
+   }
+
+   std::vector<std::tuple<int, int>> data;
 } functor;
 
 // std::function
 ([]() -> std::vector<std::tuple<int, int>>
 {
    std::vector<std::tuple<int, int>> data;
-   data.push_back(std::make_tuple(2, 1));
+
+   // connect to database
+   // fill data
+
    return data;
 }))
 {

InstallAndSetup.wiki

 hg clone ssh://hg@bitbucket.org/moswald/xunit path_to/project_root/external/xUnit++
 }}}
 
-* Add the xUnit++ library to your build process (either xUnit++/xUnit++.vcxproj for Visual Studio or xUnit++/SConstruct for SCons).
-* Create a shared object library project (.dll or .so) to store your tests.
+* Add the xUnit++ library to your build process.
+** For Windows, the supported build system is a set of MSBuild/Visual Studio projects.
+** For everything else, the build system is SCons. Currently, there is only minimal support in the {{{SConstruct}}} and related files for Windows, since MSBuild and Microsoft's C++ compiler are free (as in beer).
+* Create a shared library project (.dll or .so) to store your tests. xUnit++ comes with a suite of tests for itself; you can use this as an example.
 * Modify your test project's include path to include {{{xUnit++/xUnit++}}}.
-* Modify your test project's link settings to include {{{xUnit++/bin/xUnit++/<Configuration>/<Platform>/xUnit++.lib}}}.
+* Modify your test project's link settings to include {{{xUnit++/bin/xUnit++/xUnit++[.Debug].lib}}}.
 * Optionally (but recommended), include the xUnit++.console project in your build, and modify your test project's build to include running your tests after every succesful build. xUnit++.console will return the number of failed tests, so you can use that return value to halt your build script.
 
 ==== Packaged Release ====
 
 * Download the latest release and unpackage it somewhere within your source tree, for example {{{path_to/project_root/external/xUnit++}}}.
-* Create a shared object library project (.dll or .so) to store your tests.
+* Create a shared object library project (.dll or .so) to store your tests. xUnit++ comes with a suite of tests for itself; you can use this as an example.
 * Modify your test project's include path to include the path to {{{xUnit++/include}}}.
-* Modify your test project's link setting to include {{{xUnit++/bin/<Configuration>/<Platform>/xUnit++.lib}}}.
+* Modify your test project's link setting to include {{{xUnit++/bin/xUnit++[.Debug].lib}}}.
 * Optionally (but recommended), modify your build process to include running your test project's output, failing the build if the runner returns anything other than 0. Your platform's test runner can be found in {{{xUnit++/bin/<Configuration>/<Platform>/}}}
 
 ----

RunningTests.wiki

+== Running Tests ==
+
+xUnit++ comes with a command line test runner that inspects your test library for tests and executes them. (Actually, it just causes your test library to execute itself.) By default, it will execute all tests in your test library concurrently, with no time limit applied.
+
+The options are fairly self-explanatory:
+
+|-v|Verbose mode: include successful test timing|
+|-vv|Very verbose: write test start message|
+|-l --list|Do not run tests, just list the ones that pass the filters|
+|-s --suite <SUITE>+|Suite(s) of tests to run (substring match)|
+|-n --name <TEST>+|Test(s) to run (substring match)|
+|-i --include <NAME=[VALUE]>+|Include tests with matching <name=value> attribute(s)|
+|-e --exclude <NAME=[VALUE]>+|Exclude tests with matching <name=value> attribute(s)|
+|-t --timelimit <milliseconds>|Set the default test time limit|
+|-x --xml <FILENAME>|Output Xunit-style XML file|
+|-c --concurrent <max tests>|Set maximum number of concurrent tests|
+
+Inclusive attributes are applied with an OR operation.
+Exclusive attributes are applied with an AND operation.
+When an attribute VALUE is ommitted, any test with a matching attribute name NAME is matched.
+
+===== Visual Studio Test Adapter =====
+
+You can install the xUnit++ Visual Studio Test Adapter from [[path to visual studio adapter installer]]. This will automatically discover and run your tests for you using the built-in Text Execution Window.

SuitesAndAttributes.wiki

 
 == Attributes ==
 
-Attributes are a much more fine-grained method of grouping tests. The packaged test runner can optionally select which tests to run based on key-value pairs. Every test (whether within a suite or not) can be decorated with up to eight key-value pair attributes:
+Attributes are a much more fine-grained method of grouping tests. Every test (whether within a suite or not) can be decorated with up to eight key-value pair attributes:
 
 {{{
 #!c++
     std::this_thread::sleep_for(std::chrono::seconds(10));
 }
 }}}
+
+Importantly, skipped tests will never be instantiated, so no long-running fixture or theory setup will be run.
 }
 }}}
 
-A Theory is rather more abstract. In this example, we want to assert that a fully-constructed, valid Widget will not throw exceptions when destroyed. Widget, however, has multiple methods for constructions, and the destructors for objects created through each of these need to be tested. To do this, the {{{THEORY}}} macro takes two named parameters: the test name and the parameters to be passed to the test (combined they form the function signature), and then a collection of several {{{std::tuple}}}s that make up the test data.
+A Theory is for more abstract tests. It is intended for tests that can't verify methods with single input, single output pairs, or when the test code would otherwise need to be repeated in multiple {{{FACTS}}}.
+
+In this example, we want to assert that a fully-constructed, valid Widget will not throw exceptions when destroyed. Widget, however, has multiple methods for constructions, and the destructors for objects created through each of these need to be tested. To do this, the {{{THEORY}}} macro takes two named parameters: the test name and the parameters to be passed to the test (combined they form the function signature), and then a collection of several {{{std::tuple}}}s that make up the test data.
 
 There are several forms of {{{THEORY}}} as well.
 
 
 ====== Test Timeouts ======
 
-A note on the test timeouts: this is inherently very fragile. Once a thread begins executing, there is no guarantee that the test will actually get //n// milliseconds of CPU time. It's possible that other tests, or even other system processes, will take control of the CPU for the majority of the time, leaving the watchdog timeout thread no choice but to assume the test thread failed to complete within the allotted time.
+A note on the test timeouts: this is inherently very fragile. Once a thread begins executing, there is no guarantee that the test will actually get //n// milliseconds of CPU time. It's possible that other tests or system processes will take control of the CPU for the majority of the time, leaving the watchdog timeout thread no choice but to assume the test thread failed to complete within the allotted time.