1. Alastair Pharo
  2. pure-puritan

Overview

HTTPS SSH

Puritan

Puritan is a library for writing unit tests in Pure.

At the moment it is still quite rudimentary -- there is no convenient way to get feedback while tests are running, and when there is a failed test, the feedback isn't that easy to read. Still, it's good enough that I am now using it myself in general when writing Pure code, so you may want to use it too.

Puritan's syntax is modelled off Ruby's RSpec, but is clearly a lot less feature-rich at the moment.

Basic setup

Puritan is written entirely in Pure, so all that is required is for you to put the source code somewhere where it will be in Pure's include path when run. Either you could put it with other pure libs (/usr/lib/pure on my system); or you could put it somewhere else and run pure with -I /folder/containing/puritan, or you can specify the exact location of the file when using it (see below).

If you do want to put it with the other pure libraries, it is sufficient to copy just puritan.pure and puritan directory.

Using Puritan

You need to tell Pure to include Puritan in the file(s) where you want to write your tests:

using puritan;

Or, if Puritan is not in your PURE_INCLUDE path, you can specify the full path to the .pure file:

using "/path/to/puritan.pure";

Optionally, you can also import the puritan namespace as well, which makes the tests look nicer (and is assumed in all of the examples below), but might cause a namespace clash if you have a function with the same name as one defined by Puritan:

using puritan; // Or the longer version, as above
using namespace puritan;

Writing and running tests

The basic building block of a unit test is the infix should operator. The idea is that you can state (assert) what you expect the expression to produce. A simple example is the following:

(1+1) should return 2;

Each test takes as its left operand an expression, and as its right an assertion. In the above example, we are using the return assertion, which checks that the left operand, when evaluated in the same (in the === sense) as the right operand. More on assertions later.

Return values

When a should clause succeeds, it will return 1 (i.e. true) to whatever called it. If it fails however, then an exception will be thrown. The point of should is to always reliably do one of these two things, and to give descriptive feedback when things aren't as expected via the exception thrown. For instance, typing:

(1+1) should return 3;

into a Pure interactive session (assuming Puritan was loaded) will cause you to see something like:

<stdin>, line 2: unhandled exception 'puritan::assert_failed {puritan::test=>(1+1) puritan::should puritan::return 3,puritan::expected=>3,puritan::got=>2,puritan::reason=>puritan::test_returned_false}' while evaluating '(1+1) puritan::should puritan::return 3'

This isn't very nice to read. In the near-future, Puritan will introduce some way of pretty-printing this sort of thing. In the meantime though, you can see that the exception consists of a puritan::assert_failed header, followed by a symbolic vector of hash pairs (i.e. a record), containing the details of how the test failed. The details returned vary depending on the assertion, but they should always include at least the test itself (under the puritan::test key) and a code indicating the reason for the test failing (under puritan::reason). The three basic reasons that a test can fail are:

  1. The assertion (in this case return) returned false (puritan::test_returned_false). This happens when the assertion was not true in some straight-forward manner.
  2. An exception was thrown in the course of testing the assertion (puritan::execption_asserting). In this case, you will be informed of what the exception was under the puritan::exception key.
  3. Pure's term-rewriting engine failed to reduce the assertion (puritan::cannot_reduce). In this case, under puritan::expr you will find the unreducable expression.

Where to write tests?

At the moment, I generally just write my tests in the root context of a pure file. This means that they will be evaluated when that file is run, and any assert_failed exceptions will be dumped to stderr. This means that you need to situate them in your code so that they don't get evaluated until after all relevant rewriting rules have been evaluated. Note also that this means you don't get any feedback for tests that succeed, because the 1 returned by a successful test won't be printed.

The question then is which file? This is pretty much up to you. For example, say we have a file f.pure, containing the following:

f x = x^2 + x + 2;

We could write our tests straight into this file, so that (with the using lines added) the full definition becomes:

using puritan;
using namespace puritan;

f x = x^2 + x + 2;

f 1 should return (1^2 + 1 + 2);
f 2 should return (2^2 + 2 + 2);

Running this now defines our function and tests it in one! However, you will probably eventually want to prevent your tests from being run every time you run the file. For this, you could either make a specific f_tests.pure file, that looked like:

using puritan;
using namespace puritan;
using f;

f 1 should return (1^2 + 1 + 2);
f 2 should return (2^2 + 2 + 2);

Or you could wrap the test-related parts of f.pure in conditional compilation pragmas (see Conditional Compilation in the Pure manual). For this, you will want to designate a compilation option, say test, and then set out f.pure as follows:

// This says that if you don't specify otherwise, tests are disabled
#! --ifndef test
#! --disable test
#! --endif

// Only include Puritan when tests are enabled
#! --if test
using puritan;
using namespace puritan;
#! --endif

// Define f as before
f x = x^2 + x + 2;

// Now, wrap any tests as follows
#! --if test
f 1 should return (1^2 + 1 + 2);
f 2 should return (2^2 + 2 + 2);
#! --endif

Now, we can run these tests by invoking pure on the file with the --enable test option:

pure --enable test f.pure

I find that this last setup has a nice flow, but its up to you whether you think its worth the extra noise to have your tests in the same place as your code, or if you'd rather put them in a separate file.

Assertions

The thing to the right of each should expression needs to be an assertion. Puritan has a few different assertions bundled in, and it's pretty easy to write your own as well. should passes both the assertion and the expression to run the assertion against in quoted form to the puritan::assert function, and expects it to do one of four things:

  1. return a boolean, true indicates that the assertion was successful, false indicates that it failed trivially;
  2. throw an assert_failed exception, including a record containing information about how the assertion failed;
  3. throw any other kind of exception, indicating that the exception itself is what prevented the assertion from succeeding; or
  4. return something else. This is regarded by should as a failure to reduce successfully.

Writing your own assertions

Puritan defines an interface (called puritan::assertion) for assertions. Hence, each assertion requires a rule of the form:

puritan::assert <assertion> <any expression> = ...;

Anything that matches the above rule is considered a puritan::assertion under Pure's interface rules.

Because both parts are automatically quoted, they can take any form whatsoever. The general convention is for assertions to be written as plain Pure function/argument lists. This has the advantage of reading nicely when placed after should because of Pure's precedence rules.

As a simple example, we can write an assertion for testing that things equal one:

puritan::assert eq1 x = (eval x) === 1;

Note that eval is used because x is passed to puritan::assert in quoted form. If we don't use eval, then only the literal value 1 would succeed.

Now, we can use this assertion to test things:

using namespace puritan;

1 should eq1;
(2-1) should eq1;
(x === x) should eq1;
2 should eq1; // fails

The last expression will fail, throwing an exception like the following:

puritan::assert_failed {puritan::test=>2 puritan::should eq1,puritan::reason=>puritan::test_returned_false}

Of course, eq1 is really only of limited use as-is. We can therefore make a more general eq <something> assertion, by including an argument, using Pure's standard pattern-matching facilities:

puritan::assert (eq x) y = (eval x) === (eval y);

Now, we can check for equality of any two things:

x should eq x;
eval "y" should eq y;
(a+b) should eq (a+b);
(b+a) should eq (a+b); // fails

When a test fails, we generally want more information than just puritan::test_returned_false. For this purpose, assertions can throw puritan::assert_failed exceptions instead of just returning false. For instance, if we were concerned with checking the equality of lists, we might want to know which elements were different. To do this, we could write:

// The quoted forms aren't very handy here, so evaluate and use an auxilliary function
puritan::assert (eq x) y = assert_eq (eval x) (eval y);

assert_eq x::list y::list = all_equal ||
                            throw (puritan::assert_failed {'puritan::reason => list_values_differ,
                                                           'indices => diffs}) when
                              equalities = zipwith (===) x y;
                              all_equal  = all (===1) equalities;
                              diffs      = filter (\i -> ~equalities!i) (0..(#equalities - 1));
                            end;

// For anything that isn't a list, just check equality like before.
assert_eq x y = x === y;

Now, if we try:

[1,2,3] puritan::should eq [3,2,1];

We will get an exception informing us of the offending indices:

<stdin>, line 12: unhandled exception 'puritan::assert_failed {puritan::test=>[3,2,1] puritan::should eq [1,2,3],puritan::reason=>puritan::test_returned_false,indices=>[0,2]}' while evaluating '[3,2,1] puritan::should eq [1,2,3]'

Bundled assertions

In what follows, a brief list of the assertions bundled with Puritan is given. All of these assertions are parameterised (like eq, above), so the documentation refers to parameters, and also to the expression that the assertion is to be made against.

puritan::be

The puritan::be assertion is for making basic assertions about an expression. puritan::be leaves the expression it receives unevaluated, so it is useful where you want to check that an expression itself (rather than the result of evaluating it) is correct in some way. The parameter can be any expression, but if it is an unsaturated function, then puritan::be checks that calling this function with the expression returns true. For anything else (including a quoted unsaturated function), puritan::be checks for === equality of the result of evaluating the parameter and the expression.

Some examples of using puritan::be without a function are:

1 should be 1; // Returns true, because evaluating the right operand
               // does nothing.
1 should be '1; // Also returns true
1 should be (0+1); // Returns true
1 should be '(0+1); // Fails

The last example fails because 1 is not the same as the expression 0+1 (but it is the same as the result of evaluating the expression). Some examples of using puritan::be with a function are:

// Some test functions
test1 x = x < 1;
test2 x = eval x < 1;

0.5 should be test1; // Succeeds
(0.1+0.2) should be test1; // Fails -- the expression will not be evaluated.
(0.1+0.2) should be test2; // Succeeds -- the test does an eval.

Note that because this assertion behaves differently with functions, if you need to check that a function is equal to something, you need to make sure to quote the parameter to puritan::be, like so:

f should be 'f;

puritan::return

This is quite similar to puritan::be, but checks compares the result of evaluating the expression to the assertion's parameter, instead of the unevaluated expression. As with puritan::be, the parameter can either be an unsaturated function, or an expression. Examples:

(1+1) should return 2;
(1+2+a) should return (3+a);

// A test function
test3 x = x < 1;

(0.1+0.2) should return test3; // Succeeds because the expression is
                               // evaluated before being passed to test3.

(\x -> '(x+x)) 1 should return '(1+1); // If the function returns an
                                       // expression, you can check
                                       // it by quoting the parameter.

(\x -> '(x+x)) 1 should return (1+1); // Fails

not

Puritan reuses the bitwise not operator from the Pure core library (see Arithmetic) for assertion negation. It takes a single parameter, which should be another assertion, to be negated. Some examples:

1 should not be 2;
puts "hello" should not output "hi";
throw "my exception" should not throw "your exception";

puritan::output

puritan::output can be used to make assertions about what a function or expression causes to be written into a file or data stream (probably most useful with stdout/stderr). It does this by redirecting output to the given file descriptor. This means that in the course of running the test, the output will not actually go where you say it will, because it will be intercepted.

The puritan::output test has several forms:

  • x should output a to b states that evaluating x will cause something to be output to the file stream b, depending on what a is.
    • If a is a function, then it will be passed a file descriptor containing the output that was sent to b when x was evaluated.
    • If a is an expression of the form d exactly, then the full output from evaluating x will be compared as a string to d.
    • Otherwise, a is treated as a regular expression, and the ; see Regex Matching in the Pure manual.
  • x should output a is short-hand for x should output a to ::stdout.

Some examples:

fputs "test" ::stderr should output "t" to ::stderr;
fputs "test" ::stderr should output "test" exactly to ::stderr;
puts "test" should output "test";
puts "test" should output (\f -> fget f === "test\n");

throw

The throw assertion reuses the bare throw keyword from core Pure. It can be used to make assertions about whether or not evaluating something is expected to produce an exception.

As with other assertions, it takes a parameter which can either be a function or any other expression. If it is a function, then the function will be passed any exception that is thrown, and will be expected to return 1 if the exception is as it should be. Otherwise, the exception will be compared to the parameter using (===), except in the case where the exception is in the form of an unevaluated function (e.g. assert_failed {}). In this case, only the head exception will be compared with the parameter.

This assertion can be usefully combined with the not assertion to make statements about which exceptions expressions can and cannot throw.

Examples:

(throw blah1) should throw blah1;
(blah1 should blah2) should throw assert_failed;
(1 should throw failed_cond) should not throw failed_cond;

Domains

Often when you are performing a test, you may want to check that the result of some function call conforms to some sort of set of properties. In Haskell, QuickCheck is apparently useful for this purpose, but it depends on Haskell's somewhat sophisticated polymorphic type system.

In Pure, we could do something similar, but we would need to define a new type for every single class of value that we want to check (e.g. a type for 5 by 5 real-valued matrices). This seemed a bit cumbersome, so Puritan includes an interface for a thing called a domain. Simply put, anything can be a domain, provided we can:

  1. determine whether something is in the domain or not; and
  2. give some examples of things that belong to the domain.

To this end, Puritan provides the puritan::domain interface:

namespace puritan;

// ... snip ...

interface domain with
  from d::domain;
  in d::domain x;
end;

That is, a puritan::domain is something that has to two corresponding term rewriting rules:

  1. puritan::from takes a puritan::domain and is expected to return a lazy list of example elements belonging to that domain.
  2. puritan::in takes a domain and an expression (x), and should return true or false, depending on whether that expression is a member of the domain or not.

Testing domain membership with assertions

The key idea is that the puritan::be and puritan::return assertions allow you to pass in unsaturated function. This means that you can pass puritan::in with a domain to puritan::should in order to check whether an expression (in the case of puritan::be) or the result of evaluating an expression (in the case of puritan::return) will belong to a domain. Some examples are:

// Types
1 should be in int;
1L should be in integer;

// Vectors / matrices
{1, 1.2} should be in (int * real);
{1, 2, 3} should be in (real^3);
{1.0, 2.0, 3;
 4.0, 5.0, 6} should be in (number^(3,2));

// Sets / lists
1 should be in [1,2,3];
1 should be in (1..10);
(1,2) should be in { x,y | x = 1..10; y = 1..10; x < y };

Domains vs. types

The key distinction between a puritan::domain and a type is that domains can be any expression, not just a keyword. However, a type can work perfectly well as a domain. For instance, domain rules for ::int are as follows:

from ::int = map eval $ (repeat.quote) random;
in ::int x = intp x;

This uses random to generate random numbers as example int values (with some extra stuff to make this into a lazy list), and uses the intp type test to check for membership. A fair number of built-in types from Pure are supported in this way, and it's fairly easy to add more.

Anything that can be pattern-matched in Pure, according to the interface, can be a domain. For instance, a pattern exists for the expression x::domain ^ n::int, meaning that expressions such as int^3 are domains too (the domain of integer vectors of length three).

Sets and lists

Both sets, matrices and lists can be used (interchangably) as domains for enumerating some discrete set of possible values. For instance, [1,"x",a] is a domain of three values. puritan::from will return items from this list at random; puritan::in will search for the given value in the list.

This means that you can straight-forwardly make domains from list or matrix comprehensions, etc.

Sub domains

The sub domain system allows you to make a domain as a combination of some other domains, using the puritan::subdomains rewriting rule. For example, to make ::integer a domain that includes machine ints and bigints, you could write:

subdomains ::integer = [::int, ::bigint];

That's all. puritan::in and puritan::from will now work automatically for ::integer; the latter will draw values from the sub domains at random.

Using puritan::from to generate tests

The second function of domains is to provide randomised values to use in testing your functions. The usual way to do this is to use the take function with an array comprehension. For instance:

all id [ (x+x) should return (2*x) | x = take 100 $ from ::integer ];
all id [ sqrt x should return in real | x = take 100 $ from real; x > 0 ];

The filter in the second example means however that you won't be sure of just how many tests will be performed. The all id part is there to force the entire list to evaluate, as otherwise Pure will be lazy and return a thunk (because puritan::from returns a lazy list). Alternatively you could use the list function, or vector with matrix comprehensions:

{ (x+x) should return (2*x) | x = vector $ take 100 $ from ::integer };
{ sqrt x should return in real | x = vector $ take 100 $ from real; x > 0 };

More info

Clearly an exhaustive list of implemented domains would be helpful, and should be forthcoming. In the meantime you can browse the source in the puritan/domains folder to see the full list of domains that are already implemented.

Testing Puritan

Puritan magically tests itself using itself! Tests are written in the same files as the source code, and are triggered by the puritan::test compiler directive, in the way described in the Where to write tests? section.

You can run the Puritan tests like so:

pure --enable puritan::test puritan.pure

If you don't see any output, then things are (hopefully) working.

More information

You can browse the source for more examples of tests. If you have any queries or suggestions, you can either make them on the Pure Mailing List, or you can communicate via the "Issues" section on Bitbucket.

Alternatives

Check out the Pure's own QuickCheck library for a library that offers some overlapping functionality with this one.

License

Copyright (c) 2014, Alastair Pharo All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.