Falderal is a file format for literate test suites. It is particularly
suited for documenting programming languages (or other specifications of
ways to transform text) and testing their implementation(s) in a
language-agnostic fashion. The dumbed-down sound-bite version: "doctests
Test.Falderal is the reference implementation, in Haskell,
of tools for formatting and running tests written in Falderal.
Here's the scenario:
- You have a file format, or a language (perhaps a programming language.) Like all languages, it establishes a certain set of rules about its syntax, semantics, and so forth.
- You have one or more implementations of this language.
- You have some example programs in this language, and some expectations about what should happen when they are run (possibly including producing an error.)
- You want to be able to present those example programs in a nicely readable fashion, interleaved with some descriptive prose, perhaps ultimately formatted into a document format such as HTML. (This is certainly a reasonable way to document a language; most people are good at learning from examples.)
- You want to run those example programs to make sure they do what you expect, to find flaws that may be lurking in either those programs, or in an implementation of the language.
- You want the option of embedding those example programs and that documentation in a source code file -- perhaps a literate implementation of the very language they are testing.
If this describes you, then Falderal might help. I wrote it because I was designing yet another esoteric programming language, and while working on it I realized I was rebuilding yet another ad-hoc unit test suite, like I had done a half-dozen times before. I didn't want to keep doing this for every language I designed, and I realized that literate test suites could serve as documentation as well; the result was Falderal.
The current version of Falderal is described in the Falderal Literate Test
Format document. The current version of
Test.Falderal is 0.5 "The Loop".
Neither the file format specification, nor the
Test.Falderal API, should
be expected to be stable through the 0.x series.
Currently supported features of the framework are:
- Writing literate test suites. These may be embedded in other kinds of textual documents, such as Markdown, literate Haskell source code, and so forth.
- Formatting these test suites (i.e. converting them to other file formats, such as Markdown).
- Running these test suites. Running is accomplished by formatting the tests
to an executable format (such as a Haskell source file), running that,
and collecting the results from it. This sequence of steps is done
automatically with the
- Producing readable failure reports. Each test or group of test may be preceded by descriptive text, and this will be displayed above every failing test, along with the expected and actual output.
- Testing text-processing functionalities. A test specifies textual input to the function, and may expect a particular textual output, or that a particular error was encountered. Functionalities are abstract: each functionality defined in a Falderal file can be implemented in multiple ways. Thus the same tests can be run multiple times, once for each implementation of the functionality they test.
- Specifying that a functionality is implemented by a Haskell function of
String -> String.
- Specifying that a functionality is implemented by a shell command. The shell command can invoke arbitrary executables, allowing you to test implementations of your language written in essentially any language.
Version 0.5 "The Loop" (current released version):
- The command-line format of
falderalhas changed, for the better. The
testsubcommand no longer requires that the failure reporting style be specified; instead, it defaults to
standard, and can be changed with a command-line option. There are also command-line options for selecting the programs to run results generators, and to keep generated files around instead of cleaning them up after testing.
- The dependency on
ghcfor running Haskell results generators has been removed; these can be run by
runhaskellnow, and are run by
- Failure reporting is now consistent across languages; both Haskell and
Bourne shell results generators generate an intermediate format, which
- A new pragma
encoding:was added, so that this directive can be embedded in your Falderal document (for the benefit of your text editor) without necessarily appearing in the formatted document.
- We began giving release milestones colorful names. The naming
convention is to choose names of Chicagoland neigborhoods, suburbs,
landmarks, and institutions. Version 0.5 was named after The Loop in
recognition of its ability to shuttle test results between
falderaland the various results generators implemented in different languages. Previous versions of
Test.Falderalwere retroactively given milestone names during this release.
Version 0.4 "Blackstone Hotel":
- For ease of installation, the project is presented as a Cabal package.
- A driver executable,
falderal, is built as part of installing the Cabal package. It provides a command-line interface to formatting Falderal files and, in a limited fashion, running the tests in them.
- A shell script formatter has been written, enabling testing of shell commands. One caveat is that reporting for these tests is nowhere near as nice as for Haskell functions, but that will change in the next version.
- The Functionality-definition pragma has been implemented, making it possible to write tests in a more abstract, implementation-independent fashion.
- Falderal files written to work with Falderal 0.3 should still work with Falderal 0.4, but you are encouraged to use the Functionality-definition pragma introduced in 0.4 to make your tests more implementation-independent.
Version 0.3 "Chicago Board of Trade":
- The definition of a Falderal Literate Test Format, distinct from the
reference implementation of tools for it in Haskell (
Test.Falderal). This represented a fairly substantial departure from how previous versions of Falderal worked.
- The ability to format a Falderal file to different formats, including Markdown and Haskell.
- Running tests is now a matter of formatting a Falderal file as a Haskell
module and running the function
testModulein that module.
Version 0.2 "Dearborn Station":
- Added a test harness for Falderal itself, in the form of a simple shell
script which diffs the output of
Test.Falderal.Demoagainst a text file containing the expected content.
- Improved formatting of failure reports. Multi-line input text or expected output is always presented starting on its own line.
- Tests may be organized into groups; the entire group is preceded by some literal text, but there is no literal text between the tests in the group. When one of these tests fails, the literal text for the group is reported, along with the number of the test within the group.
- Fixed a bug where exception text which extended over multiple lines could not be expected correctly.
Version 0.1 "Haymarket Square":
- Provision of a framework for writing and running literate tests which may be embedded in literate Haskell source code.
- Testing Haskell functions of type
String -> String. A test specifies input to the function, and may expect a particular output, or that a particular exception is thrown.
- Through simple adapters, testing functions of other types such as
(Show a) => String -> a.
- Falderal started life as a Haskell-specific hack that could be embedded in a Bird-style Literate Haskell source file. I took a framework for literate tests I had already written in a project called Rho, and used it as the basis of this code.
There is also a git mirror of the repository on Github.
For Further Information
Please see the Falderal wiki on Bitbucket.