Running Tests 🔗
In Audition, the test runner is the executable program compiled with your test cases and linked against the Audition library. Audition provides its own main function so you don't have to. By implementing its own entry point, Audition can respond to program arguments, discover tests, and run tests without any manual intervention on your part.
You can review the program arguments supported by Audition by executing your test runner with the -h or --help flag. This document will discuss some of the noteworthy arguments and features of the test runner.
Test Execution Order 🔗
By default, Audition executes all test cases when you run the test runner. Suites are executed alphabetically and tests are executed in ascending order by line number (e.g. the order they are declared in the source file). This ensures execution order is deterministic.
You can execute tests in a random order by passing the --shuffle option to the test runner.
$ runner.exe --shuffle
When shuffling tests, the default test reporter will print the seed value used by the random number generator. You can pass this seed value as an argument to the --shuffle option to re-shuffle the tests exactly as they were shuffled initially. The following example executes the test runner with a shuffle seed of 1234.
$ runner.exe --shuffle=1234
Filtering Tests and Suites 🔗
You can run a specific test suite by specifying its name as a program argument. The following command will execute all tests belonging to the foo and bar suites.
$ tests.exe foo bar
Specific test cases can be executed by listing their suite name, followed by a period ., followed by the test name. The following command will execute only the bar and baz tests from the foo suite.
$ tests.exe foo.bar foo.baz
Glob Patterns 🔗
Audition lets you specify a glob pattern for the suite and test. The glob pattern syntax is documented below:
| Pattern | Description |
|---|---|
| * | Matches zero or more characters. |
| ? | Matches any single character. |
| [set] | Matches any character in the set where set is a group of characters or ranges.
|
| [!set] | Matches any character not in the set where set is a group of characters or ranges.
|
| char | Matches itself except where char is '*' or '?' or '['. |
| \char | Matches character char, including any pattern character.
|
Set ranges which can appear in the [set] and [!set] pattern syntax are written as two characters separated with a hyphen: e.g. the range 'a-z' denotes all characters between 'a' to 'z' inclusive.
The following table demonstrates various glob pattern along with the strings they match:
| Pattern Example | Matches |
|---|---|
| a*c | ac abc abbc ... |
| a?c | acc abc aXc ... |
| a[a-z]c | aac abc acc ... |
The following command will execute all tests in suites beginning with the name foo.
$ tests.exe foo*
The following command will execute all tests named bar from any suite.
$ tests.exe *.bar
Here are more complete examples:
| Pattern Example | Matches |
|---|---|
| foo.bar | foo.bar |
| foo.ba? | foo.bar, foo.baz |
| bank | bank.deposit, bank.withdrawl |
| comp*.o* | company.office, computer.on, computer.off |
Parameterized Test Patterns 🔗
You can specify which iteration of a parameterized test to run in your pattern by following the test name with a period . followed by an integer indicating the test case iteration. For example, the following command will execute iteration index 12 of the bar test that belongs to the foo suite.
$ tests.exe foo.bar.12
You can specify multiple iteration indices by using a comma as a delimiter. In the following example iterations 5, 7, and 10 of the bar test will be executed:
$ tests.exe foo.bar.5,7,10
You can specify an inclusive range of test iterations to run by providing two numbers separated by a hyphen - character. The first number is the initial test case iteration to run and the second number is the last iteration. In the following example, test iterations 9, 10, 11, 12, and 13 of the bar test are executed:
$ tests.exe foo.bar.9-13
Excluding Patterns 🔗
By default, Audition interprets test patterns as a white list. That means Audition will only execute tests that match your pattern. If no tests match your pattern, then no tests will be executed.
You can reverse this behavior by instructing Audition to interpret your test patterns as a black list by passing the -x or --exclude command-line option. This instructs Audition to only execute tests that do not match your pattern.
Debugging Patterns 🔗
You can preview which test cases will be executed by your pattern without actually executing them by passing the --which option. This option instructs Audition to print to stdout the test cases will be executed according to your patterns, but does not actually execute them. This is a quick way to learn which tests and suites match your pattern without actually running them.
Formatting Test Output 🔗
Audition supports various test reporters which define how test output is formatted. Each test reporter offers a distinct presentation style for displaying test results. You can select the test reporter on the command line using the -r <reporter> or --reporter=<reporter> option where <reporter> is an argument option that specifies the test reporter to use.
The following table lists the test reporters supported by Audition. The second column denotes the <reporter> argument option:
| Test Reporter | Argument Option |
|---|---|
| Hollywood | hollywood
|
| JUnit XML | junit
|
| Test Anything Protocol | tap
|
| Subunit | subunit
|
As an example, to use the JUnit XML test reporter you'd run your test runner like so:
$ tests.exe -r junit
or
$ tests.exe --reporter=junit
These reporters are discussed in more detail in the following subsections.
Hollywood 🔗
Hollywood is a plain text test reporter for Audition. It is also the default test reporter for Audition.
Hollywood is Audition's own custom test reporter intended for display in a terminal window. It is not intended to be parsed by external tools. For machine readable test output use the Test Anything Protocol, JUnit XML, or Subunit reporters.
Hollywood uses VT100 terminal control characters for colored terminal output. Colors can be disabled by passing the --color=never command-line option.
JUnit XML 🔗
JUnit XML is a machine readable XML-based format for reporting test results popularized by the JUnit testing framework for Java. Many popular continuous integration tools, such as Jenkins, CircleCI, and Travis CI, support JUnit XML natively.
Audition supports version 5 compatible output of the JUnit XML schema.
Test Anything Protocol 🔗
The Test Anything Protocol (TAP) is a machine readable text-based format for reporting test results. It originated in the Perl community in the early 2000s, primarily created for testing Perl applications. Its simplicity and flexibility quickly led to adoption in other programming languages and testing frameworks. Many continuous integration tools support TAP output.
Audition supports version 12, 13, and 14 compatible output of the Test Anything Protocol.
Version 13 introduced YAML blocks. These blocks allow the addition of test meta-data. The format of the YAML block is not specified by the TAP v13 protocol. Audition uses YAML blocks to emit various details about the test case.
Version 14 is largely backwards compatible with TAP version 13. The most notable change is the addition of subtests which is not a feature Audition supports.
Subunit 🔗
Subunit is a protocol designed for streaming test results, particularly useful in continuous integration and testing environments. It was developed as an alternative to formats like TAP and JUnit XML, aiming to provide a more structured and flexible way to report test outcomes.
Audition supports version 1 compatible output of the subunit protocol. This version of the protocol produces machine-readable, line-oriented text output.