Test reporters

Test reporters are responsible for the output from a test applicaton. It is up to the reporter to decide the format of the output. This could be xml, json or even messages to some other application, e.g. a GUI application for running tests. Currently there is a text reporter implemented.

Test reporter may be loaded into the test runner either at construction or by calling the test_runner::use_reporter() method.

Don’t let more than one test runner reference an instance of a test reporter at a time. If several test runners use the same reporter the output will most likely be interleaved in an unfortunate way.

The test_reporter interface

It is a fairly easy task to implement new test reporters. Each test reporter must be derived from the test_reporter base class.

class test_reporter

test_reporter()

~test_reporter()

virtual ~test_reporter()

prepare_run()

virtual void prepare_run(const run_descriptor &d)

This function is called once before any tests are run.

prepare_suite()

virtual void prepare_suite(long suiteId);

This function is called once before any tests in a given suite are run. The suiteId argument is a generated id that uniquely identifies the test suite.

prepare_test()

virtual void prepare_test(long testId)

This function is called before each test. The testId argument is a generated id that uniquely identifies the test.

complete_test()

virtual void complete_test(const run_result &r)

This function is called after each test.

complete_suite()

virtual void complete_suite(long suiteId)

This function is called after the last test in the suite has been run.

complete_run()

virtual void complete_run()

This function is called when all tests have been run.

run_error()

virtual void run_error(const run_result &r)

This function is called if some kind of error was detected.

It is up to the reporter to decide what to do on each function call. The run_error() method will be called if some error is detected during the execution. E.g it may be due to unexpected exceptions during setup or teardown. However, if any of the prepare_*() methods have been called, the corresponding complete_*() method is guaranteed to be called after a run_error(). Hence, suppose that an error occures after the first prepare_suite() method has been called. Then the call sequence will be

prepare_run(...)
prepare_suite(...)
run_error(...)
complete_suite(...)
// ...
// more calls for other suites
// ...
complete_run(...)

The text_reporter

The default reporter is the text_reporter. It will print out the result of a test run to an ostream, the default is cout. To use another stream, e.g. a file stream, construct a new text_reporter with the stream as an argument and load it into the test runner.

class text_reporter

text_reporter()

text_reporter(std::ostream &output = std::cout, mode m = DETAILED)

Constructs a text reporter that will output the result to stream output using mode m. See the text_reporter::mode for more details.

mode Mode

Gets or sets the output mode of the reporter.

type mode

The mode enumeration type is used in order to set how detailed the output from the reporter is to be. Possible values are

QUIET

No output.

DETAILED

Maximum output. This mode will print the name of each suite, followed by the description of each test. It will indicate which tests that fails and which ones that passes. This mode will also print a suite summary and a total summary. Fail and error reports will be printed after information about all suites.

SUITE_SUMMARY

This mode will print the description of each suite followed by a summary how many tests that were run, how many that passed, and how many that failed in the suite.

TOTAL_SUMMARY

This mode will only print a summary indicating how many suites that were run, how many tests that were run, and how many of them that passed and failed.

// Create a text_reporter that outputs a suite summaries to a file.
std::fstream MyFile("my_report.txt");
text_reporter MyReporter(MyFile, text_reporter::SUITE_SUMMARY);

// If test runner has not been created yet...
test_runner Tester(MyReporter);

// or if the test runner is already defined
Tester.use_reporter(MyReporter);

The multi_reporter

The multi_reporter is simply a relay for calls to test_reporter methods. It will make each call to a list of test reporters. The order in which the reporters are called is the same as the order in which they were added to the multi_reporter.

class multi_reporter

multi_reporter()

add()

void add(test_reporter *)

Add a new test_reporter to the list of test reporters to call. The multi_reporter will grab ownership of the added reporter.

// Write test results to two different reporters, a detailed
// report to a file, and a summary to the console.

std::fstream MyFile("my_report.txt");

multi_reporter reporter;
reporter.add(new text_reporter(MyFile, text_reporter::DETAILED));
reporter.add(new text_reporter(cout, text_reporter::SUITE_SUMMARY));

The run_descriptor class

The run_descriptor class is a data structure that describes the test run to be performed. It contains a list of instances of the suite_descriptor class. Each suite_descriptor contains a list of test_descriptor instances. Hence a run descriptor describes a test hierarchy.

Suppress usage of << operator

The verify function and different constraints will use the << operator in order to print values of arguments in test reports if some test fails. If the operator is not defined for some class or type this will result in a compile time error. The obvious, and a bit clumsy way around this, is to define the << operator for the type. This may be tedious to do in some meaningful way. There is another way however, by specializing the unittest_traits class for the type.

template<>
class unittest_traits<some_non_printable_class> {
public:
  typedef some_non_printable_class   value_type;
  typedef not_printable_tag          print_category;
};

If a type has the not_printable_tag set, the type will not be output via the << operator, instead the string “not printable” will be used.