23.06.2008 Writing Unit Tests with GLib

Tim Janik

Projects

Every other week, someone asks how to use the new unit testing framework in GLib (released with GLib-2.16.0) and Gtk+ (to be released with the next stable). First, here is a write-up from last December that summarizes all important aspects: Test Framework Mini Tutorial.

For people building packages that use the new framework, the following new Makefile rules will be of interest:

make test
Run all tests recursively from $(TEST_PROGS), abort on first error.
make test-report
Run all tests recursively, continue on errors and generate test-report.xml.
make perf-report
Run all tests recursively, enable performance tests (this usually takes significantly longer), continue on errors, generate perf-report.xml.
make full-report
Run all tests recursively, enable performance tests, enable slow (thorough) tests, continue on errors, generate full-report.xml.
make check
Run make test in addition to automake checks.

After GUADEC, we will be looking into getting build machines setup that’ll regularly build GLib, Gtk+ and friends, run the unit testing suites and provide reports online.

For those pondering to write unit tests, but too lazy to look at the tutorial:

Implementing a test program is very easy, the only things needed are:

// initialize test program
gtk_test_init (&argc, &argv);
// hook up your test functions
g_test_add_func ("/Simple Test Case", simple_test_case);
// run tests from the suite
return g_test_run();

In most cases, a test function can be as simple as:
```
static void
simple_test_case (void)
{
  // a suitable test
  g_assert (g_bit_storage (1) == 1);
  // a test with verbose error message
  g_assert_cmpint (g_bit_storage (1), ==, 1);
}
```
Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.

Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:

g_assert_cmpstr   (stringa, cmpop, stringb);
g_assert_cmpint   (int64a,  cmpop, int64b);
g_assert_cmpuint  (uint64a, cmpop, uint64b);
g_assert_cmphex   (uint64a, cmpop, uint64b);
g_assert_cmpfloat (doublea, cmpop, doubleb);

For instance:

char *string = "foo"; g_assert_cmpstr (string, ==, "bar");

yields:

ERROR: assertion failed (string == "bar"): ("foo" == "bar")

The framework makes it easy to test program output in unit tests:

static void
test_program_output (void)
{
  if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT |
			   G_TEST_TRAP_SILENCE_STDERR))
    {
      g_print ("some stdout text: somagic17\n");
      g_printerr ("some stderr text: semagic43\n");
      exit (0);
    }
  g_test_trap_assert_passed();
  g_test_trap_assert_stdout ("*somagic17*");
  g_test_trap_assert_stderr ("*semagic43*");
}

And it is similarly easy to test and verify intentional program abortion:

static void
test_fork_fail (void)
{
  if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR))
    {
      g_assert_not_reached();
    }
  g_test_trap_assert_failed();
  g_test_trap_assert_stderr ("*ERROR*test_fork_fail*not*reached*");
}

The above and more tests are showcased in GLib: glib/tests/testing.c.

There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:

Introduce 2D marker recognition for graphical unit testing of Gtk+ layouts (prototyped in Rapicorn).
Provide functionality to determine image similarities to allow for pixel image based unit tests (port this from Rapicorn).
Implement state dumps to automate result specification and verification in unit tests. (This will allow us to avoid adding lots of abusable testing hooks to our API.)
Integrate performance statistics (like #354457) and other related information into test reports.
Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.
Implement the unit test ideas that are outlined at the end of this email: Gtk+ unit tests (brainstorming).

Post comment via email