Jun 242008

Every other week, someone asks how to use the new unit testing framework in GLib (released with GLib-2.16.0) and Gtk+ (to be released with the next stable). First, here is a write-up from last December that summarizes all important aspects: Test Framework Mini Tutorial.

For people building packages that use the new framework, the following new Makefile rules will be of interest:

  • make test
    Run all tests recursively from $(TEST_PROGS), abort on first error.
  • make test-report
    Run all tests recursively, continue on errors and generate test-report.xml.
  • make perf-report
    Run all tests recursively, enable performance tests (this usually takes significantly longer), continue on errors, generate perf-report.xml.
  • make full-report
    Run all tests recursively, enable performance tests, enable slow (thorough) tests, continue on errors, generate full-report.xml.
  • make check
    Run make test in addition to automake checks.

After GUADEC, we will be looking into getting build machines setup that’ll regularly build GLib, Gtk+ and friends, run the unit testing suites and provide reports online.

For those pondering to write unit tests, but too lazy to look at the tutorial:

  • Implementing a test program is very easy, the only things needed are:
      // initialize test program
      gtk_test_init (&argc, &argv);
      // hook up your test functions
      g_test_add_func ("/Simple Test Case", simple_test_case);
      // run tests from the suite
      return g_test_run();
  • In most cases, a test function can be as simple as:
      static void
      simple_test_case (void)
        // a suitable test
        g_assert (g_bit_storage (1) == 1);
        // a test with verbose error message
        g_assert_cmpint (g_bit_storage (1), ==, 1);

    Tests that abort, e.g. via g_assert() or g_error(), are registered as failing tests with the framework. Also, the gtester utility used to implement the above Makefile rules will restart a test binary after a test function failed and continue to run remaining tests if g_test_add_func() has been used multiple times.

  • Checks in tests can be written with if() and g_error() or exit(1), or simply by using variants of g_assert(). For unit tests in particular, an extended set of assertions has been added, the benefit of using these are the printouts of the involved values when an assertion doesn’t hold:
      g_assert_cmpstr   (stringa, cmpop, stringb);
      g_assert_cmpint   (int64a,  cmpop, int64b);
      g_assert_cmpuint  (uint64a, cmpop, uint64b);
      g_assert_cmphex   (uint64a, cmpop, uint64b);
      g_assert_cmpfloat (doublea, cmpop, doubleb);

    For instance:
    char *string = "foo"; g_assert_cmpstr (string, ==, "bar");
    ERROR: assertion failed (string == "bar"): ("foo" == "bar")

  • The framework makes it easy to test program output in unit tests:
      static void
      test_program_output (void)
        if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT |
            g_print ("some stdout text: somagic17\n");
            g_printerr ("some stderr text: semagic43\n");
            exit (0);
        g_test_trap_assert_stdout ("*somagic17*");
        g_test_trap_assert_stderr ("*semagic43*");
  • And it is similarly easy to test and verify intentional program abortion:
      static void
      test_fork_fail (void)
        if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDERR))
        g_test_trap_assert_stderr ("*ERROR*test_fork_fail*not*reached*");
  • The above and more tests are showcased in GLib: glib/tests/testing.c.

There’s of course lots of room left to improve GLib and Gtk+ unit tests, and also to improve the current framework. For a speculative, non-comprehensive list, here are some ideas from the unit testing section of my personal TODO:

  • Introduce 2D marker recognition for graphical unit testing of Gtk+ layouts (prototyped in Rapicorn).
  • Provide functionality to determine image similarities to allow for pixel image based unit tests (port this from Rapicorn).
  • Implement state dumps to automate result specification and verification in unit tests. (This will allow us to avoid adding lots of abusable testing hooks to our API.)
  • Integrate performance statistics (like #354457) and other related information into test reports.
  • Publically install a variant of the Makefile.decl file used in Gtk+ to implement the test framework rules and Xvfb swallowing of test programs. This is needed by other projects to run unit tests the way Gtk+ does.
  • Implement the unit test ideas that are outlined at the end of this email: Gtk+ unit tests (brainstorming).

  9 Responses to “23.06.2008 Writing Unit Tests with GLib”

  1. Not sure if you know, but Google gave a few machines and our intention is to use these as build machines. Shipping those is taking much longer than intended.. so they’re not online yet. If you want specifics, ask vuntz.

  2. One feature at least I would find useful is an option to instrument the code with a test vector and include the test coverage result in the test reports.

    From experience I know that hard facts, code coverage percentage for example, encourages developers (or testers) to write complete test cases (feature wise) and eliminate unneeded or dead code.

  3. 1) Glib testing contains a lot of dirty ocde.
    For example:
    #define XXXXXX do {xxx}while(0)
    glib’s code style is: #define XXXXXX G_STMT_END{}G_STMT_END

    May be you should consult the cunit’s code (http://cunit.sourceforge.net/).

    2)I use glib’s Testing, but documentation is missing. There was no guide for how to set up your own project to use the glib testing framework or higher level overview of setup/teardown, available asserts, running, parameters, etc.

  4. shark_yang, G_STMT_START/G_STMT_END is essentially a lengthier way to write do{…}while(0).
    As for CUnit, that has been considered but didn’t meet the requirements:

  5. Wouldn’t it be better if this blog post was converted into a live.gnome.org tutorial ?

  6. liberforce, feel free to start a unit test tutorial page on live.gnome.org or to submit it as a patch against GLib for inclusion in library.gnome.org. The main tutorial contents are here:
    And this is also related:
    We appreciate various sorts of contributions, of course also documentation.

  7. Tim Janik:

    I use glib’s main event loop and iochannel, but why glib’s main event loop and iochannel don’t support epoll/kqueue/(/dev/poll)?

    The poll/select func’s efficiency is very poor.

    The libevent(http://monkey.org/~provos/libevent/) use epoll/kqueue. Why glib not?


  8. hi Tim:

    I use glib’s testunit in visual studio 2002.But when compling the code, i find warnings.

    The code:

    g_assert_cmpint(3, ==, 2);

    The warnings:
    warning C4244: conversion from ‘__int64’ to ‘long double’, possible loss of data

    In glib’s source code(gtestutils.h):
    #define g_assert_cmpint(n1, cmp, n2) \
    do { gint64 __n1 = (n1), __n2 = (n2); \
    if (__n1 cmp __n2) ; else \
    _assertion_message_cmpnum (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
    #n1 ” ” #cmp ” ” #n2, __n1, #cmp, __n2, ‘i’); } while (0)

    May be correct:
    #define g_assert_cmpint(n1, cmp, n2) \
    do { gint64 __n1 = (n1), __n2 = (n2); \
    if (__n1 cmp __n2) ; else \
    _assertion_message_cmpnum (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
    #n1 ” ” #cmp ” ” #n2, (long double)__n1, #cmp, (long double)__n2, ‘i’); } while (0)

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>