C++ unit testing with Qt Test – part 2 – advanced testing

This tutorial explores more advanced topics in C++ unit testing with Qt Test. A working example is discussed and analysed in detail. Full qmake project and C++ source code are provided.

More C++ unit testing with Qt Test

In this tutorial I am going to introduce more advanced features of Qt Test, the Qt framework for C++ unit testing. In particular I will explain how to handle a project with multiple unit tests and how to implement data driven testing. I will also give examples of more testing macros and I will show you the integration offered by Qt Creator.

This is the second post of a series dedicated to Qt Test. The posts of this series are:

I recommend to read the previous posts of the series, in particular the first part of C++ unit testing with Qt Test, to fully understand new concepts I will introduce here.

Creating a better project

Last time I showed how to create an unit test project using the “Auto Test Project” template. Another (slightly more advanced) option to do the same is the “Qt Unit Test” template:

This wizard will allow you to chose which Qt modules you want to include in the project and will offer more options in the Details section.

As seen in the first tutorial, the Qt Test philosophy is that every test case is an independent executable. In real projects you usually have hundreds or even thousands of different unit tests and running them all manually is definitely not an option. The best way to run and manage them is creating a “parent” project. The right choice in this case is the “Subdirs Project” template, which is listed in the “Other Project” group of the “New Project” dialog.

After creating the project you will get back to the templates dialog to create a first project to include. You can cancel that and proceed to add your existing projects. In the end you will get something like this:

For this tutorial I extended the TestCalculator unit test project and I created a new one called TestIntBitset. The new project tests a simplified bitset implementation. Once again, the code to test (the IntBitset class) is included in the unit test project for simplicity.

Data driven testing

An advanced feature of Qt Test is data driven testing. The idea is to separate tests and data to avoid to have a long list of similar QVERIFY or QCOMPARE macros and to replicate all the code needed to initialise a test.

To provide data to a test function you have to create another private slot with the same name of the function and add the “_data” suffix. For example the data function for testDiff() is testDiff_data().

Implementing a data function is a bit like inserting data into a database. First you define your data like if you were defining a table:

void TestCalculator::testDiff_data() { QTest::addColumn<int>("a"); QTest::addColumn<int>("b"); QTest::addColumn<int>("result");

Then you add rows of values:

QTest::newRow("all 0") << 0 << 0 << 0; QTest::newRow("same number") << 10 << 10 << 0; // ... more data ... }

Each row contains a name and a list of values. You can imagine that the previous code is converted to something like the following table:

INDEX NAME a b result 0 “all 0” 0 0 0 1 “same number” 10 10 0

Once we have defined the data function we can write the test function which is divided in 2 parts.

The fist part retrieves the data:

void TestCalculator::testDiff() { // retrieve data QFETCH(int, a); QFETCH(int, b); QFETCH(int, result); // set value mCalc.SetA(a); mCalc.SetB(b);

The second part uses the data to perform checks:

// test QCOMPARE(mCalc.Diff(), result); }

Without a data driven approach we should have repeated the instructions to set the values and the QCOMPARE check many times.

When a data driven test is executed the test function is called once per set of data and the log message looks like this:

PASS : TestCalculator::testDiff(all 0) PASS : TestCalculator::testDiff(same number) ... more lines ...

As you can notice the name of the data row is reported in the log to help you differentiate the cases.

Other useful macros

Qt Test offers few extra macros to help you handling different situations in your unit tests.

Failing a test

One of these macros is QFAIL, which makes the current test fail. It can be used when you know that something will make the test fail. In that case there’s no point in wasting time executing the test, you can just fail and move on.

void TestIntBitset::initTestCase() { if(sizeof(int) != 4) QFAIL("Int size is not 4 on this platform."); }

In my example project I used QFAIL in the initTestCase and cleanupTestCase functions, which are special functions executed before and after the test functions are executed. When initTestCase fails none of the tests in the test case is executed.

Failing a single check

In case you know that a particular QVERIFY or QCOMPARE is going to fail, but you still want to continue executing the test, you can precede a check with the macro QEXPECT_FAIL:

void TestIntBitset::testSetOff() { mBS.setAllOn(); unsigned int bitsOff = 0; mBS.setBitOff(BITS_IN_BYTE * bitsOff++); mBS.setBitOff(BITS_IN_BYTE * bitsOff++); QEXPECT_FAIL("", "isAnyOff not implemented yet", Continue); QVERIFY(mBS.isAnyOff()); // ... more test code ... }

Its first parameter identifies a row of data when doing data driven testing, but it can be set to an empty string during normal testing. The second one is a log message and the third one lets you decide if you want to Continue or Abort the test on failure.

When running the previous test the output log will show something like this:

XFAIL : TestIntBitset::testSetOff() isAnyOff not implemented yet Loc: [../../UnitTests/TestIntBitset/TestIntBitset.cpp(67)]

Skipping a test

void TestIntBitset::testOperators() { QSKIP("Operators have not been implemented yet..."); }

No code after QSKIP will be executed, but the code before will, so if any check there fails the test will be considered failed.

Running a skipped test will show the following text in the logs:

SKIP : TestIntBitset::testOperators() Operators have not been implemented yet... Loc: [../../UnitTests/TestIntBitset/TestIntBitset.cpp(28)]

Deciding when using QFAIL and when using QSKIP can be debatable sometimes. In general there is not a precise rule and it’s all about your design choices. Personally I tend to use QFAIL when I know in advance that something is going to fail and I want to highlight that, whereas QSKIP when it doesn’t matter executing a test or part of it.

Warning messages

If you want to print a warning message in the tests log you can use QWARN. This macro can be useful when you want to notify that something is not going as expected in a test.

void TestIntBitset::testSetOff() { mBS.setAllOn(); unsigned int bitsOff = 0; mBS.setBitOff(BITS_IN_BYTE * bitsOff++); // ... more test code ... // this test will trigger a warning if((BITS_IN_BYTE * bitsOff) < BITS_IN_INT) QVERIFY(!mBS.isBitOff(BITS_IN_BYTE * bitsOff)); else QWARN("trying to verify bit out of set bounds"); // ... more test code ... }

In this case the QVERIFY check will fail because input data is somehow wrong. It would be unfair to fail the test because of a possible bug in the local code, but this situation needs to be highlighted. A warning is a good way to achieve that.

When running a test containing a warning, the output log will show something like this:

WARNING: TestIntBitset::testSetOff() trying to verify value out of set bounds Loc: [../../UnitTests/TestIntBitset/TestIntBitset.cpp(75)]

The warning message will always have a message and will show where the warning was issued.

Qt Creator integration

Not surprisingly, Qt Creator offers an excellent integration with Qt Test.

One of the panels in the left sidebar is called “Tests” and it shows all the unit test found in your container project.

Using this panel you can disable some tests, run them all or run only a specific one. When doing data driven testing, you can also select which data sets are enabled. All this is extremely useful in a real project where you can have hundreds or even thousands of unit tests and you need to check/debug only one/few.

When running the unit test from the Tests panel the results are shown in the Test Results panel that you can also open with ALT-8.

This panel will show clearly which tests passed and which not, but it will also show other useful information, especially in case of failure. In particular you click on results to jump to any fail or warning in the code. Furthermore, the panel lets you filter what events you want to see in the log, so for example you can check only fails or warnings.

The 2 panels combined make a great addition to Qt Creator and Qt Test and offer a very powerful tool for free.

Source code

The full source code of this tutorial is available on GitHub and released under the Unlicense license.

In this case you will find 3 qmake projects, but you only need to load the top level one (UnitTests.pro) to build.

References

To know more about Qt Test you can check out the latest documentation of the QTest namespace.

If you want to learn more about Qt have a look at the other Qt tutorials I posted.

Conclusion

The features discussed in this tutorial make Qt Test a more usable and complete framework than the one introduced in the first part of this series. In particular the integration with Qt Creator is extremely useful and effective. Data driven testing can also be very powerful to reduce testing code and to test different cases very easily.

Things will get even more interesting when I will discuss GUI testing in the next tutorial, so stay tuned for more.

In case you need help to handle your C++ unit tests with Qt Test feel free to contact me.

Stay connected

Don’t forget to subscribe to the blog newsletter to get notified of future posts.

You can also get updates following me on Github, Google+, LinkedIn and Twitter.