Unit Testing

This week I decided to take a quick break from the series on building the core Qi library in order to talk about an important part of any code project, testing! Even though this might not be a very interesting part of the project, it's important to get into place earlier rather than later. If you have an easy-to-use unit test system and already have it in place and ready to go, it's fairly painless to add new tests as you add new features to the engine. If you wait until later, it's becomes an impossible task of writing tons of tests to exercise features that you haven't worked on for months.

You could always write your own test system but this is very solved problem with many good options out there. Previously I've made pretty extensive use of UnitTest++ which has a very simple-to-use interface and works pretty well. However, with Qi I've decided to change it up and make use of Google's gtest. This is a pretty lightweight open-source unit test framework that you can easily integrate into any project. Setting it up is pretty easy and you can follow the documentation for your specific platform.

When using gtest, you define a simple main.cpp like this:

int main(int argc, const char * argv[])
{
    ::testing::InitGoogleTest(&argc, (char **)argv);
    return RUN_ALL_TESTS();
}

gtest exposes the simple macro TEST() which you can use to declare tests in your test files. The RUN_ALL_TESTS() macro will find all uses of the the TEST() macro and put them together at runtime.

A simple test to verify that vector addition is working in Qi could look like this:

TEST(VectorTests, Addition)
{
    Vec4 v1(1, 2, 3, 4);
    Vec4 v2(5, 6, 7, 8);
    
    Vec4 sum = v1 + v2;
    EXPECT_EQ(6,  sum.x);
    EXPECT_EQ(8,  sum.y);
    EXPECT_EQ(10, sum.z);
    EXPECT_EQ(12, sum.w);
}

The first parameter to the macro specifies the test group and the second parameter is the name of this specific test. This is really just a way of grouping your tests for when you look at your output later. EXPECT_EQ() is another macro defined by gtest which states what you expect a result to be compared with the actual result returned by the calculation. This macro and its variants are the real heart of the unit test system. The current tests for Qi are located here.

When writing unit tests, I think it's important to think about what a "unit test" really is. I usually think of it as the smallest possible test which is useful to catch any errors in your code. Something as complex as rendering an entire frame isn't really a unit test. So many things can go wrong during frame rendering that a single failing test doesn't tell you much about what a problem is. However, a test which ensures that a matrix is properly transforming a vector is very easy to diagnose when failing. Therefore, Qi's unit tests are very small and likewise very quick to diagnose. Some simple test areas could include:

• Math libraries
• Do the math functions produce the correct results?
• Custom containers
• Are elements able to be added/removed?
• Do queries still work after adding data that should be found?
• Upon deletion, are there any memory leaks?
• Memory systems
• Are any leaks being properly detected?
• Do allocations/deallocations happen as expected?
• Threading
• Is the code threadsafe?

Like I said, these are just some examples. As Qi adds new subsystems/features, new unit tests will always follow along. Once you've gotten down your low-level system unit tests, you can branch into larger features which build on multiple concepts within your engine, such as:

• Initialization
• Does the engine initialize properly under different inputs
• Error checking
• Are the proper errors reported under invalid input (not crashing)
• Resource loading
• Can you load and use resources properly?
• What happens when a resource isn't found?
• Adding and removing characters to the world
• Does the engine perform properly under these conditions?
• Etc...

Remember that unit tests are your first line of defense against a logic bug (which are notoriously hard to track down). Think that change you made has no reaching side effects outside of where you made it? Run the tests to make sure. If everything passes then it means that there are no problems with your change as far as your current tests are aware of. It's important to think about that last part because you may have many tests but if they don't tests all parts of your system (and all kinds of inputs) then you can't be 100% sure the code is error-free. In fact, it's likely that it never will be :).

How often should you run your unit tests? There are many answers to this question so really it comes down to how each team wants to work. The smaller your tests are, the faster they'll be able to be ran. If you can run hundreds or even thousands of tests in a few minutes, developers will be more likely to run the tests all of the time. If the runtime starts to soar upwards of fifteen to twenty minutes you might as well forget it, nobody is ever going to run them and you'll have a whole group of changes that probably weren't well tested. For Qi, I run the test suite periodically during the development of new features and most definitely before pushing any code to the main repository. My policy is to never push code that doesn't pass all tests otherwise tracking down issues later can become incredibly difficult.

Once the engine becomes more complete, it would be nice to perform image-based testing to make sure that a resulting frame looks how it's supposed to (e.g. nothing has failed to render/moved unexpectedly). However, these tests are harder to debug and we won't be able to use them in a meaningful way until much later in the development in the engine. As things are changing all of the time, the reference images will constantly become invalid. Once there is an official release, then these kinds of tests will become useful as things will be changing less. Conversely, unit tests are small, easy to implement, and can start testing your code right away with pretty minimal effort!

Until next time!