QTest: Data-Driven Unit Tests Are Hard To Understand

Today, I looked at the data-driven unit tests I had written nearly four weeks ago. It took me a couple of minutes to understand the tests again. Understanding my own tests should have been much easier.

Data-driven unit tests in QTest have a fundamental problem. The data goes in one function, say testWriteFrame_data, and the test case working on the data goes in another function, say testWriteFrame. I must go back and forth to understand the test case. While going back and forth, I typically forget one piece of information. So, I must do another round trip.

So, I sat down and converted each row of the table created by testWriteFrame_data into a test function of its own. The resulting test cases are much easier to understand. They have about the same code size as the original solution. But see for yourself.

Here are the original data-driven test cases – split up in the data part testWriteFrame_data and the execution part testWriteFrame.

void TestReadWriteOnMockCanBus::testWriteFrame_data()
{
    QTest::addColumn<MockCanFrameCollection>("outgoingFrames");
    QTest::addColumn<MockCanFrameCollection>("expectedCanFrames");
    QTest::addColumn<bool>("isCanIoOk");
    auto frame1 = MockCanFrame{MockCanFrame::Type::Outgoing, 
        0x18ef0201U, "018A010000000000"};
    auto frame2 = MockCanFrame{MockCanFrame::Type::Outgoing, 
        0x18ef0301U, "01B5010000000000"};

    QTest::newRow("Write two frames in expected order")
            << MockCanFrameCollection{frame1, frame2}
            << MockCanFrameCollection{frame1, frame2}
            << true;
    QTest::newRow("Write more frames than expected")
            << MockCanFrameCollection{frame1, frame2}
            << MockCanFrameCollection{frame1}
            << false;
    // 4 more rows ...
}

void TestReadWriteOnMockCanBus::testWriteFrame()
{
    QFETCH(MockCanFrameCollection, outgoingFrames);
    QFETCH(MockCanFrameCollection, expectedCanFrames);
    QFETCH(bool, isCanIoOk);

    setExpectedCanFrames(m_device, expectedCanFrames);

    for (const auto &frame : outgoingFrames) {
        QVERIFY(m_device->writeFrame(frame));
    }
    QCOMPARE(actualCanFrames(m_device) == expectedCanFrames, 
             isCanIoOk);
    QCOMPARE(m_writtenSpy->size(), outgoingFrames.size());
}

The function testWriteFrame_data creates a table with the columns outgoingFrames, expectedCanFrames and isCanIoOk and six rows (only two of them shown). QTest::newRow creates a row with a tag (e.g., “Write one expected frame”) and a value for each column.

The test runner calls the function testWriteFrame for each row. The QFETCH macros extract the data from a row and assign them to the respective variables.

The function testWriteFrame specifies with the call to setExpectedCanFrames that it expects the test case to produce the expectedCanFrames on the CAN bus. It then writes all the outgoingFrames to the CAN bus and finally checks the results.

So, what is my problem with data-driven tests? I must look at two places – testWriteFrame_data and testWriteFrame – at the “same” time to understand the test case. Typically, I go back and forth between the two places a couple of time, before I understand why the test fails. This may even involve scrolling up and down through the code. I easily forget a piece of information and must take another round trip.

I want to understand the test function with one glance. All the required information must be in one place. I achieved this by converting each row into a test function of its own. Here are the test functions for the two rows shown above.

void TestReadWriteOnMockCanBus::testWriteTwoFramesInExpectedOrder()
{
    expectWriteFrame(c_frame1);
    expectWriteFrame(c_frame2);

    m_device->writeFrame(c_frame1);
    m_device->writeFrame(c_frame2);

    CHECK_FRAMES_EXPECTED_EQUAL;
}

void TestReadWriteOnMockCanBus::testWriteMoreFramesThanExpected()
{
    expectWriteFrame(c_frame1);

    m_device->writeFrame(c_frame1);
    m_device->writeFrame(c_frame2);

    CHECK_FRAMES_EXPECTED_NOT_EQUAL;
}

The first block with the calls to expectWriteFrame specifies which CAN messages the second block with the calls to writeFrame must produce. The third block is a macro to check the actual result of the second block against the expected result of the first block. The two CHECK_FRAMES macros give better diagnostic output than the two QCOMPARE macros in the original version, because they don’t have to be the same for an expected failure and for an expected success.

The intent of the rewritten tests is instantly clear. One look at the tests is enough to understand them.

This simplicity is not possible with the data-driven approach of QTest. The required information is always in two places. The actual test function tends to have a higher complexity, because it must take care of multiple execution paths (the for loop and the isCanIoOk flag in this example).

My conclusion is clear: I won’t use data-driven unit tests much in the future.

Acknowledgement. I want to thank James W. Grenning for pointing out the same problem in my solution to one of the exercises in his course Test-Driven Development for Embedded C/C++.