CTest — running tests as part of the build

1. Why do we need CTest at all if we already have doctest/Catch2?

If you have just learned a testing framework, it is natural to ask: “Okay, tests run — why do we need CTest?” Developers already deal with enough extra tools, especially when a deadline is looming. Let’s carefully separate responsibilities and see what CTest actually adds.

A test framework (doctest/Catch2) is a library inside your test binary. It can discover test cases, run them, pretty-print expect/actual and return a process exit code 0 (success) or != 0 (failure).

CTest is a tool outside. It does not “understand” your TEST_CASE or analyze your CHECK. It does something else: it organizes running tests as part of the project build process. In short, CTest is a dispatcher: “there are these registered tests in the project, let’s run them all, collect statistics, print a summary and return an overall success/failure code.”

It helps to keep the following table in mind:

2. The main idea of CTest: a test is a command that returns 0

When you hear “CTest”, a newbie might imagine some magical compiler mode: that it will “embed tests into the build” and they will run automatically by themselves. The actual model is very down-to-earth: a test is just a process (a command) you can run, and the process has an exit code.

Almost all automation in development relies on a very simple convention: 0 means success, any other number means error. CTest is no exception. It runs test commands and checks whether the exit code is 0. If it is not 0, the test is considered failed, even if it printed “everything is fine, honest”.

To feel this hands-on, it is useful to see a micro test without a framework — a homemade runner. It is not pretty, but it is very honest (like cout without formatting):

#include <iostream>

bool test_add() {
    return (2 + 3) == 5;
}

int main() {
    if (!test_add()) {
        std::cerr << "test_add FAILED\n";
        return 1;                   // <-- non-zero: failure
    }

    std::cout << "test_add OK\n";    // test_add OK
    return 0;                        // <-- 0: success
}

That is why the combination “framework + CTest” works smoothly: the framework inside the process decides pass/fail and sets the correct exit code, and CTest outside runs processes and aggregates the result.

How CTest is hooked into a CMake project

CMake can generate build files, and it can also generate a “test registry” — a list of commands considered project tests. CTest later reads that list.

Enabling CTest in CMake requires two minimal steps. First — tell CMake “this project has tests, enable testing support”. Second — register specific commands as tests.

In CMake it looks like this:

enable_testing()

add_test(NAME core_tests COMMAND core_tests)

The meaning reads almost like plain language: enabled testing, added a test named core_tests which runs the command core_tests (usually the test executable name).

Important: add_test registers a command, not a “test file”. You can register anything as a test: running your application with arguments, a script, a format checker — but for our focus we keep to unit tests and test binaries.

3. Practical example: test binary and registration in CTest

Now we will assemble a small but complete project skeleton, with logic (what we test), an application (user runs it), and tests (run by CTest). This is important methodically: if you mix everything into one main.cpp, you might have “run tests” but automating that becomes hard and maintenance is painful.

Imagine we develop a small learning app TinyCalc. It currently supports two things: clamp (clamp a number into a range) and safe_div (divide but return std::optional if division is impossible). No magic — just functions easy to test.

Core code we will test

File src/core.hpp:

#pragma once

#include <optional>

int clamp(int x, int lo, int hi);
std::optional<int> safe_div(int a, int b);

File src/core.cpp:

#include "core.hpp"

int clamp(int x, int lo, int hi) {
    if (x < lo) return lo;
    if (x > hi) return hi;
    return x;
}

std::optional<int> safe_div(int a, int b) {
    if (b == 0) return std::nullopt;
    return a / b;
}

Note the testability: no std::cin, no dependency on time, no randomness. You supply parameters and get a result. Such functions are easy to test.

Application that uses the core

File src/main.cpp:

#include <iostream>
#include "core.hpp"

int main() {
    std::cout << clamp(15, 0, 10) << '\n';   // 10
    const auto r = safe_div(10, 2);

    if (r) {
        std::cout << *r << '\n';             // 5
    }
}

This just demonstrates that the library is wired and works. You can unit-test main, but that is a separate discipline; for today it is enough to test the core.

Test binary using doctest/Catch2

File tests/core_tests.cpp:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

#include "core.hpp"
#include <optional>

TEST_CASE("clamp clamps into [lo, hi]") {
    CHECK(clamp(5, 0, 10) == 5);
    CHECK(clamp(-1, 0, 10) == 0);
    CHECK(clamp(999, 0, 10) == 10);
}

TEST_CASE("safe_div returns value or nullopt") {
    CHECK(safe_div(10, 2) == std::optional<int>{5});
    CHECK(safe_div(10, 0) == std::nullopt);
}

This file is built into a separate executable, e.g. core_tests. Run it directly — it will discover tests, execute them and return an appropriate exit code.

CMakeLists.txt: tying everything together

At the project root: CMakeLists.txt:

cmake_minimum_required(VERSION 3.20)
project(TinyCalc LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_library(core src/core.cpp)
target_include_directories(core PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)

add_executable(tinycalc src/main.cpp)
target_link_libraries(tinycalc PRIVATE core)

add_executable(core_tests tests/core_tests.cpp)
target_link_libraries(core_tests PRIVATE core)

enable_testing()
add_test(NAME core_tests COMMAND core_tests)

Two key ideas are present here.

• First idea: tests are a separate executable (target), like the application. We do not mix tests into tinycalc, otherwise later we will confuse ourselves about what we run and why.
• Second idea: add_test registers running core_tests as a CTest test. CTest will then run that binary and check the exit code.

4. How to run CTest in practice

Here usually comes the first typical “newbie pain”: someone runs ctest, gets a strange error or sees zero tests, and then proclaims: “CTest is broken”. Most often it is not CTest that is broken but the understanding of from which folder to run it and what must be built first.

CTest “lives” in the build directory, because CMake generates files there that list the tests. So the flow is: configure the project, build it (so test binaries exist), and only then run ctest.

A typical command sequence looks like this:

cmake -S . -B build
cmake --build build
ctest --test-dir build

If you run ctest from the build directory, often it is enough to do:

cd build
ctest

What will the output show? Usually CTest prints how many tests it found, which it ran and which failed. Important detail: if a test failed, CTest will exit with code != 0. That means the test step is failing for automation.

Why you sometimes see “0 tests”

If CTest reports 0 tests, this almost always means one of two things: either you forgot enable_testing() / add_test(...), or you are running ctest in the wrong folder (not where CMake generated the test configuration).

How to see the output of a failed test

Another common surprise: a test fails but you do not see detailed messages (for example, the expected/actual values). This is not a bug — it is an output policy. It is often useful to enable “show output on failure”:

ctest --test-dir build --output-on-failure

Or a “very verbose” mode:

ctest --test-dir build -V

The idea is simple: CTest can be a “quiet dispatcher”, and on request — a “chatty dispatcher”. Chatty output is useful during development; in automated checks you might prefer quiet.

Multiple tests: names and filtering

When you have one binary core_tests, everything is simple. But once the project grows, there are several test binaries: parser tests, model container tests, formatting tests. CTest becomes especially useful: it can run the test set as a “suite” and lets you select subsets.

In CMake this is just repeating add_test with different names:

add_executable(parser_tests tests/parser_tests.cpp)
target_link_libraries(parser_tests PRIVATE core)

add_test(NAME parser_tests COMMAND parser_tests)

A good habit: choose test names (NAME ...) that are human readable in reports. test1 and test2 are like variables a and b in accounting: technically fine, but later you will make life harder for yourself.

When there are many tests, CTest allows running by a filter, for example by a regular expression on the test name. Conceptually:

ctest --test-dir build -R core

That is “run only those tests whose name contains core”. You do not have to memorize it, but it is useful to know you do not need to run the whole suite when fixing one subsystem.

5. How tests become part of the build process

The phrase “as part of the build” sometimes sounds like tests execute during compilation (like constexpr). In reality “part of the build” means part of the project verification process. First the build ensures the code compiles and links. Then tests ensure behavior is not broken.

This separation of responsibilities is important. The compiler checks form: types, names, syntax, linking. Tests check meaning: “safe_div(10, 0) indeed returns nullopt”, “clamp does not escape the range”, “parser rejects garbage” — and so on.

Here CTest appears: it turns “manually run test exe” into a standard project test command. You no longer need to remember the test binary name or where it lives. You tell CTest, “run the project tests”, and it does it because CMake already registered the required commands.

It is also useful to know about multi-configuration generators (for example, some IDEs) with Debug/Release. In such cases you sometimes need to tell CTest the configuration, but the idea is: run tests from the configuration where they were built. If you built Debug — run Debug tests. If built Release — run Release tests.

6. Common mistakes when working with CTest

Mistake #1: tests registered but the binary is not built.
It is easy to call add_test(NAME core_tests COMMAND core_tests) and forget that core_tests must exist as an executable. If there is no add_executable(core_tests ...) or it failed to build due to compile errors, CTest will try to run something that does not exist. Fix the build first, then run the tests.

Mistake #2: tests are “not found” because ctest is run from the wrong folder.
CTest is not a clairvoyant: it will not search the entire disk for your build. If you run ctest from the repository root but test configuration is in build/, CTest will often report “0 tests” or be confused. Discipline: run ctest from the build directory or specify the test directory explicitly.

Mistake #3: tests rely on std::cin and wait for interaction.
CTest runs tests as automated commands. If your test waits for user input, CTest will wait too — which looks like “ctest hung”. Architect tests to exercise logic via functions with parameters; leave IO to main.

Mistake #4: a test fails but you do not see why.
Newcomers sometimes expect CTest to always show full output. It can be quiet and show only failure facts. If you need details, enable verbosity (-V) or output-on-failure (--output-on-failure). Also ensure your test framework writes diagnostics to stderr/stdout as you expect.

Mistake #5: the test prints “FAILED” but returns 0.
This is classic for homemade runners and poorly written checks. CTest trusts the exit code, not words. If you print “something is wrong” but return return 0;, CTest will consider the test passed. The convention is simple: success — 0, failure — != 0. It is best not to argue with this contract — the automation ecosystem expects it.