Anti-flakiness for external/async tests

1. Flaky tests in external/async

We already have the basic pattern: replace the external call with a deterministic stub, wait for the async effect with Awaitility, and make the observable effect a port call or the state of a fake. But even with this pattern, external/async tests are easy to make flaky if timeouts, polling, and test double state are assembled by guesswork.

Why does this happen? Because external and asynchronous things bring time and non-determinism into tests. Time means delays, queues, threads, schedulers. Non-determinism means different machine speeds, a random moment when the effect happens, and an unpredictable order of events. When a test does not agree with that explicitly, it turns into a lottery, just without the pleasant prize.

It helps to keep a simple distinction in mind. A bad test usually fails all the time, reliably, because the expectations are wrong. A flaky test fails sometimes, because the assertion is conceptually correct, but the moment of checking was chosen at random.

2. Timeouts for external and async tests

A timeout is not a “hack” but a boundary of common sense. In tests for external and async scenarios, timeouts serve two important roles: first, they protect you from stuck tests (which can hold up a build for 20 minutes); second, they make failure reasons clearer. A test that fails after 2 seconds with the message “did not observe the effect” is much more honest than a test that hangs for 10 minutes and gets killed by the pipeline limit.

The most important thing to understand is that such tests usually have two different timeouts. One applies to the external call (for example, contenthub.moderation.read-timeout), and the other applies to waiting for a condition in the test (for example, Awaitility.atMost(...)). If you confuse them, tests start either “slowing down” or “failing by luck”.

Let us put this into a table so we do not have to keep it in our heads like a spell:

External tests: fail-fast through test configuration

In tests for the RestClientModerationClient adapter, it is useful to make the configuration deterministic and fast: fixed base URL, small timeouts, and no dependency on a random application.yml that may have been left “production-like”.

Example — set the configuration directly in the test via @TestPropertySource:

import org.springframework.test.context.TestPropertySource;
import org.springframework.boot.test.autoconfigure.web.client.RestClientTest;

@RestClientTest(RestClientModerationClient.class)
// Important: set test properties explicitly so we do not inherit “production” timeouts/URL from the environment.
@TestPropertySource(properties = {
        // A fixed base URL makes the test deterministic.
        "contenthub.moderation.base-url=https://moderation.local",
        // A small timeout helps fail fast and clearly if the stub did not work.
        "contenthub.moderation.read-timeout=200ms"
})
class RestClientModerationClientTest {
}

The point here is not that “200ms is the perfect number forever”. The point is that the test must not accidentally inherit a 30-second timeout and then make you wait forever when something goes wrong. For unit/slice adapter tests, it is better to “fail fast and explain” than to “wait patiently for a long time”.

Async tests: timeout as part of the scenario

In Awaitility, the timeout should be written explicitly. Not because the library cannot work without it, but because it makes the test readable. When reading the test, you immediately understand: “we consider it normal for the effect to appear within at most 2 seconds.” If the effect does not appear, there is a problem.

import java.time.Duration;
import static org.awaitility.Awaitility.await;

// Wait for the effect to appear for at most 2 seconds — this is part of the test contract.
await().atMost(Duration.ofSeconds(2))
       // untilAsserted repeats the assertion until it passes or the timeout expires.
       .untilAsserted(() -> then(sender).should().sendPublished(42L));

If you remove atMost(...), the test gets implicit waiting logic, and that is worse. Implicit behavior in tests is almost always a synonym for “we will not remember why we did it this way later”.

3. Polling in waits

The basic await().untilAsserted(...) already solves the main problem: we wait for a condition, not just waste time. Now it is important to make polling explicit: how often we check the condition, whether we need a delay before the first check, and how to prove negative scenarios.

Explicit polling: less magic, more predictability

Here is an example of a “readable” wait:

import java.time.Duration;
import static org.awaitility.Awaitility.await;

await()
       // Do not wait before the first check: if the effect is already there, the test will pass faster.
       .pollDelay(Duration.ZERO)
       // Polling interval: too large makes the test sluggish, too small makes it noisy/expensive.
       .pollInterval(Duration.ofMillis(50))
       // Overall waiting deadline: after that, the test should fail honestly.
       .atMost(Duration.ofSeconds(2))
       // We check the observable effect specifically.
       .untilAsserted(() -> then(sender).should().sendPublished(42L));

pollDelay(Duration.ZERO) means “check immediately”, and that is usually useful: if the effect already happened quickly, the test will not wait unnecessarily. pollInterval(50ms) is a compromise: frequent enough to catch the event quickly, but not so frequent that the test turns into “check spam”.

And there is an important practical nuance here: the polling interval should not be pulled from the author’s head. If you picked 50ms, let that be visible. Then, when the test starts to flap in CI, you can discuss concrete numbers instead of mysticism.

Negative async checks: “during an interval” rather than “at one moment”

The most treacherous trap is negative checks in async code. For example: “after moderation is rejected, the notification must not be sent.” If you write then(sender).should(never())... immediately after the call, the test may “honestly” pass... and yet the notification may be sent after 100ms, after the check. In other words, the test is green, but the bug is alive — classic.

One workable approach is to require that the condition remain true for some time. Awaitility supports this through during(...).

import java.time.Duration;
import static org.awaitility.Awaitility.await;

await()
       // during: the condition must remain true for the specified time (we do not take a “one-time snapshot”).
       .during(Duration.ofMillis(300))
       // atMost: the upper bound on waiting in case the scheduler/CI is delayed.
       .atMost(Duration.ofSeconds(1))
       // Check that the call did not happen during the interval.
       .untilAsserted(() -> then(sender).should(never()).sendPublished(42L));

The idea is: we did not just “check once”, we checked for 300ms straight that the notification still had not been sent. That is already closer to proof than to “well, it does not seem to have been sent yet”.

4. Observable effect in tests

We already chose the observable effect: either wait for then(sender).should(...), or check the state of a capturing fake. Now another thing matters: the observation object itself must not become a source of randomness. If it is not thread-safe or is shared between tests without cleanup, we will create flaky behavior ourselves.

For ContentHub, a good example is PublicationNotificationSender: we either verify this port with a mock, or record it in a thread-safe fake.

Capturing fake as a “black box” for the effect

Let us make a small thread-safe fake that stores published IDs:

import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;

public class CapturingNotificationSender implements PublicationNotificationSender {
    // Thread-safe queue: the effect may be added from another thread (async executor).
    private final Queue<Long> publishedIds = new ConcurrentLinkedQueue<>();

    @Override
    public void sendPublished(long articleId) {
        // “Observable effect”: record the send so the test can check the state.
        publishedIds.add(articleId);
    }

    // For tests, people usually add an accessor (for example, publishedIds()) to read accumulated events.
}

Yes, that is “ridiculously simple”. And that is exactly why it is good: the test does not need to know how mocks work — it needs the observable fact “the ID was added”.

And now the async test can be written like this:

import java.time.Duration;
import static org.awaitility.Awaitility.await;
import static org.assertj.core.api.Assertions.assertThat;

// Note: publishedIds() is any convenient accessor to the accumulated events (getter/method).
await().atMost(Duration.ofSeconds(2))
       // Awaitility will repeat the assertion until it becomes true.
       .untilAsserted(() ->
               assertThat(sender.publishedIds()).contains(42L));

Notice that we wait for a normal AssertJ assertion to pass. And if it does not, Awaitility repeats it until the timeout. The test reads like a human sentence: “wait up to 2 seconds until sender records the send”.

State isolation: so tests do not affect each other

If you use a capturing fake (or any shared mutable object), it almost always needs a state-reset discipline. It is best when the fake is created as a new object per test, and then there is nothing to reset.

If the fake lives as a Spring bean inside @SpringBootTest, then add a cleanup method and call it from @BeforeEach. It is boring... but it is exactly the kind of boredom that saves you from “it only fails when the whole class runs”.

5. Deterministic stubs and fixtures

An external HTTP test can be made unstable even without real HTTP. It is very easy to get “shifting” expectations if you add random data, generate strings with an unpredictable field order, or compare too many details that are not part of the contract. For MockRestServiceServer, the rule is simple: the stub should be small, clear, and deterministic. Everything extra is potential noise.

This is especially important in ContentHub for moderation payloads. We do not want the test to “break” if someone adds a new field to the JSON response from the external service that we do not need. We want to fail only when something important breaks: for example, the result or reason field stops being read correctly.

Minimal JSON responses as part of the anti-flak strategy

For BLOCK, for example, we only need:

server.expect(requestTo("https://moderation.local/api/check"))
      .andRespond(withSuccess(
              // Minimal response: only the fields that actually participate in the contract.
              """
              {"result":"BLOCK","reason":"spam"}
              """,
              MediaType.APPLICATION_JSON));

This stub has no dates, no random ids, and no huge JSON tree. It cannot be made unstable “by accident”. And if the test failed, then either you did not call the correct endpoint, or you stopped reading result/reason correctly. In other words, the failure is informative.

“Timeout” in a test without a real timeout

A very common mistake is for a timeout test to do a real Thread.sleep(5_000) in the stub or to try to “wait until it times out” for real. That is both slow and unstable. It is much better to simulate a timeout as a deterministic exception.

import java.net.SocketTimeoutException;
import org.springframework.test.web.client.response.MockRestResponseCreators;

server.expect(requestTo("https://moderation.local/api/check"))
      .andRespond(MockRestResponseCreators.withException(
              // Deterministically simulate a network problem without making the test actually “sleep”.
              new SocketTimeoutException("read timed out")))

Now your adapter should translate this into a ModerationUnavailableException (or your domain-specific form). The test is fast, stable, and checks exactly what is needed: “error translation”, not “the JVM’s ability to sleep for 5 seconds”.

Remove hidden randomness from test data

Sometimes flaky behavior comes not from threads, but from the fact that you used UUID.randomUUID() or Instant.now() in a test “for convenience”. On a local machine that is often invisible, but on CI it can cause a mismatch, especially if you compare time serialization.

In this course, we are already used to fixing Clock and using stable test values. Anti-flakiness here is simple: if a value is not important to the meaning of the test, make it a constant. Flakiness almost never comes from “too stable data”.

Deterministic stubs do not eliminate real infrastructure checks altogether. They remain the base of the suite, and a heavier real-infra path should be added selectively — only where the cheap deterministic test no longer proves the risk you need.

6. Fail fast and diagnostics

When a test flaps, the first reaction is usually: “I will increase atMost(...) from 2 seconds to 10.” Sometimes that really helps temporarily... and almost always makes the suite worse: tests become slower, and the problem still remains. We need an approach of “fail fast + clear failure”. Then flaky behavior stops being “mystical” and becomes an ordinary defect, either in the code or in the test environment.

Clear messages in assertions

AssertJ lets you label assertions. In the async world this is especially useful: if the wait did not succeed, you want to see a human meaning in the failure, not “Expected true but was false”.

import static org.assertj.core.api.Assertions.assertThat;

assertThat(decision.result())
        // The message explains the meaning of the expectation: what exactly we check and why it matters.
        .as("moderation result should be mapped from external payload")
        .isEqualTo(ModerationResult.OK);

This also works in Awaitility, because Awaitility repeats the same assertion, and in the end you will see the same “human” message.

Check that the stub was used

MockRestServiceServer can verify expectations. If you did not hit the expected endpoint at all, you want to know that quickly, not indirectly.

A simple option is to call verify() explicitly at the end of the test:

import org.junit.jupiter.api.Test;

@Test
void shouldCallModerationEndpoint() {
    // ... expect + act ...
    // Important: verify() gives an honest failure “the request was not made”, not a vague “something went wrong”.
    server.verify(); // if there was no request — we fail here
}

It is a small thing, but it makes the failure honest: “the request was not made” instead of “something is wrong somewhere”.

Order of fixing flakiness: clarity first, numbers second

A good order of actions looks like this: first make the effect observable (capturing fake, explicit assertion), then make timeouts explicit, then choose polling. And only if instability remains after that, look at performance/scheduling.

A bad order of actions is to first crank timeouts up to “just to be safe”, and then live with a suite that takes 12 minutes to run. That treats the symptom (sometimes) but breaks project discipline: feedback becomes too expensive.

7. Typical mistakes in external/async tests

Mistake #1: treating a flaky test by increasing Thread.sleep(...) instead of changing the waiting strategy.
Thread.sleep does not wait for an event; it waits for time. If the event happens after 10ms, you still sleep for 2 seconds. If the event happens after 2100ms, you sleep for 2 seconds and fail. That is almost the perfect formula for instability. Awaitility waits for the condition itself, and that is the key difference.

Mistake #2: not separating the external call timeout from the test timeout.
Sometimes in a test people set await().atMost(10 seconds) and think that “this will solve external call problems”. But if your RestClient is configured with a read-timeout of 30 seconds, you may get a hang inside the call before Awaitility even starts polling anything. Timeouts must exist on both levels and be reasonable.

Mistake #3: negative async checks are done “at one moment” instead of “during an interval”.
A should(never()) check right after a method call is almost always meaningless: “did not happen” may simply mean “has not happened yet”. If you are checking that the effect must not happen, use the “condition must remain true” strategy (during(...)) or design things so the effect is synchronously forbidden at the business-logic level.

Mistake #4: use overly “rich” stubs and expect them to stay stable.
Large JSON responses with extra fields, dates, random numbers, and unclear structures create more surfaces for accidental changes. You need a small stub that checks the contract in essence. This is not “simplifying reality”; this is engineering focus in the test.

Mistake #5: a shared mutable object for the whole test class without state cleanup.
Capturing fake, a list of sent IDs, an in-memory log — all of that is great... until it starts remembering the previous test. Then you get “test order matters”, and that is almost a guaranteed path to flaky behavior. Either create a new object per test, or clean the state up disciplinedly in @BeforeEach.