System resilience: timeouts, circuit breakers, bulkheads, protection against webhook storms

1. Why think about “resilience” in a ChatGPT App at all

In a typical web app, the user at least sees the URL, the browser spinner, and can refresh the page. In ChatGPT, the user sees a single surface: the chat and your App. If something is slow, they cannot tell who is at fault — OpenAI, your Gateway, the payment provider, or a neighbouring analytics microservice. To them it is all “ChatGPT + your App”.

When a tool-call hangs for 30–60 seconds, the model waits and waits… and at best apologizes for the delay. At worst, it hallucinates an answer instead of using data from your backend. So resilience is not only about SRE and uptime — it is also about answer quality, the model’s tone, and your metrics in the Store.

In the ChatGPT App ecosystem we have several independent planes:

• ChatGPT ↔ MCP Gateway.
• Gateway ↔ your backend/REST services (Gift REST API, Commerce REST API, Analytics Service, etc.).
• Your services ↔ external APIs (LLM, payments, catalogs).
• Incoming webhooks (ACP, Stripe, any integrations) ↔ your handlers.

The issue is that a failure in one place can cause a cascade: the Gateway patiently waits on a stuck service, workers get clogged, connections run out, clients start retrying — and within a couple of minutes you have a classic meltdown: everything is on fire and sinking at once. The four patterns we discuss today protect us from exactly that:

• Timeouts — we never wait forever.
• Circuit breaker — we do not bang on a locked door.
• Bulkheads — we build “compartments” so the whole ship does not sink.
• Protection against webhook storms — we accept that webhooks come with duplicates, spikes, and retries, and we prepare for it.

2. Timeouts: we don’t wait forever

What a timeout is and why everything is bad without it

A timeout is the maximum time your code is willing to wait for a dependency to respond: a database, an MCP server, an external HTTP API, a model. If the response does not arrive within the allotted time, we consider the call unsuccessful, free resources, and return a meaningful error or a fallback.

Without timeouts, requests can:

• hang indefinitely,
• consume connections and the thread/worker pool,
• block subsequent requests,
• cause cascading failures.

The pattern is simple: “better a predictable failure in 3–5 seconds than mysterious silence for 5 minutes”.

Remember that we have timeouts at multiple levels:

• at the proxy/load balancer level (Cloudflare, Nginx),
• at the MCP Gateway level (HTTP clients to microservices),
• inside services themselves (calls to DB, external APIs, LLM).

For ChatGPT overall, it is reasonable to keep total tool-call time in the 5–10 second range for typical operations and at most 20–30 seconds for especially heavy ones. Anything longer is almost guaranteed to produce poor UX.

A simple fetchWithTimeout in TypeScript

Let’s start with practice. In the GiftGenius MCP Gateway we have a helper HTTP client that talks to the gift recommender, the commerce service, and analytics. We will wrap the standard fetch in a function with a timeout:

// src/gateway/httpClient.ts
export async function fetchWithTimeout(
  url: string,
  opts: RequestInit & { timeoutMs?: number } = {}
) {
  const { timeoutMs = 5000, ...rest } = opts;
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), timeoutMs);

  try {
    return await fetch(url, { ...rest, signal: controller.signal });
  } finally {
    clearTimeout(timeoutId);
  }
}

Now in the Gateway code we never use a “bare” fetch — only through this helper:

// src/gateway/giftClient.ts
import { fetchWithTimeout } from "./httpClient";

export async function callGiftService(path: string) {
  const res = await fetchWithTimeout(
    process.env.GIFT_SERVICE_URL + path,
    { timeoutMs: 4000 }
  );

  if (!res.ok) {
    throw new Error(`gift_service_${res.status}`);
  }
  return res.json();
}

This guarantees that even if the gift service hangs, after 4 seconds we will abort the connection and can return an MCP error to ChatGPT instead of holding the connection open until the bitter end.

Where exactly to set timeouts in GiftGenius

In our GiftGenius example:

• At the Gateway level: timeouts on calls to the Gift REST API, Commerce REST API, and the Analytics Service / REST API.
• Inside those services: timeouts on calls to the DB, ACP/payment providers, external recommendation APIs.
• At the Gateway entry: a global timeout on the request from ChatGPT so a tool-call does not turn into an “endless spinner”.

It is important that the wait time at the top level be slightly higher than at the inner levels. For example, if the Gateway waits for the backend for 5 seconds and the backend waits for the DB for 3 seconds, we have time left for processing and serializing the result.

How to explain timeouts to the ChatGPT model

For ChatGPT, it is important to return semantic errors, not silently drop connections. Instead of an abstract 500, it is better to return a structured MCP error that the model can relay to the user: “The gift recommendation service is overloaded, please try again shortly”, etc.

This means that in the Gateway on timeout you should:

1. Catch AbortError or your timeout_….
2. Form an MCP response with a meaningful code and a short description.
3. Let the model decide how to explain it to the user.

Timeouts solve stuck requests, but if a dependency starts failing en masse they do not protect you from a flood of identical failed attempts. Here we need the next level of defense — the circuit breaker.

3. Circuit breaker: a switch against dying services

Intuition: why a timeout alone is not enough

We have learned to limit the wait time of individual calls with timeouts. A timeout protects a single call. But if a dependency is truly “dead” (for example, the commerce service throws OOM — Out Of Memory — on every request), we will keep calling it, wait 3–5 seconds each time, catch an error, load the network and CPU, and wait again.

A circuit breaker adds memory: it tracks errors and timeouts and, when there are too many, stops sending requests to that service at all. Instead, it immediately returns a fast failure or a fallback. After some time it cautiously tries again in half-open mode.

Classic breaker states:

• Closed — all good, requests go through.
• Open — the service is considered “dead”, requests don’t go through, immediate error.
• Half-open — try a limited number of requests; if they succeed — go back to closed, if they fail again — back to open.

A simple circuit breaker diagram

A small diagram:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: too many errors
    Open --> HalfOpen: cooldown expired
    HalfOpen --> Closed: several consecutive successes
    HalfOpen --> Open: errors again
    Open --> Open: fast fail

A mini circuit breaker implementation in TypeScript

In production you typically use ready-made libraries (for Node.js there is, for example, opossum or lightweight self-made solutions), but to understand the mechanics, a compact class is enough.

An extremely simplified breaker around a commerce module call:

// src/gateway/circuitBreaker.ts
type State = "closed" | "open" | "half-open";

export class CircuitBreaker {
    private state: State = "closed";
    private failureCount = 0;
    private nextAttemptAt = 0;

    constructor(
        private readonly failureThreshold = 5,
               private readonly cooldownMs = 30_000
    ) {}

    async call<T>(fn: () => Promise<T>): Promise<T> {
        const now = Date.now();

        if (this.state === "open") {
            if (now < this.nextAttemptAt) {
                throw new Error("circuit_open");
            }
            this.state = "half-open";
        }

        try {
            const result = await fn();
            this.onSuccess();
            return result;
        } catch (err) {
            this.onFailure();
            throw err;
        }
    }

    private onSuccess() {
        this.failureCount = 0;
               this.state = "closed";
    }

    private onFailure() {
        this.failureCount++;
        if (this.failureCount >= this.failureThreshold) {
            this.state = "open";
            this.nextAttemptAt = Date.now() + this.cooldownMs;
        }
    }
}

And using it in the commerce service client:

// src/gateway/commerceClient.ts
const commerceBreaker = new CircuitBreaker(3, 20_000);

export async function callCommerce(path: string) {
    return commerceBreaker.call(async () => {
        const res = await fetchWithTimeout(
            process.env.COMMERCE_URL + path,
            { timeoutMs: 3000 }
        );
        if (!res.ok) throw new Error(`commerce_${res.status}`);
        return res.json();
    });
}

Here, when commerce starts replying with errors en masse or failing to meet the timeout, after several failures the breaker switches to open. In this state, for cooldownMs we do not attempt calls at all and immediately return the circuit_open error.

What ChatGPT should see when the breaker has “cut off” a service

From ChatGPT’s perspective, it is better if you:

• Respond quickly with an MCP error like “commerce_unavailable” or “gift_service_overloaded”.
• Add a clear description: “The payment service is temporarily unavailable, let’s try again later.”
• Do not hide the issue behind endless retries.

This is exactly the case where a “fast, honest failure” is better than a long hang. Especially at checkout: a user will more likely accept a clear message than stare at a spinner for 40 seconds and then get “something went wrong”.

Timeouts and breakers protect us from “bad” or down dependencies, but they do not solve the problem when one type of workload consumes all resources and starts choking the rest of the system. For that, we need another layer — bulkheads.

4. Bulkheads: compartmentalization so one part does not sink the whole ship

A ship analogy

The bulkhead pattern is named after ship partitions: if one compartment is breached, water does not flood the whole ship. In architecture this means: split resources across different work streams so that one overloaded service does not eat everything — CPU, connections, pools — and take down critical paths.

In microservices this is usually done via separate:

• HTTP connection pools,
• thread/worker pools,
• queues/topics,
• even separate DB clusters for mission-critical operations.

The idea is that if the gift recommendation service starts slowing down and sticking, it will exhaust only its own resources and not break checkout and authentication.

Bulkheads in the Node.js and MCP Gateway world

In Node.js we do not have threads in the classical sense (we have an event loop and workers), but we can limit the number of concurrent tasks per work stream.

Example: the Gateway has three external dependencies:

• Gift service (gift recommendations, heavy LLM calls).
• Commerce service (checkout, ACP).
• Analytics service (event logging).

We can introduce simple limits on concurrent requests to each of them.

For example, a small “semaphore” to limit concurrency:

// src/gateway/bulkhead.ts
export class Bulkhead {
    private active = 0;
    private queue: (() => void)[] = [];

    constructor(private readonly maxConcurrent: number) {}

    async run<T>(fn: () => Promise<T>): Promise<T> {
        if (this.active >= this.maxConcurrent) {
            await new Promise<void>((resolve) => this.queue.push(resolve));
        }
        this.active++;

        try {
            return await fn();
        } finally {
            this.active--;
            const next = this.queue.shift();
            if (next) next();
        }
    }
}

And using it for services:

// src/gateway/clients.ts
import { Bulkhead } from "./bulkhead";

const giftBulkhead = new Bulkhead(10);      // up to 10 concurrent
const commerceBulkhead = new Bulkhead(3);   // checkout is heavily limited
const analyticsBulkhead = new Bulkhead(50); // can be high

export async function callGiftWithBulkhead(fn: () => Promise<any>) {
    return giftBulkhead.run(fn);
}

export async function callCommerceWithBulkhead(fn: () => Promise<any>) {
    return commerceBulkhead.run(fn);
}

Thus, even if GPT decides to “ask for 30 complex gift recommendations at once”, they will run at most 10 concurrently, and checkout will keep working using its separate limit.

GiftGenius: which compartments we want

In GiftGenius, it makes sense to create separate compartments for:

• Gift recommendations (LLM-heavy, less critical, can be slowed down).
• Checkout/ACP (super-critical, must be protected as much as possible).
• Analytics/logging (important, but can tolerate some delay).

In a more advanced architecture you would deploy them as separate clusters with separate resources, but in this lecture what matters is the idea: do not let secondary features “consume all the oxygen”.

These three patterns — timeouts, circuit breakers, and bulkheads — are about how you call out to your dependencies. But there is another class of resilience threats: incoming event streams that can overwhelm you even if your outbound calls are perfectly tuned. The most common example is webhook storms.

5. Webhook storms: when the world sends you events faster than you can handle

How webhooks behave in reality

The fourth source of resilience issues is incoming events: webhooks from ACP, Stripe, and other systems. They can create a real “storm” even if you already have timeouts, circuit breakers, and bulkheads in place.

Webhooks are not an HTTP “on-demand request”, but “push” events from external systems (Stripe, ACP, external shops, etc.). They have several unpleasant properties:

• At-least-once delivery — duplicates are inevitable.
• Delivery order is not guaranteed.
• On errors they like to retry: first after a second, then after 10, then after a minute… until you respond with 2xx.
• At peak times (e.g., on a sale) they come in bursts, creating a “storm”.

If your handler is not idempotent and takes too long, it becomes a bottleneck, the whole queue clogs, and retries only amplify the storm. As a result, you can take down your database, the queue, worker pools — and by chain reaction the rest of the system.

Basic principles for protecting against storms

There are several ideas that greatly increase your chances of surviving a storm:

First, queue-first, process-later. Ideally, an incoming webhook should not perform heavy work synchronously. Instead, it should validate the signature/schema as quickly as possible, push a task to a queue, and return 200 OK. Processing happens asynchronously in a worker. If you need a “fast confirmation” for ChatGPT, you can keep a separate notifications path.

Second, idempotent handlers. A repeated webhook for the same operation must not “create the order again” or “charge twice”. Usually this is solved by storing an idempotency key or eventId and checking whether we have already processed this event.

Third, rate limiting and a circuit breaker on the receiver. Even if the sender is storming, you can:

• limit RPS by IP/subscription/endpoint,
• temporarily return 429 or 503 to slow down retries,
• use a breaker to avoid pouring traffic into a broken downstream (e.g., the orders DB).

Example of a Next.js webhook handler in GiftGenius

Suppose we have an ACP/payment provider that sends a webhook about an order status to POST /api/commerce/webhook. We want to:

• accept the event quickly and push it to a queue,
• not process it synchronously,
• not break on duplicates.

A simplified example (without signature verification and a real queue — those will be in the modules on security and queues):

// app/api/commerce/webhook/route.ts
import { NextRequest, NextResponse } from "next/server";

// Here we could have Redis/queue; for now we mock with an array
const inMemoryQueue: any[] = [];
const processedEvents = new Set<string>(); // idempotency (demo)

export async function POST(req: NextRequest) {
    const event = await req.json();

    const eventId = event.id as string;
    if (processedEvents.has(eventId)) {
        return NextResponse.json({ ok: true, duplicate: true });
    }

    // In reality, there will be signature and schema verification

    inMemoryQueue.push(event); // push to a queue for background processing
    // A background worker will process later and mark the ID as processed
    return NextResponse.json({ ok: true });
}

While this is a pseudo-implementation, two points matter:

1. The synchronous part is as lightweight as possible.
2. We build idempotency around event.id.

In real life you will:

• use an external queue (SQS, RabbitMQ, Kafka),
• store processed events in a DB,
• verify the webhook signature and payload schema version,
• possibly apply a dedicated Bulkhead/Breaker around the handler.

What this looks like in the context of GiftGenius

For GiftGenius integrated with ACP/Stripe via webhooks, protection against storms is especially important in peak seasons (New Year, Black Friday). There are many events:

• creating intents,
• payment confirmations,
• cancellations,
• refunds.

If your handler starts “stretching” (for example, due to calls to external APIs), you risk:

• ACP starting to retry,
• events arriving in batches,
• your orders DB and worker pool getting clogged.

The “queue first” pattern + idempotency + rate limiting at ingress serves as insurance against such scenarios.

6. How these patterns work together

Now let’s assemble these patterns into one scenario and see how they work in a real flow: “Recommend a gift and immediately place an order.”

Consider the chain “ChatGPT → Gateway → Gift Service → Commerce → webhooks” in this scenario:

The user in chat says: “Pick a gift and immediately place an order.”

1. The model decides to call your tool suggest_and_checkout.
2. The Gateway calls the gift service via fetchWithTimeout and the gift service’s bulkhead.
3. If the gift service hangs, the timeout triggers; the breaker around it will switch to open after a number of errors, and subsequent requests will immediately receive an MCP error like “gift_service_unavailable”.
4. If the gift service responds, the Gateway calls the commerce service (again with a timeout and its own bulkhead).
5. Any issues with commerce trigger a separate circuit breaker, configured stricter than the gift one (because checkout is critical).
6. A successful order results in a webhook from ACP to your /api/commerce/webhook, which enqueues the event and responds quickly; background workers process the payment, and repeated webhooks with the same eventId are ignored as duplicates.

As a result:

• A hanging recommendation service does not take down checkout.
• A hanging commerce service does not turn all tool-calls into a minute-long spinner — ChatGPT quickly receives a meaningful error.
• Webhook storms do not break your primary HTTP path.
• You control where degradation happens: it is better to temporarily disable personalized recommendations than to break payments.

7. A small practical checklist for your App (narrative form)

To generalize, in a typical ChatGPT App with MCP/Gateway it makes sense to go through the following questions in order.

First, check whether you have timeouts on all external calls. All fetch code, DB queries, and LLM calls should use a wrapper like fetchWithTimeout with sensible values. It is important that there be no places where a request can hang indefinitely.

Next, identify your most fragile dependencies. Typically these are payment providers, ACP, large external APIs, and sometimes your own orders DB. Around them, add a circuit breaker to protect against a flood of repeats to a known-dead service. At the same time, decide how ChatGPT should behave when the breaker is open.

After that, view your resources as “compartments”. Does everything go through one connection pool and one worker pool, or do critical operations (login, checkout) have their own concurrency limits, independent from the recommendation and analytics services? If not, add a basic bulkhead implementation, at least as a coarse limit on concurrent tasks.

Finally, audit all incoming webhooks. Check whether they have an idempotency key or eventId, whether you are attempting heavy work synchronously in the HTTP handler, and whether you can survive a wave of retries if your downstream temporarily fails. If not, move the logic into a queue and background workers.

This sequence yields a substantial resilience boost even without super-complex infrastructure.

8. Common mistakes with timeouts, circuit breakers, bulkheads, and webhook storms

Error #1: missing timeouts “somewhere down below”.
Developers often set a timeout only on the Gateway or only on the frontend, forgetting that inside the backend there is also a DB, external APIs, and LLMs. As a result, the outer request supposedly has a 5-second timeout, but inside a single DB or payment call can hang for minutes, blocking the connection pool and causing cascading failures.

Error #2: gigantic “just in case” timeouts.
Sometimes people set 60–120 second timeouts: “let it finish”. In the ChatGPT context this is almost always bad. The user leaves, the model starts hallucinating, and your resources remain blocked the whole time. A much better approach is an honest failure in 5–10 seconds with a clear explanation.

Error #3: a circuit breaker without a thought-through UX.
Sometimes a breaker is added “to check a box”, but when it trips the user or model gets a cryptic 500, “ECONNREFUSED”, or “axios error”. As a result, GPT cannot adequately explain what is happening and starts inventing things. You should plan the error messages so they are understandable to both people and the model.

Error #4: mixing resources without a bulkhead approach.
A classic scenario: a recommendation (or analytics) service slows down, consumes the entire DB connection pool or the thread/worker pool, and then checkout and login die after it. All because resources are not separated. A lack of even a basic bulkhead approach means a secondary feature can take down the whole production system.

Error #5: treating webhooks like ordinary requests.
Beginners often write a webhook handler like a normal controller: long business logic, calls to third-party APIs, no idempotency. With retries and duplicates this leads to double-processing of events, weird order states, and load-related failures during a storm.

Error #6: ignoring idempotency in commerce scenarios.
It is especially dangerous when a payment webhook can create an order again or update its state again. Without checking an idempotency key and storing an event’s processing status, you will sooner or later get double charges or strange duplicate orders.

Error #7: trying to fix everything with setTimeouts and “magic delays”.
Sometimes people attempt to sidestep race conditions and storm issues by “waiting 100 ms and it will be fine”. In practice, this makes behavior even more unstable and does not protect you from real failures. The right path is explicit timeouts, circuit breakers, queues, and idempotency — not delay shamanism.

Error #8: no prioritization of critical paths.
When checkout and login live under the same limits as analytics or the recommendation logic, any overload can take down both the critical and the secondary parts. In a resilient design, checkout and auth are “sacred cows”: they get separate resources, separate limits, separate alerts, and SLOs.