Gateway and perimeter defense: proxy, rate limiting, queues, and backpressure

1. Why bother protecting the ChatGPT App perimeter

In a classic web application, the user is a browser that hits your endpoints in a fairly predictable way. In the world of ChatGPT Apps, you get a new type of client: an LLM that decides on its own when and which tools to call.

The model may:

• call the same tool multiple times in a single conversation;
• experiment: “what if we call suggest_gifts again with slightly different parameters?”;
• work in parallel for hundreds of users.

Add in potential bots, test scripts, and bugs in your own code (for example, an infinite loop that constantly triggers a tool‑call), and you get a near‑perfect recipe for a voluntary DoS.

The cherry on top is cost. Each tool‑call can:

• hit paid external APIs (couriers, payments, catalogs),
• invoke other LLMs (e.g., RAG search),
• launch heavy background jobs.

Without limits and perimeter protection, a single “unlucky” client can:

• take down all your backend services behind the gateway (Gift API, Commerce API, etc.),
• exhaust external API quotas,
• and noticeably burn your model budget.

The goal of this lecture is to show how gateway/proxy + rate limiting + queues + backpressure turn this potential disaster into a manageable system.

Insight

The ChatGPT platform does not provide any mechanisms to protect your MCP server from external traffic. Any internet client can send requests to it, including utilities like MCP Jam.

All ChatGPT can offer is to limit incoming traffic by IP addresses by configuring a reverse proxy (e.g., NGINX) with an allowlist. If IP filtering is not configured, your MCP server remains completely open, which is unsafe. For you and for your users.

2. Proxy/Gateway as a “shield” in front of backend services and agents

First, let’s recall the diagram, now through the lens of protection.

Imagine a typical layout:

flowchart LR
  ChatGPT["ChatGPT / Widget"]
    --> GW["MCP Gateway (Auth, Rate Limit, Logs)"]

  GW --> GiftAPI["Gift REST API (gift selection)"]
  GW --> CommerceAPI["Commerce REST API (checkout, ACP)"]
  GW --> Analytics["Analytics Service / REST API"]

  GW --> Queue["Task queue"]
  Queue --> Worker["Background workers"]

The gateway sits between the outside world (ChatGPT, webhooks, test clients) and everything else. It:

• sees absolutely all incoming requests;
• is first to validate the token and request format;
• can drop clearly invalid stuff (bogus host, strange path, overly large body);
• decides which internal REST/HTTP service the request is even worth sending to.

At this same layer you also get:

• rate limiting — limit how many requests can be made per time window;
• basic backpressure — refuse if downstream services are already choking;
• a switch to asynchrony — heavy tasks go straight to a queue; the client gets “accepted, waiting.”

So the gateway is not only a “router,” but also a “bulletproof vest.” The main thing is not to turn it into a “monolith of all business logic,” as we already discussed in the previous lecture.

3. Which traffic flows need to be controlled

In a ChatGPT App ecosystem, there are usually three main types of traffic that matter in terms of limits and protection.

First, MCP tool‑calls from ChatGPT. That’s everything coming through the MCP protocol: calls to suggest_gifts, get_product_details, create_checkout_session, and other tools. The model can generate them quite briskly, especially when there are Agents under the hood.

Second, outgoing requests from our backends to external APIs. Inside the service we might have our own rate limits for third‑party systems: catalogs, logistics, payments. Violating them can get you throttled, fined, or degraded quality.

Third, incoming webhooks — notifications from ACP, payment providers (Stripe, etc.), couriers. They arrive independently of user activity. If our endpoint is slow or responds with errors, the external system will start making retries and can create a “storm” of repeated notifications.

For GiftGenius this looks like:

• the user and the model actively hit suggest_gifts and find_similar_gifts;
• the checkout tool calls the ACP/commerce backend;
• after payment, the provider sends webhooks payment.succeeded / payment.failed.

All these flows converge at a single point — the Gateway, which is precisely where it makes sense to put “counters, filters, and circuit breakers.”

4. Rate limiting: basic protection and saving money

What rate limiting means in our context

Rate limiting is a mechanism that caps the number of requests from a specific client per unit of time. The idea is as old as the internet, but in the context of ChatGPT Apps it immediately solves three problems:

• prevents a single client (or bug) from taking down your services;
• helps you observe external API limits;
• protects your wallet from uncontrolled model calls.

Classic algorithms include:

• fixed window,
• sliding window,
• token bucket,

and we mostly need to know them conceptually: “no more than N requests during a minute,” “each request consumes a token, tokens refill at X per second,” etc. A library or an API Gateway usually provides the implementation.

Where to place limits

Limits can be applied at different levels.

At the reverse proxy level (Nginx, Cloudflare, AWS API Gateway) it’s convenient to:

• cut off the wildest traffic by IP;
• limit the request body size;
• defend against simple DDoS patterns.

At the MCP Gateway (application) level it’s useful to do more “meaningful” rate limiting:

• by user (userId from the token),
• by organization (tenantId),
• by operation type (for example, hard‑limit create_checkout_session, be softer on search),
• by source (webhook vs tool‑call).

You can also add limits within individual microservices for especially expensive operations, but that’s the next level of detail.

How to choose a key for limits

The most common mistake is to limit by IP address. In the case of ChatGPT this is rather useless:

• all requests can originate from a single OpenAI range,
• different users will “sit” behind the same IP.

We’re much more interested in:

• userId — a specific user in your app;
• tenantId — the organization (if you do B2B and one chat is used by many employees);
• an API token or clientId if you have multiple integrations.

In GiftGenius, userId + tenantId, extracted from the token that ChatGPT passes in MCP calls, is usually sufficient.

Simple rate limiting in TypeScript

Suppose we have a small MCP Gateway on Express. Let’s add the simplest rate limiting: no more than 30 tool‑calls per minute per user.

// Primitive rate limiting: N requests per minute per userId
const WINDOW_MS = 60_000;
const MAX = 30;
const hits = new Map<string, { ts: number; count: number }>();

function rateLimit(req: Request, res: Response, next: NextFunction) {
  const userId = (req.headers["x-user-id"] as string) ?? "anonymous";
  const now = Date.now();
  const rec = hits.get(userId) ?? { ts: now, count: 0 };

  if (now - rec.ts > WINDOW_MS) {      // Window expired — start over
    rec.ts = now;
    rec.count = 0;
  }
  rec.count += 1;
  hits.set(userId, rec);

  if (rec.count > MAX) {
    return res.status(429).json({
      error: "rate_limit_exceeded",
      retryAfterSec: 60,
      message: "Too many tool calls, please retry later."
    });
  }
  next();
}

Now let’s use it in the MCP route:

// Apply middleware to all MCP tool-calls
app.post("/mcp/tools/call", rateLimit, async (req, res) => {
  const result = await callBackendForTool(req.body); // REST call to Gift/Commerce/Analytics API
  res.json(result);
});

Key points:

• we return a meaningful error (error: "rate_limit_exceeded"), not just a 500;
• the model can read this error, understand what happened, and explain it properly to the user instead of hallucinating.

In real production, counters of course don’t live in a single process’s memory; they live in Redis or another shared store so everything works in a cluster. But this is enough to understand the principle.

Rate limiting and gateway‑level limits protect us from request avalanches, but they don’t solve another problem — individual operations can still be very heavy and long‑running. Here synchronous HTTP isn’t enough, and queues and asynchronous jobs enter the stage.

5. Queues and asynchronous jobs: when synchronous is no longer viable

The ChatGPT timeout problem

Even if you carefully configure rate limiting, ChatGPT (and HTTP clients in general) don’t like when responses take too long. The platform limits tool‑call execution time, and if you wait for some “super‑recommendation” algorithm to finish, then:

• the user sees an endless spinner;
• the platform aborts the request by timeout;
• the model decides that “something went wrong” and starts making things up.

The solution is to move heavy operations to an asynchronous mode. The classic pattern:

1. The gateway accepts the request.
2. It enqueues a job.
3. It immediately returns a 202 Accepted with a jobId.
4. A separate worker takes jobs from the queue and processes them.
5. The client (our widget or even ChatGPT via an additional tool) periodically asks for status by jobId or receives a notification via an MCP event.

In ChatGPT App terms, this usually looks like two tools: the first tool accepts the request, enqueues a job, and returns a jobId; the second lets the model or widget query status and retrieve the result by that jobId. Additionally, you can duplicate progress updates via MCP notifications.

Mini‑queue for GiftGenius (code example)

Let’s say we have a heavy tool generate_large_gift_report that can run for tens of seconds. In a real app it would just return a jobId, and a separate tool get_report_status would let the model or widget query that jobId for state and pick up the result. At the Gateway level we’ll create a separate endpoint with a queue for it.

type Job = { id: string; payload: any };
const queue: Job[] = [];
const MAX_QUEUE = 100;

app.post("/mcp/tools/generate_report", (req, res) => {
  if (queue.length >= MAX_QUEUE) {
    return res.status(503).json({
      error: "system_busy",
      message: "System is busy, please retry later."
    });
  }

  const job: Job = { id: crypto.randomUUID(), payload: req.body };
  queue.push(job);
  res.status(202).json({ jobId: job.id, status: "accepted" });
});

And a primitive worker that takes one job every 200 ms:

async function processJob(job: Job) {
  // Call the real backend service or an agent workflow via REST here
  await handleHeavyGiftReport(job.payload);
}

setInterval(async () => {
  const job = queue.shift();
  if (!job) return;
  await processJob(job);
}, 200);

Obviously, this is heavily simplified:

• in real life the queue lives in Redis, SQS, Kafka, etc.;
• the job status is stored somewhere else so it can be queried;
• there are usually multiple workers.

But the concept is clear: the gateway doesn’t keep the request open until everything is done. It accepts, enqueues, and replies quickly.

6. Backpressure: how not to drown in your own queue

How backpressure differs from rate limiting

Rate limiting primarily answers: “how many requests can one client make per time interval?” It protects against “one overly active user” or a bug on a specific client.

Backpressure says: “how many tasks/requests can our system handle concurrently in total without falling apart?” This is about the overall volume of load, regardless of where it comes from.

Example:

• rate limiting: “a user cannot call suggest_gifts more than 30 times per minute”;
• backpressure: “the queue cannot have more than 100 pending tasks, otherwise we start rejecting new requests.”

Ideally, these mechanisms complement each other: rate limits keep clients in check; backpressure saves the system if a ton of traffic still arrives.

A simple implementation of limiting active tasks

One of the simplest forms of backpressure is to limit the number of active calls under the hood. For example: don’t hold more than 50 concurrent tool‑calls to a specific backend/REST service (Gift API, Commerce API, etc.) at the same time.

let activeCalls = 0;
const MAX_ACTIVE = 50;

app.post("/mcp/tools/call", async (req, res) => {
    if (activeCalls >= MAX_ACTIVE) {
        return res.status(429).json({
            error: "gateway_overloaded",
            message: "Gateway is temporarily overloaded, please retry later."
        });
    }

    activeCalls += 1;
    try {
        const result = await callBackendForTool(req.body); // REST call to Gift/Commerce/Analytics API
        res.json(result);
    } catch (err) {
        console.error("Tool call error", err);
        res.status(500).json({ error: "internal_error" });
    } finally {
        activeCalls -= 1;
    }
});

What’s happening here:

• as long as concurrently running requests are fewer than MAX_ACTIVE, we allow a new call through;
• if the limit is exhausted, we immediately respond with a meaningful error;
• it’s important to decrement the counter in finally so you don’t lose “slots” on errors.

This is the simplest backpressure: we honestly tell the client: “can’t right now, try later,” instead of mindlessly accepting everything and crashing.

Later you can:

• set different MAX_ACTIVE values for different operation types (e.g., almost always let checkout through, enforce a stricter cap on report generation),
• switch limits dynamically based on load metrics.

7. Webhooks and “storms”: protecting incoming events

So far we’ve mostly looked at requests we or ChatGPT initiate (tool‑calls, outgoing requests, async jobs). But there’s another important source of load on the gateway — incoming webhooks from external systems.

Webhooks are the flip side: if we (via the model) initiate tool‑calls, a webhook is initiated by an external service. This is that third traffic type from section 4 that we don’t control in timing or frequency, but must be able to digest without falling over. Payment, ACP, logistics — they all send notifications (webhooks) to our endpoint for each important change: “payment succeeded,” “order created,” “delivery status updated.”

Problems start when:

• our endpoint responds slowly;
• it responds with an error;
• it’s periodically unavailable.

Then the external service, following best practices, starts making retries. If you’re unlucky, you’ll get a “webhook storm” — dozens or hundreds of repeated events trying to “reach” you at any cost.

To avoid getting crushed by such care, at the gateway level you should:

1. Rate limit incoming webhooks by source: for example, “no more than 10 events per minute per event_type from a specific provider.”
2. Verify signatures before parsing JSON: an HMAC signature or similar mechanism lets you drop fake requests.
3. Make event handling idempotent: by event_id or a similar field so repeats don’t create duplicate orders or payments.
4. Under a severe storm, enable additional backpressure: temporarily respond with “503: try later” if downstream services can’t keep up.

The simplest example (the idea, not production code):

app.post("/webhooks/stripe", rateLimitWebhook, (req, res) => {
    const sig = req.headers["stripe-signature"] as string;
    if (!isValidSignature(req.rawBody, sig)) {
        return res.status(400).send("Invalid signature");
    }

    const event = JSON.parse(req.body.toString());
    if (isAlreadyProcessed(event.id)) {
        return res.json({ received: true }); // idempotency
    }

    handleStripeEvent(event);
    res.json({ received: true });
});

Here at the gateway level we:

• apply a separate rate limiting policy for webhooks;
• validate the signature before trusting the payload;
• protect against duplicates via isAlreadyProcessed.

8. Applying it to GiftGenius: example of limit and queue policies

Now let’s step away from abstractions and see how this might look for our training project GiftGenius.

Imagine three key scenarios:

1. Gift search (suggest_gifts, find_similar_gifts).
2. Order creation / checkout (create_checkout_session, confirm_order).
3. Receiving webhooks from the payment provider and ACP.

For each scenario it’s logical to define:

• which key we count the limit by;
• how many requests per minute we allow;
• what we do when the limit is exceeded.

For example:

For webhooks it’s usually more sensible to limit by the combination “provider + event type,” and filter duplicates with a separate idempotency mechanism by event_id.

In code this turns into different middleware: rateLimitSearch, rateLimitCheckout, rateLimitWebhook.

For heavy operations like “generate a large annual PDF gift report,” we use a queue and the asynchronous pattern shown above. In such a case, the gateway:

• accepts the request from ChatGPT;
• enqueues a job;
• returns a jobId and a hint for the model on how to get the status;
• limits the queue size (backpressure) so the system doesn’t overflow.

Keep in mind: both rate limiting and backpressure are not only about security and reliability, but also about UX. It’s much nicer to hear from the assistant, “The service is overloaded right now; let’s try again in a minute,” than to sit with a spinner until timeout or see “Internal Server Error.”

9. Mini‑practicum: adding protection to our MCP Gateway

To make sure the material doesn’t remain theoretical, let’s assemble a mini‑practicum you can implement in your training project.

Rate limiting for all MCP tool‑calls

Add a rateLimit middleware (like above) and attach it to /mcp/tools/call. To start, you can use a very simple limit: 30 requests per minute per userId. Then experiment:

• lower the limit and see how your App and the model react;
• set different limits for different tool types by passing, for example, toolName to the middleware.

The simplest backpressure by active calls

Add an activeCalls counter and a MAX_ACTIVE cap. Try simulating load (for example, with a script that sends a batch of requests) and see when the gateway starts responding with gateway_overloaded.

What matters here is the behavior: you don’t wait until everything falls over — you refuse to take new tasks, honestly telling the client it’s too hot right now.

A queue for a heavy tool

Pick one heavy operation (or make one “heavy” artificially — by inserting setTimeout/a long fetch) and switch it to the “queue + jobId” pattern. At a minimum:

• endpoint POST /mcp/tools/generate_report — enqueues a job and returns a jobId;
• endpoint GET /jobs/:id — returns status (pending, done, error, and possibly the result);
• a worker that calls processJob every X milliseconds.

That’s enough to understand how a real integration with BullMQ or another queue engine will look.

10. Common mistakes when protecting the perimeter

Mistake #1: Limiting only by IP.
In the world of ChatGPT Apps this is almost useless: most requests come from OpenAI addresses, and all your users end up behind the same IP. As a result, one person can burn the limit for everyone while the real culprit remains unknown. It’s better to limit by userId, tenantId, or a token, and use IP only as a very coarse filter at the reverse proxy layer.

Mistake #2: Returning a bare 500 instead of a meaningful error.
If, upon exceeding limits or during overloads, you just send 500 Internal Server Error, the model won’t understand and will start making things up. A structured error with a code (rate_limit_exceeded, gateway_overloaded) and a human‑readable description lets the LLM properly explain the situation to the user and, if needed, try again later.

Mistake #3: Building an infinite queue without backpressure.
Sometimes it seems: “let’s just queue everything and sort it out later.” In practice, the queue grows to thousands of tasks, latency increases, memory runs out, and users never see results. Always limit queue size and the number of active operations. It’s better to honestly reject new requests with 503 or 429 than to turn the queue into a black hole.

Mistake #4: Relying only on rate limiting and ignoring webhooks.
Many protect only incoming traffic from ChatGPT while leaving webhooks to “sort themselves out.” When a payment provider starts retrying, webhooks can cause a real storm. Webhook endpoints need their own limits, signature checks, and idempotent processing. Otherwise, it’s easy to get a dozen duplicates of the same order.

Mistake #5: Keeping all counters and the queue only in a single instance’s memory.
For a training project that’s fine, but in production, when you scale the gateway to multiple instances, counters on each node will “live their own lives,” limits will stop being global, and restarting a node will wipe the queue. In real systems, state for limits and queues lives in a shared store (Redis, cloud queues, etc.). We’ll talk more about this in lectures on scaling and production.

Mistake #6: Stuffing business logic into the gateway “since it’s already in the middle of everything.”
It’s tempting: “let’s decide which gifts to show right in the gateway; requests go there anyway.” You end up with a gateway turned into a monolith packed with logic — simultaneously a router, business brain, and logger. This makes scaling and maintenance much harder. The gateway should remain a network/infrastructure layer: authentication, authorization, limits, cache, routing — yes; gift selection — no.

Mistake #7: Thinking “we’re small; this doesn’t apply to us.”
People often think: “We don’t have a million users; we can do without a gateway/limits.” In reality, even a single bug in client code (or in a prompt that makes the model call a tool in a loop) can cause a small but very local apocalypse. Basic rate limiting and at least primitive backpressure are not luxuries — they’re the toothbrush of production: use them from day one, before it starts hurting.