Scaling and deployment: load balancing, backend service clusters, blue/green and canary

1. What this lecture is about and why it matters

Imagine you left GiftGenius at the stage where it lives alone on Vercel: one MCP gateway instance (which both exposes MCP externally and talks to your REST services), one backend for agents, everything “sort of works.” That’s tolerable for a pet project and the first 100 users.

But as soon as OpenAI adds your App to the Store and it suddenly lands in a front-page collection before Christmas, “one gateway on port 3000” turns into a very sad story: a queue of tool calls, timeouts, 500 errors, a rating drop in the Store, and emails from marketing like “why was everything down at the peak of sales?”

Our goal in this lecture is to learn to think about GiftGenius (and any ChatGPT App) as a system made of many identical instances behind a load balancer. Plus, we’ll cover careful release strategies and a clear plan for “how to roll back if something goes wrong.”

2. Horizontal scaling and stateless design

Let’s start with the basic idea: if your MCP Gateway or an internal backend service stores important state in the memory of a specific process, it’s almost impossible to scale it horizontally properly.

Vertical vs horizontal scaling

First, let’s clarify the terminology.

Vertical scaling is when you simply “beef up” a single server: more CPU, more RAM. It’s quick, sometimes cheap at the start, but it has a hard limit and makes one instance a single point of failure: if that big powerful box goes down, everything goes down.

Horizontal scaling is when you run multiple instances of a service behind a load balancer. Each instance is relatively small, doesn’t keep anything critical in memory, and state moves to external stores (Postgres, Redis, object storage). You can freely add or remove instances under load.

For the MCP Gateway and backend services (Gift REST API, Commerce REST API, Analytics Service / REST API, etc.) horizontal scaling is essentially mandatory: ChatGPT can suddenly send you many times more traffic (seasonality, Store promo, a viral TikTok), and you should just add instances instead of “praying that one server holds.”

What a stateless service means for the MCP Gateway and backends

For horizontal scaling to work, a service should be as stateless as possible.

Stateless in our context means:

• the service does not store unique, long‑lived user state in process memory that business logic depends on;
• any important state lives in an external database, queue, cache, or S3‑like storage;
• if a specific instance crashes, another instance can keep serving the user by simply “picking up” context from the external store.

For GiftGenius this means:

• the user’s gift selection history, likes/dislikes, and cart live, for example, in Postgres;
• queues for long‑running tasks (bulk generation of selections, email campaigns) live in a broker like Redis/Cloud Queue;
• if there is a separate service for complex agent workflows, it stores checkpoints and long‑term memory in its own store, not in the RAM of a single process.

An MCP Gateway or any backend service instance becomes “cattle, not pets”: you can ruthlessly kill and recreate it without losing business data.

Mini example: moving state from memory to external storage

Suppose you once made a very simple MCP tool add_to_cart that calls internal logic via the gateway, and that logic stores the cart in process memory (yes, demos sometimes do this — and that’s fine as long as you know you can’t do that in prod):

// BAD: cart in the backend service process memory
const inMemoryCarts = new Map<string, string[]>();

export async function addToCart(userId: string, sku: string) {
  const cart = inMemoryCarts.get(userId) ?? [];
  cart.push(sku);
  inMemoryCarts.set(userId, cart);
  return cart;
}

Horizontal scaling here is impossible: one request hits instance A, another hits instance B, and the user ends up with different carts.

The right approach is to move the cart to an external DB or cache. Roughly (heavily simplified):

// GOOD: cart in external storage
import { db } from "./db";

export async function addToCart(userId: string, sku: string) {
  await db.cartItems.insert({ userId, sku }); // simplified
  const cart = await db.cartItems.findMany({ where: { userId } });
  return cart;
}

Now it doesn’t matter which backend instance handles the request coming through the gateway: the cart is the same for all.

3. Load balancing: how traffic reaches backend clusters

As soon as you have more than one instance of a service, you need something that distributes requests among them. It’s like an order dispatcher in a busy pizzeria: lots of couriers, lots of customers, and without logic — chaos.

L4 vs L7, and why we mostly care about L7

A load balancer can operate at different layers:

• L4 (TCP/UDP) just forwards bytes from the client to one of the backends without really understanding the protocol;
• L7 (HTTP) understands that it’s dealing with an HTTP request, can look at the path, headers, cookies, and sometimes even the body.

For a ChatGPT App architecture with an MCP Gateway and REST services, an L7 load balancer is almost always what we want: everything talks over HTTP/SSE, and we want to route by path, domain, headers (e.g., for canary releases), and perform health checks.

Health checks and removing unhealthy instances from rotation

A load balancer should periodically verify instances are healthy. The simplest way is to have a GET /health or /readyz endpoint that returns 200 OK if everything is fine.

In a Node/TypeScript service acting as an MCP Gateway or backend, a health check can look like this:

// apps/gateway/src/http/health.ts
import { type Request, type Response } from "express";

export function healthHandler(req: Request, res: Response) {
  res.json({
    status: "ok",
    version: process.env.RELEASE_ID ?? "dev",
  });
}

The load balancer pings /health every N seconds. If responses start coming back as 5xx or timing out, that instance is removed from rotation and new traffic doesn’t go there.

Considerations for Streaming / SSE

The MCP Gateway often works via SSE (Server‑Sent Events), especially if you stream partial results. The load balancer should:

• support long‑lived HTTP connections;
• be able to account for such connections when choosing an instance (some LBs can consider the number of active connections, not just RPS).

This matters because a single “chatty” tool call that streams text for 2 minutes hangs as an active connection. If there are too many such connections on one instance, that instance should be temporarily “unloaded” — send new connections elsewhere.

4. Backend service clusters: split by tasks, not “everything in one pot”

A logical next step is to stop thinking about one “big backend service” and split the system into several clusters depending on the nature and criticality of the load.

Example GiftGenius architecture by clusters

All the material collected in module 16 recommends this layout for GiftGenius:

Essentially, this is the bulkheads pattern: if cluster B suddenly starts “burning CPU tokens” while generating complex texts, cluster C with payments keeps working because it has its own resource pool and scaling.

How this looks through the Gateway

The MCP Gateway described in the first lecture of the module sees all incoming MCP traffic and routes it across backend clusters. Roughly:

• tool calls list_gifts, suggest_gifts → cluster A (Gift REST API);
• tool calls generate_greeting_card or complex agent workflows → cluster B (Agents REST service or workers);
• tools create_order, confirm_payment → cluster C (Commerce REST API).

Behind this there can be a single shared load balancer or multiple load balancers (e.g., a dedicated L7 LB in front of commerce for even stronger isolation).

We can sketch the big picture:

flowchart LR
    ChatGPT((ChatGPT))
    GW[MCP Gateway]
    LBA[LB Gift API Cluster A]
    LBB[LB Agents/Workers Cluster B]
    LBC[LB Commerce API Cluster C]

    A1[Gift REST API A-1]
    A2[Gift REST API A-2]
    B1[Agents Service B-1]
    B2[Agents Service B-2]
    C1[Commerce REST API C-1]
    C2[Commerce REST API C-2]

    ChatGPT --> GW
    GW -->|tools: gifts| LBA
    GW -->|agents workflows| LBB
    GW -->|commerce| LBC

    LBA --> A1
    LBA --> A2
    LBB --> B1
    LBB --> B2
    LBC --> C1
    LBC --> C2

The diagram is slightly idealized, but it reflects the main principle: different types of load — different backend clusters behind a single MCP Gateway.

5. Deployment strategies: why we need blue/green and canary

Now let’s move on to how to update all this in a way users don’t notice — and you can sleep at night.

Anti‑example: “deploy over production”

The simplest and most dangerous strategy: you take the active cluster (e.g., Gift REST API cluster A), launch a new image over the old one, swap containers or restart processes.

What’s wrong with this:

• while some instances are new and others are old, the system may behave unpredictably (especially if the DB schema changed);
• if something goes wrong, rollback is a new deploy “as it was,” which can take minutes;
• at the moment of deployment you can easily get a brief downtime when no instance is up yet.

In Kubernetes and PaaS this is softened a bit by rolling updates, but the idea is the same: without a clear strategy you get a long “gray zone” where different code versions handle traffic simultaneously.

Blue/green deploy: two environments and instant switching

Blue/Green is an approach where you have two almost identical environments at the same time: Blue (current production) and Green (the new version).

Schematically, the process looks like this:

1. Deploy the new version (v2) into the Green environment: the same set of gateway + backend clusters, just without real traffic yet.
2. Run all necessary tests on Green: automated tests, smoke scenarios, manual checks via ChatGPT Dev Mode.
3. At release time, switch the load balancer/routing so that 100% of production traffic goes to Green.
4. Blue stays alive nearby as a “backup runway.” If something goes wrong, you switch traffic back in seconds.

For GiftGenius this could look like: you have mcp-gateway-blue.example.com and mcp-gateway-green.example.com. The ChatGPT App in prod points to the official MCP endpoint (gateway), and at release you update the DNS/LB config so that the domain name mcp-gateway.example.com points to green.

Pros:

• instant switch back and forth;
• you can fix any problem after rolling back;
• no state where “half the cluster is new, half is old.”

Cons:

During release you need to maintain two full environments, i.e., pay for resources ×2. So this strategy is typically applied to critical backend services — for example, commerce cluster C and the MCP Gateway itself, where you must not break checkout or the entry point under any circumstances.

Canary releases: a small “canary in the coal mine”

A canary release is a more economical option: you don’t spin up two full productions; you roll out the new version gradually to a small portion of traffic and watch it closely.

A rough scenario:

1. Deploy version v2 of Gift REST API cluster A into the same pool or a separate small canary pool.
2. Configure the load balancer or the MCP Gateway so that, say, 1% of tool calls related to gifts go to v2, and 99% go to v1.
3. Watch the metrics: error rate, latency, specific business metrics (conversion, successful checkouts).
4. If all is good — gradually increase the share: 1% → 5% → 10% → 50% → 100%. If not — roll back immediately.

In the context of ChatGPT Apps, canary is especially useful not only for code, but also for prompt experiments: a new system prompt version for an agent service can radically change behavior, and it’s better to test it on a small subset of users first.

The Gateway or LB can decide what counts as “canary” by different signals:

• randomly (e.g., 1% of all requests);
• by userId (some users are in the experiment permanently);
• by a special header or cookie (for internal testing).

A small example of routing logic in pseudo‑TypeScript (to illustrate the idea in the gateway):

// Pseudocode in Gateway: simple random canary 5%
function routeToGiftBackendCluster(ctx: { userId?: string | null }) {
  const rnd = Math.random();
  if (rnd < 0.05) {
    return "gift-api-v2"; // canary
  }
  return "gift-api-v1";   // stable
}

In real life you won’t do this via Math.random() in runtime code; you’ll move the rules to config/feature flags, but the logic is similar: a portion of traffic goes to canary versions of a backend service, the rest goes to stable.

6. Rollback as a mandatory part of the strategy

A long time ago I learned a good rule: rollback should be faster than the fix.

That means if errors spike after a release and users say “everything’s broken,” don’t heroically fix the bug in prod. Press the big red “roll back” button.

On platforms like Vercel (where we’ve already deployed the Next.js part of GiftGenius), this is natural: every deploy is an immutable artifact, and Vercel lets you roll back quickly to the previous one.

For the MCP Gateway and backend clusters deployed in Kubernetes or another orchestrator, kubectl rollout undo plays that role: you roll back to the previous set of pods and images.

The main thing is to log and expose the version currently serving traffic. For example, you can:

• add a version to /health and other diagnostic endpoints (we already did this above);
• propagate the release ID via headers into logs (e.g., X-Release-Id).

A tiny example: a Next.js API route that returns the build version for inspection by the ChatGPT App inside a widget:

// apps/web/app/api/version/route.ts
export async function GET() {
  return Response.json({
    version: process.env.RELEASE_ID ?? "dev",
    builtAt: process.env.BUILT_AT ?? "unknown",
  });
}

This endpoint is also handy for debugging: you can ask a prod instance which version is running right now, without guessing “did the latest build actually roll out?”

7. Capacity planning: how many instances GiftGenius needs

We’ve discussed how to release safely (blue/green, canary) and roll back quickly when things go wrong. A practical question remains: how many instances and which clusters should you run in prod so it can handle real traffic without breaking the bank?

We won’t go overboard with formulas, but a little is necessary. Scaling should be tied to load and economics: how many requests per day/second, how many heavy LLM calls, how much it costs per day.

For simplicity, think in orders of magnitude:

• at 10k requests per day to GiftGenius (roughly 0.1 RPS on average) you can easily live on one or two MCP Gateway instances and a couple of Gift REST API/Agents worker instances;
• at 100k requests per day (1–2 average RPS, higher at peak) you should already have 3–5 instances of the gateway + Gift REST API cluster, a separate cluster B for heavy agents, and a dedicated commerce cluster;
• at 1M requests per day (dozens of RPS, peak loads during holidays) you will definitely need clusters, dedicated resources for LLM agents, an aggressive cache, and an edge layer (that’s a separate lecture).

These are not strict numbers but a way to force yourself to estimate the order of magnitude and think ahead: where the bottlenecks are, how you will scale, and how much it will cost.

For GiftGenius it’s especially important to prepare for holidays: New Year, Christmas, Valentine’s Day, Black Friday. The load can multiply, and you want the system to survive it.

8. Practical mini example: GiftGenius deployment evolution

To bring it all together, let’s sketch a simple evolution of GiftGenius deployment.
We’ll apply everything we discussed: stateless design of the gateway and backend services, load balancing, separate clusters, and release strategies (blue/green, canary).

Base level: one gateway + backend on Vercel/Kubernetes

At some point in the course you already built this: a single Next.js app with the Apps SDK on Vercel, with both the MCP endpoint and simple backend logic (Gift/Commerce) in one service. It’s fairly monolithic.

Pros are obvious: simple, cheap, few places to make mistakes.

The one but critical downside: it does not scale for serious traffic and handles updates poorly.

Level 2: separate MCP Gateway + multiple backend clusters

Next step:

• extract the MCP Gateway into a separate service (Node/Go/NGINX+Lua — doesn’t matter);
• run several instances of the Gift REST API (cluster A) and several workers/services for agents (cluster B);
• split commerce into a separate service (cluster C), possibly with a separate database/infrastructure.

At this stage you already have classic L7 load balancing, health checks, and, where possible, horizontal scaling.

Level 3: release strategies

Here you add:

• Blue/Green for the commerce cluster C (and, if desired, for the MCP Gateway), so checkout and auth remain maximally stable;
• Canary releases for the Gift REST API clusters and the agent service, so you can safely experiment with new versions of tools and agents without risking the entire prod.

Schematically:

flowchart LR
    ChatGPT((ChatGPT))
    GWBlue[Gateway Blue]
    GWGreen[Gateway Green]
    LB[Traffic Switch]

    subgraph Prod
      LB --> GWBlue
      LB -.canary,% .-> GWGreen
    end

    ChatGPT --> LB

In reality it may be a bit more nuanced (separate Blue/Green only for commerce, canary only for gift clusters), but the idea stands: you always know which version goes where, while ChatGPT still sees a single MCP entry point (the gateway).

9. Small code fragments for versioning and diagnostics

We’ve already seen the health endpoint and /api/version. Let’s add another example of how to log the version and cluster in an MCP tool handler on the gateway side, so it’s easy to correlate metrics later.

Suppose the tool suggest_gifts is implemented as a REST endpoint in the Gift REST API and called via the gateway:

import { type McpToolHandler } from "@modelcontextprotocol/sdk";

export const suggestGifts: McpToolHandler<{
  occasion: string;
  budget: number;
}> = async ({ input, meta }) => {
  const releaseId = process.env.RELEASE_ID ?? "dev";
  const clusterId = process.env.CLUSTER_ID ?? "gift-api-A";

  console.log("[suggest_gifts]", {
    releaseId,
    clusterId,
    userId: meta.userId,
    occasion: input.occasion,
  });

  // here the MCP Gateway calls the Gift REST API according to the routing table,
  // and the tool itself remains a thin wrapper around the REST call
  return {
    content: [{ type: "text", text: "Gift ideas..." }],
  };
};

Here we:

• read RELEASE_ID and CLUSTER_ID from env;
• write them to structured logs;
• and it’s easy to use them for analysis later: “on which version/cluster are we seeing more errors right now?”

From the ChatGPT App’s perspective this is transparent, but for you as a developer it’s a huge plus — especially combined with canary/blue‑green.

10. Typical mistakes when scaling and deploying a ChatGPT App

Error #1: storing session/user state in gateway or backend process memory.
This kills horizontal scaling: as soon as you have a second instance, state “splits” between them. It’s especially dangerous to store the cart, search results, or workflow progress in memory. All of this must live in an external store — a DB, cache, or a specialized store for agent state.

Error #2: assuming “one powerful server” is enough.
Vertical scaling is convenient at the start but fails with real growth: the machine has a physical limit, one process becomes a single point of failure, and ChatGPT may bring unpredictable traffic spikes. For MCP Gateway and backend clusters you almost always need a stateless design and multiple instances behind a load balancer.

Error #3: rolling out new versions “over production” without a clear strategy.
If you just update containers/processes in the live cluster, you get an intermediate state where some traffic goes to the old version and some to the new, and rollback becomes “redeploy again.” It’s much safer to keep either two environments (blue/green) or at least a separate canary version of the backend service that receives a small portion of traffic.

Error #4: no fast rollback plan.
Bad scenario: the release went out, metrics are red, users complain, and only then you start thinking how to roll back. Good scenario: a pre‑built capability for instant rollback (blue/green switch, rollout undo, Vercel rollback), clear version identifiers in logs and health endpoints, and a strict rule “roll back first, investigate later.”

Error #5: one shared cluster “for everything” with no split by workload type.
If greeting text generation (LLM agents) and checkout live in one cluster, any model‑side issue (latency, timeouts, token growth) can take payments down too. Splitting into clusters by task type (Gift REST API / light tools, agents‑heavy service, Commerce REST API) and separate limits/resources per cluster is an important step toward resilience.

Error #6: no connection between architecture and economics.
It’s easy to get carried away with “let’s spin a couple more nodes,” forgetting that each LLM call and each instance cost money. Without basic capacity planning (load and cost estimation) you can either under‑scale and crash prod or over‑scale and lose margin. It helps to tie the number of requests, the share of heavy LLM operations, and hosting cost to the app’s business metrics.