Lightweight load tests and feed data quality

1. Why does a ChatGPT App need load testing at all?

In classic web, load testing is often associated with the picture “millions of RPS, a gigantic cluster, pizza for SRE.” For a ChatGPT App and MCP servers, reality is simpler and, luckily, cheaper. You’re already familiar with SLO in principle, but let’s see how SLO/observability and feed quality interact under load.

The key characteristic: ChatGPT waits for a tool call to complete before continuing to generate a response. The user sees a nice token stream, but as soon as the model decides to invoke a tool, the stream magic ends — until the backend responds. If your MCP or ACP server sometimes answers in 8–10 seconds instead of the target 2–4, the UX turns from “magical assistant” into “yet another slow site.”

There’s also a strict timeout budget: for tool invocations, OpenAI keeps an upper bound on the order of tens of seconds (exact numbers depend on the mode, but you should think in the 30–60 second range, and for UX — ideally 5–10 seconds). If, at peak load, your tool calls start landing in 25–30 seconds, you’re still formally within the limit, but from the user’s perspective you already “broke.”

Second point: what matters is not abstract RPS but concurrency. For an app from the Store, it’s quite realistic to have 50–100 concurrent active users; that’s exactly what you want to verify, not “can we handle 50k RPS of a synthetic GET /health.”

And finally, a ChatGPT App is a stack:

flowchart LR
  User --> ChatGPT
  ChatGPT -->|tools/call| MCP["MCP server GiftGenius"]
  MCP --> DB["Gift feed database"]
  MCP --> ACP["Checkout / ACP backend"]
  ACP --> PSP["Payment provider / Stripe"]

If we don’t check how this stack behaves under a small but realistic load, any promo email or being featured in the Store can quickly turn it into a slide titled “how not to build LLM products.”

In this lecture, by “lightweight load tests” we mean short runs (usually 1–10 minutes) that verify:

• whether the system handles the expected peak concurrent load;
• whether p95/p99 latency stays within your SLO;
• whether errors, timeouts, and rate limits from external APIs start popping up.

In parallel, we’ll look at the other side of quality — product feed data (hereafter just “feed”), without which no GiftGenius will be either “Gift” or “Genius.”

We’ll first cover lightweight load tests for MCP/ACP (what exactly to load and how, which metrics to watch), then ground this in observability (latency, errors, resources, webhooks, and logs), and in the second half talk about feed quality and how it unexpectedly bites under load.

2. What exactly to load: not ChatGPT, but your APIs

It’s important to fix one idea in place so we don’t confuse it later: we run load tests directly against our backend — the MCP server, ACP endpoints, webhooks — not through the ChatGPT UI.

There are several reasons.

• First, cost. If you run real tool calls through ChatGPT, you’ll pay for tokens and simultaneously hit ChatGPT limits, even though you’re testing your code.
• Second, predictability. With direct calls to /mcp or /api/checkout you control the scenario, without depending on the model’s decision to call a tool at that moment or not.
• Third, transparency. Under load you want to see clearly: here are 2,000 requests to MCP in 5 minutes, here’s the latency distribution, here’s the CPU chart. If you drive load through ChatGPT, the extra layer of noise and constraints just complicates the picture.

Typical set of endpoints for a GiftGenius load test:

• the MCP server endpoint that implements JSON‑RPC tools (/mcp or similar);
• one or two ACP endpoints to create and finalize checkout (in the payment sandbox mode);
• possibly an endpoint that handles payment webhooks, to see how it behaves at an event spike.

We’ll assume we have a Next.js 16 backend that hosts an MCP server at /api/mcp, and an ACP server with an endpoint /api/checkout/create.

3. A mini smoke‑load scenario for GiftGenius

Suppose our product managers believe in a bright future and say: “A realistic peak is 50 concurrent users, each visits, picks a gift, and sometimes proceeds to payment.”

For a lightweight load test, it’s enough to simulate, say, 30–50 “virtual users” (VU), each of whom performs the sequence:

1. Call the giftgenius.search_gifts tool (search gifts by profile and budget).
2. Call giftgenius.get_gift_details for a couple of products from the result.
3. (Sometimes) call the ACP endpoint create_checkout_session for one product.

All of this directly over HTTP to our MCP/ACP, without ChatGPT.

JSON‑RPC call to MCP

Example request body to MCP (simplified):

const body = {
  jsonrpc: "2.0",
  id: "test-" + Math.random(),
  method: "tools/call",
  params: {
    toolName: "giftgenius.search_gifts",
    arguments: {
      occasion: "birthday",
      budget: 50,
      interests: ["sport", "books"],
    },
  },
};

In a real project the structure may differ slightly, but the principle is the same: one JSON‑RPC method with the tool and its arguments inside.

4. Writing a simple load script in TypeScript

As a first step, let’s implement the simplest part of our scenario — a call to giftgenius.search_gifts on MCP. We’ll first write a minimal Node.js script in TypeScript that sends such requests to /api/mcp and measures latency, and then add checkout and more complex flows.

Basic HTTP client

Suppose we have a .env with MCP_URL=http://localhost:3000/api/mcp.

// scripts/loadTest.ts
import "dotenv/config";

const MCP_URL = process.env.MCP_URL!;

async function callSearchGifts() {
  const body = {
    jsonrpc: "2.0",
    id: `search-${Date.now()}-${Math.random()}`,
    method: "tools/call",
    params: {
      toolName: "giftgenius.search_gifts",
      arguments: { occasion: "birthday", budget: 50 },
    },
  };

  const started = Date.now();
  const res = await fetch(MCP_URL, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(body),
  });
  const latencyMs = Date.now() - started;
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  return latencyMs;
}

You can also add simple JSON response parsing here, but for latency/error rate purposes this is enough.

Concurrent execution of multiple requests

We need to control the number of simultaneous requests. For simplicity, let’s take a fixed number of “virtual users” and have each make N requests in sequence.

async function runVirtualUser(iterations: number) {
  const latencies: number[] = [];
  for (let i = 0; i < iterations; i++) {
    try {
      const ms = await callSearchGifts();
      latencies.push(ms);
    } catch (e) {
      console.error("Error in VU:", e);
      latencies.push(-1); // mark the error
    }
  }
  return latencies;
}

Now we can run, say, 20 such virtual users:

async function main() {
  const users = 20;
  const iterations = 10;

  const tasks = Array.from({ length: users }, () =>
    runVirtualUser(iterations),
  );

  const results = await Promise.all(tasks);
  const all = results.flat();
  // ...metrics calculation
}

main().catch((e) => console.error(e));

This will already produce about 200 MCP calls, some of which will run in parallel — i.e., with fairly high concurrency.

Computing p95 and error rate

Let’s add a small utility to compute percentiles and errors. Recall: p95 is the value below which 95% of requests fall.

function percentile(values: number[], p: number) {
  const sorted = values.filter(v => v >= 0).sort((a, b) => a - b);
  if (!sorted.length) return 0;
  const idx = Math.floor((p / 100) * (sorted.length - 1));
  return sorted[idx];
}

function errorRate(values: number[]) {
  const total = values.length;
  const errors = values.filter(v => v < 0).length;
  return (errors / total) * 100;
}

And in main let’s add output:

const p95 = percentile(all, 95);
const p99 = percentile(all, 99);
const errRate = errorRate(all);

console.log(`Total: ${all.length}`);
console.log(`p95: ${p95} ms, p99: ${p99} ms`);
console.log(`Error rate: ${errRate.toFixed(2)}%`);

Now you have a minimal smoke‑load script that you can run locally or on staging before a release. You don’t touch ChatGPT, don’t burn tokens, and keep the focus on your MCP.

What to do with ACP and checkout

Similarly, you can add another helper callCreateCheckoutSession that will hit the ACP endpoint. It’s important to use the test/sandbox mode of the payment system to avoid creating real orders. A typical call will look like a regular POST with JSON:

async function callCreateCheckoutSession(productId: string) {
  const started = Date.now();
  const res = await fetch("http://localhost:3000/api/checkout/create", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ productId, test: true }),
  });
  const latencyMs = Date.now() - started;
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  return latencyMs;
}

Then, in runVirtualUser, you can use a pattern like: 3 searches → 1 checkout to simulate the funnel “more searches than purchases.”

5. Heavier tools: k6 (the simple way)

A Node script is great as a “minimum entry,” but sometimes it’s convenient to use a specialized tool like k6, where scenarios are written in JavaScript and the runtime is in Go (fast).

Example of a small k6 script for MCP:

// loadtest-mcp.js
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "30s", target: 30 },
    { duration: "2m", target: 30 },
  ],
};

export default function () {
  const payload = JSON.stringify({
    jsonrpc: "2.0",
    id: `search-${Math.random()}`,
    method: "tools/call",
    params: {
      toolName: "giftgenius.search_gifts",
      arguments: { occasion: "birthday", budget: 50 },
    },
  });

  const res = http.post(__ENV.MCP_URL, payload, {
    headers: { "Content-Type": "application/json" },
  });

  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

Run command:

MCP_URL=http://localhost:3000/api/mcp k6 run loadtest-mcp.js

k6 will compute p95/p99 and error rate, generate nice reports — you can then export them to Grafana and other systems.

Importantly, even with such tools our goal remains the same: not to endure a million RPS, but to make sure that at 5–10× your expected peak the system doesn’t fall apart and p95 stays within your SLO.

6. What to watch during (and after) a load run

We’ve already discussed metrics and SLO; now let’s just “land” them in the load context.

First, latency. For MCP tools like search_gifts you’ve set a target like “p95 < 2–3 seconds.” During a smoke‑load, check whether p95/p99 crept up 2–3×. It’s important to compare with your baseline: if before the code change p95 was 400 ms and after it’s 1,500 ms, even if you’re still formally within SLO, it’s a reason to think.

Second, error rate. Under load, unexpected things often pop up: exhausted DB connection pool, surprising 429s from an external API, timeouts when calling the payment provider. Under normal load, error rate should be close to zero; during smoke‑load occasional failures are acceptable, but definitely not 5–10%.

Third, resource metrics: CPU, memory, sometimes the number of open file descriptors and connections. These depend on your infrastructure, but the key idea is simple: you don’t want to see CPU at 100% with 30 VU and GC eating half the time.

Fourth, webhooks. If you have a commerce scenario, the final order state often depends on successful processing of a payment webhook. It’s important to watch not only the request speed in ACP, but also the delay “webhook arrived → we processed it successfully.”

And finally, logs. Structured logs with trace_id/checkout_session_id let you take a couple of the slowest or failed requests after a load run and trace the chain: MCP → external API → ACP → webhook. This is especially useful if you see weird p99 tails under load.

7. Feed data quality: from structure to meaning

We looked at how latency, errors, and resources behave under load. But even if you meet all these SLOs, the user experience can still fall apart due to bad data.

Let’s move to the second big topic: data. In a commerce app like GiftGenius, the product feed is not “something on disk,” it’s literally fuel for the LLM and agents. If the feed is garbage, the model won’t “invent” the price and availability for you.

It’s useful to think of feed quality in three layers.

Structural level

This is basic data validity:

• JSON parses correctly.
• All required fields are present: id, name, price, currency, imageUrl, availability, etc.
• Value types match expectations: price is a number, availability is an enum, categories is an array of strings.
• No duplicate ids.

You’ve already covered part of this with contract tests when you defined a JSON Schema/Zod schema for the feed. Now you need to apply these schemas to real data volumes.

Example of a simple Zod schema for a GiftGenius feed item:

import { z } from "zod";

export const giftItemSchema = z.object({
  id: z.string().min(1),
  name: z.string().min(3),
  description: z.string().optional(),
  price: z.number().positive(),
  currency: z.enum(["USD", "EUR", "GBP"]),
  imageUrl: z.string().url(),
  inStock: z.boolean(),
  tags: z.array(z.string()).default([]),
});

And the whole feed schema is just z.array(giftItemSchema).

Business level (semantics)

A product can be structurally valid and still be absurd from a business standpoint:

• Price 0 or 0.01 for an expensive product.
• Currency doesn’t match the market (USD for products sold only in EUR).
• inStock = true, but the last update was half a year ago.
• Categories from 1,000 variants with no unification.

For this level it’s useful to add additional checks and “common‑sense rules.” For example:

const businessRules = (item: GiftItem) => {
  const problems: string[] = [];

  if (item.price > 10000) {
    problems.push("suspiciously high price");
  }
  if (!item.inStock && item.tags.includes("bestseller")) {
    problems.push("bestseller, but not in stock");
  }
  return problems;
};

You can run these checks as part of a nightly job or when generating a new feed.

LLM level

The model is very capable, but it has its quirks:

• Descriptions stuffed with HTML, extra tags, and technical text.
• Mixed languages (half the feed in one language, half in another) without an explicit locale.
• Very long “SEO‑style names” like “Buy the best super‑duper gift now cheap.”

At this level it’s important to make the data model‑friendly:

• Strip HTML tags or normalize to plaintext.
• Normalize description language (or at least explicitly specify locale).
• Trim overly long names and duplicate information.

You can partially automate these tasks (e.g., via pre‑processing scripts) and partially coordinate with the team that populates the feed.

8. Practice: a feed validator for GiftGenius

Let’s add a simple validateFeed.ts script to our project that reads the feed JSON, validates it via Zod, and computes basic quality metrics.

// scripts/validateFeed.ts
import { readFile } from "fs/promises";
import { giftItemSchema } from "../src/schema/giftItem";

async function main() {
  const raw = await readFile("data/gift-feed.json", "utf-8");
  const data = JSON.parse(raw);

  const items = giftItemSchema.array().parse(data);
  console.log(`Total items: ${items.length}`);

  const missingImages = items.filter(i => !i.imageUrl).length;
  console.log(`Missing images: ${missingImages}`);
}

main().catch((e) => {
  console.error("Feed validation failed:", e);
  process.exit(1);
});

Here we use the same contract as the MCP server, i.e., contract tests and feed validation share a single schema — this greatly reduces the chance of discrepancies.

Then you can add business‑rule checks and metrics like:

• share of products without a description;
• share of products with suspiciously low/high price;
• number of duplicate ids or repeated name + price.

You can send these numbers to a metrics system (Prometheus, Datadog, etc.) and maintain separate SLOs for data quality — just like you set SLOs for code.

9. How load and the feed are connected

It may seem that “performance” and “data quality” are two loosely related topics. In practice, they are quite intertwined.

Examples of connections:

• Under load, some requests start going down “rare” code paths that hardly ever occurred before. For example, products with special discount types or non‑standard shipping. If the feed is dirty there, you can get both errors and serious performance degradation (lots of validations, exceptions, fallback logic).
• If the feed is very noisy (huge HTML‑laden descriptions, meaningless tags), the MCP server has to pull and serialize more data, which directly affects the tool call processing time and response size.
• In the commerce part, a poor feed can lead to many “empty” checkout attempts when a user picks a product that suddenly is out of stock. This hurts both UX and ACP metrics (growth of unsuccessful intents).

It’s convenient to view this as a matrix:

A load run acts like a flashlight here: it helps illuminate those parts of the feed that were rarely touched in everyday usage but start to bite under active traffic.

10. Integrating this into the GiftGenius release process

Process‑wise, everything above shouldn’t be “once before first prod.” In the training plan of modules 16 (“Production, networking, and scaling”) and 17 (“Observability and quality”), this approach is embedded specifically as part of a regular release checklist: before a release, you run not only unit/contract/E2E, but also a short smoke‑load plus a feed check.

A reasonable minimal pipeline before rolling out a new version:

1. Unit + contract + integration tests are green.
2. Short smoke‑load against MCP/ACP on staging if critical code changed (search logic, DB work, checkout).
3. Feed validator runs without errors, basic feed metrics (number of broken records, share without images, etc.) are within acceptable bounds.
4. Dashboards and alerts are updated to include new endpoints and SLOs.
5. Rollback plan prepared in case of failure: either disable the feature with a flag or roll back the build.

This way your GiftGenius stops being a “DevDay demo” and becomes a service ready for life in the Store and traffic spikes.

11. Common mistakes in load testing and feed checks

Mistake #1: load testing “via ChatGPT” instead of your backend.
Sometimes people try to “test everything as in reality” and run scripts that go specifically through the ChatGPT UI. As a result, they hit OpenAI limits, burn tokens, and get extremely noisy results. Meanwhile, MCP/ACP issues could have been caught a hundred times cheaper by hitting /mcp and /api/checkout directly.

Mistake #2: focusing only on average response time.
“We have 500 ms average latency, everything is great” — while p95 is 5 seconds and gets forgotten for some reason. We’ve already discussed in the SLO topic that it’s the tail of the distribution (p95/p99) that defines the real UX. Under load, the average often stays decent while the tail grows 2–3×.

Mistake #3: trying to build “enterprise‑grade load” instead of a practical smoke‑load.
Spending months building a complex environment emulating tens of thousands of users for a ChatGPT App like GiftGenius is almost always unnecessary. It’s far more useful to have a simple but regularly run smoke‑load at 50–100 VU with clear metrics.

Mistake #4: unrealistic load scenario.
A script sends the same request over and over, without user/language/product variation, and doesn’t touch ACP and webhooks. As a result, you test one hot happy path, while the real “edges” of the system remain in the shadows. It’s better to model at least a simplified but plausible flow: different budgets, different interests, some users reach checkout, some don’t.

Mistake #5: checking the feed only “by eye” or in prod.
You assembled the feed, shipped it to prod, saw weird recommendations from the model, and started wondering what happened. Meanwhile, a simple Zod/JSON Schema script could have shown in a minute that 10% of products lack images, 5% have price 0, and 3% have currency XXX. Lack of automatic feed validation is one of the most frequent sources of embarrassment in commerce apps.

Mistake #6: hoping the LLM will “figure it out” with a bad feed.
Yes, the model can do a lot, but it won’t invent a correct price or stock status. If the same product appears in the feed with different prices or both “in stock” and “out of stock,” an agent can produce hallucinations and an inconsistent user experience. The responsibility for clean data is on you, not the model.

Mistake #7: no linkage between feed metrics and overall SLOs.
You can have a perfectly fast MCP and ACP, but if 30% of products in the feed are “broken,” the user experience will still be terrible. Teams often track only technical SLOs (latency, error rate) and ignore data quality SLOs (minimum percent of valid SKUs, maximum duplicates, etc.). As a result, “numbers look fine,” but it doesn’t feel fine.

Mistake #8: running load tests directly in production without preparation.
Sometimes someone on Friday evening decides to “quickly run k6 against prod MCP,” without notifying anyone. In the best case you’ll skew real metrics and puzzle the on‑call engineer with a traffic spike; in the worst — you’ll hit rate limits of an external API or payment system. Always run your first scenarios on staging, and if you need a prod test — do it deliberately, with maintenance windows and notifications.