Testing GiftGenius — unit, contract, E2E, and smoke in CI

1. What do we actually test in a ChatGPT App (and what we don’t)

In a classic web app, it’s straightforward: UI → backend → DB. We write unit tests for functions, integration tests for APIs, and E2E for the “user completed the flow.”

In a ChatGPT App, the picture is a bit more complex:

User ↔ ChatGPT UI ↔ Widget (Apps SDK, React)
                  ↘
                    MCP server (tools/resources)
                      ↘
                        ACP / backend / external APIs

The model inside ChatGPT decides when to call your suggest_gifts, with which arguments, how to render structuredContent from MCP, and when to show your widget.

From a testing perspective, it’s convenient to split the world into two layers:

• Infrastructure tests — what we do in this lecture. We verify that:
  • widget code doesn’t crash on a user click;
  • MCP tools accept and return data in the format promised by the schemas;
  • ACP endpoints and webhooks are alive and don’t crash on a typical JSON.
• AI behavior evals — what will come in module 20. There we’ll look at what the model actually answers: whether it explains sensibly, chooses the gift correctly by meaning, and doesn’t hallucinate.

A rough formula for today:

“We test everything around the LLM, but not the LLM itself.”.

That’s why the course plan for this topic specifically emphasizes: “We don’t test the GPT response verbatim; we test the infrastructure around it and the data contracts.”

To stay focused, we’ll use a simple testing “pyramid” for GiftGenius.

graph TD
  A["Unit tests
utils, tool business logic"] --> B[Contract tests
Zod/JSON Schema, webhooks]
  B --> C[E2E / UI tests
widget + MCP without ChatGPT]
  C --> D["Smoke in CI
“is it even alive?”"]

  style A fill:#e0f7fa,stroke:#00838f,stroke-width:1px
  style B fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
  style C fill:#fff3e0,stroke:#ef6c00,stroke-width:1px
  style D fill:#ffebee,stroke:#c62828,stroke-width:1px

We’ll now go through each level and, along the way, extend our training project GiftGenius with tests. At the end, we’ll compile a checklist of common mistakes that most often occur in testing a ChatGPT App.

2. Unit tests: breaking GiftGenius into small pieces

What counts as a unit in a ChatGPT App

A unit test in our stack is a check of a small, isolated piece of logic. No real network, no database, and, if possible, no call into the MCP framework itself.

In GiftGenius, this can be:

• a function that computes “gift relevance”;
• a filter that removes products with no price or with an unsuitable currency;
• a currency converter;
• a mapper from a “raw” product object to GiftCardProps for the UI.

Ideally, you should also split the logic inside MCP tools: the MCP route handler is a thin wrapper that calls a pure function with business logic. In unit tests, we test the pure function.

Example: a function for ranking gifts

Imagine we have a utility scoreGift that assigns a “score” based on price range and popularity:

// src/lib/scoreGift.ts
export type Gift = {
  id: string;
  price: number;
  popularity: number; // 0..1
};

export function scoreGift(gift: Gift, maxPrice: number): number {
  if (gift.price > maxPrice) return 0;
  const priceScore = 1 - gift.price / maxPrice;
  return Math.round((priceScore * 0.6 + gift.popularity * 0.4) * 100);
}

We’ll write a unit test in Jest (Vitest will look almost the same):

// src/lib/scoreGift.test.ts
import { scoreGift } from './scoreGift';

test('scoreGift lowers the score for expensive gifts', () => {
  const cheap = { id: 'c', price: 50, popularity: 0.5 };
  const expensive = { id: 'e', price: 100, popularity: 0.5 };

  const max = 100;
  const cheapScore = scoreGift(cheap, max);
  const expensiveScore = scoreGift(expensive, max);

  expect(cheapScore).toBeGreaterThan(expensiveScore);
});

This shows the basic “Arrange–Act–Assert” (we prepared data, called the function, checked the result) — exactly the structural approach you should also use in more complex tests.

Extracting business logic from the MCP handler

Right now, you probably have something like this:

// app/mcp/route.ts — heavily simplified
import { createMcpServer } from '@modelcontextprotocol/sdk';
import { scoreGift } from '@/lib/scoreGift';

server.tool('suggest_gifts', {
  // ...
  handler: async ({ input }) => {
    const gifts = await fetchFromCatalog(input);
    const scored = gifts
      .map(g => ({ ...g, score: scoreGift(g, input.maxPrice) }))
      .sort((a, b) => b.score - a.score);

    return { gifts: scored.slice(0, 10) };
  },
});

We’ve already written a unit test for scoreGift, but we also want to test the whole function: “a function that takes a list of gifts and returns a top‑10 sorted.” Let’s extract it to a separate module:

// src/lib/rankGifts.ts
import { scoreGift, Gift } from './scoreGift';

export function rankGifts(gifts: Gift[], maxPrice: number) {
  return gifts
    .map(g => ({ ...g, score: scoreGift(g, maxPrice) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, 10);
}

And a test:

// src/lib/rankGifts.test.ts
import { rankGifts } from './rankGifts';

test('rankGifts returns up to 10 gifts in descending score order', () => {
  const gifts = Array.from({ length: 20 }, (_, i) => ({
    id: `g${i}`,
    price: 10 + i,
    popularity: 0.5,
  }));

  const result = rankGifts(gifts, 100);

  expect(result).toHaveLength(10);
  expect(result[0].score).toBeGreaterThanOrEqual(result[9].score);
});

Such unit tests are fast, cheap, and provide quick feedback — that’s why they’re recommended as “the wide base of the testing pyramid” for MCP services.

Unit tests for MCP tools: mock external APIs

A common mistake is trying to “unit test” an MCP tool handler together with real HTTP requests to a catalog, Stripe, etc. The result is a slow and brittle test.

The best option is to keep only “wiring” in the handler and move all the complex logic into functions we’re already testing separately. If you really want to test the handler itself, replace dependencies with mocks. This is exactly what in‑depth reviews of MCP testing recommend: mock external APIs in tool handlers.

3. Contract tests: Zod/JSON Schema as the “agreement” with the model and ACP

What is a contract test in our context

We’ve handled the unit logic: small pure functions are under control. The next layer of the pyramid is to ensure services still understand each other by JSON contracts. Those are contract tests.

Contract testing verifies that two parties that exchange data still understand each other. The focus is not on internal algorithms but on the shape and meaning of the JSON: fields, types, requiredness.

In a ChatGPT App we have lots of such contracts:

• ChatGPT ↔ MCP: inputSchema and outputSchema of MCP tools.
• MCP ↔ commerce API (ACP): request format for create_checkout_session, response structure.
• ACP ↔ our backend via webhooks: order.created, payment_failed, etc.

If you change a schema but forget to update the code (or vice versa — you change the code but leave the schema the old one), a silent break appears. The model keeps sending the old JSON while your code expects a new field — and it crashes at runtime. Contract tests should catch exactly such situations before production.

Zod as the single source of truth

In the JavaScript/TypeScript ecosystem, Zod is a great fit, and you’ve already used it with MCP: the SDK can convert Zod schemas into JSON Schema to declare tools.

For example, let’s describe the schema of a gift and a recommendation result:

// src/schemas/gift.ts
import { z } from 'zod';

export const GiftSchema = z.object({
  id: z.string(),
  title: z.string(),
  price: z.number().nonnegative(),
  currency: z.string().length(3),
  url: z.string().url(),
});

export const SuggestGiftsResultSchema = z.object({
  gifts: z.array(GiftSchema).min(1),
});

We derive types for code via z.infer:

export type Gift = z.infer<typeof GiftSchema>;
export type SuggestGiftsResult = z.infer<typeof SuggestGiftsResultSchema>;

This is already a kind of compile‑time contract test: if you try to assign currency: 123 somewhere in code, TypeScript will light up and remind you it must be a string.

Runtime contract tests for schemas

We can go further with runtime tests that run real (or close‑to‑real) data examples through the schemas.

// src/schemas/gift.test.ts
import { GiftSchema, SuggestGiftsResultSchema } from './gift';

test('GiftSchema accepts a valid item', () => {
  const sample = {
    id: '123',
    title: 'Cat mug',
    price: 19.99,
    currency: 'USD',
    url: 'https://example.com/gift/123',
  };

  expect(() => GiftSchema.parse(sample)).not.toThrow();
});

test('SuggestGiftsResultSchema rejects an empty gift list', () => {
  const badResult = { gifts: [] };

  expect(() => SuggestGiftsResultSchema.parse(badResult)).toThrow();
});

Why this matters:

• if you show JSON examples in prompts/documentation for the model, you can put them straight into such tests and guarantee that “the example doesn’t lie”;
• if you change the schema (for example, make the url field required), the tests will immediately highlight all the old examples and fixtures that are no longer valid.

Official Apps SDK recommendations explicitly emphasize: structured content must match the declared outputSchema, otherwise the model may not understand it. Schema tests are the first line of defense against discrepancies.

Webhook and ACP contracts

The same principle applies to webhooks and ACP endpoints. Let’s say we have OrderCreated:

// src/schemas/acp.ts
import { z } from 'zod';

export const OrderCreatedSchema = z.object({
  id: z.string(),
  userId: z.string(),
  totalAmount: z.number(),
  currency: z.string().length(3),
  status: z.literal('created'),
});

Test:

// src/schemas/acp.test.ts
import { OrderCreatedSchema } from './acp';

test('OrderCreatedSchema validates a sample webhook', () => {
  const sample = {
    id: 'ord_1',
    userId: 'user_42',
    totalAmount: 59.99,
    currency: 'USD',
    status: 'created',
  };

  expect(() => OrderCreatedSchema.parse(sample)).not.toThrow();
});

Then, in the webhook handler, the first thing you do is OrderCreatedSchema.parse(body) — and you can be confident you’re working with a valid object.

OpenAI’s regression checklist for Apps also recommends keeping schemas up to date as the application evolves — contract tests ensure you won’t forget.

4. Testing the widget and “almost E2E”: how to do without chatgpt.com

Unit tests keep the logic in shape; contract tests keep the data shape between services. But that’s not the end of the pyramid: we still need to verify that the entire user journey through the widget and MCP actually works as a whole. For a ChatGPT App, this will be a special, “almost E2E” format.

Why you can’t just “point Playwright” at ChatGPT

The intuitive impulse: “Let’s open https://chatgpt.com, launch the widget, go through the whole scenario ‘pick a gift → checkout’ with Playwright, and get ourselves a real E2E.”

Sadly, no.

Problems:

• automated runs against chatgpt.com violate the ToS;
• there are protections (Cloudflare, 2FA, etc.) that really don’t like bots from CI;
• model behavior is variable: today it called your suggest_gifts, and tomorrow it decided to stick to a textual response.

Therefore, for a ChatGPT App, E2E testing is interpreted more broadly: we test the end‑to‑end path inside our own application — widget + MCP + ACP — but without the real ChatGPT UI and without the real model.

Detailed guides propose exactly this strategy: test the MCP server with a headless client, and the widget in a “test host” with a mocked window.openai.

Testing the widget as a React component

The basic option is React Testing Library. We need to:

1. Render the GiftGeniusWidget component.
2. Provide a fake window.openai with the required methods (callTool, openExternal, etc.).
3. Pretend to be a user: click buttons, type text.
4. Verify that callTool is called with the correct arguments and that the UI shows the expected result.

Suppose we have a simplified widget:

// src/app/GiftGeniusWidget.tsx
'use client';
import React from 'react';

export function GiftGeniusWidget() {
  const [loading, setLoading] = React.useState(false);

  async function handleClick() {
    setLoading(true);
    await (window as any).openai.callTool('suggest_gifts', {
      occasion: 'birthday',
    });
    setLoading(false);
  }

  return (
    <div>
      <button onClick={handleClick}>Pick a gift</button>
      {loading && <p>One moment, gathering ideas...</p>}
    </div>
  );
}

Test:

// src/app/GiftGeniusWidget.test.tsx
import { render, screen, fireEvent } from '@testing-library/react';
import { GiftGeniusWidget } from './GiftGeniusWidget';

test('the button calls suggest_gifts via window.openai.callTool', async () => {
  const callToolMock = vi.fn().mockResolvedValue({});
  (window as any).openai = { callTool: callToolMock };

  render(<GiftGeniusWidget />);

  const button = screen.getByText('Pick a gift');
  await fireEvent.click(button);

  expect(callToolMock).toHaveBeenCalledWith('suggest_gifts', {
    occasion: 'birthday',
  });
});

Here we fully control the environment:

• no real ChatGPT;
• no network;
• a clean, fast test that verifies the “UI → window.openai” link.

Apps SDK docs recommend exactly this: mock window.openai when testing the widget so you don’t depend on the real environment.

E2E‑light with Playwright: Next.js + MCP

The next level — we spin up the Next.js application locally (like in Dev Mode), but instead of entering through ChatGPT, we hit it directly from the test’s browser.

A sensible scenario to check:

1. Open the page /widget (or / — depending on your project).
2. Mimic the minimum steps: choose a gift type, click the “Show ideas” button.
3. Verify that the widget shows gift cards.
4. (Optional) Click a card, hit “Proceed to checkout,” and make sure the ACP mock returns success.

Mini Playwright test example:

// tests/e2e/gift-flow.spec.ts
import { test, expect } from '@playwright/test';

test('a user can choose a gift and see the results', async ({ page }) => {
  await page.goto('http://localhost:3000/widget');

  await page.click('text=Birthday gift');
  await page.click('text=Pick');

  await page.waitForSelector('[data-testid="gift-card"]');

  const cards = await page.locator('[data-testid="gift-card"]').all();
  expect(cards.length).toBeGreaterThan(0);
});

In a real project, you’ll also add:

• spinning up npm run dev or a dedicated test server in Playwright’s beforeAll;
• mocks for MCP/ACP so you don’t touch production services.

Even such a simple scenario catches typical “breaks” between the widget and MCP: wrong URL, CORS errors, incorrect structuredContent, etc.

5. Smoke tests in CI: check that “it even starts”

The top, lightest layer of our pyramid is smoke tests. They don’t verify the entire scenario like E2E‑light; they simply tell us whether the app is alive and starts before a deploy.

Smoke vs full E2E

You’ve already heard about a “manual” smoke test back in the second module: then we launched the very first “Hello GiftGenius,” checked that the widget renders, ChatGPT sees it, and the button opens a link. The goal was to make sure the Dev Mode + tunnel + Apps SDK config was correct.

Now the task is similar, only automated and in CI:

• we don’t try to model all user scenarios;
• we don’t talk to the real ChatGPT;
• we only check that:
  • the Next.js app starts;
  • the MCP server responds at least to tools/list / tools/call;
  • the ACP endpoint is alive and returns 200 on a test JSON.

This is especially important before deploying to production or submitting a new version to the Store: it’s easier to catch “everything fell over and won’t start” in CI than to learn it from users.

Example smoke test for MCP tools

Suppose we have a helper module that spins up an MCP server in the test or uses the SDK’s MCP client. Conceptually, the test looks like this:

// tests/smoke/mcp-tools.smoke.test.ts
import { createTestMcpClient } from './testClient';

test('MCP responds to tools.list and tools.call(suggest_gifts)', async () => {
  const client = await createTestMcpClient(); // spins up the server or connects to it

  const tools = await client.listTools();
  expect(tools.some(t => t.name === 'suggest_gifts')).toBe(true);

  const result = await client.callTool('suggest_gifts', {
    occasion: 'birthday',
    budget: { currency: 'USD', max: 50 },
  });

  expect(result.gifts.length).toBeGreaterThan(0);
});

In deeper MCP testing write‑ups, this is exactly the approach they advise: use an MCP client in tests to verify the full JSON‑RPC cycle — list → call → response.

You can hide the implementation of createTestMcpClient in utilities: it either starts the server in the same process or connects to an already running instance.

Smoke test for ACP/checkout

Similarly, you can write the simplest test for the commerce layer without simulating a real payment:

// tests/smoke/acp.smoke.test.ts
import fetch from 'node-fetch';

test('ACP test-intent returns 200', async () => {
  const res = await fetch('http://localhost:3000/api/acp/test-intent', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({
      amount: 10,
      currency: 'USD',
    }),
  });

  expect(res.ok).toBe(true);
});

Here it doesn’t matter what test-intent does — it can simply check DB access and return {"status":"ok"}. The main thing is that CI will catch:

• a forgotten env key;
• a broken route;
• bad JSON parsing.

Minimal CI pipeline

We’ll cover CI/CD in detail in the deployment modules, but a basic pipeline might look like this (for example, on GitHub Actions):

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [ main ]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm test           # unit + contract
      - run: npm run test:e2e   # e2e/ui
      - run: npm run test:smoke # smoke mcp/acp

The npm run test:e2e and npm run test:smoke commands can spin up a dev server inside, wait for it to be ready, and then run Playwright / Node scripts.

6. Mini map of tests for GiftGenius

So we don’t get lost, let’s put everything into a single table — what we test at each level and which questions that answers.

This set is the very “minimum viable test suite” for a ChatGPT App mentioned in the module plan: without an Enterprise QA team, but with baseline guarantees that production doesn’t fall over at every sneeze.

7. Typical mistakes when testing a ChatGPT App

Mistake No. 1: trying to deterministically test model outputs.
Sometimes developers try to write tests like “I expect the GPT to answer with the string Here are 5 gift ideas.” Such tests are brittle by definition: the model is not obliged to repeat the exact phrasing, and the model itself can be updated. In this module we don’t touch the content of the answers at all — we only verify that tools are called, schemas are valid, and the flow doesn’t crash. Evaluating text quality is a separate discipline (M20, LLM evals).

Mistake No. 2: no contract tests for MCP schemas.
It’s very tempting to describe a Zod schema once and forget it. Then you add a discount field to a tool’s result, update the code, but don’t update the schema. The model keeps sending the old format, while your code expects a new field — strange crashes begin in production. Contract tests via Zod/JSON Schema prevent exactly such “silent” failures, so neglecting them is a popular and very painful misstep.

Mistake No. 3: trying to run E2E against chatgpt.com from CI.
Someone will still try: they run Playwright against real ChatGPT, log in, click through the UI — and get banned by Cloudflare, end up with unstable tests, and potentially violate the terms of use. The right path is to test your own Next.js host + MCP in isolation, mocking window.openai and external APIs, as Apps SDK and MCP guides recommend.

Mistake No. 4: writing only E2E and forgetting the unit layer.
Sometimes you see a project with one “huge” E2E test clicking through half the app and zero unit tests. This approach gives a false sense of safety: the test is either green or red, but it’s almost impossible to localize the cause, and each run takes minutes. It’s much more effective to have dozens of fast unit tests for pure functions and a couple of neat E2E‑light scenarios for critical paths.

Mistake No. 5: using real external APIs in ordinary tests.
Stripe, external catalogs, CRM — all of these are great for integration tests in a controlled environment, but not for a regular npm test. If your tests depend on the network, other people’s rate limits, and someone’s production server, they’ll fail for reasons unrelated to your code. The best approach is to mock external APIs (via nock, msw, etc.) and separately have a few “live” checks in a special environment.

Mistake No. 6: forgetting smoke tests before a deploy.
You glued a feature, updated an MCP schema, tweaked the UI, hit “Deploy” — and Next.js doesn’t start because someone broke next.config or deleted .env. Without automated smoke tests, CI lets obvious failures slip into prod. One simple smoke suite that checks “the server started,” “MCP responds to a basic call,” and “the ACP test endpoint returns 200” saves hours of firefighting and lots of nerves.

Mistake No. 7: overcomplicating the test setup at an early stage.
Sometimes, inspired by big‑company best practices, you want to spin up a dozen environments right away, complex contract tests with data generation, load scenarios, etc. As a result, the team spends weeks on infrastructure and stops shipping features. To start with a ChatGPT App, the “Sanity Suite” we discussed is enough: unit + contract + a couple of E2E‑light + smoke in CI. You can evolve further as traffic and requirements grow.