Release process — versions, notes, SDK/spec migrations, feature flags, rollback

1. Why the release process for a ChatGPT App is harder than a usual deploy

In the previous part of the module we discussed how to observe the quality and stability of the App (logs, metrics, SLO). Now let’s figure out how to set up the release process itself so that these metrics don’t tank with every deploy.

In a typical web application things are relatively simple: you deploy a new version of the backend and frontend — the user refreshes the page and lives on the new version. If something breaks, you can often just roll back the deploy.

In a ChatGPT App the stack is trickier. You have at least four layers that live different lives:

• Manifest and tool schema (MCP tools / OpenAPI);
• Infrastructure: the Next.js application and the MCP/Agents/ACP server;
• System prompt and other prompts;
• Data: product feed, settings, configs.

The problem is that the model lives in its own “mental world.” Chat context can last for hours and days. The manifest and tool descriptions are loaded and cached on the OpenAI side and don’t update instantly across all existing conversations. If you change a tool signature (for example, remove a field from the input schema), then in old conversations the model will keep sending the old payload, and your new backend will reject it. As a result — 400 errors, strange replies in the chat, and very unhappy users.

Therefore, in the world of a ChatGPT App, a “release” is not just “deploy a new Docker.” It’s a coordinated change across multiple layers, with careful version management, feature flags, and the ability to roll back.

2. Version matrix: what to version at all

It’s useful to look not just at “App version 1.3,” but immediately at a version matrix across layers. For GiftGenius it looks roughly like this.

giftgenius-app 1.4.2

tools-schema v1.7

prompt v3.1

feed v2, acp v1.3

Note that these are different axes. You can release application version 1.4.2 in which the tool schema remains v1.7, while the prompt moves to v3.2. This should be visible in the logs; that way it’s easier later to understand why conversion to checkout suddenly dropped after prompt v3.2.

For code it’s convenient to use semantic versioning (SemVer): MAJOR.MINOR.PATCH. MAJOR — breaking change, MINOR — new features without breaking changes, PATCH — bug fixes. For interface versions (schema, prompts) the logic is similar: MAJOR signals a breaking change.

Separately, call out the tool interface version. For an LLM this is critically important: if you change a tool’s contract while the model “thinks” the contract is old, chaos ensues. The usual policy is: in MINOR releases we only add new optional fields and don’t rename or remove old ones; breaking changes — only through a new tool, for example suggest_gifts_v2.

Now that we’ve split layers by versions, let’s see how these versions “live” in the release cycle — from dev to prod.

3. Basic release flow for GiftGenius

First let’s agree on the stages. You probably already have environments (from the module about deploy), but now let’s view them through the lens of releases.

Usually there are:

• dev — local development, Dev Mode in ChatGPT, tunnels;
• staging — as close to prod as possible: same DB type, same MCP/ACP type, but test data and payments in a sandbox;
• prod — the production environment, where the production ChatGPT App / Store listing is connected.

The process might look like this:

flowchart TD
  A[Dev: feature/* branch] --> B[PR → main]
  B --> C[CI: unit + contract + lint]
  C --> D[Deploy to Staging]
  D --> E[Smoke/E2E + manual checks]
  E --> F[Deploy to Prod]
  F --> G[Monitoring metrics and logs]

At each step we add “fuses.” In the dev environment a developer can do whatever they want, but every merge into main triggers CI, which runs unit/contract tests. If everything is ok — we automatically roll out to staging. On staging we run a short set of E2E/smoke scenarios: for example, one full gift selection flow and a “fake” checkout in the test payment provider.

Only after that do we press the deploy button to prod. Ideally — with the version tag and a link to the CHANGELOG. Already in prod we keep watching p95, error rate, and business metrics (conversion from recommendation to checkout). If something drops after the release — we should have a clear rollback plan, which we’ll discuss below.

4. Release notes and changelog: what exactly to record

If you don’t have release notes, in a month you’ll be looking at a chart: “why did our checkout p95 double 3 weeks ago?” and answer yourself “no idea, but we had a big release.” Not the best strategy.

There are usually at least two kinds of notes.

Internal changelog — technical. It’s for developers, SREs, and everyone poking around in the code. You note which features appeared, which bugs were fixed, and whether there were breaking changes. You can use the canonical Keep a Changelog format: sections Added, Changed, Fixed, Removed.

External release notes — human-readable notes for users and for the Store. You don’t need to write “migrate MCP SDK 0.4 → 0.5”; it’s better to write “Added recommendations for the Thanksgiving holiday,” “Fixed a rare bug that caused some gifts not to be added to the cart.”

Example snippet of CHANGELOG.md for GiftGenius:

## [1.4.0] - 2025-11-20
### Added
- New tool `suggest_gifts_v2` with support for interest tags.
- A/B test of the new system prompt (flag: GG_PROMPT_V3).

### Changed
- Improved ACP checkout error handling (more user-friendly messages).

### Fixed
- Fixed a bug where items without an image were hidden from recommendations.

It’s helpful to store the prompt version somewhere nearby too: even just the commit hash of the system prompt file, which you can later recall: “aha, after prompt b3f9c2d users clicked “Buy” less often.”

5. SDK and spec migrations: how to live in a fast-moving ecosystem

The ecosystem around the Apps SDK, MCP, and Agents evolves quickly: new SDK versions, MCP protocol changes, ACP updates, etc. That’s good (new capabilities) but also painful: things can break quickly too.

Pinning versions and “don’t update on release day”

First simple recommendation: pin dependency versions. Not ^0.4.0, but 0.4.0. For SDKs that often break APIs (MCP/Agents/Apps SDK), this is especially critical. I’ve read studies on this topic, and they specifically mention the phenomenon of “SDK fatigue” — fatigue from constant incompatible updates, especially for those who left dependencies with floating versions (like ^0.4.0) and once suddenly hit a pile of errors after npm install.

Mini example:

// package.json (fragment)
{
  "dependencies": {
    "@modelcontextprotocol/sdk": "0.4.2",
    "@openai/applications-sdk": "0.3.1"
  }
}

Second recommendation: don’t bundle both business features and large SDK migrations into the same release. If you need to update the MCP SDK from 0.3.x to 0.4.x, it’s better to do a separate technical release: first the SDK update, tests, and stabilization — and only then a new checkout flow.

SDK migration strategy

A reasonable plan usually looks like this:

In the dev environment, create a separate branch upgrade/mcp-sdk-0.4. Update the dependency, fix the code, run all unit/contract tests, and locally run the main GiftGenius flow via Dev Mode.

Then deploy this branch to a separate staging URL and run E2E/smoke tests against it. You can even do a small load test: 50–100 calls to suggest_gifts in a row.

If everything is ok — merge into main, deploy to the primary staging, run smoke again, and only then — to prod.

If not — you have an obvious rollback: simply don’t merge this branch or revert it. That’s the point of separating SDK migrations from product releases.

Migrations of tool schemas and API

The most unpleasant part is interface changes: the model doesn’t learn about them instantly. Studies on this topic emphasize an important rule: “Extend, don’t break” (additive-only). If you need to add a new argument interests: string[] to suggest_gifts, make it optional rather than required; old scenarios will continue to work.

Example of evolving a Zod schema for input data:

// v1
const suggestGiftsInputV1 = z.object({
  recipientName: z.string(),
  budget: z.number()
});

// v1.1 (added optional fields)
const suggestGiftsInputV1_1 = suggestGiftsInputV1.extend({
  interests: z.array(z.string()).optional(),
  occasion: z.string().optional()
});

Note: we don’t touch existing fields, only extend.

If you really need to break the contract (for example, replace budget with minBudget + maxBudget), it’s better to create a new tool suggest_gifts_v2 and indicate in the description that it’s an improved version. You can leave the old API as deprecated and gradually turn it off once you’re sure the model and users have migrated.

Data migrations: product feed and ACP

We’ve already talked about SDK and tool schema migrations. The product feed is also a contract. If you change the format of SKUs, currencies, or localizations, this must be coordinated: update both the feed schema and the handling in MCP/ACP, and any preprocessors. Commerce documentation for the ChatGPT App notes that feed errors (broken fields, duplicates, weird prices) can kill recommendation quality even with perfect code.

Typical approach:

1. First add new fields to the feed schema as optional and teach GiftGenius to use them when present.
2. Then update the pipeline that builds the feed so it starts populating these fields.
3. Run a feed validator (contract tests on data) and only after that start depending on these fields in the logic.

6. Feature flags: separate deploy from release

Feature flags are one of the main survival tools in the world of ChatGPT Apps. The core idea is simple: you deploy code, but you don’t necessarily enable new functionality immediately. First it lives “behind a flag” — enabled only for developers, or for 1% of users, or disabled altogether.

This is especially important when:

• you roll out a new recommendation algorithm;
• you change the system prompt (which can radically change the model’s behaviour);
• you connect an expensive or slow tool;
• you experiment with a new checkout flow.

Simple flag via environment variables

At the bare minimum a feature flag can be implemented via an environment variable:

// lib/features.ts
export const isNewRecoEnabled =
  process.env.NEXT_PUBLIC_GG_NEW_RECO === "1";

Then in the MCP tool code you can use it like this:

if (isNewRecoEnabled) {
  return runNewRecoAlgorithm(input);
}
return runOldRecoAlgorithm(input);

The plus of this approach is simplicity. The minus is that toggling flags at runtime is harder: you need to deploy a new environment or at least reinitialize.

Centralized helper with context

A slightly more mature approach is to have a centralized helper that considers not only global flags, but also user context (tenant, segment, A/B group).

// lib/featureFlags.ts
type Feature = "new-reco" | "checkout-v2";

type Context = { userId: string; tenantId?: string };

export function isFeatureEnabled(
  feature: Feature,
  ctx: Context
): boolean {
  // logic may live here: env, DB, external flag service
  if (feature === "new-reco") {
    return process.env.GG_NEW_RECO === "1";
  }
  return false;
}

In the MCP handler:

const enabled = isFeatureEnabled("new-reco", { userId });
const result = enabled
  ? await runNewReco(input)
  : await runOldReco(input);

If you ever hook up an external flag service (LaunchDarkly, Statsig, etc.), you’ll only need to change the implementation of isFeatureEnabled, not all the code.

Example scenarios for GiftGenius

A/B test of the system prompt. You create PROMPT_V2 and enable it only for 10% of users with a tenantId in a specific list. MCP tools and the widget don’t change, and you compare conversion.

Kill switch for an expensive tool. Suppose you created a tool that makes a complex external request (for example, to some ML recommendation model that’s expensive). If this external service starts failing or suddenly gets much more expensive, you want to disable it in a second without stopping GiftGenius as a whole. A feature flag is the simplest way.

Gradual rollout of a new checkout flow. You first enable Checkout v2 only for company employees and a couple of trusted customers. If metrics look ok — expand the audience until it’s enabled for everyone.

7. Rollback: how to roll back quickly when everything is on fire

Even with perfect tests and flags, something will break. It’s important to have a clear strategy: what you do in the first 5–15 minutes after you see an error spike or a metric drop.

Fast code rollback

If the problem is a purely technical bug (NPE, wrong variable, incorrect URL of an external service), it’s usually enough to roll back the deploy to the previous version. For example, Vercel has an instant rollback to the previous deployment.

Your task is to always know which deployment corresponds to which version and how to roll it back. Ideally this is described in the on-call README: “if everything is on fire after release 1.4.0, roll back to deployment X.”

A second quick lever is feature flags. If only the new feature (checkout-v2) broke, it’s easier to turn it off with a flag than to immediately roll back the entire release.

Dangerous changes: manifests and schemas

Manifests and schemas are trickier. If you pushed a new manifest with an incorrect tool schema to the Store and OpenAI has already approved it, rolling back can take days. The reason is simple: each manifest change also goes through OpenAI review. Analytics across various Stores explicitly state that changes to the manifest and schemas are “dangerous” releases that must be prepared and tested especially carefully.

So it’s better to separate:

• safe releases: changes to MCP/Next.js code, prompt edits, new features hidden behind flags;
• dangerous releases: changes to the list of tools, input/output schemas, ACP contracts.

“Dangerous” releases should be rolled out separately, with additional checks and, possibly, first only via Dev Mode and a staging App (without publishing to the Store).

Rollback of data and the feed

With data it’s even more complicated (and more painful). If you migrated the product feed to a new schema and removed old fields in the process, reverting isn’t always trivial. Therefore data migrations should be idempotent and reversible: either you keep an old copy, or you do the migration in two steps (first duplicate the data, then switch reads).

A simple approach is to store old and new fields in parallel for some time and give yourself the ability to switch between them via a feature flag. If something breaks — you simply go back to the old reads.

8. Mini design of a release process for GiftGenius

Let’s assemble everything we discussed above (versions, SDK migrations, feature flags, rollback) into one practical scenario.

Imagine you’re building release 1.4.0, in which you:

• add a new tool suggest_gifts_v2 with an expanded input;
• enable a new system prompt for a portion of users;
• update the MCP SDK from 0.3 → 0.4;
• change the product feed format (add a tags field).

A reasonable plan would look like this.

First, a separate technical release 1.3.1: the MCP SDK update + minimal code tweaks, without schema or feature changes. Run CI, staging, smoke tests. If everything is stable — live with it for a couple of days.

Then branch feature/reco-v2. There you add suggest_gifts_v2 as a new tool (the old suggest_gifts remains). Its input schema expands only by adding new optional fields. Prepare the new system prompt, but wrap it in a flag GG_PROMPT_V3. In ACP/the feed add the new tags field as optional, and implement reading so that everything continues to work if it’s absent.

In CI add a few new contract tests: that suggest_gifts_v2 accepts both old and new payloads, that a feed with tags is valid, and that old records without tags don’t break the server either.

After merging into main:

• CI runs unit/contract tests;
• on staging, 1–2 E2E scenarios run through the new tool;
• enable the new prompt and tool only for a test tenant via a feature flag.

Watch the metrics: tool p95, error rate, conversion to checkout. If everything is ok — expand the flag to a larger group of users. Only then, after you’re convinced of stability, update the manifest for the Store (if needed) and the app’s promo description.

If something breaks along the way — you know how to roll back: either turn the flag off manually, or roll back the deployment, or, in extreme cases, roll the manifest version back (but that’s the scenario you’d rather not allow at all).

9. Common mistakes in a ChatGPT App release process

Mistake #1: one abstract “App version” instead of a version matrix.
When you only have “GiftGenius v1.4” but tool schema, prompt, and product feed versions aren’t recorded anywhere, you won’t be able to answer later: “after which exact change did our checkout drop?” Split versions by layers and log them in structured logs.

Mistake #2: breaking changes in tools without a new name/version.
The most painful thing: you went and renamed or removed a field in the input schema without changing the tool name. In old chats the model keeps sending the old payload, the backend returns 400, GPT starts to “hallucinate” in response, users don’t understand anything. Make any breaking change via a new tool (foo_v2) or a new API version, and keep the old interface for a transition period.

Mistake #3: updating the SDK “on the way” to a feature.
Classic: you’re adding a new business feature and along the way update @modelcontextprotocol/sdk from 0.3 to 0.5 “well, to keep things fresh.” As a result, if something breaks, it’s unclear whether the culprit is the new code, the new schema, or the new SDK. SDK migrations are better done as separate technical releases, with a clear test plan and a rollback option.

Mistake #4: no feature flags and no instant kill switches.
Rolling out a new recommendation algorithm immediately to 100% of users is fun until it starts giving strange results or takes down an external service. Without a feature flag your only lever is a full release rollback, which can affect not only the new feature, but also a dozen harmless improvements. Implement at least simple flags via env or a small config.

Mistake #5: hoping for an “instant” manifest update.
A common misconception is to think that as soon as you change tools or openapi.yaml, the model immediately learns about the new schema. In practice, the manifest and tool descriptions are cached, and in already-open chats can live long. Ignoring this leads to non-obvious bugs: in new chats everything works, in old ones — it breaks. Plan schema changes with this behaviour in mind and test them via Dev Mode and staging before rolling out to the Store.

Mistake #6: no clear rollback plan and release documentation.
If nobody in your team can answer “how do we roll back a release in 5 minutes?”, “which version will we roll back to?”, “how do we return to the old feed schema?” — consider that you have no rollback. The on-call should have a short but concrete playbook: which buttons to press, which variables to change, and where to check that the rollback worked.

Mistake #7: “silent” releases without release notes and metric linkage.
Shipping releases without even a minimal changelog means you’ll be guessing in a couple of months. When p95 suddenly grows and conversion drops, you’ll be asking: “what changed back then?” Building a habit of writing at least minimal release notes and linking them to deploy dates and versions makes not only quality audits easier, but the whole team’s life too.

1

Task

ChatGPT Apps, level 17, lesson 3

Locked

Release Guard — a script for release notes and pinned dependencies

Release Guard — a script for release notes and pinned dependencies

Open