Self‑improving App: a closed loop of improvements

1. Why an app needs a closed loop of improvements

Any LLM‑App lives in a changing world. You get new users and scenarios, OpenAI rolls out new models and safety mechanisms, you update your MCP server, Agents SDK, UI widget… And even if all your code were perfect (yes, yes, I know you almost do it that way), a single model or prompt change can quietly break half of your cases.

Without an improvement loop, life looks like this. A user writes to support: “GiftGenius started suggesting gifts above the budget” or “it sits and thinks for 15 seconds.” You open logs, tweak something in the system prompt, switch the model, deploy. A week later it repeats, but elsewhere. The result is a chronic fire drill and a bunch of “magical” changes that no one can explain.

With a loop, it’s different. You have:

• a clear set of signals across tech, money, product, and quality;
• a regular ritual: look at them and pick hypotheses;
• controlled changes in code/config;
• automated checks: golden cases + LLM evals + experiments.

Then the app turns from a “frozen prompt” into a living system that:

• shows where it hurts and why;
• suggests exactly what to improve;
• protects against silent degradation after an “innocent” prompt change.

Bonus: this loop fits very well with everything you’ve already built in previous modules. Logs and SLO from M17 provide technical signals, cost instrumentation and AARRR from M19 — economic and product signals, golden cases and LLM evals from M20.1–2 — quality signals. All that’s left is to connect this into a clear operational loop.

2. A signal map for improvements

For the app to suggest where to improve itself, you need to clearly understand what signals you have and where they come from. It’s convenient to think in four groups.

Technical signals come from your observability stack: tool_invocation logs, latency and error‑rate metrics, MCP/ACP health checks, OAuth failures. They answer the question “is the service alive and how stable is it.”

Economic signals arise in cost instrumentation and billing: cost_per_tool_call, cost_per_task, cost_per_user, unexpected cost spikes for specific tools or scenarios. They say: “we’re burning tokens and money more than expected,” or conversely, “there’s headroom, we can improve quality.”

Product signals are formed from events like app_opened, workflow_started, workflow_completed, checkout_*. That’s activation rate, step‑to‑step funnel conversions, cohort retention. They show what’s happening with real user behavior.

Quality and behavior signals include LLM eval results on golden cases (scores for correctness/helpfulness/style/safety), sampled manual reviews of dialogs, thumbs up/down, complaints and reviews in the Store. This is what’s closest to the feeling “the app got smarter/dumber.”

It’s convenient to organize this in a table:

Key point: all these signals are logged with a link to a specific scenario and app version. That is, events include at least scenario, appVersion, and experimentId. Then, seeing that helpfulness on golden cases fell from 8.5 to 6.2, you can say not just “it got worse,” but “it got worse in the "gift_selection" scenario after release "1.3.0", in experiment variant "B".”

In code, you can even introduce a simple structure to describe a signal:

// lib/improvement/signals.ts
export type SignalKind = "slo" | "cost" | "product" | "quality";

export type ImprovementSignal = {
  kind: SignalKind;
  scenario: string;       // e.g. "gift_selection"
  metric: string;         // e.g. "p95_latency_ms", "cost_per_task"
  value: number;
  previous?: number;      // "before" value
};

Such objects are convenient to feed both dashboards and your future improvements assistant.

3. Canonical feedback loop: 4 steps

Now let’s assemble a canonical improvement loop you can rely on every time. It’s very down to earth: four steps.

flowchart TD
  A[Signal: something is wrong
or there is a growth opportunity] --> B[Hypothesis:
what we change and how]
  B --> C[Controlled change
in code/config]
  C --> D[Verification:
offline + online]
  D --> A

Step 1. Detect a problem or opportunity

At this step you answer the question “where to look.” Examples:

• LLM eval on budgeted golden cases shows a drop in helpfulness.
• p95 latency for the "suggest_gifts" tool grew from 1.2s to 3.8s.
• cost_per_task for the "gift_selection" scenario increased by 40% after switching to a reasoning model.
• Conversion workflow_completed → checkout_success dropped by 5 pp after changing the UX wizard.
• 10 complaints appeared in the Store over a week: “gifts are more expensive than the specified limit.”

Important nuance: sometimes a signal doesn’t say “bad,” it says “can be better.” For example, you see that cost_per_task is noticeably below the acceptable threshold, and the quality score is already 9/10. That means you can try a more expensive model or a “smarter” scenario — maybe conversion will grow.

Step 2. Formulate a hypothesis

A hypothesis is not “we’ll rewrite the prompt,” but “we think that a specific change X in component Y will improve metric Z, because…”. Without the “because,” it’s a desire, not a hypothesis.

Examples:

• “If we enforce strict budget adherence in the system prompt and ask to always explain how the budget was used, helpfulness in budgeted cases will increase by at least 2 points.”
• “If we replace the expensive model with "gpt-mini" on the rerank step, cost_per_task will drop by 30%, and conversion won’t decrease by more than 1 pp.”
• “If we simplify the wizard (combine two steps into one) and rewrite the CTA, activation rate will increase by 5 pp.”

In code, it’s convenient to describe such a hypothesis explicitly, at least as an object:

// lib/improvement/hypothesis.ts
export type ImprovementHypothesis = {
  id: string;
  scenario: string;
  description: string;   // "What we change and why"
  targetMetric: string;  // e.g. "quality.helpfulness" or "conversion.checkout"
  successCriteria: string;
};

Yes, it’s almost like a Jira ticket, but at least in a typed form.

Step 3. Make a controlled change

Two words matter here: controlled and separate.

Controlled means the change is a PR/commit, linked to the hypothesis, has a changelog and, if possible, a feature flag or version. You didn’t just “tweak a prompt in prod,” you can say which release brought it.

Separate means you try not to mix three different hypotheses in one release. If you simultaneously:

• change the model,
• rewrite half of the system prompt,
• and add a new step to the UX,

then even if metrics improve, it will be hard to understand what actually worked. It’s much more honest to move in small increments.

Technically, changes can be anywhere:

• the agent’s system prompt (lib/prompt/systemPrompt.ts);
• MCP tool descriptions (description, inputSchema, annotations);
• the agent config (limits for reasoning steps, model choice for a specific tool);
• the widget code (CTA, step order, error texts).

Step 4. Verify whether it got better

Verification is divided into offline and online.

Offline verification is running golden cases through the new app version without real users. Here you already have:

• LLM evals (module 20.1);
• threshold/baseline logic (module 20.2).

You check how the quality scores changed for target scenarios: went up, down, or stayed the same. Also check safety cases: any prompt edits must first pass the safety suite.

Online verification is an experiment on real traffic. In the simplest version, you enable the new version for N% of users and compare:

• conversion to the target action (checkout, a repeat scenario run);
• cost_per_task;
• complaints/feedback.

After that, the loop either closes (the hypothesis is confirmed/rejected and documented) or spawns a new hypothesis.

4. An internal “improvements assistant” as a separate agent

Now the fun part: let’s give the LLM another role — not only to answer users but also to help you improve the app. This is an internal agent, separate from the user‑facing GiftGenius.

What it is

This assistant lives in your internal environment (in the same repo, in Dev Mode, in a separate ChatGPT App). It:

• reads logs and dialog samples;
• looks at metrics;
• analyzes the system prompt and tool descriptions;
• helps formulate problems and change proposals.

Essentially, you get a “virtual product/analyst” who:

• doesn’t get tired of reading dialogs;
• quickly finds repeating patterns;
• can write prompt and changelog drafts.

What input it accepts

It helps to formalize the input to make it easier for the agent. For example, a type:

// lib/improvement/assistant.ts
export type BadDialogExample = {
  id: string;
  userMessages: string[];
  appMessages: string[];
  qualityScore?: number;
};

export type ImprovementInput = {
  scenario: string;
  signals: ImprovementSignal[];      // from the previous section
  examples: BadDialogExample[];
  systemPrompt: string;
  toolsDescription: string;
};

You can build such an object by exporting 10–20 unsuccessful dialogs from logs for the "gift_selection" scenario and attaching the current system prompt and tool descriptions.

What output it should provide

It’s also convenient to fix the expected response:

export type ImprovementSuggestion = {
  patterns: string[];     // repeating problems
  promptPatches: string[]; // suggested fragments for the system prompt
  toolsPatches: string[];  // ideas for tool descriptions
  uxCopyIdeas: string[];   // UX text variants
  changelog: string[];     // short list of "what to change"
};

A meta‑prompt for such an agent will be roughly:

• “analyze the examples and signals”;
• “describe 2–3 problem patterns”;
• “propose 1–2 changes for the prompt, tool descriptions, and UX texts”;
• “return everything in strictly defined JSON.”

Then you manually take these fragments, discuss with the team, integrate into code, and run through evals and experiments. An important principle: this assistant does not change anything in production by itself. It generates ideas and text, not commits.

5. Types of changes: what you can actually “tune”

When you have such an assistant and a clear loop, it’s very easy to reduce everything to “give me a new version of the system prompt.” In reality, the improvement surface is much wider.

Prompt and instructions

The system prompt sets the role, tone, priorities, and hard rules (e.g., budget, safety, order of steps). You can:

• simplify it by removing contradictory or duplicate instructions;
• strengthen it by adding missing rules (as in the budget example);
• adapt it to different scenarios (separate sub‑prompts for "gift_selection", "post_purchase_help", etc.).

Tool descriptions help the model understand when to call a specific tool and what it does. Here improvements often boil down to:

• more explicit “Use this when… / Do not use when…”;
• clarifying wording (reduce overlap with other tools);
• adding information about consequences (destructiveHint, isConsequential).

Safety rules are the part of the prompt/descriptions responsible for behavior in sensitive domains. It’s best to touch them only after good safety evals.

Behavior architecture

Sometimes a prompt can’t fix the problem: you need to change the order of actions.

Examples:

• add a mandatory parameter clarification step before calling an expensive tool;
• move part of the computation into a separate, cheaper step (e.g., preliminary filtering on the backend);
• limit the number of consecutive tool calls in a single scenario.

These changes are usually described in the agent config or the MCP layer, not just in the prompt.

UX and copywriting

Yes, button and error texts are also part of quality. A GiftGenius that says:

“Error 500. Please contact the administrator.”,

creates a very different impression from:

“We couldn’t get a response from the store. Your selected ideas are safe — please try to complete the purchase a bit later.”.

Intermediate screens, hints, the structure of wizards — all of this affects that very activation rate and conversions you measure.

Economics

Here we play the well‑known “quality ↔ cost” game:

• model choice (expensive with reasoning, fast/cheap);
• depth of reasoning/agent steps (how many iterations we allow);
• dialog limits (e.g., no more than N re‑computes of a selection in one session);
• “cheap” fallback modes when the token/limit budget is exhausted.

Signals from here feed cost_per_task, cost_per_user, margin and are combined with the quality score.

6. Guardrails: what must not be handed over to the LLM

When an “improvements assistant” and a convenient loop appear in the system, it’s very tempting to press the “Auto‑optimize” button and go grab coffee. Let’s agree right away where that button is forbidden.

Changes in permissions (OAuth scopes, MCP tools, data access) are always a human‑in‑the‑loop zone. No model should decide on its own that the app can now read orders, touch payments, or send emails to users. The same applies to the commerce flow: any changes around ACP/Stripe, limits, payment types, and refunds go through human review and testing.

The app’s safety profile (which domains it has the right to advise on, where it must refuse) is also not something to delegate to an LLM. A model can help draft the rules text, but the decision about which topics you trust the app with remains yours.

Data and logging policy (what we log, how long we store it, how we respond to deletion requests) — in the same list. An LLM can suggest a Privacy Policy structure but must not change retention in code without your involvement.

What can be semi‑automated? Wording of prompts, tool descriptions, and UX texts, tool priorities (within reason), additional clarifying questions. You can trust the assistant as a source of ideas for all this, but the final say and verification belong to a human and eval scripts.

7. End‑to‑end improvement loop example (GiftGenius)

Back to our hero.

Problem

The user specifies a budget, but GiftGenius often suggests gifts above this limit. Logs show many dialog continuations like “no, that’s too expensive” and “make it cheaper.”

Signals

First you see quality: an LLM eval on golden cases like “pick a gift up to $50” gives a helpfulness of about 6/10. The judge regularly writes in reason that “gifts exceed the budget” or “it’s not explained how the budget is accounted for.”

In parallel, product and economic signals arrive:

• conversion to checkout_success for budgeted scenarios is lower than without a budget;
• some users abandon the scenario after seeing options that are too expensive;
• cost_per_task for such scenarios is higher because the app recomputes the gift selection several times on “make it cheaper” requests.

Analysis with the improvements assistant

You collect 20 dialogs where users complained about price and assemble an ImprovementInput:

• scenario = "gift_selection_with_budget";
• signals showing drops in helpfulness and conversion;
• examples with dialogs;
• the current system prompt and tool descriptions.

You feed this to the internal improvements agent. In response it returns, for example:

• Patterns:
  • a budget phrased as “around $50” is interpreted too loosely;
  • the system prompt lacks an explicit requirement to “never suggest gifts above the limit”;
  • answers don’t explain to the user how exactly the budget is taken into account.
• Suggestions:
  • add an instruction to the system prompt about strict adherence to the limit;
  • ask the model to always state that “all options ≤ X”;
  • add a clarifying question when the budget sounds vague (“roughly,” “about”).

Hypothesis and change

You formulate a hypothesis:

“If we explicitly fix strict adherence to the limit and ask to explain budget use, helpfulness for budgeted cases will increase to at least 8/10, and purchase conversion will grow by 3 pp.”

You add the following fragment to the system prompt:

export const budgetRule = `
If the user specifies a budget (for example, "up to $50" or "around €30"),
treat this amount as a STRICT upper limit.

Never suggest options that exceed this limit.
In every reply, explicitly state that all options fit within the budget
and how exactly (for example: "all gifts are no more than $45").
`.trim();

And you add budgetRule to the overall GiftGenius system prompt.

Offline validation

You run the “budgeted gifts” golden case suite against the new version:

• LLM eval helpfulness rises from 6.0 to 8.5;
• correctness also grows: the budget is respected;
• safety does not worsen (at least).

If instead helpfulness doesn’t grow or safety drops, the change doesn’t pass and the hypothesis is revised.

Online test

Then you launch an experiment:

• 10% of users get the new prompt version (variant B);
• the remaining 90% — the old one (variant A).

A week later you look:

• conversion workflow_completed → checkout_success for B becomes, say, 13% instead of 10% for A;
• cost_per_task barely changes;
• the share of dialogs with “too expensive” complaints drops.

The experiment is deemed successful.

Rollout

After that:

• roll out the new prompt to 100% of traffic;
• add a new golden case “soft budget up to $50” to the regression suite;
• record the result in the changelog and, possibly, in tech notes so that in six months it’s clear why there is such a strict budget phrasing in the prompt.

The loop is closed. The next iteration might address, for example, recommendations for people with very rare hobbies or optimizing the cost of selection.

8. A mini roadmap for a self‑improving app

To avoid this looking like “another bajillion tasks on top,” it helps to see the minimum needed for the app to already be considered self‑improving, and what can be added later.

Version 1.0: minimal improvement loop

At the first step, three things are enough.

First, structured logs and basic SLOs. You already know how to log tool_invocation, workflow_completed, checkout_* with requestId, userId, scenario, appVersion, costEstimateUsd. SLOs are built on top of this: latency, error rate, checkout success.

Second, 10–20 golden cases and an LLM eval script. A small but well‑chosen set of examples for key scenarios + safety cases. One CLI script that runs them through the app and the judge and outputs JSON scores.

Third, a simple ritual every 2 weeks. Sit down (alone or with the team), open:

• 2–3 dashboards for SLO, cost, and product metrics;
• the LLM eval report on golden cases;

and choose 1–2 hypotheses for the next cycle. Formulate, document, make a small release, verify.

Version 2.0: a “smart” improvement loop

At the next level you get:

An internal improvements assistant. This is a separately defined agent (or ChatGPT App) that accepts an ImprovementInput and returns an ImprovementSuggestion. It helps you avoid spending hours manually combing through dialogs.

Automated reports. Based on logs and evals you can generate:

• clusters of problematic cases (e.g., all instances where the user re‑wrote the budget);
• ready‑to‑use drafts of prompt and tool description changes;
• a short changelog.

CI hooks. Any change to prompts, agent configs, or tool descriptions automatically triggers:

• the functional golden suite;
• the safety suite.

If safety cases fail or the quality of key scenarios no longer meets thresholds, the build is red — no release.

9. Practice

Exercise 1. A mini feedback loop for your app

Take one key scenario of your application. For GiftGenius we already chose “gift selection and purchase”; you can pick a similar or your own scenario.

Describe in free form (or create a small improvement-plan.md):

Which signals you will track. One each:

• technical: for example, p95 latency for the tool that does the main work;
• economic: cost_per_task for this scenario;
• product: conversion workflow_started → workflow_completed or to the target action;
• quality: average quality_score across several golden cases for this scenario.

Next, fix which thresholds you consider degradation. Not “we’ll look someday,” but quite concrete:

• p95 > 5 seconds — bad;
• cost_per_task increased by more than 30% without revenue growth — an alarm;
• quality_score fell below 7 — needs investigation.

And formulate the first hypothesis. For example:

“I suspect we ask too many clarifying questions. If we shorten the wizard by one step and make the questions a bit more general, the activation rate will increase and helpfulness will hardly suffer. I will verify this with a small UX change and an A/B test.”

This is no longer just “we’ll improve the UX,” but a quite specific step in the loop.

Exercise 2. A meta‑prompt for the improvements assistant

Try drafting a prompt for the internal improvements agent for your app. You can start simple:

• Describe who it is: “You are the quality analyst of App N who…”
• List the inputs it receives: dialogs, signals, system prompt, descriptions.
• Formulate tasks:
  • find 2–3 problem patterns;
  • propose 1–2 changes in the prompt, tools, and UX texts;
  • compile a short changelog.
• Describe the response format: structured JSON with fields patterns, suggestions, changelog.

Even if you don’t call this agent from code yet, a single such prompt will already help you better structure your thinking about how you improve the app.

10. Common mistakes when building an improvement loop

Mistake #1: optimizing for a single metric only.
It’s very tempting to focus on a single thing. You can chase only cost, be happy about a lower OpenAI invoice — and miss how the quality score, conversions, and retention start sliding. You can, conversely, crank quality to 10/10 on reasoning models and forget that each scenario now costs a dollar. An improvement loop must look at a bundle of metrics: quality ↔ money ↔ user behavior.

Mistake #2: “magical” prompt changes without measurement.
Saying “I rewrote the system prompt, it should be better now” without golden cases and evals is a way to surprise yourself in a couple of weeks. Any prompt change — especially in production — must go through a clear pipeline: a case set, LLM eval before/after, and, if necessary, an online experiment. Otherwise you’re not improving the app — you’re scattering quality in a random direction.

Mistake #3: auto‑deploying changes from the LLM assistant.
Even if the improvements assistant writes incredibly nice prompt fragments and convincing tool descriptions, that is no reason to push them to prod without review. The model may not see all business constraints, safety risks, and the context of your metrics. Its role is a consultant, not a DevOps with release rights.

Mistake #4: no link between changes, hypothesis, and signals.
Sometimes teams make edits “by feel” and don’t record which signal led to the change and which hypothesis they’re testing. As a result, a month later no one remembers why a strange paragraph appeared in the prompt and who needed it. A good quality PR must answer at least three questions: “what was hurting?”, “what are we changing?”, “by which metric will we know it got better?”

Mistake #5: forgetting about safety while improving.
In the pursuit of usefulness it’s easy to relax safety constraints, remove “excess refusals” from the prompt, or forget to run the safety suite. Any change to the system prompt, tool descriptions, and agent behavior must first pass the safety eval set. If even one case that used to pass now fails, that’s a stop signal — no matter how pretty the other metrics look.

Mistake #6: trying to “optimize everything at once.”
Rewriting half the prompt, changing the model, adding another MCP tool and a new wizard — all in one release — sounds productive, but completely kills the ability to understand which change created the effect. An improvement loop is effective only when it’s iterative and focused: one or two hypotheses, a small set of changes, a clear rollback plan.

Mistake #7: turning the improvement loop into a rare “big cleanup day.”
If you run a grand “quality day” once every six months and live without evals and metric analysis the rest of the time, the app will be “floating with the current” most of the time. A small but regular ritual is far more useful: look at signals every week or two, update hypotheses, and take small steps. This is how your ChatGPT App stops being static and starts truly self‑improving.