Monetization, pricing, and “cost ↔ quality” experiments

1. Why think about monetization right now

Up to this module, the main question was “does this even work?”. Now we add the next level of difficulty: “does it pay for itself?”.

For LLM apps this is especially painful: variable costs (LLM tokens, rerank models, embeddings) can blow up your budget compared to your usual “a $20 server and a $15 Postgres”. Ignoring this means getting an end‑of‑month bill from OpenAI comparable to a mortgage.

So today we have three big topics:

1. What monetization models exist for a ChatGPT App and for our GiftGenius in particular.
2. How to connect pricing ↔ cost_per_task and avoid selling “100 gift selections for $1” when a single selection already costs $0.15.
3. How to set up A/B experiments “cost ↔ quality”: change the model, prompts, UX and log it so that in a couple of weeks you can make data‑driven decisions rather than going by gut feel.

At the same time, we gently prepare the ground for the next module on LLM evals and quality_score, but we won’t dive into code there yet.

2. Monetization models for a ChatGPT App: B2C, B2B, freemium, and upsell

If you strip away the LLM magic, monetization models here look very similar to those used in regular SaaS and mobile apps. But a conversational interface has nuances: users often don’t feel “what’s free and what’s paid”, and you need to design that carefully in the UX.

Let’s go through the main options using GiftGenius.

B2C: everyday users and gifts

Here your customers are everyday people who come to ChatGPT and ask: “pick a gift for a space fan for 50 dollars”. You don’t have to sell your own products — you can only recommend gifts for the user.

Typical B2C models:

1. One‑off purchase.
   The user pays for a specific scenario. For example: 3 ideas for free, then a paid “pack” of 10 more ideas for a single recipient.
2. Subscription.
   A monthly fee for access. For GiftGenius this could be “up to 100 selections per month” or “unlimited selections for a frequent gifter”.
3. Freemium (free vs paid tiers).
   The basic scenario is free (up to N selections per month, reduced functionality), and the paid tier gives more limits, a stronger model, additional selection formats, and history. This is the most typical model for a ChatGPT App: “inside ChatGPT — basically free, premium features — paid”.
4. In‑app upsell.
   A user runs a free basic selection, sees a decent result, and you gently offer: “want me to do a deep selection for $X using wishlist, social networks, etc.?” or “buy a gift card right away”.

B2B: teams, companies, and corporate gifting

This is where HR, marketing departments, and “people responsible for gifts for employees/clients” come into play.

The traditional set:

• Per‑seat license.
  For example, an “HR‑Team” plan for 10 people, each with access to GiftGenius, and gift/budget reports.
• Per‑company license.
  “Up to 500 employees, a fixed monthly price, unlimited selections inside.”
• Additional enterprise features.
  A separate admin console, HRIS/CRM integrations, custom reports, SLA.

In both cases you’re not counting “how much does one selection cost”, but rather cost_per_user_per_month or cost_per_tenant_per_month and comparing that to the license price.

How to choose a model for GiftGenius

To avoid getting lost in theory, you can take a simple starting option:

• B2C: freemium.
  3 selections per month for free; beyond that — a $5/month subscription with unlimited selections and a premium model.
• B2B: per company.
  The “HR team” plan for $99/month, including up to 500 employee selections, HR‑system integration, and reporting.

Later, when you have real data on cost_per_task and conversion, you can tune all this. In practice, these numbers are currently ballpark guesses: they seem reasonable, but we haven’t yet verified how much a completed scenario costs us. In the next section we’ll tie such plans to actual unit cost — cost_per_task.

3. The link between price and unit cost: what is cost_per_task

Now we move to the most important part: how not to turn yourself into a charity named after GPT‑5.

Intuition: price ≥ cost_per_task × margin

In the previous topic you already saw the concept of cost_per_task — the total cost of one successful scenario: from “the user started a selection” to “received the result” (and possibly paid for something).

It includes:

• LLM costs (tokens * price_per_token on input/output, possibly including rerank models, embeddings, etc.);
• a share of infrastructure cost per task (servers, DB, queues, MCP gateway) — often computed from aggregated data;
• optionally — transactional costs (Stripe fee, fraud checks), if you measure cost_per_task up to “cash in hand”.

The idea is simple: the price of a scenario or subscription must be higher than the average unit cost of one scenario multiplied by the “margin buffer”.

If we greatly simplify:

price_per_task >= cost_per_task * ( 1 + margin )

We won’t clutter the lecture with numbers and margins in percentages; the key is the intuitive rule.

Example for GiftGenius

Suppose you’ve already implemented cost logging from the previous lecture and you have an aggregated report:

• average cost_per_task (one completed gift selection) = 0.15 USD;
• this already includes LLM tokens (several calls to suggest_gifts, rerank, and the final summary) and a share of infrastructure.

Then you look at the scenario:

• a free user runs a selection and sometimes buys a $50 gift card;
• conversion to purchase among completed selections, say, 5%.

We won’t dive into full unit economics yet, but we can already estimate: if out of 100 selections:

• you spend 100 × $0.15 = $15;
• 5 end with a $50 checkout;
• revenue is 5 × $50 = $250.

Looks good: roughly ($250 – $15), plus Stripe fees, taxes, and other pain. But it’s important to understand that with an overly generous subscription (say, 100 selections for $1) you can easily go negative.

Mini code example: saving cost_per_task in TypeScript

Suppose you have an MCP tool that completes the selection workflow and knows its total cost:

// Type for final scenario metrics
type TaskMetrics = {
  taskId: string;
  userId: string;
  costPerTaskUsd: number;
  modelName: string;
  completedAt: string;
};

// A simple function for logging metrics
async function logTaskMetrics(metrics: TaskMetrics) {
  console.log(JSON.stringify({ 
    level: "info",
    event: "workflow_completed",
    ...metrics,
  }));
}

// Somewhere in the completion handler for the selection:
await logTaskMetrics({
  taskId: context.taskId,
  userId: context.userId,
  costPerTaskUsd: context.costEstimateUsd, // computed from tokens
  modelName: context.modelName,
  completedAt: new Date().toISOString(),
});

Such a log is easy to aggregate in a dashboard to see the distribution of cost_per_task by model, user, and scenario.

4. Pricing: how to turn cost_per_task into real prices

Now that we have cost_per_task, we need to decide what to charge and how much.

A simple rule for B2C

For B2C you can choose an empirical rule:

“We are willing to spend no more than X% of revenue on LLM+infra.”

For example, you decide you don’t want to spend more than 20% of turnover on LLM costs. Then:

• if cost_per_task = $0.15, the minimum price for one paid scenario should be ≈ $0.75, so that 0.15 is around 20% of 0.75;
• if you sell a subscription, estimate how many average scenarios will be attributed to one subscriber per month and multiply.

It’s perfectly fine to start by eyeballing it and then adjust prices when real data appears (spoiler: it won’t appear immediately).

A simple rule for B2B

In B2B you usually look at:

• cost_per_user_per_month or cost_per_tenant_per_month;
• willingness to pay (the magnitude of the problem you solve).

For example, if an HR team uses GiftGenius to distribute gifts worth tens of thousands of dollars a year, a $99/month subscription looks modest, even if your LLM spend on that team is only $10/month. The main thing is to understand that you do not want to end up where cost_per_tenant = $80 while the subscription is $50.

And yes, that happens if you go “we’re AI, let’s keep everything free for now and we’ll figure it out later”.

A small “guard” function on the server

You can have a simple guard in code that nudges you when costs go out of reasonable bounds for a chosen price:

function checkPricingSafety(params: {
  avgCostPerTaskUsd: number;
  plannedPricePerTaskUsd: number;
  maxCostShare: number; // e.g., 0.3 = 30%
}): boolean {
  const share = params.avgCostPerTaskUsd / params.plannedPricePerTaskUsd;
  return share <= params.maxCostShare;
}

// Example:
checkPricingSafety({
  avgCostPerTaskUsd: 0.15,
  plannedPricePerTaskUsd: 0.75,
  maxCostShare: 0.3,
}); // true — OK, 20% < 30%

This doesn’t replace a financial model, but it gives a quick sanity check (a “does this make sense” check), especially when you experiment with prices.

5. Experiments: model/agent vs cost and conversion

Now to the most interesting part: A/B experiments.

The intuition is straightforward:

• Variant A — a more expensive model / more complex workflow;
• Variant B — a cheaper model / simplified workflow;
• we want to understand how this simultaneously affects:
  • cost_per_task,
  • result quality (as perceived by users and by future LLM ratings),
  • business metrics (conversion, revenue).

What exactly to experiment with

There are three main axes of experiments:

1. Model.
   For example, GPT‑5 vs GPT‑5‑mini or a completely different family. Typically, an expensive model gives better quality and higher cost_per_task, a cheaper one — the opposite.
2. Agent logic / prompts.
   More detailed steps, longer prompts, complex reasoning — higher quality but more expensive; minimalist logic — cheaper and sometimes almost as good.
3. UX format.
   A long wizard with lots of fields and hints vs a quick inline mode. Even with the same model, the number of tokens and steps can differ drastically.

You can already implement all these variations — the important part is to wrap them into experiments with logging.

What fields to log for experiments

On top of the fields you log for cost (tokens, model, cost_estimate, user_id, request_id, etc.), add experiment fields:

• experiment_id — a unique experiment identifier (for example, "gift_model_ab_2025_11").
• variant — which branch a specific user is in: "A", "B", "control", "treatment", etc.
• model_name or agent_version — so you don’t have to remember which configuration was used.
• scenario outcome:
  • whether workflow_completed;
  • whether checkout_success;
  • final cost_per_task.
• optional — quality_score (more on it soon — this bridges to the LLM evals module).

Example of an experiment event JSON log

A typical log event might look like this:

{
  "level": "info",
  "timestamp": "2025-11-21T20:15:03.123Z",
  "event": "experiment_task_result",
  "experiment_id": "gift_model_ab_2025_11",
  "variant": "A",
  "user_id": "user_123",
  "task_id": "task_456",
  "model_name": "gpt-5.2",
  "workflow_completed": true,
  "checkout_success": false,
  "cost_per_task_usd": 0.18,
  "quality_score": null,
  "request_id": "req_abc",
  "trace_id": "trace_xyz"
}

These records aggregate nicely in any analytics: you can build tables like “variant A vs B by cost/conversion/revenue”.

Code example: logging an experiment in an MCP tool

Imagine your MCP server has already computed the scenario cost (cost_per_task) and knows which experiment branch the user is in:

type ExperimentContext = {
  experimentId: string;
  variant: "A" | "B";
};

async function logExperimentResult(params: {
  ctx: ExperimentContext;
  userId: string;
  taskId: string;
  modelName: string;
  costPerTaskUsd: number;
  workflowCompleted: boolean;
  checkoutSuccess: boolean;
}) {
  const event = {
    level: "info" as const,
    event: "experiment_task_result",
    timestamp: new Date().toISOString(),
    experiment_id: params.ctx.experimentId,
    variant: params.ctx.variant,
    user_id: params.userId,
    task_id: params.taskId,
    model_name: params.modelName,
    cost_per_task_usd: params.costPerTaskUsd,
    workflow_completed: params.workflowCompleted,
    checkout_success: params.checkoutSuccess,
  };

  console.log(JSON.stringify(event));
}

Somewhere up the stack you decide which variant the user belongs to (by user_id, tenant_id, or randomly) and pass the ExperimentContext into the workflow handler. At this level, we’ve fixed how to log experiments: which fields are needed and where to write them. Next, we’ll talk about how to turn such experiments into clear product hypotheses and pricing decisions rather than just a pile of logs.

6. A bit about quality_score and LLM evals

I’ll cover this topic in detail in the 20th module; for now, just the idea: quality_score is a quality score of the answer/solution on a scale, say, from 0 to 10, often from a separate LLM “judge” model. I’ll talk more about LLM‑as‑judge in module 20.

We don’t need implementation details now — that’s the next module — but it’s important to grasp the concept:

• besides money, we want to measure quality;
• we can ask either a human or a second model to assess: “how well did GiftGenius select a gift on a 10‑point scale?”;
• then we can see how quality_score correlates with:
  • conversion to purchase;
  • user retention;
  • willingness to pay.

From a logging perspective, it’s just one more field:

type ExperimentResultEvent = {
  experiment_id: string;
  variant: string;
  user_id: string;
  task_id: string;
  cost_per_task_usd: number;
  quality_score?: number; // 0-10, may be undefined
};

We’ll stop here for today’s lecture: the details of LLM evals, golden cases, and “LLM‑as‑judge” will appear later in the course. For now it’s enough to understand where to plug this score into experiments. It’s exactly quality_score that saves you from the classic “optimize only for cost” mistake: it lets you numerically see where you’ve cut costs too far, started losing quality, and with it — conversion and revenue.

7. How to use experiments for pricing and monetization

Now we’re not just logging experiments — we frame them as clear business hypotheses with success metrics and monetization impact. Simply logging an experiment_id is not enough: it’s important to frame product changes as hypotheses with explicit success metrics.

Hypothesis example: expensive model vs cheap model

Imagine this experiment for GiftGenius:

• Variant A — expensive model (GPT‑5), rich reasoning, a long wizard.
• Variant B — cheap model (GPT‑5‑mini), a slightly simpler prompt, and a shorter conversation.

Hypothesis: switching to the cheaper model will reduce cost_per_task by at least 50%. At the same time, quality by user perception and LLM rating (our quality_score) will drop by no more than 5–10%, and checkout conversion won’t fall.

Technically, for each task you log the same fields as in section 5.2:

• experiment_id = "gift_model_ab_2025_11";
• variant = "A" or "B";
• model_name;
• cost_per_task_usd;
• workflow_completed;
• checkout_success;
• quality_score (when LLM evals are in place).

In a week or two you can:

• compare average cost_per_task for A and B;
• compare checkout rate (the share of scenarios with successful payment);
• compare average quality_score, if available.

If B barely loses in quality but is twice as cheap, you can either:

• switch to B and increase your margin;
• or keep the model but lower the subscription price for users (thus boosting growth).

Hypothesis example: a quality upsell

Another hypothesis: if, after 3 free ideas, you show a premium upsell “full gift report + greeting‑text recommendations” for $4.99, checkout conversion will increase by at least 2 percentage points (2 pp). At the same time, cost_per_task will increase by no more than $0.05.

This experiment is less about the model and more about UX and product logic. But technically it’s the same:

• different UX variants via variant;
• logging cost and revenue per scenario;
• uplift analysis (how much more money the new logic brings without blowing up costs).

Code example: logging simple revenue per task

Sometimes it’s convenient to log revenue alongside cost for the scenario:

type RevenueEvent = {
  taskId: string;
  userId: string;
  experimentId?: string;
  variant?: string;
  revenueUsd: number;
  checkoutSuccess: boolean;
};

async function logRevenue(event: RevenueEvent) {
  console.log(JSON.stringify({
    level: "info",
    event: "task_revenue",
    timestamp: new Date().toISOString(),
    ...event,
  }));
}

By joining task_revenue and experiment_task_result on taskId, you can compute for each branch:

• average revenue_per_task;
• average cost_per_task;
• and build a simple ROI.

8. Practical exercise: an A/B experiment for GiftGenius

To have something concrete and ground the theory in practice, let’s articulate what the “expensive vs cheap model” experiment would look like for GiftGenius — step by step, as a practical exercise.

What we change

• Variant A:
  • model gpt-5;
  • a more detailed system prompt and agent steps;
  • possibly more intermediate reasoning calls.
• Variant B:
  • model gpt-5-mini;
  • a slightly more compact prompt;
  • fewer auxiliary tool calls, a simplified flow.

How we assign users to variants

The simplest way — by a hash of user_id:

function assignVariant(userId: string): "A" | "B" {
  const hash = Array.from(userId).reduce((acc, ch) => acc + ch.charCodeAt(0), 0);
  return hash % 2 === 0 ? "A" : "B";
}

This guarantees roughly even distribution, and a given user always lands in the same variant.

What we log

When the selection workflow completes, you log the same set of fields as in the previous sections (5.2 and 7.1) and add revenue:

• experiment_id = "gift_model_ab_2025_11";
• variant from the function above;
• model_name, the model actually used;
• cost_per_task_usd, the total cost of tokens and infrastructure;
• workflow_completed (true/false);
• checkout_success (true/false);
• revenue_usd (0 or the purchase amount).

Optionally (later in the course) you’ll add quality_score.

These data then land in your logs/analytics where you can lay them out in tables:

From a table like this, you can see that B yields almost the same money at half the cost — a strong argument in its favor.

9. Visual diagram: what the “cost ↔ quality ↔ monetization” loop looks like

To put it all together, let’s draw a small “data goes back and forth” diagram:

flowchart TD
    U[User in ChatGPT] --> A["ChatGPT App (GiftGenius)"]
    A --> E[Experiment module
assigns variant A/B]
    E --> AG[Agent / MCP tools
with different models]
    AG -->|LLM calls| L[Usage & cost logs]
    AG -->|Selection results| UI[Widget / chat reply]
    UI -->|behavior: clicks, purchases| BE[Commerce backend]

    L --> M[Metrics: cost_per_task,
cost_per_user]
    BE --> M
    M --> D[Pricing & experiments dashboard]

    subgraph "Future module 20"
      J[LLM judge
quality_score]
      J --> M
    end

Right now you are in the middle of this diagram: you can log cost and revenue, and you are adding experiments and pricing on top. In the next module, the Judge Dredd topic will arrive (LLM judge).

10. Common mistakes when working on monetization and experiments

When you have the overall picture in mind, it’s easier to see where things usually break. Below is a set of common mistakes that’s handy as a checklist when working on monetization and cost experiments.

Mistake #1: optimizing only for cost and ignoring quality.
A common scenario: you switch to a cheaper model, see a nicer OpenAI bill, and declare victory. A month later you notice that users buy gift cards less often, come back less, and support volume grows with “I got some nonsense”. If you log neither quality_score nor at least proxy metrics (clicks on ideas, saves, conversions), you can easily end up “cheap but useless”.

Mistake #2: computing cost_per_task only for LLM while ignoring infrastructure and payments.
Sometimes developers carefully count tokens but forget about Redis, queues, third‑party APIs, Stripe fees, and so on. As a result, cost_per_task ends up severely underestimated, and prices look more comfortable than they really are. Infrastructure is usually computed from aggregated data, but its share still needs to be included in the scenario’s unit cost.

Mistake #3: changing models/UX without explicit experiment_id and variant.
“We tweaked the prompt a bit; it seems better” — a month later, no one remembers when it changed, what data it was based on, or what it led to. Without explicitly tagging experiments in logs (experiment_id, variant) and tying them to specific releases, it’s hard to analyze the past and prove improvements aren’t accidental.

Mistake #4: making decisions on too little data or too early.
If an experiment runs for two days and, based on ten purchases, you decide that model B is “far more profitable,” that’s a textbook example of a noisy statistical inference. You need at least a minimal horizon — a week, preferably more — and enough scenarios to compare averages and conversions. We don’t go into statistics in this lecture, but keep the rule “don’t draw conclusions from 5 events” in mind.

Mistake #5: using complex pricing without a simple mental rule.
You can design a three‑tier plan, prices in different currencies, and referral discounts, yet not have a simple product rule like “we’re willing to spend no more than X% of revenue on LLM cost” or “scenario price must not drop below 3× the average cost_per_task”. Without such guardrails, it’s easy to lose margin and not notice until month‑end.

Mistake #6: forgetting the link between monetization, marketing, and growth.
Monetization and pricing don’t live in a vacuum: the more expensive the subscription, the higher the churn and the lower the conversion; the lower the price, the higher the demands on cost optimization. The mistake is to look only at “how much we earn now” and not tie it to acquisition/activation/retention metrics, which we’ll cover in the next topic of the module. It’s best to log pricing experiments in the same way as quality and cost experiments to keep the full picture.