Metrics and SLO: p95, error‑rate, webhooks, availability

1. Why metrics and SLO matter in a ChatGPT App

Imagine two team states.

In the first, everyone lives by the principle “seems to work.” As long as users don’t contact support and don’t post angry tweets, everything is fine. From time to time someone opens the logs, scrolls through a long wall of lines, nods, and closes the tab.

In the second, the team has a few simple dashboards:

• p95 of latency for the main MCP tool.
• error‑rate for MCP and for checkout.
• webhook availability.
• conversion to payment along the funnel.

And there are 3–5 SLO: “95% of calls to the recommend_gifts tool are faster than 2 seconds,” “share of MCP‑tools errors < 1%,” “checkout availability ≥ 99.5%,” “conversion from widget to successful payment ≥ 10%.”

In the second scenario you:

• quickly understand that something is wrong with the app, even before complaints arrive;
• can measure the effect of any change (a new recommendation algorithm, SDK migration, a new provider plan);
• and by the time you get to the module about the Store and review, you can honestly say: “we have quality criteria and we know how well we meet them.”

We’ll talk about very straightforward things: what p95 is and why it’s better than the average, how to compute error‑rate and availability, which metrics matter for the commerce part with webhooks, and how to assemble simple but useful SLO from all of this. We’ll break it down using our training app GiftGenius — a commerce scenario with a recommendation MCP tool, checkout, and payment webhooks.

2. Basic metrics for a ChatGPT App: what exactly to measure

Latency: p50/p95/p99 instead of the “overall average”

Latency is the time from the start of an operation to its logical end. In our stack there are several such operations:

• calling the MCP tool recommend_gifts (gift selection);
• calling a tool that creates a checkout intent in ACP;
• processing a webhook for a successful payment.

It’s important to understand that we only partially control the overall delay the user sees in ChatGPT. There is the platform’s contribution (model inference, OpenAI network latency), and there is our contribution — tools, backend, DB, payment provider. For a SLI (Service Level Indicator — a measurable service quality indicator) on latency, you usually measure our portion: from the request entering the App/MCP to our server’s response being ready.

The average response time is deceptive here. If 90% of requests arrive in 100 ms and 10% take 5 seconds, the average will be around half a second and the chart looks “green.” But every tenth user experiences a 5-second freeze — UX suffers noticeably.

That’s why percentiles are used: p50 (median), p95, p99. p95 is a threshold below which 95% of requests fall. As SRE guides emphasize, percentiles “don’t let outliers hide in the average” and make visible the experience of those 5–10% of users who always end up in the tail of the distribution.

An easy way to think about it: p50 tells you how the “typical” user lives, and p95/p99 show how painful it is for the patient folks for whom things “keep lagging.”

Error‑rate: the fraction of unsuccessful requests

Error‑rate is the ratio of unsuccessful requests to the total number of requests over a period. The main data source is usually either structured logs or metrics with labels status="success" | "error".

In our stack there are several natural error‑rate targets:

• at the MCP‑tools level: the share of recommend_gifts calls that ended with an error (exception, external API HTTP 5xx, timeout);
• at the ACP/checkout level: the share of unsuccessful checkout attempts (API error, payment provider unavailable);
• at the webhook handler level: the share of webhooks whose processing ended unsuccessfully.

A tricky point: ChatGPT Apps may have “silent” errors. For example, the MCP tool returned an error, the model apologized and continued the conversation without surfacing the technical error in the header. Formally ChatGPT shows a human reply, but from a reliability perspective your stack failed. Therefore, for error‑rate it’s important to count exactly the engineering statuses, not “the final UX reply from GPT.”

Availability: service availability

Availability is the percentage of requests successfully served over a period. Conceptually this is the same as error‑rate, just inverted: “how many were successful” rather than “how many were unsuccessful.” A classic example: availability = successful_requests / total_requests * 100%.

Applied to our stack:

• MCP server availability: how many JSON‑RPC calls from ChatGPT completed with a correct response in the last hour;
• checkout‑API availability: what share of calls to /api/checkout ended with HTTP 2xx;
• webhook endpoint availability: what share of incoming webhooks from the payment provider received a correct response from us (usually HTTP 200).

If a payment provider’s documentation promises, say, 99.9% availability and you have 97%, the problem is most likely not on their side.

Commerce metrics: conversion and funnel

GiftGenius is a commerce scenario. Here not only the tech matters (responses are fast, few errors) but also the business outcome — how often recommendations turn into paid orders.

Enter the conversion funnel:

• users who were shown a widget (view);
• users who selected a gift (selection);
• users who clicked “start checkout” (checkout started);
• users whose order status became “paid” (paid).

From this funnel you can compute conversion “from widget to payment,” and that’s also an SLI — a measurable service quality indicator. For example, “successful payments / users who saw the widget over a day = 7%.” If latency suddenly grows or checkout sometimes times out, the funnel will start narrowing at unexpected stages.

If you use Instant Checkout

All the described commerce metrics also apply to Instant Checkout based on the Agentic Commerce Protocol. In this case, instead of your own /api/checkout, you have a standardized Agentic Checkout API: ChatGPT calls your REST endpoints POST /checkout_sessions, POST /checkout_sessions/{id}, POST /checkout_sessions/{id}/complete (and optionally — cancel and GET), each time receiving the “true” state of the cart and checkout from you.

Metrics‑wise this changes nothing: you’ll calculate p95 latency, error-rate, and availability not for an arbitrary /api/checkout, but for these standard endpoints. SLOs like “checkout p95 < 3 seconds, error-rate < 2%” simply apply to checkout_sessions calls rather than to a custom API.

A world of its own: webhook metrics

Webhooks are asynchronous events (for example, payment_succeeded from the payment provider) that move an order through its lifecycle. If webhooks are processed poorly, the app may nicely recommend gifts, but orders will “hang” with a status of “waiting for payment” or in some undefined state.

If you integrate not directly with Stripe/the payment provider but via Instant Checkout, the role of “payment provider webhooks” is assumed by order events from Agentic Checkout: your backend sends OpenAI events like order.created and order.updated to a dedicated webhook URL. It’s exactly the same class of entities: asynchronous events that determine the final order status; the recipient is ChatGPT, not your frontend.

Accordingly, the very same metrics — success rate, latency, error-rate — will be computed not for Stripe webhooks, but for your own order‑event webhooks toward OpenAI. In SLO you can explicitly state “at least 99% of order.* events are delivered and successfully processed over 7 days; p95 processing delay < 500 ms” — and that will be a correct SLI/SLO specifically for Instant Checkout.

For webhooks you typically track:

• webhook success rate — the share of webhooks whose business logic completed successfully (including retries);
• webhook latency — the time from webhook ingress to the end of processing;
• webhook error rate — the share of cases where webhook processing ended with an error (validation failed, DB unavailable, timeout, and so on).

For example, you can set an SLO: “at least 99% of webhooks are processed successfully over 7 days” and “webhook processing p95 < 500 ms.”

3. How to technically measure metrics in GiftGenius

Let’s move from theory to code. We need to understand where to instrument measurements so we can later aggregate data into p95, error‑rate, and other metrics.

To avoid dragging a specific Prometheus or Datadog into the lecture, we’ll assume we have a simple logMetric function or JSON event logging. Any observability system can build the necessary charts on top of this.

Measuring an MCP tool’s latency

Suppose we have a GiftGenius MCP server in TypeScript and a recommend_gifts tool. Wrap the business logic with a timer:

// mcp/tools/recommendGifts.ts
import { logMetric } from "../observability/metrics"; // hypothetical helper

export async function recommendGiftsTool(input: RecommendInput) {
  const startedAt = performance.now(); // start time
  try {
    const result = await recommendGifts(input); // business logic
    const duration = performance.now() - startedAt;

    logMetric("tool_latency_ms", duration, {
      tool: "recommend_gifts",
      status: "success",
    });

    return result;
  } catch (error) {
    const duration = performance.now() - startedAt;

    logMetric("tool_latency_ms", duration, {
      tool: "recommend_gifts",
      status: "error",
      error_type: "exception",
    });

    throw error;
  }
}

Here logMetric can simply write a JSON log to stdout:

// observability/metrics.ts
export function logMetric(
  name: string,
  value: number,
  labels: Record<string, string | number>
) {
  // In reality this would be a Prometheus/DataDog client
  console.log(
    JSON.stringify({
      type: "metric",
      name,
      value,
      labels,
      timestamp: new Date().toISOString(),
    })
  );
}

With this approach you get a stream of tool_latency_ms events, and via the tool and status labels you can compute p95 only for successful calls or, conversely, see how long requests that end in error live.

Counting error‑rate

A similar way can log a separate error metric:

// inside the same tool handler
logMetric("tool_error_total", 1, {
  tool: "recommend_gifts",
  error_type: "external_api_timeout",
});

And for successful requests:

logMetric("tool_success_total", 1, {
  tool: "recommend_gifts",
});

Then in the metrics system you build error_rate = tool_error_total / (tool_error_total + tool_success_total) for a period. This is aggregated on the metrics system side; in the application it’s important to reliably emit events.

If you want absolute minimalism, you can even get by with logs without a separate error_total. Then error‑rate is calculated based on the status field.

Checkout/ACP metrics

For a Next.js checkout endpoint, the logic is the same: wrap the handler with a timer and count statuses.

// app/api/checkout/route.ts
import { NextRequest, NextResponse } from "next/server";
import { logMetric } from "@/observability/metrics";

export async function POST(req: NextRequest) {
  const startedAt = performance.now();

  try {
    const body = await req.json();
    const result = await createCheckoutSession(body); // call to ACP/Stripe
    const duration = performance.now() - startedAt;

    logMetric("checkout_latency_ms", duration, { status: "success" });
    logMetric("checkout_total", 1, { status: "success" });

    return NextResponse.json(result, { status: 200 });
  } catch (error) {
    const duration = performance.now() - startedAt;

    logMetric("checkout_latency_ms", duration, { status: "error" });
    logMetric("checkout_total", 1, { status: "error" });

    return NextResponse.json(
      { error: "Checkout failed" },
      { status: 500 }
    );
  }
}

Now you can look at p95 for checkout_latency_ms and error‑rate via checkout_total. Based on these SLI it’s easy to set an SLO like “p95 < 3 seconds, error‑rate < 2%.”

Webhook metrics

And, of course, webhooks. We’ve already discussed which metrics matter (see section 2), now let’s see exactly how to capture them in code. There, not only time but also the fact of successful processing is important: otherwise a user paid, but the order never moved to “paid.”

// app/api/webhooks/payment/route.ts
import { NextRequest, NextResponse } from "next/server";
import { logMetric } from "@/observability/metrics";

export async function POST(req: NextRequest) {
  const startedAt = performance.now();

  try {
    const payload = await req.text(); // raw payload
    const sig = req.headers.get("stripe-signature") || "";
    const event = verifyStripeSignature(payload, sig); // validation

    await handlePaymentEvent(event); // update the order

    const duration = performance.now() - startedAt;

    logMetric("webhook_latency_ms", duration, {
      type: event.type,
      status: "success",
    });

    logMetric("webhook_total", 1, {
      type: event.type,
      status: "success",
    });

    return new NextResponse("ok", { status: 200 });
  } catch (error) {
    const duration = performance.now() - startedAt;

    logMetric("webhook_latency_ms", duration, {
      type: "unknown",
      status: "error",
    });

    logMetric("webhook_total", 1, {
      type: "unknown",
      status: "error",
    });

    return new NextResponse("error", { status: 500 });
  }
}

Based on these events you can define SLI:

• webhook success rate = success / (success + error);
• p95 of webhook_latency_ms for payment_succeeded.

And then set SLO for them, for example “99% of webhooks are processed successfully over 7 days, p95 < 500 ms.”

4. Percentile statistics: a bit deeper

We’ve relied on p50/p95/p99 several times; now let’s look a bit more formally at what they mean. In real life, your metrics system will compute percentiles, but it’s useful to understand what’s going on there.

Percentile pX is a value below which X percent of measurements lie. If you sort the latency array in ascending order, p95 will be somewhere in the “tail,” closer to the maximum values. In code (for example, on Node, if you suddenly want to locally compute p95 for a set of values) it might look roughly like this:

// simple percentile calculation function
export function percentile(values: number[], p: number): number {
  if (values.length === 0) return 0;
  const sorted = [...values].sort((a, b) => a - b);
  const index = Math.ceil((p / 100) * sorted.length) - 1;
  return sorted[Math.max(0, Math.min(index, sorted.length - 1))];
}

You can use such a function in tests or in a simple script that plays with data and shows how much p95 differs from the average. In prod, Prometheus, Datadog, New Relic, and other grown‑ups will take care of this functionality.

5. SLI, SLO, and SLA: in plain English

Three acronyms that look scary but are actually very simple.

SLI — Service Level Indicator

SLI is a concrete measurable metric, a formula. For example:

• p95(latency) of the recommend_gifts tool over the last 24 hours;
• error‑rate of MCP tools over the last 7 days;
• widget → successful payment conversion for a week.

SLI is not a goal and not a promise, it’s just a “thermometer.”

SLO — Service Level Objective

SLO is a target for an SLI, essentially a condition for the service being “healthy.” For example:

• “p95(latency of recommend_gifts) < 2 seconds over a 7‑day window”;
• “error‑rate of all MCP tools < 1% over a 30‑day window”;
• “checkout API availability ≥ 99.5% over a quarter”;
• “conversion from widget to successful payment ≥ 15% per month.”

A good practice is to first measure the current level and set an SLO slightly stricter than what you already consistently meet; otherwise it becomes either “mission impossible” or “it’s already fine.”

SLA — Service Level Agreement

SLA is no longer an internal target but an external agreement with users or partners. An SLA defines commitments (for example, “99.9% uptime”) and consequences of violations (compensation, penalties, subscription extension, etc.). SLO is often stricter than SLA to leave room for an “error budget.”

Within the training GiftGenius you probably don’t need an SLA, but it’s important to understand the ladder:

SLI → SLO → SLA
metric → objective → external promise

Error budget

Error budget is, simply put, the allowable amount of “imperfection.” If your SLO is “availability 99.9% over 30 days,” then 0.1% is the error budget that can be “spent” on:

• planned releases with downtime;
• experiments;
• unforeseen incidents.

When the budget is “burned” (for example, availability ends up at 99.5% for the month), it’s time to slow down new features and fix stability.

6. Special metrics for GiftGenius: commerce + webhooks

Let’s put it all together and look at GiftGenius as a whole product.

Funnel and conversion

Imagine a simple user path:

flowchart TD
  A[Opened the app with the GiftGenius widget] --> B[Received recommendations]
  B --> C[Selected a gift]
  C --> D[Clicked 'proceed to payment']
  D --> E[Successful payment]

For each step we can log an event:

logMetric("funnel_step_total", 1, {
  step: "widget_view",
});

logMetric("funnel_step_total", 1, {
  step: "gift_selected",
});

logMetric("funnel_step_total", 1, {
  step: "checkout_started",
});

logMetric("funnel_step_total", 1, {
  step: "checkout_paid",
});

Then, knowing the counts at widget_view and checkout_paid, we can compute conversion: paid / view * 100%. And set an SLO: “conversion ≥ 10%.” If conversion suddenly drops to 3% while the technical SLI (latency/error‑rate) look fine, the issue may be UX, model instructions, or the quality of the product feed, not the infrastructure.

Webhook metrics as part of conversion

We already discussed the technical metrics for webhooks and how to implement them. Now it’s important to connect them to conversion: for commerce scenarios webhooks are critical. A user sees “payment succeeded,” but the final order status depends on how quickly your backend accepts and processes payment_succeeded.

SLI for webhooks:

• webhook success rate;
• webhook latency p95;
• rough availability webhook_endpoint_availability.

If webhook success rate drops, you’re literally losing orders.

7. Simple alerts on top of SLO

A full alerting system is a topic closer to operations modules, but we can already outline the basics.

The idea is to send alerts not for every single error but specifically for SLO violations. This is often called symptom‑based alerts — alerts tied to symptoms of service degradation: you’re interested not in “an error occurred,” but in “the service has started systematically performing worse than promised.”

Typical examples:

• error‑rate of MCP tools over the last 5 minutes > 2% when the SLO is < 1%;
• p95(latency of recommend_gifts) over 10 minutes > 3 seconds when the SLO is 2 seconds;
• no successful checkout_paid in the last 15 minutes — suspected integration outage;
• webhook success rate over 10 minutes < 95%.

The alert text should be human, not “Alert: metric 123 > 456.” An example “Slack notification” for GiftGenius:

[ALERT][GiftGenius] High error rate on MCP tools:
error_rate = 3.2% (>1% SLO) for last 5 minutes.

Impact: part of users cannot receive gift recommendations.
Actions: check MCP logs and external gift API health.

Such messages are tied to SLO and help the on‑call developer quickly grasp the magnitude of the issue.

8. Mini‑practice: drafting SLOs for GiftGenius

Let’s try to formulate a set of SLO for our app. You can do this even on paper, without a full metrics system.

A “starter” example:

1. For the main MCP tool recommend_gifts:
   • SLI: p95(latency) of successful calls over the last 7 days.
   • SLO: p95(latency) < 2 seconds.
2. For MCP tool errors:
   • SLI: error‑rate = errors / (errors + successes) over 30 days.
   • SLO: error‑rate < 1%.
3. For the checkout API:
   • SLI: availability = the share of HTTP 2xx out of all requests.
   • SLO: availability ≥ 99.5% per month.
4. For payment webhooks:
   • SLI: webhook success rate.
   • SLO: at least 99% successful webhook processing over 7 days; webhook processing p95 < 500 ms.
5. For the business outcome:
   • SLI: widget → paid order conversion over a month.
   • SLO: conversion ≥ 10–15% (depending on niche).

Even such a small set already gives you a “frame” for decision‑making: if you roll out a new model version, change the prompt or the recommendation algorithm and see that p95 and error‑rate stayed within the SLO, and conversion increased — the experiment likely succeeded.

If instead of your own checkout endpoint you use Instant Checkout, then “checkout API” effectively means Agentic Checkout calls (/checkout_sessions create/update/complete), and “payment webhooks” are your order events (order.created, order.updated) toward OpenAI. The metrics and SLO wording remain the same; only the specific protocol changes.

9. Common mistakes when working with metrics and SLO

Mistake #1: measuring only the average response time.
The average is convenient but dangerous. It easily masks the tail of slow requests. In a ChatGPT App this is critical: even if most users get everything quickly, regular 5–10‑second lags for a small slice of users will feel like “the app is always slow.” So for latency always use at least p50 and p95, not just the mean.

Mistake #2: confusing SLI, SLO, and SLA in your head and in docs.
Sometimes developers write “our SLA is p95 < 2 seconds,” while it’s not recorded anywhere for external users and isn’t tied to any obligations. In reality, that’s an SLO. An SLA is a contract with customers that includes consequences of violations. If you mix it all up, it’s unclear what you actually promise to whom.

Mistake #3: no link between alerts and SLO.
A classic trap is to set alerts “on every error,” on every timeout, on any metric deviation by 0.01. As a result, the on‑call engineer lives in a notification hell and stops looking at them. It’s much more useful to build alerts on SLO violations: if error‑rate grows significantly beyond the target level or p95 “falls out” of bounds, that’s when it makes sense to wake people up.

Mistake #4: ignoring webhook and other asynchronous metrics.
Called out separately, but a problem we already know — ignoring webhook and asynchronous metrics. Many teams monitor only HTTP responses of the main APIs but ignore webhooks and background processing. In commerce scenarios, that’s where the nastiest bugs hide: the payment went through, the webhook didn’t get processed, the order “stalled.” Without success rate and latency metrics for webhooks, you can think “everything works” for a long time — until accounting and your database numbers start diverging badly.

Mistake #5: unrealistic SLO or none at all.
Sometimes SLO are stated as “100% uptime” or “no errors at all.” In practice this is unachievable and demotivates the team. The other extreme is to not state SLO at all and live in “seems fine” mode. The golden mean is to start by measuring current SLI, set a slightly stricter but attainable goal, and gradually tighten it as the system matures.

Mistake #6: metrics just for show, without product linkage.
This happens too: the metrics system is full of charts, p95, p99, dozens of dashboards, but no one can answer a simple question: “If this line jumps — what suffers? The user? The money? The reputation?” Especially in LLM products, it’s important to connect technical indicators (latency, error‑rate) to business metrics (conversion, successful payments). Otherwise observability turns into a pretty but useless TV.