UX analytics for the workflow: measuring step effectiveness

1. Why measure the workflow at all

In short: without analytics you live in “I think” mode, not in “I know” mode.

In previous lectures of this module we already split the scenario into steps, assigned roles to GPT, the widget, and MCP, discussed tool gating and storing state between steps. Now let’s look at the same construction through the lens of analytics: does all this work as intended, and where users actually get stuck.

On the regular web everyone is used to funnels: how many people landed on a page, how many added an item to the cart, how many reached payment. In the ChatGPT App it’s the same, except instead of pages you have workflow steps, and instead of “clicking the Buy button” you have a combination of a user message, a tool call, and widget interaction.

When you build a complex scenario without metrics, you don’t see:

• at which step people most often drop off;
• where they linger and read for a minute (or they just went for tea and never came back);
• which step brings no value and only annoys;
• how changes in prompts or tool gating affect behavior.

The goal of step-level analytics is simple: learn to increase the share of completed scenarios, reduce time to result, and decrease errors and support tickets.

From this moment a workflow is not only an architectural object and not only a UX quest. It’s also a measurable thing with numbers.

2. The scenario funnel in the ChatGPT App

In classic web the funnel is linear: Landing → Product → Cart → Checkout. In the ChatGPT App the picture is a bit livelier: the user can “skip” a step with words, the model can sometimes skip a step, and the widget and the text dialogue can get out of sync.

Nevertheless, the basic idea is the same: we have a sequence of steps, and at each one some users proceed and some do not.

Let’s take our GiftGenius:

1. collect_recipient — ChatGPT and the widget collect basic data about the recipient (gender, age, relationship, interests).
2. collect_budget — clarify budget and currency.
3. suggest_ideas — the MCP/agent selects ideas and returns gift cards in the widget.
4. review_selection — the user likes/hides ideas and picks 1–2 favorites.
5. checkout — a commerce intent is created and the order is placed.

You can draw this as a funnel like this:

flowchart TD
    A[Workflow start] --> B["1\. Recipient"]
    B --> C["2\. Budget"]
    C --> D["3\. Gift ideas"]
    D --> E["4\. Gift selection"]
    E --> F["5\. Checkout"]

But it’s important to remember: the user can write in chat “Let’s go straight to payment” or “Show expensive options first,” and the model may decide to skip some steps. Therefore step analytics in the ChatGPT App is not only about UI screens but also about model behavior: which steps are actually taken, in what order, and who initiated the transition — the user, the widget, or GPT.

3. Core step-level metrics

Let’s start with classic product analytics and adapt it a bit to the ChatGPT App.

For each workflow we need at least four basic indicators.

For convenience, let’s summarize them in a table:

We almost always add effort metrics:

• average time per step (where people “get stuck”);
• number of interactions per step (how many messages/clicks it took);
• the share of steps that ended in error or required a retry.

In the context of LLM scenarios even more specific things get added, such as tool-selection accuracy by the model or the share of hallucinated responses on a specific step, but that’s advanced and we’ll get to it in the final modules.

For commerce scenarios, business metrics are layered on top of the steps:

• conversion to payment from workflow start;
• conversion to payment from a specific step (for example, from “suggesting ideas”);
• average order value;
• share of cancellations/returns.

Important: these numbers don’t live in isolation; there’s a causal story between them. A step with a high drop-off isn’t always bad: it may be filtering out unqualified users so only those for whom the scenario is truly useful continue. Analytics is not just “calculating percentages,” but being able to tell a story with data.

To compute all these percentages and funnels at all, we need raw events: who, when, and which step they completed (or did not). In the next section we’ll agree on the format of such events.

4. What analytics events look like

Before writing code, we need to agree on the format of the event we will send from the widget and backend.

An analytics event usually contains:

• who: a user ID or at least a session ID;
• which workflow and its version;
• which step;
• what happened (event type);
• whether it succeeded, how long it took;
• a bit of metadata (locale, device, etc.).

A simplified event schema for a workflow can be described like this:

export type WorkflowEventType =
  | "workflow_started"
  | "workflow_finished"
  | "step_started"
  | "step_completed"
  | "step_failed";

export interface WorkflowAnalyticsEvent {
  eventId: string;              // uuid
  timestamp: string;            // ISO string
  userId?: string;              // if you can de-anonymize
  conversationId?: string;      // ChatGPT conversation ID (if available)
  workflowId: string;           // our internal identifier
  workflowType: "gift_selection";
  workflowVersion: string;      // for example, "1.2.0" or "1.2.0-A"
  stepName?: string;            // collect_budget, suggest_ideas, etc.
  eventType: WorkflowEventType;
  toolName?: string;            // if related to a tool call
  success?: boolean;
  errorCode?: string | null;
  durationMs?: number;
  metadata?: Record<string, unknown>;
}

A few nuances:

First, workflowVersion is very important if you plan to run A/B tests: without it you will never know which variant of the scenario yields better numbers.

Second, conversationId or another correlation ID lets you link events: steps in the widget, tool calls in MCP, and the text dialogue. We’ll talk more about tracing and observability in later modules, but the habit of thinking about end-to-end identifiers from the start is very useful.

Third, you don’t need to shove everything into the event: full message texts, emails, addresses, and other PII are better avoided or strictly anonymized — we’ll talk more about this closer to the end.

5. Instrumentation in the widget (Next.js + Apps SDK)

Now the fun part: how to make our GiftGenius widget quietly report steps when the user goes through the scenario.

Assume that in previous lectures you already built something like:

// components/GiftWizard.tsx
type StepId = "recipient" | "budget" | "ideas" | "review" | "checkout";

export function GiftWizard() {
  const [currentStep, setCurrentStep] = useState<StepId>("recipient");
  const [workflowId] = useState(() => crypto.randomUUID());

  // ... rendering of different steps here
}

Let’s add a small “analytics layer” as a hook.

Hook useWorkflowAnalytics

We’ll make a wrapper that knows about workflowId, workflowVersion and can send an event to our Next.js API route /api/workflow-analytics.

// lib/useWorkflowAnalytics.ts
import { useCallback } from "react";
import type { WorkflowAnalyticsEvent, WorkflowEventType } from "./types";

const WORKFLOW_VERSION = "1.0.0";

export function useWorkflowAnalytics(
  workflowId: string,
  workflowType: WorkflowAnalyticsEvent["workflowType"] = "gift_selection"
) {
  const sendEvent = useCallback(
    async (payload: Omit<WorkflowAnalyticsEvent, "eventId" | "timestamp" | "workflowType" | "workflowVersion" | "workflowId">) => {
      const event: WorkflowAnalyticsEvent = {
        eventId: crypto.randomUUID(),
        timestamp: new Date().toISOString(),
        workflowId,
        workflowType,
        workflowVersion: WORKFLOW_VERSION,
        ...payload,
      };

      // simple POST to the API; in prod you can add a buffer/debounce
      await fetch("/api/workflow-analytics", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(event),
      });
    },
    [workflowId, workflowType]
  );

  const trackStepEvent = useCallback(
    async (stepName: string, eventType: WorkflowEventType, extra?: Partial<WorkflowAnalyticsEvent>) => {
      await sendEvent({ stepName, eventType, ...extra });
    },
    [sendEvent]
  );

  return { sendEvent, trackStepEvent };
}

It’s important that the hook does not depend on a specific UI step. It only knows what stepName and eventType are. Specific components will tell it: “I started the step,” “I finished the step,” and so on.

Sending workflow_started and workflow_finished

In the GiftWizard component you can register the start and end of the scenario on mount and unmount:

// components/GiftWizard.tsx
export function GiftWizard() {
  const [currentStep, setCurrentStep] = useState<StepId>("recipient");
  const [workflowId] = useState(() => crypto.randomUUID());
  const { sendEvent } = useWorkflowAnalytics(workflowId);

  useEffect(() => {
    void sendEvent({ eventType: "workflow_started" });
    return () => {
      void sendEvent({ eventType: "workflow_finished" });
    };
  }, [sendEvent]);

  // ...
}

Obviously, finishing on unmount is a rough approximation: a user can just minimize the chat or go to another conversation. But even such a coarse metric already gives a sense of how many scenarios “got anywhere” at all.

Tracking step events

Now let’s make each step report itself to analytics. First add a simple wrapper:

interface StepProps {
  stepId: StepId;
  onNext: () => void;
  trackStepEvent: (stepName: string, eventType: WorkflowEventType, extra?: Partial<WorkflowAnalyticsEvent>) => Promise<void>;
}

function StepRecipient({ stepId, onNext, trackStepEvent }: StepProps) {
  useEffect(() => {
    void trackStepEvent(stepId, "step_started");
  }, [stepId, trackStepEvent]);

  const handleSubmit = async () => {
    // ... validation, save to widgetState
    await trackStepEvent(stepId, "step_completed");
    onNext();
  };

  return (
    <div>
      {/* recipient form fields */}
      <button onClick={handleSubmit}>Next</button>
    </div>
  );
}

In GiftWizard pass trackStepEvent:

export function GiftWizard() {
  // ...
  const { trackStepEvent } = useWorkflowAnalytics(workflowId);

  const goToNext = () => {
    setCurrentStep((prev) => NEXT_STEP[prev]);
  };

  if (currentStep === "recipient") {
    return (
      <StepRecipient
        stepId="recipient"
        onNext={goToNext}
        trackStepEvent={trackStepEvent}
      />
    );
  }

  // other steps...
}

Similarly, in steps where there are potential errors (for example, a request to an external API in suggest_ideas), you can send "step_failed" with an errorCode on failure, and "step_completed" on successful loading of options.

This gives us:

• a clear list of events: when steps start and finish;
• the ability to calculate step time: the difference between "step_started" and "step_completed";
• visibility into which steps most often end with "step_failed".

6. Instrumentation on the backend / MCP

Client-side analytics is great, but the widget lives in a rather fragile world: the user’s browser, an iframe, sandbox constraints, and everything that comes with it. Therefore, in parallel, you should log events on the server side — in MCP tools or backend APIs of your App.

For example, you have a suggest_gifts tool that does the heavy lifting: it queries a product feed, applies filters, and returns gifts. Inside such a tool you can log both business logic and analytics events.

A hypothetical MCP tool handler in TypeScript might look like this:

// mcp/tools/suggestGifts.ts
import type { SuggestGiftsArgs } from "../schemas";
import { logWorkflowEvent } from "../analytics/log";

export async function handleSuggestGifts(args: SuggestGiftsArgs, context: { workflowId: string; stepName: string }) {
  const startedAt = Date.now();

  try {
    // ... main logic for suggesting ideas

    await logWorkflowEvent({
      workflowId: context.workflowId,
      workflowType: "gift_selection",
      workflowVersion: "1.0.0",
      stepName: context.stepName,
      eventType: "step_completed",
      toolName: "suggest_gifts",
      success: true,
      durationMs: Date.now() - startedAt,
    });

    return {
      content: [{ type: "text", text: "Found 5 gift ideas." }],
      _meta: {
        // raw data for the widget
      },
    };
  } catch (e) {
    await logWorkflowEvent({
      workflowId: context.workflowId,
      workflowType: "gift_selection",
      workflowVersion: "1.0.0",
      stepName: context.stepName,
      eventType: "step_failed",
      toolName: "suggest_gifts",
      success: false,
      errorCode: "SUGGEST_FAILED",
      durationMs: Date.now() - startedAt,
    });
    throw e;
  }
}

And logWorkflowEvent can write to the same table/storage where events from the frontend go, just with a "source" label: "backend".

Why server-side analytics is more reliable

First, a tool call either happened or it didn’t — it’s a hard fact, not a heuristic like “the user apparently clicked the button.”

Second, it’s easier to aggregate data on the server: you can count how many times each tool is called, its average durationMs, and what share of calls end in error.

Third, you see the difference between a UX problem (the user doesn’t reach the step where the tool is called) and a technical problem (they do reach the step, but the tool often fails).

7. How to read the data: finding bottlenecks

Suppose you’ve already implemented sending "workflow_started", "step_started", "step_completed", and "step_failed" events from the widget and MCP, and enough data has accumulated in storage. Let’s assume you’ve already collected some data for GiftGenius and obtained a step summary. The table below shows hypothetical numbers for 1,000 started workflows:

What to look at here:

First, the budget step is an obvious bottleneck. High drop-off (26%) and noticeably higher average time. Maybe you’re asking too much about currencies/taxes, the wording is unclear, or people are simply unsure about their budget. This is a good candidate to simplify the step, split it into two substeps, or change question wording.

Second, review also has heavy drop-off. Perhaps the gift card UI is overloaded, or it’s unclear to the user what it means to “like” a gift. Maybe the model returns too many options and the widget feels like an endless list. Here you should look not only at the numbers but also at screenshots/session recordings (if you have them), or at least go through the scenario manually as a user.

Third, checkout loses 16% — for a commerce scenario that’s a lot of money. But you need to understand where they’re lost: during order creation, due to payment provider errors, or because the user simply changed their mind. This is no longer a pure UX question, but a combination of UX plus business constraints.

It’s important to distinguish UI problems from model problems.

• If users often return to the previous step and change answers, that signals an unclear or poorly worded question.
• If a step finishes quickly but the tool call on it often fails, that’s a backend/MCP problem.
• If a step lasts a long time with no errors and no returns, maybe the user is just reading a long block of text they don’t really need.

8. Experiments and A/B tests of the workflow

Numbers by themselves don’t improve anything. For analytics to be useful, you need to be able to run experiments: change steps and compare whether things got better.

In the context of the ChatGPT App a typical experiment compares two versions of a step or sequence of steps:

• a long wizard with several simple screens versus one complex form;
• different question wordings;
• different step order (for example, ask about the budget earlier or later);
• different tool-gating strategies (fewer tools on the first step, more on the second).

A good habit is to pin the scenario version in workflowVersion and add an experiment identifier there, for example "1.3.0-A" and "1.3.0-B".

The simplest A/B split in the widget

Of course, in production you’ll want stable assignment at the user or session level (via the backend), but for a learning example a random choice of variant is enough.

// lib/useWorkflowVariant.ts
import { useMemo } from "react";

export type WorkflowVariant = "A" | "B";

export function useWorkflowVariant(): WorkflowVariant {
  return useMemo(() => {
    return Math.random() < 0.5 ? "A" : "B";
  }, []);
}

In GiftWizard we determine the variant and pass it to analytics:

export function GiftWizard() {
  const [workflowId] = useState(() => crypto.randomUUID());
  const variant = useWorkflowVariant();
  const { sendEvent, trackStepEvent } = useWorkflowAnalytics(
    workflowId,
    "gift_selection"
  );

  useEffect(() => {
    void sendEvent({
      eventType: "workflow_started",
      metadata: { variant },
    });
  }, [sendEvent, variant]);

  // you can then change the texts/structure of steps depending on variant
}

On the server you can replace the hardcoded "1.0.0" with something like "1.1.0-A" and "1.1.0-B", or simply log metadata.variant and group by it in analytics.

The main point of an A/B test: choose the target metric in advance. For example: “we want to raise the scenario completion rate from 42% to 50%,” or “reduce time on the budget step by 20%.” Without a target metric, any workflow restructuring will feel like moving furniture around because “it looks nicer.”

9. Privacy and data ethics

We already briefly mentioned that it’s better not to pass PII straight into metadata and analytics events. While discussing metrics, it’s easy to get carried away and start logging everything. But remember that you’re working inside ChatGPT, and a user can reasonably expect that their personal messages won’t be shipped to external analytics in raw form.

A few simple rules to follow already now, before the modules on security and the Store:

• First, do not log full texts of user messages. Instead, store message length, the type of response (number, “yes/no,” choice from a list), or anonymized features like “empty/incomplete/edited response.”
• Second, do not log explicitly identifying information (PII) unless you need it for business logic: email, phone numbers, addresses, full names. If you cannot avoid it, store them in another, protected environment and strictly limit access.
• Third, be careful with conversation context. If you store a conversationId, make sure you don’t try to stitch separate conversations into super-profiles in analytics without a solid reason and legal basis.
• Fourth, pay attention to OpenAI policies and Store requirements (we’ll cover this in detail in the modules on publishing and security), which clearly state what data can be taken out of ChatGPT and what cannot. When designing analytics, it’s helpful to build in anonymization and data minimization from the start so you don’t have to rewrite half the system later.

And finally, remember that UX analytics is not total surveillance and definitely not Big Brother–style monitoring. The goal is to improve scenarios and reduce user frustration, not to build a Big Brother dashboard of “who at 2:37 a.m. didn’t make it to checkout.”

10. Common mistakes in workflow UX analytics

Mistake #1: “We’re not in prod yet; metrics later.”
Very often developers start thinking about analytics when the App is already used by real people. As a result, events are introduced after the fact, data is patchy, and comparing old and new versions of the scenario is almost impossible. It’s best to bake in a minimal funnel ("workflow_started", "step_started", "step_completed", "workflow_finished") right away while the code is still relatively simple.

Mistake #2: Logging only successes and ignoring errors.
Sometimes logs only contain "step_completed", but no one writes "step_failed" because “well, it shouldn’t fail.” As a result, you see that few people reach a step, but you can’t tell whether they left on their own or got kicked out by errors. Always log both successful step completion and failures, with at least a coarse errorCode.

Mistake #3: No linkage to the workflow version.
You change texts, step order, introduce tool gating, but the events always show the same workflowVersion: "1.0.0". A month later you look at the graphs and can’t tell what was before changes and what came after. Pinning the scenario version and, if needed, the A/B variant is a must-have element of analytics.

Mistake #4: Overly detailed analytics without a reason.
The opposite extreme is to immediately build the “perfect” event schema with 50 fields, log every pixel click and every retyped character. First, this may violate privacy. Second, such data is hard to analyze and you will drown in noise. It’s better to start with a small set of events and metrics that truly answer concrete product questions, and then evolve the system as needed.

Mistake #5: Inconsistent names for steps and scenarios.
Sometimes the step is called budget in code, collect_budget in analytics, and “the step with the money question” in the report. In a couple of weeks no one remembers what is what. When designing the workflow, it’s helpful to agree on stable step identifiers (stepName) and use them in the UI, logs, and reports.

Mistake #6: Metrics that no one uses.
The saddest story: you carefully collected lots of data, set up event sending from the widget and MCP, but no one opens dashboards or makes decisions. Analytics for the sake of analytics is useless; always ask yourself: “What decision will I make based on this metric?” If there’s no answer, you don’t need that metric yet.