Production and security: permissions, sandbox, secrets, monitoring

1. Why an agent needs a “production mindset”

When you write a regular backend, the very idea of “going to prod” automatically enables paranoia mode: authorisation, logging, error handling, limits, secrets in .env rather than in code.

With an agent, you need the same mode — only stricter. The reason is simple: a regular backend does exactly what you wrote, while an agent does what the model decides on its own within the tools and instructions you gave it. The illusion of control is stronger here than in classic code: it feels like the prompt describes everything, but in reality you control only the environment and allowed actions, not all of the model’s thoughts.

So in this lecture we will gradually surround our agent with “defence layers”:

• first limit exactly what it can do (tool permissions and agent separation),
• then isolate the execution environment (sandbox and limits),
• tidy up secrets and PII,
• and finally enable observability: logs, metrics, and basic tracing.

To keep it concrete, we’ll continue with our GiftGenius story: an agent that helps pick a gift and dips a little into the commerce world (through order and checkout, but without ACP details yet — that’ll come later).

2. Permissions: an agent doesn’t need “every button in the world”

Least privilege principle

Rule one: an agent doesn’t need to do everything. The more tools it has, the higher the chance it will call the “wrong” function at the “wrong” time. Instead of one monstrous manageEverything() that reads and writes anything, we design small, precise functions split at least into reads and writes.

This is especially clear for GiftGenius: reading the gift list and user preferences is one thing; creating or confirming an order (that’s money) is another. So we usually build:

• a set of safe read‑only tools (gift search, view details),
• separate write tools (create order draft, cancel order),
• and, if needed, an extra tier for particularly dangerous operations (payment confirmation, bulk changes).

Different agents for different jobs

Another powerful technique is to split agents by responsibility area. One agent — “gift selection,” another — “order management.” Then even if the model in the gift agent goes a bit off the rails, it physically can’t call a payment tool because it simply isn’t in its configuration.

Let’s imagine a minimalist type for agent and tool configuration:

// Simplified types to explain the ideas
type ToolName = 'suggest_gifts' | 'get_gift_details' |
  'create_order_draft' | 'confirm_order';

type AgentConfig = {
  id: string;
  allowedTools: ToolName[];
  maxSteps: number;
};

Now let’s define two GiftGenius agents:

export const giftPlannerAgent: AgentConfig = {
  id: 'gift-planner',
  allowedTools: ['suggest_gifts', 'get_gift_details'],
  maxSteps: 6,
};

export const orderAgent: AgentConfig = {
  id: 'order-manager',
  allowedTools: ['create_order_draft', 'confirm_order'],
  maxSteps: 4,
};

Yes, this is an abstraction, but the gist is simple: even if the code contains all four tools, a specific agent receives only the necessary subset.

Binding permissions to the user and roles

Remember we have two different entities:

• the user and their permissions (whether this user_id can buy, cancel, see history at all),
• the agent and its allowed tools.

Ideally each tool call should pass both checks: “is the agent allowed?” and “is the user also allowed?”

Roughly:

type UserRole = 'guest' | 'customer' | 'admin';

function canUserCallTool(role: UserRole, tool: ToolName): boolean {
  if (tool === 'confirm_order') {
    return role === 'customer' || role === 'admin';
  }
  if (tool === 'create_order_draft') {
    return role !== 'guest';
  }
  return true; // allow reads to everyone
}

On the MCP/backend side, when handling a tool call we can double‑check:

function assertToolAllowed(
  agent: AgentConfig,
  userRole: UserRole,
  tool: ToolName,
) {
  if (!agent.allowedTools.includes(tool)) {
    throw new Error(`Tool ${tool} is not allowed for agent ${agent.id}`);
  }
  if (!canUserCallTool(userRole, tool)) {
    throw new Error(`A user with role ${userRole} cannot call ${tool}`);
  }
}

As a result, even if the model suddenly decides to call confirm_order from the wrong agent or on behalf of a guest, the call will hit this check and turn into a controlled error rather than an unplanned charge.

Different configurations per environment

In dev and staging environments you often want to give the agent more freedom: test tools, fake payment services, experimental features. In production, on the contrary, the configuration is as strict as possible: some tools are disabled, endpoints are production‑only, tokens are real‑only.

Simplest scheme:

type Env = 'dev' | 'staging' | 'production';

const env = (process.env.APP_ENV as Env) ?? 'dev';

const orderAgentByEnv: Record<Env, AgentConfig> = {
  dev: {
    id: 'order-manager-dev',
    allowedTools: ['create_order_draft', 'confirm_order'],
    maxSteps: 8,
  },
  staging: {
    id: 'order-manager-staging',
    allowedTools: ['create_order_draft', 'confirm_order'],
    maxSteps: 6,
  },
  production: {
    id: 'order-manager-prod',
    allowedTools: ['create_order_draft'], // confirm only via a separate path
    maxSteps: 4,
  },
};

export const currentOrderAgent = orderAgentByEnv[env];

In prod, confirm_order can be moved to a separate “dangerous” agent that you invoke only after an explicit “Confirm order” click in the widget and additional checks.

3. Sandbox: an agent doesn’t need root access to your universe

Isolation levels

After we set permissions for agents and users, we move to the next level of protection — sandboxing and execution environment isolation.

You can roughly split a sandbox for an agent and its tools into several levels:

1. Tool code level. We restrict access to the file system, network, and process resources: don’t allow arbitrary writes, random domains, infinite CPU spinning, or eating gigabytes of memory.
2. Agents SDK level. We set limits on run cycle steps, number of tool calls, and context size (token limit). The model can’t “think” forever and spawn endless tool calls — at some point the run ends with “step limit” or “time limit.”

All this forms a classic “defensive architecture,” which is convenient to visualise with a diagram.

graph TD
    A[Prompt / system instructions] --> B[Tools JSON Schema]
    B --> C[Agent and user permissions]
    C --> D[Infrastructure sandbox]
    D --> E[External services / DB]

    subgraph Agent
      A
      B
      C
    end

    subgraph Infrastructure
      D
    end

The prompt is the weakest defence; the real strength begins where you physically restrict what your code can do and which APIs are available.

Limits on the run cycle: steps, time, tool calls

Part of the sandbox can be expressed directly in the agent’s configuration: maximum number of steps, total execution time, limit of tool calls. This protects not only against runaway loops but also controls costs.

Example of an abstract run options configuration:

type RunLimits = {
  maxSteps: number;
  maxToolCalls: number;
  timeoutMs: number;
};

const defaultLimits: RunLimits = {
  maxSteps: 8,
  maxToolCalls: 10,
  timeoutMs: 30_000,
};

You then pass such limits into the wrapper that runs the agent. If the model decides to call a tool for the 11th time — you interrupt the agent and honestly tell the user that the task is too complex, rather than letting the agent burn the budget unchecked.

Code and network isolation

At the container/process level, common practices include:

MCP server code and/or the agent service runs in a container with a read‑only file system (except for a specifically designated working directory) and limited resources (CPU, RAM). The network is configured via an allow list: it can call only the necessary external services (your commerce backend, payments, a couple of external APIs), not the arbitrary internet.

This is especially critical for agent scenarios: the model might try to reach some “random” API or read unexpected files, and it’s good if even with such attempts it physically lacks privileges to reach extra resources.

In code this usually doesn’t look like a “magic TypeScript line,” but like settings of your orchestrator (Docker Compose, Kubernetes, Vercel, Fly.io, etc.). Still, it’s helpful to think about this already at the design stage:

• a tool that runs third‑party code (e.g., generating a report with shell commands) must work in a separate, strongly isolated environment;
• tools must not be able to read other people’s files, secrets, configs;
• network access is better explicitly restricted by domains or IPs.

4. Secrets and confidential data: what an agent doesn’t need to know

Where secrets should and shouldn’t live

Basic rule: no secrets — API keys, passwords, access tokens — should end up in the model’s prompt, in the widget, in logs, or in the repository. They should live:

• in environment variables (process.env.SOMETHING),
• in a secret manager (AWS Secrets Manager, GCP Secret Manager, Vault, etc.),
• in separate encrypted stores with strictly controlled access.

In our GiftGenius, for example, there’s a key to the store’s commerce API. We need the agent to create an order draft via an MCP tool, but the model must not see the key itself.

// mcp/tools/createOrderDraft.ts
const COMMERCE_API_KEY = process.env.COMMERCE_API_KEY!;

export async function createOrderDraft(args: {
  userId: string;
  giftId: string;
  quantity: number;
}) {
  // The model will never see COMMERCE_API_KEY — it stays here on the server
  const res = await fetch(`${process.env.COMMERCE_API_URL}/orders/draft`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${COMMERCE_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(args),
  });

  if (!res.ok) {
    throw new Error(`Commerce API returned ${res.status}`);
  }

  return res.json(); // Return a safe object back to the agent
}

Important: in the tool’s response you must not “smuggle” keys or other sensitive details. The agent only needs to know draftOrderId, the line items list and, perhaps, the status.

PII and minimising data in context

Besides secrets, there’s the PII category (personal data): names, phone numbers, shipping addresses, emails, etc. The agent often doesn’t need all this raw text. A structured profile is enough: “likes board games,” “age 30–35,” “approximate budget $50–$70.”

Instead of dumping the user’s entire order history into the prompt, you can create a tool get_user_profile_summary that returns an aggregated and de‑identified profile.

type ProfileSummary = {
  ageRange: '18-25' | '26-35' | '36-50' | '50+';
  interests: string[];
  preferredBudget: { min: number; max: number };
};

export async function getUserProfileSummary(userId: string): Promise<ProfileSummary> {
  // You query the DB here, but expose only aggregated information
  return {
    ageRange: '26-35',
    interests: ['board games', 'gadgets'],
    preferredBudget: { min: 30, max: 80 },
  };
}

The model sees exactly as much as needed to pick a gift — and no more.

Scrubbing logs

Logs are the natural place where secrets and PII pop up by accident. Especially if you write a “convenient” logger like console.log(...) and print “everything.”

A good approach is to have a central logger that walks the payload and masks sensitive fields before printing.

type LogPayload = Record<string, unknown>;

const SENSITIVE_KEYS = ['email', 'phone', 'cardNumber', 'token'];

function scrub(payload: LogPayload): LogPayload {
  const result: LogPayload = {};
  for (const [key, value] of Object.entries(payload)) {
    if (SENSITIVE_KEYS.includes(key)) {
      result[key] = '***redacted***';
    } else {
      result[key] = value;
    }
  }
  return result;
}

export function logEvent(event: string, payload: LogPayload) {
  const safe = scrub(payload);
  console.log(JSON.stringify({ event, ...safe }));
}

Now, instead of someday catching a production incident like “we’ve been logging customers’ phones and tokens for half a year,” you design the system so it’s simply impossible. This is not just tidiness, but also future compliance needs (GDPR and local laws): the less PII in logs, the easier life is for the product.

5. Monitoring and observability of the agent

What exactly you need to see

We limited what the agent can do, which data it sees, and what goes to logs. The next question is — how do we know that in this whole zoo the agent behaves in prod as intended?

A simple “service up/down” monitor is almost useless for an agent. We need not only to know the process is alive, but also understand its behaviour: which steps it takes, which tools it calls, where it fails, where it loops.

Minimal data set for each run:

• agent_run_id — a unique run identifier;
• anonymous user_id or a session ID;
• agent name and environment;
• the list of tools called: name, count, total time;
• workflow steps and at which step we stopped;
• final status: success, partial_success, failed, canceled, timeout, limits_exceeded.

You can model it as a structure:

type RunStatus =
  | 'success'
  | 'partial_success'
  | 'failed'
  | 'canceled'
  | 'timeout'
  | 'limits_exceeded';

type ToolCallLog = {
  name: ToolName;
  durationMs: number;
  success: boolean;
};

type AgentRunLog = {
  runId: string;
  agentId: string;
  userId: string;
  env: Env;
  startedAt: string;
  finishedAt: string;
  status: RunStatus;
  toolCalls: ToolCallLog[];
  errorMessage?: string;
};

Example “wrapper” around an agent run

Suppose you have a runAgent function that encapsulates the real Agents SDK call. Let’s wrap it with monitoring:

async function runAgentWithLogging(
  agent: AgentConfig,
  input: string,
  userId: string,
): Promise<string> {
  const runId = crypto.randomUUID();
  const startedAt = new Date();

  const toolCalls: ToolCallLog[] = [];

  try {
    const result = await runAgent(agent, input, {
      userId,
      limits: defaultLimits,
      onToolCall: (name, durationMs, success) => {
        toolCalls.push({ name, durationMs, success });
      },
    });

    const finishedAt = new Date();

    const log: AgentRunLog = {
      runId,
      agentId: agent.id,
      userId,
      env,
      startedAt: startedAt.toISOString(),
      finishedAt: finishedAt.toISOString(),
      status: 'success',
      toolCalls,
    };

    logEvent('agent_run', log);
    return result;
  } catch (err) {
    const finishedAt = new Date();
    const log: AgentRunLog = {
      runId,
      agentId: agent.id,
      userId,
      env,
      startedAt: startedAt.toISOString(),
      finishedAt: finishedAt.toISOString(),
      status: 'failed',
      toolCalls,
      errorMessage: (err as Error).message,
    };
    logEvent('agent_run', log);
    throw err;
  }
}

Here, runAgent is a black box that can be implemented via a real Agents SDK; we’re showing how to add observability without relying on a specific API.

Logs vs metrics vs tracing

It’s useful to distinguish three observability levels:

Metrics answer “is everything OK in general?” (e.g., last hour’s error rate). Logs and tracing help answer “why is it bad right here?” and reproduce a specific problematic run.

You can implement a simple start for metrics on top of logs: a periodic job aggregates agent_run events and computes p95 durations, error counts, etc.

6. What it looks like end‑to‑end in GiftGenius

So it doesn’t feel like a pile of abstractions, let’s assemble the picture for our training app.

The gift-planner agent in the production environment has only safe tools: gift suggestions and fetching details. It can’t see payments or order management. Its system instructions tell it not to promise the user “I’ll pay for you,” but at most to prepare recommendations and, possibly, a draft list of gifts.

The order-manager agent exists separately and only works with orders. In production it can create only an order draft (create_order_draft), while confirming an order (confirm_order) is either performed by a human via an explicit UI trigger in the widget, or available only in dev/staging. Its tools use secrets (shop API keys) exclusively on the backend side, and proxy only the necessary fields in the response.

Both agents are launched through the runAgentWithLogging wrapper, which applies limits and writes logs with agent_run_id, userId, environment, and the tool list. There are no emails or phone numbers in the logs; these fields are scrubbed in advance. The user profile is used in de‑identified form: age range, interests, budget, but not the full text of the purchase history.

The infrastructure hosting the MCP server and the agent service is isolated: containers with a read‑only file system (except for /tmp or a specifically designated directory), CPU/RAM limits, and an allow‑list network. If the agent suddenly tries to call “something off‑limits,” it simply won’t be able to reach it physically.

If at some point you see a spike in the metric “share of runs with status limits_exceeded” or “average number of tool calls > 10,” you understand that either the prompt became overly verbose, or one of the tools is buggy and makes the agent restart steps.

That’s already the behaviour of a mature service, not an experimental “good enough” agent.

7. Common mistakes when taking agents to production

Everything we discussed above is the “proper” picture of a production agent. In practice, the most common pitfalls keep showing up. Let’s collect them into one list: if you avoid at least these mistakes, the launch to prod will go much more smoothly.

Mistake #1: you “gave the agent access to everything.”
A common scenario: you’ve described a bunch of MCP tools (search, modify, delete, payments), and when creating the agent you simply fed it the entire list. As a result, the model can accidentally call deletion or payment where you intended read‑only. Fix by splitting tools by roles and creating several narrower agents, each with its own allowedTools.

Mistake #2: permission checks only in the prompt.
Sometimes developers write in system instructions: “never buy anything without user confirmation” and call it a day. But the prompt is a weak defence, and jailbreaks and simple mistakes still happen. You need real checks on the backend level: “this tool is allowed for the agent” and “this tool is allowed for the user,” otherwise one careless generation can lead to actions no one intended.

Mistake #3: secrets in prompts and logs.
It’s tempting to “speed up integration” by putting an API key in the system prompt or passing it in tool arguments so the agent can call an external API on its own. As a result, the key ends up in model logs and potentially in third‑party systems. This is a direct path to leaks and a ban in the Store. Secrets must live only server‑side, in environment variables or a secret manager, and never enter the model’s context.

Mistake #4: “raw” logs without scrubbing.
During debugging it’s convenient to write console.log(...) and forget about it. A couple of months later it turns out the logs contain user addresses, phone numbers, and order numbers with PII. Especially unpleasant in the world of GDPR and other regulations. It’s better to set up a central logger and automatic masking of sensitive fields right away, even if it feels like “we only log on dev.”

Mistake #5: no limits on agent behaviour.
Without limits on steps, time, and number of tool calls, an agent can loop: repeatedly call the same tool, try to fix the same error infinitely, spend loads of tokens, and burden external APIs. In the best case you get enormous model bills; in the worst, you take down the backend and anger all users. Run‑cycle limits and sane defaults for timeouts are a mandatory part of configuration.

Mistake #6: mixing read and write operations in one tool.
Sometimes people create “convenient” methods like getOrCreateOrder that create a new order if none exists. For a classic backend this is an acceptable pattern, but in the agent world it can lead to unexpected side effects: the model wanted only to learn the state, but the tool created something. It’s much safer to split get_order_details and create_order_draft, so even with repeated calls the consequences are more controlled.

Mistake #7: ignoring observability.
Many start with “we’ll add logs and metrics later, for now it just needs to work.” Agents without monitoring are a black box: you don’t know which tools they call, how many steps they take, or where they fail. Any user complaint turns into an investigation in a dark room. It’s much easier to lay out a log structure from the start (agent_run_id, tools, status) and basic metrics than to try to bolt it on later over chaotic code.