LLM Agents in Production: Tools, Guardrails and Evaluation

An LLM agent is a model that can decide to call tools — search a database, create a ticket, send an email, update a record — observe the result, and continue until a task is done. That ability to act is what makes agents useful, and it is also what makes them risky. A chatbot that gives a wrong answer is embarrassing. An agent that takes a wrong action can cost money, leak data or corrupt records.

Running AI agents in production is mostly ordinary software engineering applied carefully: narrow interfaces, permissions, tests, logs and budgets. Here is how we approach it.

Do you need an agent at all?

Many problems described as agentic are better solved with a fixed workflow: a sequence of steps where the model fills in specific parts, such as classifying a request or extracting fields. Workflows are easier to test, cheaper to run and more predictable. Reach for an open-ended agent only when the path genuinely varies from case to case and cannot be enumerated in advance. Often the right design is a workflow with one or two agentic steps inside it.

Design tools like a public API

Tools are the agent's interface to your systems, and their design matters more than the prompt. Good tools are narrow, well named, strictly typed and safe to call twice:

• Prefer specific tools over general ones. `refund_order(order_id, amount)` is safer than `run_sql(query)`.
• Validate every argument on the server, never trusting the model's output. Reject and explain, so the agent can correct itself.
• Return compact, structured results. Large raw payloads waste context and confuse the model.
• Make actions idempotent where possible, so a retry does not create a duplicate payment or ticket.
• Separate read and write tools, and put the write tools behind stronger checks.

Guardrails and permissions

An agent should never have more access than the user it is acting for. Pass the user's identity through to every tool call and enforce permissions in the tool, not in the prompt. Prompts can be overridden — including by text the agent reads from a web page, email or document. That is prompt injection, and the only reliable defence is limiting what a manipulated agent could do.

const tools = {
  lookupOrder: readTool(async ({ orderId }, ctx) =>
    orders.findForUser(orderId, ctx.user)),

  issueRefund: writeTool(
    async ({ orderId, amount }, ctx) => {
      const order = await orders.findForUser(orderId, ctx.user);
      if (amount > order.total) throw new ToolError("Amount exceeds order total");
      return refunds.create({ orderId, amount, requestedBy: ctx.user.id });
    },
    { requiresApproval: (args) => args.amount > 100 },
  ),
};

Other guardrails worth building in: a maximum number of steps per task, a spending limit per run, human approval above a risk threshold, allow-lists for external domains, and output checks before anything is sent to a customer.

Evaluating agents

Agents are harder to evaluate than single prompts because there are many paths to an outcome. We evaluate at three levels:

1. Tool level — given a situation, does the agent choose the right tool with valid arguments?
2. Task level — across a set of realistic scenarios, does it reach the correct end state? Check the state of the system, not just the final message.
3. Safety level — adversarial cases: injected instructions in retrieved content, requests outside the user's permissions, ambiguous requests that should trigger a clarifying question.

Run scenarios against a sandboxed copy of your systems, several times each, because agent behaviour varies between runs. Track success rate, steps taken and cost per task, and re-run the suite whenever prompts, tools or models change.

Observability

When an agent goes wrong in production, you need to see exactly what it saw and did. Trace every run end to end: the input, each model call with its prompt and response, each tool call with arguments and results, timings and token counts. Link traces to the user and the business object involved so support teams can answer “why did this happen?” without engineering help. Sample traces regularly for review, and turn failures into new evaluation cases.

Cost control

Agents can loop, and loops are expensive. Set hard limits on steps and tokens per task. Use a smaller, cheaper model for routing and simple steps, and a stronger one only where reasoning is needed. Cache stable context such as system prompts and tool definitions where your provider supports it. Trim tool outputs to the fields the agent needs. And measure cost per successful task, not per call — a cheaper model that fails more often may cost more overall.

Memory and state

Agents that work across long tasks or multiple sessions need somewhere to keep state, and that is another place where ordinary engineering beats cleverness. Store task progress, decisions and intermediate results in your own database with a clear schema, rather than relying on an ever-growing conversation history. It makes runs resumable after a failure, keeps the context window small, and lets you inspect exactly what the agent believed at each step.

Be deliberate about long-term memory. Remembering a user's preferences can be helpful; silently accumulating personal data in a memory store is a privacy problem. Decide what is stored, for how long, and how a user can see or delete it, exactly as you would for any other feature.

Treat external content the same way. Anything an agent reads from a web page, an inbound email or an uploaded file should be stored and passed along as data, clearly separated from instructions, and never promoted into memory without a check. Otherwise a single malicious document can influence every future run.

Rolling agents out

Start with read-only agents that research and draft, with a person taking the action. Add write actions one at a time, with approval steps, and remove approvals only when the evidence supports it. This staged approach is the same one we recommend for any custom AI development project, and it works especially well for agents.

Building agents with Amygdal

We design and build agentic systems, tool integrations and the evaluation and observability around them for clients in the UK, Germany, Switzerland and Qatar. Many agents also depend on good retrieval, covered in our RAG development guide. See our custom AI development service or contact us to talk through your use case.