Agent reliability, the whole picture: from demo to production

Most agents that fail in production worked fine in the demo. What they lacked was the engineering around the model: evals, guardrails, tracing, and cost controls. This is the map of how those four pieces fit together.

Pushpendra Singh

· 11 min read

Share:

Every AI agent has a version that works beautifully: the one in the demo. The founder drives it, the inputs are friendly, the tools all respond, and the model happens to be having a good day. Put the same agent in front of real users, real data, and a real bill, and it behaves like a different system. It invents a policy that doesn’t exist. It calls a tool in a loop until someone notices the invoice. It copies a customer’s email address into a prompt. It gets worse after someone “just tweaks the prompt”, and nobody notices.

A better model fixes none of these. They are engineering problems with engineering fixes. We’ve spent the last couple of years building and auditing agentic systems for clients, and the same fixes keep coming up. We call the set agent reliability: the practices that take an agent from “works in the demo” to “works in production”.

This post is the map. We’ve written about individual pieces before, including evals, AI cost, and the Agent Reliability Kit we open-sourced for measuring agents. Each piece depends on the others, so this post shows how they fit around the model.

Why Agents Break Differently Than Software

Traditional software fails deterministically. The same input produces the same bug, the stack trace points at the line, and you can verify the fix. Agents break all three of those assumptions.

The Four Ways Agents Fail That Software Doesn't

Each one breaks a debugging habit your team has spent years building

Non-determinism

The same input can produce different outputs on different runs, so “it worked when I tried it” means very little. You can no longer count on reproducing a bug, which is the first step of every debugging workflow you know.

Unbounded input space

Users type anything. Retrieved documents contain anything, including instructions aimed at your agent instead of your user. No validation schema covers natural language.

Compounding steps

A multi-step agent multiplies its per-step error rates. An agent that’s 95% reliable per step is about 60% reliable across ten steps (0.95 to the tenth power). Nobody would ship a service with 60% availability, yet teams ship ten-step agents without ever measuring per-step reliability.

Silent economic failure

Software that misbehaves usually crashes. An agent that misbehaves often succeeds expensively: the right answer, after twelve tool calls, for four dollars. No alert fires, and the bill keeps growing.

Most teams try to fix these with prompt engineering. Prompts matter, but the model can ignore a prompt. Reliability has to come from code around the model that checks what goes in and what comes out.

The Mental Model: Nothing Touches a Model Unwrapped

Our reference architecture fits in one sentence:

Every model call is wrapped by guardrails, instrumented by traces, metered for cost, and verifiable by evals.

Nothing else is allowed to call a model directly: not a quick internal endpoint, not a background summarizer, not the “temporary” script that ends up in production. There is one wrapped entry point, and every call goes through it.

Application code calls one reliable model client. Inside it, every call passes through input guardrails, a cost meter and router, the model provider, output guardrails, and an eval recorder, while OpenTelemetry tracing observes every step.Application code calls one reliable model client. Inside it, every call passes through input guardrails, a cost meter and router, the model provider, output guardrails, and an eval recorder, while OpenTelemetry tracing observes every step.

In code, the wrapper is smaller than the diagram suggests. The skeleton is about a decorator’s worth of logic:

def complete(prompt, *, tenant, feature, budget_usd=0.05):
prompt = input_guardrails.run(prompt) # redact PII, reject injection
model = router.pick(prompt) # cheap first, escalate on need
check_budget(tenant, projected_cost(prompt, model), budget_usd)
with tracer.start_span("gen_ai.chat.completions") as span:
response = call_with_retries(model, prompt, fallback="backup-model")
span.set_attributes(usage_attributes(response, tenant, feature))
output_guardrails.run(response) # schema, toxicity, grounding
eval_recorder.record(prompt, response, span) # feeds the dataset flywheel
return response

We deliberately don’t make this a framework. Use LangGraph, Pydantic AI, the Claude Agent SDK, or your own loop; the reliability layer wraps the model calls inside whatever orchestration you chose. We don’t run it as a gateway or proxy either. Every check that wraps a call is explicit, in your own code, and shows up in the trace, because hidden behavior is exactly what you can’t debug at 2am.

The wrapper rests on four pillars: evals, guardrails, tracing, and cost.

Pillar 1: Evals

Every conversation about agent quality eventually comes back to one question from leadership: “is the system improving?” Without an eval harness, the honest answer is “we think so”. With one, it’s a number with a trend line.

The harness is simple: a dataset of inputs with expected behavior, a scoring function, and a runner wired into CI. The work is in what fills it. We build three suites per agent:

  • Golden. Known-good inputs with expected outputs. This is the regression suite. It should almost always pass, and a drop blocks the merge.
  • Adversarial. Known-bad inputs: jailbreaks, personal data, prompt injection, malformed data, and requests at the edge of what the agent should do. This suite tells you how the agent holds up under attack.
  • Synthetic. LLM-generated variations of the golden cases, reviewed by a person before they can block a merge. This is how the suite grows faster than your team can write cases by hand.

Two practices separate eval suites that keep working from ones that decay.

Calibrate the judge. You need LLM-as-judge scorers for fuzzy criteria, but an uncalibrated judge is guesswork with extra steps. Every judge should ship with a small set of human-labeled cases, and its agreement rate with those labels should be reported next to the scores. If the judge agrees with people 70% of the time, reviewers should know that before trusting an 85% pass rate.

Feed production back in. The eval dataset should grow from production as well as from what the team can imagine. Grade recent production traces with scorers that don’t need a reference answer, and turn the failures into regression cases, so every real failure becomes a permanent test. When failures pile up, group them by cause. A team that fixes “the agent mishandles refund requests with several order IDs” makes more progress in a week than a team working through twelve separate bug reports.

The runner code and CI wiring are in Evals before features.

Pillar 2: Guardrails

The most common setup we find is a single guardrail that leaves no record. There’s a regex for prompt injection at the input and nothing at the output or on tool use, and no log of what it blocked or when.

Guardrails work when they’re layered, so each layer catches what the one before it missed:

LayerWhat it checks
InputPII detection (redact or reject, per policy), prompt-injection heuristics, hidden-content scanning on externally-sourced text, schema validation
Tool-use policyAllowed tool set per agent role, parameter validation, forbidden arguments, rate limits
OutputSchema validation, toxicity/safety, hallucination checks against grounding sources where verifiable
Human-in-the-loopThreshold-based approval on high-stakes actions: cost, blast radius, low confidence

Three design rules keep the layers working:

  1. Every decision is logged. Each guardrail decision is recorded on the trace, so an audit can answer “which guardrail tripped, on which message, and when” without digging. A guardrail that blocks without a record teaches you nothing.
  2. Policy lives in config. Each agent’s guardrails are set in a YAML file, so a policy change is a reviewable diff that doesn’t redeploy application code, and security review and prompt changes stop colliding.
  3. The tool layer matters most. Input and output filters limit what the agent says. Tool policy limits what it does, and actions are where the damage happens. Give each agent role an explicit list of allowed tools, the same way you give a service account IAM permissions, and for the same reason.

Guardrails need their own adversarial tests, as much as the agent does. We sort ours by the six attack classes in DeepMind’s “AI Agent Traps” paper and set a pass threshold for each class separately, because one overall pass rate can hide a class that fails every time. A class with no test cases isn’t low risk; nobody has measured it yet. The Agent Reliability Kit post shows how that works with a real example.

Pillar 3: Tracing

Tracing answers a concrete question: a user says the agent gave them a wrong answer on Tuesday. Can you see what happened? If answering means grepping application logs and guessing, the system isn’t observable.

A production agent trace looks like this:

agent.invoke(session_id, user_id, tenant_id)
├── guardrail.input(name=pii, decision=allow)
├── cost.budget_check(budget_usd=0.05, projected=0.012)
├── gen_ai.chat.completions(model=gpt-5-mini, prompt_tokens=412, cost_usd=0.0008)
├── tool.call(name=lookup_order, latency_ms=124)
├── gen_ai.chat.completions(model=gpt-5, prompt_tokens=890, cost_usd=0.0042)
├── guardrail.output(name=schema, decision=allow)
└── guardrail.output(name=toxicity, decision=allow)

Every model call, tool call, and guardrail decision is a span, tied to the user, session, and tenant. Given a user report, you pull the trace and see the conversation: which model ran, what it cost, which tools fired, and what each guardrail decided.

The decision that matters most here is the format: OpenTelemetry’s gen_ai.* semantic conventions instead of a vendor SDK. That gives you two things. You can change vendors, because Langfuse, Braintrust, Datadog, or your existing OTel backend are all just exporters, and switching doesn’t touch application code. And frameworks that already emit gen_ai.* spans (Google ADK, Agno, LangChain via OpenLLMetry) get cost metering and trace analysis without changing a single call site.

Traces feed the other pillars too. The eval suite samples production failures from them, cost attribution adds them up, and replay depends on them: capture a production run, re-run it against a new prompt or model, and compare the outcomes side by side. When the next model generation ships, replay turns “should we upgrade?” from a debate into a table of equivalent / improved / regressed verdicts over real traffic. We keep the verdict cautious: any change in the structure of the output becomes mixed, so a person looks at it.

Pillar 4: Cost

Teams put off cost longest, because nothing breaks while they do. Then the workload scales and the bill becomes the incident.

Do it in two steps. Meter first, always on: every model call records tokens in, tokens out, and dollars on the same traces you already collect, tagged with tenant, feature, and model. Add controls second, once you can see the spend: per-request budgets that warn before they block, and model routing that tries a cheap model first and escalates only when a check fails. On workloads we’ve measured, routing cuts model spend by 40 to 70% with no quality change that the evals can detect, which is why cost controls come after evals and not before.

Cost and latency also belong in the eval harness, reported on every run next to accuracy. A prompt change that improves quality 2% and triples token spend is a trade-off someone should make on purpose, in review, with the numbers in front of them. The worked cost example and the questions finance will ask are in FinOps for AI workloads.

Measuring the Whole Thing: The Reliability Scorecard

The pillars are how you build. A scorecard tells you where you stand. We assess systems across eight weighted dimensions, each scored from 0 to 5:

DimensionWeightThe question it answers
Eval coverage20%Is agent behavior measured, per-step and end-to-end?
Regression detection15%Will a bad prompt or model change be caught before users see it?
Guardrails15%Are inputs, outputs, and tool calls bounded, and logged?
Observability15%Can you debug a failed conversation from a user report?
Cost controls10%Is spend attributed, budgeted, and routed?
Failure-mode mapping10%Is “what can go wrong” written down, with detection and playbooks?
Security & data handling10%Would this pass a security review tomorrow?
Deployment & operations5%Can the team ship, roll back, and run on-call for it?

A 3 in every dimension is the production bar. The full rubric is on our site, and the Agent Reliability Kit turns it into a YAML file you can render and compare between reviews.

Where teams actually land

Across our audits, the same two dimensions score lowest: regression detection and failure-mode mapping. Teams usually have some evals, so eval coverage lands at 2 or 3. But the evals don’t run in CI (regression detection: 0 to 1), and nobody has written down what can go wrong (failure-mode mapping: 0 to 1). So the team is one confident prompt tweak away from a regression nobody catches, and one incident away from improvising a response at 2am.

Where to Start: The Fix Order

If your system is below the bar, do these in order. Each step makes the next one safer:

1. Lock the versions

Pin prompts, tools, and models in source control. You cannot detect change against a moving baseline, and every later step assumes the baseline holds still.

2. Wire evals into CI

Thirty golden cases that block merges are worth more than 300 that nobody runs. From here on, every prompt tweak, tool change, and model swap is compared against a baseline automatically.

3. Write the failure-mode doc

One page: what can go wrong, how you would detect it, and what you would do. Hallucination, tool failure, infinite loops, runaway cost, provider outage. This document tells you which guardrails you need, which is why it comes before them.

4. Layer in guardrails

Chosen from the failure-mode list instead of a generic checklist. Input, tool-use policy, output, and human checkpoints, each logged and each with its own adversarial tests.

5. Instrument tracing

OTel gen_ai.* spans on every model call, tool call, and guardrail decision, tied to user and session. This makes the guardrails auditable and the next two steps possible.

6. Bring in cost controls

Metering first, always on. Budgets and model routing once you have sized the workload, because controls tuned against guesses cause mysterious failures instead of savings.

7. Document the runbook

So the on-call engineer at 2am does not have to be the person who built the system. AI incidents (a spike in hallucinations, latency, runaway cost) get the same treatment as any other production incident.

Teams usually reach for guardrails first, because they feel like safety. They’re fourth here because without an eval suite you can’t verify a guardrail works, and without tracing you can’t see what it did.

Honest caveats

  • A throwaway prototype doesn’t need this. If you’re a week into finding out whether an idea works at all, wrapping every call in guardrails is overhead. Build the reliability layer once you decide the agent is going to live, and before it meets real users.
  • The bar scales with the damage an agent can do. An internal document summarizer and an agent that moves money shouldn’t be held to the same standard. The scorecard weights are a default; for a high-stakes system, treat guardrails and security as merge blockers whatever their weight.
  • This is a floor. Everything here makes failures visible and bounded, and makes regressions catchable. It doesn’t make the model smarter. If the task is beyond the model, this work will tell you so quickly and precisely, which is useful, but it won’t fix it.

Where this leaves you

The model in the demo and the model in production are the same model. What differs is the engineering around it: evals that prove behavior, guardrails that bound it, traces that explain it, and cost controls that keep it affordable.

Teams that build this layer ship agents that hold up with real users, and they keep shipping improvements with confidence, because every change is measured against a baseline. Teams that skip it are left with an agent that worked once, in the demo.

To see where your system stands, score it honestly against the eight dimensions above; the gaps will be obvious. If you’d like experienced eyes on it, talk to us.