Open rubric · Apache-2.0

Agent Reliability Scorecard

Eight dimensions, each scored 0 to 5 and weighted into one number. A 3 on every dimension is the production bar. Score your agent yourself, or have us run it.

Get the kit on GitHubHow engagements work

Scoring scale

  1. 0
    Absent

    Nothing exists.

  2. 1
    Ad hoc

    Tried once, not maintained.

  3. 2
    Inconsistent

    Covers some flows, not others.

  4. Production bar
    3
    Adequate

    Happy path and the obvious edge cases.

  5. 4
    Mature

    Versioned, automated and owned.

  6. 5
    Exemplary

    Failure modes found before users find them.

The eight dimensions

Weights add up to 100%. Back every score with evidence: a file, a trace, a CI run.

01

Eval coverage

20%

Is agent behaviour measured?

  • Versioned golden set in source control
  • Adversarial set for jailbreaks, PII leaks and tool misuse
  • Per-step evals for multi-step agents
  • LLM judges checked against human labels
  • Production samples feed back into the dataset
02

Regression detection

15%

Is a bad change caught before users see it?

  • Evals run in CI on every prompt, tool or model change
  • Merges blocked past a set threshold
  • Pass rates tracked over time
  • Model versions pinned, and upgrades compared on evals
03

Guardrails

15%

Are inputs, outputs and tool calls checked?

  • Input: PII, prompt injection, schema
  • Output: schema, verifiable facts, safety
  • Tools: allowed per role, parameters checked, rate limited
  • Human approval on high-stakes actions
  • Every guardrail decision logged
04

Observability

15%

Can you debug a conversation from a user report?

  • Every model call, tool call and guardrail decision traced
  • OpenTelemetry gen_ai conventions
  • Traces linked to user, session and tenant
  • At least 30 days of production traces kept
  • AI traces next to the rest of your telemetry
05

Cost controls

10%

Is the bill predictable, and can you find leaks?

  • Token and dollar cost per request
  • Cost by tenant, feature and model
  • Per-request budgets, soft or hard
  • Cheap model first, escalating when needed
  • Prompt caching where it pays
06

Failure-mode mapping

10%

Is what can go wrong written down?

  • A list covering hallucination, tool failure, loops, runaway cost and outages
  • A way to detect each one
  • A mitigation or playbook for each one
  • Updated after every incident
07

Security and data handling

10%

Would it pass a security review tomorrow?

  • No PII or secrets in prompts unless redacted
  • Least-privilege access for every tool
  • Audit log of every action on user data
  • Provider set to no training and a pinned region
  • All six attack classes from the AI Agent Traps research tested, each with its own threshold
08

Deployment and operations

5%

Can the team ship and roll back safely?

  • Infrastructure as code; prompts and tool config versioned
  • Feature flags or staged rollouts
  • A runbook for AI incidents: hallucination spikes, latency, cost
  • Rollback without redeploying everything

Overall score

Multiply each score by its weight and add them up: sum(score × weight).

  • below 2.0 Not production ready. Don’t scale traffic.
  • 2.0 to 2.9 Pilot quality. Real users will hit problems.
  • 3.0 to 3.9 Fine for non-critical workloads, with a plan for the gaps.
  • 4.0 to 4.5 Mature. Ready for critical workloads with monitoring.
  • above 4.5 Industry leading, and rare.

Where teams usually score low

Typically some evals exist (2 or 3), nothing runs them in CI (0 or 1), and nobody has written down the failure modes (0 or 1). Fix in this order; once versions are pinned and evals block merges, every later change is measured.

  1. 1 Pin prompt, tool and model versions
  2. 2 Run evals in CI and block merges on regression
  3. 3 Write the failure-mode list
  4. 4 Add guardrails
  5. 5 Add tracing
  6. 6 Add cost controls
  7. 7 Write the runbook

Want it run on your agent?

A Snapshot takes 3 to 5 days at a fixed price and needs only a staging endpoint and a trace export. A two-week Audit adds evidence from your code and a prioritised roadmap.

Book a Snapshot