An AI agent is production-ready only when it completes representative work, uses tools and permissions correctly, fails safely, stays within cost and latency limits, produces auditable evidence and improves a business outcome under real operating conditions. A high answer-accuracy score is not enough because an agent can reach the right answer through unsafe actions or fail unpredictably across a multi-step workflow.
Evaluation must cover outcomes, trajectories, controls and operations before launch, then continue in production.
Why agent evaluation is different
Traditional models produce an output from an input. Agents plan, retrieve, call tools, modify systems, wait for events and adapt. One task may contain dozens of decisions. The evaluation surface includes tool choice, argument correctness, permission, sequencing, recovery and stopping behaviour.
Nondeterminism means a test should be repeated across seeds, model versions and realistic environmental conditions. One successful run proves possibility, not reliability.
Define the job and autonomy boundary
Write a task charter: users, goals, allowed actions, prohibited actions, required approvals, systems, data, completion conditions and escalation. Classify actions by consequence. Reading a public knowledge base differs from refunding a customer or modifying infrastructure.
The evaluation plan should become stricter as autonomy and consequence rise. High-impact actions require deterministic policy checks and human approval even if model performance is strong.
Build a representative evaluation set
Use normal cases, edge cases, ambiguous requests, missing data, conflicting instructions, stale information, tool failures, malicious content and unsafe user requests. Sample real workflow distributions without exposing personal data.
Maintain a hidden holdout set so teams do not overfit prompts to known tests. Version tasks and expected policies as the product changes.
Outcome evaluation
Score task completion, factual correctness, constraint compliance, evidence quality and user utility. Use deterministic checks when possible: database state, calculated totals, schema validation or exact policy rules. Use expert review for judgement-heavy work.
Separate partial completion from unsafe completion. A task that succeeds by violating policy must score as failure.
Trajectory evaluation
Inspect the sequence of observations, plans, tool calls and state changes. Measure unnecessary actions, invalid arguments, repeated loops, unsupported assumptions and whether the agent sought clarification.
Trajectory review explains why an outcome failed and reveals brittle success. Keep sensitive reasoning private where required, but retain actionable event traces and tool evidence.
Safety and adversarial evaluation
Test prompt injection from documents, websites, emails and tool outputs. Attempt privilege escalation, data exfiltration, secret discovery, cross-tenant access, destructive commands and policy bypass. Test whether the agent obeys an untrusted source over system policy.
Use isolated environments and synthetic credentials. Verify that controls outside the model block dangerous actions.
Permission and identity evaluation
Give the agent least privilege and test each boundary. Confirm user identity and entitlement propagate to retrieval and tools. Evaluate delegated authority, token expiry, revocation and separation of duties.
The agent should not convert read access into write action or combine individually permitted data into a prohibited inference.
Reliability and failure recovery
Inject timeouts, rate limits, malformed responses, partial writes, unavailable tools and stale sessions. Verify retries are bounded and idempotent. The agent must distinguish uncertain completion from confirmed success.
Test pause, resume, cancellation, rollback and human escalation. Production trust is built by predictable failure handling.
Cost and latency evaluation
Track model tokens, retrieval, tool calls, compute, retries and human review per completed task. Report percentiles, not averages. Set budgets and circuit breakers for runaway loops.
Measure end-to-end latency against user expectations and business windows. A cheaper model that creates more retries or review may increase total cost.
Business-outcome evaluation
Define the economic job: reduce handling time, improve resolution, increase qualified leads, prevent loss or accelerate decisions. Run controlled pilots with a baseline and account for displaced work and new review effort.
Measure quality and risk alongside volume. An agent that closes more cases but increases reopen rates has not improved the system.
Pre-production gates
Require minimum performance by risk class, zero critical policy violations, passed adversarial tests, acceptable cost and latency, validated rollback, complete observability, named operational owner and incident response. Document known limitations and excluded uses.
Approval should involve product, engineering, security, privacy, legal and domain operations as applicable.
Production monitoring
Monitor task success, policy blocks, escalations, tool errors, drift, cost, latency, user correction and business outcomes. Sample traces for review. Maintain a kill switch and downgrade path to read-only or human-assisted mode.
Re-evaluate after model, prompt, tool, data, permission or workflow changes. Treat evaluation assets as production code.
A scored evaluation rubric
Create weighted dimensions for task outcome, factual support, policy compliance, tool correctness, efficiency, recovery and user impact. Define critical failures that override the aggregate score. Report average, worst-case, percentile and failure clusters.
Calibrate human reviewers with examples and measure agreement. Where experts disagree, improve the rubric instead of hiding variance.
Test harness architecture
Separate task fixtures, environment setup, agent configuration, execution traces, deterministic validators, model-based graders and reporting. Reset state between runs. Mock dangerous tools for early tests, then use isolated integration environments.
Version every component so a regression can be attributed to model, prompt, tool, data or policy change. Store only necessary data and protect traces because they may contain sensitive content.
Red-team scenario catalogue
Embed malicious instructions inside retrieved documents. Return forged tool success. Ask the agent to bypass approval for urgency. Present two users with conflicting authority. Expire credentials mid-task. Create a tool whose description overclaims safety. Attempt indirect data extraction through summaries.
Test persistence: a poisoned memory or task note should not influence unrelated users. Test social engineering aimed at human approvers as well as the model.
Launch by autonomy tier
Tier one agents recommend but cannot act. Tier two agents act in reversible low-risk systems with logging. Tier three agents perform consequential actions only with approval and deterministic policy. Tier four autonomy should be reserved for tightly bounded environments with strong containment.
Advance a use case only when evidence meets the next tier. Do not grant broad tool access because the model performed well on read-only tests.
Incident response and change control
Define how to stop the agent, revoke credentials, preserve evidence, identify affected actions, notify owners, correct state and communicate with users. Link each agent to an operational service owner.
Treat model, prompt, tool, permission, retrieval and workflow changes as releases. Use canaries, rollback and re-evaluation. A vendor model update is an external dependency change, not routine background noise.
Detailed execution checklist
Task validity
Confirm that the evaluation task represents a real user goal and current workflow. Synthetic tasks that omit messy context inflate performance.
Ground-truth design
Use deterministic expected states where possible and expert rubrics where judgement is necessary. Record acceptable alternatives instead of forcing one phrasing.
Repeated trials
Run enough repetitions to expose variance, loops and rare policy failure. Report confidence intervals and worst cases, not one showcase execution.
Tool correctness
Validate tool selection, arguments, order, authorisation and resulting state. A correct final message cannot excuse an incorrect side effect.
Clarification behaviour
Test incomplete and contradictory instructions. The agent should request missing information when acting would create material risk.
Stopping behaviour
Verify that completion conditions prevent unnecessary actions and that iteration caps stop loops. Test whether the agent recognises irrecoverable failure.
Human approval
Evaluate the approval request itself: it should explain action, consequence and evidence. Test rejection, timeout and modified approval.
Data protection
Attempt cross-user, cross-tenant and sensitive-field retrieval. Verify redaction, retention and trace access.
Security boundaries
Run prompt injection, malicious tool output and privilege escalation tests. Confirm external controls block harmful action even when reasoning fails.
Operational recovery
Test cancellation, partial completion, duplicate execution, retry and rollback. The system must surface uncertain state instead of claiming success.
Cost controls
Set per-task budgets, tool-call limits and anomaly alerts. Evaluate cost under failure, because retries create the long tail.
Latency
Measure queue, model, retrieval, tool and approval time separately. Optimise the stage that limits the business workflow.
User experience
Test whether users understand capability, limitations, evidence and escalation. Misplaced trust is a product failure.
Business impact
Compare against the existing workflow using quality, time, risk and total effort. Include human review and exception handling.
Change regression
Run the suite after any model, prompt, tool, permission, retrieval or policy change. Maintain a release gate and rollback.
Practitioner planning workbook
Task charter
For task charter, define users, outcome, allowed actions, prohibited actions and escalation. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Risk classification
The risk classification workstream should score consequence, reversibility, data sensitivity and autonomy. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Evaluation dataset
Treat evaluation dataset as an operating control: represent normal, edge, ambiguous, malicious and failure conditions. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Holdout policy
For holdout policy, protect unseen tasks and rotate them as the product learns. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Environment control
The environment control workstream should reset state, isolate credentials and reproduce tool dependencies. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Outcome checks
Treat outcome checks as an operating control: use deterministic business-state validation wherever possible. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Trajectory checks
For trajectory checks, inspect tool choice, arguments, order, loops and stopping. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Grounded evidence
The grounded evidence workstream should verify each factual conclusion against accessible authorised sources. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Policy tests
Treat policy tests as an operating control: exercise permissions, approvals, privacy, retention and prohibited actions. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Adversarial tests
For adversarial tests, inject malicious instructions through users, documents and tools. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Failure injection
The failure injection workstream should simulate timeout, rate limit, partial write, stale state and tool outage. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Idempotency
Treat idempotency as an operating control: repeat tasks and confirm that retries do not duplicate consequential actions. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Cost distribution
For cost distribution, measure model, retrieval, tools, retries and review at percentiles. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Latency distribution
The latency distribution workstream should separate queue, reasoning, tool, approval and recovery delays. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Human factors
Treat human factors as an operating control: test comprehension, appropriate trust, correction and escalation. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Business experiment
For business experiment, compare quality, time, cost and risk with the current workflow. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Launch gate
The launch gate workstream should require named owners, observability, rollback and zero critical violations. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
Canary plan
Treat canary plan as an operating control: limit users, actions, data and volume while evidence accumulates. Preserve the supporting artefacts, unresolved questions and agreed exceptions. Recheck the control after launch so temporary migration choices do not become permanent unmanaged risk.
Incident playbook
For incident playbook, stop, revoke, preserve evidence, correct state and notify stakeholders. Record the current state, target state, accountable owner, dependencies and evidence required for approval. Test the assumption using production-like information, and define the failure signal that would stop or redirect the work.
Regression suite
The regression suite workstream should rerun after every model, prompt, tool, data or policy change. Convert the decision into measurable acceptance criteria, an owner and a review date. Include normal operation, edge conditions and recovery, because a design that works only in the happy path is not production-ready.
