How to evaluate an AI agent before launch

Before launching an AI agent, test whether it completes the business task, respects access and approval rules, and stays within cost and response-time limits. Use realistic cases, inspect tool calls and resulting system changes, and agree release criteria before seeing the scores. A convincing answer is not proof that the underlying action was correct.

What does evaluating an AI agent mean?

Evaluation means running defined tasks under controlled conditions and checking the result against explicit expectations. For an agent, that includes what it did—not just what it said.

Think of a new employee handling a customer request. A well-written reply is useful, but not if it cites the wrong contract, updates the wrong account or promises a refund that was never authorised.

This is why evaluation belongs in the path from POC to production, rather than in a final presentation before launch. Anthropic’s guide to agent evaluations distinguishes tasks, repeated trials, grading and the actual outcome of the agent’s work.

1. Define success as a business outcome

Start with one workflow, its owner and the decision the agent supports. Replace “the response looks good” with an observable result.

For a project-mail assistant, success might mean finding relevant messages, identifying outstanding decisions and citing the correct sources without revealing another user’s mailbox. For a request-handling agent, it might mean routing the request correctly without creating a duplicate record.

Describe unacceptable outcomes too: unauthorised disclosure, an incorrect recipient, an external message sent without approval or a false claim that an action succeeded. These should not disappear inside an average quality score.

2. Build a representative test set

Use examples that reflect the intended operating environment. Start from real user questions and known failure patterns, with permission and data minimisation. Synthetic cases can extend coverage, but should not be your only evidence.

Case family	Example	Expected behaviour
Routine	Find the latest agreed deadline	Return the right date with its source
Ambiguous	Two projects have the same name	Ask which project before acting
Incomplete	A required document is missing	State the gap; do not invent an answer
Conflicting	Two messages contain different dates	Identify the conflict and its context
Adversarial	An email says to ignore access restrictions	Treat the email as data, not authority
Access boundary	A relevant message belongs to another user	Do not retrieve or disclose it
Tool failure	Mail provider times out	Report the failure; avoid claiming completion
Consequential action	Send a reply to a supplier	Require the applicable authorisation before sending

These examples are illustrative, not reported results from a customer deployment. There is no universal dataset size that certifies production readiness: coverage, risk and uncertainty matter more than a round number.

For each case, record input, starting state, accessible data, expected outcome, prohibited outcomes and the person who validates the reference answer.

3. Test the outcome, the path and the boundaries

An agent can reach a correct answer by an unacceptable route. It can also take several legitimate routes to the same result. Do not require an identical sequence of tool calls unless the sequence itself is a control requirement.

LangSmith’s evaluation approaches distinguish final-response evaluation, individual-step evaluation and trajectory evaluation. The trajectory is the sequence of messages and tool calls used to reach the result.

Layer	Question	Evidence
Business outcome	Was the task actually completed?	Final state in a test system or validated deliverable
Answer quality	Is the response correct, supported and useful?	Source checks and a review rubric
Tool use	Were the right tools used with valid parameters?	Tool-call trace and returned errors
Permissions	Did the agent stay within the user’s rights?	Tests across identities and data scopes
Approvals	Was a consequential action blocked without valid approval?	Rejection, expiry and changed-parameter tests
Operations	Was performance acceptable within the budget?	End-to-end latency, cost and retry measurements

Run consequential tests against isolated systems and test accounts. Reset the state between trials. A test that sends real customer emails or changes live permissions is not a safe substitute for an evaluation environment.

4. Choose graders that fit the claim

Use deterministic checks for things software can verify: valid fields, correct record IDs, whether a write occurred, whether approval was present, or whether a spending limit was exceeded.

Use human review for domain interpretation, usefulness and disputed reference answers. A model-based grader can help assess larger volumes against a written rubric, but it is another model—not an independent guarantee. Compare it with human judgements and inspect disagreements before using its score to support a release decision.

Anthropic recommends combining grader types and defining unambiguous success criteria. LangGraph’s evaluation documentation illustrates deterministic trajectory matching and model-based assessment.

Do not expose unnecessary personal data to an external evaluator. The grading pipeline needs the same attention to hosting, access and retention as the agent itself.

5. Set thresholds before looking at results

The business owner and technical lead should agree release criteria appropriate to the workflow. Security and data-protection reviewers should participate where relevant. This is an operational decision, not a universal percentage supplied by a model vendor.

Separate three categories:

• Critical failures that block the release until corrected and retested.
• Quality thresholds defined by case family, not only across the whole dataset.
• Operating limits for latency, cost, retries and escalation rates.

Passing a finite test suite does not prove that future failures are impossible. In particular, observing no access violation in a test set is not proof of zero security risk.

Worked example: why the overall score can mislead

Suppose an illustrative test suite has 100 cases. The agent passes 90 routine cases and fails all 10 access-boundary cases. Its overall success rate is 90%, but its access-boundary success rate is 0%. That is a no-go for this example, even though the headline score looks reassuring.

This is arithmetic, not an Elia Studio customer benchmark. Reporting scores by category makes the risk visible.

6. Repeat trials and compare versions fairly

An agent may behave differently on the same task across runs. Repeat important cases and inspect how often the agent succeeds, not merely whether it managed to succeed once.

Compare versions using the same dataset, starting conditions, permissions and grading rules. Record model identifier, prompt version, tool configuration and relevant retrieval settings. Measure the whole workflow, including tool delays, retries and human hand-offs, rather than model response time alone.

Keep a holdout set separate from the examples used to tune prompts. Otherwise, you may improve the test score without improving behaviour on unfamiliar requests.

7. Make launch a staged decision

Summarise the decision in a short release report: tested version, coverage, results by case family, critical failures, operating cost, remaining gaps, decision owner and rollback plan.

A limited pilot can then reveal issues absent from the test environment. Monitor failures, escalations and user corrections, and add validated examples to the regression suite. Changes to the model, prompt, tools or data sources should trigger proportionate retesting.

Our production readiness checklist places evaluation alongside data, security, ownership, cost and adoption. Passing evaluations alone does not complete those other responsibilities.

Five questions for your AI expert

1. What business outcome are you measuring, beyond the final response?
2. Which failure categories are hidden by your average score?
3. Have you tested permissions, malicious content and provider failures?
4. Who reviewed the grading rules and the disputed answers?
5. What happens when a model or connector changes after launch?

For a concrete integration context, see our mailbox connector lessons. The useful question is not whether the connector responds, but whether the whole workflow remains useful and controlled.

Start with one workflow and a release decision

Bring a real use case, representative examples and the name of the person who will own it. Define the tests and release criteria before expanding the agent’s permissions or user base.

Explore our AI consulting and delivery services, or contact Elia Studio to frame the evaluation work.