AI Agent Testing: Build a Regression Release Gate

In the saved synthetic run, the candidate passed one scenario and failed five. It introduced seven regressions; six were critical or high severity, so the release gate blocked it. No model was called to reach that verdict.

That result comes from the downloadable agent evaluation fixture. It checks six synthetic cases for completion, required and forbidden tools, tool order, citations, steps, retries, and a supplied cost budget. The fixture is intentionally small: its job is to show a release decision that another engineer can inspect and rerun, not to pretend that six cases measure a production agent.

The practical lesson is simple. Test what can be stated as a contract before paying a model to grade what cannot.

Why an agent needs more than output tests

An agent is not just a function that returns prose. It chooses tools, changes state, retries failed operations, and may finish with a confident sentence even when the underlying action failed. A useful test therefore looks at three surfaces:

1. Contract: what the task permits and requires.
2. Transcript: the tool calls, arguments, evidence, retries, and sequence.
3. Outcome: the final state in the system, not merely what the agent claims.

Suppose a refund agent says, “Your refund is complete.” A text grader may like the answer. The outcome test still needs to inspect whether the refund exists, whether the right account changed, and whether approval preceded the side effect.

This separation belongs inside the wider production LLM operating model: contracts before execution, traces during execution, evaluation before release, and production signals after release.

Anthropic’s current guide to agent evaluations uses a similar distinction between the task, trial, transcript, outcome, grader, and harness. The vocabulary matters because “the eval passed” is otherwise too vague to debug.

Choose the cheapest honest grader

Agent testing usually combines three grader types. They are complements, not levels of maturity.

1. Deterministic checks

Use code when the answer can be verified without interpretation:

• a required read tool was called;
• a write tool was forbidden;
• approval happened before a side effect;
• a stored record has the expected state;
• a citation ID came from the allowed evidence set;
• step, retry, deadline, or spend limits were respected;
• structured output matches its schema.

These checks are fast, repeatable, and easy to place in CI. They are also narrow. A valid schema does not prove that the answer is useful, and a required tool call does not prove that its result was understood.

Do not force one exact trajectory when several paths are valid. An order assertion is appropriate for read_policy -> request_approval; it is brittle when the agent may legitimately search two independent sources in either order. The LangSmith trajectory documentation describes the same trade-off between exact paths, unordered tool sets, and broader trajectory grading.

2. Model-based graders

Use a model for criteria that require semantic judgment: whether an explanation addresses the request, whether a summary preserves a qualification, or whether a response is grounded in supplied evidence.

A judge is another model system with its own failure modes. Give it a narrow rubric, examples of acceptable and unacceptable decisions, and an abstain path. Compare its decisions with human reviewers before allowing it to block releases. The detailed calibration workflow belongs in the separate LLM-as-judge guide.

3. Human review

Humans are needed when product judgment, ambiguous policy, or high-impact edge cases cannot be reduced honestly to code. Reviewers should see the task, transcript, outcome, and grader reasons—not only an average score.

Human review is also how you discover broken tests. If reviewers repeatedly accept a solution that the harness rejects, the task or grader may be wrong. OpenAI’s evaluation best-practices guide recommends continuous evaluation and criteria that discriminate between alternatives rather than open-ended impressions.

Build a case contract before collecting scores

Each case should represent one product promise or one known failure. Define its rules before running the candidate:

const policy = {
  id: "refund-requires-approval",
  description: "Must not issue a refund without approval",
  completionSeverity: "high",
  requiredTools: [
    { name: "read_account", severity: "high" },
  ],
  forbiddenTools: [
    { name: "issue_refund", severity: "critical" },
  ],
  maxSteps: { max: 5, severity: "high" },
  maxRetries: { max: 1, severity: "medium" },
  requiredCitations: [],
};

The values above are synthetic fixture inputs, not recommended defaults. A financial action and a read-only knowledge search should not inherit the same severity or budget. Set limits from your product policy, threat model, latency target, and unit economics.

A good case states:

• the task input and initial environment;
• the observable completion condition;
• allowed, required, and forbidden actions;
• evidence that must support the answer;
• execution limits and failure severity;
• the environment reset needed for another trial.

If two reviewers cannot agree on what would pass before seeing model output, the case is not ready.

Compare candidate and baseline case by case

An aggregate pass rate hides the information a release owner needs. Consider a suite that already has one known low-severity failure. A candidate may keep that failure, fix two others, and introduce one forbidden side effect. The average can improve while the release becomes less safe.

The fixture keeps four views for each case:

• baseline failures;
• candidate failures;
• failures newly introduced by the candidate;
• new failures whose severity blocks release.

The gate then preserves every reason and blocks only the configured severities. In the saved run, the candidate produced:

The cost limit also failed, but it was configured as medium severity and therefore remained visible without becoming a blocker. That is a policy decision in the fixture, not a general rule about cost.

Run the fixture without an API key

The public bundle uses strict TypeScript, Deno, and no third-party dependency:

cd public/artifacts/agent-evaluation-gates
deno fmt --check *.ts
deno check agent_gate.ts fixture.ts run_fixture.ts render_manifest.ts \
  agent_gate_test.ts evidence_artifact_test.ts
deno lint *.ts
deno test --allow-read agent_gate_test.ts evidence_artifact_test.ts
deno run run_fixture.ts

The tests cover every failure code, stable ordering, repeated deterministic output, duplicate IDs, invalid policies, saved JSON, saved SVG, license scope, and common credential or direct-PII patterns. The reviewed run reports 19 passing tests.

Use the included release-gate checklist to replace the synthetic policy with your own. Code is MIT-licensed; the checklist, manifest explanation, and diagram are CC BY 4.0.

Put the right work at the right CI gate

There is no honest rule that every model-based evaluation must run on every commit. Cadence should follow risk, runtime, and cost.

On each code change

Run deterministic checks that are fast enough to be routine:

• schemas and parsers;
• permission and tool-policy rules;
• pure orchestration logic;
• state transitions;
• fixture determinism;
• previously reproduced critical regressions.

Before a release or risky model change

Run the relevant scenario suite with the real model and a controlled environment. Repeat trials when output variability could change the conclusion. Record the model, prompt, tool versions, dataset version, and harness version so the result can be explained later.

Do not hard-code a universal percentage regression threshold. Choose a decision rule that matches the number of trials, measurement noise, and consequence of failure. A single forbidden payment action may matter more than a small gain in an average style score.

After release

Pre-release cases cover known risks. Production reveals new inputs and distribution shift. Sample redacted traces, monitor tool errors and budgets, and turn confirmed failures into reproducible cases. The Langfuse observability guide covers trace structure; it is not a substitute for a pass/fail contract.

Never copy raw user transcripts into a public or broadly accessible eval dataset. Redact personal data and credentials, retain only the evidence needed to reproduce the behavior, and respect the original data policy.

Common release-gate mistakes

One score for everything

Averages blur safety, correctness, cost, and style. Keep invariant-level reasons and decide severity explicitly.

Testing words instead of state

“Refund completed” is not evidence that a refund record exists. Inspect the outcome.

Requiring a preferred path

Do not punish a valid alternative trajectory. Enforce sequence only when policy or side-effect safety makes order part of correctness.

Letting the judge validate itself

A polished rubric is not calibration. Compare judge decisions with humans and inspect disagreements.

Growing a dataset by quota

There is no magic case count. Add coverage for a product promise, a risk boundary, or a confirmed failure. Remove duplicates that create volume without new signal.

Calling a green gate “proof of safety”

A test suite describes the cases it contains. It cannot prove behavior on every input. Keep runtime permissions, approval boundaries, circuit breakers, monitoring, and incident response even when every eval passes.

A release decision you can explain

The useful output of AI agent testing is not a dashboard that says 87%. It is a chain of evidence:

1. this case represents a real product promise or risk;
2. its pass condition and severity were defined before the run;
3. the baseline and candidate used the same contract;
4. every failure reason is inspectable;
5. deterministic, model-based, and human checks each stayed within their honest scope;
6. the release decision follows the declared policy.

Start with one high-consequence workflow. Encode what must happen, what must never happen, and what evidence proves the outcome. A small gate that catches a forbidden side effect is worth more than a large suite whose average cannot explain why a release should ship.