Reproducible agent-evaluation evidence · MIT + CC BY 4.0

Deterministic agent release gate

Compare a candidate agent with its baseline case by case. The fixture checks completion, tool policy, order, grounding, steps, retries, and supplied cost without calling a model or an external API.

This deliberately failing candidate proves the gate mechanics. It does not measure model quality, real latency, production cost, dataset coverage, or judge calibration.

Six synthetic agent cases with one pass, seven new failures, and a blocked release gate

Files

agent_gate.ts Strict TypeScript release-gate evaluator fixture.ts Six synthetic baseline and candidate scenarios agent_gate_test.ts Determinism, policy, and failure-code tests run_fixture.ts Dependency-free manifest runner agent-evaluation-manifest-2026-08-29.json Reviewed failing-candidate manifest render_manifest.ts Deterministic SVG renderer agent-evaluation-gate-2026-08-29.svg Diagram generated from the saved manifest agent-release-gate-checklist.md Checklist for adapting the gate to a product evidence_artifact_test.ts Stored evidence and credential-pattern checks evidence-run-2026-08-29.md Commands, hashes, results, and limits README.md File map and independent rerun commands

Per-case contracts

Each scenario owns its required tools, forbidden effects, order, grounding, and execution budgets. There is no universal score threshold.

Regression-aware gate

Existing failures remain visible, while only new failures with configured critical or high severity block this candidate.

Inspectable output

The manifest keeps every reason in stable order, and both the JSON and SVG byte-match a fresh deterministic run.

Verified scope

Nineteen tests cover each failure code, invalid policies, duplicate cases, stable ordering, baseline comparison, saved JSON, saved SVG, licensing, and credential or direct-PII patterns. The fixture uses six synthetic cases and no paid request.

MIT license CC BY 4.0 license Read the agent testing guide