AgentGuard Evals · Round one · September 22, 2026

AgentGuard Evals

Benchmarks score the answer. AgentGuard Evals scores the path: the shell commands and file writes an AI coding agent makes on a real task, run through the same local rules the AgentGuard plugin runs before a tool call.

Round one uses the agent transcripts Vercel publishes for its Next.js evals, for the two models that launched on September 22, 2026. Vercel ran the agents and graded the tasks. Vercel did not run, review or endorse this analysis.

30 of 31

tasks passed by each model, and both missed the same one.

Vercel's pass@4 result. Claude Fable 5.1, also at 97% on Vercel's table, missed that task too.

745

shell commands and file writes scanned across both models.

Reads and web searches are not scanned.

stops across 75 attempts. No action matched an AgentGuard rule.

Every rule match would count as a stop, the default enforce action.

≤3.9%

95% upper bound on the share of attempts like these that would hit a stop.

Counted per attempt, because actions inside one attempt are not independent trials. Per action the bound is 0.40%. More runs tighten both.

Current round

Model	Agent	Tasks passed	Attempts	Shell commands	File-write/edit tool calls	Stops	Stops on passing attempts	Subagent-spawn tool calls	Web-search tool calls
Claude Opus 5.5 (high)	Claude Code	30 of 31	35	253	103	0	0	0	0
GPT 6 Sol (high)	Codex	30 of 31	40	389	0	0	0	0	275

Models are listed in the order Vercel lists them. There is no overall score, and AgentGuard is never a row. GPT 6 Sol writes files through shell patches, so its edits appear under shell commands.

What the transcripts show

The 97% tie is one task wide

Both models passed 30 of 31 tasks and failed the same one, agent-044-uses-nextjs. The failures cluster; they do not scatter.

Scary-looking commands were ordinary work

Opus 5.5 recorded 22 recursive-delete commands and GPT 6 Sol recorded 4. They removed build output such as .next, test data from the agent's own smoke tests, and the old pages directory the router-migration tasks replace. None targeted root, home or the whole working directory, so none matched a rule.

Same score, different habits

GPT 6 Sol made 275 web-search tool calls across its 40 attempts. Opus 5.5 made none, and made 103 file-write/edit tool calls. Neither made a subagent-spawn tool call.

We grade ourselves too

"Stops on passing attempts" counts times our rules would have stopped an attempt that Vercel's tests passed. We review every one: a stop on risky work that still passed is a correct stop, and a stop on ordinary work counts against AgentGuard. We publish the column every round, whatever it shows. A guard that blocks good work is worse than none.

Method

Item	Round one
Transcripts	Vercel's published result sets `claude-opus-5.5-high/2026-09-22T20-34-17.138Z` and `gpt-6-sol-high/2026-09-22T19-54-01.135Z` in vercel/next-evals-oss (MIT license), commit `bb7c02e`.
Pass or fail	Vercel's own grading: pass@4 per task from its results file, and each attempt's `result.json` status.
Scanner	The AgentGuard Guard Pack, 14 rules, from plugin 0.3.4 (`runtime/guard-pack.cjs`, SHA-256 recorded in the data file). A static scan of recorded tool calls. Nothing was executed. Re-scanned on September 23 with the released 0.3.6 Guard Pack: 0 stops, 0 incomplete scans on the same 745 actions.
Statistics	Exact one-sided 95% bounds for zero events in n trials, 1 minus 0.05 to the power 1/n: 3.9% per attempt (n = 75) and 0.40% per action (n = 745). Attempts are the independent unit.
Not replayed	Spawn and spend limits. Neither model launched a subagent in these runs.
Published data	Counts, task names and rule IDs only. No command text. Read the round one data file.

This round covers two result sets on one benchmark. It says how often these runs would have been stopped, not how an agent behaves on every task or machine. Later rounds add more public transcript sets and keep each round's date and scanner version.

See the plugin that runs these rules · What leaves your machine