- Published on
Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills
- Authors
- Name
- DeepHorus
On September 11, Anthropic introduced a plugin evaluation workflow for Claude Code. It measures whether a skill triggers, survives edits or model changes, and beats a bare model. The claude plugin eval command runs a plugin against realistic prompts, grades Claude's output, and compares it with a run without the plugin.
Previously, plugin developers had no behavioral testing. claude plugin validate only checks manifest syntax and schema, not real performance. The new workflow answers three questions developers could not measure before.
An eval suite lives in an evals/ directory inside the plugin. Each case contains a prompt.md and a graders/ folder. Six grader types exist: regex, tool_used, tool_order, and file_exists are free; llm and baseline call a judge model and add to the bill. Every case runs twice, with and without the plugin; the difference, Ξ, shows the plugin's contribution. One example scored WITH 1.00, W/OUT 0.33, Ξ +0.67 across six runs, costing $0.41 and taking 74 seconds.
Anthropic says the most common first finding is a Ξ near zero with the tool_used: Skill grader failing, meaning Claude is not choosing the skill on natural phrasing. That is a defect claude plugin validate cannot see. Results land under evals/results/ /report.html and can be published to claude.ai where supported.
The workflow runs on Claude Code v2.1.269 or later with any plugin manifest or skills-directory plugin. A documented CI invocation uses a --threshold 0.8 gate, letting teams block changes that degrade performance. Each eval run and judge grader is a real model call billed to the user's plan or API account.
By adding behavioral evals and a CI gate, Anthropic is pushing plugin development toward measurable quality. Developers can start with claude plugin eval init to generate cases and graders automatically. This comparative testing could become standard as models and plugins evolve.
