- Published on
Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
- Authors
- Name
- DeepHorus
NVIDIA has introduced SkillEvaluator, an open-source evaluation layer for AI agent skills. The tool measures how packaged instructions, examples, and tool guidance change agent performance by comparing task runs with and without each skill. It reports the difference as Skill Lift. The effort matters because agents can waste tokens and steps searching for tools even when using capable models.
The first benchmark covers more than 300 verified skills across over 30 NVIDIA products. Each skill was evaluated on two independent agent harnesses. Verified Skills are signed capability descriptors that tell an agent what an NVIDIA product does, when to invoke it, and how to call it. The verified label depends on measurement, not just documentation.
Before publication, skills pass three evaluation tiers. Tier 1 runs static checks for schema and frontmatter validation, quality scoring, prompt-injection and data-exfiltration scanning, secret and PII detection, license checks, and script linting. Tier 2 uses embedding similarity to identify duplicated guidance and overlapping coverage across the catalog. Tier 3 runs live evaluations against generated tasks in isolated sandboxes, once with the skill and once without.
Tier 3 uses Harbor, an open-source framework for repeatable agent evaluations. SkillEvaluator converts cases into Harbor task bundles, runs both conditions, grades them, and calculates impact. Each run uses the same prompt, model, task inputs, and grading criteria; the only experimental variable is whether the skill is installed. The comparison repeats across two agent harnesses, and the difference is reported as Skill Lift in points.
NVIDIA publishes plugins for Claude Code, Codex, and Cursor, and the same skills are available through Skills.sh, ClawHub, and Hermes Hub. A command-line example shows the workflow: skillevaluator create-eval-dataset ./my-skill --full creates evals/evals.json with explicit, implicit, contextual, and negative cases. Users then run skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker.
The live results use an August 12, 2026 snapshot of benchmarks.json at commit 738d79e. Scores are macro-averaged across published skill–harness results, giving each pair equal weight. Skill Lift is the with-skill score minus the without-skill score. As agents take on specialized work, NVIDIA's evaluation layer could help developers decide which skills are worth installing.
