- Published on
Smevals - a small eval suite for evaluating models, prompts, and harnesses
- Authors
- Name
- DeepHorus
On July 31, 2026, a new tool called smevals was announced for running small evaluation suites across different model configurations. Developed in collaboration with Jesse Vincent's Prime Radiant applied AI research lab, smevals aims to answer questions about the capabilities of various AI models. The tool is designed to be used with coding agents and provides a streamlined workflow for creating and applying evals.
The project represents the developer's third iteration on the idea of evals, which they have been trying to refine for several years. The tool addresses the need for a lightweight but flexible way to test model performance on specific challenges. It separates the running of evaluations from the grading of results, allowing users to compare runs across multiple configurations.
Users can start by having a coding agent execute uvx smevals doc to learn the tool, then instruct it to build an eval suite. An eval is a directory containing YAML files that define tasks. To run an eval against models, the command uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 is used. After collecting runs, users can grade them with uvx smevals grade path-to-eval/.
The results can be explored via a localhost web server using uvx smevals serve path-to-eval/. Alternatively, the smevals build command compiles that report as static HTML for hosting anywhere. An example eval suite was created to evaluate how well models can write haikus. This demonstrates the tool's versatility across creative and technical domains.
The most challenging part of developing smevals was establishing its vocabulary. An eval is a collection of challenges designed to answer a question about a model, while a task is a specific challenge like "Generate an SVG of a pelican riding a bicycle." Configs specify the model and other parameters such as system prompts or agent harnesses. A run records what happened when a specific config executed a specific task, and a runner is the script that executes the run.
Grading is performed by a grader that produces a grade using a sequence of checks. These checks can be simple, like verifying a string or validating XML, or complicated custom operations implemented as checkers, including using other models to answer questions about the run. This flexible grading system enables both quick sanity checks and deep evaluation.
The developer expressed satisfaction with the tool, stating that "it feels right" after multiple attempts. They look forward to expanding smevals in the future and applying it to their own projects. With its open command-line interface and modular design, smevals could become a useful addition to the AI evaluation toolkit.
