- Published on
2.2x faster, 27% cheaper
- Authors
- Name
- DeepHorus
Ploy, an AI agent that builds and edits real marketing websites, has migrated its production system to GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. The switch follows months of testing where no model could challenge Claude Opus due to Ploy's high quality bar. After a head-to-head evaluation, GPT-5.6 Sol became the default model powering every Ploy workspace, marking a significant change from the Claude Opus 4.8 that held the default slot for four months.
Ploy's agent handles complex tasks: planning a page, reading codebases, writing components, generating imagery, and self-screening its own work. This job description sets a very high bar for a model, and every frontier release was tested against it. For four months, Claude Opus (first Opus 4.7, then 4.8) remained unmatched until GPT-5.6 Sol demonstrated superior performance in both speed and cost.
The migration yielded impressive metrics: builds finishing in less than half the wall-clock time at 27% lower cost, with output quality at or above the incumbent. GPT-5.6 writes leaner code—on one matched pair, it produced a 2,508-character globals.css with 45 CSS variables compared to Opus's 17,957-character file with 174 variables. This efficiency translated to about half the output tokens, making GPT-5.6 both faster and more economical.
However, the transition was not seamless. Despite using Vercel's AI SDK, a universal LLM SDK, switching from Claude Opus to GPT-5.6 required discovering provider-specific behaviors that the entire stack had quietly specialized around. These included differences in how models fill in tool arguments, handle prompt caching, and replay reasoning between turns. Fixing these issues involved updating the eval harness, tool schemas, caching logic, and reasoning replay mechanisms.
A critical lesson emerged: the eval harness was tuned to the incumbent model. Roughly a third of initial failures in cross-model runs traced back to harness assumptions, not model behavior. For instance, the harness's tool-call budgets were sized for Opus's sequential style, while GPT-5.6 fans out parallel calls, causing false failures. Similarly, the executor didn't support batched file reads, which GPT-5.6 uses heavily. Only after triaging these traces did the true performance gains become apparent.
Ploy's team emphasizes that fair evaluation requires adjusting for each model's strengths. They discovered a dataset that omitted its minScore threshold silently inherited a default of 1.0, causing GPT-5.6 to "fail" a hero it scored 0.98 on while Opus passed others inconsistently. Once the harness was cleaned up, GPT-5.6 demonstrated clear superiority: 2.2× faster to a finished page, 27% cheaper, and consistently high quality.
With migration complete, GPT-5.6 Sol now powers every Ploy workspace, offering immediate improvements in speed and cost. The move signals a shift in the AI agent landscape, where newer models can outperform established leaders if integrations are handled carefully. Ploy's experience provides a blueprint for other teams considering similar migrations, highlighting the importance of robust eval harnesses and provider-specific optimizations.
