- Published on
OpenAI's Two Improvements Boost GPT-5.6 Sol's Score by 188%
- Authors
- Name
- DeepHorus
OpenAI today (July 31) announced that by improving its external framework, its latest model GPT-5.6 Sol saw a 188% score increase on the most stringent AI interactive reasoning benchmark, ARC-AGI-3, jumping from 13.3% to 38.3%. This achievement not only set a new record for the benchmark but also highlighted the significant impact of a model's external architecture (i.e., the "harness") on AI performance, sparking widespread industry interest in the emerging field of "harness engineering."
ARC-AGI-3, introduced by the ARC Prize team, is currently recognized as the most cutting-edge interactive reasoning evaluation benchmark for measuring AI general intelligence. Unlike traditional static tests, ARC-AGI-3 places AI in completely unfamiliar "game environments," requiring it to explore, learn, and adapt in real time like a human. This test is designed to examine AI's generalization ability in unknown tasks. Previously, GPT-5.6 Sol's raw score on this benchmark was only 7.8%, far below OpenAI's expectations, and the previous generation GPT-5.5 scored a mere 0.4%, underscoring the extreme difficulty of the task.
After a review, OpenAI found that the problem was not with the model itself but with two key mechanical flaws in the official test harness. First, after each game action, the model's internal private reasoning process was completely discarded, causing the model to re-understand the game after every step. Although it could see past action records and brief notes, it lost the plans and thoughts that led to those actions. Second, the framework employed a rolling truncation window. When conversation history exceeded 175,000 characters, earlier action records disappeared from context, meaning the model not only lost short-term thinking but also failed to retain long-term memory.
To address these flaws, OpenAI proposed two solutions: reasoning retention and context compression. For reasoning retention, the team reimplemented the test harness using the Responses API. By passing the previous response ID, the model could automatically retain its internal reasoning process during tool calls and multi-turn interactions. Once enabled, GPT-5.6 Sol no longer needed to re-understand the game each round, significantly reducing thinking time and allowing it to learn based on coherent long-term strategies. For context compression, OpenAI completely replaced the rolling truncation mechanism, allowing the model to continuously retain early observations and actions during long tasks, preventing performance degradation from a full context window.
The actual test results were astonishing: under the official harness, GPT-5.6 Sol scored 13.3%, but with the two new mechanisms enabled, the score soared to 38.3%, an improvement of 188.42%, while the output token count dropped to one-sixth of the original. This means less computational resources for higher reasoning capability, with qualitative leaps in the model's "memory" and "thinking continuity" on complex tasks. OpenAI emphasized that both tests used exactly the same model weights and temperature parameters, with only the harness configuration changed, so the difference came entirely from external engineering optimization.
This achievement quickly sparked heated industry discussion. The ARC Prize team praised OpenAI's optimization for "revealing a key variable often overlooked in intelligence evaluation." Several AI analysts noted that the industry has long focused excessively on model architecture and training data, underestimating the impact of how models are "steered" on their true capabilities. One researcher commented: "It's like putting the same car on different tracksāthe quality of the track design directly determines the result." The open-source community observed that reasoning retention and context compression are general techniques that can be adopted by other players, potentially fueling new discussions about fairness in competition.
For developers, this finding means that while pursuing model scale, equal attention must be paid to context management and memory persistence during inference. OpenAI plans to integrate these two improvements into its official API, so all users calling the GPT-5.6 series will automatically benefit in the future. Additionally, OpenAI stated it will continue to optimize harness robustness and explore more advanced memory compression algorithms to handle longer-duration, more interactive complex tasks. Industry experts predict that the next focus of competition will shift from "model capability" to "engineering steering capability," and ARC-AGI-3 may become the proving ground for new standards.
From a broader industry trend perspective, AI evaluation is moving from static question-answering to dynamic interaction, and how to fairly assess intelligence has become a greater challenge than training. OpenAI's adjustment is not just a technical fix but also a wake-up call: AI's true potential depends not only on compute and parameters but also on how humans design the "reins" for it. As the AGI race enters deeper waters, challenges like ARC-AGI-3 will force the entire industry to re-examine the definition of "intelligence," and harness engineering is poised to become the next disruptive innovation field.
