- Published on
AI agents have bad taste in long-term decisions and a new benchmark proves it
- Authors
- Name
- DeepHorus
Microsoft and City University of Hong Kong researchers introduced Taste-Bench, a benchmark for evaluating AI agents' long-term decision quality. The best model barely exceeded 60%. It matters because agents increasingly handle multi-step research and software tasks where early choices have delayed costs.
Existing benchmarks only check completion, not step quality. The authors call good long-term choice taste. Taste-Bench measures decisions at decision forks, where trajectories branch and later outcomes reveal which path was better.
The benchmark contains 502 questions. Each includes the original task, the trajectory up to the fork, and two possible directions. Data came from thousands of scored runs on coding and AI research tasks, including SWE-bench Pro and METR's MALT. A generator model filtered suitable forks, removing trivial or undecidable cases, and human review confirmed almost all labels.
The team tested 14 current models from Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Questions were asked twice with flipped options, and only consistent correct answers counted. Pure guessing scores 25%; GPT-5.6 Sol averaged 59.7%, followed by GPT-5.5 at 59.5%.
Accuracy depended on when evidence appeared. When key clues were visible in prior steps, average accuracy reached 62.3%. When clues appeared only after extensive later work, accuracy fell to 21%, near random guessing. Detour forks were harder than parallel forks.
Longer reasoning did not solve the problem. Switching two models from lowest to highest reasoning settings barely changed performance. The results suggest current agents struggle to predict future consequences and need better methods for long-horizon judgment. No model came close to solving the benchmark despite every question being a binary choice.
