- Published on
A benchmark for the failure modes of agent memory
- Authors
- Name
- DeepHorus
A new open-source benchmark aims to expose the hidden failure modes of AI agent memory systems, directly addressing why agents often produce confidently wrong answers in real-world use. Called agent-memory-bench, the tool scores four specific failure categoriesâcollision, retraction, conflict, and recallâthat cause an agent to retrieve stale data, attribute facts to wrong entities, or contradict itself. Unlike typical evaluations that only measure retrieval quality, this benchmark runs completely offline with no API keys, ensuring reproducible results with a single command.
The benchmark's creator argues that current evaluation methods are dangerously shallow. Most teams test memory by checking if a relevant chunk was fetched, but agents rarely fail because of a missed retrieval. Instead, they fail when the retrieved fact is outdated, belongs to another entity, or conflicts with other stored information. agent-memory-bench directly scores these four bugs through 13 ordered scenarios across four categories, with each query specifying both required answer substrings and forbidden stale substrings.
Three reference baselines are included: keyword, recency, and typed-constraint. The results reveal stark differences: keyword retrieval achieves 0% on retraction and conflict due to its lack of time awareness, while recency collapses on collision and recall by drifting to recent look-alikes. The typed-constraint baseline, which models time and entity identity, scores 92% overall but still fails on one deliberate multi-hop recall scenarioâensuring the benchmark is not saturated.
The critical insight is that retrieval-quality metrics would rate all three systems similarly, while their actual answer correctness ranges from 23% to 92%. This gap, the creator emphasizes, is the whole point of the benchmark. It demonstrates that agents can appear to have good memory by traditional measures yet deliver wrong answers most of the time.
The developer notes that the benchmark is designed for easy integration: any memory systemâembeddings, hosted products, or LLM-backed extractorsâcan plug in via a simple interface of remember and query methods. Users can run the full suite with npm run bench or test individual scenarios to see exactly where their system fails. An adversarial test suite pins the scoring core and baselines, preventing regressions.
Looking ahead, version 0.1 covers four categories with 13 scenarios, but the roadmap includes broadening each category with more distractor-heavy scenarios, adding temporal and preference-drift categories, and introducing an optional LLM-judge mode for free-form answers. The project also invites external contributions, especially adversarial scenarios that break the typed-constraint baseline, which the creator considers the most valuable additions.
By shifting focus from retrieval accuracy to factual correctness, agent-memory-bench provides a more realistic evaluation of agent reliability. Its reproducibility and zero-dependency design make it accessible for both researchers and practitioners to diagnose the precise failure modes of their memory systems before deployment.
