EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents

Abstract

We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games. Rather than using a fixed set of questions, EMemBench generates questions from environment-grounded trajectories, covering both text-only and visual game environments. Each template computes verifiable ground truth from underlying game signals, with controlled answerability and balanced coverage over memory skills: single/multi-hop recall, induction, temporal, spatial, logical, and adversarial. We evaluate memory agents with strong LMs/VLMs as backbones, using in-context prompting as baselines. Across 15 text games and multiple visual seeds, results are far from saturated: induction and spatial reasoning are persistent bottlenecks, especially in visual settings. Persistent memory yields clear gains for open backbones on text games, but improvements are less consistent for VLM agents, suggesting that visually grounded episodic memory remains an open challenge. A human study further contextualizes the difficulty and interpretability of EMemBench.

Author abstract · arXiv abstract · v2

Publication

Findings of Empirical Methods in Natural Language Processing (Findings of EMNLP), 2026

Paper and resources

• Paper · arXiv v2 (PDF)
• arXiv:2601.16690
• Google Scholar
• Code · InternLM/EMemBench

Research topics

EMemBench · Episodic memory · Interactive evaluation · VLM agents · Long-term memory · Memory induction · Trajectory-grounded questions · Spatial memory

Research problem and approach

A fixed question set cannot fully probe interactive episodic memory. EMemBench generates verifiable questions from game trajectories across text-only and visual environments. arXiv abstract · v2

Main contributions

• Controls answerability and coverage across recall, induction, temporal, spatial, logical and adversarial memory skills. arXiv abstract · v2
• Compares persistent-memory agents with in-context baselines and includes a human study. arXiv abstract · v2

Method comparison

arXiv abstract · v2

Selected results

• For text-based Qwen2.5-32B, A-MEM raises overall accuracy from the in-context baseline of 40.8 to 51.4 and F1 from 39.9 to 48.1. For visual GPT-5.1, accuracy instead changes from 43.8 to 42.1 and F1 from 39.0 to 35.5, showing that external memory is not uniformly beneficial. Table 3 · episodic-memory evaluation · arXiv v2
• Human open-book accuracy is 65.6, versus 23.2 in the closed-book condition; corresponding F1 scores are 63.8 and 19.8. These results quantify the effect of access to past experience under the benchmark protocol. Table 4 · human reference conditions · arXiv v2

Cite this paper

Xinze Li, Ziyue Zhu, Siyuan Liu, Yubo Ma, Yuhang Zang, Yixin Cao, Aixin Sun. EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents. arXiv preprint arXiv:2601.16690, 2026.

@article{arxiv260116690,
  title     = {{EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents}},
  author    = {Xinze Li and Ziyue Zhu and Siyuan Liu and Yubo Ma and Yuhang Zang and Yixin Cao and Aixin Sun},
  journal   = {arXiv preprint arXiv:2601.16690},
  year      = {2026},
  url       = {https://arxiv.org/abs/2601.16690}
}