EMNLP · 2026
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
Key takeaway
RNG-Bench separates memory failures from action-selection errors in interactive visual games, finding that forgotten observations account for most residual errors in the evaluated models. arXiv abstract · v1
Abstract
Deploying multimodal foundation models as closed-loop policies increasingly requires conditioning actions on observations that are no longer visible. However, existing benchmarks either expose the full state, conflate hidden-state reconstruction with other agent skills, or test recall only after an episode has ended. We introduce RNG-Bench (Reconstructive Non-Markov Games), a benchmark suite designed to isolate a base model's ability to reconstruct past observations and act on them during multi-step interaction. RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map. Both games are evaluated under a unified harness with three controlled difficulty axes: grid size, visual pattern, and observation modality. The benchmark further introduces a head-to-head duel protocol to control for instance-level variance and a Memory Gap metric that disentangles forgetting from poor action selection. The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs. Memory Gap analysis shows that most residual errors stem from forgetting earlier observations rather than from suboptimal decision making. Finally, fine-tuning Qwen3.5-9B on optimal-policy rollouts and filtered model demonstrations improves performance on RNG-Bench and transfers to existing benchmarks without degrading general multimodal capability.
Author abstract · arXiv abstract · v1
Publication
Empirical Methods in Natural Language Processing (EMNLP), 2026
Paper and resources
Research topics
RNG-Bench · Non-Markov games · Multimodal agents · Visual memory · Hidden-state reconstruction · Memory Gap · Closed-loop interaction · Spatial mapping · Long-context evaluation
Research problem and approach
Closed-loop agents must act on observations that are no longer visible. RNG-Bench isolates this requirement using Matching Pairs and egocentric 3D Maze games with controlled difficulty and access to past observations. arXiv abstract · v1
Main contributions
- Controls grid size, visual pattern and observation modality within one evaluation harness. arXiv abstract · v1
- Introduces head-to-head duels and the Memory Gap metric to distinguish forgetting from poor decisions. arXiv abstract · v1
Method comparison
| Approach | Key difference |
|---|---|
| Full-state or post-episode evaluation | Exposes the current state or tests recall after interaction, rather than isolating the use of hidden past observations. |
| RNG-Bench | Requires memory during closed-loop games and separates forgetting from action selection with Memory Gap. |
Selected results
- On 10×10 Matching Pairs, GPT-5.4 scores 62.3% and Gemini-3.1-Pro 50.0%; on the 13×13 3D Maze, their aggregate game scores are 30.5 and 49.7, respectively. The ranking changes with the game and metric, so the benchmark does not support a single model winning every hidden-state task. Table 2 · single-player evaluation · arXiv v1
- For Qwen3.5-397B on Matching Pairs, the score falls from 100% with text input to 75.8% with ASCII images and 38.3% with noisy images. This controlled comparison isolates a substantial visual-input bottleneck within the same game. Table 4 · input-modality ablation · arXiv v1
Cite this paper
Shengyuan Ding, Xilin Wei, Xinyu Fang, Haodong Duan, Dahua Lin, Jiaqi Wang, Yuhang Zang. Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games. arXiv preprint arXiv:2606.19338, 2026.
@article{arxiv260619338,
title = {{Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games}},
author = {Shengyuan Ding and Xilin Wei and Xinyu Fang and Haodong Duan and Dahua Lin and Jiaqi Wang and Yuhang Zang},
journal = {arXiv preprint arXiv:2606.19338},
year = {2026},
url = {https://arxiv.org/abs/2606.19338}
}