Spatiotemporal reasoning
Track object trajectories, speed, and direction from short high-FPS frame sequences.
NeurIPS 2026 ยท Evaluations and Datasets
AtariBench
A benchmark for testing whether multimodal LLMs can ground actions in fast Atari gameplay.
- -
- model settings
- -
- Atari games
Leaderboard
Model Ranking
Benchmark loop
Frame, Prompt, Act, Repeat
ObserveAtari frame sequence
Promptstructured history or append-only context
Actmodel emits joystick actions
Scorereward, frames, turns, tokens, video
What AtariBench tests
From Video Understanding to Control
Models observe frame sequences, receive game-specific prompts, emit joystick actions, and are scored from closed-loop trajectories across 20 Atari games.
Sequential decisions
Convert the current visual state into the next joystick action under tight feedback loops.
Visual grounding
Update actions from newly appended frames instead of relying on stale text history.
Observed failure modes: Models miss motion cues that are obvious to humans.
Check our paper additional diagnostic examples and failure-mode breakdowns.
Motion direction can be wrong even when objects are visible.
In Breakout, the model identifies the ball and paddle but infers a direction that contradicts the ground-truth trajectory.
Gameplay
Representative Runs
Atari 2600 games are the property of their respective copyright holders. Gameplay clips are shown for research illustration only.
Citation
BibTeX
@inproceedings{huo2026ataribench,
title = {Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay},
author = {Huo, Mingjia and Fu, Yao and Chang, Bo and Wang, Yaqing and Zhu, Dawei and Zhang, Dylan and Yi, Xinyang and Hong, Lichan and Chi, Ed H. and Zhang, Hao and Tang, Jiaxi},
booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
year = {2026}
}