World Models in the Loop: Towards Application-Driven World Model Evaluation

The next challenge for intelligent systems is not only perception but also anticipating and acting in physical space under interventions and uncertainty. World models support simulation, data generation, and planning by rolling out futures conditioned on observations and actions, in applications such as robot manipulation, autonomous driving, and embodied navigation.

Yet recent rapid progress in world models has outpaced evaluation: beyond visual fidelity, we must assess controllability, physical plausibility, and robustness to distribution shift. As world models are getting embedded in larger systems and are effectively used in-the-loop for planning or training and evaluating perception and control policies, open-loop benchmarks provide limited insight. In such settings, errors compound, and missing controllability, physical plausibility, or robustness under distribution shift directly translate into inaccurate rollouts, suboptimal decisions, degraded downstream performance, or potentially catastrophic errors in safety-critical applications. Such properties are task-dependent and, thus, require task-specific evaluation criteria.