How Far Are Video Models from True Multimodal Reasoning?

Despite remarkable progress toward general-purpose videomodels, a critical question remains unanswered: how far are these modelsfrom achieving true multimodal reasoning? Existing benchmarks fail toaddress this question rigorously, as they remain constrained by straight-forward task designs and fragmented evaluation metrics that neglectcomplex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models’ zero-shot reasoning capabilities via Context Learning in Video Generation.CLVG-Bench comprises more than 1,000 high-quality, manually anno-tated metadata across 6 categories and 47 subcategories, covering com-plex scenarios including physical simulation, logical reasoning, and inter-active contexts. To enable rigorous and scalable assessment, we furtherpropose an Adaptive Video Evaluator (AVE) that aligns with human ex-pert perception using minimal annotations, delivering interpretable tex-tual feedback across diverse video context tasks. Extensive experimentsreveal a striking answer to our central question: while state-of-the-art(SOTA) video models, such as Seedance 2.0, demonstrate competenceon certain understanding and reasoning subtasks, they fall substantiallyshort with logically grounded and interactive generation tasks (achiev-ing success rates < 25% and ∼ 0%, respectively), exposing multimodalreasoning and physical grounding as critical bottlenecks. By systemati-cally quantifying these limitations, the proposed method provides action-able feedbacks and a clear roadmap toward truly robust, general-purposevideo models. CLVG-Bench and code are released at here.