SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence

True video intelligence demands more than recognizing whatis visible: it requires reasoning about why events unfold, predicting whatwould change under different conditions, and deciding what to do next.We refer to this full progression—from perception through causal reason-ing and simulation to strategic planning—as Strategic Video Intelligence(SVI). No existing benchmark evaluates this capability stack: in-the-wildvideos lack verifiable ground truth for causal and strategic questions,while synthetic environments sacrifice the complexity of real multi-agentsystems. To bridge this gap, we introduce SVI-Bench, a large-scale bench-mark that leverages team sports as a dynamic microworld, a domain thatuniquely combines the complexity of real-world multi-agent interactionwith the verifiability of explicit rules and definitive outcomes. SVI-Benchcomprises ∼35K hours of broadcast video, ∼15M annotated actions,∼15K hours of expert commentary, ∼23K game reports, and ∼103Kstructured statistical records across basketball, soccer, and hockey, allconstructed via a data engine that transforms raw game data into adense, cross-referenced corpus. We organize evaluation into 9 tasks span-ning a progressive four-pillar hierarchy: Dynamic Scene Understanding,Causal Reasoning, Strategic Simulation, and Agentic Synthesis. Evaluat-ing strong multimodal and agentic baselines, we find a capability cliff :models perform competently on perceptual tasks (achieving ∼74% onfine-grained action QA) but degrade sharply at higher levels of the stack.Agentic tasks prove hardest of all: the strongest model achieves only 5%accuracy when required to autonomously gather and integrate evidenceacross a corpus of 1.8M clips. We release the full benchmark to catalyzeprogress toward AI systems capable of strategic intelligence in complex,dynamic multi-agent environments.§ Code github.com/texaser/svi-benchõ Data huggingface.co/mvp-group/svi-bench€ Website svi-bench.github.ioA Extended Paper svi-bench.github.io/svibenchextended.pdfPillar 1: Perceptionparsing who is where, doing what and whenPillar 2: Reasoningexplaining why something happenedPillar 3: Simulationgenerating plausible alternative futuresPillar 4: Agencyautonomous analysis over multimodal evidenceThe Strategic Video Intelligence StackFig. 1: Overview of SVI-Bench, illustrated through a single play from the 2022NCAA Final Four. SVI-Bench is the first large-scale video benchmark evaluatingthe full SVI stack: Perception (describing what happens), Reasoning (explaining why),Simulation (generating plausible alternatives), and Agency (autonomous analysis).