ScenarioControl: Vision-language Controllable Vectorized Latent Scenario Generation

We introduce ScenarioControl, the first vision-language con-trol mechanism for learned driving scenario generation. Given a textprompt or an input image, ScenarioControl synthesizes diverse, realistic3D scenario rollouts – including map, 3D boxes of reactive actors overtime, pedestrians, driving infrastructure, and ego camera observations.The method generates scenes in a vectorized latent space that representsroad structure and dynamic agents jointly. To connect multimodal con-trol with sparse vectorized scene elements, we propose a cross-global con-trol mechanism that integrates cross-attention with a lightweight global-context branch, enabling fine-grained control over road layout and traf-fic conditions while preserving realism. The method produces tempo-rally consistent scenario rollouts from the perspectives different actorsin the scene, supporting long-horizon continuation of driving scenarios.Extensive experiments validate that the control adherence and fidelityof ScenarioControl compare favorable to all tested methods across allexperiments. To facilitate training and evaluation, we release a datasetwith text annotations aligned to vectorized map structures at the projectwebpage: https://light.princeton.edu/ScenarioControl