CL4D: Contrastive Language–4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

4D understanding and reasoning is a fundamental capabil-ity for embodied AI agents operating in dynamic physical environments.However, existing vision encoders are largely limited to static 2D im-ages or 3D point clouds without temporal modeling, or to 2D videosthat lack accurate geometric depth reasoning. Consequently, current ap-proaches fail to jointly capture spatial structure and motion evolutionin dynamic scenes. We present CL4D, the first foundational 4D visionencoder that directly operates on dynamic point clouds, trained witha contrastive learning objective to align spatio-temporal geometric rep-resentations with natural language descriptions. By learning a shared1 ∗These authors contributed equally. † These authors jointly supervised this work.embedding space between text and 4D scene dynamics, CL4D enableszero-shot motion-to-text and text-to-motion retrieval in dynamic en-vironments and serves as a foundational 4D vision encoder for down-stream 4D vision–language tasks. Building on this encoder, we intro-duce 4DVLM, a 4D vision–language model that conditions languagegeneration on dynamic geometric representations. 4DVLM is the firstVLM designed to operate directly on 4D point clouds without relyingon 2D images, 2D videos, or static 3D point clouds. We train CL4D andsubsequently 4DVLM on a newly constructed dataset termed DynAc-tion4D capturing diverse human motions across varying object inter-actions and scene environments. Extensive experiments across multiple4D human action benchmarks demonstrate that CL4D achieves state-of-the-art performance, with improvements of approximately ∼16.75% overprior methods. Furthermore, 4DVLM outperforms frontier video VLMssuch as Gemini and GPT-5 even when these models are provided withRGB video sequences corresponding to the same scenes represented as4D point clouds for 4DVLM.