Tempo-SAM3D: Monocular Video to 4D via Temporal Memory-Guided Generation

Generating temporally coherent 4D content from monocularvideo is a challenging yet practically important problem. Recent advancesin layout-aware 3D generation have enabled producing objects with theirspatial arrangements from a single image, naturally providing scene-levelunderstanding that is valuable for 4D generation. However, applyingsuch single-image models independently to each video frame yields severetemporal inconsistencies—flickering textures, structural discontinuities,and jittering poses. We present Tempo-SAM3D, a training-free frame-work that extends layout-aware 3D generation to coherent 4D generationfrom monocular video. Our approach introduces Dual-Path TemporalAttention, which maintains importance-driven KV caches in both cross-attention and self-attention to propagate visual memory and structuralmemory across frames, respectively. We further incorporate gradient-guided velocity correction that constrains the flow matching samplingtrajectory, and HexPlane-based temporal smoothing that regularizes la-tent representations via low-rank spatiotemporal decomposition. Tempo-SAM3D requires no model retraining and inherits layout awareness forspatially-grounded 4D generation. Experiments demonstrate clear im-provements over per-frame baselines and existing video-to-4D methods.