Zero-Shot Novel Depth Synthesis Using Foundation Models Scene Representations

3D Foundation Models (3DFMs) such as VGGT have re-cently pushed the boundaries of 3D vision by predicting rich unifiedrepresentations with feed-foward transformers. The scene representationslearned by these models enable strong performance on multiple 3D visiontasks. In this paper, we investigate using their internal representationsto infer 3D in the scene from new views. Our hypothesis is that in or-der to solve the task of 3D reconstruction, these models need to learn arepresentation that includes a large amount of general knowledge about3D scenes. After showing that it is possible to decode hidden surfacesfrom internal 3DFM representations, we propose a method, Z3D, thatestimates pointmaps in unseen views by doing latent diffusion on 3DFMrepresentation. We show that Z3D can predict realistic depthmaps fornew views across multiple datasets.