SegVGGT: Joint 3D Reconstruction and Instance Segmentation from Multi-View Images

3D instance segmentation methods typically rely on high-quality point clouds or posed RGB-D scans, requiring complex multi-stage processing pipelines, and are highly sensitive to reconstructionnoise. While recent feed-forward transformers have revolutionized multi-view 3D reconstruction, they remain decoupled from high-level semanticunderstanding. In this work, we present SegVGGT, a unified end-to-endframework that simultaneously performs feed-forward 3D reconstructionand instance segmentation directly from multi-view RGB images. Byintroducing object queries that interact with multi-level geometric fea-tures, our method deeply integrates instance identification into the visualgeometry grounded transformer. To address the severe attention disper-sion problem caused by the massive number of global image tokens, wepropose the Frame-level Attention Distribution Alignment (FADA) strat-egy. FADA explicitly guides object queries to attend to instance-relevantframes during training, providing structured supervision without extrainference overhead. Extensive experiments demonstrate that SegVGGTachieves the state-of-the-art performance on ScanNetv2 and ScanNet200,outperforming both recent joint models and RGB-D-based approaches,while showing favorable generalization to ScanNet++. The code is avail-able at https://github.com/IDEA-Research/SegVGGT.