Group3D: MLLM-Guided Semantic Grouping for Open-Vocabulary 3D Object Detection

Open-vocabulary 3D object detection aims to localize andrecognize objects beyond a fixed training taxonomy. In multi-view RGBsettings, recent approaches often decouple geometry-based instance con-struction from semantic labeling, generating class-agnostic fragments andassigning open-vocabulary categories post hoc. While flexible, such de-coupling leaves instance construction governed primarily by geometricconsistency, without semantic constraints during merging. When geomet-ric evidence is view-dependent and incomplete, this geometry-only merg-ing can lead to irreversible association errors, including over-mergingof distinct objects or fragmentation of a single instance. We proposeGroup3D, a multi-view open-vocabulary 3D detection framework thatintegrates semantic constraints directly into the instance constructionprocess. Group3D maintains a scene-adaptive vocabulary derived from amultimodal large language model (MLLM) and organizes it into semanticcompatibility groups that encode plausible cross-view category equiva-lence. These groups act as merge-time constraints: 3D fragments are asso-ciated only when they satisfy both semantic compatibility and geometricconsistency. This semantically gated merging mitigates geometry-drivenover-merging while absorbing multi-view category variability. Group3Dsupports both pose-known and pose-free settings, relying only on RGBobservations. Experiments on ScanNet and ARKitScenes demonstratethat Group3D achieves state-of-the-art performance in multi-view open-vocabulary 3D detection, while exhibiting strong generalization in zero-shot scenarios. The project page is available at https://ubin108.github.io/Group3D/.