Holo-Captioning: A Comprehensive Textual View of 3D Scenes

This work introduces holo-captioning, a novel task that strivesto seek the text equivalent of 3D scenes. As the initial step, we formu-late holo-captioning as generating a structured textual description thatcomprehensively depicts all entities within a 3D scene—including theirsemantic tags, spatial locations, attributes, and inter-entity relations. Totackle this challenging task, we first develop an effective captioning en-gine to produce detailed descriptions of individual entity instances andinstance pairs, and contribute a large-scale benchmark comprising over15K scenes for training and evaluation. Building upon this foundation,we propose HoloScribe, a novel model that features an instance-awaredecoupled pipeline for generating structured holo-captions, and furtherincorporates anchor-aware instance linking to identify relational instancepairs. Additionally, we propose a comprehensive evaluation metric namedHoloScore, and provide a human-curated test set to ensure reliable modelassessment. Experimental results demonstrate that HoloScribe signifi-cantly outperforms state-of-the-art 3D dense captioners and 3D LLMgeneralists, underscoring the effectiveness of our approach. Project page:https://visual-ai.github.io/holocap/