Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models

Recent advancements in 3D generation have significantly en-hanced the fidelity and geometric details of synthesized 3D assets. How-ever, due to the inherent ambiguity of single-view input and the lackof robust global structural priors–caused by limited 3D training data–the unseen regions generated by existing models remain stochastic anddifficult to control. This often results in geometries that are either phys-ically implausible or misaligned with user intent. In this paper, we pro-pose Know3D, a novel framework designed to incorporate rich knowl-edge from Multimodal Large Language Models (MLLMs) into 3D gen-eration processes. By leveraging latent hidden-state injection, Know3Dsupports language-controllable generation of the back-view for 3D as-sets. We utilize a VLM-diffusion-based architecture: the Vision LanguageModel (VLM) is used to provide high-level semantic understanding, whilethe diffusion model serves as a bridge, transferring semantic knowledgeinto the 3D generation model. Extensive experiments demonstrate thatKnow3D effectively bridges the gap between abstract textual instruc-tions and the geometric reconstruction of invisible regions. By trans-forming the traditionally stochastic back-view hallucination into a se-mantically controllable process, Know3D offers a promising direction forhighly plausible and user-friendly 3D generation in the future. Projectpage: https://xishuxishu.github.io/Know3D.github.io/.