BrepLLM: Enabling Large Language Models to Understand Boundary Representations

Current token-sequence-based Large Language Models (LLMs)struggle to directly process 3D Boundary Representation (B-rep) mod-els that contain complex geometric and topological information. To thisend, we propose BrepLLM, the first multimodal framework that enablesLLMs to directly parse and reason over raw B-rep data. BrepLLM adoptsa two-stage training pipeline: cross-modal alignment pre-training andtwo-stage LLM fine-tuning. In the first stage, we design an adaptiveUV sampling strategy to convert B-reps into graph representations thatintegrate geometric and topological information. Subsequently, we con-struct a hierarchical BrepEncoder to extract features from geometricelements (faces and edges) and topology, generating a global token anda sequence of node tokens. Then, via contrastive learning, we conduct aninitial alignment between this global token and the text embeddings of afrozen CLIP text encoder (ViT-L/14). In the second stage, we integratethe pre-trained BrepEncoder into the LLM and employ a two-stage pro-gressive strategy to align the sequence of node tokens: (1) training anMLP-based semantic mapping network that utilizes the prior knowledgeof a 2D-VLM to align the B-rep representation to the 2D visual semanticspace; (2) utilizing LoRA for parameter-efficient fine-tuning of the Q-Former and the LLM backbone network to achieve the final 3D-languagegeneration capability. Furthermore, we construct the Brep2Text dataset,which contains 269,444 B-rep and text question-answer pairs. Experi-ments demonstrate that BrepLLM achieves SOTA performance on 3Dobject classification and captioning tasks.The project page is availableat https://user-deng.github.io/BrepLLM/.