ZAP: Zero-Shot Assembly Planning with Large Language Models

Achieving general-purpose robotic assembly is a longstand-ing goal in AI. Early data-driven methods, particularly those based onReinforcement Learning (RL), made progress but are fundamentally lim-ited by poor sample efficiency and struggle to generalize to unseen ob-jects in zero-shot scenarios. To overcome these issues, more structuredapproaches have emerged. Classical planners like ASAP explicitly enforcegeometric feasibility but require precise, pre-existing geometric models,which can hinder deployment under real-world perception noise. Con-versely, recent Vision-Language Model (VLM)-based systems like Man-ual2Skill excel at semantic understanding but are critically dependenton pre-authored manuals, restricting their autonomy in unstructuredenvironments. To address both planning-time model dependency andmanual dependency, we introduce ZAP, a novel framework for zero-shotVLM-guided assembly planning with geometry-based simulation verifi-cation. ZAP constructs an internal structural representation from per-ception without relying on task-specific manuals, pre-authored assemblysequences, or hard-coded assembly logic. Our framework features twocore modules: a VLM-Parser that analyzes segmentation-based multi-view RGB-D observations to infer an “implicit manual” and compile itinto structured part dossiers, and a VLM-Planner that performs Chain-of-Thought (CoT) reasoning over these dossiers to produce physicallyplausible, robot-executable assembly sequences. We validate candidateplans using a geometry-based digital-twin verifier and further exam-ine representative executions on a robotic arm. Our experiments showthat ZAP remains effective in manual-free scenarios where methods likeManual2Skill are less directly applicable. Furthermore, by reasoning di-rectly from visual perception, ZAP can be more flexible than geometry-dependent classical planners like ASAP in the evaluated settings. Theframework generates and executes plans for complex items such as multi-part LEGO models and household furniture in our evaluation setting,providing evidence toward more autonomous assembly without task-specific manuals.