Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents

Vision-Language Models (VLMs) exhibit strong visual rea-soning capabilities, yet they still struggle with 3D understanding. Inparticular, VLMs often fail to infer a text-consistent goal 6D pose of atarget object in a 3D scene. However, we find that with some inference-time techniques and iterative reasoning, VLMs can achieve dramatic per-formance gains. Concretely, given a 3D scene represented by an RGB-Dimage (or a compositional scene of 3D meshes) and a text instructionspecifying a desired state change, we repeat the following loop: observethe current scene; evaluate whether it is faithful to the instruction; pro-pose a pose update for the target object; apply the update; and ren-der the updated scene. Through this closed-loop interaction, the VLMeffectively acts as an agent. We further introduce three inference-timetechniques that are essential to this closed-loop process: (i) multi-viewreasoning with supporting view selection, (ii) object-centered coordinatesystem visualization, and (iii) single-axis rotation prediction. Withoutany additional fine-tuning or new modules, our approach surpasses priormethods at predicting the text-guided goal 6D pose of the target object.It works consistently across both closed-source and open-source VLMs.Moreover, when combining our 6D pose prediction with simple robot mo-tion planning, it enables more successful robot manipulation than recentVision-Language-Action models (VLAs). Finally, we conduct an ablationstudy to demonstrate the necessity of each proposed technique.