CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates

Vision Language Models (VLMs) have shown promising plan-ning capabilities, yet their success remains confined to the text domain,leaving visual decision-making relatively underexplored. Addressing thisgap, we introduce Corrective Sequence Planning (CoSPlan) bench-mark, where VLMs must plan a sequence of visual actions from an initialscene to a target scene. CoSPlan evaluates models on their ability to imag-ine and execute a coherent set of visual steps required to reach the goal(Step Completion). To prevent any shortcuts that simply describethe final scene, we introduce an erroneous action in decision making,which must be detected (Error Detection) and corrected to reach thegoal, enabling a deeper understanding of the task. CoSPlan spans across4 tasks: maze navigation, block re-arrangement, image reconstruction,and object re-organization. Despite using advanced reasoning strategiessuch as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan,while still showing promising performance in the text domain. Addressingthis, we propose Scene Graph Incremental updates (SGI), a noveltraining-free method to transform images into ‘textual’ scene graphs, en-abling step-by-step reasoning through iterative scene graph refinement.SGI yields an average of ≃ 4.4% ↑ on CoSPlan w/ generalization onPlanBench and VQA. Link for solving puzzles on the project page.