InstanceControl: Controllable Complex Image Generation without Instance Labeling

Controllable image generation methods, such as ControlNet,have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methodsoften struggle with complex multi-instance scenes, frequently leading toattribute confusion among instances. While recent approaches attemptto mitigate this via manual instance labeling, such requirements arelabor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for in-stance labeling. We identify the primary bottleneck in existing meth-ods as the inability to accurately associate instance descriptions withtheir corresponding regions within visual conditions. To address this, weleverage the Vision-Language Model (VLM) to establish instance-levelcorrespondences between text prompts and visual conditions. Specifi-cally, the VLM automatically parses instance descriptions from the textprompts and simultaneously predicts instance masks based on the visualconditions. Furthermore, since the predicted masks may contain noise,we introduce an adaptive mask refinement strategy that dynamically re-fines these instance masks during the generation process. Extensive ex-periments demonstrate that our approach outperforms state-of-the-artmethods, achieving superior fidelity and precise instance-level control.The code and dataset will be made publicly available.