CoT-PL: Chain-of-Thought Pseudo-Labeling for Open-Vocabulary Object Detection

Open-vocabulary object detection (OVD) aims to recognizeand localize object categories beyond the training set. Recent approachesleverage vision-language models to generate pseudo-labels using image-text alignment, allowing detectors to generalize to unseen classes withoutexplicit supervision. However, these methods depend heavily on single-step image-text matching, neglecting the intermediate reasoning stepscrucial for interpreting semantically complex visual contexts, such ascrowding or occlusion. In this paper, we introduce MSPL, a frameworkthat incorporates multi-step visual reasoning into the pseudo-labelingprocess for OVD. It decomposes complex scene understanding into threeinterpretable steps—object localization, category recognition, and back-ground grounding—where these intermediate reasoning states serve asrich supervision sources. Extensive experiments on standard OVD evalu-ation protocols demonstrate that MSPL achieves state-of-the-art perfor-mance with superior pseudo-labeling efficiency, outperforming the strongHuman Pseudo PseudoVLMs MLLMAnnotation Annotation AnnotationImage-Text RPN🔥 A boy sits near a dog Visual SAMMatching while watching television. ReasoningImage-Captions Input ImageSingle-step Assignment Multi-step Assignment🔥 🔥 OV Noisy PLs 🔥 OV Accurate PLsBase Detector Novel Large-Scale Novel Require captions Open-Vocab. Novel No captionsDetector DetectorClasses Classes Base Classes Classes BG collapse Base Classes Classes BG grounding(a) Manual Pseudo-Labeling Designs (b) Single-Step OV Pseudo-Labeling Methods (c) Our Method: Multi-Step OV Pseudo-LabelingFig. 2: (a) Manual pseudo-labels for novel classes is costly and does not scale. (b) Re-cent approaches automate this process via single-step semantic assignment with vision-language models and image captions, struggling in complex scenes. (c) Our caption-freemethod leverages multi-step reasoning to interpret semantically complex scenes.baseline by 9.4 AP50 for novel classes on OV-COCO and improving boxand mask APr by 3.2 and 2.2, respectively, on OV-LVIS.