DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

Multi-Modal LLMs (MLLMs) demonstrate strong visualgrounding capabilities on popular object detection benchmarks likeOdinW-13 and RefCOCO. However, state-of-the-art models still struggleto generalize to out-of-distribution classes, tasks and imaging modalitiesnot typically found in their pre-training. While in-context promptingis a common strategy to improve performance across diverse tasks, wefind that it often yields lower detection accuracy than prompting withclass names alone. This suggests that current MLLMs cannot yet effec-tively leverage few-shot visual examples and rich textual descriptionsfor object detection. Since frontier MLLMs are typically only accessi-ble via APIs, and state-of-the-art open-weights models are prohibitivelyexpensive to fine-tune on consumer-grade hardware, we instead exploreblack-box prompt optimization for few-shot object detection. To this end,we propose Detection Prompt Optimization (DetPO), a gradient-freetest-time optimization approach that refines text-only prompts by max-imizing detection accuracy on few-shot visual training examples whilecalibrating prediction confidence. Our proposed approach yields consistentimprovements across generalist MLLMs on Roboflow20-VL and LVIS,outperforming prior black-box approaches by up to 9.7 mAP. Our codeand optimized prompts are available on our project page.