GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks

Systematic failures of vision models on semantically coherentsubsets, known as error slices, reveal limitations in robustness and evalu-ation. Existing slice discovery approaches largely model slices as clustersin representation space or combinations of predefined attributes. Whileeffective for image-level classification, such formulations are insufficientfor instance-level tasks such as object detection and segmentation, wherefailures often arise from contextual, relational, and spatially groundedvisual patterns. We propose GH-ESD (Grounded Hypothesis-Driven ErrorSlice Discovery), a generate-and-verify framework that reformulates slicediscovery as grounded hypothesis generation and statistical verification.GH-ESD constructs relational failure hypotheses using LLM priors andgrounded visual evidence, discovers hypothesis slices at the instance levelvia Vision-Language Models, and verifies them through statistical trendanalysis over instance-level errors. We also introduce GESD (GroundedError Slice Dataset), a new benchmark for instance-level error slice discov-ery, providing expert-defined and spatially grounded slices derived fromdetection and segmentation failures. Extensive experiments demonstratethat GH-ESD consistently outperforms baselines, improving Precision@10by 0.10 (0.73 vs. 0.63) on the GESD benchmark for detection tasks, whilealso supporting segmentation scenarios. GH-ESD identifies interpretableslices that facilitate actionable model improvements.