SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

Measuring structured object understanding in vision founda-tion models remains challenging due to inconsistent evaluation protocolsand limited part-level supervision. Semantic correspondence (SC) eval-uates this capability by testing whether object parts can be matchedacross instances and categories under large variations in appearance,viewpoint, and geometry. To enable a systematic SC evaluation, we in-troduce SOCO, a new benchmark for Semantic Object Correspondencethat introduces a taxonomy of correspondence types and provides con-sistent, functionally meaningful keypoint annotations across 100 cate-gories and over 1M correspondence pairs. In addition, SOCO includeskeypoint language descriptions, enabling the evaluation of large vision-language models (LVLMs) and their fine-grained part-level understand-ing. Comprehensive experiments reveal that (i) vision foundation back-bones encode strong semantic structure but transfer correspondencespoorly across related categories and only partially capture object-partposition, (ii) LVLMs are stronger at text-prompted part localizationthan at visual-reference cross-image matching, exposing a gap betweenlanguage-grounded localization and fine-grained visual correspondence,and (iii) correspondence performance predicts dense downstream tasks—segmentation, tracking, 3D pose estimation, and 3D detection—morestrongly than ImageNet classification. Together, these findings positionSOCO as a benchmark for structured, part-level representation qualityin vision and multimodal foundation models.