Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval

Composed Image Retrieval (CIR), a task towards personalizing visual intelligence, aims to retrieve a target image from a database based on users’ multimodal query, which contains a reference image and a modification text. The text specifies how to alter the reference image to form a “mental image”, based on which CIR should find the target image in the database. The fundamental challenge of CIR is that this “mental image” is not physically available and is only implicitly defined by the query. The contemporary literature pursues zero-shot methods and uses a Large Multimodal Model (LMM) to generate a textual description for a given multimodal query, and then employs a Vision-Language Model (VLM) for textual-visual matching to search for the target image. In contrast, we address CIR from first principles by directly generating the “mental image” for more accurate matching. Particularly, we prompt an LMM to generate a “mental image” for a given multimodal query and propose to use this “mental image” to search for the target image. As the “mental image” has a synthetic-to-real domain gap with real images, we also generate a synthetic counterpart for each real image in the database to facilitate matching. In this sense, our method uses LMM to construct a “paracosm”, where it matches the multimodal query and database images. Hence, we call this method Paracosm. Notably, Paracosm is a training-free zero-shot CIR method. It significantly outperforms existing zero-shot methods on challenging benchmarks, achieving state-of-the-art performance for zero-shot CIR.