Steerable Vision Transformers

Pretrained Vision Transformers (ViTs) such as DINOv2 andMAE provide generic image features that can be applied to a varietyof downstream tasks such as retrieval, classification, and segmentation.However, such representations tend to focus on the most salient visualcues in the image, with no way to direct them toward less prominentconcepts of interest. In contrast, Multimodal LLMs can be guided withtextual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To addressthis, we introduce Steerable Visual Representations, a new class of vi-sual representations, whose global and local features can be steered withnatural language. While most vision-language models (e.g., CLIP) fusetext with visual features after encoding (late fusion), we inject text di-rectly into the layers of the visual encoder (early fusion) via lightweightcross-attention. We introduce benchmarks for measuring representationalsteerability, and demonstrate that our steerable visual features can focuson any desired object in an image while preserving the underlying rep-resentation quality. Our method also matches or outperforms dedicatedapproaches on anomaly detection and personalized object discrimination,exhibiting zero-shot generalization to out-of-distribution tasks.Project Website: jonaruthardt.github.io/project/SteerViTPrompt CLS AttentionSteerable Visual RepresentationsGoal: control what vision features encode New Pareto FrontierEval: retrieving images w/ prompted objects Previous SoTAUse: task-specific adaptationFig. 2: SteerViT produces high-quality visual representations that can besteered by text. Left: Traditional (non-steerable) representations like DINOv2 tendto focus on the dominant object in an image and retrieve images with the same object.SteerViT can adapt to a text prompt, enabling retrieval of images even with smallobjects of interest. Right: We compare SteerViT to prior work in terms of its ability toadapt to text (measured by text-guided image retrieval (cf. Sec. 4.1)) and the qualityof the visual representation (measured by the accuracy of linear probing for the CLSfeature and semantic segmentation for patch features). While models typically tradeoff steerability for representation quality, SteerViT preserves both. By modulating agating factor (Eq. (2)), SteerViT achieves a new Pareto frontier.