Personalize Your Large Vision-language Models With In-context Prompt Tuning

Large vision-language models (LVLMs) have demonstratedstrong general multimodal capability and are increasingly deployed indownstream systems. This trend has driven growing interest in LVLMpersonalization, which aims to enable models to quickly and effectivelylearn out-of-distribution multimodal concepts to meet user-specific needs.However, many existing methods rely on inference-time training, whichreduces efficiency. They also struggle to maintain accuracy in complexmulti-image, multi-concept settings. These limitations restrict the broaderdeployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweightprojection module capable of operating in complex scenarios to extractfine-grained visual semantics from multiple reference images, seamlesslytransforming these features alongside identity-label mappings into con-tinuous prompts. To maximize computational efficiency, this moduleadaptively determines the prompt length based on the intrinsic visualcomplexity of each concept. Crucially, to overcome the environmental bi-ases and cross-concept interference prevalent in real-world applications,we introduce two novel geometric regularizations. These constraints re-fine prompt representations by decoupling key identities from transientenvironmental states and separating concepts to avoid semantic confu-sion. Extensive experiments show that ICPT achieves state-of-the-artpersonalization accuracy across diverse tasks and LVLM backbones.