Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generation

Personalized Text-to-Image (PT2I) generation aims to pro-duce customized images based on reference images. A prominent interestpertains to the integration of an image prompt adapter to facilitate zero-shot PT2I without test-time fine-tuning. However, current methods grap-ple with three fundamental challenges: 1. the elusive equilibrium betweenConcept Preservation (CP) and Prompt Following (PF), 2. the difficultyin retaining fine-grained concept details in reference images, and 3. therestricted scalability to extend to multi-subject personalization. To tacklethese challenges, we present Dynamic Image Prompt Adapter (DynaIP),a cutting-edge plugin to enhance the fine-grained concept fidelity, CP·PFbalance, and subject scalability of state-of-the-art T2I multimodal diffu-sion transformers (MM-DiT) for PT2I generation. Our key finding is thatMM-DiT inherently exhibit decoupling learning behavior when injectingZ. Wang and T. Chu—Equal contribution. ∗ Corresponding author.reference image features into its dual branches via cross attentions. Basedon this, we design an innovative Dynamic Decoupling Strategy that re-moves the interference of concept-agnostic information during inference,significantly enhancing the CP·PF balance and further bolstering thescalability of multi-subject compositions. Moreover, we identify the vi-sual encoder as a key factor affecting fine-grained CP and reveal thatthe hierarchical features of commonly used CLIP can capture visual in-formation at diverse granularity levels. Therefore, we introduce a novelHierarchical Mixture-of-Experts Feature Fusion Module to fully leveragethe hierarchical features of CLIP, remarkably elevating the fine-grainedconcept fidelity while also providing flexible control of visual granularity.Extensive experiments across single- and multi-subject PT2I tasks verifythat our DynaIP outperforms existing approaches, while requiring onlysingle-subject training datasets.