Auto-Prompting: Layer-Specific Prompt Fusion Discovery via Differentiable Search

Visual prompt tuning has emerged as a parameter-e!cientfine-tuning approach for adapting large-scale Vision Transformers (ViTs)to downstream tasks. As its learnable prompts are applied in input andfeature spaces, prior to jointly going through attention in transformerlayers, the most commonly used scheme for fusing image and prompttokens is concatenation or addition. In this paper, we aim to study afundamental yet essential problem in visual prompt tuning: whethera single fusion scheme tends to yield better results, and whether thatwould be beneficial to develop a hybrid fusion scheme. To this end,we formulate the task as a bi-level optimization problem, and solve itleveraging di"erentiable architecture search. In this context, the learnableprompts and their fusion schemes are jointly optimized. To enrich thesearch space in the architecture search, we propose two additional fusionschemes, namely, a!ne transformation and cross-attention, in additionto concatenation and addition. Extensive experiments on 34 datasetsspanning VTAB-1k, FGVC, and HTA show consistent gains over prompt-tuning baselines. With a frozen ViT backbone, our method delivers afavorable accuracy–latency–parameter trade-o" compared with VPT-Deep and recent variants. Our findings reveal that how prompts fusewith image tokens plays a significant role in visual prompt tuning, anda hybrid fusion fashion can more e"ectively leverage layer semantics ofViTs, contributing a novel perspective for visual prompt-tuning research.