Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

Parameter-efficient fine-tuning (PEFT) has become a prac-tical solution for adapting large pretrained vision transformers (ViTs)to downstream tasks while updating only a small subset of parame-ters. However, existing adapter-based methods perform adaptation in-dependently for each token, implicitly assuming that token refinementsshould be learned in isolation. This token-wise formulation overlooksthe structured relationships among tokens that naturally arise in visualscenes, potentially leading to redundant updates and spatially inconsis-tent feature refinement. In this work, we revisit the design of parameter-efficient adapters and propose to perform adaptation in hyperedge spacerather than token space. We introduce HyperAdapter, a hypergraph-based adapter architecture that enables structured, group-aware adap-tation through soft token routing. HyperAdapter constructs a soft hy-pergraph over ViT tokens using prototype-based assignments, aggregatestoken features into latent hyperedge representations, applies lightweightbottleneck adaptation at the hyperedge level, and diffuses the result-ing updates back to tokens via the hypergraph incidence structure. Thisdesign injects an explicit structural inductive bias into PEFT while pre-serving the modularity and efficiency of standard adapters. Extensiveexperiments across diverse visual benchmarks demonstrate that struc-tured hyperedge adaptation consistently outperforms strong PEFT base-lines under comparable parameter budgets, with particularly pronouncedgains on tasks requiring structured reasoning. Our results suggest thatthe choice of adaptation space is a critical yet underexplored dimensionin parameter-efficient transfer for ViTs.