VQT: Vector Quantization Tuning for Efficient Fine-tuning and Compression of Pre-trained Vision Transformers

Pre-trained Vision Transformers (ViTs) have shown strongperformance across diverse tasks, creating increasing demand for efficientfine-tuning and compression for downstream deployment. Existing VectorQuantization (VQ) methods achieve high compression ratios, yet strug-gle to balance fine-tuning efficiency and downstream performance. In thiswork, we propose Vector Quantization Tuning (VQT), a novel frameworkfor efficient fine-tuning and compression of pre-trained ViTs. We iden-tify Coupled Vector Quantization Noise (CVQN) as a key factor limitingperformance. It arises from the interaction between Low-Rank Adapta-tion (LoRA) and the Straight-Through Estimator (STE), which degradesoptimization effectiveness under vector quantization. To address this,VQT employs a VQ-aware fine-tuning process that stochastically replacesvector-quantized weights with continuous counterparts to reduce CVQN,while progressively aligning the model toward fully vector-quantized rep-resentations to ensure consistency between fine-tuning and deployment.Furthermore, VQT enables efficient storage for multi-task deploymentby storing only lightweight task-specific parameters in the cloud, withless than 1 bit per parameter needed to reconstruct compressed models.Extensive experiments show the advantages of VQT over state-of-the-art baselines in performance and efficiency at high compression ratios.For example, at 1-bit compression, VQT surpasses the advanced baselineby 9.5% on the VTAB-1K benchmark while fine-tuning only 0.8% of theparameters. The code is available: https://github.com/leenas233/VQT.