BLOB-Q: Boosting Low Bit ViT Quantization via Global Optimization on Model Distortion

In this paper, we present BLOB-Q, a novel Mixed-PrecisionPost Training Quantization (MPQ) approach for Vision Transformers(ViTs). MPQ aims to assign different bit-widths across layers under amodel size budget. Most existing MPQ methods either rely on layerwisesensitivity heuristics or surrogate objectives. These approaches simplifyoptimization, but they no longer optimize the same model-level objec-tive that determines accuracy. Others solve the global MPQ problemby non-heuristic and non-analytical algorithms, such as ReinforcementLearning, genetic algorithms or gradient-based learning. However, theyare inefficient for modern large vision transformers. In this work, we revisitthe global MPQ problem and show that this problem can be solved byefficient analytical algorithms while still maintaining its global-optimality.This comes from two key insights. Firstly, through systematic signalanalysis on quantization errors, we discover that under 4„6-bit range,the quantization error on converged ViTs are indeed small perturbations.This encourages that the global MPQ objective is possible to be quadrat-ically decomposed for empirical ViT models. Secondly, motivated by thesmall perturbation observation, we further discover that the global MPQobjective satisfies an additivity property. Utilizing the additivity property,the NP-hard global MPQ optimization problem can be decomposed intosub-problems and solved practically, while still retain the global optimal-ity. Specifically, we solve the decomposed analytical optimization problemusing dynamic programming algorithm, which efficiently find the globallyoptimal solution with only linear time complexity. Extensive experimentson numerous ViT models demonstrate the effectiveness of our approach.Results show that BLOB-Q significantly improves state-of-the-art andcan further reduce the size of ViT models to 4 bits to 6 bits withouthurting ImageNet accuracy. Moreover, BLOB-Q is highly efficient, onlyrequiring on average sub-2 minutes for regular ViT-S to ViT-B models.