Stabilizing Ultra-Low-Bit Quantization of Multimodal LLMs via Global Bit Allocation

Weight-only post-training quantization (PTQ) has proven highly effective for deploying large language models under memory-bound inference constraints. However, extending PTQ to Multimodal LLMs at ultra-low bit-widths (2–3 bits) presents a significant challenge. Existing uniform-precision methods suffer severe accuracy collapse due to their implicit assumption that all output channels within a layer are equally sensitive to quantization noise. This uniform bit-width assignment fails to account for the heterogeneous sensitivity of different channels. By lifting the analysis to the Transformer block level, we reveal a two-level anisotropy—inter-layer and intra-layer inter-channel. This anisotropy is further amplified by an outlier–sensitivity resonance mechanism, causing different output channels to contribute vastly different amounts of quantization error to the block output. Together, these findings provide the rigorous foundation for per-output-channel mixed-precision quantization. Building on the derived optimality condition, we propose GloBitQ, a framework that unfolds the anisotropic loss into principled bit allocation. It integrates three intermediate proxies: power-law balanced layer weighting, rank–value hybrid channel scoring, and global top-τ bit assignment. The resulting allocation provably approximates the optimum of the block-level quantization objective. On six multimodal benchmarks spanning Qwen2-VL-7B, Qwen2.5-VL-7B, and LLaVA-OneVision, GloBitQ consistently surpasses GPTQ, GPTAQ, and VLMQ: the 3-bit model performs within 2.1% of full precision(81.70 vs. 83.76), while the 2-bit model remains stable where prior methods collapse—all within a calibration-only, finetuning-free, hardware-compatible pipeline. Code is available in the supplementary material.