When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

Vision Transformers (ViTs) are strong backbones for seman-tic segmentation, but their computational cost limits deployment. Recenttoken compression methods for efficient transformer-based segmentationreduce this cost by decreasing the number of tokens. However, exist-ing evaluations primarily focus on low-to-moderate compression, leavingtheir behavior under aggressive compression and corrupted inputs un-clear. Meanwhile, structural pruning provides an orthogonal route toefficiency by removing redundant components in the ViT architecture,but is rarely compared to token compression under a unified protocol.To bridge this gap, we benchmark representative token compression andstructural pruning methods for ViT-based semantic segmentation un-der matched FLOPs on ADE20K and Cityscapes, together with theircommon-corruption variants ADE20K-C and Cityscapes-C. Our resultsreveal a consistent trend on both clean and corrupted inputs: token com-pression is highly effective at mild reductions but degrades sharply whencompression becomes severe, consistent with substantial information lossfrom overly aggressive token reduction. In contrast, structural prun-ing exhibits a smoother degradation curve and is more stable at highcompression. Motivated by these findings, we study a prune-then-mergepipeline that applies moderate token compression on top of a moder-ately pruned backbone. At comparable FLOPs, this combined strategyconsistently achieves a better accuracy-robustness trade-off at high com-pression, offering a practical recipe for deployment-oriented ViT segmen-tation. Code is available at https://github.com/phatnguyencs/vit-seg-compression.