MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding

Vision encoders are a critical component of vision-languagemodels, and scaling their capacity effectively improves performance. How-ever, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having en-abled efficient scaling in LLMs, yet the MoE design space for CLIP-stylevision encoders remains underexplored at State-of-the-Art (SOTA) lev-els. In this work, we systematically study MoE designs for vision encoderscaling and find that fine-grained MoE topologies yield substantial gainsover both dense and standard MoE counterparts. We further proposean auxiliary-loss-free balancing variant for better expert utilization, anddesign a specialized MoE kernel to mitigate inference latency overhead.To enhance video capabilities while preserving image knowledge, we in-troduce frame-level distillation paired with a novel freezing mechanism.We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE)across a range of sizes, all consistently outperforming their dense counter-parts. Our largest model matches the zero-shot performance of a SOTAencoder 1.7× its size at 76% of its latency. When aligned with an LLM,MoE-ViE surpasses all compared encoders on image and video bench-marks, including those with up to 5× more activated parameters.