MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

Vision and language models (VLMs) hold immense promiseto transform biomedical imaging workflows, from detecting lesions inchest X-rays to profiling cellular features in microscopy. Realizing this po-tential, however, requires robust and fine-grained visual perception. Mod-els need to correctly interpret subtle features in images, and they mustdo so across diverse biomedical modalities, scales, and contexts. Never-theless, current benchmarks remain limited. To address these gaps, weintroduce the Massive Multimodal Biomedical Understanding (MMBU)benchmark. It is the largest biomedical vision and language benchmarkto date, covering 35 submodalities with rich structured metadata. Itincludes both open and closed versions of ungrounded classification,grounded classification, and object detection, enabling systematic evalu-ation of model performance across biological scales, clinical settings, andimaging modalities. Evaluating 15 open-weight and 2 frontier VLMs, wefind that while medical adaptation provides measurable gains for somemodels, the high accuracy often reported on established benchmarks canmask deficiencies in visual perception and domain generalization.