Comprehensive language–image pre-training for 3D medical image understanding

In the 3D medical image domain, vision-language pre-training is used to create vision-language encoders (VLEs) that can support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, three challenges limit the capabilities of current 3D VLEs: data scarcity due to privacy concerns, high computational costs resulting from the volumetric nature of the images, and a domain shift between the long reports used for training and the short prompts used during inference for, e.g., zeroshot classification. As a consequence, natural-image VLE recipes do not directly transfer to 3D medical imaging. In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training, allowing us to leverage both image-only and paired image-text 3D datasets. Further, we propose a novel loss that addresses the domain shift between long reports and short textual prompts. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language–Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model weights and inference code are freely available at https: //huggingface.co/microsoft/colipri.