Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

Generative models for volumetric medical images have foundmany applications in medical imaging, ranging from data augmentationto serving as priors for inverse problems. For these applications, generat-ing high-resolution 3D images with strong controllability is essential butremains highly challenging. Existing approaches typically control gener-ation either through radiology reports used as text prompts or throughfull image segmentation. While text-based prompting is flexible, it pro-vides limited spatial control over the location, shape, and boundary ofabnormalities. In contrast, segmentation-based methods receive precisespatial guidance but are restrictive in requiring full-organ annotations.In this work, we propose MedSynV2, a flexible multimodal framework forcontrollable volumetric image generation that supports input from radi-ology reports and segmentation prompts (both optional). Our approachallows users to provide segmentation of a specific anatomy or abnormalitywithout requiring full-organ annotations. The semantic meaning of thesegmentation mask is specified through an accompanying text descrip-tion, resulting in a highly flexible and scalable conditioning mechanism.We develop a memory-e!cient architecture based on a modified di"usiontransformer that jointly processes image and segmentation tokens. Themodel further incorporates gated attention to e"ectively attend to longradiology reports. Experiments demonstrate that our method achievesstate-of-the-art perceptual and semantic scores (e.g., → 24% relative im-provement in mean FID), generates high-resolution anatomically consis-tent CT volumes, and improves data e!ciency when used for data aug-mentation. Radiologists’ evaluation further confirms strong alignmentbetween generated and real medical images. We also explore the abilityof the model to generalize towards concepts beyond training data. Codewill be released at https: // github. com/ batmanlab/ MedSynV2 .