MedRepBench: Benchmarking Structured Understanding of Medical Report Images

Medical report understanding from real-world document im-ages is essential for generating patient-facing explanations and enablingstructured information exchange in clinical systems.Existing VLMs and LLMs have shown strong performance on documentunderstanding, but structured understanding of medical reports remainsinsufficiently benchmarked. Therefore, we introduce MedRepBench, abenchmark with 1,925 de-identified Chinese medical report images span-ning diverse departments, patient demographics, and acquisition formats.In MedRepBench, we mainly focus on report-grounded interpretationrather than evaluating diagnostic reasoning, treatment recommendation,or the integration of patient history. The interpretation is defined asstructured extraction of report fields (e.g., item, value, unit, referencerange, abnormal flag) plus a patient-facing explanation grounded strictlyin the report content.The benchmark primarily evaluates end-to-end VLMs, and also includesa controlled text-only setting (high-quality OCR + LLM) to approximatean upper bound when character recognition errors are minimized. Ourevaluation framework provides two complementary protocols: (1) an ob-jective protocol measuring field-level recall of structured items, and (2)an automated subjective protocol that uses an LLM-based judge to scorefactuality, interpretability, and reasoning quality under a fixed prompt.Using the objective metric as a reward signal, we also provide a lightweightGRPO-based alignment baseline for a mid-sized VLM, which improvesfield-level recall by up to 6%.Finally, we analyze practical limitations of OCR+LLM pipelines, includ-ing layout-related errors and additional system latency, showing the needfor robust end-to-end vision-based medical report understanding. Thedataset and evaluation resources are publicly available on HuggingFace.