Medinsight AI: Explainable Chest X-Ray Analysis and Automated Radiology Report Generation with Knowledge Verification

Small vision-language models (VLMs) can draft chest X-ray reports, but after fine-tuning they often fall back on one generic “normal” report, and they give no hint about when a draft should not be trusted. We studied a two-model pipeline on the Indiana University (IU) X-ray collection: a calibrated EfficientNetV2-S classifier and a MedGemma-4B generator fine-tuned with QLoRA. We asked whether the classifier’s calibrated image-level predictions can send unreliable drafts to a human reader, and what training-free visual contrastive decoding (VCD) does to finding-level quality and diversity. On 240 held-out studies, VCD raised NIH-14 macro-F1 from 0.262 to 0.399 (paired bootstrap p = 0.002; Holm-adjusted p = 0.012) and cut the share of the single most common report from 71.2% to 35.8%. BLEU-4, ROUGE-L and CheXbert micro-F1 stayed flat or dropped slightly, no other clinical gain survived multiplicity correction, and because the greedy baseline used classifier hints in its prompt while VCD did not, the gains cannot be attributed to VCD alone. For verification, the classifier was adapted to IU with five-fold cross-fitting and every threshold was frozen before testing. At its 90%-sensitivity operating point the gate auto-approved 29.2% of VCD reports (70 of 240). Every approved report was a normal-reading draft, so the gate works as a screen for omissions and missed abnormalities in such drafts. Among approved reports, 4.3% (3 of 70; exact 95% CI 0.9–12.0%) had a labeler-detected label mismatch against the reference report, and 71 of the 74 label-mismatched reports were routed to review. On 1,033 test studies with retrieval-generated reports, the same gate approved 23.8% with a 6.9% mismatch rate (4.1–10.8%). A pre-declared second referee trained on CheXpert and MIMIC-CXR transferred poorly to IU (abnormality AUC 0.74) and halved coverage without a significant drop in risk. On 3,000 images from an external hospital system (VinDr-CXR, Vietnam) with radiologist labels, the frozen classifier reached a normal-versus-abnormal AUC of 0.891, IU-fitted calibration cut the mean expected calibration error from 0.30 to 0.037, and at the same operating point 28.7% of images would pass the gate, of which 3.6% (exact CI 2.5–5.1%) were abnormal by radiologist consensus. The report-level results are label-mismatch rates from a single institution with report-derived labels and no radiologist review of the generated reports; they are not radiologist-confirmed error rates and say nothing about clinical safety.

chest X-ray, radiology report generation, vision-language models, visual contrastive decoding, calibration, selective prediction

1. A. Sellergren et al., “MedGemma technical report,” arXiv:2507.05201, 2025. [Google Scholar] [Crossref]

2. Z. Babar, T. van Laarhoven, and E. Marchiori, “Encoder-decoder models for chest X-ray report generation perform no better than un-conditioned baselines,” PLoS ONE, vol. 16, no. 11, Art. no. e0259639, 2021, doi: 10.1371/journal.pone.0259639. [Google Scholar] [Crossref]

3. S. Leng, H. Zhang, G. Chen, X. Li, S. Lu, C. Miao, and L. Bing, “Mit-igating object hallucinations in large vision-language models through visual contrastive decoding,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 13872–13882. [Google Scholar] [Crossref]

4. Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2017. [Google Scholar] [Crossref]

5. X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common tho-rax diseases,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 2097–2106. [Google Scholar] [Crossref]

6. M. Tan and Q. V. Le, “EfficientNetV2: Smaller models and faster training,” in Proc. Int. Conf. Mach. Learn. (ICML), 2021, pp. 10096–10106. [Google Scholar] [Crossref]

7. Z. Chen, Y. Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2020, pp. 1439–1449. [Google Scholar] [Crossref]

8. H. Jin, H. Che, Y. Lin, and H. Chen, “PromptMRG: Diagnosis-driven prompts for medical report generation,” in Proc. AAAI Conf. Artif. Intell., vol. 38, no. 3, 2024, pp. 2607–2615, doi: 10.1609/aaai.v38i3.28038. [Google Scholar] [Crossref]

9. E. J. Hu et al., “LoRA: Low-rank adaptation of large language mod-els,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2022. [Google Scholar] [Crossref]

10. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2023. [Google Scholar] [Crossref]

11. H. Abdaoui, S. Barbaria, A. Al Kuwaiti, N. Raouafi, R. Ben Salah, and H. Boussi Rahmouni, “Swin-Qwen3: A three-stage vision–language framework for automated radiology report generation with multi-agent verification,” Front. Radiol., vol. 6, Art. no. 1875905, 2026, doi: 10.3389/fradi.2026.1875905. [Google Scholar] [Crossref]

12. R. Mahmood et al., “Phrase-grounded fact-checking for automatically generated chest X-ray reports,” arXiv:2509.21356, 2025. [Google Scholar] [Crossref]

13. A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. P. Lungren, “CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2020, pp. 1500–1519. [Google Scholar] [Crossref]

14. J. C. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, A. J. Smola, P. Bartlett, B. Schölkopf, and D. Schuurmans, Eds. Cambridge, MA, USA: MIT Press, 1999, pp. 61–74. [Google Scholar] [Crossref]

15. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. Int. Conf. Mach. Learn. (ICML), 2017, pp. 1321–1330. [Google Scholar] [Crossref]

16. R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2017, pp. 618–626. [Google Scholar] [Crossref]

17. D. Demner-Fushman et al., “Preparing a collection of radiology examinations for distribution and retrieval,” J. Am. Med. Inform. Assoc., vol. 23, no. 2, pp. 304–310, 2016, doi: 10.1093/jamia/ocv080. [Google Scholar] [Crossref]

18. J. P. Cohen et al., “TorchXRayVision: A library of chest X-ray datasets and models,” in Proc. Int. Conf. Medical Imaging with Deep Learning (MIDL), PMLR, vol. 172, 2022, pp. 231–249. [Google Scholar] [Crossref]

19. J. Irvin et al., “CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proc. AAAI Conf. Artif. Intell., vol. 33, 2019, pp. 590–597, doi: 10.1609/aaai.v33i01.3301590. [Google Scholar] [Crossref]

20. A. E. W. Johnson et al., “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports,” Sci. Data, vol. 6, Art. no. 317, 2019, doi: 10.1038/s41597-019-0322-0. [Google Scholar] [Crossref]

21. H. Q. Nguyen et al., “VinDr-CXR: An open dataset of chest X-rays with radiologist’s annotations,” Sci. Data, vol. 9, Art. no. 429, 2022, doi: 10.1038/s41597-022-01498-w. [Google Scholar] [Crossref]