A Comparison of Large Language Models for Common Patient Questions on the Interventional Radiology Treatment of Hepatocellular Carcinoma

Monday, March 31, 2025

3:09 PM – 3:18 PM CT

Location: Music City Center, 205 A

Presenting Author(s)

HB

Harrison S. Blume (he/him/his)

Medical Student
Albert Einstein College of Medicine, United States

Author/Co-author(s)

DW

Darnell K. Adrian Williams, Jr., BS

MD/PhD Student
Albert Einstein College of Medicine, United States
Arvind Dev

Medical Student
Albert Einstein College of Medicine, United States
Christine Yoon (she/her/hers)

Medical Student
Albert Einstein College of Medicine, United States
Jacob Cynamon, MD, FACR, FSIR

Attending
Montefiore Medical Center, United States
KW

Kapil Wattamwar, MD

Interventional Radiology Fellow
Montefiore Medical Center, United States

Purpose: To compare the efficacy of widely used Large Language Models (LLMs) in answering common patient questions about interventional radiology (IR) treatments for hepatocellular carcinoma (HCC).

Materials and Methods: Ten frequently asked patient questions regarding IR treatments for HCC were developed by interventional radiologists and researchers, addressing key concerns of patients undergoing embolization and/or ablation. Three widely accessible, free LLMs—ChatGPT 4o Mini (OpenAI), Gemini (Google), and Copilot (Microsoft)—were prompted to generate responses to the questions, which were anonymized and randomized to prevent source LLM identification. Two interventional radiology attendings independently evaluated each response using a structured survey instrument, assessing five domains using a numerical scale from 1 to 100: accuracy, comprehensiveness, readability, compassion, and overall quality. Appropriate statistical methods were applied to compare the LLM performances across domains.

Results:
Preliminary findings indicate that all three LLMs can generate readable and generally accurate responses to common patient questions about IR treatments for HCC. It is expected that:
• ChatGPT 4o Mini may score higher in readability and compassion due to its conversational design.
• Google Gemini might excel in accuracy and comprehensiveness given its integration with up-to-date medical information.
• Microsoft Copilot could perform consistently across all domains but may show variability depending on the specificity of the question.
Statistical analysis did not reveal a significant difference in performance across the LLMs within domains (p > 0.05).

Conclusion: While LLMs show considerable promise in addressing patient concerns about IR treatments for HCC, they are not yet ready to be used independently. The variability in accuracy, comprehensiveness, and compassion among the models underscore the need for professional oversight. These tools could serve as valuable adjuncts to expert guidance, and further studies incorporating patient feedback are necessary to refine these models and assess their impact on patient understanding and satisfaction.