Skip to main content
SIR 2026 Event Main banner
Merit Medical_Embosphere_Left Banner
MediView_Experience_Left Banner

SIR 2026

Scientific Session

General IR

Session: General IR 2

Evaluation of Large Language Models (LLM) for Patient-Oriented Explanations of Liver Directed Therapies (LDT) in Hepatocellular Carcinoma (HCC)

Tuesday, April 14, 2026
3:36 PM - 3:45 PM ET
Location: 714B
Purpose: To evaluate whether contemporary LLM can generate accurate, comprehensive, readable, and safe explanations of minimally invasive LDT for HCC in English and Spanish.

Materials and Methods: A structured framework of patient-oriented questions was developed based on guideline-endorsed management of HCC. {1-3} 8 questions were applied across 11 therapies spanning 3 categories (ablation, transarterial, external radiation). One additional question addressed minimally invasive treatments overall, yielding 9 domains in total. Two LLMs (GPT-4.1 mini and Gemini-2.5) were queried in English and Spanish at 3 temperature settings (0.0, 0.4, 0.7), generating n = 1,068 outputs.


Responses were evaluated for factual concordance with guidelines, readability (Flesch Reading Ease), and linguistic variability (perplexity, lexical entropy). Automated metrics were supplemented with clinician review to provide multi-layer validation of accuracy, comprehensiveness, and accessibility, while integration of fact checking indices (FactCC, FEVER, SMOG) further enhanced the robustness of the evaluation framework.

Results: Gemini demonstrated superior interpretability, with a higher reading score and lower grade level compared to GPT (simple words, short sentences). Readability for both versions were considered at college level; Gemini achieved slightly better scores in the Flesch Reading Ease (26.6) and Flesch-Kincaid Grade Level (14.66) compared to GPT (20.75 and 15.24, respectively), though responses by both were deemed challenging for the general public. GPT displayed significantly lower lexical entropy (6.93) than Gemini (7.40), proving Gemini responses favor word variety and richness, whereas GPT responses favor consistency and simpler language. Differences across all tests reached significance (Mann Whitney test, all p < 0.01). When matched on question, language, and temperature, the average median cosine similarity score between the two models (0.83) demonstrates that GPT and Gemini responses are semantically very close, using similar vocabulary, and following similar structures. However, in regard to style, nuance, and information, the models are distinct enough to matter for medical comprehensibility and safeness.

Conclusion: LLMs generate accurate explanations of HCC therapies but remain limited by readability, depth, and


recency. Gemini was most accurate, yet outputs stayed at college level, underscoring the need for clinical and AI expert oversight, literacy-focused tuning, and cross-language validation.