120 - Clinical Utility of Consensus Multimodel LLMs for Interventional Radiology: A Proof-of-Concept Evaluation

Purpose: To develop and evaluate an LLM‑driven procedure planning and recommendation system for interventional radiology (IR) that integrates multiple large language models with patient data and clinical guidelines, and to compare true image interpretation against a template‑based approach for clinical utility and accuracy.

Materials and Methods: A multi‑model architecture integrated three LLMs (OpenAI GPT‑4o, Anthropic Claude 4, and xAI Grok‑3) via APIs into an aggregation engine that synthesized consensus outputs into guideline‑concordant recommendations. In a retrospective study (IRB‑approved), 10 de‑identified IR cases spanning venous thromboembolism, arterial interventions, trauma, and interventional oncology were processed by two pipelines: (1) direct image analysis (LLMs interpreted CT/MRI/angiography with consensus and confidence scoring) and (2) template‑based analysis (generic recommendations from case descriptions without image review). Clinical data included imaging, laboratory values, demographics, and history. Quality metrics included agreement with procedures performed (Cohen’s kappa), intervention specificity, risk stratification accuracy, guideline compliance, and decision support utility.

Results: Direct image analysis produced case‑specific findings and recommendations (e.g., active upper GI hemorrhage with targeted embolization; traumatic aortic pseudoaneurysm with TEVAR; lesion‑specific HCC therapy), whereas template‑based outputs were generic, imaging‑agnostic, and often misaligned with clinical context. Agreement with procedures performed was strong for image analysis (kappa > 0.8) and poor for template‑based analysis (kappa < 0.3). Intervention specificity was 90% for image analysis versus 20% for template‑based; correct urgency/risk stratification was 85% versus 40%, respectively. Quality ratings favored image‑based over template‑based outputs for clinical relevance (9/10 vs 4/10), evidence base (9/10 vs 2/10), intervention specificity (9/10 vs 3/10), and risk assessment accuracy (8/10 vs 4/10).

Conclusion: In this proof‑of‑concept evaluation, LLM‑driven decision support for IR was clinically useful only when real image interpretation was incorporated. Template‑based, non‑image pipelines underperformed on accuracy, specificity, and risk assessment, underscoring the necessity of image based LLM analysis for safe, evidence‑based IR recommendations.