539 - Retrieval-Augmented Language Models for Patient-Centered Periprocedural Anticoagulation in Interventional Radiology

Purpose: Managing periprocedural anticoagulation (AC) in interventional radiology (IR) is a complex task that requires careful consideration of patient-specific factors. Current tools, such as the Society of Interventional Radiology (SIR) Guidelines App, offer static recommendations that lack individualization. In this study, we assess the ability of large language models (LLMs) enhanced with Retrieval-Augmented Generation (RAG) to produce patient-tailored, guideline-concordant AC recommendations. We further compare the performance of RAG-augmented models with their non-RAG counterparts.

Materials and Methods: IR procedure data were obtained from the MIMIC-IV database, resulting in 394 unique procedure–anticoagulant pairs. Five state-of-the-art LLMs, Claude-3.5-Sonnet, GPT-4o, LLaMA-3.1, Mistral-Large-2411, and Qwen-2.5, were implemented within a RAG system built using LlamaIndex, which indexed the SIR guidelines as the external knowledge source. Claude was also evaluated without RAG integration (Base Claude). Each model was prompted with the procedure–AC pairs to generate recommendations on: (1) whether to withhold anticoagulation, (2) withholding and reinitiation timeframes, and (3) platelet and International Normalized Ratio (INR) thresholds. Model outputs were evaluated by a human assessor for concordance with SIR guidelines using a 3-point scale (2 = complete concordance, 1 = partial concordance, 0 = no concordance).

Results: LLMs display high accuracy in generating patient-specific AC recommendations. Claude achieved the highest overall concordance (1.51/2). Claude, GPT-4o, and Mistral performed similarly in withholding decisions (1.88, 1.86, 1.88, respectively). Claude and GPT-4o were comparable in timeframe accuracy (withholding: 1.56, 1.55; reinitiation: 1.34, 1.37). Mistral outperformed on platelet thresholds (1.52), while Claude led INR guidance (1.37). LLaMA and Qwen underperformed, particularly for timeframes and INR. Significant differences (p< 0.001) separated top-performing models (Claude, GPT-4o, Mistral) from lower-performing ones (LLaMA, Qwen). Claude outperformed Base Claude across all categories except platelet thresholds.

Conclusion: RAG-integrated LLMs, particularly Claude, GPT-4o, and Mistral, demonstrated strong potential in generating patient-specific, guideline-concordant anticoagulation recommendations for IR. These results underscore the promise of both proprietary and open-source models as safe, efficient, and adaptable decision-support tools for periprocedural anticoagulation management.