Leveraging LLMs to Automatically Identify Kyphoplasty Inclusion and Exclusion Criteria from Clinical Notes

Wednesday, April 2, 2025

11:06 AM – 11:15 AM CT

Location: Music City Center, 202 BC

Presenting Author(s)

Jenny Nguyen

Student Author
Emory University School of Medicine, United States

Author/Co-author(s)

RW

Richard Wu, BS

Medical Student
Emory University School of Medicine, United States
NZ

Nathan Zhang, BS

Medical Student
Alabama College of Osteopathic Medicine, United States
WB

Wasif Bala, MD

Resident Physician
Emory University, United States
JM

John T. Moon, MD

Resident Physician
Emory University, United States
HL

Hanzhou Li, MD

Resident Physician
Emory University, United States
J. David Prologo, MD

Division Director, Interventional Radiology
Emory University School of Medicine, United States
Jay H. Shah, MD

Associate Professor
Emory University School of Medicine, United States

Purpose: Kyphoplasty remains underutilized for treating vertebral compression fractures despite its potential to improve patient outcomes. Large Language Models (LLMs) have demonstrated high accuracy in extracting medical information from clinical texts. This study aims to evaluate the efficacy of privacy-preserving LLMs in automatically identifying kyphoplasty inclusion and exclusion criteria from clinical notes.

Materials and Methods: Inpatient encounters for kyphoplasty were identified by CPT codes. For each encounter, the following note types were evaluated: H&P, IR consult, discharge summary, and a spinal imaging report. Four human reviewers labeled the clinical notes for the presence of inclusion and exclusion criteria, which were established based on a review of interventional radiology literature. Multiple LLMs (Claude 3.5 Sonnet, Claude 3 Haiku, LLama 3 70B) were tasked with identifying these criteria via a private instance of the Amazon Web Services Bedrock platform. Model performance was evaluated using a zero-shot (no example output provided) approach. Precision, recall, and F1 scores were calculated as metrics.

Results: Our study analyzed 49 patient encounters, corresponding to 196 individual patient notes. 15 unique inclusion and exclusion criteria classes were defined, and human experts identified a total of 255 instances of these classes across all notes. The highest performing LLM was Claude 3.5 Sonnet, achieving a global precision, recall, and F-1 score of 62.6%, 80.0%, and 70.2%, respectively. The lowest performing LLM was Claude 3 Haiku, achieving a global precision, recall, and F-1 score of 52.3%, 76.1%, and 62.0%, respectively. The most accurately identified criteria included back pain (F-1 98.9%), acute compression fracture (F-1 98.0%) and limited mobility (F-1 87.3%). Challenges were encountered in identifying intolerance to narcotics (F-1 0%) and disrupted posterior cortex (F-1 27.2%).

Conclusion: In this pilot study, LLMs demonstrated moderately successful zero-shot performance on extracting kyphoplasty inclusion and exclusion criteria from clinical notes. Our results show that the largest commercial model performed best, in line with expectations. These findings suggest that LLMs can be effectively utilized to enhance the identification of suitable kyphoplasty candidates, potentially increasing appropriate utilization of this procedure. Future work will focus on multiple-shot prompting approaches and model fine-tuning, which are expected to substantially improve performance.