294 - Multimodal Vision Transformer Modeling of Survival and Transplant Eligibility Following Radioembolization for Hepatocellular Carcinoma

Purpose: To evaluate a multimodal 3D vision transformer model for predicting clinical outcomes following Yttrium-90 (Y-90) transarterial radioembolization (TARE) for hepatocellular carcinoma (HCC), using pre-treatment MRI, 99mTc-MAA SPECT/CT, and early post-treatment MRI.

Materials and Methods: Thirty patients with biopsy-confirmed HCC who underwent TARE using Y-90 were retrospectively analyzed. Pre-treatment liver MRI (T2-weighted, arterial-phase T1, DWI), 99mTc-MAA SPECT/CT maps, and post-treatment liver MRI (~1 month) were collected. Segment Anything 2 (SAM2) was fine-tuned to segment lesions across modalities. A multimodal, multi-dimensional vision transformer (Multi-D ViT) was developed to fuse voxel-level SPECT dose and serial MRI features using hierarchical 3D patch embeddings, shifted window attention, and cross-modal integration.

Outcome heads included softmax classifiers for 6-month Milan Criteria status and mRECIST class, and Cox-loss survival heads for predicting progression-free survival (PFS) and overall survival (OS). The model was trained on 75% of patients, validated on 10%, and tested on 15%. Evaluation metrics included AUC, F1 score, R², concordance index, and log-rank p-values.

Results: The model showed modest performance in predicting both short-term response and long-term outcomes. For 6-month Milan Criteria status, AUC was 0.75, with sensitivity and specificity of 76% and 70%, respectively. mRECIST classification yielded an AUC of 0.78.

Survival prediction yielded R² values of 0.74 for PFS and 0.71 for OS. Median concordance index across five cross-validation folds was 0.73 (range: 0.69–0.77). Log-rank tests were not statistically significant (p = 0.08 for PFS, p = 0.12 for OS) indicating no significant difference between actual and predicted survivial outcomes.

Conclusion: A multimodal vision transformer model incorporating pre- and post-treatment MRI with 99mTc-MAA SPECT/CT dosimetry predicted survival and 6-month clinical endpoints after Y-90 TARE in HCC. This approach leverages dose distribution heterogeneity and delta imaging features to enable early risk stratification and early prediction of treatment outcomes. The model offers a scalable framework for optimizing bridge-to-transplant decision-making as well as guiding locoregional and systemic therapies. The model's predictions were accurate despite the small sample size. Future iterations of this study will utilize increased sample sizes for more robust and durable metric calculations.