The FDA has announced the five winners of the Veterans Cardiac Health and AI Model Predictions (V-CHAMPS) Challenge. They are:

1. VHeartAI
2. ZS Associates ADS
3. DSS
4. Cognizant Good HeartML
5. D&D

The V-CHAMPS challenge was developed by the Veterans Health Administration (VHA) Innovation Ecosystem (IE), the Digital Health Center of Excellence (DHCoE) at the FDA, the FDA Office of Digital Transformation (ODT)’s precisionFDA, and the UK Medicines and Healthcare products Regulatory Agency (MHRA) to call on the scientific and data analytics community to develop and evaluate AI/ML models to predict cardiovascular health related outcomes in Veterans.

The Challenge included two phases. Phase 1 was focused on synthetic data and ran from May 25 to August 2, 2023. In this Phase of the Challenge, AI/ML models were developed by participants and trained and tested on the synthetic data sets provided to them, with a view towards predicting outcome variables for Veterans who have been diagnosed with chronic heart failure. Phase 2 focused on validating and further exploring the limits of the AI/ML models. During this Phase, high-performing AI/ML models from Phase 1 were brought into the VA system and validated on real-world Veterans health data within the VHA.

More than 300 people from 16 countries registered for the Challenge, and the entries were judged by a combination of clinical and data science subject matter experts, and were evaluated on several aspects, including:

• Innovation in clinical predictors
• Completeness of data science approach
• Statistical metrics
• Exploration of demographics measures

Additionally, as a performance check, their model predictions on the QC dataset were compared to the ground truth in the QC data.

From the 13 entries that made it to Phase 2 of the Challenge, a final group of nine teams successfully completed the Challenge requirements and entered the final judging process. The FDA noted that teams that could not complete Phase 2 were affected mainly by using proprietary or third-party libraries that could not be deployed within the U.S. Department of Veterans Affairs (VA) cloud environment.

The final group of Phase 2 entries was dual-anonymously judged by a multi-disciplinary team of three data scientists and four clinicians from the VHA, FDA and National Institute of Standards and Technology (NIST). Models were evaluated based on performance on real-world VA data and were ranked according to the following criteria:

• High performance on clinical factors
• High performance on real-world data (RWD)
• High performance on aggregate statistical measures

Extreme Gradient Boosting (XGBoost) was the most common approach among the better-performing teams, and ensemble learning (using multiple ML models to build better predictions) using Scikit-Learn.ensemble was a common theme.

“The V-CHAMPS Challenge showed us that artificial intelligence (AI) models that performed well on the synthetic patient data in Phase 1 also performed well on the RWD during Phase 2, highlighting the potential value of using synthetic data in AI model development,” the FDA stated in its announcement of the winning entries. “Teams that employed ensemble learning approaches also tended to perform the best. Overall, isolating the clinical features that drove model performance was considered the most critical differentiator in evaluating whether AI models could potentially be valuable aids to clinicians treating patients with cardiovascular conditions such as heart failure. A formal publication is planned to provide more detailed information on the V-CHAMPS Challenge, the challenge process, the lessons learned and details of the models entered into the challenge.”