FDA Logo

FDA Announces Cardiac Health and AI Model Predictions (V-CHAMPS) Challenge Winners

By MedTech Intelligence Staff
No Comments
FDA Logo

“The V-CHAMPS Challenge showed us that artificial intelligence (AI) models that performed well on the synthetic patient data in Phase 1 also performed well on the RWD during Phase 2, highlighting the potential value of using synthetic data in AI model development.”

The FDA has announced the five winners of the Veterans Cardiac Health and AI Model Predictions (V-CHAMPS) Challenge. They are:

  1. VHeartAI
  2. ZS Associates ADS
  3. DSS
  4. Cognizant Good HeartML
  5. D&D

The V-CHAMPS challenge was developed by the Veterans Health Administration (VHA) Innovation Ecosystem (IE), the Digital Health Center of Excellence (DHCoE) at the FDA, the FDA Office of Digital Transformation (ODT)’s precisionFDA, and the UK Medicines and Healthcare products Regulatory Agency (MHRA) to call on the scientific and data analytics community to develop and evaluate AI/ML models to predict cardiovascular health related outcomes in Veterans.

The Challenge included two phases. Phase 1 was focused on synthetic data and ran from May 25 to August 2, 2023. In this Phase of the Challenge, AI/ML models were developed by participants and trained and tested on the synthetic data sets provided to them, with a view towards predicting outcome variables for Veterans who have been diagnosed with chronic heart failure. Phase 2 focused on validating and further exploring the limits of the AI/ML models. During this Phase, high-performing AI/ML models from Phase 1 were brought into the VA system and validated on real-world Veterans health data within the VHA.

More than 300 people from 16 countries registered for the Challenge, and the entries were judged by a combination of clinical and data science subject matter experts, and were evaluated on several aspects, including:

  • Innovation in clinical predictors
  • Completeness of data science approach
  • Statistical metrics
  • Exploration of demographics measures

Additionally, as a performance check, their model predictions on the QC dataset were compared to the ground truth in the QC data.

From the 13 entries that made it to Phase 2 of the Challenge, a final group of nine teams successfully completed the Challenge requirements and entered the final judging process. The FDA noted that teams that could not complete Phase 2 were affected mainly by using proprietary or third-party libraries that could not be deployed within the U.S. Department of Veterans Affairs (VA) cloud environment.

The final group of Phase 2 entries was dual-anonymously judged by a multi-disciplinary team of three data scientists and four clinicians from the VHA, FDA and National Institute of Standards and Technology (NIST). Models were evaluated based on performance on real-world VA data and were ranked according to the following criteria:

  • High performance on clinical factors
  • High performance on real-world data (RWD)
  • High performance on aggregate statistical measures

Extreme Gradient Boosting (XGBoost) was the most common approach among the better-performing teams, and ensemble learning (using multiple ML models to build better predictions) using Scikit-Learn.ensemble was a common theme.

“The V-CHAMPS Challenge showed us that artificial intelligence (AI) models that performed well on the synthetic patient data in Phase 1 also performed well on the RWD during Phase 2, highlighting the potential value of using synthetic data in AI model development,” the FDA stated in its announcement of the winning entries. “Teams that employed ensemble learning approaches also tended to perform the best. Overall, isolating the clinical features that drove model performance was considered the most critical differentiator in evaluating whether AI models could potentially be valuable aids to clinicians treating patients with cardiovascular conditions such as heart failure. A formal publication is planned to provide more detailed information on the V-CHAMPS Challenge, the challenge process, the lessons learned and details of the models entered into the challenge.”

 

 

Related Articles

  • Advamed Logo

    “There will be a great deal of legislative, regulatory, and payment policies to be implemented in the coming years with the proliferation of AI and digital health platforms into our healthcare system, and AdvaMed looks forward to working with lawmakers…

  • Rogers-Headshot

    Digital therapeutics (DTx) have the potential to lower healthcare costs and improve patient compliance, outcomes and access to care. In the first part of this two-article series we look at the current and potential applications for this growing sector of…

  • 3M logo

    “The innovation, security and reliability of AWS helps us accelerate the delivery of high-quality clinical documentation. Our overarching goal is to create a better, more sustainable solution and to continue to be a trusted partner that our clients can rely…

  • Sudhir Kesavan

    In his new role, Kesavan will assume responsibility for leading CitiusTech's global delivery organization. This will encompass enhancing practice capability, service delivery maturity, market innovation to drive profitable business growth by collaborating across practices, markets, delivery and functions.

About The Author

MedTech Intelligence

Leave a Reply

Your email address will not be published. Required fields are marked *