Health AI Validation Claims: 8 Red Flags to Check

How we grouped these

These eight red flags are our own editorial grouping, not a list defined by FDA or any single paper. Each is tied to a published finding or to FDA’s Good Machine Learning Practice (GMLP) principles, now an IMDRF final document.

1. Retrospective-only evidence presented as clinical proof

A strong retrospective result is not evidence of benefit in clinical use.

Evidence: In a systematic review of deep learning versus clinicians in imaging, only nine of 81 non-randomised studies were prospective and just six were tested in a real clinical setting; 58 of 81 were at high risk of bias (Nagendran M et al., BMJ 2020). For how FDA-authorized AI devices have been evaluated, see Wu E et al., “How medical AI devices are evaluated,” Nat Med 2021.

Ask for: Whether any results are prospective, and in what setting.

2. No external validation

Single-site performance often does not transfer.

Evidence: Pneumonia models performed better internally than externally in 3 of 5 comparisons, and the best internal model fell from AUC 0.931 to 0.815 at an outside site (Zech JR et al., PLoS Med 2018). An external validation of a widely deployed sepsis model found a hospitalization-level AUC of 0.63 in one health system's data (Wong A et al., JAMA Intern Med 2021).

Ask for: Results from at least one site that contributed no training data.

3. Weak or undefined reference standard

If “ground truth” is vague, accuracy numbers are hard to interpret.

Evidence: GMLP asks that models be evaluated against a clinically relevant, well-characterized reference standard (FDA, Good Machine Learning Practice guiding principles). STARD-AI centres AI diagnostic accuracy reporting on the reference standard (STARD-AI, Nat Med 2025).

Ask for: Who labeled the test set, with what qualifications and protocol, and how disagreements were adjudicated.

4. Data leakage between training and test sets

Leakage inflates headline numbers.

Evidence: GMLP says training data should be independent of test sets (FDA, Good Machine Learning Practice guiding principles). Kapoor and Narayanan found leakage in 17 fields affecting 294 papers, in some cases leading to “wildly overoptimistic conclusions” (Kapoor S, Narayanan A, Patterns 2023).

Ask for: How patients (not just records) were split, and how duplicates were removed.

5. No subgroup performance

Average performance can hide worse performance in specific groups.

Evidence: Chest X-ray classifiers “consistently and selectively underdiagnosed under-served patient populations” (Seyyed-Kalantari L et al., Nat Med 2021). GMLP asks that data represent the intended patient population (FDA, Good Machine Learning Practice guiding principles).

Ask for: Performance by sex, age, race and ethnicity, site and device, with sample sizes.

6. A headline metric that hides clinically important failures

An AUC can look fine while a clinically important subset fails, or while alert burden is unworkable.

Evidence: See Oakden-Rayner L et al., “Hidden stratification…,” CHIL 2020 on hidden stratification. In the sepsis model validation, the model alerted on 18% of all hospitalizations while missing 67% of sepsis cases (Wong A et al., JAMA Intern Med 2021).

Ask for: Operating-point metrics: sensitivity, PPV and alert rate at the threshold actually used.

7. Confounders and shortcut learning

Models can learn the site, scanner or workflow rather than the disease.

Evidence: In the pneumonia study, models identified the source hospital system for nearly all radiographs, and prevalence differences alone gave an AUC of 0.861 just by sorting by hospital system (Zech JR et al., PLoS Med 2018).

Ask for: Analyses that rule out site, device and workflow shortcuts.

8. Incomplete reporting

Without a reporting guideline, data or code, claims can't be checked.

Evidence: In the imaging review, datasets were unavailable in 95% of studies and code in 93% (Nagendran M et al., BMJ 2020). Use Collins GS et al., TRIPOD+AI, BMJ 2024 for prediction models, Mongan J et al., CLAIM, Radiol Artif Intell 2020 (and the Tejani AS et al., CLAIM 2024 update) for imaging, and STARD-AI, Nat Med 2025 for diagnostic accuracy.

Ask for: A report that follows the relevant guideline, with a completed checklist.

What it means for startups

• Have prospective, multi-site and subgroup numbers ready, or state their absence plainly.
• Document the reference standard and how you split patients; leakage and weak ground truth are the most common ways headline numbers get inflated.
• Pair AUC with operating-point metrics buyers actually experience.

Go deeper

• Clinical prediction model validation
• Inter-rater agreement in medical data labeling
• FDA-authorized AI neurology devices (data)

Sources (accessed October 27, 2026)

1. Nagendran M et al., BMJ 2020
2. Wu E et al., “How medical AI devices are evaluated,” Nat Med 2021 (cited for topic only)
3. Zech JR et al., PLoS Med 2018
4. Wong A et al., JAMA Intern Med 2021 (one health system, 2018–2019; later model versions may differ)
5. FDA, Good Machine Learning Practice guiding principles
6. STARD-AI, Nat Med 2025
7. Kapoor S, Narayanan A, Patterns 2023
8. Seyyed-Kalantari L et al., Nat Med 2021
9. Oakden-Rayner L et al., “Hidden stratification…,” CHIL 2020 (cited for topic only)
10. Collins GS et al., TRIPOD+AI, BMJ 2024
11. Mongan J et al., CLAIM, Radiol Artif Intell 2020
12. Tejani AS et al., CLAIM 2024 update

Related resources

• Healthcare AI consulting
• Technical due diligence
• Expert clinical labeling