What does it mean to validate a clinical prediction model?
A clinical prediction model estimates a person's risk of having, or developing, a health outcome from several inputs. Examples include a risk score, a regression model, or a machine learning model inside a digital health product. Validating it means measuring how well its predictions hold up on data other than the exact data used to build it. Performance measured on the development data alone is usually optimistic. The TRIPOD+AI statement (Collins et al., BMJ 2024) sets out what a study that develops or evaluates a prediction model should report, whether it uses regression or machine learning.
What are the steps from development to external validation?
| Step | What it is | What it answers |
|---|---|---|
| Development | Fit the model on a development dataset, with predictors and outcome defined in advance. | What is the model, and how was it built? |
| Internal validation | Re-estimate performance using resampling of the development data, such as bootstrapping or (repeated) cross-validation. | How optimistic is the apparent performance, and how much will it shrink on new patients from the same source? |
| Temporal validation | Test on patients from the same setting but a later time period. | Does performance hold as practice, devices and populations drift over time? |
| External validation | Test on data from different sites, populations or devices, collected separately from the development data. | Does the model transport to the settings where it will actually be used? |
| Recalibration or updating | Adjust the intercept or slope, or refit, when validation shows miscalibration in a new setting. | Can the model be fixed for a new population without starting over? |
Sources: TRIPOD+AI (BMJ 2024); Van Calster et al., BMC Medicine 2019.
Which performance metrics should you report?
- Discrimination (AUROC / C-statistic): how well the model ranks people who have the outcome above those who don't. 0.5 is no better than chance and 1.0 is perfect ranking. It says nothing about whether the predicted risks are the right size.
- Calibration (calibration plot, calibration slope and intercept): whether predicted risks match observed risks. If the model says 20%, about 20 in 100 such people should have the outcome. A slope below 1 usually means predictions are too extreme (often from overfitting). Van Calster et al. call calibration "the Achilles heel of predictive analytics."
- Clinical usefulness (decision curve analysis): whether acting on the model's predictions does more good than harm across a range of decision thresholds, compared with treating everyone or no one (Vickers & Elkin, Med Decis Making 2006).
What does TRIPOD+AI ask you to report?
TRIPOD+AI is a 27-item checklist, with a separate checklist for abstracts. It supersedes TRIPOD 2015, which its authors say should no longer be used. In summary, it asks for:
- the intended use and target population of the model
- data sources, eligibility, and how predictors and outcomes were defined and measured
- sample size and missing-data handling
- the modeling approach and any tuning
- how performance was evaluated (discrimination, calibration, and clinical utility where relevant)
- fairness across subgroups
- openness of data and code
- limitations
Use the official checklist from the TRIPOD website rather than this summary. To appraise someone else's model, or your own before submission, PROBAST and its update PROBAST+AI (Moons et al., BMJ 2025) assess risk of bias and applicability.
What are the most common mistakes?
- Overfitting: too many candidate predictors for the number of outcome events. Riley et al. (Stat Med 2019) give methods to calculate the minimum sample size for development.
- Data leakage: the same patient, or information from the future, appears in both training and test data. Split at the patient level and respect time order.
- Reporting apparent performance as validation: metrics computed on the development data without correcting for optimism.
- Ignoring calibration: reporting only AUROC, so a model that ranks well but over- or under-estimates risk looks fine.
- No external validation before claims: performance at one site is assumed to carry over to others.
- Unreported subgroups: no check of performance across relevant demographic or clinical groups.
Where does FDA come in?
If the model is part of a medical device, FDA's Good Machine Learning Practice guiding principles and the January 2025 draft guidance on AI-enabled device software functions (draft, not for implementation) set out expectations, including test data that are independent of training data and performance across relevant subgroups. Whether your product is a device, and what FDA will expect, depends on its intended use. Confirm that with regulatory counsel.
From our research
Mike Hittle co-authored a study that developed and internally validated a prediction model for migraine attacks: Holsteen KK, Hittle M, Barad M, Nelson LM. "Development and Internal Validation of a Multivariable Prediction Model for Individual Episodic Migraine Attacks Based on Daily Trigger Exposures." Headache. 2020;60(10):2364–2379. doi:10.1111/head.13960. As the abstract reports, 178 adults with episodic migraine completed a 90-day daily-diary study through a custom iPhone research app. The models were multivariable multilevel logistic regression, internally validated with repeated cross-validation. The abstract states that "the multivariable model predicted migraine risk only slightly better than chance (within-person C-statistic: 0.56, 95% CI: 0.54, 0.58)." The authors concluded that attacks "were not predictable based on self-prediction or on self-reported exposure to common trigger factors." A negative result like this, reported honestly with internal validation, is part of what good prediction research looks like.
How PAHL helps
We help digital health and health-AI teams:
- plan the development and validation of prediction models
- choose and pre-specify metrics
- design temporal and external validation
- write TRIPOD+AI-aligned reports
We are not regulatory counsel and do not submit to FDA on your behalf. This page is general information, not clinical or regulatory advice.
Related
Frequently asked questions
What is the difference between internal and external validation?
Internal validation estimates how optimistic a model's performance is using the development data itself, for example by bootstrapping or cross-validation. External validation tests the model on separately collected data from different sites, times or populations, which shows whether it transports to new settings.
Is a high AUROC enough to show a model works?
No. AUROC measures discrimination only. A model can rank patients well but give risks that are systematically too high or too low, so calibration should also be reported, and clinical usefulness assessed where the model will drive decisions.
Should I still use the TRIPOD 2015 checklist?
No. TRIPOD+AI (BMJ 2024) supersedes TRIPOD 2015 for prediction models built with regression or machine learning, and its authors say the 2015 checklist should no longer be used.
Scope it in 30 minutes
Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.
Book a 30-min scoping callPrefer to write? Send a short note.