How Big Should Your Medical AI Test Set Be? Sample Size, Patient-Level Splits and Subgroup Coverage

Test-set size is a precision question

The question "how many cases do I need?" really means "how wide a confidence interval can I accept around the number I will report?" FDA's draft guidance for AI-enabled devices says "All performance estimates should be provided with confidence intervals," and FDA's statistical guidance for diagnostic tests lists "estimates of sensitivity and specificity pairs, likelihood ratio of positive and negative result pairs, and ROC … analysis along with confidence intervals" as appropriate measures. A small test set gives wide intervals. Riley and colleagues found that "Many existing validation studies are too small, which leads to wide confidence intervals of performance estimates and potentially misleading claims about a model's reliability."

Sources: FDA AI-enabled device software functions draft guidance, Jan 2025; FDA statistical guidance on diagnostic tests, 2007; Riley et al., BMJ 2024. This page is a summary, not regulatory or statistical advice for a specific submission.

How many cases for a target sensitivity?

Sensitivity is estimated only from cases that have the condition, so you size those first and then divide by prevalence to get the total number of cases to collect. Buderer (1996) set out this prevalence-based approach: sample size is "calculated based on a clinically acceptable degree of precision, the hypothesized values of sensitivity and specificity, and the estimated prevalence of disease."

The numbers below are our own calculation with the normal-approximation formula:

cases with condition = z² × Se × (1 − Se) / d²,   total cases = cases with condition / prevalence

where z = 1.96 for a 95% interval, Se is the sensitivity you expect, and d is the half-width you can accept. Results are rounded up.

Read the first row this way: if you expect 90% sensitivity and want the 95% interval to be about 85% to 95%, you need 139 positive cases. If only 5% of the cases you collect are positive, that means about 2,780 cases in total. Halving the half-width roughly quadruples the cases needed. Specificity works the same way using cases without the condition: divide by (1 − prevalence) instead.

Sources: Buderer 1996 (method, cited by reference); table values computed for this page. The normal approximation is rough when sensitivity is close to 100% or counts are small; in those cases a statistician will usually use an exact or Wilson interval (our note).

How many cases for a target AUC?

For a model that outputs a score, the area under the ROC curve (AUC) "represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject." Hanley and McNeil give "closed-form expressions for the approximate magnitude of the sampling variability" of the AUC, and guidance on "the size of the sample required." The table applies their standard-error formula with equal numbers of cases with and without the condition.

SE(AUC) = √[ (A(1−A) + (n₁−1)(Q₁−A²) + (n₂−1)(Q₂−A²)) / (n₁n₂) ], with Q₁ = A/(2−A), Q₂ = 2A²/(1+A), half-width = 1.96 × SE

Sources: Hanley & McNeil 1982; table values computed for this page.

Validating a clinical prediction model

For risk models, a single sensitivity or AUC target is not enough. Riley and colleagues note that "rules of thumb for sample size have been proposed, such as having at least 100 events and 100 non-events," but "Such rules of thumb provide a starting point but are problematic." Their approach sizes an external validation study for the precision of "R2, calibration curve, c statistic, and net benefit," and "The pmvalsampsize package in Stata and R allows researchers to implement the approach with one line of code." Separate methods papers cover binary and time-to-event outcomes.

Sources: Riley et al., BMJ 2024; Riley et al., Stat Med 2021; Riley et al., Stat Med 2022.

Patient-level splits and leakage

Good Machine Learning Practice calls for training data sets to be independent of test data sets. FDA's draft AI guidance adds that "it is also important for FDA reviewers to understand whether the test data are independent (e.g., sampled from completely different clinical sites) from the training data and are sequestered from the model developers and the model development stage."

In practice (our view), that means:

• Split by patient, not by image, recording or visit. Several scans or EEG segments from one person must all sit on the same side of the split.
• Hold out whole sites or devices when you can, so the test set measures performance on data the model has never seen.
• Lock the test set before model development and keep it away from the people tuning the model.
• Count cases in patients, not in rows, when you apply the tables above.

Sources: FDA/Health Canada/MHRA Good Machine Learning Practice, 2021 (paraphrased); FDA AI-enabled device software functions draft guidance, Jan 2025. The checklist is our practice recommendation.

Subgroup and site coverage

Good Machine Learning Practice also says clinical study participants and data sets should be representative of the intended patient population. FDA's draft AI guidance states: "Generally, subgroup analysis by patient characteristics (e.g., sex, age, race, ethnicity, disease severity), geographic sites, and data collection equipment are appropriate."

This drives size more than founders expect. If you want a usable interval for each subgroup, each subgroup needs enough positive cases on its own. Using the table above, a ±10-point interval around 90% sensitivity takes 35 positive cases per subgroup, so four age bands need about 140 positive cases before you account for prevalence (our arithmetic).

Sources: FDA/Health Canada/MHRA Good Machine Learning Practice, 2021 (paraphrased); FDA AI-enabled device software functions draft guidance, Jan 2025. FDA's January 2025 AI guidance is a draft and is not for implementation.

Reporting the result

Report the test-set size in patients and in positive cases, how it was chosen, the split method, and every performance estimate with its confidence interval, overall and by subgroup. For prediction models, the TRIPOD+AI reporting guideline lists what to include.

Sources: Collins et al., TRIPOD+AI, BMJ 2024; FDA AI-enabled device software functions draft guidance, Jan 2025.

Related resources

• Clinical prediction model validation
• Inter-rater agreement for medical data labeling
• Choosing a medical data labeling approach
• Is my app a medical device?
• Labeling scoping calculator
• Clinical study design services