Download the template
A fill-in annotation guideline for medical AI datasets: scope, reference standard, provenance and splits, population coverage, annotator qualifications, labeling rules, adjudication and quality control.
What regulators expect of training data
In January 2025, the IMDRF released a final document with 10 Good Machine Learning Practice (GMLP) guiding principles, building on principles FDA, Health Canada and the UK MHRA released in October 2021. In the 2021 wording, three principles bear directly on annotation:
| Principle | 2021 wording | Template section |
|---|---|---|
| Population (principle 3) | Data should represent "the relevant characteristics of the intended patient population (e.g., age, gender, sex, race, and ethnicity), use, and measurement inputs." | Log demographics and use conditions of the annotated set |
| Independence (principle 4) | Training and test sets "are selected and maintained to be appropriately independent," considering "patient, data acquisition, and site factors." | Record patient, site and acquisition source per item |
| Reference standard (principle 5) | "Accepted, best available methods for developing a reference dataset ... ensure that clinically relevant and well characterized data are collected and the limitations of the reference are understood." | Define who sets ground truth and document its limits |
Sources: FDA GMLP page; FDA slide deck, 2021 principles. Quotes are the 2021 wording; check the 2025 IMDRF text before relying on exact language.
Reporting guidelines to plan for
If the dataset will support a publication or submission, check what these guidelines ask authors to report about ground truth and annotation: CLAIM for imaging AI, TRIPOD+AI for prediction models, and STARD-AI for diagnostic accuracy studies (note its 2026 author correction). For domain scoring rules, cite the governing manual, such as the AASM scoring manual for sleep, rather than restating it.
Sources: CLAIM, Mongan et al. 2020; CLAIM 2024 Update, Tejani et al.; TRIPOD+AI, Collins et al. 2024; STARD-AI, Sounderajah et al. 2025; author correction, 2026.
Related resources
Frequently asked questions
What should a medical annotation guideline include?
Scope, the reference standard and its limits, data provenance and splits, population coverage, annotator qualifications, labeling rules, adjudication and quality control.
Which GMLP principles apply to data annotation?
In the 2021 FDA, Health Canada and MHRA wording, the principles on representative data, independent training and test sets, and reference standards apply most directly.
Is the template free?
Yes. Download it as Word or Markdown and adapt it to your dataset.
Scope it in 30 minutes
Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.
Book a 30-min scoping callPrefer to write? Send a short note.