# Medical Data Annotation Guideline Template

Palo Alto Health Labs. Template, version 1.0 (Oct 27, 2026). Replace every [bracketed] field. This template is not regulatory advice.

## 1. Scope and intended use
- Dataset name and version: [ ]
- Intended use of the model the data supports: [ ]
- Target population and care setting: [ ]
- Data types (e.g., images, signals, notes): [ ]

## 2. Reference standard
GMLP principle 5 (2021 wording): "Accepted, best available methods for developing a reference dataset (that is, a reference standard) ensure that clinically relevant and well characterized data are collected and the limitations of the reference are understood."
- What defines ground truth (expert read, test result, follow-up outcome): [ ]
- Who sets it, and their qualifications: [ ]
- Governing scoring manual, cited by name (e.g., AASM scoring manual): [ ]
- Known limitations of the reference standard: [ ]

## 3. Data provenance and splits
GMLP principle 4 (2021 wording): training and test sets "are selected and maintained to be appropriately independent," considering "patient, data acquisition, and site factors."
- Fields recorded per item: patient ID (pseudonymized), site, device or acquisition source, date: [ ]
- Split rule (by patient / by site / by time): [ ]
- How leakage between splits is prevented and checked: [ ]

## 4. Population coverage
GMLP principle 3 (2021 wording): data should represent "the relevant characteristics of the intended patient population (e.g., age, gender, sex, race, and ethnicity), use, and measurement inputs."
- Demographics logged: [ ]
- Use conditions and measurement inputs logged: [ ]
- Target vs. achieved distribution, and gaps: [ ]

## 5. Annotators
- Roles (primary annotator, reviewer, adjudicator): [ ]
- Required qualifications and training: [ ]
- Calibration set and pass criterion before live labeling: [ ]
- Blinding (to other annotators, to model output, to clinical data): [ ]

## 6. Label definitions and rules
| Label | Definition | Include when | Exclude when | Example ID |
|---|---|---|---|---|
| [ ] | [ ] | [ ] | [ ] | [ ] |

- Handling of uncertain or unreadable items: [ ]
- Tool and export format: [ ]

## 7. Adjudication and agreement
- Number of independent reads per item: [ ]
- Disagreement rule (consensus, majority, adjudicator): [ ]
- Agreement statistic and target, and how often it is measured: [ ]

## 8. Quality control
- Audit sample size and frequency: [ ]
- Error categories tracked: [ ]
- Re-training trigger: [ ]
- Change log for guideline versions: [ ]

## 9. Reporting
Check what the applicable reporting guideline asks authors to report about ground truth and annotation:
- CLAIM (imaging AI): Mongan J et al., Radiol Artif Intell 2020; 2024 Update, Tejani AS et al.
- TRIPOD+AI (prediction models): Collins GS et al., BMJ 2024.
- STARD-AI (diagnostic accuracy): Sounderajah V et al., Nat Med 2025, and its 2026 author correction.

## Sources
- FDA, Good Machine Learning Practice for Medical Device Development: Guiding Principles. https://www.fda.gov/medical-devices/artificial-intelligence-enabled-medical-devices/good-machine-learning-practice-medical-device-development-guiding-principles
- FDA slide deck reproducing the 2021 principles. https://www.fda.gov/media/160125/download