How do the three sourcing models compare?
Whichever model you pick, FDA's January 2025 draft guidance on AI-enabled device software recommends that a marketing submission describe the annotators' expertise, the instructions they were given and whether they were blinded to each other, how annotation quality and consistency were checked and disagreements adjudicated, and the plan for fixing incorrect labels. The draft is not final and is not for implementation, but it is the clearest public statement of what reviewers expect. The table compares the three models on those points. Cells marked "our view" are general considerations, not sourced figures.
| Factor | In-house clinicians | Crowd platforms | Specialist expert labeling |
|---|---|---|---|
| Annotator expertise you can document | Your own clinicians; credentials on file | Varies by task; ask what screening and qualification tests are used | Named specialties and credentials per project |
| Independent, blinded reads | Possible, but colleagues often share context; plan for it | Typically many independent workers per item | Set by protocol (e.g., two independent reads plus adjudication) |
| Quality control and adjudication | You design and run it | Platform filtering of crowd inputs (pre-evaluation, post-evaluation, quality-control items) | Agreement metrics and adjudication run by the vendor, reported to you |
| PHI exposure | Stays inside the covered entity | Usually requires de-identified data first | De-identified data, or PHI under a BAA that covers subcontractors |
| Turnaround (our view) | Limited by clinical schedules | Fast for simple, high-volume tasks | Depends on specialist availability and protocol |
| Cost drivers (our view) | Clinician time taken from care | Items x workers per item x QC overhead | Specialist hours, number of reads, adjudication rate |
Sources: FDA draft guidance page (Jan 7, 2025); FDA draft guidance PDF; Petrović et al., Health Informatics J 2020 (systematic review); HHS de-identification guidance; HHS sample BAA provisions. No official source covers turnaround or cost, so those rows have no numbers.
Which approach fits your project?
Start from what the labels are for. If they will form the reference standard for an FDA submission, the draft guidance asks how that standard was set, how uncertain it is, how equivocal or missing cases were handled and, when clinicians grade, the grading and blinding protocols, how many clinicians read and their qualifications, and their intra- or inter-reader variability. That favors documented specialists, whether your own or a vendor's. Test data should also be independent of training data, for example from different sites, and kept away from the model developers, so plan who labels the test set separately from who labels training data.
Crowdsourcing has a real evidence base: a systematic review of 43 studies of crowdsourced medical image analysis concluded that crowdsourcing "can successfully solve medical image analysis problems" but that its use "is still limited and more studies should be conducted to obtain generalizable results." It fits best for de-identified data and tasks a non-specialist can be trained and tested on. In our view, many teams mix models: a crowd or junior pass for screening and specialists for the reference standard and the test set.
Sources: FDA draft guidance PDF; Petrović et al., Health Informatics J 2020 (systematic review); FDA Good Machine Learning Practice page.
15 questions to ask a labeling vendor
Expertise
- Who will label our data, and what are their specialties, credentials and years of practice?
- How are annotators trained on our guideline, and is passing a qualification test required before live work?
- Can you give us the written annotator instructions as a deliverable we can put in a submission?
Quality and consistency
- Does each annotator read independently, without seeing other annotators' labels?
- How many reads per item, and how are disagreements adjudicated?
- Which agreement metric do you report (for example kappa or ICC), per label and per annotator?
- How do you find and correct incorrect labels after delivery?
- How do you handle equivocal or unreadable cases?
FDA readiness
- Can you document the reference-standard process: grading protocol, blinding, number and qualifications of readers, and reader variability?
- Can different annotators label the test set than the training set, and is the test set kept from our model developers?
- Will you keep an audit trail of who labeled what, and when?
Security and HIPAA
- Will annotators see PHI? If yes, will you sign a BAA?
- Are any annotators subcontractors, and does the BAA flow down to them?
- If you de-identify for us, is that authorized in the BAA, and do you use Safe Harbor or Expert Determination, including free text and image headers?
- What happens to our data at the end: return or destruction, and how is it confirmed?
Sources: FDA draft guidance PDF; HHS de-identification guidance; HHS sample BAA provisions. The question wording is ours; each item maps to a point in these sources.
Related resources
Frequently asked questions
Does FDA require independent annotators?
FDA's January 2025 draft guidance on AI-enabled device software recommends independent assessments by each annotator, without knowledge of the other annotators' decisions. It is a draft and not for implementation, so it is a recommendation rather than a requirement, but it signals what reviewers look for.
Can a labeling vendor see PHI?
Only under a business associate agreement. HHS's sample BAA provisions require safeguards, breach reporting, the same terms for subcontractors, and return or destruction of PHI at termination. A vendor may de-identify PHI for a covered entity only as far as the BAA authorizes it.
Is crowdsourced labeling good enough for medical AI?
It can be. A 2020 systematic review of 43 studies found that crowdsourcing can successfully solve medical image analysis problems, but said its use is still limited and more studies are needed for generalizable results. It works best on de-identified data with strong quality control.
Scope it in 30 minutes
Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.
Book a 30-min scoping callPrefer to write? Send a short note.