Clinical NLP on EHR notes: annotating free text for AI, public datasets, de-identification and LLM extraction accuracy

The short answer

To build or evaluate an AI system that reads clinical notes, you need human-annotated text. Annotators mark spans for clinical concepts (problems, tests, treatments, medications). They label each concept's assertion status (present, absent, possible, hypothetical, about someone else). They link concepts with relations (for example, drug to adverse event). Then you score the model against those labels with precision, recall and F1, and you say whether spans had to match exactly or only overlap.

The public benchmarks are the n2c2/i2b2 shared-task corpora and MIMIC-IV-Note. Both sit behind data use agreements. As of October 27, 2026, the n2c2 portal lists its datasets as temporarily unavailable. MIMIC-IV-Note is raw de-identified text with no entity labels, and PhysioNet forbids sending it to LLM services that retain data. HIPAA applies to free text just as it does to structured fields, and no automated de-identification tool is perfect, so de-identification is a measured step, not a checkbox.

Sources: Uzuner et al., JAMIA 2011 (2010 i2b2/VA challenge); DBMI Data Portal: n2c2 NLP Research Data Sets; MIMIC-IV-Note v2.2; PhysioNet LLM guidance (Sept 24, 2025); HHS de-identification guidance.

Annotation layers for clinical notes

Most clinical NLP projects stack several layers of labels on the same text. Each layer needs its own guideline section, its own agreement check and its own metric. The public shared tasks give tested schemes you can adopt or map to.

Store labels as character offsets into the unchanged note text, not as edited copies of the note. The brat standoff format is a common example: a .txt file holds the document, and an .ann file holds one annotation per line with a type and start and end offsets, plus relation lines that point to two annotation IDs. Offsets let you score exact and overlapping matches later, and they keep the source text auditable.

Assertion is where simple keyword systems fail. NegEx, a regular-expression negation algorithm, reported 77.8% sensitivity and 84.5% positive predictive value on 1,235 findings in 1,000 discharge-summary sentences. ConText, its extension, found that deciding whether a condition is historical needs knowledge beyond surface cues, and that conditions experienced by someone other than the patient were very rare in its test reports. Rare classes like these need deliberate sampling if you want to measure them.

Sources: Uzuner et al. 2011; Henry et al., JAMIA 2020 (2018 n2c2 Track 2); Stubbs et al., J Biomed Inform 2015 (2014 de-identification track); DBMI n2c2 portal; Chapman et al. 2001 (NegEx); Harkema et al. 2009 (ConText); brat standoff format; NLM UMLS; NLM SNOMED CT; NLM RxNorm.

Public datasets: n2c2/i2b2 and MIMIC-IV-Note

n2c2/i2b2. These corpora come from the i2b2 shared tasks (2006 to 2014) and, since 2018, the National NLP Clinical Challenges (n2c2). Harvard Medical School's Department of Biomedical Informatics is the steward. They cover de-identification, smoking status, obesity, medications, concepts/assertions/relations, coreference, temporal relations, cohort selection and ADE extraction. They are the only widely used clinical-note sets with gold labels and published baselines. For example, the 2010 set has 394 annotated training reports and 477 test reports, released under data use agreements. As of October 27, 2026, the DBMI portal says the n2c2 datasets are “temporarily unavailable” and registration is not open. It notes that 2019 tracks 1 and 2 are available directly through Mayo Clinic. Check the portal before you plan around them.

MIMIC-IV-Note v2.2. This set holds 331,794 de-identified discharge summaries from 145,915 patients and 2,321,355 radiology reports for 237,427 patients, all from Beth Israel Deaconess Medical Center. It links to the MIMIC-IV structured data. It does not ship entity, assertion or relation labels; any annotation is yours to do. Access rules:

• You need PhysioNet credentialing and the CITI “Data or Specimens Only Research” course, and you must sign the Credentialed Health Data Use Agreement 1.5.0. PhysioNet says approval typically takes 1–2 weeks.
• Access is individual. Sharing within teams or classes is not permitted.
• The agreement forbids re-identification attempts and requires you to report any identifying information you find.
• PhysioNet's LLM guidance says the agreement prohibits sharing the data with third parties, including through APIs or online platforms. It requires zero data retention and strongly recommends locally run models. If you use a cloud service, you must confirm that it keeps no data, does not train on it and has no human review. If that is unclear, do not use it.

Sources: DBMI n2c2 portal; Uzuner et al. 2011; MIMIC-IV-Note v2.2; PhysioNet Credentialed Health Data Use Agreement 1.5.0; PhysioNet LLM guidance; PhysioNet FAQs.

De-identifying clinical text

HHS is explicit that the HIPAA de-identification standard “makes no distinction between data entered into standardized fields and information entered as free text.” Identifiers in a note must be removed wherever they appear. Narrative context can still identify someone; HHS gives “Executive Vice President of the state university” as an example. Under Safe Harbor, provider names do not have to be removed.

Score de-identification mainly on recall, because one missed name matters more than one over-redacted word. The Philter authors reported 99.46% recall (78.28% precision) on a UCSF test corpus, against 95.30% for NLM-Scrubber and 85.10% for the PhysioNet deid tool. They also warned that even at 95% recall, the PHI left across millions of notes “would be staggering.” (The Philter authors disclose a patent filing on the tool.) In the 2014 i2b2 track, three of ten teams exceeded an F1 of 0.90.

Sources: HHS de-identification guidance (Q3.8, Q3.10); 45 CFR 164.514; NLM-Scrubber; Norgeot et al., npj Digit Med 2020 (Philter); Philter source code; Stubbs et al. 2015; MIMIC-IV-Note v2.2.

Measuring extraction accuracy

An F1 score means little until you say what counted as a match. The same system in the 2010 i2b2 challenge scored 0.852 F1 under exact matching and 0.924 under inexact matching. Always report the matching rule, the unit and the averaging method alongside the number.

Score assertion and relations separately from concept spans, and also end to end. In the 2018 n2c2 ADE track, the best systems reached 0.9418 F1 for concept extraction and 0.9630 for relation classification, but 0.8905 end to end. Scores were much lower for reasons and ADEs, where the authors noted that local context was not enough.

Sources: Uzuner et al. 2011; Stubbs et al. 2015 (scoring scripts: i2b2_evaluation_scripts); Henry et al. 2020; Norgeot et al. 2020.

Evaluating an LLM as the extractor

LLMs are now often used to pull structured data out of notes. Evaluate them on the same gold labels, with the same matching rules, as any other extractor. One JAMIA study tested GPT-3.5 and GPT-4 on clinical named-entity recognition, using the 2010 i2b2 concept types on MTSamples notes. With a baseline prompt, relaxed-match F1 was 0.634 for GPT-3.5 and 0.804 for GPT-4. Prompts that added annotation-guideline definitions, error-analysis-based instructions and examples raised them to 0.794 and 0.861. A fine-tuned BioClinicalBERT reached 0.901 on the same task. The same study used exact match (identical boundaries and type) and relaxed match (overlap with the same type), which is the reporting pattern to copy.

• Write the annotation guideline first, and give the model the same definitions your annotators use.
• Report exact and relaxed scores per entity type, plus assertion accuracy, against a fine-tuned or rule-based baseline.
• Check that extracted spans actually appear in the note. Generated text that paraphrases or invents a finding cannot be scored as a span match.
• Keep a held-out, freshly annotated test set from your own sites. Public corpora are widely circulated and give no guarantee that a model has not seen them.
• Do not send MIMIC or other agreement-bound notes to a hosted model unless its terms meet PhysioNet's zero-retention, no-training and no-human-review conditions.

This page covers extracting facts from existing notes. For judging notes an AI writes, see How to evaluate an AI medical scribe. For broader LLM evaluation (safety, factuality, rubric design), see How to evaluate a health LLM.

Sources: Hu et al., JAMIA 2024;31(9):1812–1820; PhysioNet LLM guidance.

Annotating your own notes: a checklist

• Settle de-identification and data-access terms before any annotator sees a note.
• Adopt or map to a public scheme (2010 i2b2 concepts and assertions, 2018 n2c2 medications) so you can compare against published baselines. Write it up using a medical data annotation guideline.
• Annotate with offsets in a standoff format. Keep raw text unchanged.
• Double-annotate a sample, measure inter-rater agreement per layer (spans, assertion, relations), then adjudicate.
• Sample on purpose for rare but important classes (hypothetical, family history, ADEs), and size the test set so per-class confidence intervals are useful. See test set sample size.
• Choose who annotates (clinicians, trained annotators, or a mix) based on the layer. See choosing a labeling approach.

Key references

• Uzuner, South, Shen & DuVall, JAMIA 2011;18(5):552–6: the 2010 i2b2/VA concepts, assertions and relations challenge.
• Stubbs, Kotfila & Uzuner, J Biomed Inform 2015;58 Suppl:S11–S19: the 2014 i2b2/UTHealth de-identification track.
• Henry et al., JAMIA 2020;27(1):3–12: the 2018 n2c2 ADE and medication extraction track.
• MIMIC-IV-Note v2.2 (Johnson et al., PhysioNet, 2023): de-identified discharge summaries and radiology reports.
• HHS guidance on HIPAA de-identification: Safe Harbor and Expert Determination.

Related resources

• Medical data labeling
• How to evaluate an AI medical scribe
• How to evaluate a health LLM
• Inter-rater agreement for medical data labeling
• Medical data annotation guideline template
• How big should a medical AI test set be?
• Choosing a medical data labeling approach