How to evaluate an AI medical scribe (ambient clinical documentation): hallucination and omission rates, PDQI-9, a pilot protocol and vendor questions

The short answer

To evaluate an AI medical scribe (ambient clinical documentation), have blinded clinicians compare each AI-drafted note against the recording or transcript of the visit. Count two kinds of error separately. A hallucination is something in the note that did not happen in the visit. An omission is something clinically relevant that happened but is missing from the note. Grade each error by whether it could change diagnosis or management. Score overall note quality with a validated rubric such as PDQI-9. Run the same test on every vendor with the same recordings, then pilot the winner in real clinics with monitoring.

Do not rely on clinician satisfaction alone. In one head-to-head study, the scribe physicians liked best was not the one with the fewest errors on a line-by-line audit. A peer-reviewed VA evaluation calls for “independent, vendor-neutral evaluations of note quality” before large-scale deployment.

Sources: Asgari et al., npj Digit Med 2025; Stetson et al., Appl Clin Inform 2012 (PDQI-9); Weiss et al., JAMIA 2026; Reddy et al., Ann Intern Med 2026. This page is a planning guide, not legal, regulatory or clinical advice.

What to measure: hallucinations, omissions and severity

The most detailed published method checks every sentence. Two clinicians review each note. Every note sentence must be supported by the transcript; if it is not, it is a hallucination. Every transcript sentence is checked for clinically relevant content that the note leaves out; that is an omission. Errors that could change diagnosis or management if not corrected are graded major, the rest minor. A senior clinician resolves disagreements.

Using that method on 450 notes made from simulated primary care transcripts, the authors found hallucinations in 191 of 12,999 note sentences (1.47%) and omissions in 1,712 of 49,590 transcript sentences (3.45%). 44% of hallucinations and 16.7% of omissions were major. Hallucinations were fabrications (43%), negations (30%), context errors (17%) and wrong causality (10%), and the major ones were most common in the plan. Note that the two rates use different denominators, so they are not directly comparable. All authors worked for a scribe vendor, so treat the numbers as an example of the method, not a benchmark.

Whatever you count, state the denominator. “Percent of sentences” and “percent of notes with at least one error” give very different numbers for the same notes. The metrics we would track:

The CHAI framework names similar measures, including the share of notes with at least one clinically significant error, recall of medical terms reported separately for diagnoses, medications, procedures, labs, symptoms and allergies, and a transcription equity measure comparing the worst and best demographic subgroups. CHAI says its benchmark values are “reference points drawn from the cited literature, not universal pass/fail cut points,” and recommends setting thresholds against your own baseline.

Sources: Asgari et al., npj Digit Med 2025; CHAI Ambient AI Testing & Evaluation Framework; Wang et al., npj Digit Med 2025. The metrics table is our practice summary.

Note-quality rubrics: PDQI-9 and its variants

The Physician Documentation Quality Instrument (PDQI-9) has nine items. Each is rated on a 5-point scale, with “Not at all” over 1 and “Extremely” over 5, and a description of the ideal note anchoring the top. The full instrument is in the paper’s appendix.

• Up-to-Date
• Accurate
• Thorough
• Useful
• Organized
• Comprehensible
• Succinct
• Synthesized
• Consistent

In its validation study on admission, progress and discharge notes, PDQI-9 had an inter-rater intraclass correlation of .83 and internal consistency (Cronbach’s alpha) of .87 to .94. It grew out of an earlier 22-item instrument (Stetson et al., JAMIA 2008).

Know its limits before you use it on AI notes. The authors state that PDQI-9 is not designed to check whether specific note components are present or absent, and that other tools are needed to assess redundancy. A high PDQI-9 score does not prove a note is free of hallucinations. Pair it with the error audit above.

Most scribe studies modify the rubric. A VA evaluation used a 10-domain version (maximum 50). Another study used 11 criteria plus a yes/no hallucination check. For LLM summaries, the PDSQI-9 adapts the idea and was validated with 7 physician raters on 779 summaries (Cronbach’s alpha 0.879, ICC 0.867). If you modify a rubric, publish the exact version and check rater agreement on your own notes.

Sources: Stetson et al., Appl Clin Inform 2012 (PDQI-9); Stetson et al., JAMIA 2008; Reddy et al., Ann Intern Med 2026; Palm et al., Front Artif Intell 2025; Croxford et al., JAMIA 2025 (PDSQI-9).

What the published evaluations show

Two lessons stand out. First, results depend heavily on the case: in the VA study the low back pain case had the largest gap (human 43.8 vs AI 20.3 out of 50). Second, how you run the study changes the answer. In an emergency department crossover of two scribes, apparent differences mostly reflected which tool clinicians used first. In simulated English and Spanish visits, scribes carried interpreter errors into the notes, so a scribe can only be as accurate as the conversation it hears.

Sources: Asgari et al. 2025; Reddy et al., Ann Intern Med 2026; Palm et al., Front Artif Intell 2025; Weiss et al., JAMIA 2026; Guo et al., JAMIA 2026; Lukac et al., NEJM AI 2025; Afshar et al., NEJM AI 2025; Webb et al., Appl Clin Inform 2026; Rabotin et al., JMIR Med Inform 2026. Conflict-of-interest notes are from the articles’ own disclosures.

A pilot test protocol

1. Set the decision first. Write down the specialties, visit types and languages in scope, the metrics, and the error rates you would accept, set against your current notes.
2. Build a test set. Record standardized visits with standardized patients (as the VA did), plus consented real visits where policy allows. Include hard cases: multiple problems, medication changes, interpreters, accents, background noise and more than two speakers.
3. Give every vendor the same audio. Same recordings, same note template, same settings. Log the product version.
4. Blind the raters. Strip vendor formatting where you can and mix in human-written notes as a control.
5. Run two reviews per note. A PDQI-9 (or documented modified) rating, and a sentence-level audit against the transcript for hallucinations and omissions, graded major or minor. Use two independent clinician reviewers with adjudication, and report agreement.
6. Break results down. Report by visit type, specialty, note section (the plan matters most), speaker accent and language, and interpreter use.
7. Pilot in clinic. Randomize or stagger rollout if you can, and counterbalance tool order if you compare two. Track time in notes, clinician edits by category, and safety reports.
8. Keep monitoring. Re-audit a sample of signed notes on a schedule and after every model or product update.

Sources: Reddy et al., Ann Intern Med 2026; Weiss et al., JAMIA 2026; Asgari et al. 2025; Webb et al., Appl Clin Inform 2026; Guo et al., JAMIA 2026; CHAI framework. The protocol is our practice summary.

Questions to ask scribe vendors

• What are your hallucination and omission rates, how were they measured, by whom, and with what denominator? Was the evaluation independent?
• Can we test on our own recordings, before signing, with a version that will match production?
• How is the model versioned, how are we notified of changes, and how do you monitor for drift?
• How does the product enforce clinician review and sign-off before a note enters the record?
• How long are audio and transcripts kept, where, and can we set retention ourselves?
• Is our data used to train or improve models? Can we opt out, in the contract?
• Which sub-processors and model providers handle our data, in which countries, and will you sign a BAA?
• How does transcription accuracy vary by accent, language and interpreter use, and can you show the data?
• Can we export notes, transcripts and audit logs for our own review?

Sources: CHAI Ambient AI Testing & Evaluation Framework (sign-off enforcement, retention, training-data use, sub-processor and data residency controls, transcription equity). The question list is our practice summary.

Governance and patient communication

NHS England guidance for professionals using ambient scribes says to tell patients at the start of a session that the tool is being used and to respect objections. It says explicit consent is generally not needed for individual care in England, but local rules differ elsewhere. The clinician remains “responsible and accountable” for the accuracy of the record, should check outputs every time, and should take extra care when a scribe has translated. For organizations, NHS England has also published the NHS England, Guidance on the use of AI-enabled ambient scribing products in health and care settings (April 2025).

In the US, the Coalition for Health AI released a voluntary, vendor-agnostic implementation playbook and testing and evaluation framework for ambient AI in July 2026, written by a work group of health systems, practices and ambient AI developers. Its best practices include treating ambient documentation as a recording governed by state law and building workflows for patients to withdraw consent mid-visit (CHAI announcement (July 2026)). For HIPAA questions about sending visit audio and transcripts to an AI service, see our guide to LLM APIs, PHI and BAAs below.

Sources: NHS England guidance for health and care professionals; CHAI announcement (July 2026); CHAI Ambient AI Testing & Evaluation Framework. England-specific rules do not apply in the US.

Related resources

• Healthcare AI consulting
• Expert clinical labeling
• How to evaluate a health LLM
• LLM APIs, PHI and HIPAA BAAs
• Inter-rater agreement for medical data labeling
• Medical data annotation guideline template
• Clinical prediction model validation