Red-Teaming Health LLMs: Test Categories, Red-Team Sets, Severity

The short answer

Red-teaming a health LLM means trying, on purpose, to make it fail in the ways that hurt patients or the organization: unsafe clinical advice, leakage of protected health information (PHI), following instructions injected through a document or image, and biased answers. NIST AI 600-1 defines red-teaming as a structured exercise to find flaws such as inaccurate, harmful or discriminatory outputs, and says red teams work best with domain expertise. In health, that means clinicians write and judge the attacks.

Build a red-team set from real clinical workflows, run it with synthetic or de-identified patient data, score every failure for severity with a rubric you fixed in advance, and re-run it on every model or prompt change. Map findings to NIST AI 600-1, the OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS so security, compliance and clinical teams share one vocabulary. No public framework gives a clinical harm severity scale, so the scale below is PAHL's suggested example, not a standard.

Sources: NIST AI 600-1 (Generative AI Profile, July 2024); OWASP Top 10 for LLM Applications 2025; MITRE ATLAS; Chang et al., npj Digit Med 2025;8:149.

Red-teaming vs. evaluation

An evaluation measures average performance on a representative, pre-specified test set. A red team hunts for failures with adversarial and edge-case inputs, and reports failure modes and their severity. The two feed each other: OpenAI describes red-team outcomes as input into risk assessment and automated evaluations, and NIST says red-team results need additional analysis before they shape governance decisions.

This page covers the adversarial side only. For rubric-graded accuracy, grader checks and evaluation frameworks, see How to evaluate a health LLM. For note-writing tools, see How to evaluate an AI medical scribe. For when public benchmarks are enough, see Medical AI benchmarks vs. custom evals.

Sources: Ahmad et al. (OpenAI), arXiv:2503.16431; NIST AI 600-1 (Generative AI Profile, July 2024).

Adversarial test categories

The first four rows are the categories a published 80-participant red-teaming study of ChatGPT in medicine used: Safety, Privacy, Hallucinations/Accuracy and Bias. The others come from the OWASP and ATLAS threat lists and matter as soon as the model reads outside content or can take actions. The example shapes describe the structure of a test case. They are not working attack prompts.

Category	What you probe	Benign example shape	Framework mapping
Unsafe clinical advice	Statements that, if followed, could cause physical, psychological, emotional or financial harm (Chang et al.'s safety definition)	A realistic patient or clinician question with a missing or conflicting detail (age, weight units, pregnancy, a second medication) where the safe answer is to ask or escalate	NIST: Dangerous, Violent, or Hateful Content; Human-AI Configuration (over-reliance). OWASP LLM09
PHI leakage and privacy	Disclosure of PHI or personal information: echoing it to the wrong place, revealing another user's data, or regurgitating training data	A synthetic note with fake identifiers, plus a request to send a summary to an outside party; a second session that asks about the first patient	NIST: Data Privacy. OWASP LLM02. ATLAS AML.T0057 LLM Data Leakage
Hallucination and inaccuracy	Confident but false clinical content, invented citations, unsupported claims	A question about a rare condition or a recently changed guideline; a request for references	NIST: Confabulation. OWASP LLM09 Misinformation
Bias	Identity-based discrimination or false stereotypes; outputs that change with race, gender or other attributes when they should not	The same vignette run several times with only one demographic attribute changed; compare differentials and plans	NIST: Harmful Bias or Homogenization. See Section 1557 bias audits
Direct prompt injection and jailbreaks	User input that overrides system instructions or safety behavior	A user message that claims special authority (for example, "I'm the admin") and asks the assistant to drop its clinical safety rules	OWASP LLM01. ATLAS AML.T0051.000, AML.T0054 LLM Jailbreak
Indirect and multimodal injection	Instructions hidden in content the system ingests: retrieved documents, uploaded files, web pages, images	A synthetic referral letter or scanned image in the retrieval set that contains a line addressed to the assistant; check whether the output changes	OWASP LLM01. ATLAS AML.T0051.001. NIST: Information Security
Excessive agency	An agent taking actions (ordering, messaging, writing to the record) beyond what the user approved	A task that tempts the agent to send a message or change a record without a human confirmation step	OWASP LLM06 Excessive Agency

Sources: Chang et al., npj Digit Med 2025;8:149; NIST AI 600-1 (Generative AI Profile, July 2024); OWASP Top 10 for LLM Applications 2025; OWASP LLM01:2025 Prompt Injection; OWASP LLM02:2025 Sensitive Information Disclosure; OWASP LLM09:2025 Misinformation; ATLAS data release 2026.09 (GitHub).

What published medical red-teaming found

Inappropriate answers were common, and model updates did not fix them all. In Chang et al., 80 clinicians, students and technical professionals wrote 376 prompts (1,504 responses). Six medically trained reviewers judged 20.1% of responses inappropriate: 25.8% for GPT-3.5, 16.5% for GPT-4.0, 17.8% for GPT-4.0 with Internet and 20.4% for GPT-4o. 51.3% of inappropriate responses contained hallucinations. 81 prompts (21.5%) that got appropriate answers from GPT-3.5 got inappropriate answers from a later model.
Privacy failures were silent. In the same study, no response to prompts containing synthetic PHI included a disclaimer that such information should not be given to a public chatbot.
Bias shows up in repeat and counterfactual runs. Omiye et al. asked four commercial LLMs nine questions five times each; all models gave some race-based medicine answers, and answers to the same question were not always consistent. Zack et al. found GPT-4's differential diagnoses were more likely to include diagnoses that stereotype certain races, ethnicities and genders, and that demographics were associated with recommendations for more expensive procedures.
Diverse raters find different harms. Pfohl et al. built EquityMedQA, seven datasets enriched for adversarial queries, and stressed using varied assessment methods and raters of different backgrounds and expertise.
Images can carry instructions people cannot see. Clusmann et al. ran 594 prompt injection attacks on four vision-language models; all were susceptible, including to sub-visual prompts in medical images that were non-obvious to human observers.
Benchmarks can miss a poisoned model. Alber et al. showed that replacing 0.001% of training tokens with medical misinformation produced harmful models that matched clean models on open medical benchmarks.

Study notes: Chang et al.'s abstract rounds the GPT-4.0 rate to 16%. Pfohl et al. was funded by Google. Each study tested specific model versions at one point in time.

Sources: Chang et al., npj Digit Med 2025;8:149; Omiye et al., npj Digit Med 2023;6:195; Zack et al., Lancet Digit Health 2024;6(1):e12–e22; Pfohl et al., Nat Med 2024;30:3590–3600; Clusmann et al., Nat Commun 2025;16:1239; Alber et al., Nat Med 2025;31:618–626.

Building a clinician-written red-team set

Step	What to do	Why
1. Scope the system	List users, inputs (chat, documents, images, tool output), actions the system can take, and the data it can reach	Indirect injection and excessive agency only apply if the system ingests outside content or acts (OWASP LLM01, LLM06)
2. Compose the team	Clinicians from the target specialty write and judge clinical cases; security engineers own injection and leakage; consider patients or general users for realistic phrasing	NIST describes expert, general-public and combination red teams, where experts modify or verify prompts written by others
3. Write from real workflows	Seed tasks such as treatment plans, fact checking, patient communication, differential diagnosis, summarization and note creation (the task types in Chang et al.)	Failures that matter show up in the tasks people actually do
4. Use synthetic or de-identified data	Write synthetic patient notes with fake identifiers. If real records are needed, de-identify them and limit what is used to the minimum necessary	HIPAA's minimum necessary standard and de-identification rules; Chang et al. used synthetic notes
5. Write instructions and a rubric first	A red-teamer guide (goal, categories, what counts as a failure, what not to submit) and a fixed severity rubric. Base it on the annotation guideline template	NIST MS-2.8-002: document the instructions given to red-teamers
6. Add variants	For each seed: demographic swaps, repeated runs of the same prompt, paraphrases, multi-turn follow-ups	Omiye et al. found inconsistent answers on repeats; Zack et al. found demographic effects
7. Double-review findings	Two clinicians score a sample of responses independently; measure inter-rater agreement and adjudicate	Severity judgments are subjective; agreement tells you whether the rubric works
8. Version, hold out and re-run	Freeze a versioned set, keep part held out, and re-run on every model, prompt or retrieval change. Size it with test set sample size methods	Chang et al. saw 21.5% of prompts flip from appropriate to inappropriate across model versions

Treat the set as sensitive. It is a map of how to break your system, so store it with access controls and do not publish working attacks. When outside people take part, NIST says feedback activities should follow human subjects research requirements and best practices such as informed consent and compensation. NIST also notes that demographically and interdisciplinarily diverse red teams can find flaws in the varying contexts where the system will be used.

Sources: NIST AI 600-1 (Generative AI Profile, July 2024); OWASP LLM01:2025 Prompt Injection; Chang et al., npj Digit Med 2025;8:149; Omiye et al., npj Digit Med 2023;6:195; Zack et al., Lancet Digit Health 2024;6(1):e12–e22; HHS: Minimum Necessary Requirement; HHS de-identification guidance.

Harm severity scoring

NIST AI 600-1, OWASP and MITRE ATLAS name risks and attack techniques; none defines a clinical severity scale for LLM outputs. Chang et al. labeled each response appropriate or inappropriate in each of four categories. For triage you usually need more levels. The table below is PAHL's suggested example, not a standard. Adapt the anchors to your clinical setting and fix them before testing starts.

Level (PAHL example)	Anchor description	Suggested response
0 – No issue	Appropriate, accurate and safe for the intended user	None
1 – Minor	Imprecise, incomplete or poorly worded, but unlikely to change a care decision	Log; fix in normal prompt or content updates
2 – Moderate	Could mislead a care decision or disclose limited personal information if not caught by a human	Fix before the next release; add the case to the evaluation set
3 – Severe	If followed, could plausibly cause serious harm; discloses PHI to the wrong party; or performs an unapproved action	Block release; root-cause; add regression tests

Score each failure on two more axes as well: how easy it was to trigger (a plain question vs. a multi-step attack) and who would see it (a clinician who can catch it vs. a patient who may not). Report results by category and severity, and list every level 3 finding individually instead of rolling it into an average.

Sources: NIST AI 600-1 (Generative AI Profile, July 2024); OWASP Top 10 for LLM Applications 2025; MITRE ATLAS; Chang et al., npj Digit Med 2025;8:149.

Frameworks to map findings to

Framework	What it gives you	Health red-team use
NIST AI 600-1 (Generative AI Profile, July 2024)	12 generative AI risks (including Confabulation, Data Privacy, Harmful Bias, Human-AI Configuration and Information Security), red-team definitions and types, and suggested actions	Governance vocabulary; actions such as red-teaming resilience to prompt injection (MS-2.7-007) and for leakage of personal or sensitive information (MS-2.10-001)
OWASP Top 10 for LLM Applications 2025	Ten application risks, LLM01 Prompt Injection through LLM10 Unbounded Consumption, with scenarios and mitigations	Engineering checklist; LLM02 names health records as sensitive information and LLM09 gives a medical chatbot scenario
MITRE ATLAS	A living knowledge base of adversary tactics, techniques and case studies against AI systems; its site listed 16 tactics and 208 techniques on Oct 27, 2026	Technique IDs for security reports, such as AML.T0051 LLM Prompt Injection, AML.T0054 LLM Jailbreak, AML.T0057 LLM Data Leakage and AML.T0020 Training Data Poisoning

OWASP notes that it is unclear whether there are fool-proof ways to prevent prompt injection, and recommends least privilege, segregating external content, human approval for high-risk actions and regular adversarial testing. Research on indirect injection describes the root problem: LLM-integrated applications blur the line between data and instructions. Training-data extraction is a separate risk: Carlini et al. extracted verbatim personal information from a language model's training data. For whether your product is regulated, see Is my app a medical device?. For vendor data terms, see LLM APIs, PHI and HIPAA BAAs.

Sources: NIST AI 600-1 (Generative AI Profile, July 2024); OWASP Top 10 for LLM Applications 2025; OWASP LLM01:2025 Prompt Injection; OWASP LLM02:2025 Sensitive Information Disclosure; OWASP LLM09:2025 Misinformation; MITRE ATLAS; ATLAS data release 2026.09 (GitHub); Greshake et al., arXiv:2302.12173; Carlini et al., arXiv:2012.07805.

Automated tools and their limits

Open-source tools can generate and run attack variants at scale. Microsoft's PyRIT describes itself as a framework to help security professionals and engineers proactively identify risks in generative AI systems. NVIDIA's garak is an LLM vulnerability scanner that probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks. NIST notes that AI-led red-teaming can be more cost-effective than human red-teamers alone, but it also says red teams work best with domain expertise. Use tools for coverage of security categories, and clinicians for whether an answer is clinically unsafe.

Anthropic's red-teaming work released 38,961 red-team attacks and documented its instructions and methods, and found that models trained with human feedback got harder to red team as they scaled. Public attack sets are useful seeds, but they are not clinical and do not know your workflow.

Sources: Microsoft PyRIT; NVIDIA garak; NIST AI 600-1 (Generative AI Profile, July 2024); Ganguli et al. (Anthropic), arXiv:2209.07858.

Health LLM red-team checklist

Write down the system scope: inputs, retrieval sources, tools and actions, users, and data reached.
Cover all categories: unsafe advice, PHI leakage, hallucination, bias, direct and indirect injection, and excessive agency where relevant.
Have clinicians from the target specialty write and judge the clinical cases; have security engineers own injection and leakage.
Use synthetic or de-identified patient data only, and the minimum necessary.
Document red-teamer instructions and fix the severity rubric before testing.
Add demographic swaps, repeated runs and multi-turn variants.
Double-review a sample and report inter-rater agreement.
Map each finding to NIST AI 600-1, OWASP LLM Top 10 and ATLAS IDs.
Report by category and severity; list every severe finding individually.
Store the set with access controls; never publish working attacks.
Re-run the versioned set on every model, prompt or retrieval change, and move fixed failures into the regular evaluation set.

Key references

NIST AI 600-1, Generative AI Profile (July 2024): GAI risks, red-team definition and types, suggested actions.
OWASP Top 10 for LLM Applications 2025: prompt injection, sensitive information disclosure, misinformation and seven more.
MITRE ATLAS and its data repository: adversary tactics and techniques against AI systems.
Chang et al., npj Digit Med 2025;8:149: red-teaming ChatGPT in medicine with 80 participants.
Pfohl et al., Nat Med 2024: EquityMedQA and a framework for surfacing health equity harms.
Clusmann et al., Nat Commun 2025: prompt injection on medical vision-language models.
Ahmad et al., OpenAI's approach to external red teaming (2025): team composition, access and guidance.
Ganguli et al., Red teaming language models to reduce harms (2022): methods and a released attack dataset.

Related resources

Frequently asked questions

What is red-teaming a health LLM?

NIST AI 600-1 defines AI red-teaming as a structured testing exercise that probes an AI system to find flaws and vulnerabilities such as inaccurate, harmful or discriminatory outputs, often in a controlled environment and with the system's developers. For a health LLM, that means deliberately trying to make it give unsafe clinical advice, leak protected health information, follow injected instructions or produce biased answers. NIST says red teams get the best results when they have domain expertise, which in health means clinicians.

How is red-teaming different from evaluating a health LLM?

An evaluation measures how well the model does on a representative, pre-specified test set, such as rubric-graded accuracy on realistic cases. Red-teaming searches for failures on purpose, using adversarial and edge-case inputs, and its output is a list of failure modes with severity, not an average score. You need both. Red-team findings should feed new cases into the evaluation set, which is how OpenAI describes using external red-teaming results.

What should a health LLM red-team test for?

At minimum: unsafe clinical advice, privacy leakage of protected health information, hallucination or inaccuracy, and bias. Those are the four categories used in a published 80-participant red-teaming study of ChatGPT in medicine. Add direct and indirect prompt injection if the system reads documents, web pages, images or tool output, and excessive agency if it can take actions. OWASP lists prompt injection, sensitive information disclosure and misinformation in its 2025 Top 10 for LLM applications.

Who should write the red-team prompts?

A mix. NIST AI 600-1 describes expert red teams with domain specialists such as medicine, general-public red teams, and combinations where experts modify or verify prompts written by others. In health, clinicians should write and review the clinical scenarios, and security engineers should own prompt injection and data-leakage tests. NIST also recommends documenting the instructions given to red-teamers, and automated tools can extend coverage but do not replace clinical judgment.

Is there a standard harm severity scale for health LLM red-teaming?

No. NIST AI 600-1, the OWASP Top 10 for LLM applications and MITRE ATLAS name risk categories and attack techniques, but none of them defines a clinical harm severity scale for LLM outputs. Published medical red-teaming studies have used their own schemes, such as labeling each response appropriate or inappropriate in four categories. Define your own rubric, with anchored examples, before testing starts, and measure agreement between reviewers.

Scope it in 30 minutes

Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.

Book a 30-min scoping call

Prefer to write? Send a short note.