How to Evaluate a Health LLM

A practical sequence

1. Define the task and the harm. Write down who uses the feature, what decision it touches, and what a harmful answer looks like. This drives everything else, including whether FDA rules may apply.
2. Don't stop at exam questions. Multiple-choice medical exam benchmarks measure knowledge recall, not how a model behaves in a conversation. Singhal K et al., “Large language models encode clinical knowledge,” Nature 2023 paired exam-style datasets with human evaluation across axes such as factuality, possible harm and bias for this reason.
3. Grade against rubrics written by clinicians. HealthBench grades multi-turn health conversations against physician-written rubrics: 5,000 conversations, 48,562 unique rubric criteria, built with 262 physicians (OpenAI, “Introducing HealthBench” (May 12, 2025); Arora RK, Wei J et al., HealthBench, arXiv:2505.08775 (2025)). Its axes include accuracy, communication quality, context awareness and instruction following. You can adopt the rubric approach for your own task. (We don't reproduce HealthBench examples here, at OpenAI's request.)
4. Measure reliability, not just the average. Run each case several times and look at the worst response, not only the mean. HealthBench reports worst-of-n reliability for this reason.
5. Check the grader. If a model grades responses, measure its agreement with clinicians on a sample before trusting it at scale. HealthBench reports this kind of meta-evaluation.
6. Map to a risk framework. Use the Measure function of NIST AI RMF 1.0 (NIST AI 100-1) to document what you test for and why.
7. Move from offline to live evaluation carefully. Offline scores don't show workflow effects. Vasey B et al., DECIDE-AI, Nat Med 2022 describes how to report early live clinical evaluation of AI decision support.
8. Check the regulatory question. Some LLM features may be device software functions. Walk through FDA, Clinical Decision Support Software guidance, and see FDA, Artificial Intelligence-Enabled Medical Devices.

The regulatory step is not legal or regulatory advice; consult qualified counsel.

What it means for startups

• Build a private, versioned eval set from your own task and users; re-run it whenever the underlying model changes.
• Report safety and worst-case behavior alongside averages.
• Write the rubric before you see model outputs.

Go deeper

• LLM APIs, PHI and HIPAA BAAs
• Inter-rater agreement for clinician raters
• Is my app a medical device?

Sources (accessed October 27, 2026)

1. OpenAI, “Introducing HealthBench” (May 12, 2025)
2. Arora RK, Wei J et al., HealthBench, arXiv:2505.08775 (2025)
3. Singhal K et al., “Large language models encode clinical knowledge,” Nature 2023
4. NIST AI RMF 1.0 (NIST AI 100-1)
5. Vasey B et al., DECIDE-AI, Nat Med 2022
6. FDA, Clinical Decision Support Software guidance
7. FDA, Artificial Intelligence-Enabled Medical Devices

Related resources

• Healthcare AI consulting
• Expert clinical labeling
• SaMD consulting
• How to evaluate an AI medical scribe