A primer on evaluation methods for large language models in healthcare

Resource information

Authors

Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan, Timothy J Keyes, Kameron C Black, John L Havlik, David JH Wu, Jack W O'Sullivan, Ethan Goh, Sarah F Mercaldo, James M Hillis

Research area

Language technology

Application domain

Healthcare

AI methods

Large language models, Machine learning

Research topics

cs.CL, cs.AI

arXiv record date

2026-10-15

Publication

arXiv

Upstream last modified

Oct 16, 2026

Last record update

Nov 4, 2026

Abstract

Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and not harm. This evaluation can be more challenging than traditional machine learning for many reasons, including probabilistic and open-ended outputs, and behavior that shifts with prompt design and accumulated context. This review covers four key areas of LLM evaluation: principles of study design, statistical methods, capability evaluation and clinical application evaluation. Capability evaluation considers different benchmarks, including multiple-choice, agentic and multi-turn benchmarks, alongside operational metrics like token usage and latency. Clinical application evaluation addresses establishing accuracy of free text outputs, such as human review and LLM-as-a-judge, and clinical trial approaches. Across sections, we describe underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question. Together, this article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs.