Learn how we're leveraging AI and Natural Language Processing (NLP) to advance medical education assessment — from delivering actionable feedback that enhances skill development and better facilitates learning to constructing new scoring systems and more.
U
Is your program built on a collection of assessments or a system?
This post is the fifth and final in a five-part series drawing on Formative Assessment and Feedback in Medical Education: A Practical Guide, AMEE Guide No. 189. Each post takes a question medical educators are actively grappling with and work through what the research and the practice actually suggest.
This post is the fourth in a five-part series drawing on Formative Assessment and Feedback in Medical Education: A Practical Guide, AMEE Guide No. 189. Each post will take a question medical educators are actively grappling with and work through what the research and the practice actually suggest.
This post is the third in a five-part series drawing on Formative Assessment and Feedback in Medical Education: A Practical Guide, AMEE Guide No. 189. Each post will take a question medical educators are actively grappling with and work through what the research and the practice actually suggest.
This post is the second in a five-part series drawing on Formative Assessment and Feedback in Medical Education: A Practical Guide, AMEE Guide No. 189. Each post will take a question medical educators are actively grappling with and work through what the research and the practice actually suggest.
This post is the first in a five-part series drawing on Formative Assessment and Feedback in Medical Education: A Practical Guide, AMEE Guide No. 189. Each post will take a question medical educators are actively grappling with and work through what the research and the practice actually suggest.
Learn why strong, patient-centered communication remains vital as AI becomes more embedded in medical education and health care. This webinar explores why communication skills are more critical than ever, how they must evolve in AI-enabled environments, and the role of formative assessment, feedback, and reflection in developing empathetic, responsive clinicians.
Communication Learning Assessment: Improving Patient-Centered Communication Skills
In this video, Director of Curriculum Affairs Stephanie Mann, MD, MS, HPED discusses how faculty can use the Communication Learning Assessment (CLA) to track student progress and view class-level performance data on key communication behaviors.
Communication Learning Assessment: The Importance of Formative Assessment and Feedback
In this video, Vice President of the Office of Research Strategy Krista Mattern, PhD explains how the Communication Learning Assessment (CLA) was designed and discusses the results of an efficacy study used to evaluate the effectiveness of the tool.
Assessment for Learning in Medical Education: Supporting Students’ Growth and Development
Learn about the impact of formative assessment — also known as assessment for learning — on medical education and discover strategies for implementing this approach within your own curriculum to support student growth and development.
Formative assessment and feedback in medical education: A practical guide
In September 2025, NBME researchers published a paper to provide medical educators with guidance on formative assessment and feedback in medical education. By addressing persistent challenges and illustrating principles with real-world examples, the paper provides actionable strategies for educators to support continuous learner growth and skill development. Learn more in the AMEE Guide today:
In this video, Anastasiya Lipnevich, PhD, Professor of Educational Psychology, CUNY and NBME Visiting Scholar, discusses how assessment for learning, or formative assessment, can cultivate medical students’ competencies and help educators better address individual needs.
How can guided reflection and thoughtful feedback transform learners into empathetic and effective communicators?
Anastasiya Lipnevich, PhD, Professor of Educational Psychology, CUNY and NBME Visiting Scholar, provides insights on effective feedback strategies in medical education and key considerations for guiding learners toward clinical competence and professional growth.
In this Ask the Expert video, Senior Measurement Scientist Ann King, PhD defines the different facets of communications skills and explains why this area is both so challenging and important to assess.
Lung cancer screening: What do physicians say to reluctant patients and what do patients want them to say?
In this paper, Research Associate Angelo D’Addario, Senior Manager of Measurement Science Ann King, and their coauthors examine the communication strategies physicians employ when discussing lung cancer screening with reluctant patients, and compares these approaches to the preferences of the patients themselves. The study aims to identify potential gaps between physician communication techniques and patient expectations,…
How can formative assessment actively cultivate medical learners’ competencies and help educators better address individual needs?
Anastasiya Lipnevich, PhD, Professor of Educational Psychology, CUNY and NBME Visiting Scholar, shares her insights on the components of formative assessment – or assessment for learning – that incorporates learning objectives and timely feedback to support a system focused on improving learner competencies throughout their journey.
Vice President of Educational Strategy Christopher Feddock, MD shares his thoughts on the challenges of assessing clinical reasoning and the steps NBME is taking to address this critical area within medical education.
Creating community: The OSCE for clinical reasoning project
Hear from members of the OSCE for Clinical Reasoning Creative Community as they share insights and learnings from this two-year journey and discuss how this work has impacted their own approach to teaching and assessment.
Innovative Uses of AI in Medical Education Assessment
NLP Data Scientist, Victoria Yaneva, discusses NBME’s approach to innovation in AI solutions for assessment, driving research excellence, and promoting transparency through collaboration and active community engagement.
In this video from our Ask the Expert series, Manager of NLP Research Victoria Yaneva, PhD discusses the potential of NLP and AI within medical education assessment and the importance of a responsible approach in applying this technology.
In the first video from our Ask the Expert series, Senior Measurement Scientist Christopher Runyon, PhD discusses clinical reasoning and the importance of research in this area to improve teaching and learning.
The future of natural language processing (NLP) in medical education assessment
Read some of the highlights from recent articles published by NBME researchers as they continue to explore the use of NLP in transforming assessment development and scoring.
In this Ask the Expert video, Senior Measurement Scientist Monica Cuddy, PhD, discusses the importance of test validity and shares an example of a validity study being conducted on a new item type designed to assess clinical reasoning.
Improving how we evaluate clinical reasoning skills through innovations in assessment
Senior Measurement Scientist Christopher Runyon discusses his background, research in using Natural Language Processing to assess clinical reasoning and what drives his passion for the work.
Assessment of clinical skills: a case study in constructing an NLP-based scoring system for patient notes
In this chapter of Advancing Natural Language Processing in Educational Assessment, Director of Research Polina Harik, PhD, Measurement Scientist Janet Mee, PhD, Senior Measurement Scientist Christopher Runyon, PhD and Distinguished Research Scientist Brian Clauser, PhD describe INCITE, a natural language processing (NLP)-based system for scoring free-text responses. The system was developed by NBME to score responses produced as…
What are the challenges of assessing clinical reasoning?
Members of NBME’s OSCE for Clinical Reasoning Creative Community give insight into the importance and challenges of assessing clinical reasoning and share their experience working on this collaborative initiative so far.
The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156
In this AMEE guide, Lead Measurement Scientist Monica Cuddy, PhD and coauthors describe the practical considerations involved in conducting and interpreting medical education studies using AI approaches. After introducing basic terminology and identifying which problems and data are well-suited for the use of different AI methods, they then address how to evaluate methodological rigor and consider…
In this video from our Ask the Expert series, Senior Measurement Scientist Su Somay, PhD discusses the importance of construct development and shares how the use of evidence-centered design (ECD) contributes to the validity of this process.
Video-based communication assessment for weight management counseling training in medical residents: a mixed methods study
In this study, Manager of Research Analysis Angelo D’Addario, Ann King, PhD, and coauthors demonstrate how the Video-based Communication Assessment (VCA) can be used to help train medical residents on delivering weight management counseling. In VCA debriefing interviews with residents, five themes emerged:
Effects of practicing with and obtaining crowdsourced feedback from the Video-Based Communication Assessment app on resident physicians’ adverse event communication skills: pre-post trial
In this study, Manager of Research Analysis Angelo D’Addario, Ann King, PhD, and coauthors evaluate the efficacy of using Video-Based Communication Assessment (VCA) practice and feedback as a stand-alone intervention for the development of residents’ error disclosure skills. The results show that residents at all training levels can potentially improve error disclosure skills with VCA….
Leveraging technology to keep assessments up-to-date, relevant and flexible without sacrificing quality
From finding better ways to develop exam content to discovering innovative methods for scoring competency-based assessments, NBME is exploring how technology can be used to solve the most pressing challenges facing the medical education community.
SHARP (SHort Answer, Rationale Provision): A new item format to assess clinical reasoning
In this study, Senior Measurement Scientist Christopher Runyon, PhD, Director of Research Polina Harik, PhD, and their coauthors looked at a new 2-step item format called SHARP (SHort Answer, Rationale Provision). In collaboration with physician-educators, they integrated short-answer questions into a patient medical record–based item starting in October 2021 and arrived at an innovative item…
“Best practice” for patient-centered communication: A narrative review
In this narrative review, Senior Manager of Measurement Science Ann King, MA and her coauthor, Ruth Hoppe, MD, aim to summarize the work that has been conducted in physician-patient communication that supports the efficacy of good communications skills. A review of the literature shows it contains impressive evidence supporting positive associations between physician communication behaviors…
Innovative Uses of AI in Medical Education Assessment
NLP Data Scientist, Victoria Yaneva, discusses NBME’s approach to innovation in AI solutions for assessment, driving research excellence, and promoting transparency through collaboration and active community engagement.
In this video from our Ask the Expert series, Manager of NLP Research Victoria Yaneva, PhD discusses the potential of NLP and AI within medical education assessment and the importance of a responsible approach in applying this technology.
The future of natural language processing (NLP) in medical education assessment
Read some of the highlights from recent articles published by NBME researchers as they continue to explore the use of NLP in transforming assessment development and scoring.
