The Video-Based Communication Assessment (VCA) from NBME is a unique formative assessment tool designed to improve physicians’ patient-centered communication skills. Using an app that’s accessible on both desktop and mobile devices, physicians record their spoken responses to a series of vignettes that simulate real-life telemedicine appointments.

A pool of online raters then listens to the recorded responses and evaluates their effectiveness. Physicians receive feedback, including ratings and examples of highly-rated responses, helping them to further develop their communication skills.

To help advance the area of clinical text scoring, NBME sponsored a Kaggle competition on the topic of Natural Language Processing (NLP). Data scientists from all over the world were invited to develop an automated method to map clinical concepts from an exam rubric (e.g., “diminished appetite”) to various ways in which these concepts are expressed in clinical patient notes written by medical students (e.g., “eating less,” “clothes fit looser”).

Nearly 1,500 teams enrolled and the May 2022 results remain open source so others can benefit and continue developing shared value for the community.

After running an initial proof-of-concept pilot in early 2020, NBME trained 38 subject matter experts from our test development committees on creating Automated Item Generation (AIG) models, which led to the generation of questions that were pretested on four NBME® Clinical Science Subject Exams and the Health & Wellness Coach Certifying Exam. Item analyses indicate the AIG questions are performing at least as well as traditionally written items.

The success of this pilot inspired the creation of a new AIG tool specially tailored to NBME test items called IMAGE (Item Modeling and Automated Generation Engine), which is currently being used to generate questions for additional NBME exams.