Research fellows tap technology to break down barriers and fuel learning

A virtual reality simulation designed to assess communication skills may help make simulation training more accessible, while a new AI model could enable medical school faculty to devote more of their time to teaching.

When artificial intelligence went mainstream in 2023, a torrent of negative headlines followed. The challenges facing higher learning have been particularly well-documented, with college professors struggling to counter rampant plagiarism and students frustrated by teachers who used AI to craft slide presentations and feedback.

But a research project by some of NBME’s Strategic Educators Enhancement Fellows is aiming to use AI to enhance faculty and student engagement in medical schools. Their idea — using AI to grade medical students’ notes — is not a new one, but they do believe they’ve mastered the technology to execute it.

Attempts to computerize the grading of human writing date back to the 1990s, says Tama S. Thé, MD, one of the fellows involved in the project. This is because, even though faculty receive training on grading, there remains a lot of variability in how it’s done, even by a single professor. “Someone’s assessments are going to look different 10, 20 and 30 grades in,” Thé says.

It's a natural consequence of what can be a labor-intensive and time-consuming responsibility. To that end, Thé has worked out that if their technology was used at his school, it would save 432 physician hours over the course of an academic year.

With the heightened scrutiny that AI draws these days, Christopher Nash, MD, says he and his fellow investigators understood their tests would need to be rigorous for the results to carry any weight. And so, they ran their AI model 17,000 times, according to Thé, and not once did it yield a hallucination, the term used to describe AI-generated information that’s false or nonsensical.

The potential implications are profound. Thé says faculty positions at his medical school aren’t coveted because of the time demands they entail. If their AI program could take over the grading, that could entice more physicians to become teachers, and teachers would have more time to engage in small-group and one-on-one learning activities with their students.

“I know this sounds backward, but I think artificial intelligence is going to make medical education a little bit more human in these ways,” Thé says.

An unforeseen finding could reshape simulation training

The NBME Strategic Educators Enhancement Fellowship (SEEF) enables researchers and medical educators early in their careers to collaborate on research projects with NBME assessment experts and faculty mentors. This cohort, the program’s second, began its fellowship in June 2024, and it will conclude this upcoming June. 

The eight fellows are divided between two research projects. The other group is trying to solve a similarly elusive challenge in medical education: objectively assessing communication.

Working in collaboration with MedVR Education, the fellows have designed a virtual reality-based simulation scenario involving a patient in cardiac arrest. The case selection was intentional, says fellow Natascha Heise, PhD. Because treatment must be administered in a fast and efficient manner, it becomes a litmus test for communication skills.

To assess them, Heise and her fellow researchers explored a range of tools before landing on one that has been previously used only for in-person simulations. As of this writing, they were planning their first test run. But already, they’ve discovered another potential benefit of their efforts.

Simulation training in medical schools began in the 1950s and became more widespread in the 1960s. While high-fidelity simulators and virtual reality systems are shaping the next frontier in simulation training, much of it still centers on standardized patients who learn a case, based on a real patient, and are interviewed, examined and treated by students as though they were the actual patient.

This type of training can be cost-prohibitive for rural and otherwise isolated medical centers and clinics, says Laura Mulvey, MD. Recruiting people to play standardized patients can also be a challenge in communities with small populations. But a virtual reality simulation removes both of these barriers.

“We’re working toward the ideal, which is being able to measure some of these skills that are critical to becoming a good physician by actually seeing students perform them, even when a simulation lab or standardized patients aren’t available,” Mariela Lane, MD, another fellow working on the project. 

Perhaps paving the path to progress

Thé and his group are also working toward removing all potential barriers to accessibility. They’re publishing everything an institution would need to implement their AI model, including the prompt engineering, software and instructions on how to use it all.

Should it take root, Candace Pau, MD, a fellow involved in developing the AI model, believes the technology has the potential to help educators better understand what makes a good rubric. Thé thinks, in 10 to 15 years, it could unlock new, holistic methods of assessing students.

“The more tools we have for assessment and the more we understand the theory about why those tools do or don’t work and in what contexts they can work,” Pau says, “the better assessment is going to be at achieving its ultimate goal, which is to really help learners be able to progress in their development.”

The 2024 SEEF Cohort

Listed according to their research groups

AI model

Nayef Chahin, MD, MEd

Assistant professor of pediatrics, associate program director of the neonatal-perinatal fellowship and associate program director of the pediatrics residency program at the Virginia Commonwealth University School of Medicine

Christopher Nash, MD

Emergency medicine instructor at the Duke University School of Medicine

Candace Pau, MD

Associate professor of clinical science at the Kaiser Permanente Bernard J. Tyson School of Medicine

Tama S. Thé, MD

Assistant professor of pediatric emergency medicine at the University of Kentucky College of Medicine

VR simulation

Natascha Heise, PhD

Assistant professor of anatomy at Macon and Joan Brock Virginia Health Sciences at Old Dominion University.

Mariela Lane, MD

Assistant professor of internal medicine at the Paul L. Foster School of Medicine at Texas Tech University

Laura Mulvey, MD

Assistant professor and clerkship director of emergency medicine and the director of evidence-based practice at The Robert Larner, MD, College of Medicine at the University of Vermont

Nuno S. Osório, PhD

Researcher at the Life and Health Sciences Research Institute (ICVS) and ICVS/3B’s – PT Government Associate Laboratory and invited assistant professor at the School of Medicine at the University of Minho