Over 250 medical school faculty attain new skills at NBME Invitational Conference for Educators (NICE)

The NICE Conference was launched to support professional growth and, over time, build a core group of educators on medical school campuses who have specialized training in assessment and can then be resources for their peers. The annual conference is designed to benefit educators who are new or inexperienced in updating assessments through a series of interactive workshops.

When medical school faculty were recently asked, “What is the most difficult part of writing good test questions?” the top responses contained a surprise. “Everything!” appeared as one of the top submissions, in addition to “answer distractors,” “vignettes,” and “higher-order thinking.” These valuable and spirited answers helped planners customize the 2022 NBME Invitational Conference for Educators (NICE) curriculum, which ran June 14–15 as a hybrid event, hosted in Philadelphia.

“We offer the NICE Conference as a unique professional development opportunity to share our secret sauce with assessment novices; that is, how we bring together smart, committed people from a variety of universities and institutions to do quality work with our test development experts,” said Dr. Peter Katsufrakis, President and CEO of NBME. “You serve as medical educators for love and passion. We’re here to offer the training and insights to make your assessments even more effective. After all, measures drive learning, and we all want to create the best outcomes for aspiring physicians and the public we serve.”

Offered as a community contribution for the fourth year, NICE was attended by over 250 US medical school educators, who participated either virtually or in person.

Dr. Christopher Knight (University of Washington), NBME’s Strategic Educators Enhancement Fund (SEEF) Oversight Committee Chair and member, was joined by Dr. Tracy Yarbrough (California Northstate University College of Medicine), also a member of the SEEF Oversight Committee, for the opening session, Paying It Forward – Equipping the Creators of Next Generation Assessment. Dr. Knight said, “We’ve seen how the medical education community values the importance of quality assessment because it is very challenging.” Knight continued, “What we can do is to provide the toolkit, a strong network for support, and offer continuous encouragement.”

Assessment Roundtable Discussions

A collective 60 years’ experience from NBME assessment experts offered a range of topics over the two days:

• Identifying Bias and Supporting Fairness in Assessment led by NBME psychometricians Amanda Clauser, EdD, and Francis O’Donnell, PhD
• Training Raters to Produce Unbiased Ratings led by Su Somay, NBME Sr. Measurement Scientist
• Patient Characteristics and Portrayals in Test Questions by NBME Editorial Staff Kris DeRuchie and Jen Scotese, with discussion moderated by Andrea Anderson, MD (The George Washington School of Medicine and Health Sciences)
• Assessment of Communications Skills led by NBME researchers Ann King, Sr. Measurement Scientist, and Angelo D’Addario, Measurement Analyst

Item Writing Workshops

NBME Test Development Staff members Kris DeRuchie, Amy Morales, and Ally Kulesher presented at a plenary session on Good Item Writing: An Interactive Didactic, which generated many questions and interactions from both the in-person and online audience and provided an opportunity for additional item writing training for the attendees.

Later in the conference, attendees were encouraged to share their item writing challenges in small group settings. Participants brought sample multiple-choice test questions with them, and through group discussion, they were able to explore potential structural issues and further their learning through this practical application.

Each participant was provided with a complimentary copy of NBME’s comprehensive “Item Writing Guide.” The guide includes some of the material presented during NICE, as well as additional concepts and templates. The Item Writing Guide is always available for download free of charge.

SEEF Panel Discussions on Clinical Reasoning and Bias in Assessment

NBME SEEF Fellows presented their work and responded to thoughtful questions during panel discussions. The panel discussions were led by Ann King, Sr. Measurement Scientist at NBME, and Dr. Sally Santen (Wake Forest University School of Medicine). The SEEF Fellowship supports faculty in the early stages of their career to develop skills in medical education and assessment research. They also have a commitment to working with a team of interested colleagues.

The first panel discussion focused on the current state of clinical reasoning in the medical education community. SEEF Program Mentor Dr. Larry D. Gruppen (Professor, University of Michigan Medical School) explained the challenges. “Different components each need different methods of assessment, such as knowledge versus information gathering.” The SEEF Fellows described their investigation as a pursuit to integrate those methods and were also joined by NBME Researcher Polina Harik to all discuss their projects on Clinical Reasoning. The full list of panelists included:

• Todd Guth, MD, Associate Professor at the University of Colorado Anschutz Medical School (SEEF Fellow)
• Ofelia Martinez, MD, Director of Clinical Skills at Hackensack Meridian School of Medicine (SEEF Fellow)
• Rachel Wolfe, MD, Assistant Professor, Rheumatology and Immunology at Wake Forest School of Medicine (SEEF Fellow)
• Danielle Roussel, MD, Assistant Dean for Clinical Curriculum & Associate Professor (Clinical), Anesthesiology at the University of Utah School of Medicine (SEEF Fellow – not in attendance)
• Polina Harik, NBME, Manager, Measurement Science

The second SEEF Fellow presentation focused on bias in assessment. Dr. Jennifer R. Kogan (Associate Dean for Student Success and Professional Development, Perelman School of Medicine, University of Pennsylvania) serves as one of the program mentors. She described the two-year fellowship program’s focus on the role of bias in medical education, our learning environment, and assessment. “Bias undoubtedly influences the educational experiences that our learners have and the way they progress, and develop their competence.”

“We know from the literature that bias is present in the assessments that our assessors make. And bias can impact multiple variables of the utility equation. Consider how a learner’s willingness to prepare for or take the feedback from an assessment may be influenced if there’s bias.” The SEEF Fellows who presented their research studies on bias in assessment were also joined by NBME Researcher Dr. Amanda Clauser. The full list of panelists included:

• Todd Cassese, MD, FACP, Associate Dean for Medical Education & Professor of Medicine at Albert Einstein College of Medicine
• Shimae Fitzgibbons, MD, Associate Professor Surgery & Associate Program Director at MedStar Georgetown University Hospital
• Aimee Medeiros, PhD, Associate Professor, Anthropology, History & Social Medicine at the University of California, San Francisco School of Medicine
• Nora Yusuf Osman, MD, Director, Medical Student Education, Department of Medicine at Brigham and Women’s Hospital & Assistant Professor at Harvard Medical School
• Amanda Clauser, EdD, NBME, Vice President, Psychometrics and Data Analysis

Racial Biology in Medical Education and Assessment

The keynote session, Racial Biology and Medical Misconceptions: From Curriculum to Assessment and Back Again featured a lively discussion with Drs. Andrea Deyrup (Duke University) and Joseph Graves (North Carolina A&T University) about challenges with the topic of race/ethnicity in medical education and assessment. This discussion was facilitated by NBME’s Medical Advisor and Associate Vice President of Assessment Operations and Medical Education, Dr. Miguel Paniagua, and included a structured discussion on the impact of using race when determining medical diagnosis on assessments and in test questions, followed by a moderated question-and-answer session with the audience.

To prepare attendees for the keynote session discussion, NBME provided attendees with access to the New England Journal of Medicine article on Racial Biology and Medical Misconceptions (published Feb. 10, 2022) by Drs. Andrea Deyrup and Joseph Graves. Upon successful completion of the post-conference survey, respondents received a complimentary digital copy of Racism, Not Race: Answers to Frequently Asked Questions by Dr. Joseph Graves.

The keynote was very well-received by attendees, as indicated by feedback and comments we received in the post-event survey. One attendee noted “excellent discussants and discussion, really appreciate the open discussion, this was a highlight of the conference for me,” while another commented, “Wow, this was extremely interesting and challenging, thank you for including this last session! There are things that I need to think about now that I didn't even know I needed to know.”

NBME will continue to develop programming that addresses challenges in medical education and provides resources for medical students and educators. Our website is regularly updated for information about the next NICE event, Item Writing Workshops, and other professional development opportunities.