Engaging with Faculty at Item-Writing Workshops
On March 4, 2019, NBME’s Director of Test Development Services (TDS) conducted a workshop for 20 participants at the American College of Obstetricians and Gynecologists (ACOG). For the past 10 years ACOG has reached out to NBME for a spring workshop to train subject matter experts writing items for ACOG’s PROLOG (self-directed learning) and CREOG (residency in-training) assessments.
NBME has developed test items and examinations for licensure and other assessment purposes since 1915. NBME staff members are internationally recognized experts in the field of test development, psychometrics, and medical education research. For over 20 years, members of NBME’s professional test development staff have served as a resource to medical school faculty wishing to improve their item-writing techniques. Additionally, faculty from other health professions (e.g. veterinary, physical therapy, podiatry) reach out to NBME to conduct workshops to improve their assessments.
As part of its goal of collaborating with organizations to develop better assessment tools for learning, measurement, education, and patient outcomes, TDS conducts many off- and on-site domestic workshops throughout the year. In 2018, TDS facilitated 18 offsite workshops, in addition to many onsite workshops for the United States Medical Licensing Examination® (USMLE®) and its client exam programs. For 2019, 16 have already occurred or are planned through mid-year.
The item-writing workshops are designed to help faculty members improve the quality of multiple-choice questions written for their examinations. Through informative presentations and discussions, faculty members are able to explore structural issues relating to constructing quality multiple-choice items for testing of basic science and/or clinical science content. An overview of item formats concentrating on the traditional one-best-answer item formats that require examinees to select the single best response is presented. One-best-answer questions are the most widely used multiple-choice-item format. Several examples of well-constructed one-best-answer items are presented. Each consists of a focused item-stem (e.g., a clinical vignette) and a lead-in question, followed by a series of options with one correct answer for each item. Several examples of technical item flaws are presented and discussed during the session.
Each participant is provided with a printed copy of the manual “Constructing Written Test Questions for the Basic and Clinical Sciences,” which is published by NBME. The manual includes the material that is presented during the session and additional templates that can be used to assist item writers develop items. Copies of this manual can be obtained at www.nbme.org.
The workshop day consists of a short educational session followed by a rewriting exercise. NBME provides workshop participants with a handout of poorly structured items (constructed by our own in-house medical experts); participants rewrite items in small groups, and then present the rewritten items to the full group. The group discusses and provides both content and structural suggestions. The facilitator will also display the NBME version of the rewritten item.
Feedback from the workshops has been very positive. Participants have expressed that they feel that these workshops improve their ability to create their own assessments, recognize test question flaws, and rehabilitate poor questions. Participants are surveyed at the end of the session to assess their knowledge gained and obtain feedback on how sessions can be improved.
You can learn more about how you can have NBME conduct a workshop at your institution here.