Update on Web-Based Examinations and Pilot Projects
Date Posted: July 10, 2009
Clinical Progress Tests
Progress tests can provide medical educators with an invaluable tool with which to obtain a comprehensive assessment of students’ knowledge and skills. Progress tests are longitudinal assessments aimed at charting growth of knowledge in a comprehensive number of disciplines as students proceed through various phases of their medical education. Due to the typically comprehensive nature of the domains targeted by these types of exams, progress tests are also useful in assessing the extent to which curricular goals and objectives are met.
In collaboration with Case Western Reserve University School of Medicine and the University of South Florida College of Medicine, the NBME developed a clinical progress test to explore the development and use of a form of progress testing in clerkship education. Faculty from the participating schools identified all of the key learning goals and objectives of clerkship instruction and developed test specifications that covered all of them. Multiple test forms were then built to those specifications, and students periodically (typically every three months) take one of the forms to determine if they are making good overall progress in mastering objectives and to identify areas of strength and weakness for individual students.
The study is being expanded in 2009 to include additional schools. In mid-January the NBME sent an RFP to all LCME- and AOA-accredited medical schools with information on the study design and test administration plan. Interested schools were asked to provide information that included a description of how the test would be used, the class year(s) to be tested, how students would be motivated, and approximate test dates. Proposals were received from seven allopathic schools and one osteopathic school. The following schools were selected for the study:
Approximately ten test administrations are planned for the 2009-2010 academic year across all schools.
A collaborative project is also underway with medical schools in the United Kingdom and Ireland to design and implement a multi-school progress testing program. The purpose of the project is to design a progress test suitable for use at multiple medical schools, construct and translate multiple (equated) forms of that examination, and to periodically administer and report scores on the test beginning in the 2008-2009 academic year.
NBME staff members and the Multi-School Progress Testing Committee, which included faculty members from participating schools, met in December 2007 to finalize the study design and progress test blueprint and to review sample test items to aid in assembly of draft test forms. A subsequent meeting was conducted in March 2008 to review and finalize the eight test forms. NBME staff members scheduled web-based conferences with the participating schools to discuss the logistics for test administration and issues related to score reporting. The participating schools include:
All four successfully administered examinations during school-identified testing windows starting in November 2008 through early July 2009.
Until equating and scaling can be done across all eight forms of the examination, schools receive total test and content category percent correct scores for each examinee. One of the project challenges is to recruit a sufficient number of examinees to participate in the “linking” administrations, which will produce data for the equating and scaling processes.
Senior (Year 5) students scheduled to graduate in 2009 are the focus of this effort. Further research will be conducted in progress and achievement testing in collaboration with a number of US and international medical schools and NBME. These projects provide participating schools and the NBME with an opportunity to engage in and publish the results of collaborative research efforts on the use of longitudinally administered tests for assessment of student progress and for evaluation and improvement of instructional programs. Staff members anticipate that this may be a service available to medical schools in the future.