International Programs

The NBME has provided consultation, psychometric, and examination services for international examination programs for more than a decade. At the 2001 Annual Meeting, members of the National Board identified global assessment services as a priority area for strategic development of new programs and services for the NBME. At planning sessions in the 2002 Annual Meeting, board members recognized that the public served by the NBME extends beyond North America, and provided guidance for staff members in the development of a coordinated strategy for international activity. Based on initial results, in 2003 the Board created an International Task Force to explore global opportunities and risks to the NBME, review the inventory of NBME resources, and develop strategies for engaging in projects outside of the US and Canada.

The NBME Membership received and endorsed the recommendations of this Task Force at the 2005 annual meeting. The Membership authorized prudent investments over an initial five-year period that might yield sustainable evaluation programs compatible with NBME’s mission and values. The Membership further recognized the NBME’s responsibility to provide assistance to developing nations. While concentrating on Europe and Latin America, the Membership also authorized staff to evaluate opportunities in other regions that might arise. To manage these activities, the Membership endorsed the formation of an International Programs division in Assessment Programs. The following report highlights some of the activities of this unit over the past four years.

Foundations of Medicine Pilot Program
The Foundations of Medicine Examination is the result of a three-year collaborative research effort between the NBME and a consortium of the Universities of Bologna, Firenze, Ferrara, Milano, Parma and the Catholic University of Rome, the University of Minho in Portugal, and the Catholic University of Leuven in Belgium. Initially, the study was designed to determine the approximate percentage of materials contained in NBME item pools that have international relevance and to study the issues involved with translation of materials to other languages. In addition, European collaborators wished to evaluate the applicability and acceptability of NBME-style multiple- choice questions to students and faculty members in European curricula, to correlate results with internal measures of performance, and to determine the role of standardized assessments in the overall evaluation of student performance.

The examination is intended to provide a common measure of knowledge for students who are nearing the completion of, or who have recently completed, the second cycle of the European curriculum in medicine. Prototypes were designed and validated by faculty members from the collaborating universities with the assistance of the NBME. The exam consists of 200 multiple-choice test questions drawn from NBME secure item banks. Approximately 40% of the items cover the basic sciences that are fundamental to clinical practice. The remainder covers the clinical disciplines of medicine, surgery, pediatrics, obstetrics/gynecology, and psychiatry. The examination was provided in Italian translation in Italy. English language versions of the examination were used in Belgium and Portugal. The exam was published in paper-and-pencil format, and administered under secure, proctored conditions.

Pilot tests of the examination were conducted in Italy in 2007. In 2008, the examination was also administered at the University of Minho in Portugal and the Catholic University of Leuven in Belgium. Standards for satisfactory and excellent performance for the 2008 spring administration were established by faculty members from the participating European universities using the standard setting procedures employed by the NBME for its other high-stakes secure assessment programs. All students received individual performance profiles in each of the content areas covered by the examination. Participating universities received more detailed reports that permit longitudinal evaluation of curriculum effectiveness.

To date, more than 2,000 students in Italy, Portugal, and Belgium have completed the examination. Based on results to date, the NBME and its European partners have concluded that secure examinations built from patient-centered, vignette based multiple- choice questions are a valuable addition to the essay, short answer, and oral examinations traditionally used in Europe. More than 80% of the basic science and clinical science items contained in current NBME item banks can be used in Europe without significant changes. With the exception of a few idiomatic expressions and materials that refer to unique North American sports activities, translation does not pose a major problem. Faculty members from the collaborating countries were able to come to agreement on performance standards without difficulty. Furthermore, the examination provides a solution to the European challenge of documenting an individual’s knowledge base for students seeking transnational educational experiences.

The NBME is encouraged by these results, and intends to transition the Foundations of Medicine Examination from research to operational status over the next two years.