NBME logo
Date Updated: April 17, 2012

Center for Innovation Projects

Open or close all project information


The NBME Center for Innovation's mission is to assist in formulating and actualizing NBME's strategic vision. Active projects include:

Data Commons

The Data Commons is a potentially large "virtual" repository of data, useful for the compilation of data reports that support and drive a wide variety of functions and purposes. The Data Commons would be a kind of "membership organization" of data repositories from the medical education and assessment world. The kinds of data reports the Data Commons could provide would be useful to healthcare providers (HCPs), researchers, and the education and assessment organizations themselves. While the founders of the Data Commons (AAMC and NBME) collectively bring a large amount of data to bear, it would be even more meaningful and important with the addition of other institutional repositories.

Another potential source of data would be new repositories that do not currently exist. One example of this is the current initiative of the Association of Pediatric Program Directors (APPD) to build a comprehensive database on all Pediatric residents, and the Center is currently negotiating with APPD as to the possible affiliation of this important new source of GME data within the Data Commons. Further discussions are also underway with AAMC to define the rules of engagement, and the optimal business model for the future. Other progress involves further build-out of the infrastructure, and the ability to offer near real time AAMC: NBME data-sourcing using the Data Commons infrastructure.

eFolio Connector

This pilot grew out of the Invitational Conference on Portfolios that the NBME co-convened with the AAMC, FSMB and ACGME in October 2007 and the need to drive inter-operability of data contained in a rapidly increasing number of very disparate portfolio systems.

Learn More...

After successful creation and refinement of a technical specification – "educational trajectory" - by the MedBiquitous Working Group, feasibility testing for the infrastructure of the eFolio is now being done. The AAMC and NBME are working together to fine-tune such necessary systems as ID matching across the institutions, security, and report generation. The first prototype report will be My Scores, pulling assessment information from both AAMC and NBME databases to create a useful, meaningful data set for an individual medical student, resident or practicing physician.

Several committees are working regularly on different aspects of the eFolio – an eFolio Oversight Team to deal with high-level operational issues, an eFolio Development Team to work on the technology and infrastructure, an eFolio Advisory Group to provide guidance in functionality and usability, and a MedBiquitous Working Group to help coordinate the development of relevant technical specifications. The AAMC is also forming focus groups of medical students to provide feedback on mock reports for design, pricing, and marketing purposes.

Using Clinical Data for Practice Measurement and Improvement ("Making the Data Sing")

There is growing interest in analyzing clinical data in near real-time to better understand the complexities of modern healthcare and its delivery, and to provide physicians with reality springboards to better understand and improve the performance of themselves, their teams and practices. Assessment organizations such as the NBME could develop derivative assessment tools based on real data obtained from real settings, a new domain of measurement broadly termed Assessment at Work.

Learn More...

Performance improvement, a critical goal of healthcare, cannot be fully realized without first understanding what care is being delivered, and to whom and how it is being delivered. This requires a clear understanding of the practice of each individual and the teams in which they work ("My Practice"). Using the data gleaned from a practice (or individual physician) Electronic Health Record system could be used to create a "Practice Profile": a structured report on patient demographics, frequent diagnoses, commonly used medications and treatments, and more. This Practice Profile would act as a snapshot of the doctor’s or practice’s community of patients, and allow them a more sophisticated understanding of who they are treating and how.

Measurement can then be tailored to this real-life practice data to facilitate identification of both strengths and weaknesses for targeted improvement. The end result of this process could be more effective lifelong learning, together with mastery and maintenance of appropriate knowledge, to care for those patients actually being seen. Broadening the scope of lifelong learning and improvement beyond the traditional "minimal competence in scope of practice" paradigm, to include a "continuous improvement in actual practice", is becoming an increasingly important concept in clinical medicine.

Assessment of Milestones

A system of Pediatric Milestones has recently been drafted by a group with broad representation of the academic Pediatric community. For each of the sub-competencies within the six ACGME domains, this Working Group developed a series of narrative anchors for observable or demonstrable behaviors; these anchors are meant to assist in the delineation of proposed developmental stages in the achievement of each sub-competency. Delineation of these Milestones represents a new approach to assessment of individuals based on the general competencies promulgated by the ACGME and ABMS, and an important step toward a new view of education and training that adds a focus on outcomes to the previous emphasis on process. The next step is to begin the process of determining best practices for their assessment to support competency-based progression decisions.

Learn More...

The pilot aims to study the feasibility of implementing a nationally defined set of Pediatric Milestones locally in participating institutions, with a cohort of medical students who complete pediatric sub-internships and then continue on as Pediatric interns at the same institution. This study will allow comparison of milestone achievement prior to internship with performance of the same or related constructs later in internship. This should also provide useful information for decision-making in selection of residents, as well as help define expected level of performance in particular areas at the start of internship. Developmental work might also have implications for strategies to orient, train and calibrate faculty participating in assessment and progression decisions.

Computational Linguistics (CL)

Recent trends in computational linguistics have resulted in sets of tools that could be applied in the assessment and healthcare domains in innovative, useful ways.

Learn More...

The ability to automatically manipulate and process text via CL can potentially support a number of applications, including analysis of data from electronic health records (EHRs), portfolio data aggregation and analysis, generation of test items, and essay scoring. The Center has been exploring related applications through a series of feasibility pilots, in collaboration with a company that has expertise in latent semantic analysis as well as a computational linguistics group at the University of Wolverhampton.

A pilot involving the study of computer-generated item coding is currently in the planning stages. At the NBME, multiple choice questions are assigned codes which are used to select items from a large pool and assign them to test forms. The codes contain information about the item deemed useful in assembling content to match a test design, as well as information useful for categorizing items to provide sub-scores. The codes are assigned by humans, are limited in number and are of pre-determined interest (e.g., patient age, sex, location of the encounter, organ system, and diagnosis). Such coding is resource-intensive and error prone. In contrast, computers are now theoretically capable of achieving comparable coding accurately, consistently and instantaneously; and computers could assign much more expansive coding or descriptors of items based on ALL available information and meta-information. This additional information could enhance a number of item and test development activities including, but not limited to, quality control and security.

A pilot related to computer scoring of patient notes is also underway, and another related to predicting item difficulty is planned.

Learn More about NLP Pilots...

The first pilot, termed Phrase Analysis studied the utility of latent semantic analysis for modeling the ratings experts assigned to patient notes in the USMLE Clinical Skills examination. While the initial results were promising, the research was put on hold because of competing priorities. That work has been picked up by another research group at the NBME.

The second pilot, termed Rapid Item Generation (RIG), led to the creation of a system that automatically produced a number of low-quality MCQs that, with post-editing, were viewed as usable for low-stakes assessment.

The third pilot focused on how information contained in vignette-based MCQs might be automatically extracted and used as a basis for editing tools. This pilot, named Computer-Assisted Item Development (CAID), led to the creation of a parsing engine (coined "Lucy") that automatically identifies, tags, extracts, and organizes the key components of a vignette-based MCQ (e.g., patient age, gender, chief complaint).

A fourth pilot named Pool Search Engine grew naturally out of work on Lucy, and focused on expanding the types of metadata the system parses to include types of lab values and drugs (e.g., to enable efficient mining of our item pool) and terms that might vary across cultures (e.g., for purposes of translation from English to British-English).

Scouting Activities

Virtual scanning for relevant topics, themes and trends is essential to the Center, not only for future pilot development, but also to inform existing projects and refine activities already underway.

Learn More...

In the last year, some examples of Scouting Activities include the following:

  • Avatar and animation authoring
  • Cognitive errors
  • Ethnography/ethnomethodology
  • Evolution of the patient-physician relationship
  • Expert networks
  • Expert patients
  • Health literacy
  • Lean manufacturing principles
  • Metacognition
  • Novel business models enabled by the Internet
  • Participatory medicine
  • Patient input/satisfaction
  • Patient safety and quality improvement
  • Physician information management skills
  • Physician rating systems
  • Portfolio assessment and data transparency
  • Practice profiles
  • Professional fatigue
  • Semantic web
  • Usability metrics

Stemmler Fund


NBME logo