Stemmler Fund Prior Grant Information

Rationale: Health professions’ admission committees across the world are faced with the difficult task of selecting, from among many eligible applicants, the select few who will be admitted to their training programs. The determinants of this admissions process are often a combination of cognitive measures, such as Grade Point Average (GPA) or standardized tests such as the Medical College Admission Test (MCAT) and non-cognitive measures, including interviews and essays. However, there has been limited success in the development of evaluation tools that will provide reliable and valid measures of an applicant’s non-cognitive qualities. The exception to this is the MMI, in essence an admissions OSCE. The MMI has been shown to predict intramural and licensing examination performance. However, like any OSCE the MMI has practical limitations; the sheer volume of candidates for many institutions makes it necessary to develop a reliable and valid strategy for screening candidates’ non-cognitive attributes in a more efficient fashion. To this end a new measure, using video scenarios and written or audio responses was developed and a pilot study was completed. In 2006, 110 applicants to McMaster’s medical school completed this Computer-based Multiple Sample valuation of Non-cognitive Skills (CMSENS). Of those applicants, 78 completed the CMSENS by verbally recording their responses in an audio file while 32 typed their responses. The overall test generalizability was .86 for the audio CMSENS, and .72 for the written. The written CMSENS also demonstrated predictive validity, correlating with the MMI at .51. However, conclusions from this study are limited because of the small sample and the one-time nature of the findings.

Objectives:

1. Assess the impact of varied test time and proctoring on the reliability and validity
2. Determine the predictive validity of CMSENS
3. Determine the reliability of the pilot results with a larger, more diverse sample

Methods: To achieve the first objective, applicants to McMaster’s medical school will be invited to participate in the CMSENS in winter 2008. These applicants will complete a CMSENS in which the length of response time and proctoring will be manipulated. In addition to reliability and validity analysis to determine the optimal testing format, participants invited to interview will have their scores compared to MMI performance. To assess objective 2, several methods will be followed. In-program performance results will be assessed for the about 25 members of the medical class of 2009 who participated in the CMSENS pilot project and gained admission. In addition, construct validity will be examined by recruiting students in their final year of medical school and second year residents to participate in a mock-CMSENS, thereby allowing a comparison between (a) scores assigned to the medical school applicants and more senior trainees, and (b) for the final year students and residents, comparison between scores assigned on the CMSENS and those received on the Canadian qualifying examinations; specifically Part II of the MCCQE ( an OSCE) which evaluates both cognitive and non-cognitive characteristics of medical trainees. To satisfy the 3rd objective, a sample of applicants applying in the winter 2009, will be administered the CMSENS. A larger sample in this year will permit accurate assessment of reliability and facilitate comparison of CMSENS performance to MMI & in-program results.

Significance for Medical Education and Practice: This innovative assessment tool, if it is proven reliable and valid, has the potential to allow educators in the health professions to efficiently assess the non-cognitive qualities of the thousands of applicants to training programs, for whom reliable and valid was previously impossible.

Project objectives: The ACGME has directed that all residents must meet six competencies. The sixth competency, Systems-Based Practice, has presented a challenge for assessment.

The objective of this project is: to develop a standardized, objective, innovative method to measure the sixth competency: Systems Based Practice.

An OSSIE (Objective Structured System Interaction Examination) will evaluate the resident’s ability to interact with the health care team, deal with aspects of the health care system, coordinate effective care across settings, and provide cost-effective care.

Rationale and primary methods to be employed: In today’s health care system, not only must physicians be competent in their knowledge and practice of medical care, but they must be leaders of teams composed of other health care providers. They must be able to assist their patients to navigate the health care system and insure that continuity of care is promoted across health care settings. They must be cognizant of costs of various treatments.

Four required skills are identified by the ACGME that must be attained in order to be judged competent in Systems-Based Practice. The resident must:

1. be able to understand the interaction of physician practices with the larger system, resources, and providers.
2. have knowledge of practice and delivery systems.
3. practice cost-effective care.
4. advocate for patients within the health care system.

Three scenarios involving patients, families, and members of the health care team will be developed to test each of these four required skills for a total of twelve scenarios. The scenarios will be presented in the format of an OSCE. The new examination for residents, the OSSIE, will be given to PGY2 residents in the middle of their second year. It will be a formative examination that will evaluate the residents’ competence and enable tailoring of their third year to improve these competencies. Generalizability analysis will be used to determine inter-case reliability. Correlations between exam scenarios and ratings by observers in practice situations will determine the validity of the OSSIE.

How the proposed research will advance assessment in resident education: Currently, there are few measures of Systems-Based Practice to use in assessing residents. The proposed research will advance assessment in resident education by investigating an innovative, objective method of assessing the ACGME competency of Systems Based Practice. This new method of assessment, the OSSIE, will be a modification of the traditional OSCE. A simulation of interaction with other members of the health care team will capture abilities needed for physicians in today’s complex health care system. By using the OSSIE, faculty will be able to provide constructive feedback to each resident to enable improvement in Systems-Based Practice.

The overall goal of this research project is to demonstrate the feasibility and examine the usefulnessof evaluating the quality of clinical training programs by assessing the clinical outcomes of the patients later cared for by the graduates of those training programs. The concept is premised on the view that although medical education serves a collection of intermediate goals, in the end the most important clinical goal is to improve the health of individuals and populations. We may mean many different things when we say that a medical school or a residency program is good, or that one medical school or residency program is better than another. However, stakeholders, including prospective trainees, health systems, and patients, could be justified in expecting at least one specific meaning: that graduates of good training programs in general take care of patients well, and that graduates of better training programs in general take care of patients better.

This project represents a study of “proof of concept,” using as our test case the analysis of maternal birth treatment and outcomes to inform the assessment of residency training in obstetrics and gynecology. We will use data from all hospital-based deliveries in New York and Florida between 1992 and 2006 to test the relationship between residency program, physician characteristics, and maternal outcomes. Our measures of performance will be:

a) use of Caesarean section; (b) whether a woman who delivered vaginally had a 4th degree perineal laceration; (c) whether a woman experienced any adverse outcome, as defined by HealthGrades(2004); and (d) a complication measure developed by Epstein, Ketcham and Nicholson (2006) that assigns larger positive values to complications that result in long hospital stays for the mother. Using these data and measures, we will address the following specific questions:

1) How much variation in inter-physician performance is explained by residency program and year of residency graduation?

2) Can residency programs be categorized reliably according to the treatment patterns and patient outcomes of their graduates?

3) Is there systematic variation in residency program effects?

We believe these notions are consistent with the goals of the Stemmler Fund and the National Board of Medical Examiners more generally, because they incorporate new methods of assessment in which the influence of training programs on clinical outcomes is assessed independently of patient and physicians characteristics. At the conclusion of this project, we expect to have a deeper and more specific understanding of the promise and limitations of evaluating medical training programs using clinical outcomes; we expect to have a series of manuscripts describing our conceptual view, analytic approach, and results; and we expect to have identified the next steps toward further development and evaluation of this assessment concept.

Objectives: We propose to assess the reliability and validity of a shared decision making checklist as a tool for evaluating medical student cultural competence in standardized patient encounters. We will validate shared decision making checklist ratings by correlating scores with global assessments of cultural competence made by clinician experts in cultural competence. Additionally, we will compare ratings of cultural competence to ratings of general communication skills as measured by the Common Ground instrument in these standardized patient encounters to determine the extent to which cultural competence overlaps with communication skills proficiency.

Background: Failure to develop care plans that incorporate information about patients’ cultural backgrounds and values contributes to important disparities in health care. Medical schools and residency training programs are now required to teach and assess trainees’ cultural competence skills. However, a review of the literature indicates that measures of cultural competency suffer from significant deficits. Most studies rely on measurements of attitudes or skill selfassessments, and the minority of studies that do have skills-based outcomes assess only a subset of relevant competencies. Thus, while medical schools nationally are emphasizing the importance of patient-centered care and cultural competence, they lack the ability to measure the degree to which students are mastering these core concepts.

Methods: We will determine the reliability and validity of a shared decision making checklist as a measure of cultural competence. Using purposeful sampling to identify students of different genders, clinical skills competence, and races, we will select 200 videotaped encounters from 50 third year medical students’ interactions with four standardized patients. Trained coders will score the student-standardized patient encounters using a shared decision making checklist. Two faculty cultural competence experts who will be blinded to the study hypothesis, checklist content, and scores will perform global assessments of cultural competence based on review of the same videotaped encounters. The standardized patients’ ratings of general communication skills using the Common Ground instrument will be used to assess trainees’ communication skills. Reliability of the three ratings instruments will be calculated. We will assess validity by correlating the shared decision making scores to the global assessments of cultural competence. We will assess concurrent validity by correlating the shared decision making results with the communication skills scores. We will explore reliability by conducting several generalizability studies. This analysis will determine the number of raters and cases needed to obtain reliable cultural competence scores.

Implications for assessment: Our results will inform the assessment literature by evaluating the use of a shared decision making checklist for assessing cultural competence and determining the degree to which cultural competence correlates with communication skills. These results will facilitate evaluation of the efficacy of cultural competence curricula.

The public in general and professional organizations in particular are increasingly demanding evidence of competence in medical practice and a physician’s ability to meet the demands of today’s society (IOM, 2001; 2003). Medical education has responded with a focus on educational outcomes (Whitcomb, 2004), case-based, authentic, curricula (Friedman, 2001; Kincade, 2005) and experiences that support the development of physicians within a complex health care system (ACGME, 2005; AAMC Report V, 2001). The emerging definition of professional competence is difficult to evaluate using traditional assessment. The portfolio addresses the current limitations of assessment by integrating professional outcomes and placing them within an authentic learning context. Challenges in portfolio assessment include insufficient inter-rater reliabilities, questions of generalizability, a substantial faculty and learner time commitment, and balancing a prescriptive, standardized approach with individualization (Friedman, et. al. 2001; Case, 1994; Des Marchais et al 1995; Challis 1999: LeMahieu, et.al 1993; Herman et al, 1995).

The University of Missouri has developed a set of key competencies for our graduates (MU2020 key characteristics) that are consistent with national and international discussions of professional competence. To our knowledge few medical schools have successfully engaged faculty in developing an approach for assessment of professional competencies. This proposed research draws on the prior work at MU to address two global questions: 1) How does the development of a set of descriptive anchors for each of the key characteristics influence the validity, reliability, reproducibility and trustworthiness of portfolio assessment? 2) How do student contributions to the portfolio influence faculty assessment of portfolios?

Descriptive anchors of exemplary performance for each of the professional outcomes will be derived from the literature, clinical faculty, medical students and patients. These anchors will be used to develop a portfolio assessment tool. A twostep judgmental review process will establish the content validity of the descriptive anchors. Inter and intra rater reproducibility will be established by using the assessment tool to evaluate the portfolios of third year medical students. Predictive validity will be determined by correlation of portfolio assessment with traditional measures of student success. The influence of student contributions to the portfolio assessment will be evaluated by determining the differences between individual faculty ratings of students portfolios rated with only required documentation and rated a second time with student contributions included. An external advisory board will provide guidance to the research team and will review the appropriateness of the intermediate research projects.

The outcome of this project will be a portfolio assessment tool to evaluate student outcomes. It will be a useful addition in the assessment of learners and promote an enhanced understanding of professional competence.

Currently, no psychometrically rigorous and developmentally informative instrument exists for assessing resident competencies in the cognitive domains encompassed by Practice Based Learning and Improvement (PBLI), as defined by the ACGME. Using an iterative process model, we propose to develop and empirically validate four cognitive measures tapped by a comprehensive PBLI instrument that permits periodic formative and summative assessments of residents competency levels as they progress through their programs in different specialties. Based upon preliminary experience with a relevant pilot project, we will develop an item pool addressing the following PBLI sub-competency domains: 1) analyzing practice experience, 2) using information technology to manage information and locate evidence from scientific studies related to patients’ health problems, 3) applying knowledge of study designs and statistical methods to the appraisal of clinical studies and other information on diagnostic and therapeutic effectiveness, and 4) assimilate evidence from scientific studies related to patient’s health problems. We propose that these domains, which conform to the standard cognitive domains of evidence based medicine (EBM), and are frequently summarized as ‘ask, acquire, appraise, and apply’, are best-suited for measuring the cognitive aspects of PBLI.

Initially, we will generate a pool of 100-150 Written-Structured Response items tied to sub-competency domain specifications, using different item formats (such as multiple choice, true-false items). The item pool will represent the relevant, observable facets of each of the sub-competency domains. Parallel forms of the PBLI instrument will be generated next, aligned with common assessment specifications that stipulate a weighting distribution for items tied to different competency domains and cognitive levels (Phases 1-2, Year 1). We will implement a rigorous content- and empirical- validation plan by testing each parallel form of the PBLI on samples of resident volunteers in medicine, pediatrics, and emergency medicine at New York Presbyterian Hospital, as well as upon residents in accredited programs from these specialties outside of our institution. We will use Rasch modeling techniques combined with methods from classical measurement theory to examine validity and reliability of the PBLI measures through these empirical trials. We will supplement the PBLI data with a structured survey to identify programs conforming to ‘best practices’ criteria in the target domains of EBM. Convergent validity evidence can thus be gathered and evaluated, along with evidence of resident group differences on PBLI measures in programs that are more or less compliant with EBM practices and teaching.

We believe that the resulting PBLI instrument(s) will provide a unique and critically important vehicle that, combined with existing performance-based assessment modalities, will make possible a comprehensive approach to evaluation of residents’competencies across a broad range of specialties. We believe that the PBLI instruments thus produced will fill a gap in the area of outcome assessments in residency programs, the absence of which currently limits the quality.

Standardized patients (SPs) are widely used to teach and assess clinical skills. Normal SPs, however, are limited in their ability to display abnormal physical findings. Non-SP simulations could be used (e.g., listening to pre-recorded abnormal heart sounds on a computer), but that method excludes interaction with a live person, and thus is less realistic and probably a less accurate representation of students' skills in real settings. Augmented reality (AR) can expand what an SP can do. AR is a methodology that overlays artificial or virtual components (visual, aural, etc.) over the natural environment to provide the user with helpful information. The proposed augmented SPs (ASPs) will combine the assessment technologies of SPs and computer-driven simulations, allowing each to offset limitations of the other. This project augments the SP by permitting the learner to hear abnormal heart and lung sounds from an SP whose own sounds are actually normal.

We have developed a functioning prototype of the technology for this augmentation. The prototype allows the listener to hear pre-recorded heart and lung sounds when auscultating any of 26 locations on a mannequin. Prior to the award date, we will have moved the system from the mannequin to a variety of SPs of different body morphologies, with a learner hearing the selected sounds rather than those of the SP through a modified stethoscope.

The primary objective of the proposed project is to continue to make this system more realistic by minimizing the cues that AR is being used. We will improve the stethoscope's appearance and performance, make the sounds audible over a wider variety of locations on the ASP, and create a database of abnormal heart and lung sounds. These changes will move this new assessment technology from a laboratory prototype to a functional system for routine assessment of students' auscultation skills in any SP-based examination.

We will test the improved system with students in a required annual M4 OSCE. Product development needs will drive formative evaluation studies of students using the system in OSCE-like assessments throughout the project period. Through surveys and interviews we will gather students' views on certain aspects of the ASP. We will also examine the validity of using the ASP through analyses of students' performance in several situations (e.g., ASP with normal findings versus traditional SP with normal findings; ASP with normal findings versus "placebo" ASP which provides the SPs own sounds through a system similar in appearance to that of the ASP; and diagnosis of pathologies indicated by abnormal ASP findings.)

This innovative approach to assessment using augmented standardized patients to assess heart/lung auscultation skills will expand the range of physical abnormalities that can be tested in SP-based assessments.

Rationale

As part of the assessment of clinical performance during the Canadian national specialty examination in internal medicine, candidates' physical examination skills are tested in a series of bedside stations. At each station, a candidate performs a focused physical examination on a standardized patient. Since 2003, we have integrated simulation technology into the physical examination stations in order to test candidates' ability to recognize common internal medicine physical abnormalities.

Objectives

1. To establish the relationship between competence in physical examination as assessed using simulation technology compared to real patients.
2. To assess whether physical exam technique is a separate competency from the recognition of abnormalities on physical examination.

Methods

Internists' physical examination skills and diagnostic accuracy on real patients and simulations will be assessed during a 10 station OSCE. The OSCE will consist of 5 stations using patients with real cardiac abnormalities and 5 stations using standardized patients lacking physical abnormalities combined with audio-video simulation of cardiac auscultatory abnormalities.

Contribution to Assessment

Our integration of simulation technology into a high-stakes assessment of clinical performance is a novel contribution to the field of assessment. In addition, we will examine the relationship between the transfer of physical examination skills between simulations and real clinical performance, which has not been previously established. Our approach to integrating simulation technology into an examinee's patient assessment may be generalized to other testing formats and settings.

In contrast to current checklist-based SP assessment procedures, that focus primarily on assessing physical exam maneuvers or history taking, the proposed hypothesis-driven assessment procedure brings together all key elements of physical diagnosis, namely generating a limited set of diagnostic hypotheses, anticipating discriminating findings, performing maneuvers and appreciating the findings, and interpreting the finding by proposing a working diagnosis. The assessment task requires students to think in action, while gathering the data. The findings from the scientific literature that were used to build this assessment procedure, namely co-selection, prototypes, discriminating features, and transfer, provide a strong conceptual framework for the proposed procedure. By implementing this approach as an assessment procedure, it also automatically guides learning (knowing that students learn what they are assessed on). It promotes contextualized, integrated, and meaningful learning, and provides, as advocated by medical educators, a more parsimonious, selective approach to physical diagnosis, focusing on key, discriminating findings as well as an array of structural patterns (diagnostic sets) that can facilitate transfer when students go from pre-clinical to clinical settings and from patient to patient. The procedure is based on 18 complaints, 145 physical exam maneuvers, and 59 diagnostic alternatives, a sound foundation upon which students can build their physical diagnosis. The student and class profiles generated from this procedure provide a well-organized and detailed framework for providing feedback to students and educators, where various sources of strengths and weaknesses in physical diagnosis can be parceled out, such as distinguishing anticipation errors from execution or interpretation errors (an important asset in an era of reducing medical errors). An example of a student profile following a case would include: "Good anticipation of clinical findings, some faulty physical exam maneuvers, and incorrect diagnosis." Finally, the assessment procedure and the various scores derived from the observations, such as anticipation scores, diagnostic interpretation scores, and overall physical exam scores (8 profiles), offer the possibility of better distinguishing among levels of expertise. The purpose of this proposed project is to begin to validate this hypothesis-driven assessment procedure for physical diagnosis of medical students and residents. Both a three-step and a four-step procedure will be studied, where the four-step procedure includes generating hypotheses while the three-step procedure does not. Six pilot testing and validation studies are proposed, each testing various aspects of construct validity and reliability:

C-I Pilot testing the materials and 3-step procedure with M3 students
C-II Content validation with a blue ribbon panel of clinical educators
C-III Estimating reliability, feasibility, and consequential validity of the 3-step procedure
with M3 students
C-IV Estimating reliability and learning effects with early M4 students
C-V Pilot testing 4-step procedure and estimating reliability with PGY-1 & -2 residents
C-VI Estimating expert-novice differences.

Reliability will be assessed with G and D generalizability studies; feasibility using time on task and reliability data; consequential validity using a questionnaire; and instructional feedback from the assessment profiles generated using observational data. (A group of Japanese educators are testing the three-step procedure with pre-clinical students.) The main strengths of the proposed hypothesis-driven assessment procedure are its sound theoretical foundation, its relative procedural simplicity (including 3 and 4 steps), and its potential for informative and structured feedback to students and educators, and in distinguishing levels of expertise.

Lifelong learning is an essential element of professionalism. In response to the demand for an operational measure of lifelong learning, we developed the Jefferson Scale of Physician Lifelong Learning (JSPLL, 19 Likert-type items). By surveying 444 physicians from the Greater Philadelphia region we provided evidence in support of the psychometric properties of the JSPLL in a previous research study supported by the NBME Stemmler Fund. Nonetheless, the following three questions remain to be addressed by using a nationwide sample of physicians:

I. Is it feasible to generate a general (G) and a specific (S) component (subscale) of the JSPLL, each applicable to a different group of physicians? The two G and S components of the JSPLL will be identified based on the results of factor analysis and content analysis, so that the G component will be more applicable to physicians in patient care who are not involved in teaching and research activities (Group 1), whereas the S component will be applicable to academic physicians who are involved in teaching or research in addition to clinical responsibilities (Group 2). The feasibility of generating the G and S components will be addressed by comparing their psychometric properties and differential validity for physicians in Group 1 and Group 2.

II. What are the predictors of physician lifelong learning? We will examine the contribution of the following measures in predicting the JSPLL (components and total) scores: Academic performance prior to medical school (MCAT, undergraduate GPAs), during medical school (performance in the basic and clinical sciences, rating of clinical competence in core clerkships), scores of the medical licensing examinations (Steps 1, 2, and 3, of the USMLE, formerly Parts I, II and III of the NBME), and ratings of postgraduate clinical competence in three areas of "data gathering," "interpersonal skills," and "socioeconomic aspects of patient care."

III. What are the professional outcomes of physician lifelong learning? We will examine the associations between the JSPLL (components and total) scores and professional outcomes such as board certification, employment status, satisfaction with career, work setting, patient load, teaching, research, publications, and other practice variables. A survey will be mailed to a nationwide sample of 5,412 physicians who graduated from Jefferson Medical College between 1975 and 2000. Multivariate statistical analyses (MANOVA and regression) will be employed. The study will lead to a better understanding of the predictors and professional outcomes of lifelong learning, and a refined assessment instrument useful for the evaluation of lifelong learning among different groups of physicians.