Stemmler Fund Current Grant Information

In addition to accessing specific grant abstracts separately, you can also click here to open or close all abstracts for all grantees on this page.

Clinical decision making requires two distinct skills: the ability to classify patients' conditions into diagnostic and management categories that permit the application of "best evidence" guidelines, and the ability to individualize or - more precisely - to contextualize care for patients whose circumstances and needs require variation from the standard approach to care. Most assessment in medical education places heavy emphasis on biomedical decision-making with little emphasis on how to incorporate contextual factors that may be essential to planning patients' care.

The goal of this project is to demonstrate and provide validity evidence for an innovative standardized patient (SP) method of assessing medical students in the clinical years on their ability to detect and respond to individual contextual factors in a patient encounter that overcomes the aforementioned challenges. As such, the project is designed to directly address the Stemmler Fund goal of research and development of innovative assessment approaches to enhance the evaluation of those preparing to practice medicine.

During the project, 144 fourth-year medical students participating in a Medicine sub-internship will be randomized to an intervention group or a control group; the intervention group will receive additional training in the application of qualitative methodology to elicit and incorporate contextual factors in the clinical encounter. All students will participate in an SP assessment consisting of four SPs, blinded to trial arm, presenting cases with and without important biomedical and contextual factors in a counterbalanced factorial design. Performance will be compared between trial arms. In addition, performance will be compared with USMLE Step 2 clinical knowledge scores to determine whether contextualizing ability is independent of clinical knowledge, and consistency of performance across individual SP cases will be studied to determine the number of cases necessary to achieve sufficient reliability for the assessment to be used.

The outcomes of this project, which will be widely disseminated to permit replication at other medical schools, will include: (1) a well-documented method for developing SP assessments designed to test the ability of a trainee to contextualize care, (2) evidence for the ability of the assessments to distinguish between trainees with differing levels of skill in contextualization using a randomized controlled educational trial, (3) evidence that assessment scores are not predicted by clinical knowledge, and (4) evidence for internal consistency in scoring of the assessments.

Rationale: Research into effective approaches to significant learning has generated two powerful descriptive models: formative assessment and deliberate practice. Although the two models approach student learning from somewhat different perspectives, the congruence between them is striking. Unfortunately, realizing the full power of either model requires significant manpower, time, and expertise - resources that are difficult to achieve and to sustain. Technological approaches can reduce the human resource requirements and make the learning benefits of these models more accessible for medical educators and their students.

Objectives: In support of the overarching goal of improving medical students' clinical reasoning skills, this proposal specifically aims to evaluate the core, shared strengths of the formative assessment and deliberate practice models as implemented through a resource-sparing, web-based technology - Calibrated Peer ReviewT. The objectives are to answer the following questions. While using Calibrated Peer Review do learners:

1. Focus on improving performance in the well-defined task of patient note-writing?
2. Recognize and give informative feedback?
3. Incorporate feedback into iterative practice to improve performance?
4. Report that the new learning is integrated successfully into new performances?

Methods: This proposal begins with archival data from patient notes written in Calibrated Peer Review T by four medical student classes (2005-2008) and continues gathering note-writing data, along with survey and focus group information, from two additional classes (2010 & 2011). Four studies are proposed. 1) The archival study evaluates individual factor correlations using the scores from 3,468 completed "assignments," as well as trends in score deviation of the peer review and self-assessment over time. 2) The quality of feedback study codes students' narrative feedback to analyze quality trends over assignment iterations. 3) The perceptions of feedback study asks students' opinions about sample feedback of varying quality. 4) The student survey and focus group study constructs questions to assess students' habits in preparation for note-writing during the performance examinations, their perceptions of the CPR system, and how it aids their learning. Descriptive statistics, inferential statistics, qualitative analyses, and analysis of change via Hierarchical Linear Modeling, as relevant to each research question, will be conducted.

Significance for Assessment in Medical Education: This web-based program applies the principles of formative assessment and deliberate practice to provide a powerful learning experience for students. It emphasizes the importance of the learner's focus on improving performance, accurate observation and informative feedback, repetition, and the formation of cognitions that reflect the true complexity of the clinical task and thus bring the learner to integrate that improvement into his or her everyday work. The judicious use of this technological approach reduces the intensive resources required for effective learning - a true innovation in 'assessment as learning' in medical education.

Background and Rationale: Successful delivery of health care relies on effective team coordination and communication. A major shortcoming of the recent push toward greater education of teamwork skills in medicine is our limited ability to assess the effectiveness of different forms of team skill training with respect to behavior change. There is a critical need to develop valid, reliable, and feasible methods of assessing health care team coordination skills. A number of observer-based rating tools have been implemented, with limited success. While some successes have been noted in the case of specialty-based tools for specific work environments, general team skill assessment tools intended to apply to a broader audience, or for the purpose of medical and nursing education, have not been strongly supported. Limitations of current assessment methods include: (1) a failure to attain high levels of inter-rater agreement with observer-based team skill rating scales, and (2) a failure to adequately assess teamwork skills with respect to managing difficult situations or difficult team members.

Objective: Our long-term goal is to develop, validate, and share a standardized team skill assessment (STSA) tool focused on evaluating critical health care teamwork behaviors. We have developed a STSA tool that embeds trainees with standardized team members (actors playing the roles of health care team members, similar to the accepted practice of using standardized patients) in difficult teamwork scenarios. Scoring of performance is observer-based rating of scenario-specific ideal team skill behaviors. This tool overcomes limitations of current assessment methods by (1) assessing trainees in a standardized scenario that is the same for each trainee (not dependent on the participation of other trainees) (2) using scenarios that stress critical teamwork skills in difficult scenarios, and (3) basing scoring on specific observable behaviors that are easily identified, within the context of each scenario. The primary objective of this project is to assess the feasibility, reliability, and validity of this STSA.

Methods: We will use the STSA tool to assess 30 medical and nursing students before and after teamwork training content is delivered in their capstone courses. We will assess reliability, validity, and generalizability through multivariate generalizability analysis of behavioral ratings by standardized team members (immediately following interaction) and trained observers (from videotape). Generalizability analysis is a statistic useful for assessing the reliability of a measure when there are multiple sources of variability in the measure. We will assess the proportion of variance in team skill ratings attributable to: (1) trainee, (2) pre- vs post-training, (3) rater type (actor/live vs. observer/tape), (4) medical vs. nursing student, (5) rater, and (6) scenario. We hypothesize a large proportion of variability in student scores attributed to trainee and pre- vs. post-training. We expect a low proportion of the variability attributed to rater and rater type. We will predict the number of repetitions (e.g., number of scenarios) required to achieve a reliable team skill score. We will also assess feasibility and validity of the STSA through subjective surveys of clinician-teachers, trained observers, standardized team members, and students.

Advancing assessment in medical education and practice: This research is innovative because it presents a novel approach to assessing team skills and a rigorous method of analyzing the feasibility, reliability, and validity of that approach. We expect the results of this effort to advance knowledge in medical education assessment as it relates specifically to measuring health care teamwork skills. This includes advancement in knowledge related to details such as scenario presentation, actor involvement, and rater training for the purposes of low fidelity simulation-based assessment. We also expect to advance knowledge related to methods of analyzing the validity, reliability, and generalizability of these types of assessment tools.