Stemmler Fund Current Grant Information

In addition to accessing specific grant abstracts separately, you can also click here to open or close all abstracts for all grantees on this page.

Study objectives:  The ability to effectively function as a healthcare team member
is now regarded as a prerequisite skill for all medical professionals. One of the
challenges of an intern’s first days as a physician is taking call, and responding to
nurses’ pages about patients in need of immediate attention. Despite the anxiety
associated with this universal responsibility, little attention has been paid to trainees’
preparation to safely and effectively perform this task. The purpose of our study is to
evaluate the validity and feasibility of an instrument for assessing the interpersonal skills
and clinical decision-making in a common physician task that has unique
communication requirements, phone consultations with nurses.

Rationale:  Interprofessional communication is a critical aspect of safe team
function, and general assessments of communication do not readily capture the nature
of specific team interactions. A validated instrument designed to measure the
communication and decision-making involved in phone consultations about patients will
afford us the opportunity to conduct competency assessment and standard-setting in
this important patient care skill.

Methods:  Simulated surgical cases for physician-nurse consultations, or mock
pages, will be developed by a multi-disciplinary team of physicians, nurses, and
educational specialists. Each mock page will include a two-part evaluation: one part
specific to the case, and another which will be general to phone communication with a
nurse. The evaluation instruments to be refined were adapted from the literature and
have been used in an earlier pilot project. The assessment will be composed of 10
standardized cases, implemented and evaluated by a nurse. Approximately 140 senior
medical students (70 per year) and 30 surgery residents from 5 institutions will be given
pre-determined times to be on call. A nurse from the paging team will page and present
participants with a hypothetical surgical scenario, and evaluate their performance. A
random sample of one-third of the calls will be independently evaluated by two raters, a
nurse and a surgeon, from the audiotapes of those case performances. In Year 2, a
subset of student participants will be selected for follow-up in the clinical setting. Ratings
of their in situ phone consultation performances will be gathered from nurses. Analyses
will be conducted on raters’ scores to determine interrater reliability and internal
consistency. Construct validity will be assessed by comparing student (novice) and
senior resident (experienced) performances on each case. Predictive validity will be
determined with a correlation between Year 1 student performance and their
performance as a surgery intern. A standard-setting method will be employed to identify
performance standards for determining competency on phone consultations. Feasibility
will be studied via cost analysis of hours of time spent and equipment costs.

Expected outcomes:  The validation of an instrument to measure performance
during phone consultations with nurses will be useful in determining medical student
and resident competence in this important skill. In addition to its value as an
assessment instrument, its use would serve to enhance the specificity of feedback
about physician communication. The feasibility of our assessment strategy will be
studied, to inform the possible extension to disciplines beyond surgery. The expected
outcome of our research is to provide a feasible and worthwhile model for educational
interventions and competency assessment that could be applied across disciplines.

Rationale:  National accreditation bodies including the Accreditation Council for Graduate Medical Education (ACGME) and the American Association of Medical Colleges (AAMC) have called for competency based curriculum and assessment models for training programs. Many medical schools have responded with education portfolios to capture student experiences in real and simulated environments matched to shared competency goals. However, education portfolios have not achieved widespread adoption, partially because current methods require significant manual entry of a limited amount of clinical data. Automatic and valid methods of capturing the richness of students’ clinical experiences are needed. In this project, we will apply advanced informatics techniques to a complete repository of medical students’ clinical notes to automatically locate students’ experiences and assess student proficiency according to the AAMC’s medical student geriatrics competencies.

Project objectives:

1. Develop descriptive anchors that express the AAMC geriatric competencies as a national standard for medical education.
2. Develop and validate automated tools to identify a student’s experience with a given competency based on student notes.
3. Develop and validate automated tools to assess student proficiency based on student notes.
4. Compare the student clinical note representation of geriatric experience and proficiency to external validations using standardized patient interactions and a validated medical knowledge exam.

Methods:  Our project leverages two existing transformational systems. We have developed an automated concept indexer that uses natural language processing techniques to map free-text clinical notes to standardized terminologies. The second system, Learning Portfolio, collects all student notes from patient encounters as they are created in the electronic medical record (EMR). We will apply existing algorithms to accurately locate all medical concepts in the clinical notes, separated by document section (e.g., “chief complaint” and “past medical history”). We will develop sets of descriptive terms that represent geriatrics competencies in clinical notes through literature review and local and national collaboration with geriatric educators. We will identify exposures to each geriatric competency using these descriptors, and then rank proficiency with which the competency was addressed (“advanced”, “intermediate”, or “novice”). We will internally validate the algorithms by comparing the computer ratings of notes to a gold standard of physician review of the notes. We will externally validate the algorithms using student completion of observed structured clinical examinations and a validated geriatrics knowledge exam.

Significance for medical education assessment:  This project will produce a tool that automatically rates clinical notes according to national geriatric competencies, and will provide an evaluation of the role of clinical notes in competency assessment. This work will establish criteria for the application of the AAMC geriatric competencies at other institutions through a set of descriptive terms that could be applied to other sets. This work introduces new methods to identify and rate performance outcomes that can be applied to other topics, competencies, and modalities of assessment, such as reflections, narrative evaluations, essays, and other written work.