Frequently Asked Questions
If you do not find your question answered here, please contact us here or call the Observational Assessment team at 215-495-6099.
Why is NBME involved in measuring professionalism?
Our mission is to protect the health of the public through state-of-the-art assessments. By addressing professional behaviors, we apply that mission to an area in which the public and physicians themselves see the need for the profession to grow.
Why is NBME offering a formative assessment?
The NBME is offering the APB program as a formative assessment, because it is designed to support continuous learning. Therefore the APB program uses a multisource feedback instrument to collect specific behavioral feedback which can be used by recipients for discussions with their mentor and goal setting.
What is the added value of a standardized approach to assessment of professional behaviors?
A standardized approach creates a common understanding and vocabulary that can focus discussion and attention around professional behaviors within programs, departments, and institutions. Also, our approach benefits from rigorous field testing across specialties and healthcare settings in about 30 programs, a breadth of testing hard for other assessments to achieve. Our program addresses ACGME competencies and LCME and Joint Commission requirements, and will continue to benefit from ongoing research and NBME’s expertise.
What ACGME competencies are addressed in the APB Program?
The APB Program addresses professionalism, interpersonal and communications skills, and practice-based learning. At the invitational conference on Embedding Professionalism in Medical Education in 2002, the AAMC and the NBME convened experts with diverse perspectives on professionalism in part to produce an initial list of professional behaviors that could be used in a formative assessment. The APB instrument was initially constructed from this list and has been updated throughout the years by consulting experts and published literature from the fields of professionalism, interpersonal and communications, and practice-based learning.
What roles do program participants fill?
Program participants fill several roles, including:
Observees are residents, fellows, students, physicians, or other healthcare professionals whose behavior is observed and rated by others using the APB instrument. An observee may also be an observer of others.
Observers are supervisors, mentors, peers, and other healthcare professionals who observe and rate the behavior of observees. A participant may serve as both observee and observer.
Feedback providers are the individuals, usually faculty/mentors, who sit down with an observee and use the APB report to discuss the feedback.
Can we have only faculty evaluate residents?
Yes, but we do not recommend it. When people with varied perspectives and positions provide similar feedback on a behavior, the impact is particularly powerful for observees.
How many people need to participate?
There is no defined number. You do want to make sure you have a large enough pool of observers that no one individual is burdened by completing surveys.
How many observers should provide ratings on an observee in order for us to get a reliable report?
We will provide a feedback report to any observee who has at least three evaluations completed on his/her behavior. Three is a minimum set to maintain the confidentiality of the observers. The more observers you have complete an evaluation on any given observee, the better the reliability. We recommend collecting 15 to 20 surveys per observee per reporting period in order to ensure stability in the ratings and adequate variability in the information to provide useful feedback to the observee.
Is the APB Program appropriate for medical schools, pharmacy schools, physician assistant programs, and other educational programs?
The APB instrument and program reflect what we learned in pilot tests with graduate medical education programs. If you represent another educational program, and are interested in implementing the APB Program, please contact us.
Can we use the APB survey to assess the professional behavior of nurses and other healthcare professionals outside medicine?
If you feel the behaviors on the APB survey instrument are relevant to the evaluation of other healthcare professionals in your program, please contact us, so we can ensure the proper setup of the instrument to conduct this evaluation.
What are the validity and reliability/psychometric properties of the APB instrument?
Reliability refers to the consistency of ratings and is influenced by many factors, including the institutional setting and the number observers. To examine the reliability of the APB scores, we conducted generalizability analyses and examined factors such as training level, number of behavioral items, and number of observers. Results of these analyses support the use of the APB tool as a formative assessment. Please note that as the APB does not produce a total score; Cronbach’s α is not a valid index of reliability.
The validation of the APB instrument is an on-going process of gathering evidence to support the inferences that are made from the APB instrument. The APB instrument was developed using expert opinion and existing research, thus providing a strong foundation for valid inferences. Features of the APB program, including participant training and the ability to indicate “unable to observe”, contribute to the validity evidence base.
How were the survey items selected?
A task force of experts in measurement and professionalism developed a list of behaviors for testing. NBME then conducted a study with residency programs to further test specific items. Residents, nurses, attending physicians, and other respondents rated the importance, observability, and clarity of the items. In keeping with NBME practices, research will contribute to continuous improvement throughout the life of the program.
Is the instrument appropriate to use in making high-stakes decisions (e.g., weeding out bad apples)?
Using results from the APB for high-stakes decisions is not recommended as the APB instrument does not meet the standards necessary to support high-stakes decisions. Nonetheless, APB results can be used as evidence in conjunction with additional sources of information to inform decision-making processes.
Using the APB Program for high-stakes purposes can have negative consequences for the entire program implementation: observers may be less likely to provide honest ratings if they believe the results could affect someone’s career through a high-stakes decision.
Can I see the APB survey instrument?
Yes, if you would like us to send it to you, please contact us.
How will the NBME use participant data?
We keep all data in a dedicated APB database that is not connected to any other database at NBME. There are no common identifiers that would facilitate matching data from the APB Program with data from any other NBME program.
How much time will it take to implement the APB Program?
It depends on your institution and the size of your program. Good planning upfront is extremely important to program success. You will need to think about who will be observing whom and what groups (e.g. nurses, other healthcare professionals) you want to involve. You may also need to get buy-in from other groups to prepare the ground for a strong program. Once you have completed planning, your program administrator can begin scheduling observations, distributing survey instruments, and following up with individuals to help them take advantage of training and other supports. You will receive help with all these decisions and tasks through training and APB team members, who are available to provide support. NBME takes care of loading the survey into your evaluation system and providing you with the feedback reports.
How can I use the results?
The APB provides aggregated feedback in easy-to-read graphical reports that identify areas for improvement. Feedback reports can be used to form the basis of a mentoring discussion designed to highlight strengths and plan improvements.
Can you tell me what other residency programs are currently using this instrument?
We can tell you the number of programs by specialty, but we cannot give out a list of specific programs. We can let you know if another program at your institution is participating.