NBME OPEN ENROLLMENT WORKSHOPS 

Workshop Format

• Synchronous virtual (Zoom) meeting
• Two-hour session
• No pre-work required for participants

Workshop Description

In this interactive workshop, you will learn the fundamental measurement concepts of reliability and validity. We will cover the importance of identifying the intended purpose and use of an assessment, how an assessment’s intended purpose and use relate to the validity of its test scores, and why score reliability is critical in supporting the claims and inference we make about examinees. You will also learn approaches to assess reliability and collect validity evidence throughout the assessment design and evaluation process.

By the end of the workshop, you will be able to:

• Articulate the importance of identifying the intended purpose and use of an assessment
• Define the measurement concepts of reliability and validity
• Explain how reliability and validity are related and why they are important to the quality of an assessment
• Describe different approaches to assess score reliability
• Identify and gather relevant evidence to support score use

Workshop Description

This workshop equips attendees with the skills to confidently interpret and apply information from score reports and dashboards. Participants will explore core statistical concepts alongside principles for interpreting assessment results and recognizing meaningful performance signals. Multiple examples of reports and dashboards will be presented and critiqued, fostering interactive discussion of best practices and practical tips. The workshop will be led by a senior psychometrician with over a decade of experience in communicating and interpreting results from assessments. Pre-read materials will be provided.

By the end of the workshop, participants will be able to:

• Interpret score reports and interactive dashboards to identify meaningful performance signals (e.g., trends, outliers, subgroup gaps)
• Apply core statistical concepts—such as reliability/SEM, distributions/percentiles, aggregation levels, and criterion vs. norm interpretations—to draw conclusions from sample reports/dashboards and avoid common misinterpretation pitfalls.
• Evaluate the design and messaging of score reports or dashboard
  views and propose concrete revisions for improvement.

Workshop Format

• Synchronous virtual (Zoom) meeting
• Two-hour session
• No pre-work required for participants

Workshop Description

In this workshop, you will be introduced to the foundational principles of Evidence-Centered Design (ECD) and how they guide the development of high-quality assessments. We will cover how ECD offers a structured, theory-driven framework for designing assessments that are aligned with their intended interpretations and uses. Through this framework, assessment tasks are explicitly linked to the knowledge, skills, and abilities they are intended to measure, ensuring the evidence gathered supports meaningful and defensible inferences.

By the end of the workshop, you will be able to:

• Describe the core principles of ECD
• Explain how ECD supports the development of validity arguments

Workshop Description

In NBME’s item-writing workshops, participants learn how to design, review, and refine multiple-choice questions (MCQs). This workshop takes the next step: it helps you understand how those questions perform once students respond to them. During this highly interactive session, participants will work through sample item statistics, practice interpreting difficulty and discrimination values, and discuss what these numbers mean for their own assessments. Through collaborative exercise, participants will review example MCQ items, identify potential issues, and consider practical steps for improving test quality. This workshop is designed to be hands-on, supportive, and directly applicable to the assessments that participants use every day.

By the end of the workshop, participants will understand:

• The importance of the intended purpose and use on an assessment
• What it means for a test (score) to be reliable and valid
• How to calculate and interpret common item statistics, such as item
  difficulty and discrimination
• How to interpret common reliability indices
• How item analysis results can be used in the assessment development
  process.