Preface to Third Addition
This manual was written to help faculty members improve the quality of the multiple-choice questions written for their examinations. The manual provides an overview of item formats, concentrating on the traditional one-best-answer and matching formats. It reviews issues related to technical item flaws and issues related to item content. The manual also provides basic information to help faculty review statistical indices of item quality after test administration. An overview of standard-setting techniques is also provided. Issues related to exam blueprinting are not addressed in any detail. We have focused almost exclusively on the item level, leaving exam level planning for another manuscript.
We anticipate that this manual will be useful primarily by faculty who are teaching medical students in basic science courses and clinical clerkships. The examples focus on undergraduate medical education, though the general approach to item writing may be useful for assessing examinees at other levels.
This manual reflects lessons that we have learned in developing items and tests over the past 20 years. During this period, we have reviewed (quite literally) tens of thousands of multiple-choice questions and have conducted item-writing workshops for thousands of item writers preparing USMLE, NBME, and specialty board examinations as well as faculty at more than 60 medical schools developing test questions for their own examinations. Each workshop attendee has helped us to frame our thoughts regarding how to write better quality test questions, and, over the years, we have become better able (we believe) to articulate the why's and wherefore's. We hope this manual helps to communicate these thoughts.
Susan M. Case, PhD
David B. Swanson, PhD
January 1998
Table of Contents
Section I
- Issues Regarding Format and Structure of Test Questions
Chapter 1. Introduction
- Assessment: An Important Component of Instruction
- Issues of Sampling
- Importance of Psychometric Considerations
Chapter 2. Multiple-Choice-Item Formats
- True/False vs One-Best-Answer Questions
- The True/False Family
- The One-Best-Answer Family
- The Bottom Line on Item Formats
Chapter 3. Technical Item Flaws
- Issues Related to Testwiseness
- Issues Related to Irrelevant Difficulty
- Summary of Technical Item Flaws
- Use of Imprecise Terms in Examination Questions
Section II
- Writing One-Best-Answer Questions for the Basic and Clinical Sciences
- The Basic Rules for One-Best-Answer Items
Chapter 4. Item Content: Testing Application of Basic Science Knowledge
- Item Content for the Basic Sciences
- Item Templates
- Additional Templates
- Types of Questions and Sample Lead-ins and Option Lists
- Writing the Options: Altering Item Difficulty
- Item Shape
- Problem-Based Learning and Use of Case Clusters
- Sample Items for the Basic Sciences
Chapter 5. Item Content: Testing Application of Clinical Science Knowledge
- Methods for Assessment
- General Issues Regarding What to Test
- Testing Recall of Isolated Facts or Application of Knowledge
- Writing One-Best-Answer Items
- Fine Points on Item Stems
- Verbosity, Window Dressing, and Red Herrings: Do They Make a Better Test Item?
- Writing Items Related to Physician Tasks
- Writing Items on Difficult Topics
Section III Extended-Matching Items
Chapter 6. Extended-Matching (R-Type) Items
- Avoiding Flaws When You Write Extended-Matching Items for Your Own Examination
- Sample Lead-ins and Topics for Option Lists
- More on Options for R-Sets
- Writing the Item Stems
- Sample Good and Bad Item Stems Using the Same Option List
- Overview of the Steps for Writing Extended-Matching Items
- Sample Extended-Matching Sets
- Steps for Organizing a Group to Write Clinical R-sets
- Form for Writing R-Sets
- Sample SPSSX Code to Score Multiple-Choice Tests Including Extended-Matching Items
- Comparison of Items in Five-Option and Extended-Matching Format
- A's to R's and Back Again
Chapter 7. Pick N Items: An Extension of the Extended-Matching Format
Section IV
Chapter 8. Interpretation of Item Analysis Results
Chapter 9. Establishing a Pass/Fail Standard
- Definitions and Basic Principles
- Two Standard-Setting Methods Based on Judgment about Items
- Relative/Absolute Compromise Standards: The Hofstee Method
Chapter 10. Miscellaneous Thoughts on Topics Related to Testing
Appendix A. The Graveyard of NBME Item Formats
Appendix B. Sample Item-Writing Templates, Items, Lead-Ins, and Option Lists for the Basic and Clinical Science
