Simulations in the United States Medical Licensing Examination (USMLE).

Over the last several years there has been much attention focused on the detection and remediation of problems that pose potential threats to patient safety and that interfere with the provision of effective care. It has been noted that changes in medical education and assessment are integral to eventual improvement in this area. Within the assessment system used to licence physicians in the United States, there has been an evolution of assessment formats intended to improve the measurement of knowledge and skills, including the recent development of computer based patient simulations and clinical skills assessments. A number of new testing formats intended to further enhance assessment of critical knowledge and skills will be available in the near future.

Review of the score-reporting policy for the United States Medical Licensing Examination.

In 1998, the authors, acting on behalf of the National Board of Medical Examiners (NBME), undertook a review of the scoring policy for the United States Medical Licensing Examination (USMLE). The main goal was to determine the likely effect of changing from numeric score reporting to reporting pass-fail status. Several groups were surveyed across the nation to learn how they felt they would be affected by such a change, and why: all 54 medical boards; 1,600 randomly selected examinees (including 250 foreign medical graduates) who had recently taken either Step 1, Step 2, or Step 3 of the USMLE; 2,000 residency directors; the deans, education deans, and student affairs deans at all 125 U.S. medical schools accredited by the Liaison Committee on Medical Education; and all 17 members of the Council of Medical Specialty Societies. Responses from the different groups surveyed varied from 80% to a little less than half. The authors describe in detail the various views of the respondents and their reasons. Some members in each group favored each of the reporting formats, but the trend was to favor numeric score reporting. The majority of the responding examinees desired that their USMLE scores be sent to them in numeric form but sent to their schools and to residency directors in pass-fail form. Based on the responses and a thorough discussion of their implications, the Composite Committee (which determines USMLE score-reporting policy) decided that there is no basis at this time for changing the current policy, but that it would review the policy in the future when necessary.

Standard-setting plans for the NBME comprehensive Part I and Part II examinations.

The National Board of Medical Examiners (NBME) has reviewed its procedure for setting pass-fail standards in conjunction with the introduction of its comprehensive Part I and Part II examinations in 1991. This report gives background information on the procedures used for the past decade to set pass-fail standards for the Part I and Part II examinations, an overview of the NBME's research on standard setting, under way since 1987, and a statement of its plans for determining pass-fail standards for these examinations. In 1981 the NBME changed from the norm-referenced standard, used since the 1950s, to a criterion-group approach to setting pass-fail standards. Although the criterion-group system resulted in more stable standards, it still meant that the standard moved whenever the performance of the reference group changed. After conducting research, surveying constituencies, and examining alternatives, the NBME has adopted a new standard-setting plan that has the following components: a content-based standard-setting procedure; determination of standards by an appropriate group; use of a fixed standard; and periodic review of standards and standard-setting procedures. This new process will produce three types of improvements: it will incorporate deliberations informed by a wide range of information, including content review; annual review of examinees' performances and pass-fail results and triennial restudy of the process will add further quality control; and a fixed standard will mean that comparable performances will be required across administrations in order to pass.

Continued research on computer-based testing.

The National Board of Medical Examiners has developed computer-based examination formats for use in evaluating physicians in training. This paper describes continued research on these formats including attitudes about computers and effects of factors not related to the trait being measured; differences between paper-administered and computer-administered multiple-choice questions; and the characteristics of simulation formats. The implications for computer-based testing and further research are discussed.