Collaboration across the pond: the multi-school progress testing project.

This collaborative project between the National Board of Medical Examiners and four schools in the UK is investigating the feasibility and utility of a cross-school progress testing program drawing on test material recently retired from the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) examination. This article describes the design of the progress test; the process used to build, translate (localize), review, and finalize test forms; the approach taken to (web-based) test administration; and the procedure used to calculate and report scores. Results to date have demonstrated that it is feasible to use test items written for the US licensing examination as a base for developing progress test forms for use in the UK. Some content areas can be localized more readily than others, and care is clearly needed in review and revision of test materials to ensure that it is clinically appropriate and suitably phrased for use in the UK. Involvement of content experts in review and vetting of the test material is essential, and it is clearly desirable to supplement expert review with the use of quality control procedures based on the item statistics as a final check on the appropriateness of individual test items.

Cumulative achievement testing: progress testing in reverse.

This collaborative project between the National Board of Medical Examiners (NBME) and Case Western Reserve University (CWRU) School of Medicine explored the design and use of cumulative achievement tests in basic science education. In cumulative achievement testing, integrative end-of-unit tests are deliberately constructed to systematically retest topics covered in previous units as well as material from the just-completed unit. CWRU faculty developed and administered a series of six web-based cumulative achievement tests using retired United States Medical Licensing Examination (USMLE) step 1 test material and tools provided by NBME's Customized Assessment Services, and trends in student performance were examined as the new CWRU basic science curriculum unfolded. This article provides the background information about test design and administration, as well as samples of score reporting information for students and faculty. While firm conclusions about the effectiveness of cumulative achievement testing are not warranted after a pilot test at a single school, preliminary results suggest that cumulative achievement testing may be an effective complement to progress testing, with the former used to encourage retention of already-covered material and the latter used to assess growth toward the knowledge and skills expected of a graduating student.

Use of a committee review process to improve the quality of course examinations.

PURPOSE: In conjunction with curricular changes, a process to develop integrated examinations was implemented. Pre-established guidelines were provided favoring vignettes, clinically relevant material, and application of knowledge rather than simple recall. Questions were read aloud in a committee including all course directors, and a reviewer with National Board of Medical Examiners (NBME) item writing and review experience. This study examines the effectiveness of this process to improve the quality of in-house examinations. METHODS: Five hundred and twenty items were randomly selected from two academic years for initial comparison; 270 from 2000 to 2001, and 250 from 2001 to 2002. The first set of items represented the style, content and format when courses and tests were departmentally/discipline based, assembled by course directors, and administered separately. The latter group represented similar characteristics when courses and tests were organ-system-based, committee-reviewed and administered in an integrated examination. Items were randomized, blinded for year of origin, and rated by three NBME staff members with extensive item review experience. A five-point rating scale was used: one indicated a technically flawed item assessing recall of an isolated fact; five indicated a technically unflawed item assessing application of knowledge. To assess continued improvement, a follow-up set of 250 items from the 2002 to 2003 academic year was submitted to the same three reviewers who were not informed of the purpose or origin of this set of test items. RESULTS: The mean rating for items from 2000 to 2001 was 2.51 +/- 1.27; analogous values for 2001-2002 were 3.16 +/- 1.33, (t = 5.83; p < 0.0001), and in 2002-2003; 3.59 +/- 1.15 (t = 10.11; p<0.0001). CONCLUSION: Pre-established guidelines and an interdisciplinary review process resulted in improved item quality for in-house examinations.

Simulations in the United States Medical Licensing Examination (USMLE).

Over the last several years there has been much attention focused on the detection and remediation of problems that pose potential threats to patient safety and that interfere with the provision of effective care. It has been noted that changes in medical education and assessment are integral to eventual improvement in this area. Within the assessment system used to licence physicians in the United States, there has been an evolution of assessment formats intended to improve the measurement of knowledge and skills, including the recent development of computer based patient simulations and clinical skills assessments. A number of new testing formats intended to further enhance assessment of critical knowledge and skills will be available in the near future.

Review of the score-reporting policy for the United States Medical Licensing Examination.

In 1998, the authors, acting on behalf of the National Board of Medical Examiners (NBME), undertook a review of the scoring policy for the United States Medical Licensing Examination (USMLE). The main goal was to determine the likely effect of changing from numeric score reporting to reporting pass-fail status. Several groups were surveyed across the nation to learn how they felt they would be affected by such a change, and why: all 54 medical boards; 1,600 randomly selected examinees (including 250 foreign medical graduates) who had recently taken either Step 1, Step 2, or Step 3 of the USMLE; 2,000 residency directors; the deans, education deans, and student affairs deans at all 125 U.S. medical schools accredited by the Liaison Committee on Medical Education; and all 17 members of the Council of Medical Specialty Societies. Responses from the different groups surveyed varied from 80% to a little less than half. The authors describe in detail the various views of the respondents and their reasons. Some members in each group favored each of the reporting formats, but the trend was to favor numeric score reporting. The majority of the responding examinees desired that their USMLE scores be sent to them in numeric form but sent to their schools and to residency directors in pass-fail form. Based on the responses and a thorough discussion of their implications, the Composite Committee (which determines USMLE score-reporting policy) decided that there is no basis at this time for changing the current policy, but that it would review the policy in the future when necessary.

Assessment in basic science instruction: directions for practice and research.

In this essay, we discuss assessment of students' understanding of the basic biomedical sciences during the basic science component of medical education. Because we strongly believe that assessment should follow from and be congruent with course and curricular goals, the first section discusses recent trends in basic science education: where it's been and where it appears to be headed. The second section discusses considerations in the design of an assessment system that reflects curricular goals and encourages students to achieve them; it also identifies some research issues that we think merit attention. In the last section, we outline some of the special assessment issues and problems that arise in the context of problem-based learning (PBL) and discuss the merits of some of these novel assessment approaches, as well as the use of more traditional methods in PBL curricula.

Relationship between students' performances on the NBME Comprehensive Basic Science Examination and the USMLE Step 1: a longitudinal investigation at one school.

PURPOSE: To examine students' growth in basic science knowledge during medical school and to evaluate the accuracy of students' scores on the National Board of Medical Examiners Comprehensive Basic Science Subject Examination (CBSE) as predictors of their performances on Step 1 of the United States Medical Licensing Examination (USMLE). METHOD: A public medical school in the southwestern United States evaluated 58 students from the entering class of 1993 by administering the CBSE in April 1994, December 1994, and February 1996. These students then sat for the USMLE Step 1 in June 1996. For each CBSE administration, descriptive statistics were calculated and least-squares regression analyses were performed to predict the students' Step 1 scores from their CBSE scores. RESULTS: The students' CBSE scores improved as they progressed through their basic science course work and clinical clerkships. The strongest correlation (r = .85) between the students' CBSE scores and their Step 1 scores was for the second CBSE administration; the weakest correlation (r = .73) was for the first CBSE administration. CONCLUSION: These results indicate that basic science knowledge continues to grow throughout the first three years of medical school and that the CBSE is a useful tool for the identification of students at risk for failing the USMLE Step 1.