Evidence for the validity of a Rasch model technique for identifying differential item functioning.

This paper presents an analysis of differential item functioning (DIF) in a certification examination for a medical specialty. The groups analyzed were (1) physicians from different subspecialties within this area and (2) physicians who qualified for the examination through two different experiential pathways. The DIF analyses were performed using a simple Rasch model procedure. The results were shown to be readily interpretable in terms of the known differences between the groups being compared. These results serve as validity evidence for the Rasch model procedure as a means for evaluating DIF in examinations. The conclusion is drawn that complex procedures are not required to generate interpretable results if relevant differences between the groups being compared are known. This suggests that the inability of many researchers to interpret results for racial/ethnic or gender groups is not due to inadequacies of the methods, but more likely to lack of pertinent knowledge about group differences.

The in-training examination in internal medicine.

OBJECTIVE: The In-Training Examination in Internal Medicine (ITE-IM) has been offered to internal medicine trainees annually since 1988 as an instrument for self-assessment. This report outlines the manner in which the test is prepared, reviews the results of annual examinations, and analyzes trends during the past 6 years. DESIGN: Results of each examination were reviewed with regard to the demographic characteristics of persons taking the test, their previous medical training, and their present program affiliations. RESULTS: Then number of residents participating in the ITE-IM has increased steadily over the past 6 years. In 1993, more than 12,000 residents from more than 90% of internal medicine training programs in the United States participated in the examination; the percentage of international medical school graduates taking the examination increased from 27% in 1988 to 47% in 1993. Statistical analyses of each examination have shown it to be reliable, internally consistent, and discriminating. Over the past 6 years, graduates of U.S. medical schools have scored consistently higher than those of international medical schools and schools of osteopathic medicine on all annual examinations. However, in 1993, for residents at all levels of training, the differences in scores between graduates of U.S. medical schools and graduates of international medical schools narrowed substantially. From 1988 to 1993, there has been a trend toward lower scores by every cohort on each subsequent examination. The decreases in scores are most pronounced for graduates of U.S. medical school and those of schools of osteopathic medicine. The lower scores may be caused by either an increased level of difficulty in the examination or decreased knowledge among examinees. CONCLUSIONS: The ITE-IM is a useful instrument to assess the knowledge base of residents and program directors with a reliable evaluation of themselves and their programs in comparison to their national peer groups. It also provides objective data to monitor trends over time in residents' scores and relates them to the changing demographic characteristics of trainees and to innovations in the clinical curricula of internal medicine training programs.