Clinical performance-based test sensitivity and specificity in predicting first-year residency performance.

To assess the use of a standardized-patient-based postclerkship examination (PCX) to predict students' performances in their first year of residency, the authors used data from 202 students of the classes of 1987, 1988, and 1990 at the Southern Illinois University School of Medicine. The PCX was found to be more sensitive than specific in that it identified more correctly those students who received high ratings in their first year of residency than those who received low ratings. Consequently, while the rate of false negatives was low across the three classes, the rate of false positives was relatively high. Analyses of the supervisors' written comments on residents' performances indicated that more than half of the false positives and false negatives resulted mostly from problems found with the supervisors' ratings, rather than from the inaccuracy of the PCX in predicting students' residency performances. The supervisors' ratings did not always match their written comments and did not always represent pure assessments of the residents' cognitive performances. Finally, because of the large number of missing residency performance ratings among the graduates who had performed unsatisfactorily on the PCX, it is expected that the sensitivity of the PCX may be overestimated and its specificity underestimated. Overall, the results suggest that the standardized-patient-based PCX is a useful indicator of students' readiness for and performance in residency.

Examination of nonresponse bias in a major residency follow-up study.

Nonresponse bias is a potential problem for follow-up studies that track medical school graduates into their residencies. There are two main types of nonresponse: when residents' supervisors do not complete the study evaluation forms and when medical students refuse permission to allow follow-up evaluations during their residencies. The present study discloses the biases that each of these types of nonresponse produced on a set of standard measures of 508 residents from eight 1980s medical school classes at Southern Illinois University. The standard measures (e.g., MCAT scores, grade-point averages, NBME scores) were collected when the residents were medical students. The findings clearly indicate that the scores of the medical students who granted permission for follow-up evaluations had significantly larger means than did the scores on the same measures of those students not granting permission, clearly a source of nonresponse bias. On the other hand, no significant difference in scores (thus negligible bias) was found in cases where supervisors did not complete forms. The authors offer suggestions for medical school policies based on these findings.

The use of performance-based assessment scores and traditional measures in predicting first-year residency performance.

The purpose of this study is to examine the relationship between students' performance on a performance-based clinical examination and their subsequent performance during the first year of residency. Additionally, its purpose is to further examine the relationship between the performance-based clinical examination and existing measures of clinical competence (clerkship ratings, and NBME Part I and II scores).