The effect of examinee and patient ethnicity in clinical-skills assessment with standardized patients.

CONTEXT: Ethnicity has been a continuing concern for the valid assessment of clinical performance with standardized patients (SPs). The concern is that examinee ethnicity and SP ethnicity might interact, such that examinees might score higher in encounters with SPs of the same ethnicity. OBJECTIVE: To test for an interaction of examinee ethnicity and SP ethnicity on clinical performance in an SP examination. MAIN OUTCOME MEASURES: History-taking and physical-examination scores and interpersonal-and communication-skills scores, both based on checklists completed by SPs. Poststation scores for answers to case-related questions concerning pathophysiology, diagnosis, test selection, and test interpretation. SETTING AND PARTICIPANTS: Two graduating classes of over 1,000 fourth-year medical students each in the New York City Consortium were tested on the SP assessment administered at The Morchand Center of Mount Sinai School of Medicine. DESIGN AND ANALYSIS: The primary analyses were two-way (2 x 2) analyses, to test the main and interaction effects of examinee ethnicity and SP ethnicity. Effect-size measures (standardized mean differences, d) were computed to provide a sharper picture of the effects. RESULTS: Of the 24 interaction analyses, only three were statistically significant (not significantly more than expected by chance) and the results were mixed: one analysis showed better examinee performance in encounters with SPs of the same ethnic background and the other two showed the opposite. For all 24 interactions, significant or not, the results showed weak effects and no consistent pattern. White examinees scored on average 0.12 standard deviations above black examinees in encounters with white SPs, and 0.11 standard deviations higher in encounters with black SPs. CONCLUSIONS: These initial results are encouraging and should dispel some of the concern about ethnicity in SP assessment, at least about the operation of an examinee-by-SP-ethnicity interaction that would pose a serious threat to the validity of the examination scores.

Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment.

PURPOSE: To evaluate the relationship between clinical competence and interpersonal and communication skills, in an attempt to clarify current thinking about these two dimensions as measured with standardized-patient (SP) examinations. METHOD: Simple Pearson correlations were computed between total examination scores for clinical competence and interpersonal and communication skills. Three sets of different types of data involving 15 separate examinations were used to explore the generality of the findings. To control for a within-case halo effect and measurement error, corrected cross-half correlations and corrected cross-case correlations were also computed. RESULTS: The simple correlations and the corrected cross-half and cross-case correlations showed moderate and above relationships between these two dimensions in the clinical context. The simple correlations centered around .50, and the corrected cross-half and cross-case correlations were slightly higher, centering around .65 and .70, respectively. CONCLUSION: The authors' thinking is that the moderate relationship between clinical competence and interpersonal and communication skills is not due to a flaw in the measurement of clinical competence, as has been suggested, but rather is a natural consequence of the clinical encounter, which exacts an interdependence of these two dimensions. At least, this possibility must be seriously considered so medical educators can think and act appropriately in the assessment of clinical performance.

A southern Illinois eyecare provider survey.

PURPOSE: To determine opinions and knowledge of southern illinois eyecare patients about the different types of eyecare providers and the services they perform. METHODS: A two page survey (Appendix) was presented to 500 consecutive patients registering at the Southern Illinois University Eye Center in April, 1995. Patients were asked to respond to questions which tested their knowledge of the type of provider they were to visit, i.e., ophthalmologist, optometrist, or optician and the services provided. RESULTS: There was pronounced confusion and uncertainty especially with regard to the terms "ophthalmologist" and "optometrist." Approximately one-quarter of those polled believed an optometrist to be a physician, and another one-quarter was unsure. Only one-third of those surveyed reported that "OD" referred to "optometrist." Approximately one-third were unsure if an optician was a physician. Over eighty percent felt there was confusion concerning the different types of eyecare providers and only 49% felt they were well informed with regard to providers and their services. CONCLUSION: The public, even among eyecare patients, is poorly informed with regard to the different types of eyecare providers and their services.

The effect of formal feedback sessions on test security for a clinical practice examination using standardized patients.

OBJECTIVE: To determine the effects of formal feedback on test security for a clinical practice examination administered to successive rotations over a year's time. METHOD: A seven case clinical performance examination was administered to ten rotations of students throughout an academic year in a required Ambulatory Care Clerkship. Three of the cases were developed at Saint Louis University School of Medicine (SLU); four of the cases were developed by the National Board of Medical Examiners (NBME). Immediately following each examination, the examinees participated in a feedback session for just the three SLU cases: checklists were displayed, and details of the cases were discussed. No feedback on the four NBME cases was provided. Student performance in successive rotations on the three cases in which students were given detailed feedback by the faculty was compared with performance on the four cases in which no feedback was given. SAMPLE: Data were collected from 119 students. RESULTS: The interaction of case and group was not significant, indicating no difference in the trend over the ten groups between the three cases for which feedback was provided and the four cases for which it was not. The results showed no significant increasing trend over the ten rotation groups for either cases where feedback was given or withheld. CONCLUSION: The results suggest that the formal feedback sessions did not pose a threat to test security.

Validity of simple approach to scoring and standard setting for standardized-patient cases in an examination of clinical competence.

The results are disappointing, providing little support for the validity of the case-passing decisions based on this simple approach to scoring and standard setting. The case-passing decisions predicted what the case author intended for about only 73% or 74% of the students on average and, with agreement expected by chance removed, predicted what the case author intended for about only 25% of the students. Even with the use of the optimal pass/fail cutoffs and the dropping of students with ambiguous borderline global ratings, the case-passing decisions failed to agree with the case authors' global ratings for 15% to 30% of the students. The findings might be dismissed as simply due to low reliabilities of passing decisions and global ratings based on a single case. Although this concern would apply to intercase reliabilities, which would be subject to case specificity, the appropriate reliabilities here would seem to be intracase (i.e., intrarater), which should be fairly high (if they could be computed). Nevertheless, it seems reasonable to expect much better agreement between results of case scoring and of standard setting developed by the case author and the case author's global ratings of performance on that case, given that the case author might recall the checklist, assign a weight to each item, and so forth. Also, case-passing decisions would possibly agree more with global ratings of live or videotaped performances than with ratings of written summaries of performance; however, that question remains a challenge for further research. In conclusion, the study provides only weak evidence, at best, for the validity of the scoring and standard setting commonly used with SP assessment. The results do not undermine claims about the realism of the SP approach, however, nor do they call into question the standardization afforded by this method of assessing clinical competence. The results do raise serious concerns about this simple approach to scoring and standard setting for SP-based assessments and suggest that we should focus more on the observation and evaluation of actual student performance on SP cases in the development of valid scoring and standard setting.

A large-scale study of the reliabilities of checklist scores and ratings of interpersonal and communication skills evaluated on a standardized-patient examination.

Checklists and rating scales are both used for assessing examinees in standardized-patient (SP) examinations. A common presumption, is that checklists are more objective than and hence superior to rating scales with respect to psychometric properties such as reliability. Recently, this presumption has been questioned and studies have been cited which support the clain that "objectified methods do not inherently provide more reliable scores." The purpose of this study was to further explore this issue, using a specially designed checklist/rating scale form for assessing interpersonal and communication skills. Methods. A 26-item checklist/rating scale form was developed, which consisted of five sections of 3 to 7 checklist items each, with a space provided at the end of each section for the SP to make a single global rating of the items in that section. Analyses were performed on data for 1,048 fourth-year medical students in the eight schools in the New York City Consortium, tested on the 7-case SP examination administered at The Morchand Center of Mount Sinai School of Medicine. Results/Conclusion. The intercase reliabilities for the checklist scores and ratings, respectively, were: 0.33 and 0.39 for eliciting information, 0.33 and 0.39 for non-verbal behavior, 0.55 and 0.52 for patient education, 0.48 and 0.45 for professional manner, and 0.49 and 0.52 for patient satisfaction. The correlations between the checklist scores and ratings for the five sections were 0.66, 0.60, 0.80, 0.69, and 0.75, respectively. The pattern of these results provides further support for the claim that "objectified" measures, like checklists, "do not inherently provide more reliable scores."

Comparing checklists and databases with physicians' ratings as measures of students' history and physical-examination skills.

PURPOSE: To compare two methods of rating students' performances on history and physical examination: (1) by using checklists completed by standardized patients (SPs) and databases completed by students, and (2) by using ratings of students by three physicians for each SP-student encounter. METHOD: Four cases were chosen for the study, and 30 students were examined per case. The students were all in their fourth year at the Southern Illinois University School of Medicine in the spring of 1991. Two of the cases had both checklists and databases, and the remaining two had databases only. Each SP-student encounter was videotaped and was viewed independently by three physicians unfamiliar with the contents of the checklists and databases. The physicians' pooled ratings were then compared with the checklist and database scores. Uncorrected and corrected correlations were obtained, with the generalizability coefficient used as the index of reliability. RESULTS: Interrater generalizability of physicians' ratings was very good, ranging from .65 to .93 for overall ratings. Generalizability of physicians' ratings pooled across the four cases was .85. Checklist scores tended to correlate higher with physicians' ratings than did database scores: across the cases, correlation coefficients between physicians' ratings and checklist scores and database scores were .65 and .39, respectively. CONCLUSION: The checklist scores correlated strongly with the physicians' ratings of history and physical-examination skills, providing some evidence of validity for their use. The checklist scores correlated much better with the physicians' ratings than did the database scores. Possible explanations for this finding are discussed.

Effects of examinee gender, standardized-patient gender, and their interaction on standardized patients' ratings of examinees' interpersonal and communication skills.

PURPOSE: To assess the effects of examinee gender, standardized-patient (SP) gender, and, in particular, their interaction on ratings made by SPs of examinees' interpersonal and communication skills in a performance-based examination of clinical competence. METHOD: The examination was administered to four classes of senior medical students (about 70 per class) at Southern Illinois University School of Medicine, 1988-1991. The skill dimensions tested were clarity of communication, thoroughness of explanation, professional manner, personal manner, and overall patient satisfaction. Split-plot analyses of variance were used. RESULTS: There was no interaction of examinee gender and SP gender for any of the five rating scales. There was no main effect of examinee gender for four of the five scales; however, for personal manner, women students were rated slightly higher than men students. There was a main effect of SP gender, but the effect was not consistent from rating scale to rating scale or from class to class. Nevertheless, differences in ratings given by men and women SPs should not be of psychometric concern, since the ratings of men and women examinees are necessarily affected alike. CONCLUSIONS: Except for the women examinee's higher performance in personal manner, the men and women examinees generally performed equally well with respect to interpersonal and communication skills, and they performed equally well regardless of the gender of the SP.

Effects of using two or more standardized patients to simulate the same case on case means and case failure rates.

The effects of using two or more standardized patients (multiple SPs) to simulate the same case in a performance-based examination were studied at the case level by comparing case means and case failure rates for multiple SPs simulating the same case, using data from the classes of 1988, 1989, and 1990 at the Southern Illinois University School of Medicine. For total scores and scores on the students' written answers, the effects on means and failure rates were negligible and could be explained as due to sampling error. For scores on the checklists completed by the SPs, there were more significant differences than would be expected by chance alone, even though the number of significant differences was relatively small. The results demonstrate a need for caution in the interpretation of scores obtained from a case checklist completed by multiple SPs, particularly in regard to making pass-fail decisions.

Stability of peripheral hematological parameters after chronic acetylcholinesterase inhibition in man.

Acetylcholinesterase (AChE; EC 3.1.1.7) is present in both primitive and mature erythroid cells, and a role has been suggested for the enzyme in regulation of differentiation in the human erythron. AChE is also a major enzyme in the central nervous system; alteration of its activity has been proposed as a therapeutic strategy in Alzheimer disease. We recently treated 18 Alzheimer disease patients with metrifonate, a long-acting AChE inhibitor, over periods up to 7 months, with resulting erythrocyte AChE inhibition as high as 82 per cent of baseline values. Despite chronic reduction of enzyme activity, no significant alterations were noted in erythrocyte, leukocyte or platelet characteristics or numbers that would suggest a deleterious effect of AChE inhibition on normal differentiation. Thus, any modification of developmental pathways appears to be compensated by other regulatory mechanisms in the intact organism.