Assessment of English proficiency in international medical graduates by physician examiners and standardized patients.

CONTEXT: Since 1986, the Ontario Ministry of Health has provided a medical licensure preparation programme for international medical graduates. Because of the diversity in candidates' oral English proficiency, this competency has been viewed as a particularly important selection criterion. OBJECTIVES: To assess and compare the quality of ratings of oral English proficiency of international medical graduates provided by physician examiners and by standardized patients (SPs). PARTICIPANTS AND MATERIALS: The study samples consisted of 73 candidates for the Ontario International Medical Graduate (IMG) Program, and physician examiners and SPs in five 10-minute encounter objective structured clinical examination (OSCE) stations. Materials used were a seven-item speaking performance rating instrument prepared for the Ontario IMG Program. METHODS: Rating sheets were scanned and the results analysed using SPSS 9.0 for Windows. RESULTS: Correlations between the physician and SP ratings on the seven items ranged from 0.52 to 0.70. The SPs provided more lenient ratings. Mean alpha reliability for the physicians' ratings on the seven items was 0.59, and for the SPs' 0.64. There was poor agreement between the two sets of raters in identifying problematic candidates. CONCLUSIONS: Notwithstanding the sizable correlations between the ratings provided by the two rater groups, the results demonstrated that there was little agreement between the two groups in identifying the potentially problematic candidates. The physicians were less prone than the SPs to rate candidates as problematic. SPs may be better placed than the physician examiners to directly assess IMG candidates' oral English proficiency.

A comparison of physician examiners', standardized patients', and communication experts' ratings of international medical graduates' English proficiency.

PURPOSE: To assess the quality of ratings of interviewing skills and oral English proficiency provided on a clinical skills OSCE by physician examiners, standardized patients (SPs), and communication skills experts. METHOD: In 1998, 73 candidates to the Ontario International Medical Graduate (OIMG) Program completed a 29-station OSCE-type clinical skills selection examination. Physician examiners, SPs, and communication skills experts assessed components of oral English proficiency and interview performance. Based on these results, the frequency and generalizability of English-language flags, physician examiners' indications that spoken English skills were bad enough to significantly impede communication with patients; the reliability of the OIMG's Interview and Oral Performance Scales and generalizability of overall interview and oral performance ratings; and comparisons of repeated assessments by experts were calculated. Principal-components analysis was applied to the panels' ratings to determine a more economical expression of the language proficiency and interview communication skills results. RESULTS: The mean number of English-language flags per candidate was 2.1, the median was 1.0, and Cronbach's alpha of the ratings was 0.63. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the interview performance scale were 9.15/10, 0.43, 0.36, and 9.30/10, 0. 56, 0.50, respectively. Corresponding values for overall interview performance ratings were 3.08/4, 0.30, 0.33, and 3.34/4, 0.32, 0.47. Means, SDs, and alphas of the physician examiners' and SPs' ratings of the oral performance scale were 8.54/10, 0.74, 0.78, and 8.74/10, 1.00, 0.76. Corresponding values for overall ratings of oral performance were 3.85/5, 0.51, 0.68, and 4.08/5, 0.60, 0.68. For the two experts' ratings of two contiguous five-minute interview stations, internal consistencies were 0.88 and 0.78. For the two experts' ratings of standardized ten-minute interviews, internal consistencies were 0.81 and 0.92. Correlations between the mean values of the experts' ratings of the ten- and five-minute stations were 0.45 and 0.51. Three factors emerged from the PCA, language proficiency, physician examiners' ratings of interview proficiency, and SPs' ratings of interview proficiency. CONCLUSIONS: Consistency between the physician examiners' and SPs' ratings of English proficiency was observed; less agreement was observed in their ratings of interviewing skills, and little agreement was observed between the experts' ratings. Communication skills results may be validly expressed by three measures: one overall global rating of language proficiency provided by physician examiners or SPs, and overall global ratings of interview proficiency provided separately by physician examiners and SPs.

Differences unrelated to clinical competence in the results of repeated multiple-station tests of clinical skills.

PURPOSE: To determine the extent to which systematic differences not related to group differences in clinical competence could be observed in the results of six administrations of the same multiple-station test of clinical skills. METHOD: The same 16 stations were administered as part of the Ontario International Medical Graduates Program clinical skills screening examination on three different occasions, and on each occasion in two sessions. The interval between the first (fall 1986) and second (March 1990) administrations was four years, and that between the second and third (July 1990) administrations, four months. International medical graduates were the candidates in the first two administrations; the third administration was to a combined sample of fourth-year clinical clerks and interns. A different item functioning (DIF) approach with sessions within administration as the grouping variable was used to describe the extent of differences unrelated to clinical competence in the results of the different test administrations. For the generalizability and DIF calculations the designs were balanced by sampling down to 33 cases in each session. RESULTS: Station DIF values varied considerably from station to station. DIF effects between sessions with administrations were less than those between administrations, and were less between the two administrations separated by four months than between the two administrations separated by four years. CONCLUSION: Hypotheses concerning the relative magnitudes of station DIF and total-test aggregated values of DIF, as a function of the time intervals between test occasions, were substantiated, demonstrating that the greater the time interval between test administrations, the greater the magnitude of DIF.

The consistency and uncertainty in examiners' definitions of pass/fail performance on OSCE (objective structured clinical examination) stations.

The Medical Council of Canada has made use of examiners' pass/fail classifications of candidates' behaviors in objective structured clinical examination (OSCE) stations in defining cutting scores for these stations. This process assumes that there is consistency in the judgments of examiners employed in the same stations at different testing sites and in the cutting scores derived from these judgments. These assumptions were tested using the results of the fall 1993 administration of part 2 of the Medical Council of Canada's Evaluating Examination to 744 candidates. The results of this study provided evidence of the consistency of the pass/fail and cutting score definitions for the stations used across examiners.

The use of global ratings in OSCE station scores.

The Medical Council of Canada makes use of examiners' pass/borderline/fail judgments of candidates' performances in OSCE stations in defining cutting scores for these stations. This process assumes that there is consistency in the judgments of different examiners used in the same stations at different testing sites. This assumption was tested using the results of the fall 1994 administration of part 2 of the Medical Council of Canada Qualifying Examination. The Council anticipated using the examiner based global ratings as part of the OSCE station scores in the fall 1995 administration of the examination. In this study, the fall 1994 results were used to estimate to what extent test reliability would increase with the addition of the global ratings.

Understanding the objective structured clinical examination.

A number of factors have contributed to the relatively rapid development and growth of the Objective Structured Clinical Examination (OSCA) as a viable and popular assessment tool. Among these was an increased need, expressed both by training institutions and certifying and licensing bodies, for valid performance-based assessment approaches. That this need became apparent at a time when concern and interest in performance-based assessment in education generally was growing, was fortuitous and contributed to the developments in medical education. In medical education important consequences were the growing interest and expertise in two distinct areas, the psychometrics of performance assessment and the use of standardized (simulated) patients in the teaching and assessment of clinical skills. Collaboration between people working in these two areas greatly facilitated the rapid development and refinement of the OSCA. Currently, there appears to be general acceptance of the legitimacy of this assessment approach. Research now seems to be directed at resolving remaining important technical problems (e.g. standard setting); the design and implementation of large scale national and regional certifying and licensing examinations are the primary current development challenges. There is a large and still growing body of related literature and many presentations about this assessment approach are given at medical education conferences.

Station gender bias in a multiple-station test of clinical skills.

PURPOSE: To apply differential item functioning (DIF) procedures to investigate station gender bias in multiple-station tests of clinical skills, and to compare these results with those obtained by comparing the station-score distributions of men and women examinees. METHOD: The data were from 23 stations used in the selection of seven successive cohorts (1987-1993) of candidates to the Ontario Pre-Internship Program for graduates of foreign medical schools. The stations had been used on at least three occasions, with a minimum sample of about 210 candidates per station. Each station's score was expressed as both a binary score and a continuous score, and DIF was assessed using the Mantel-Haenszel procedure with the binary scores and analysis of covariance with the continuous scores. For each station, DIF effect sizes were calculated and compared with the gender-group mean differences. RESULTS: Using the binary scores, significant DIF was observed for three stations; using the continuous scores, significant DIF was observed for five stations. Significant gender differences were observed in the scores of nine stations. In eight, these differences favored women. Overall, in more stations the direction of DIF favored the men, while the women demonstrated higher levels of ability. CONCLUSION: The results suggest the importance of using a DIF approach for controlling the "ability factor" in studies of this kind: although significant gender differences were observed in the continuous score distributions of nine stations, generally these differences were not indicative of station gender bias.

Validity and generalizability of global ratings in an objective structured clinical examination.

The performance of foreign medical graduates on multistation standardized patient-based tests and used to determine the validity and generalizability of global ratings of their clinical competence made by expert examiners. Data were derived from the entrance examinations of the 1989 and 1990 applicants to the Ontario Pre-Internship Program and the exit examination of 24 participants from the 1989 cohort. For each candidate, the examiners completed a detailed checklist and two five-point global ratings dealing with the candidate's approach to the patients' problem and attitude toward the patient. Generalizability coefficients for both ratings were satisfactory and stable across cohorts. Construct validity of the global ratings was demonstrated by comparing entry and exit ratings and by evidence of significant and positive correlations between the global ratings and total test scores. Tentative evidence of criterion validity of the global ratings was demonstrated. These findings suggest that global ratings by expert examiners can be used as an effective form of assessment in multistation standardized patient examinations.

Validity and reliability of a domain-referenced test of clinical competence for foreign medical graduates.

The authors determined the reliability and validity of domain-referenced scores obtained from a multiple-station clinical skills examination administered at the University of Toronto to a group of foreign medical graduates in early 1990. Issues associated with the identification of essential checklist items and the setting of minimum passing scores (cutting scores) for each station and for the total test were addressed. In addition, they examined the consistency of classification of candidates and the perceived appropriateness of the number of candidates classified "competent." The findings provide evidence of the validity of standard-setting procedures based on both the performance of criterion groups and the judgment of "experts."

Evaluating the clinical skills of foreign medical school graduates participating in an internship preparation program.

In the fall of 1986, the Ministry of Health of the province of Ontario, Canada, implemented a special 36-week internship preparation program for graduates of foreign medical schools. The program accepts 24 candidates per year and is offered at the five Ontario medical schools. At the time of this study, two cohorts of participants had completed the program. As part of the evaluation process, each participant was administered a battery of the same Objective Structured Clinical Examination-type test stations, prior to and on completion of the pre-internship experience. Repeating the same examination permitted investigation of the predictive and construct validities of the clinical skills tests, as well as the stability of the validity and high reliability estimates that emerged from the testing of the first cohort. The results provide convincing evidence of the testing approach's predictive and construct validity and reliability.