Does composition medium affect the psychometric properties of scores on an exercise designed to assess written medical communication skills?

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools are ready to enter graduate training programs in the United States. The performance-based patient note exercise is specifically used to assess an examinee's ability to summarize, synthesize and interpret the data collected in a patient interview. The purpose of this study was to investigate whether choice of composition method (typing or writing) affected the psychometric properties of the scores. Using data for a 1-year period, the validity and reliability of typed and written notes was contrasted. Although the characteristics of individuals who chose whether or not to type were different, the statistical analyses indicated that, controlling for examinee ability, physician examiner ratings of the written summaries were not influenced by composition method. The results of this study suggest that the psychometric properties of the patient note scores are invariant with respect to composition method.

The international medical graduate pipeline: recent trends in certification and residency training.

International medical graduates (IMGs) represent a large proportion of the population entering graduate medical education (GME) programs. Many of these internationally trained physicians go on to practice medicine in the United States. To be eligible for GME, IMGs must be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). The number of certificates issued by the ECFMG has varied over time and historically has exceeded the number of available training positions. More detailed longitudinal analyses are required to better understand the interwoven issues of physician supply, consumers' needs, and the role of IMGs in the U.S. health care system.

Commentary: Coming to America: the integration of international medical graduates into the American medical culture.

This Commentary is a companion piece to two Research Reports appearing in this issue: "Behavioral Science Education and the International Medical Graduate," by Searight and Gafford, and "International Medical Graduates and the Diagnosis and Treatment of Late-Life Depression," by Kales et al. International medical graduates (IMGs) come to America from diverse cultures around the world to complete their graduate medical education (GME). These residents are and will continue to be a fundamental part of the American health care delivery system. IMGs' acculturation into the norms and standards of medicine as practiced in the U.S. is crucial to their education as well as to quality patient care. The time has come for GME to begin to systematically and effectively address the cultural challenges that IMGs face not only within the context of American medicine and GME, but in the larger context of American culture. Specific programs and strategies need to be developed and put in place early in the GME experience-or even before entry into GME-to assist IMGs in understanding the context for, and issues associated with, providing optimum health care in the United States. The author reflects on the findings of the two Research Reports, and calls for increased attention in the medical education community to acculturating and educating IMGs for optimal patient care.

Specialty board certification among U.S. citizen and non-U.S. citizen graduates of international medical schools.

BACKGROUND: Graduates of international medical schools (IMGs) make up approximately one-quarter of the physician population and play a key role in the provision of health care in the United States. This study investigated whether they differ from U.S. medical graduates (USMGs) in specialty board certification. METHOD: The study compared USMGs, U.S. citizen IMGs (USIMGs), and non-U.S. citizen IMGs (non-USIMGs) who graduated from medical school between 1958 and 1994 and were involved in direct patient care in 2003. RESULTS: There is variability among the specialties, but overall USMGs have the highest specialty certification rates followed by non-USIMGs, and USIMGs. Among recent medical school graduates, non-USIMGs have certification rates that are comparable to USMGs. CONCLUSIONS: IMGs have lower board-certification rates than USMGs, although a sizeable majority has achieved board certification in the specialty they practice. There are differences between non-USIMGs and USIMGs, with the former more likely to become board certified.

Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA).

Throughout the 40 year history of standardized patient assessments and OSCEs, there have been numerous advancements, including many that involve scoring the simulated clinical encounters. While there is no clear agreement on how examinees' performance should be documented or scored in an encounter, there is a consensus that several well-chosen SP encounters are required to produce reliable examinee scores. There also continues to be some debate as to who should do the scoring on an SP-based assessment. While logistics and cost will certainly play a role, it is probably best to use the person who is most familiar with the domain being assessed. In some instances this will be the SP; in others, an outside observer or content expert. Finally, with the growing use of OSCEs for summative purposes (e.g. certification, licensure), special attention must be paid to fairness issues. Since the same test form cannot be used day after day, examinee scores must be 'equated', taking into account the psychometric properties of scores from individual cases and individual SPs. To date, the CSA has been one of the highest-volume, high-stakes, standardized patient assessments to be developed and successfully administered. In 2003 alone, over 11 500 IMGs were tested. The early conceptual framework for this assessment was synthesized from the research endeavours of several notable individuals, including, amongst many others, Harden et al. 1975, Swanson & Stillman, 1990, Newble & Swanson, 1988, Vu et al. 1992 and Colliver, 1995. The early prototype administrations of the CSA, including many operational research studies, were supported and guided by Dr Friedman Ben-David, Friedman et al. 1991, 1993, Stillman et al. 1992, and Sutnick et al. 1993, 1995.

Assessing the written communication skills of medical school graduates.

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN ratings can be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.

The effect of task exposure on repeat candidate scores in a high-stakes standardized patient assessment.

BACKGROUND: Failing candidates often make multiple attempts on licensure and certification examinations. For performance-based assessments, where available test material is often limited, overlap in examination content is frequently inevitable. PURPOSE: The purpose of this study was to investigate the performance of repeat candidates, both on new and exposed material, on a standardized patient clinical skills assessment. METHODS: Analysis of variance techniques were used to investigate the longitudinal performance of repeat candidates. Score differences for the second test session were analyzed as function of prior exposure to assessment materials. RESULTS: Although the performance of candidates improved between the first and second assessment, score increases could not be attributed to the random exposure of examination material. CONCLUSIONS: The exposure of case content on clinical skills examinations does not appear to provide any advantage, or disadvantage, to repeat test takers. This finding, although based on average scores for repeat examinees, suggests that valid assessment scores can be obtained even if there is some overlap in assessment content.

Quality assurance methods for performance-based assessments.

Performance assessments are subject to many potential error sources. For performance-based assessments, including standardized patient (SP) examinations, these error sources, if left unchecked, can compromise the validity and reliability of scores. Quality assurance (QA) measures, both quantitative and qualitative, can be used to ensure that candidate scores are accurate and reasonably free from measurement error. The purpose of this paper is to outline several QA strategies that can be used to identify potential content- and score-related problems with SP assessments. These approaches include case analyses and various comparisons of primary and observer scores. Specific examples from the ECFMG Clinical Skills Assessment (CSA) are used to educate the reader concerning appropriate statistical methods and legitimate data interpretations. The results presented in this investigation highlight the need for well-defined training regimes, regular feedback to those involved in rating/scoring performances, and detailed statistical analyses of all scores.

Evaluating the spoken English proficiency of international medical graduates: detecting threats to the validity of standardised patient ratings.

PURPOSE: To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations. METHOD: Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages). RESULTS: No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings. DISCUSSION: The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.

The changing pool of international medical graduates seeking certification training in US graduate medical education programs.

International medical graduates (IMGs) consistently represent approximately one fourth of both the physician workforce and the graduate medical education (GME) population of the United States. To enter into accredited US GME programs, IMGs must be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). Changes in the number and characteristics of those seeking certification directly affect the GME population and the future physician workforce in the United States. In July 1998, in response to concerns that IMGs might be lacking in basic clinical skills (eg, history taking, physical examinations, communicating with patients in spoken English), the ECFMG initiated a requirement that IMGs pass a clinical skills assessment (CSA) to achieve ECFMG certification. In this study we examined the pool of IMGs seeking certification, using databases reporting on all individuals beginning the certification process from 1995 through 2001. For this period, we found that the number of IMG candidates taking the Step 1 examination decreased by 45.5% (36,983 vs 16,828), and the number of IMGs registered to take Step 2 decreased by 38.1% (31 751 vs 12 122). The number of ECFMG certificates issued annually decreased, from a range of 9000 to 12,000 (1995-1998) to fewer than 6000 (1999-2001). Although the number of IMGs annually seeking and receiving certification has decreased, the quality of the applicants appears to have improved and the number of IMGs certified annually continues to adequately fill GME positions not taken by US medical graduates.

Attrition rates of residents in postgraduate training programs.

BACKGROUND: One goal of program directors is to attract and retain high-quality residents. It is therefore important to study attrition of residents to determine specific cohorts that may be less likely to complete residency training. PURPOSE: The purpose of this study was to investigate attrition rates in targeted programs. METHOD: Surveys were sent to 540 program directors whose programs contained at least 25% international medical graduates. The program directors reported both voluntary attrition and termination rates. RESULTS: Overall attrition rates for international medical graduates and U.S. medical graduates were similar, although U.S. medical graduates were terminated at a slightly lower rate than international medical graduates (2.5% vs. 3.6%) and were somewhat more likely to transfer to other programs (9.9% and 8.9%). CONCLUSIONS: Attrition from these programs remains a small but persistent issue. In addition, the differences in termination and transfer rates for the 2 years of the study were greater than the differences in rates between U.S. and international medical graduates.

Assessing the comparability of standardized patient and physician evaluations of clinical skills.

Accumulating evidence to defend decisions based on scores from evaluations is an ongoing process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment (CSA). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine's Mini-CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores.