Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards.

BACKGROUND: Physicians who are disciplined by state licensing boards are more likely to have demonstrated unprofessional behavior in medical school. Information is limited on whether similar performance measures taken during residency can predict performance as practicing physicians. OBJECTIVE: To determine whether performance measures during residency predict the likelihood of future disciplinary actions against practicing internists. DESIGN: Retrospective cohort study. SETTING: State licensing board disciplinary actions against physicians from 1990 to 2006. PARTICIPANTS: 66,171 physicians who entered internal medicine residency training in the United States from 1990 to 2000 and became diplomates. MEASUREMENTS: Predictor variables included components of the Residents' Annual Evaluation Summary ratings and American Board of Internal Medicine (ABIM) certification examination scores. RESULTS: 2 performance measures independently predicted disciplinary action. A low professionalism rating on the Residents' Annual Evaluation Summary predicted increased risk for disciplinary action (hazard ratio, 1.7 [95% CI, 1.3 to 2.2]), and high performance on the ABIM certification examination predicted decreased risk for disciplinary action (hazard ratio, 0.7 [CI, 0.60 to 0.70] for American or Canadian medical school graduates and 0.9 [CI, 0.80 to 1.0] for international medical school graduates). Progressively better professionalism ratings and ABIM certification examination scores were associated with less risk for subsequent disciplinary actions; the risk ranged from 4.0% for the lowest professionalism rating to 0.5% for the highest and from 2.5% for the lowest examination scores to 0.0% for the highest. LIMITATIONS: The study was retrospective. Some diplomates may have practiced outside of the United States. Nondiplomates were excluded. CONCLUSION: Poor performance on behavioral and cognitive measures during residency are associated with greater risk for state licensing board actions against practicing physicians at every point on a performance continuum. These findings support the Accreditation Council for Graduate Medical Education standards for professionalism and cognitive performance and the development of best practices to remediate these deficiencies.

Measuring professionalism: a review of studies with instruments reported in the literature between 1982 and 2002.

PURPOSE: To describe the measurement properties of instruments reported in the literature that faculty might use to measure professionalism in medical students and residents. METHOD: The authors reviewed studies published between 1982 and 2002 that had been located using Medline and four other databases. A national panel of 12 experts in measurement and research in medical education extracted data from research reports using a structured critique form. RESULTS: A total of 134 empirical studies related to the concept of professionalism were identified. The content of 114 involved specific elements of professionalism, such as ethics, humanism, and multiculturalism, or associated phenomena in the educational environment such as abuse and cheating. Few studies addressed professionalism as a comprehensive construct (11 studies) or as a distinct facet of clinical competence (nine studies). The purpose of 109 studies was research or program evaluation, rather than summative or formative assessment. Sixty five used self-administered instruments with no independent observation of the participants' professional behavior. Evidence of reliability was reported in 62 studies. Although content validity was reported in 86 studies, only 34 provided strong evidence. Evidence of concurrent or predictive validity was provided in 43 and 16 studies, respectively. CONCLUSIONS: There are few well-documented studies of instruments that can be used to measure professionalism in formative or summative evaluation. When evaluating the tools described in published research it is essential for faculty to look critically for evidence related to the three fundamental measurement properties of content validity, reliability, and practicality.

The mini-CEX: a method for assessing clinical skills.

OBJECTIVE: To evaluate the mini-clinical evaluation exercise (mini-CEX), which assesses the clinical skills of residents. DESIGN: Observational study and psychometric assessment of the mini-CEX. SETTING: 21 internal medicine training programs. PARTICIPANTS: Data from 1228 mini-CEX encounters involving 421 residents and 316 evaluators. INTERVENTION: The encounters were assessed for the type of visit, sex and complexity of the patient, when the encounter occurred, length of the encounter, ratings provided, and the satisfaction of the examiners. Using this information, we determined the overall average ratings for residents in all categories, the reliability of the mini-CEX scores, and the effects of the characteristics of the patients and encounters. MEASUREMENTS: Interviewing skills, physical examination, professionalism, clinical judgment, counseling, organization and efficiency, and overall competence were evaluated. RESULTS: Residents were assessed in various clinical settings with a diverse set of patient problems. Residents received the lowest ratings in the physical examination and the highest ratings in professionalism. Comparisons over the first year of training showed statistically significant improvement in all aspects of competence, and the method generated reliable ratings. CONCLUSIONS: The measurement characteristics of the mini-CEX are similar to those of other performance assessments, such as standardized patients. Unlike these assessments, the difficulty of the examination will vary with the patients that a resident encounters. This effect is mitigated to a degree by the examiners, who slightly overcompensate for patient difficulty, and by the fact that each resident interacts with several patients. Furthermore, the mini-CEX has higher fidelity than these formats, permits evaluation based on a much broader set of clinical settings and patient problems, and is administered on site.