The importance of measuring competency-based outcomes: standard evaluation measures are not surrogates for clinical performance of internal medicine residents.

BACKGROUND: Despite recent emphasis on educational outcomes, program directors still rely on standard evaluation techniques such as tests of knowledge and subjective ratings. PURPOSES: To assess the correlation of standard internal medicine (IM) residency evaluation scores (attending global evaluations, In-Training examination, and Mini-Clinical Examination Exercise) with documented performance of preventive measures for continuity clinic patients. METHODS: Cross-sectional study of 132 IM residents attending an IM teaching clinic, July 2000 to June 2003, comparing standard evaluations with chart audit. RESULTS: Mean resident performance ranged from 53% (SD = 24) through 89% (SD = 20) across the 6 preventive measures abstracted from 1,102 patient charts. We found weak and mostly not significant correlations between standard measures and performance of preventive services. CONCLUSIONS: Standard measures are not adequate surrogates for measuring clinical outcomes. This supports the Accreditation Council for Graduate Medical Education's recommendations to incorporate novel Toolbox measures, like chart audit, into residency evaluations.

Implementing achievable benchmarks in preventive health: a controlled trial in residency education.

PURPOSE: To evaluate the Preventive Health Achievable Benchmarks Curriculum, a multifaceted improvement intervention that included an objective, practice-based performance evaluation of internal medicine and pediatric residents' delivery of preventive services. METHOD: The authors conducted a nonrandomized experiment of intervention versus control group residents with baseline and follow-up of performance audited for 2001-2004. All 130 internal medicine and 78 pediatric residents at two continuity clinics at the University of Alabama School of Medicine, Birmingham, participated. Performance of preventive care was assessed by structured chart review. The multifaceted feedback curriculum included individualized performance feedback, academic detailing by faculty, and collective didactic sessions. The main outcome was difference in receipt of preventive care for patients seen by intervention and control residents, comparing baseline and follow-up. RESULTS: Charts were reviewed for 3,958 patients. Receipt of preventive care increased for patients of intervention residents, but not for patients of control residents. For the intervention group, significant increases occurred for five of six indicators in internal medicine: smoking screening, quit smoking advice, colon cancer screening, pneumonia vaccine, and lipid screening; and four of six in pediatrics: parental quit smoking advice, car seats, car restraints, and eye alignment (p < .05 for all). For control residents, no consistent improvements were seen. There was greater improvement for intervention than for control residents for four of six indicators in internal medicine, and two of six in pediatrics. CONCLUSIONS: Using a multifaceted feedback curriculum, the authors taught residents about the care they provide and improved documented patient care.

Measuring resident physicians' performance of preventive care. Comparing chart review with patient survey.

BACKGROUND: The Accreditation Council for Graduate Medical Education has suggested various methods for evaluation of practice-based learning and improvement competency, but data on implementation of these methods are limited. OBJECTIVE: To compare medical record review and patient surveys on evaluating physician performance in preventive services in an outpatient resident clinic. DESIGN: Within an ongoing quality improvement project, we collected baseline performance data on preventive services provided for patients at the University of Alabama at Birmingham (UAB) Internal Medicine Residents' ambulatory clinic. PARTICIPANTS: Seventy internal medicine and medicine-pediatrics residents from the UAB Internal Medicine Residency program. MEASUREMENTS: Resident- and clinic-level comparisons of aggregated patient survey and chart documentation rates of (1) screening for smoking status, (2) advising smokers to quit, (3) cholesterol screening, (4) mammography screening, and (5) pneumonia vaccination. RESULTS: Six hundred and fifty-nine patient surveys and 761 charts were abstracted. At the clinic level, rates for screening of smoking status, recommending mammogram, and for cholesterol screening were similar (difference <5%) between the 2 methods. Higher rates for pneumonia vaccination (76% vs 67%) and advice to quit smoking (66% vs 52%) were seen on medical record review versus patient surveys. However, within-resident (N=70) comparison of 2 methods of estimating screening rates contained significant variability. The cost of medical record review was substantially higher ($107 vs $17/physician). CONCLUSIONS: Medical record review and patient surveys provided similar rates for selected preventive health measures at the clinic level, with the exception of pneumonia vaccination and advising to quit smoking. A large variation among individual resident providers was noted.

Improving the quality of clinical care for patients with osteoporosis.

Osteoporosis, leading to hip fractures and other fragility fractures, is prevalent in countries with a high life expectancy. Fractures have a high associated societal cost and disease burden. As a result, many countries have established screening guidelines for osteoporosis, especially targeting those at a higher risk. However, primary- and secondary-fracture prevention is currently suboptimal despite existence of effective medications. In addition, adherence to therapy is low. This special report outlines some of these challenges and evaluates different techniques for improving the quality of clinical care for patients with osteoporosis.

Differences in preventive health quality by residency year. Is seniority better?

BACKGROUND: It is assumed that the performance of more senior residents is superior to that of interns, but this has not been assessed objectively. OBJECTIVE: To determine whether adherence to national guidelines for outpatient preventive health services differs by year of residency training. DESIGN: Cross-sectional study. PARTICIPANTS: One hundred twenty Internal Medicine residents, postgraduate year (PGY)- 1 and PGY -2, attending a University Internal Medicine teaching clinic between June 2000 and May 2003. MEASUREMENTS: We studied 6 preventive health care services offered or received by patients by abstracting data from 1,017 patient records. We examined the differences in performance between PGY-1 and PGY-2 residents. RESULTS: Postgraduate year-2 residents did not statistically outperform PGY-1 residents on any measure. The overall proportion of patients receiving appropriate preventive health services for pneumococcal vaccination, advising tobacco cessation, breast and colon cancer screening, and lipid screening was similar across levels of training. PGY-1s outperformed PGY-2s for tobacco use screening (58%, 51%, P = .03). These results were consistent after accounting for clustering of patients within provider and adjusting for patient age, gender, race and insurance, resident gender, and number of visits during the measurement year. CONCLUSIONS: Overall, patients cared for by PGY-2 residents did not receive more outpatient preventive health services than those cared for by PGY-1 residents. Efforts should be made to ensure quality patient care in the outpatient setting for all levels of training.