The ethical problem of false positives: a prospective evaluation of physician reporting in the medical record.

OBJECTIVE: To determine if the medical record might overestimate the quality of care through false, and potentially unethical, documentation by physicians. DESIGN: Prospective trial comparing two methods for measuring the quality of care for four common outpatient conditions: (1) structured reports by standardised patients (SPs) who presented unannounced to the physicians' clinics, and (2) abstraction of the medical records generated during these visits. SETTING: The general medicine clinics of two veterans affairs medical centres. PARTICIPANTS: Twenty randomly selected physicians (10 at each site) from among eligible second and third year internal medicine residents and attending physicians. MAIN MEASUREMENTS: Explicit criteria were used to score the medical records of physicians and the reports of SPs generated during 160 visits (8 cases x 20 physicians). Individual scoring items were categorised into four domains of clinical performance: history, physical examination, treatment, and diagnosis. To determine the false positive rate, physician entries were classified as false positive (documented in the record but not reported by the SP), false negative, true positive, and true negative. RESULTS: False positives were identified in the medical record for 6.4% of measured items. The false positive rate was higher for physical examination (0.330) and diagnosis (0.304) than for history (0.166) and treatment (0.082). For individual physician subjects, the false positive rate ranged from 0.098 to 0.397. CONCLUSIONS: These data indicate that the medical record falsely overestimates the quality of important dimensions of care such as the physical examination. Though it is doubtful that most subjects in our study participated in regular, intentional falsification, we cannot exclude the possibility that false positives were in some instances intentional, and therefore fraudulent, misrepresentations. Further research is needed to explore the questions raised but incompletely answered by this research.

Effect of pravastatin-to-simvastatin conversion on low-density-lipoprotein cholesterol.

The effects of a pravastatin-to-simvastatin conversion program on low-density-lipoprotein (LDL) cholesterol levels were studied. Patients receiving pravastatin at a Veterans Affairs medical center were switched to simvastatin beginning in 1997. The dosage of simvastatin was based on the additional percent reduction in LDL cholesterol needed to achieve the goal specified by the National Cholesterol Education Program. The primary endpoint was the change in the percentage of patients meeting their LDL cholesterol goal at baseline and follow-up. Changes in lipid indices, the relative risk (RR) of coronary heart disease (CHD), and program costs were also evaluated. A total of 1032 patients completed the program. The mean +/- S.D. daily doses of pravastatin and simvastatin were 25.2 +/- 11.3 and 22.7 +/- 13.3 mg, respectively. Median baseline and follow-up LDL cholesterol concentrations were 116 and 99 mg/dL, respectively (p < 0.001). Overall, 44% of the patients met their LDL cholesterol goal while taking pravastatin, compared with 69% after conversion to simvastatin (p < 0.001). The predicted mean RR of a future CHD event (based on changes in serum lipids) was 0.87 (95% confidence interval, 0.83-0.91) four years after conversion. The total cost of the program was $40,644 in the first year, and there was a net saving thereafter. Therapeutic interchange between pravastatin and simvastatin increased the number of patients meeting their LDL cholesterol goal.

Measuring compliance with preventive care guidelines: standardized patients, clinical vignettes, and the medical record.

OBJECTIVE: To determine how accurately preventive care reported in the medical record reflects actual physician practice or competence. DESIGN: Scoring criteria based on national guidelines were developed for 7 separate items of preventive care. The preventive care provided by randomly selected physicians was measured prospectively for each of the 7 items. Three measurement methods were used for comparison: (1) the abstracted medical record from a standardized patient (SP) visit; (2) explicit reports of physician practice during those visits from the SPs, who were actors trained to present undetected as patients; and (3) physician responses to written case scenarios (vignettes) identical to the SP presentations. SETTING: The general medicine primary care clinics of two university-afflliated VA medical centers. PARTICIPANTS: Twenty randomly selected physicians (10 at each site) from among eligible second- and third-year general internal medicine residents and attending physicians. MEASUREMENTS AND MAIN RESULTS: Physicians saw 160 SPs (8 cases x 20 physicians). We calculated the percentage of visits in which each prevention item was recorded in the chart, determined the marginal percentage improvement of SP checklists and vignettes over chart abstraction alone, and compared the three methods using an analysis-of-variance model. We found that chart abstraction underestimated overall prevention compliance by 16% (P < .01) compared with SP checklists. Chart abstraction scores were lower than SP checklists for all seven items and lower than vignettes for four items. The marginal percentage improvement of SP checklists and vignettes to performance as measured by chart abstraction was significant for all seven prevention items and raised the overall prevention scores from 46% to 72% (P < .0001). CONCLUSIONS: These data indicate that physicians perform more preventive care than they report in the medical record. Thus, benchmarks of preventive care by individual physicians and institutions that rely solely on the medical record may be misleading, at best.

How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record.

PURPOSE: Despite widespread reliance on chart abstraction for quality measurement, concerns persist about its reliability and validity. We prospectively evaluated the validity of chart abstraction by directly comparing it with the gold standard of reports by standardized patients. SUBJECTS AND METHODS: Twenty randomly selected general internal medicine residents and attending faculty physicians at the primary care clinics of two Veterans Affairs Medical Centers blindly evaluated and treated actor-patients (standardized patients) who had one of four common diseases: diabetes, chronic obstructive pulmonary disease, coronary artery disease, or low back pain. Charts from the visits were abstracted using explicit quality criteria; standardized patients completed a checklist containing the same criteria. For each physician, quality was measured for two different cases of the four conditions (a total of 160 physician-patient encounters). We compared chart abstraction with standardized-patient reports for four aspects of the encounter: taking the history, examining the patient, making the diagnosis, and prescribing appropriate treatment. The sensitivity and specificity of chart abstraction were calculated. RESULTS: The mean (+/- SD) chart abstraction score was 54% +/- 9%, substantially less than the mean score on the standardized-patient checklist of 68% +/- 9% (P <0.001). This finding was similar for all four conditions and at both sites. "False positives"-chart-recorded necessary care actions not reported by the standardized patients-resulted in a specificity of only 81%. The overall sensitivity of chart abstraction for necessary care was only 70%. CONCLUSIONS: Chart abstraction underestimates the quality of care for common outpatient general medical conditions when compared with standardized-patient reports. The medical record is neither sensitive nor specific. Quality measurements derived from chart abstraction may have important shortcomings, particularly as the basis for drawing policy conclusions or making management decisions.

Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality.

CONTEXT: Better health care quality is a universal goal, yet measuring quality has proven to be difficult and problematic. A central problem has been isolating physician practices from other effects of the health care system. OBJECTIVE: To validate clinical vignettes as a method for measuring the competence of physicians and the quality of their actual practice. DESIGN: Prospective trial conducted in 1997 comparing 3 methods for measuring the quality of care for 4 common outpatient conditions: (1) structured reports by standardized patients (SPs), trained actors who presented unannounced to physicians' clinics (the gold standard); (2) abstraction of medical records for those same visits; and (3) physicians' responses to clinical vignettes that exactly corresponded to the SPs' presentations. Setting Outpatient primary care clinics at 2 Veterans Affairs medical centers. PARTICIPANTS: Ninety-eight (97%) of 101 general internal medicine staff physicians, faculty, and second- and third-year residents consented to be randomized for the study. From this group, 10 physicians at each site were randomly selected for inclusion. MAIN OUTCOME MEASURES: A total of 160 quality scores (8 cases x 20 physicians) were generated for each method using identical explicit criteria based on national guidelines and local expert panels. Scores were defined as the percentage of process criteria correctly met and were compared among the 3 methods. RESULTS: The quality of care, as measured by all 3 methods, ranged from 76.2% (SPs) to 71.0% (vignettes) to 65.6% (chart abstraction). Measuring quality using vignettes consistently produced scores closer to the gold standard of SP scores than using chart abstraction. This pattern was robust when the scores were disaggregated by the 4 conditions (P<.001 to <.05), by case complexity (P<.001), by site (P<.001), and by level of physician training (P values from <.001 to <.05). The pattern persisted, although less dominantly, when we assessed the component domains of the clinical encounter--history, physical examination, diagnosis, and treatment. Vignettes were responsive to expected directions of variation in quality between sites and levels of training. The vignette responses did not appear to be sensitive to physicians' having seen an SP presenting with the same case. CONCLUSIONS: Our data indicate that quality of health care can be measured in an outpatient setting by using clinical vignettes. Vignettes appear to be a valid and comprehensive method that directly focuses on the process of care provided in actual clinical practice. Vignettes show promise as an inexpensive case-mix adjusted method for measuring the quality of care provided by a group of physicians.

Analyzing the time and value of housestaff inpatient work.

OBJECTIVE: To determine time allocation and the perceived value to education and patient care of the weekday activities of internal medicine housestaff on inpatient rotations and to compare the work activities of interns and residents. DESIGN: An observational study. We classified activities along five dimensions (association, location, activity, time, and value), developed a computer-assisted self-interview survey, and demonstrated its face and content validity, internal consistency, and interrater reliability. Subjects were assigned survey computers for 5 consecutive weekdays over a 24-week period, into which they entered data when prompted several times a day. SETTING: The medical service of a university-affiliated Veterans Administration Medical Center. PARTICIPANTS: Sixty housestaff (36 interns, 24 residents) rotating on the inpatient wards. MEASUREMENTS AND MAIN RESULTS: We analyzed activities according to content (direct patient care, indirect patient care, education), association, and location. Likert-scale ratings of perceived value to education and patient care were also obtained. Housestaff provided complete responses to 3,812 (95%) of 3,992 prompts by a median of 11 seconds; 93% of responses were logically consistent across the measured dimensions. Housestaff spent more time in indirect patient care (56%) than in direct patient care (14%) or educational activities (45%). Formal educational activities had the highest educational value (66 on 0-100 scale), and direct care had the highest value to patient care (81). Over 30% of time was spent in administrative activities, which had low educational value(40). Compared with residents, interns allocated significantly less time to educational activities (38% vs 57%) and more time to lower-value activities such as documentation (19% vs 12%). CONCLUSIONS: Improved data collection methods demonstrate that housestaff in our program, particularly interns, spend much of their workday in activities that are low in educational and patient care value. Selective elimination or delegation of such activities would preserve higher-value experiences during reductions in overall inpatient training time. Planners can use automated random sampling to guide the rational redesign of housestaff work.