MEASUREMENT: ACCOUNTING FOR RELIABILITY IN PERFORMANCE ESTIMATES.

When evaluating physician performance measures, physician leaders are faced with the quandary of determining whether departures from expected physician performance measurements represent a true signal or random error. This uncertainty impedes the physician leader's ability and confidence to take appropriate performance improvement actions based on physician performance measurements. Incorporating reliability adjustment into physician performance measurement is a valuable way of reducing the impact of random error in the measurements, such as those caused by small sample sizes. Consequently, the physician executive has more confidence that the results represent true performance and is positioned to make better physician performance improvement decisions. Applying reliability adjustment to physician-level performance data is relatively new. As others have noted previously, it's important to keep in mind that reliability adjustment adds significant complexity to the production, interpretation and utilization of results. Furthermore, the methods explored in this case study only scratch the surface of the range of available Bayesian methods that can be used for reliability adjustment; further study is needed to test and compare these methods in practice and to examine important extensions for handling specialty-specific concerns (e.g., average case volumes, which have been shown to be important in cardiac surgery outcomes). Moreover, it's important to note that the provider group average as a basis for shrinkage is one of several possible choices that could be employed in practice and deserves further exploration in future research. With these caveats, our results demonstrate that incorporating reliability adjustment into physician performance measurements is feasible and can notably reduce the incidence of "real" signals relative to what one would expect to see using more traditional approaches. A physician leader who is interested in catalyzing performance improvement through focused, effective physician performance improvement is well advised to consider the value of incorporating reliability adjustments into their performance measurement system.

Hospital performance with myocardial reperfusion therapy: are hospitals capable of meeting established guidelines?

OBJECTIVES: To determine whether hospitals are capable of delivering myocardial reperfusion therapy in a manner consistent with the American College of Cardiology/American Heart Association guidelines. DATA SOURCE AND STUDY SETTING: Data from the National Registry of Myocardial Infarction (NRMI)-2 and NRMI-3 were used. NRMI is an observational study, sponsored by Genentech, conducted from June 1994 through June 2000 and involving 1876 hospitals and 1,310,030 patients across the United States. The protocol calls for collecting data on all patients with a diagnosis of acute myocardial infarction. The setting was community and tertiary hospitals in the United States. STUDY DESIGN: This observational study used process capability analysis. PRINCIPAL FINDINGS: Overall, no hospital was deemed capable of delivering myocardial reperfusion therapy consistent with the American College of Cardiology/American Heart Association guidelines. The highest thrombolytic and angioplasty CPUs were 0.44 and 0.52, respectively-well below the traditional value of 1.0 signifying minimum capability. In addition, among the hospitals examined, there remained a wide degree of variability in process capability, ranging from -0.69 to 0.52. CONCLUSIONS: Myocardial reperfusion therapy performance measurement systems relying solely on mean time-to-reperfusion conceal true process performance, thereby obscuring quality improvement opportunities and strategies for improvement. Health care providers, purchasers, regulators, and other organizations interested in measuring and improving health care quality are encouraged to incorporate process capability analysis into their myocardial reperfusion therapy performance measurement and quality management systems.

Monitoring pharmacy expert system performance using statistical process control methodology.

Automated expert systems provide a reliable and effective way to improve patient safety in a hospital environment. Their ability to analyze large amounts of data without fatigue is a decided advantage over clinicians who perform the same tasks. As dependence on expert systems increase and the systems become more complex, it is important to closely monitor their performance. Failure to generate alerts can jeopardize the health and safety of patients, while generating excessive false positive alerts can lead to valid alerts being dismissed as noise. In this study, statistical process control charts were used to monitor an expert system, and the strengths and weaknesses of this technology are presented.