Bayesian methods in reporting and managing Australian clinical indicators.

Sustained clinical improvement is unlikely without appropriate measuring and reporting techniques. Clinical indicators are tools to help assess whether a standard of care is being met. They are used to evaluate the potential to improve the care provided by healthcare organisations (HCOs). The analysis and reporting of these indicators for the Australian Council on Healthcare Standards have used a methodology which estimates, for each of the 338 clinical indicators, the gains in the system that would result from shifting the mean proportion to the 20(th) centile. The results are used to provide a relative measure to help prioritise quality improvement activity within clinical areas, rather than simply focus on "poorer performing" HCOs. The method draws attention to clinical areas exhibiting larger between-HCO variation and affecting larger numbers of patients. HCOs report data in six-month periods, resulting in estimated clinical indicator proportions which may be affected by small samples and sampling variation. Failing to address such issues would result in HCOs exhibiting extremely small and large estimated proportions and inflated estimates of the potential gains in the system. This paper describes the 20(th) centile method of calculating potential gains for the healthcare system by using Bayesian hierarchical models and shrinkage estimators to correct for the effects of sampling variation, and provides an example case in Emergency Medicine as well as example expert commentary from colleges based upon the reports. The application of these Bayesian methods enables all collated data to be used, irrespective of an HCO's size, and facilitates more realistic estimates of potential system gains.

Clinical indicators in surgery: a critical review of the Australian experience.

BACKGROUND: A set of clinical measures (indicators), developed by an Australian Council on Healthcare Standards (ACHS) and Royal Australasian College of Surgeons (RACS) working party, was introduced into the accreditation programme in 1997. Although early qualitative and quantitative reporting by health-care organizations (HCOs) reflected their value in stimulating change, the number of HCOs reporting data on this set of clinical indicators (CIs) has declined, despite an increase in the number of HCOs reporting data on the CIs programme overall. Possible reasons for this decline were sought. METHODS: A retrospective review of prospectively collected surgical CI data was performed, a national survey of stakeholders in the ACHS programme was conducted and a comparison was made with published international data. RESULTS: From a maximum of 247 HCOs reporting data in 2002, the number fell to 168 by 2011. While favourable trends were evident with some CIs, for example, a decline in the rate of negative histology in childhood appendicectomy and in the rate of in-hospital infection in total hip joint replacement, there was minimal change with many of the CIs, suggesting limited responsiveness as measures of care. In the national survey, stakeholder's response was positive overall, but there was a requirement for regular review of CIs. Although some colleges viewed the CIs as simplistic and not reliable, comparisons with similar measures available in the international literature were favourable. CONCLUSIONS: Possible reasons for the declining number of HCOs reporting surgical CI data are a lack of a recent revision of the CIs and a lack of engagement of clinicians from the RACS. Revision of the surgical CI set is required.

Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study.

BACKGROUND: Despite the widespread use of accreditation in many countries, and prevailing beliefs that accreditation is associated with variables contributing to clinical care and organisational outcomes, little systematic research has been conducted to examine its validity as a predictor of healthcare performance. OBJECTIVE: To determine whether accreditation performance is associated with self-reported clinical performance and independent ratings of four aspects of organisational performance. DESIGN: Independent blinded assessment of these variables in a random, stratified sample of health service organisations. SETTINGS: Acute care: large, medium and small health-service organisations in Australia. Study participants Nineteen health service organisations employing 16 448 staff treating 321 289 inpatients and 1 971 087 non-inpatient services annually, representing approximately 5% of the Australian acute care health system. MAIN MEASURES: Correlations of accreditation performance with organisational culture, organisational climate, consumer involvement, leadership and clinical performance. Results Accreditation performance was significantly positively correlated with organisational culture (rho=0.618, p=0.005) and leadership (rho=0.616, p=0.005). There was a trend between accreditation and clinical performance (rho=0.450, p=0.080). Accreditation was unrelated to organisational climate (rho=0.378, p=0.110) and consumer involvement (rho=0.215, p=0.377). CONCLUSIONS: Accreditation results predict leadership behaviours and cultural characteristics of healthcare organisations but not organisational climate or consumer participation, and a positive trend between accreditation and clinical performance is noted.

Assessing the impact of socio-economic status on cancer survival in New South Wales, Australia 1996-2001.

OBJECTIVE: To assess the impact of socio-economic status (SES) on cancer survival in the state of New South Wales (NSW), Australia. METHODS: Patients diagnosed with one of 13 major cancers during 1992-2000 in NSW were followed-up to the end of 2001. The effect of SES on survival was estimated for each individual cancer and all 13 cancers combined using multivariable modeling. The numbers of lives that could be extended if all people had the same level of excess risk of death due to cancer as patients in the highest SES areas were also estimated. RESULTS: There were highly statistically significant variations in survival across SES groups for four cancers: stomach, liver, lung, and breast and all 13 cancers combined. Variation remained highly significant after adjusting for disease stage. Patients in lower SES areas had 10-20% higher excess risk than those in the highest SES areas. In total, there were 3,346 lives potentially extendable beyond 5 years; the highest number was for lung cancer (756). CONCLUSION: The significantly worse survival in lower SES areas from cancers of the stomach, liver, lung, and breast may be due to poorer access to high-quality cancer care. Estimates of the number of lives potentially extendable by improving cancer survival in lower SES areas suggest that priority should be given to improving lung cancer care in lower SES areas in NSW, Australia.

Evaluation of surgical performance using V-POSSUM risk-adjusted mortality rates.

Vascular-Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity (V-POSSUM) is a risk-adjusted scoring system for predicting 30-day mortality in patients undergoing vascular surgery. It can assess surgical performance by comparing predicted deaths with observed deaths. The aim of this analysis was to assess trends in surgical performance over time using risk-adjusted 30-day mortality as the primary outcome. Major vascular surgery procedures (n = 454) were prospectively scored for V-POSSUM between 1995 and 2006. Procedures were divided into 11 consecutive time bands. Observed and predicted deaths were compared using the logistic regression equation derived for V-POSSUM. The observed death rates decreased over time, as did the predicted number of deaths calculated from the V-POSSUM scores. The overall predicted mortality rate was 17.2% and the rate varied with the 12-month period, with a high of 23.9% and a low of 9.2%. The downward trend in the predicted rate shows that the patient risk factors have changed over time and that the risk of dying has declined by almost 50% (from 21.6 to 11.1%). There was a trend towards improved surgical performance over time, with a drop in the observed to predicted ratios of deaths. Observed and predicted deaths changed over the study periods. There was a trend towards improved performance compared with the risk-adjusted predicted mortality. V-POSSUM is a useful tool in the longitudinal assessment of performance in major vascular surgery.

Predictors of adverse events in surgical admissions in Australia.

OBJECTIVE: The purpose of this study was to determine risk factors of adverse events in five surgical procedures. DESIGN: Retrospective record review was used to determine adverse events and risk factors of 1,177 surgical admissions. Procedures included in this study were transurethral resection of prostate, hysterectomy, hip and knee arthroplasty, cholecystectomy and herniorrhaphy. Risk factors included comorbidity, lifestyle factors and medications. Stepwise multiple logistic regression was used to determine predictors of adverse events. SETTING: Two teaching hospitals in regional New South Wales, Australia. PARTICIPANTS: 1,177 surgical admissions for five high volume procedures. MAIN OUTCOME MEASURES: Identified predictors of adverse events in surgical admissions. RESULTS: The adverse event rate was 23.1% for all procedures (range 17.5-33.7% for the five procedures). Two factors were strongly predictive of an adverse event in all surgical admissions: age >70 years [odds ratio (OR) 1.9, 95% confidence intervals (CI) 1.3-2.6] and duration of operation (P = 0.005). Other predictive factors were: contaminated surgical site (OR 2.1, 95% CI 1.2-3.7) and anaemia (OR 1.8, 95% CI 1.1-2.8). Predictive factors of individual procedures included: urine retention (transurethral resection of the prostate); extended duration of operation and asthma (hysterectomy); acute admissions and extended duration of operation (cholecystectomy); and warfarin type drugs, ethanol abuse, failed prostheses, GI ulcer/inflammation, rheumatoid arthritis, and ischaemic heart disease (hip and knee joint arthroplasty). CONCLUSIONS: The results of this study suggest that five factors should be routinely monitored for patients undergoing these procedures: age >70 years, type of procedure, duration of operation >2 h, contaminated surgical site and anaemia.

Cancer incidence and mortality in aircraft maintenance workers.

BACKGROUND: A cancer incidence and mortality study was conducted in response to health concerns raised by workers from F-111 aircraft deseal/reseal fuel tank maintenance programs, to determine whether personnel exposed to deseal/reseal had an excess of cancers and mortality. METHODS: Number of deaths and cancers for individuals involved in F-111 DSRS activities were matched against two Air Force comparison groups. Analyses were weighted to adjust for differences in age, exposure period and rank. RESULTS: Eight hundred seventy-three exposed, 7,577 comparison group one, and 9,408 comparison group two individuals were matched against death and cancer data, with 431 cancers and 431 deaths. Cancer incidence was higher in the exposed group, with marginally significant increases of 40-50% (cancer incidence rate ratio range 1.45-1.62). Exposed group mortality was significantly lower than both comparison groups, likely due to survivor bias in the exposed group (mortality rate ratio range 0.33-0.44). CONCLUSIONS: On the balance of probabilities, there is an increased risk of cancer associated with participation in F-111 deseal/reseal activities.

Misclassification of colorectal cancer stage and area variation in survival.

We previously investigated the impact of health area of residence on colon and rectal cancer survival by estimating area-specific relative excess risk of death (RER), stratified by stage at diagnosis. The aims of this study were to quantify errors in colorectal cancer stage obtained from an Australian population-based cancer registry and assess the potential impact of errors in stage on these estimates. For a subset of cases, we compared the cancer registry stage with that from a survey of treating surgeons. We then randomly reallocated all cases to a simulated "corrected" stage according to the estimated misclassification probabilities and repeated the analysis of area variation stratified by simulated stage 1,000 times. We found 70% agreement between the Registry and Survey stage. This reallocation of the Registry cases by stage resulted in substantial variation in area-specific RERs across the simulated samples. Area variation in survival for localized colon and localized rectal cancer, which were previously statistically significant when classified using Registry stage, appeared no longer to be so. Misclassification of cancer registry stage can have an important impact on estimates of spatial variation in stage-specific colon and rectal cancer survival. If population-based cancer registry data are to be effectively used in evaluating and improving cancer care, the quality of the stage data may need to be improved.

A prospective, multi-method, multi-disciplinary, multi-level, collaborative, social-organisational design for researching health sector accreditation [LP0560737].

BACKGROUND: Accreditation has become ubiquitous across the international health care landscape. Award of full accreditation status in health care is viewed, as it is in other sectors, as a valid indicator of high quality organisational performance. However, few studies have empirically demonstrated this assertion. The value of accreditation, therefore, remains uncertain, and this persists as a central legitimacy problem for accreditation providers, policymakers and researchers. The question arises as to how best to research the validity, impact and value of accreditation processes in health care. Most health care organisations participate in some sort of accreditation process and thus it is not possible to study its merits using a randomised controlled strategy. Further, tools and processes for accreditation and organisational performance are multifaceted. METHODS/DESIGN: To understand the relationship between them a multi-method research approach is required which incorporates both quantitative and qualitative data. The generic nature of accreditation standard development and inspection within different sectors enhances the extent to which the findings of in-depth study of accreditation process in one industry can be generalised to other industries. This paper presents a research design which comprises a prospective, multi-method, multi-level, multi-disciplinary approach to assess the validity, impact and value of accreditation. DISCUSSION: The accreditation program which assesses over 1,000 health services in Australia is used as an exemplar for testing this design. The paper proposes this design as a framework suitable for application to future international research into accreditation. Our aim is to stimulate debate on the role of accreditation and how to research it.

Trends in survival and excess risk of death after diagnosis of cancer in 1980-1996 in New South Wales, Australia.

Survival from almost all cancers has improved during the last 30 years. There is debate over the reasons for the improvement. We examined trends in survival for 28 cancers from 1980 to 1996 in New South Wales (NSW), Australia, with adjustment for disease spread at diagnosis. NSW Central Cancer Registry data were used to estimate 5-year relative survival and relative excess risk of death for patients diagnosed in 1980-84, 1985-88, 1989-92 and 1993-96. Statistical significance of variation in excess deaths between periods of diagnosis was assessed using Poisson regression, with adjustment for age, sex, duration of follow-up, histology and spread of disease at diagnosis. There were statistically significant falls in excess deaths for 20 of the cancers with a 25% fall for all cancers combined. Cancers of the prostate, liver, thyroid, breast, gallbladder, body of uterus, rectum, cervix and ovary had falls of >30%. The falls varied by spread of disease; the largest being in localised and regionally spread tumours. Overall survival, when unadjusted for spread of cancer, generally fell in parallel with that in the specific categories of spread, which implies that stage migration did not contribute importantly to survival trends. While acknowledging the limitations of incomplete data on stage of cancer at diagnosis, we conclude that falls in excess deaths in NSW from 1980 to 1996 are unlikely, for many cancers, to be attributed to earlier diagnosis or stage migration; thus advances in cancer treatment have almost certainly contributed to them.

A population-based study from New South Wales, Australia 1996-2001: area variation in survival from colorectal cancer.

In this study, we have investigated the impact of area of residence on survival from colon and rectal cancer. Relative survival and relative excess risk of death from cancer were calculated for each of 17 health areas in New South Wales, Australia. There were statistically significant differences in survival across areas for both cancers after adjusting for demographic factors. The variation remained for colon cancer but was reduced for rectal cancer after adjustment for spread of disease at diagnosis. This persistent variation in colon cancer survival suggests that variation in treatment contributes to it, and there is separate evidence for such variation. Of the 7186 patients whose deaths within five years were attributable to colorectal cancer, 784 could have had their survival increased to more than five years if the excess risk of death in all areas was reduced to the 20th centile of its distribution. Estimates such as this can assist in prioritizing improvements in cancer services.

Surgeon and hospital volume and the management of colorectal cancer patients in Australia.

BACKGROUND: The evidence for a relationship between patient outcomes and clinician and hospital volume is increasing. The National Colorectal Cancer Care Survey was undertaken to determine the management patterns in Australia for individuals newly diagnosed with colorectal cancer in a 3 month period in the year 2000. METHODS: All new cases of colorectal cancer registered at each Australian State Cancer Registry were entered into the survey. This generated a questionnaire that was sent to the treating surgeon. Chi-squared tests and logistic regression analyses were used to determine levels of statistical significance. RESULTS: Of 2,383 surgical questionnaires generated, 2,015 (85%) were completed. The majority (58%) of surgeons treated one or two patients with colorectal cancer over the 3 months of the survey. There was variation across surgeon cohorts for preoperative measures including the use of deep vein thrombosis prophylaxis. Patients seen by low volume surgeons were most likely to be given a permanent stoma (P < 0.0001). Patients with rectal cancer who were operated on by high volume surgeons were significantly more likely to receive a colonic pouch (P < 0.0001). CONCLUSION: This nationwide population-based survey of the treatment of colorectal cancer patients suggests that the delivery of care by surgeons (the majority) who treat patients with rectal cancer infrequently should be evaluated.

Estimating regional variation in cancer survival: a tool for improving cancer care.

OBJECTIVE: To improve estimation of regional variation in cancer survival and identify cancers to which priority might be given to increase survival. METHODS: Survival measures were calculated for 25 major cancer types diagnosed in each of 17 health service regions in New South Wales, Australia, from 1991 to 1998. Region-specific risks of excess death due to cancer were estimated adjusting for age, sex, and extent of disease at, and years since, diagnosis. Empirical Bayes (EB) methods were used to shrink the estimates. The additional numbers of patients who would survive beyond five years were estimated by shifting the State average risk to the 20th centile. RESULTS: Statistically significant regional variation in the shrunken estimates of risk of excess death was found for nine of the 25 cancer types. The lives of 2903 people (6.4%) out of the 45,047 whose deaths within 5 years were attributable to cancer could be extended with the highest number being for lung cancer (791). CONCLUSIONS: The EB approach gives more precise estimates of region-specific risk of excess death and is preferable to standard methods for identifying cancer sites where gains in survival might be made. The estimated number of lives that could be extended can assist health authorities in prioritising investigation of and attention to causes of regional variation in survival.

Using indicators to quantify the potential to improve the quality of health care.

PURPOSE: Although clinical indicators allow individual providers to monitor and improve their own performance and quality of care, another important role for the indicators is to provide comparative information across all providers. We show that the 'league table' approach is ineffective, and provide an alternative method that uses the comparative rates to quantify the potential for improvement at both the provider and the national level. DATA SOURCES: The methods are applied to English and Australian hospital clinical indicators. METHODS: The key is to regard clinical indicators as screening tools that measure performance in one or more dimensions. All screening processes require explicit tests to determine whether the result should be classified as either positive (requires further investigation) or negative (requires continued monitoring). A clinical indicator will be defined as positive if any of the three following criteria are met: (1) large variation between all areas or hospitals, as defined by the 20th centile gains: requires improvement in the health care system; (2) large variation between strata (rural/urban, teaching/non-teaching, public/private, State): requires action in the relevant stratum; (3) outlier hospitals: requires quality improvement in the individual hospitals. Two techniques are used to determine whether any of the three criteria are positive: (1) empirical Bayesian estimation to calculate 'shrunken' rates; and (2) use of the 20th centile to quantify the potential gains or improvement. RESULTS: For 185 Australian indicators, 55 clinical indicators had system gains involving better outcomes for at least 1000 patients per indicator. Using a set of criteria and subjective judgement, we identified some key areas for quality improvement in Australia. CONCLUSION: Ranking of hospitals does not quantify the potential gains that could be achieved. Indicators that measure health care processes should be reported by quantifying the potential gains, thus encouraging action. Estimating the gains across many indicators allows priorities to be established, such as identifying the areas with the greatest potential for improvement. The main tasks are to then provide the tools and resources to tackle those areas with the most gains.

Complications after discharge for surgical patients.

AIM: To measure the type and frequency of complications for surgical patients 1 month after discharge. METHODS: A post-discharge patient survey was conducted in 2000 for patients who had undergone one of five elective operations: transurethral resection of the prostate, hysterectomy, major joint replacement, cholecystectomy, herniorrhaphy. Two hundred and fourteen patients (74%) returned the survey forms, which were sent 1 month after surgery. Patients were recruited from two teaching hospitals in the Hunter Area Health Service, New South Wales, Australia. RESULTS: One hundred and thirty-five (63%) patients reported one or more complications and 78 (37%) received treatment for 109 complications. Eighty-six per cent reported pain after discharge and 41% reported moderate to severe pain. Seventeen per cent reported infections after discharge and 94% of these patients were given treatment. Twenty-eight per cent reported bleeding after discharge and 20% of these were given treatment. Eleven (5%) patients were readmitted for treatment of problems related to their surgery including four who required further surgery. One hundred and seventy-two patients accessed a range of health services during the first month after discharge, resulting in 266 occasions of service. Twenty-eight per cent of post-discharge services were unplanned. CONCLUSIONS: The lack of post-discharge monitoring conceals information about surgical outcomes. Patient reporting is an effective method of monitoring post-discharge outcomes. There is scope to develop post-discharge services to improve the quality of care in the areas of post-discharge pain management, the use of prophylactic measures and to provide treatment for complications that occur during this period.

The effective use of a summary table and decision tree methodology to analyze very large healthcare datasets.

Very large datasets typically consists of millions of records, with many variables. Such datasets are stored and maintained by organizations because of the perceived potential information they contain. However, the problem with very large datasets is that traditional methods of data mining are not capable of retrieving this information because the software may be overwhelmed by the memory or computing requirements. In this article we outline a method that can analyze very large datasets. The method initially performs a data reduction step through the use of a summary table, which is then used as a reference dataset that is recursively partitioned to grow a decision tree.

Using hierarchical models to analyse clinical indicators: a comparison of the gamma-Poisson and beta-binomial models.

BACKGROUND: Clinical indicators (CIs) are used to assess, compare and determine the potential to improve the care provided by hospitals and physicians. The results for Australian hospitals in 1998-2000 have been reported using a new methodology. The gamma-Poisson hierarchical model was used to correct for the effects of sampling variation by obtaining the empirical Bayesian shrunken estimates for the CI proportions for each hospital. Then, an estimate of the potential system gains that could be achieved if the mean proportion was shifted to the 20th centile is obtained for each of the 185 CIs. The results are sed to prioritize quality improvement activity. OBJECTIVES: To describe the 20th centile method of calculating potential system gains in the health care system; to determine the impact of using the beta-binomial model rather than the gamma-Poisson model to obtain shrunken estimates for the CI proportions; and to compare the computationally simpler Method of Moments (MoM) with the maximum likelihood (ML) method for parameter estimation. METHODS: The formulae for the gamma-Poisson and beta-binomial shrinkage estimators were compared analytically. Each of the shrinkage estimators and the two methods of parameter estimation were applied to the Obstetric and Gynecological CIs, and the results compared. RESULTS The comparison of the formulae for the two shrinkage estimators showed that the gamma-Poisson model results in: greater shrinkage towards the overall mean. This was verified empirically using the clinical indicators. Additionally, the MoM was not a viable alternative to the ML method. CONCLUSIONS: The gamma-Poisson model provided smaller estimates of the potential system gains by up to 6.7% of the numerator for the clinical indicators. The difference in estimation increased with increasing mean proportions and between-hospital variation. We recommend that the beta-binomial model should be used on the basis of both theoretical and empirical grounds.