Using Both Clinical Registry and Administrative Claims Data to Measure Risk-adjusted Surgical Outcomes.

OBJECTIVE: To examine the validity of hybrid quality measures that use both clinical registry and administrative claims data, capitalizing on the strengths of each data source. BACKGROUND: Previous studies demonstrate substantial disagreement between clinical registry and administrative claims data on the occurrence of postoperative complications. Clinical data have greater validity than claims data for quality measurement but can be burdensome for hospitals to collect. METHODS: American College of Surgeons National Surgical Quality Improvement Program records were linked to Medicare inpatient claims (2005-2008). National Quality Forum-endorsed risk-adjusted measures of 30-day postoperative complications or death assessed hospital quality for patients undergoing colectomy, lower extremity bypass, or all surgical procedures. Measures use hierarchical multivariable logistic regression to identify statistical outliers. Measures were applied using clinical data, claims data, or a hybrid of both data sources. Kappa statistics assessed agreement on determinations of hospital quality. RESULTS: A total of 111,984 patients participated from 206 hospitals. Agreement on hospital quality between clinical and claims data was poor. Hybrid models using claims data to risk-adjust complications identified by clinical data had moderate agreement with all clinical data models, whereas hybrid models using clinical data to risk-adjust complications identified by claims data had routinely poor agreement with all clinical data models. CONCLUSIONS: Assessments of hospital quality differ substantially when using clinical registry versus administrative claims data. A hybrid approach using claims data for risk adjustment and clinical data for complications may be a valid alternative with lower data collection burden. For quality measures focused on postoperative complications to be meaningful, such policies should require, at a minimum, collection of clinical outcomes data.

Profiling Individual Surgeon Performance Using Information from a High-Quality Clinical Registry: Opportunities and Limitations.

BACKGROUND: There is increasing interest in profiling the quality of individual medical providers. Valid assessment of individuals should highlight improvement opportunities, but must be considered in the context of limitations. STUDY DESIGN: High quality clinical data from the American College of Surgeons NSQIP, gathered in accordance with strict policies and specifications, was used to construct individual surgeon-level assessments. There were 39,976 cases evaluated, performed by 197 surgeons across 9 hospitals. Both 2-level (cases by surgeon) and 3-level (cases by surgeon by hospital) risk-adjusted, hierarchical regression analyses were performed. Outcomes were 30-day postoperative morbidity, surgical site infection, and mortality. Surgeon performance was compared in both absolute and relative terms. "Signal-to-noise" reliability was calculated for surgeons and models. Projected case requirements for reliability levels were generated. RESULTS: Surgeon performances could be distinguished to different degrees: morbidity distinguished best, mortality least. Outliers could be identified for morbidity and infection, but not mortality. Reliability was also highest for morbidity and lowest for mortality. Even models with high overall reliability did not assess all providers reliably. Incorporating institutional effects had predictable effects: penalizing providers at "good" institutions, benefiting providers at "poor" institutions. CONCLUSIONS: Individual surgeon profiles can, at times, be distinguished with moderate or good reliability, but to different degrees in different models. Absolute and relative comparisons are feasible. Incorporating institutional level effects in individual provider modeling presents an interesting policy dilemma, appearing to benefit providers at "poor-performing" institutions, but penalizing those at "high-performing" ones. No portrayal of individual medical provider quality should be accepted without consideration of modeling rationale and, critically, reliability.

Comparison between clinical registry and medicare claims data on the classification of hospital quality of surgical care.

OBJECTIVE: To compare the classification of hospital statistical outlier status as better or worse performance than expected for postoperative complications using Medicare claims versus clinical registry data. BACKGROUND: Controversy remains as to the most favorable data source for measuring postoperative complications for pay-for-performance and public reporting polices. METHODS: Patient-level records (2005-2008) were linked between the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) and Medicare inpatient claims. Hospital statistical outlier status for better or worse performance than expected was assessed using each data source for superficial surgical site infection (SSI), deep/organ-space SSI, any SSI, urinary tract infection, pneumonia, sepsis, deep venous thrombosis, pulmonary embolism, venous thromboembolism, and myocardial infarction by developing hierarchical multivariable logistic regression models. Kappa statistics and correlation coefficients assessed agreement between the data sources. RESULTS: A total of 192 hospitals with 110,987 surgical patients were included. Agreement on hospital rank for complication rates between Medicare claims and ACS-NSQIP was poor-to-moderate (weighted κ: 0.18-0.48). Of hospitals identified as statistical outliers for better or worse performance by Medicare claims, 26% were also identified as outliers by ACS-NSQIP. Of outliers identified by ACS-NSQIP, 16% were also identified as outliers by Medicare claims. Agreement between the data sources on hospital outlier status classification was uniformly poor (weighted κ: -0.02-0.34). CONCLUSIONS: Despite using the same statistical methodology with each data source, classification of hospital outlier status as better or worse performance than expected for postoperative complications differed substantially between ACS-NSQIP and Medicare claims.

A Comprehensive Evaluation of Statistical Reliability in ACS NSQIP Profiling Models.

OBJECTIVE: To assess statistical reliability of hospital profiling models in ACS NSQIP (American College of Surgeons' National Surgical Quality Improvement Program). BACKGROUND: The ACS NSQIP January 2013 Semiannual Report provided risk-adjusted hospital quality assessments for 137 models. METHODS: Median reliability and percentage of hospitals achieving acceptable reliability were computed for each model. Average median reliability was computed across models with common outcomes. RESULTS: Median reliability varied across the 137 models, from a high of 0.91 for "All Cases Morbidity" to a low of 0.005 for "Procedure-Targeted Total Hip Arthroplasty Surgical Site Infection." Generally, reliability was greatest for models with larger sample sizes and higher outcome event rates. Among "Essentials" models, 72% attained a median reliability of 0.40 or more, and 24% of 0.70 or more. Among "Procedure-Targeted" models, 29% attained a median reliability of 0.40 or more, and 3% of 0.70 or more. Percentage of hospitals achieving an acceptable reliability of 0.40 ranged from 98% for "All Cases Morbidity" to 0% for "Procedure-Targeted Pancreatectomy Mortality." For Essentials models, average median reliability for each outcome, except mortality, was more than 0.40. However, for Procedure-Targeted models the average median was less than 0.40. CONCLUSIONS: For a large proportion of ACS NSQIP Essentials models, statistical reliability is adequate for assessing surgical quality and differentiating hospital performance. The Procedure-Targeted program is evolving in terms of statistical reliability, with promising results to date. These results also argue for broader discussions of statistical reliability in performance assessments for the profession.

Are procedures codes in claims data a reliable indicator of intraoperative splenic injury compared with clinical registry data?

BACKGROUND: Identifying iatrogenic injuries using existing data sources is important for improved transparency in the occurrence of intraoperative events. There is evidence that procedure codes are reliably recorded in claims data. The objective of this study was to assess whether concurrent splenic procedure codes in patients undergoing colectomy procedures are reliably coded in claims data as compared with clinical registry data. STUDY DESIGN: Patients who underwent colectomy procedures in the absence of neoplastic diagnosis codes were identified from American College of Surgeons (ACS) NSQIP data linked with Medicare inpatient claims data file (2005 to 2008). A κ statistic was used to assess coding concordance between ACS NSQIP and Medicare inpatient claims, with ACS NSQIP serving as the reference standard. RESULTS: A total of 11,367 colectomy patients were identified from 212 hospitals. There were 114 patients (1%) who had a concurrent splenic procedure code recorded in either ACS NSQIP or Medicare inpatient claims. There were 7 patients who had a splenic injury diagnosis code recorded in either data source. Agreement of splenic procedure codes between the data sources was substantial (κ statistic 0.72; 95% CI, 0.64-0.79). Medicare inpatient claims identified 81% of the splenic procedure codes recorded in ACS NSQIP, and 99% of the patients without a splenic procedure code. CONCLUSIONS: It is feasible to use Medicare claims data to identify splenic injuries occurring during colectomy procedures, as claims data have moderate sensitivity and excellent specificity for capturing concurrent splenic procedure codes compared with ACS NSQIP.

Identification of modifiable factors for reducing readmission after colectomy: a national analysis.

BACKGROUND: Rates of hospital readmission are currently used for public reporting and pay for performance. Colectomy procedures account for a large number of readmissions among operative procedures. Our objective was to compare the importance of 3 groups of clinical variables (demographics, preoperative risk factors, and postoperative complications) in predicting readmission after colectomy procedures. METHODS: Patient records (2005-2008) from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) were linked to Medicare inpatient claims. Patient demographics (n = 2), preoperative risk factors (n = 23), and 30-day postoperative complications (n = 17) were identified from ACS-NSQIP, whereas 30-day postoperative readmissions and costs were determined from Medicare. Multivariable logistic regression models were used to examine risk-adjusted predictors of colectomy readmission. RESULTS: Among 12,981 colectomy patients, the 30-day postoperative readmission rate was 13.5%. Readmitted patients had slightly greater rates of comorbidities and indicators of clinical severity and substantially greater rates of complications than non-readmitted patients. After risk adjustment, patients with a complication were 3.3 times as likely to be readmitted as patients without a complication. Among individual complications, progressive renal failure and organ-space surgical site infection had the highest risk-adjusted relative risks of readmission (4.6 and 4.0, respectively). Demographic, preoperative risk factor, and postoperative complication variables increased the ability to discriminate readmissions (reflected by the c-statistic) by 5.3%, 23.3%, and 35.4%, respectively. CONCLUSION: Postoperative complications after colectomy are more predictive of readmission than traditional risk factors. Focusing quality improvement efforts on preventing and managing postoperative complications may be the most important step toward reducing readmission rates.

Risk factors for superficial vs deep/organ-space surgical site infections: implications for quality improvement initiatives.

IMPORTANCE: Surgical site infections (SSIs) are the focus of numerous quality improvement initiatives because they are a common and costly cause of potentially preventable patient morbidity. Superficial and deep/organ-space SSIs differ in terms of anatomical location and clinical severity. OBJECTIVE: To identify risk factors that are uniquely predictive of superficial vs deep/organ-space SSIs occurring after colectomy procedures. DESIGN: Retrospective cohort study. SETTING: American College of Surgeons National Surgical Quality Improvement Program. PARTICIPANTS: Patients undergoing colectomy procedures in 2011 were identified by Current Procedural Terminology codes. INTERVENTION: Colectomy procedures. MAIN OUTCOMES AND MEASURES: We compared rates of superficial SSI and deep/organ-space SSI associated with perioperative variables of interest: demographics; preoperative clinical severity, risk factors, and comorbidities and variables related to the hospitalization or procedure. Hierarchical multivariable logistic regression models were developed to identify risk-adjusted predictors of each SSI type. RESULTS: Among 27 011 patients identified from 305 hospitals, 6.2% developed a superficial SSI and 4.7% developed a deep/organ-space SSI. Risk factors common to the occurrence of both SSI types were identified: open surgery (vs laparoscopic) and current smoker. Risk factors with differential effects on each SSI type included specific postoperative diagnoses, disseminated cancer, and irradiation therapy, which were all associated with increased odds of deep/organ-space SSI only. The graded relationship between increasing body mass index and SSI occurrence appeared to be stronger for superficial SSI. CONCLUSIONS AND RELEVANCE: Risk factors for superficial SSI and deep/organ-space SSI vary in terms of magnitude and significance, suggesting that these SSI types are somewhat different disease processes. Groups interested in preventing SSIs might improve success by considering these SSI types independently for root-cause analyses and development of best practices and interventions.

Reliability of evaluating hospital quality by colorectal surgical site infection type.

OBJECTIVE: To determine whether risk-adjusted colorectal SSI rates are statistically reliable as hospital quality measures. BACKGROUND: Policymakers use surgical site infections (SSI) for public reporting of hospital quality and pay-for-performance because they are a relatively common and costly cause of patient morbidity. METHODS: Patients who underwent a colorectal procedure in 2009 were identified from the American College of Surgeons National Surgical Quality Improvement Program. We developed hierarchical multivariate logistic models for (1) superficial SSI, (2) deep/organ-space SSI, and (3) "any SSI" and compared how each model ranked hospital-level risk-adjusted performance. Statistical reliability of hospital quality measurements was estimated on a scale from 0 to 1; with 0 indicating that apparent variation between a hospital's quality measurement and the average hospital is statistically unreliable, and 1 indicating that any observed variation is due to a real difference in performance. RESULTS: Mean reliability of hospital-level quality measurements was 0.650 for superficial, 0.404 for deep/organ-space, and 0.586 for "any SSI." Lower reliability was accounted for by relatively little variation in risk-adjusted SSI rates between hospitals and insufficient numbers of colorectal cases submitted by individual hospitals. In 2009, we estimate that 22.1% of all US hospitals performed a sufficient number of colorectal cases to report superficial SSI rates at a high standard of statistical reliability and 1.0% did for deep/organ-space SSI. CONCLUSIONS: As currently constructed, colorectal SSI quality measures might not meet a high standard of statistical reliability for most hospitals, limiting their ability to confidently differentiate high and low performance. Despite an expectation of improving statistical power, combining superficial and deep/organ-space SSI into an "any SSI" measure worsens reliability.

Association between occurrence of a postoperative complication and readmission: implications for quality improvement and cost savings.

OBJECTIVE: To estimate the effect of preventing postoperative complications on readmission rates and costs. BACKGROUND: Policymakers are targeting readmission for quality improvement and cost savings. Little is known regarding mutable factors associated with postoperative readmissions. METHODS: Patient records (2005-2008) from the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) were linked to Medicare inpatient claims. Risk factors, procedure, and 30-day postoperative complications were determined from ACS-NSQIP. The 30-day postoperative readmission and costs were determined from Medicare. Occurrence of a postoperative complication included surgical site infections and cardiac, pulmonary, neurologic, and renal complications. Multivariate regression models predicted the effect of reducing complication rates on risk-adjusted readmission rates and costs by procedure. RESULTS: The 30-day postoperative readmission rate was 12.8%. Complication rates for readmitted and nonreadmitted patients were 53% and 16% (P < 0.001). Patients with a postoperative complication had higher predicted probability of readmission and cost of readmission than patients without a complication. For the 20 procedures accounting for the greatest number of readmissions, reducing ACS-NSQIP complication rates by a relative 5% could result in prevention of 2092 readmissions per year and a savings to Medicare of $31.0 million per year. Preventing all ACS-NSQIP complications for these procedures could result in prevention of 41,846 readmissions per year and a savings of $620.3 million per year. CONCLUSIONS: This study provides substantial evidence that efforts to reduce postoperative readmissions should begin by focusing on postoperative complications that can be reliably and validly measured. Such an approach will not eliminate all postoperative readmissions but will likely have a major effect on readmission rates.

Identifying worsening surgical site infection performance: control charts versus risk-adjusted rate outlier status.

Control charts are used in industry to monitor performance and are being used increasingly in hospitals as a quality improvement tool. The authors' objective was to determine if control charts using surgical site infection (SSI) rates predict changes in outlier status for risk-adjusted SSI rates using data from a surgical registry, the American College of Surgeons National Surgical Quality Improvement Program. Control charts of monthly SSI rates for 100 hospitals were analyzed for indicators of a performance change in 2009 (vs 2008) using standard rules. Hospitals also were classified as having better, worse, or no change in outlier status for risk-adjusted SSI rates in 2009 (vs 2008). There was moderate agreement between these methods (weighted κ = 0.401). Control charts predicted nonworsening performance well (specificity = 92.9%) and identified changes in SSI performance sooner; however, they failed to identify 31.2% of hospitals with worsened outlier status. This study demonstrates that these quality measurement tools have unique strengths and weaknesses and are complementary uses of the same clinical data source.

Morbidity and mortality after colorectal procedures: comparison of data from the American College of Surgeons case log system and the ACS NSQIP.

BACKGROUND: Improving the quality of surgical care depends upon collection of robust data. The American College of Surgeons Case Log System enables surgeons to self-report patient risk factors and outcomes. In contrast, the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) uses trained data abstractors to record similar data and uses a strict data collection methodology. The objective of this study was to assess bias in data entry for colorectal cases by comparing data in these 2 registries. STUDY DESIGN: One year of NSQIP (July 1, 2008 to June 30, 2009) and 7 years of Case Log (2003 to 2010) data were examined. Colorectal cases were identified by current procedural terminology code. The frequencies of comparably defined variables were compared, and mortality models were developed using logistic regression. Observed and expected mortality rates were compared. RESULTS: Rates of most risk factor and outcome variables were significantly higher in NSQIP than those in Case Log. NSQIP had a higher unadjusted mortality rate (4.46% versus 3.69%, p < 0.001); however, the adjusted odds of mortality was significantly higher in Case Log (odds ratio 1.32, p < 0.05). The Case Log model overpredicted mortality in NSQIP by 22%, whereas the NSQIP model underpredicted mortality in Case Log by 12%. CONCLUSIONS: Significant differences exist between risk factor and outcome data in NSQIP and Case Log for colorectal procedures. These differences demonstrate the need for standardized data collection methods, as is required by NSQIP, including use of standard definitions, adherence to a follow-up period for outcomes, and use of audits. These measures would improve the validity of using a self-reported database to evaluate and benchmark performance.

Taking disclosure seriously: disclosing financial conflicts of interest at the American College of Surgeons.

BACKGROUND: The accurate disclosure of financial conflicts of interest has come to light as a sound component of managing relationships between surgeons and industry. STUDY DESIGN: In this study, we summarize and categorize 4 years of disclosures (2006-2009) given by presenters at the Annual Clinical Congress of the American College of Surgeons. RESULTS: We report 3,122 disclosures by 480 individuals. "Colorectal surgeon" was the most common profession among disclosers. The most common type of disclosure was "consulting." The company with the highest number of disclosures was Covidien. Disclosers used 195 different terms to describe their relationships. CONCLUSIONS: We propose a standard nomenclature for use by surgeons when disclosing future conflicts of interest. As the attention to disclosures increases, sound policy decisions would be facilitated by such a standardized nomenclature system.

A current profile and assessment of north american cholecystectomy: results from the american college of surgeons national surgical quality improvement program.

BACKGROUND: Cholecystectomy is among the most common surgical procedures performed in the United States. The current state of cholecystectomy outcomes, including variations in hospital performance, is unclear. The objective of this study is to compare the risk factors, indications, and 30-day outcomes, as well as variations in hospital performance associated with laparoscopic (LC) versus open cholecystectomy (OC) at 221 hospitals during a 4-year period. STUDY DESIGN: Using the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) database (2005-2008), patients were identified who underwent cholecystectomy and related procedures (cholangiogram and/or common bile duct exploration). Four outcomes were studied, ie, 30-day overall morbidity, serious morbidity, surgical site infections, and mortality. Forward stepwise logistic regressions yielded patient-level predicted probabilities, and hospital-level observed-to-expected ratios were determined. RESULTS: Of 65,511 patients, 58,659 (89.5%) underwent LC; 6,852 (10.5%) underwent OC. OC patients were considerably older with a higher comorbidity burden. LC patients were less likely to experience any morbidity (3.1% versus 17.8%; p < 0.0001), a serious morbidity (1.4% versus 11.1%; p < 0.0001), or a surgical site infection (1.3% versus 8.4%; p < 0.0001), and less likely to die (0.3% versus 2.8%; p < 0.0001). Observed-to-expected ratios for overall morbidity ranged from 0 to 3.55; for serious morbidity, 0 to 3.23; for surgical site infection, 0 to 7.02; for mortality, 0 to 13.05. CONCLUSIONS: Although overall incidence of adverse events is low after LC, substantial morbidity and mortality are associated with OC. Additionally, controlling for patient- and operation-related factors, considerable variations exist in hospital performance when evaluating 30-day outcomes after cholecystectomy.

Quantitative weighting of postoperative complications based on the accordion severity grading system: demonstration of potential impact using the american college of surgeons national surgical quality improvement program.

BACKGROUND: To quantify severity of postoperative complications based on the Accordion Severity Grading System, determine the ability of severity grading to enhance National Surgical Quality Improvement Program (NSQIP) data, and develop an aggregate measure of severity of complications (the postoperative morbidity index). STUDY DESIGN: Forty-three surgical experts rated case vignettes containing postoperative complications on a severity scale. Vignettes were based on the Accordion Severity Grading System derived from the Toronto Severity Grading System. The system was adjusted using the expert severity scale results and applied to 1 year of NSQIP outcomes (1,857 patients, 704 complications) at a large tertiary care center. RESULTS: Experts initially distinguished the 6 grades of severity in a highly significant manner (t-test probabilities all < 0.005), with 1 exception. They rated reoperation and single-system organ failure without reoperation as similar, rather than distinct, in severity. The Accordion System was adjusted to reflect this. Distinction of grades thereafter was highly significant (t-test probabilities all < 0.005). Application to American College of Surgeons NSQIP data provided important novel insights. For example, complications in 6 American College of Surgeons NSQIP categories spanned 4 or more severity grades. Severity-weighted outcomes revealed that quantitatively the greatest burden of outcomes was due to wound infection, shock, and return to the operating room, which is not revealed by unweighted outcomes. Based on this information, an aggregate measure of severity of complications-the postoperative morbidity index-was proposed. CONCLUSIONS: Quantitative severity weighting of complications is feasible. Adjustment of American College of Surgeons NSQIP outcomes using this quantitative severity grading system provides uniquely informative representations of relative burdens of morbidities.

Missing data in the American College of Surgeons National Surgical Quality Improvement Program are not missing at random: implications and potential impact on quality assessments.

BACKGROUND: Studying risk-adjusted outcomes in health care relies on statistical approaches to handling missing data. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) provides risk-adjusted assessments of surgical programs, traditionally imputing certain missing data points using a single round of multivariable imputation. Such imputation assumes that data are missing at random-without systematic bias-and does not incorporate estimation uncertainty. Alternative approaches, including using multiple imputation to incorporate uncertainty or using an indicator of missingness, can enhance robustness of evaluations. STUDY DESIGN: One year of de-identified data from the ACS NSQIP, representing 117 institutions and 106,113 patients, was analyzed. Using albumin variables as the missing data modeled, several imputation/adjustment models were compared, including the traditional NSQIP imputation, a new single imputation, a multiple imputation, and use of a missing indicator. RESULTS: Coefficients for albumin values changed under new single imputation and multiple imputation approaches. Multiple imputation resulted in increased standard errors, as expected. An indicator of missingness was highly explanatory, disproving the missing-at-random assumption. The effects of changes in approach differed for different outcomes, such as mortality and morbidity, and effects were greatest in smaller datasets. However, ultimate changes in patient risk assessment and institutional assessment were minimal. CONCLUSIONS: Newer statistical approaches to modeling missing (albumin) values result in noticeable statistical distinctions, including improved incorporation of imputation uncertainty. In addition, the missing-at-random assumption is incorrect for albumin. Despite these findings, effects on institutional assessments are small. Although effects can be most important with smaller data-sets, the current approach to imputing missing values in the ACS NSQIP appears reasonably robust.

Comparison of hospital performance in nonemergency versus emergency colorectal operations at 142 hospitals.

BACKGROUND: Quality improvement efforts have demonstrated considerable hospital-to-hospital variation in surgical outcomes. However, information about the quality of emergency surgical care is lacking. The objective of this study was to assess whether hospitals have comparable outcomes for emergency and nonemergency operations. STUDY DESIGN: Patients undergoing colorectal resections were identified from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) 2005 to 2007 dataset. Logistic regression models for 30-day morbidity and mortality after emergency and nonemergency colorectal resections were constructed. Hospital risk-adjusted outcomes as measured by observed to expected (O/E) ratios, outlier status, and rank-order differences were compared. RESULTS: Of 25,710 nonemergency colorectal resections performed at 142 ACS NSQIP hospitals, 6,138 (23.9%) patients experienced at least 1 complication, and 492 (1.9%) patients died. There were 5,083 emergency colorectal resections; 2,442 (48%) patients experienced at least 1 complication, and 780 (15.3%) patients died. Outcomes for nonemergency versus emergency operations were weakly correlated for morbidity and mortality (Pearson correlation coefficient: 0.28 versus 0.13). Median differences in hospital rankings based on O/E ratios between nonemergency and emergency performance were 30.5 ranks (interquartile range [IQR] 13 to 59) for morbidity and 34 ranks (interquartile ratio 17 to 61) for mortality. CONCLUSIONS: Hospitals with favorable outcomes after nonemergency colorectal resections do not necessarily have similar outcomes for emergency operations. Hospitals should specifically examine their performance on emergency surgical procedures to identify quality improvement opportunities and focus quality improvement efforts appropriately.

Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling.

BACKGROUND: Although logistic regression has commonly been used to adjust for risk differences in patient and case mix to permit quality comparisons across hospitals, hierarchical modeling has been advocated as the preferred methodology, because it accounts for clustering of patients within hospitals. It is unclear whether hierarchical models would yield important differences in quality assessments compared with logistic models when applied to American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) data. Our objective was to evaluate differences in logistic versus hierarchical modeling for identifying hospitals with outlying outcomes in the ACS-NSQIP. STUDY DESIGN: Data from ACS-NSQIP patients who underwent colorectal operations in 2008 at hospitals that reported at least 100 operations were used to generate logistic and hierarchical prediction models for 30-day morbidity and mortality. Differences in risk-adjusted performance (ratio of observed-to-expected events) and outlier detections from the two models were compared. RESULTS: Logistic and hierarchical models identified the same 25 hospitals as morbidity outliers (14 low and 11 high outliers), but the hierarchical model identified 2 additional high outliers. Both models identified the same eight hospitals as mortality outliers (five low and three high outliers). The values of observed-to-expected events ratios and p values from the two models were highly correlated. Results were similar when data were permitted from hospitals providing < 100 patients. CONCLUSIONS: When applied to ACS-NSQIP data, logistic and hierarchical models provided nearly identical results with respect to identification of hospitals' observed-to-expected events ratio outliers. As hierarchical models are prone to implementation problems, logistic regression will remain an accurate and efficient method for performing risk adjustment of hospital quality comparisons.

Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery.

BACKGROUND: Surgical decision-making and informed patient consent both benefit from having accurate information about risk. But currently available risk estimating systems have one or more limitations associated with lack of specificity to operation type, size of sample (reliability), range of outcomes predicted, and appreciation of hospital effects. STUDY DESIGN: Data from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) patients who underwent colorectal surgery in 2006 to 2007 were used to generate logistic prediction models for 30-day morbidity, serious morbidity, and mortality. Results for these three models were then used to construct a universal multivariable model to predict risk for all three outcomes. Model performance was externally validated against 2005 data. RESULTS: For 2006 to 2007, 28,863 patients were identified who underwent major colorectal operations at 182 hospitals. A single 15-variable predictor model exhibited discrimination (c-statistic) close to that observed for the separate models on all three outcomes. Similar discrimination was found when the 2006 to 2007 universal model was applied to 3,037 operations conducted in 2005 at 37 hospitals. CONCLUSIONS: The ACS NSQIP colorectal risk calculator allows surgeons to preoperatively provide patients with detailed information about their personal risks of overall morbidity, serious morbidity, and mortality. Because ACS NSQIP can also categorize hospitals as performing better or worse than expected (or as expected), surgeons have the opportunity to adjust risk probabilities for patients at their institutions accordingly.

The impact of surgeon specialization on patient mortality: examination of a continuous Herfindahl-Hirschman index.

OBJECTIVE: To examine the effect of surgeon specialization on patient outcomes, controlling for volume. BACKGROUND: There is great interest in the degree to which surgical specialization affects outcomes, particularly considering drives to measure and reward quality in healthcare. Although surgical specialization has been previously analyzed with respect to outcomes, most studies have treated it as a dichotomous variable based on academic credentials. We treat it here as a continuous variable defined quantitatively by procedural diversity. METHODS: We used 2002 to 2005 patient data from the National Surgical Quality Improvement Program for the Department of Surgery, Barnes Jewish Hospital, St. Louis, Missouri. To quantitate procedural specialization, Herfindahl-Hirschman indices for surgeons were calculated using billing codes. These indices were calculated according to 3 different levels of procedural aggregation. Using conditional logit models, we examined the relationship between these indices and 30-day postoperative mortality rates. RESULTS: Surgeon specialization was inversely related to mortality rates after adjusting for case volume when indices were calculated using medium procedural aggregation (odds ratio for mortality = 0.580 per 0.1 unit Herfindahl increase; P = 0.025) or low aggregation (odds ratio for mortality = 0.510 per 0.1 unit Herfindahl increase; P = 0.015). No relationship was observed at the high level of aggregation. CONCLUSIONS: The procedural concentration component of surgical specialization is correlated with improved mortality rates independently of case volume. However, how broadly or narrowly "specialization" is defined has an impact on this relationship.

Effect of subjective preoperative variables on risk-adjusted assessment of hospital morbidity and mortality.

OBJECTIVE: To examine the influence of American Society of Anesthesiologists Physical Status Classification (ASA) and preoperative Functional Health Status (FHS) variables on risk-adjusted estimates of surgical quality and to assess whether classifications are inflated at some hospitals. BACKGROUND: ASA and FHS are influential in risk-adjusted comparisons of surgical quality. However, because ASA and FHS are subjective they can be inflated, making patients appear more ill than they actually are, and crediting hospitals for a sicker patient population. METHODS: We identified 28,751 colorectal surgery patients at 170 hospitals participating in the American College of Surgeon's National Surgical Quality Improvement Program (ACS NSQIP) during 2006 to 2007. Logistic regression models were developed for morbidity and mortality with and without inclusion of ASA and FHS. Hospital quality rankings from the different models were compared. RESULTS: Morbidity and mortality rates were 24.3% and 3.9%, respectively. Percents of patients in ASA classes I through V were 3.3%, 46.4%, 41.5%,8.3%, and 0.7% and that were independent or partially or totally dependent were 89.2%, 7.2%, and 3.6%, respectively. Models that included ASA and FHS exhibited slightly better fit (Hosmer-Lemshow statistic) and discrimination(c-statistic) than models without both these variables, though magnitudes of differences were consistent with chance. There was inconsistent evidence for improper assignment of ASA and FHS. CONCLUSIONS: The small improvements in model quality when both ASA and FHS are present versus absent, suggest that they make a unique contribution to assessing severity of preoperative risk. With little indication that these subjective variables are subject to an important level of institutional bias, it is appropriate that they be used to assess risk-adjusted surgical quality. Periodic monitoring for inappropriate inflation of ASA status is warranted.