Profiling Individual Surgeon Performance Using Information from a High-Quality Clinical Registry: Opportunities and Limitations.

BACKGROUND: There is increasing interest in profiling the quality of individual medical providers. Valid assessment of individuals should highlight improvement opportunities, but must be considered in the context of limitations. STUDY DESIGN: High quality clinical data from the American College of Surgeons NSQIP, gathered in accordance with strict policies and specifications, was used to construct individual surgeon-level assessments. There were 39,976 cases evaluated, performed by 197 surgeons across 9 hospitals. Both 2-level (cases by surgeon) and 3-level (cases by surgeon by hospital) risk-adjusted, hierarchical regression analyses were performed. Outcomes were 30-day postoperative morbidity, surgical site infection, and mortality. Surgeon performance was compared in both absolute and relative terms. "Signal-to-noise" reliability was calculated for surgeons and models. Projected case requirements for reliability levels were generated. RESULTS: Surgeon performances could be distinguished to different degrees: morbidity distinguished best, mortality least. Outliers could be identified for morbidity and infection, but not mortality. Reliability was also highest for morbidity and lowest for mortality. Even models with high overall reliability did not assess all providers reliably. Incorporating institutional effects had predictable effects: penalizing providers at "good" institutions, benefiting providers at "poor" institutions. CONCLUSIONS: Individual surgeon profiles can, at times, be distinguished with moderate or good reliability, but to different degrees in different models. Absolute and relative comparisons are feasible. Incorporating institutional level effects in individual provider modeling presents an interesting policy dilemma, appearing to benefit providers at "poor-performing" institutions, but penalizing those at "high-performing" ones. No portrayal of individual medical provider quality should be accepted without consideration of modeling rationale and, critically, reliability.

Risk-adjusted hospital outcomes for children's surgery.

UNLABELLED: BACKGROUND The American College of Surgeons National Surgical Quality Improvement Program-Pediatric was initiated in 2008 to drive quality improvement in children's surgery. Low mortality and morbidity in previous analyses limited differentiation of hospital performance. METHODS: Participating institutions included children's units within general hospitals and free-standing children's hospitals. Cases selected by Current Procedural Terminology codes encompassed procedures within pediatric general, otolaryngologic, orthopedic, urologic, plastic, neurologic, thoracic, and gynecologic surgery. Trained personnel abstracted demographic, surgical profile, preoperative, intraoperative, and postoperative variables. Incorporating procedure-specific risk, hierarchical models for 30-day mortality and morbidities were developed with significant predictors identified by stepwise logistic regression. Reliability was estimated to assess the balance of information versus error within models. RESULTS: In 2011, 46 281 patients from 43 hospitals were accrued; 1467 codes were aggregated into 226 groupings. Overall mortality was 0.3%, composite morbidity 5.8%, and surgical site infection (SSI) 1.8%. Hierarchical models revealed outlier hospitals with above or below expected performance for composite morbidity in the entire cohort, pediatric abdominal subgroup, and spine subgroup; SSI in the entire cohort and pediatric abdominal subgroup; and urinary tract infection in the entire cohort. Based on reliability estimates, mortality discriminates performance poorly due to very low event rate; however, reliable model construction for composite morbidity and SSI that differentiate institutions is feasible. CONCLUSIONS: The National Surgical Quality Improvement Program-Pediatric expansion has yielded risk-adjusted models to differentiate hospital performance in composite and specific morbidities. However, mortality has low utility as a children's surgery performance indicator. Programmatic improvements have resulted in actionable data.

American College of Surgeons National Surgical Quality Improvement Program Pediatric: a beta phase report.

PURPOSE: The American College of Surgeons (ACS) National Surgical Quality Improvement Program Pediatric (NSQIP-P) expanded to beta phase testing with the enrollment of 29 institutions. Data collection and analysis were aimed at program refinement and development of risk-adjusted models for inter-institutional comparisons. METHODS: Data from the first full year of beta-phase NSQIP-P were analyzed. Patient accrual used ACS-NSQIP methodology tailored to pediatric specialties. Preliminary risk adjusted modeling for all pediatric and neonatal operations and pediatric (excluding neonatal) abdominal operations was performed for all cause morbidity (other than death) and surgical site infections (SSI) using hierarchical logistic regression methodology and eight predictor variables. Results were expressed as odds ratios with 95% confidence intervals. RESULTS: During calendar year 2010, 29 institutions enrolled 37,141 patients. 1644 total CPT codes were entered, of which 456 accounted for 90% of the cases. 450 codes were entered only once (1.2% of cases). For all cases, overall mortality was 0.25%, overall morbidity 7.9%, and the SSI rate 1.8%. For neonatal cases, mortality was 2.39%, morbidity 18.7%, and the SSI rate 3%. For the all operations model, risk-adjusted morbidity institutional odds ratios ranged 0.48-2.63, with 9/29 hospitals categorized as low outliers and 9/29 high outliers, while risk-adjusted SSI institutional odds ratios ranged 0.36-2.04, with 2/29 hospitals low outliers and 7/29 high outliers. CONCLUSION: This report represents the first risk-adjusted hospital-level comparison of surgical outcomes in infants and children using NSQIP-P data. Programmatic and analytic modifications will improve the impact of this program as it moves into full implementation. These results indicate that NSQIP-P has the potential to serve as a model for determining risk-adjusted outcomes in the neonatal and pediatric population with the goal of developing quality improvement initiatives for the surgical care of children.

Pediatric American College of Surgeons National Surgical Quality Improvement Program: feasibility of a novel, prospective assessment of surgical outcomes.

PURPOSE: The American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP) provides validated assessment of surgical outcomes. This study reports initiation of an ACS NSQIP Pediatric at 4 children's hospitals. METHODS: From October 2008 to June 2009, 121 data variables were prospectively collected for 3315 patients, including 30-day outcomes and tailoring the ACS NSQIP methodology to children's surgical specialties. RESULTS: Three hundred seven postoperative complications/occurrences were detected in 231 patients representing 7.0% of the study population. Of the patients with complications, 175 (75.7%) had 1, 39 (16.9%) had 2, and 17 (7.4%) had 3 or more complications. There were 13 deaths (0.39%) and 14 intraoperative occurrences (0.42%) detected. The most common complications were infection, 105 (34%) (SSI, 54; sepsis, 31; pneumonia, 13; urinary tract infection, 7); airway/respiratory events, 27 (9%); wound disruption, 18 (6%); neurologic events, 8 (3%) (nerve injury, 4; stroke/vascular event, 2; hemorrhage, 2); deep vein thrombosis, 3 (<1%); renal failure, 3 (<1%); and cardiac events, 3 (<1%). Current sampling captures 17.5% of cases across institutions with unadjusted complication rates ranging from 6.8% to 10.2%. Completeness of data collection for all variables exceeded 95% with 98% interrater reliability and 87% of patients having full 30-day follow-up. CONCLUSION: These data represent the first multiinstitutional prospective assessment of specialty-specific surgical outcomes in children. The ACS NSQIP Pediatric is poised for institutional expansion and future development of risk-adjusted models.

American College of Surgeons National Surgical Quality Improvement Program Pediatric: a phase 1 report.

BACKGROUND: There has been a long-standing desire to implement a multi-institutional, multispecialty program to address surgical quality improvement for children. This report documents results of the initial phase of the American College of Surgeons National Surgical Quality Improvement Program Pediatric. STUDY DESIGN: From October 2008 to December 2009, patients from 4 pediatric referral centers were sampled using American College of Surgeons National Surgical Quality Improvement Program methodology tailored to children. RESULTS: A total of 7,287 patients were sampled, representing general/thoracic surgery (n = 2,237; 30.7%), otolaryngology (n = 1,687; 23.2%), orthopaedic surgery (n = 1,367; 18.8%), urology (n = 893; 12.3%), neurosurgery (n = 697; 9.6%), and plastic surgery (n = 406; 5.6%). Overall mortality rate detected was 0.3% and 287 (3.9%) patients had postoperative occurrences. After accounting for demographic, preoperative, and operative factors, occurrences were 4 times more likely in those undergoing inpatient versus outpatient procedures (odds ratio [OR] = 4.71; 95% CI, 3.01-7.35). Other factors associated with higher likelihood of postoperative occurrences included nutritional/immune history, such as preoperative weight loss/chronic steroid use (OR = 1.49; 95% CI, 1.03-2.15), as well as physiologic compromise, such as sepsis/inotrope use before surgery (OR = 1.68; 95% CI, 1.10-1.95). Operative factors associated with occurrences included multiple procedures under the same anesthetic (OR = 1.58; 95% CI, 1.21-2.06) and American Society of Anesthesiologists classification category 4/5 versus 1 (OR = 5.74; 95% CI, 2.94-11.24). Specialty complication rates varied from 1.5% for otolaryngology to 9.0% for neurosurgery (p < 0.001), with specific procedural groupings within each specialty accounting for the majority of complications. Although infectious complications were the predominant outcomes identified across all specialties, distribution of complications varied by specialty. CONCLUSIONS: Based on this initial phase of development, the highly anticipated American College of Surgeons National Surgical Quality Improvement Program Pediatric has the potential to identify outcomes of children's surgical care that can be targeted for quality improvement efforts.

Association of surgical care improvement project infection-related process measure compliance with risk-adjusted outcomes: implications for quality measurement.

BACKGROUND: Facility-level process measure adherence is being publicly reported. However, the association between measure adherence and surgical outcomes is not well-established. Our objective was to determine the degree to which Surgical Care Improvement Project (SCIP) process measures are associated with American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) risk-adjusted outcomes. STUDY DESIGN: This cross-sectional study included hospitals participating in the ACS NSQIP and SCIP (n = 200). ACS NSQIP outcomes (30-day overall morbidity, serious morbidity, surgical site infections [SSI], and mortality) and adherence to SCIP SSI-related process measures (from the Hospital Compare database) were collected from January 1, 2008, through December 31, 2008. Hospital-level correlation coefficients between compliance with 4 process measures (ie, antibiotic administration within 1 hour before incision [SCIP-1]; appropriate antibiotic prophylaxis [SCIP-2]; antibiotic discontinuation within 24 hours after surgery [SCIP-3]; and appropriate hair removal [SCIP 6]) and 4 risk-adjusted outcomes were calculated. Regression analyses estimated the contribution of process measure adherence to risk-adjusted outcomes. RESULTS: Of 211 ACS NSQIP hospitals, 95% had data reported by Hospital Compare. Depending on the measure, hospital-level compliance ranged from 60% to 100%. Of the 16 correlations, 15 demonstrated nonsignificant associations with risk-adjusted outcomes. The exception was the relationship between SCIP-2 and SSI (p = 0.004). SCIP-1 demonstrated an intriguing but nonsignificant relationship with SSI (p = 0.08) and overall morbidity (p = 0.08). Although adherence to SCIP-2 was a significant predictor of risk-adjusted SSI (p < 0.0001) and overall morbidity (p < 0.0001), inclusion of compliance for SCIP-1 and SCIP-2 caused only slight improvement in model quality. CONCLUSIONS: Better adherence to infection-related process measures over the observed range was not significantly associated with better outcomes with one exception. Different measures of quality might be needed for surgical infection.

Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach.

The history and development of the NSQIP, from its inception in the Veterans Administration Health System to its implementation within the private sector sponsored by the ACS, documents the growth of a program that has substantially improved the quality of surgical care and has had a considerable influence on the culture of quality improvement in the profession. The success of the ACS NSQIP is the result of providing hospitals with rigorous, clinical data, networking opportunities, and resources to improve their risk-adjusted outcomes. In this manner, the ACS NSQIP challenges its hospitals and health care providers to continually improve the care they provide. In addition to reducing the complications and mortality experienced by patients after surgical procedures, hospitals that participate in the ACS NSQIP have seen the financial rewards of their quality improvement efforts. Continued growth of the ACS NSQIP will facilitate achievement of the primary goal surrounding the current health care reform debate: efficient, high-quality care.

Comparison of hospital performance in nonemergency versus emergency colorectal operations at 142 hospitals.

BACKGROUND: Quality improvement efforts have demonstrated considerable hospital-to-hospital variation in surgical outcomes. However, information about the quality of emergency surgical care is lacking. The objective of this study was to assess whether hospitals have comparable outcomes for emergency and nonemergency operations. STUDY DESIGN: Patients undergoing colorectal resections were identified from the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) 2005 to 2007 dataset. Logistic regression models for 30-day morbidity and mortality after emergency and nonemergency colorectal resections were constructed. Hospital risk-adjusted outcomes as measured by observed to expected (O/E) ratios, outlier status, and rank-order differences were compared. RESULTS: Of 25,710 nonemergency colorectal resections performed at 142 ACS NSQIP hospitals, 6,138 (23.9%) patients experienced at least 1 complication, and 492 (1.9%) patients died. There were 5,083 emergency colorectal resections; 2,442 (48%) patients experienced at least 1 complication, and 780 (15.3%) patients died. Outcomes for nonemergency versus emergency operations were weakly correlated for morbidity and mortality (Pearson correlation coefficient: 0.28 versus 0.13). Median differences in hospital rankings based on O/E ratios between nonemergency and emergency performance were 30.5 ranks (interquartile range [IQR] 13 to 59) for morbidity and 34 ranks (interquartile ratio 17 to 61) for mortality. CONCLUSIONS: Hospitals with favorable outcomes after nonemergency colorectal resections do not necessarily have similar outcomes for emergency operations. Hospitals should specifically examine their performance on emergency surgical procedures to identify quality improvement opportunities and focus quality improvement efforts appropriately.

Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program.

BACKGROUND: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year. RESULTS: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.