Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program.

BACKGROUND: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year. RESULTS: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.

Use of national surgical quality improvement program data as a catalyst for quality improvement.

BACKGROUND: Semiannually, the National Surgical Quality Improvement Program (NSQIP) provides its participating sites with observed-to-expected (O/E) ratios for 30-day postoperative mortality and morbidity. At each reporting period, there is typically a small group of hospitals with statistically significantly high O/E ratios, meaning that their patients have experienced more adverse events than would be expected on the basis of the population characteristics. An important issue is to determine which actions a surgical service should take in the presence of a high O/E ratio. STUDY DESIGN: This article reviews case studies of how some of the Department of Veterans Affairs and private-sector NSQIP participating sites used the clinically rich NSQIP database for local quality improvement efforts. Data on postoperative adverse events before and after these local quality improvement efforts are presented. RESULTS: After local quality improvement efforts, wound complication rates were reduced at the Salt Lake City Veterans Affairs medical center by 47%, surgical site infections in patients undergoing intraabdominal surgery were reduced at the University of Virginia by 36%, and urinary tract infections in vascular patients were reduced at the Massachusetts General Hospital by 74%. At some sites participating in the NSQIP, notably the Massachusetts General Hospital and the University of Virginia, the NSQIP has served as the basis for surgical service-wide outcomes research and quality improvement programs. CONCLUSIONS: The NSQIP not only provides participating sites with risk-adjusted surgical mortality and morbidity outcomes semiannually, but the clinically rich NSQIP database can also serve as a catalyst for local quality improvement programs to significantly reduce postoperative adverse event rates.

Identification of surgical complications and deaths: an assessment of the traditional surgical morbidity and mortality conference compared with the American College of Surgeons-National Surgical Quality Improvement Program.

BACKGROUND: Despite advances by surgeons in assessing quality and safety, the traditional surgical morbidity and mortality (M&M) conference has mostly remained unchallenged and unchanged. The goal of this study was to compare data as reported in a traditional M&M conference to data collected using the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP) techniques. STUDY DESIGN: A retrospective study was performed comparing data from the M&M conference in a general surgery division, in which complications and deaths were identified by residents or attendings, to data compiled by a nationally audited nurse reviewer from the ACS-NSQIP from July 1, 2002, to June 30, 2003. RESULTS: Mortality rates calculated by traditional M&M conference (53 deaths in 5,905 patients), compared with the ACS-NSQIP nurse reviewer (28 deaths in 1,439 patients; 24% sample), were 0.9% versus 1.9%, respectively (p=0.001). Complication rates reported in M&M were 6.4% versus 28.9% ACS-NSQIP (p<0.0001). Subgroup analyses showed that mortality rates, as reported in conference, were substantially lower for both in-hospital and postdischarge patients, when compared with ACS-NSQIP. All subclassifications of complications, as presented in conference, were also lower, compared with ACS-NSQIP. CONCLUSIONS: Traditional surgical M&M reporting considerably underreports both in-hospital and postdischarge complications and deaths as compared with ACS-NSQIP. Approximately one of two deaths and three of four complications were not reported in the M&M conference at our institution. A Web-based reporting system based on an ACS-NSQIP platform was created to automate, facilitate, and standardize data on surgical morbidity and mortality.

Early outcomes of endovascular versus open abdominal aortic aneurysm repair in the National Surgical Quality Improvement Program-Private Sector (NSQIP-PS).

BACKGROUND: There remains no consensus on the appropriate application of endovascular abdominal aortic aneurysm repair (EVAR). Information from administrative databases, industry-sponsored trials, and single institutions has inherent deficiencies. This study was designed to compare early outcomes of open (OPEN) versus EVAR in a contemporary (2000 to 2003) large, multicenter prospective cohort. METHODS: Fourteen academic medical centers contributed data to the National Surgical Quality Improvement Program-Private Sector (NSQIP-PS), which ensures uniform, comprehensive, prospective, and previously validated data entry by trained, independent nurse reviewers. A battery of clinical and demographic features was assessed with multivariate analysis for association with the principal study end points of 30-day operative mortality and morbidity. RESULTS: One thousand forty-two patients underwent elective infrarenal abdominal aortic aneurysm (AAA) repairs: 460 EVAR and 582 OPEN. EVAR patients were older (74 vs 71 years, P < .0001), included more men (84.6% vs 79.6%, P < .05), and had a higher incidence of chronic obstructive pulmonary disease (25.4% vs 17.9%, P < .01). EVAR resulted in significantly reduced overall morbidity (24% vs 35%, P < .0001) and hospital stay (4 vs 9 days, P < .0001). Cardiopulmonary and renal function-related comorbidities had the expected significant impact on mortality for both procedures at univariate analysis ( P < .05). While crude mortality rates between EVAR and OPEN did not differ significantly (2.8% vs 4.0%) ( P = 0.32). After multivariate analysis, correlates of operative mortality included OPEN (odds ratio [OR], 2.44; 95% confidence interval [CI], 1.03 to 5.78; P < .05), advanced age (OR, 1.11; P < .001), history of angina (OR, 5.54; P < .01), poor functional status (OR, 5.78; P < .001), history of weight loss (OR, 7.42; P < .01), and preoperative dialysis (OR, 51.4; P < .0001). EVAR also compared favorably to OPEN (OR, 2.14; 95% CI, 1.58 to 2.89; P < .0001) for overall morbidity. CONCLUSION: Significant morbidity accompanies AAA repair, even at major academic medical centers. These data strongly endorse EVAR as the preferred approach in the presence of significant cardiopulmonary or renal comorbidities, or poor preoperative functional status.

Improving surgical wound classification--why it matters.

Surgical wound classification is an important predictor of the risk of postoperative surgical site infections. Wound classification also is used to analyze clinical, economic, and educational outcomes in national reports on quality. As integral members of the health care team, nurses and physicians need to ensure that their data are correct, consistent, and reliable. This article delineates how one institution developed a multifaceted education program that resulted in a 26% improvement in the rate of correctly classified wounds. The education program provided regular feedback of results that helped identify opportunities for improvement on a widespread level.