Toward robust information: data quality and inter-rater reliability in the American College of Surgeons National Surgical Quality Improvement Program.

BACKGROUND: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure. STUDY DESIGN: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year. RESULTS: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables. CONCLUSIONS: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year.

The patient safety in surgery study: background, study design, and patient populations.

BACKGROUND: The purpose of this article is to describe the background, design, and patient populations of the Patient Safety in Surgery Study, as a preliminary to the articles in this journal that will report the results of the Study. STUDY DESIGN: The Patient Safety in Surgery Study was a prospective cohort study. Trained nurses collected preoperative risk factors, operative variables, and 30-day postoperative mortality and morbidity outcomes in patients undergoing major general and vascular operations at 128 Veterans Affairs (VA) medical centers and 14 selected university medical centers between October 1, 2001 and September 30, 2004. An Internet-based data collection system was used to input data from the different private medical centers. Semiannual feedback of observed to expected mortality and morbidity ratios was provided to the participating medical centers. RESULTS: During the 3-year study, total accrual in general surgery was 145,618 patients, including 68.5% from the VA and 31.5% from the private sector. Accrual in vascular surgery totaled 39,225 patients, including 77.8% from the VA and 22.2% from the private sector. VA patients were older and included a larger proportion of male patients and African Americans and Hispanics. The VA population included more inguinal, umbilical, and ventral hernia repairs, although the private-sector population included more thyroid and parathyroid, appendectomy, and operations for breast cancer. Preoperative comorbidities were similar in the two populations, but the rates of comorbidities were higher in the VA. American Society of Anesthesiologists classification tended to be higher in the VA. CONCLUSIONS: The National Surgical Quality Improvement Program methodology was successfully implemented in the 14 university medical centers. The data from the study provided the basis for the articles in this issue of the Journal of the American College of Surgeons.