Detecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators.

BACKGROUND: The Patient Safety Indicators (PSIs) use administrative data to screen for select adverse events (AEs). In this study, VA Surgical Quality Improvement Program (VASQIP) chart review data were used as the gold standard to measure the criterion validity of 5 surgical PSIs. Independent chart review was also used to determine reasons for PSI errors. METHODS: The sensitivity, specificity, and positive predictive value of PSI software version 4.1a were calculated among Veterans Health Administration hospitalizations (2003-2007) reviewed by VASQIP (n = 268,771). Nurses re-reviewed a sample of hospitalizations for which PSI and VASQIP AE detection disagreed. RESULTS: Sensitivities ranged from 31% to 68%, specificities from 99.1% to 99.8%, and positive predictive values from 31% to 72%. Reviewers found that coding errors accounted for some PSI-VASQIP disagreement; some disagreement was also the result of differences in AE definitions. CONCLUSIONS: These results suggest that the PSIs have moderate criterion validity; however, some surgical PSIs detect different AEs than VASQIP. Future research should explore using both methods to evaluate surgical quality.

Development and testing of tools to detect ambulatory surgical adverse events.

OBJECTIVES: Numerous health-care systems in the United States, including the Veterans Health Administration (VA), use the National Surgical Quality Improvement Program (NSQIP) to detect surgical adverse events (AEs). VASQIP sampling methodology excludes many routine ambulatory surgeries from review. Triggers, algorithms derived from clinical logic to flag cases where AEs have most likely occurred, could complement VASQIP by detecting a higher yield of ambulatory surgeries with a true surgical AE. METHODS: We developed and tested a set of ambulatory surgical AE trigger algorithms using a sample of fiscal year 2008 ambulatory surgeries from the VA Boston Healthcare System. We used VA Boston VASQIP-assessed cases to refine triggers and VASQIP-excluded cases to test how many trigger-flagged surgeries had a nurse chart review-detected surgical AE. Chart review was performed using the VA electronic medical record. We calculated the ratio of cases with a true surgical AE over flagged cases (i.e., the positive predictive value [PPV]), and the 95% confidence interval for each trigger. RESULTS: Compared with the VASQIP rate (9 AEs, or 2.8%, of the 322 charts assessed), nurse chart review of the 198 trigger-flagged surgeries yielded more cases with at least 1 AE (47 surgeries with an AE, or 6.0%, of the 782 VASQIP-excluded ambulatory surgeries). Individual trigger PPVs ranged from 12.4% to 58.3%. CONCLUSIONS: In comparison with VASQIP, our set of triggers identified a higher rate of surgeries with AEs in fewer chart-reviewed cases. Because our results are based on a relatively small sample, further research is necessary to confirm these findings.

How valid is the AHRQ Patient Safety Indicator "postoperative hemorrhage or hematoma"?

BACKGROUND: Postoperative hemorrhage or hematoma (PHH), an Agency for Healthcare Research and Quality Patient Safety Indicator, uses administrative data to detect cases of potentially preventable postsurgical bleeding requiring a reparative procedure. How accurately it identifies true events is unknown. We therefore determined PHH's positive predictive value. STUDY DESIGN: Using Patient Safety Indicator software (v.3.1a) and fiscal year 2003-2007 discharge data from 28 Veterans Health Administration hospitals, we identified 112 possible cases of PHH. Based on medical record abstraction, we characterized cases as true (TPs) or false positives (FPs), calculated positive predictive value, and analyzed FPs to ascertain reasons for incorrect identification and TPs to determine PHH-associated clinical consequences and risk factors. RESULTS: Eighty-four cases were TPs (positive predictive value, 75%; 95% CI, 66-83%); 63% had a hematoma diagnosis, 30% had a hemorrhage diagnosis, 7% had both. Reasons for FPs included events present on admission (29%); hemorrhage/hematoma identified and controlled during the original procedure rather than postoperatively (21%); or postoperative hemorrhage/hematoma that did not require a procedure (18%). Most TPs (82%) returned to the operating room for hemorrhage/hematoma management; 64% required blood products and 7% died in-hospital. The most common index procedures resulting in postoperative hemorrhage/hematoma were vascular (38%); 56% were performed by a physician-in-training (under supervision). We found no substantial association between physician training status or perioperative anticoagulant use and bleeding risk. CONCLUSIONS: PHH's accuracy could be improved by coding enhancements, such as adopting present on admission codes or associating a timing factor with codes dealing with bleeding control. The ability of PHH to identify events representing quality of care problems requires additional evaluation.

Validity of selected Patient Safety Indicators: opportunities and concerns.

BACKGROUND: The Agency for Healthcare Research and Quality (AHRQ) recently designed the Patient Safety Indicators (PSIs) to detect potential safety-related adverse events. The National Quality Forum has endorsed several of these ICD-9-CM-based indicators as quality-of-care measures. We examined the positive predictive value (PPV) of 3 surgical PSIs: postoperative pulmonary embolus and deep vein thrombosis (pPE/DVT), iatrogenic pneumothorax (iPTX), and accidental puncture and laceration (APL). STUDY DESIGN: We applied the AHRQ PSI software (v.3.1a) to fiscal year 2003 to 2007 Veterans Health Administration (VA) administrative data to identify (flag) patients suspected of having a pPE/DVT, iPTX, or APL. Two trained nurse abstractors reviewed a sample of 336 flagged medical records (112 records per PSI) using a standardized instrument. Inter-rater reliability was assessed. RESULTS: Of 2,343,088 admissions, 6,080 were flagged for pPE/DVT (0.26%), 1,402 for iPTX (0.06%), and 7,203 for APL (0.31%). For pPE/DVT, the PPV was 43% (95% CI, 34% to 53%); 21% of cases had inaccurate coding (eg, arterial not venous thrombosis); and 36% featured thromboembolism present on admission or preoperatively. For iPTX, the PPV was 73% (95% CI, 64% to 81%); 18% had inaccurate coding (eg, spontaneous pneumothorax), and 9% were pneumothoraces present on admission. For APL, the PPV was 85% (95% CI, 77% to 91%); 10% of cases had coding inaccuracies and 5% indicated injuries present on admission. However, 27% of true APLs were minor injuries requiring no surgical repair (eg, small serosal bowel tear). Inter-rater reliability was >90% for all 3 PSIs. CONCLUSIONS: Until coding revisions are implemented, these PSIs, especially pPE/DVT, should be used primarily for screening and case-finding. Their utility for public reporting and pay-for-performance needs to be reassessed.