Precision quality control: a dynamic model for risk-based analysis of analytical quality.

OBJECTIVES: There is continuing pressure to improve the cost effectiveness of quality control (QC) for clinical laboratory testing. Risk-based approaches are promising but recent research has uncovered problems in some common methods. There is a need for improvements in risk-based methods for quality control. METHODS: We provide an overview of a dynamic model for assay behavior. We demonstrate the practical application of the model using simulation and compare the performance of simple Shewhart QC monitoring against Westgard rules. We also demonstrate the utility of trade-off curves for analysis of QC performance. RESULTS: Westgard rules outperform simple Shewhart control over a narrow range of the trade-off curve of false-positive and false negative risk. The risk trade-off can be visualized in terms of risk, risk vs. cost, or in terms of cost. Risk trade-off curves can be "smoothed" by log transformation. CONCLUSIONS: Dynamic risk-models may provide advantages relative to static models for risk-based QC analysis.

Risk Analysis for Quality Control Part 3: Practical Application of the Precision Quality Control Model.

BACKGROUND: We developed a theoretical framework (Precision Quality Control [PQC]) to minimize the cost of quality, but it is not known whether the method can be applied in practice. METHODS: We used data for 2 analytes, cadmium and carbohydrate-deficient transferrin (CDT), and applied the PQC framework to find the optimal control limits. These analytes were selected because they differed with respect to sigma values that are major determinants of control limits. We explored different ways to visualize the results: (a) risk trade-off (false-positive risk vs false-negative risk), (b) cost-risk trade-off (false-positive cost vs false-negative risk), and (c) cost minimization. RESULTS: We were able to use the PQC limit to produce 3 different visualizations to suggest control limits. The risk-based analysis was the simplest to apply, but the most difficult to interpret. The cost vs risk method was easy to apply but was still difficult to interpret. The cost minimization method was easy to interpret but required users to declare a willingness to pay that may be difficult to estimate. CONCLUSIONS: The PQC method can be used to find control limits that minimize the cost of quality.

Risk Analysis for Quality Control Part 1: The Impact of Transition Assumptions in the Parvin Model.

BACKGROUND: Setting quality control (QC) limits involves balancing the risk of false-positive results and false-negative results. Recent approaches to QC have focused on the assessment of false-negative results. The Parvin model is the most-used model for risk analysis. The Parvin model assumes that the system makes a transition from an in-control to an out-of-control (OOC) state but makes no further transitions after moving to the OOC state. The implications of this assumption are unclear. METHODS: We used simulation experiments to compare the performance of QC systems based on no OOC transitions allowed (NOOCTA) vs systems where OOC transitions were allowed (OOCTA). RESULTS: The NOOCTA assumption leads to paradoxical tradeoff curves between false-positive results and false-negative results. Predictions of a false-negative result based on NOOCTA were about 10 times lower than models based on OOCTA. CONCLUSIONS: The most common models for QC risk analysis underestimate false-negative results. There is a need to develop better risk-based methods for QC analysis.

Risk Analysis for Quality Control Part 2: Theoretical Foundations for Risk Analysis.

BACKGROUND: Risk analysis can be used to determine control limits for quality control (QC). The Parvin model is the most commonly used method for risk analysis; however; the Parvin model rests on assumptions that have been shown to produce paradoxical results and to underestimate risk. There is a need for an improved framework for risk analysis. METHODS: We developed a dynamic model (Markov Reward Model) to analyze the long-term behavior of an assay under the influence of a QC monitoring system. The model is flexible and accounts for different patterns of assay behavior (shift frequency, shift distribution) and the impact of error on patient outcomes. The model determines the distribution of undetected reported errors and the frequency of false-positive laboratory results as a function of QC settings. The model accounts for the competing risks (false detections, shifts in the mean) that cause an assay to move from an in-control state to an out-of-control state. RESULTS: The model provides a tradeoff curve that expresses the cost to prevent an unacceptable reported result in terms of laboratory cost (false-positive QC). The model can be used to optimize settings of a particular QC method or to compare the performance of different methods. CONCLUSIONS: We developed a method to evaluate that determines the cost to reduce the risk to patients (reported results with unacceptable errors) in terms of laboratory costs (false-positive QC).

Impact of assay stability on the false negative and false positive rates in quality control.

BACKGROUND: The dynamic Precision QC (PQC) model can be used to evaluate the performance of quality control (QC) monitoring systems. The model depends on inputs that describe the intrinsic shift behavior (i.e., stability) of an assay. The output of the model is a trade-off curve that shows the relationship between false negative (FN) and false positive (FP) risk events. The relationship between the inputs and outputs of this model has not yet been explored. METHODS: We used Monte Carlo simulation to generate trade-off curves using the PQC. We varied the input parameters that determine assay stability (shift probability and shift size distribution) and studied the impact of these inputs on the output (i.e., the trade-off curve relating FN risk to FP risk). RESULTS: FN risk is sensitive to the shift probability and the width of the control limits. FN risk is sensitive to the shape of the shift size distribution when the standard deviation (SD) of the shift size distribution is relatively narrow (i.e., SD < 2) but is less sensitive to the width of the shift size distribution when the SD is relatively large (i.e., SD > 2). CONCLUSIONS: Practical use of the PQC model may require the estimation of the shift probability and shift size distribution.

Bacterial culture time to detection in platelet components: An evidence synthesis and estimation of detection failures.

BACKGROUND: Non-pathogen reduction platelet bacterial risk control strategies in the US FDA guidance include at least one culture. Almost all of these strategies have a culture hold time of ≥12 h. Studies have reported time to detection (TTD) of bacterial cultures inoculated with bacteria from contaminated platelets, but these data and estimates of risk associated with detection failures have not been synthesized. METHODS: We performed a literature search to identify studies reporting TTD for samples obtained from spiked platelet components. Using extracted data, regression analysis was used to estimate TTD for culture bottles at different inoculum sizes. Detection failures were defined as events in which contaminated components are transfused to a patient. We then used published data on time of transfusion (ToT) to estimate the risk of detection failures in practice. RESULTS: The search identified 1427 studies, of which 16 were included for analysis. TTD data were available for 16 different organisms, including 14 in aerobic cultures and 11 in anaerobic cultures. For inocula of 1 colony forming unit (CFU), the average TTD for aerobic organisms was 19.2 h while it was 24.9 h in anaerobic organisms, but there was substantial overall variation. A hold time of 12 versus 24 h had minimal effect for most organisms. CONCLUSION: TTD variation occurs between bacterial species and within a particular species. Under typical inventory management, the relative contribution of culture detection failures is much smaller than the residual risk from sampling failures. Increasing the hold period beyond 12 h has limited value.

The impact of the sample time of secondary bacterial culture on the risk of exposure to contaminated platelet components: A mathematical analysis.

BACKGROUND: The US Food and Drug Administration (FDA) issued a guidance for bacterial risk control strategies for platelet components in September 2019 that includes strategies using secondary bacterial culture (SBC). While an SBC likely increases safety, the optimal timing of the SBC is unknown. Our aim was to develop a model to provide insight into the best time for SBC sampling. STUDY DESIGN AND METHODS: We developed a mathematical model based on the conditional probability of a bacterial contamination event. The model evaluates the impact of secondary culture sampling time over a range of bacterial contamination scenarios (lag and doubling times), with the primary outcome being the optimal secondary sampling time and the associated risk. RESULTS: Residual risk of exposure decreased with increasing inoculum size, later sampling times for primary culture, and using higher thresholds of exposure (in colony-forming units per milliliter). Given a level of exposure, the optimal sampling time for secondary culture depended on the timing of primary culture and on the expiration time. In general, the optimal sampling time for secondary culture was approximately halfway between the time of primary culture and the expiration time. CONCLUSION: Our model supports that the FDA guidance is quite reasonable and that sampling earlier in the specified secondary culture windows may be most optimal for safety.

The costs and benefits of cross-level quality control rules.

BACKGROUND: Quality is often monitored by multi-rule schemes that are applied at each level of QC material. Cross Level (CL) quality control rules have been proposed but have not been investigated. METHODS: We used computer simulation to study the impact of CL rules on time to detection and the false positive rate in a system using multirules (3-1s, 2-2s, 4-1s, and 10x) with 2 levels of QC material We also studied the effect of correlation between shifts at each level. The performance of QC policies was compared using simulation analysis. We also compared the detection rates of QC policies (with and without QC rules) using laboratory QC data. RESULTS: Implementing the CL rule increased the false positive rate and increased the detection rate for small shifts (around 1 standard deviation). CL rules had a greater impact when the correlation of shifts between levels was high. CONCLUSIONS: CL rules have the potential to increase detection rates, but also increase false positive rates. It is difficult to identify the circumstances where the benefits of increased detection outweigh the costs of false positives. Alternative approaches to QC should be explored.