The value of Sigma-metrics in laboratory medicine.

While Six Sigma is used in different disciplines to improve quality, Tony Badric and Elvar Theodorsson in a recent paper in CCLM have questioned Six Sigma application in medical laboratory concluding Six Sigma has provided no value to medical laboratory. In addition, the authors have expanded their criticism to Total Analytical Error (TAE) model and statistical quality control. To address their arguments, we have explained the basics of TAE model and Six Sigma and have shown the value of Six Sigma to medical laboratory.

Multirule procedures vs moving average algorithms for IQC: An appropriate comparison reveals how best to combine their strengths.

BACKGROUND: Moving Average Algorithms (MAA) have been widely recommended for use in Patient Based Real Time Quality Control applications (PBRTQC) to supplement or replace traditional Internal Quality Control (IQC) techniques. A recent "proof of concept" study recommends applying MAAs to IQC data to replace traditional IQC procedures because they "outperform Westgard Rules," which is a current standard of practice for IQC. METHODS: We generated power curves for multi-rule procedures with 2 and 4 control measurements per QC event, as well as a Simple Moving Average having block sizes of 5, 10, and 20 control measurements. We also assessed time to detection in terms of the Average Number of QC Events required to detect different sizes of systematic errors. RESULTS: As expected, the more control measurements included in the control technique, the better the error detection. However, when QC performance is considered on the Sigma Scale, high Sigma methods require only 1 or 2 control measurements to detect medically important systematic errors. MAAs have very low ability to detect error at the first few QC events following shift, so they suffer a lag phase in detecting medically important errors. MAAs are most useful for methods having 4.0 Sigma performance or less. Even then, large systematic shifts are more quickly detected by simple single and multirule procedures. CONCLUSIONS: Choice of control techniques (rules, means, ranges, etc.) should consider the Sigma-metric of the method. For methods having Sigmas of 4 or greater, traditional single rule and multirule procedures with Ns up to 4 are most effective; below 4 Sigma, a multirule coupled with a Simple Moving Average (SMA) rule with Ns of 4 to 8 can improve error detection.

A multi-test planning model for risk based statistical quality control strategies.

BACKGROUND: Efforts to improve QC for multi-test analytic systems should focus on risk-based bracketed SQC strategies, as recommended in the CLSI C24-Ed4 guidance for QC practices. The objective is to limit patient risk by controlling the expected number of erroneous patient test results that would be reported over the period an error condition goes undetected. METHODS: A planning model is described to provide a structured process for considering critical variables for the development of SQC strategies for continuous production multi-test analytic systems. The model aligns with the principles of the CLSI C24-Ed4 "roadmap" and calculation of QC frequency, or run size, based on Parvin's patient risk model. Calculations are performed using an electronic spreadsheet to facilitate application of the planning model. RESULTS: Three examples of published validation data are examined to demonstrate the application of the planning model for multi-test chemistry and enzyme analyzers. The ability to assess "what if" conditions is key to identifying the changes and improvements that are necessary to simplify the overall system to a manageable number of SQC procedures. CONCLUSIONS: The planning of risk based SQC strategies should align operational requirements for workload and reporting intervals with QC frequency in terms of the run size or the number of patient samples between QC events. Computer tools that support the calculation of run sizes greatly facilitate the planning process and make it practical for medical laboratories to quickly assess the effects of critical variables.

Planning SQC strategies and adapting QC frequency for patient risk.

BACKGROUND: Risk-based Statistical QC strategies are recommended by the CLSI guidance for Statistical Quality Control (C24-Ed4). Using Parvin's patient risk model, QC frequency can be determined in terms of run size, i.e., the number of patient samples between QC events. Run size provides a practical goal for planning SQC strategies to achieve desired test reporting intervals. METHODS: A QC Frequency calculator is utilized to evaluate critical factors (quality required for test, precision and bias observed for method, rejection characteristics of SQC procedure) and also to consider patient risk as a variable for adjusting run size. RESULTS: We illustrate the planning of SQC strategies for a HbA1c test where two levels of controls show different sigma performance, for three different HbA1c analyzers used to achieve a common quality goal in a network of laboratories, and for an 18 test chemistry analyzer where a common run size is achieved by changes in control rules and adjustments for the patient risk of different tests. CONCLUSIONS: Run size provides a practical characteristic for adapting QC frequency to systematize the SQC strategies for multiple levels of controls or multiple tests in a chemistry analyzer. Patient risk can be an important variable for adapting run size to fit the laboratory's desired reporting intervals for high volume continuous production analyzers.

POCT Analysts' Perspective: Practices and Wants for Improvement.

BACKGROUND: Point-of-care testing (POCT) continues to expand worldwide. Concerns remain about result quality despite guidelines and standards that specify testing practices. To better understand POCT testing worldwide, we polled analysts to obtain their views on actual practices and needs for improvement. METHODS: An online questionnaire was constructed on SurveyMonkey, a commercially available website for conducting such surveys. POCT analysts were sought worldwide from a pool of healthcare providers subscribed to a westgard.com newsletter or visitors to westgard.com and/or LinkedIn to one of the authors. RESULTS: Seventy-three percent of testing occurred in hospitals with 64% conducted in specialty settings. Regulatory mandates were followed by 88%. For most, less than 100 tests were performed per day fewer less than 25 devices. Nurses top the list of analysts. All but 5% of analysts received some form of training primarily from manufacturers. Eighty-seven percent verified devices/methods prior to implementation. Five percent do not perform daily QC; all analyzed external QC at least once per month. When QC limits exceed acceptable limits, 92% stop testing. Expired materials were used by 5%. The majority collected data for quality improvements. Eleven percent thought their organization's POCT is acceptable. The majority of respondents believe improvements need to be made in POCT. CONCLUSIONS: Analysts' POCT practices have and are improving to contribute positively to patients' healthcare and safety. Analysts do recognize problems and their wants/needs provide important information to improve their practices. Most participants desire more in-house and/or manufacturer training, explicit directions from manufacturers, manufacturer built-in quality and function checks, and oversight.

Establishing Evidence-Based Statistical Quality Control Practices.

OBJECTIVES: To establish an objective, scientific, evidence-based process for planning statistical quality control (SQC) procedures based on quality required for a test, precision and bias observed for a measurement procedure, probabilities of error detection and false rejection for different control rules and numbers of control measurements, and frequency of QC events (or run size) to minimize patient risk. METHODS: A Sigma-Metric Run Size Nomogram and Power Function Graphs have been used to guide the selection of control rules, numbers of control measurements, and frequency of QC events (or patient run size). RESULTS: A tabular summary is provided by a Sigma-Metric Run Size Matrix, with a graphical summary of Westgard Sigma Rules with Run Sizes. CONCLUSION: Medical laboratories can plan evidence-based SQC practices using simple tools that relate the Sigma-Metric of a testing process to the control rules, number of control measurements, and run size (or frequency of QC events).

Planning Risk-Based SQC Schedules for Bracketed Operation of Continuous Production Analyzers.

BACKGROUND: To minimize patient risk, "bracketed" statistical quality control (SQC) is recommended in the new CLSI guidelines for SQC (C24-Ed4). Bracketed SQC requires that a QC event both precedes and follows (brackets) a group of patient samples. In optimizing a QC schedule, the frequency of QC or run size becomes an important planning consideration to maintain quality and also facilitate responsive reporting of results from continuous operation of high production analytic systems. METHODS: Different plans for optimizing a bracketed SQC schedule were investigated on the basis of Parvin's model for patient risk and CLSI C24-Ed4's recommendations for establishing QC schedules. A Sigma-metric run size nomogram was used to evaluate different QC schedules for processes of different sigma performance. RESULTS: For high Sigma performance, an effective SQC approach is to employ a multistage QC procedure utilizing a "startup" design at the beginning of production and a "monitor" design periodically throughout production. Example QC schedules are illustrated for applications with measurement procedures having 6-σ, 5-σ, and 4-σ performance. CONCLUSIONS: Continuous production analyzers that demonstrate high σ performance can be effectively controlled with multistage SQC designs that employ a startup QC event followed by periodic monitoring or bracketing QC events. Such designs can be optimized to minimize the risk of harm to patients.

Selecting a Risk-Based SQC Procedure for a HbA1c Total QC Plan.

BACKGROUND: Recent US practice guidelines and laboratory regulations for quality control (QC) emphasize the development of QC plans and the application of risk management principles. The US Clinical Laboratory Improvement Amendments (CLIA) now includes an option to comply with QC regulations by developing an individualized QC plan (IQCP) based on a risk assessment of the total testing process. The Clinical and Laboratory Standards Institute (CLSI) has provided new practice guidelines for application of risk management to QC plans and statistical QC (SQC). METHODS: We describe an alternative approach for developing a total QC plan (TQCP) that includes a risk-based SQC procedure. CLIA compliance is maintained by analyzing at least 2 levels of controls per day. A Sigma-Metric SQC Run Size nomogram provides a graphical tool to simplify the selection of risk-based SQC procedures. APPLICATIONS: Current HbA1c method performance, as demonstrated by published method validation studies, is estimated to be 4-Sigma quality at best. Optimal SQC strategies require more QC than the CLIA minimum requirement of 2 levels per day. More complex control algorithms, more control measurements, and a bracketed mode of operation are needed to assure the intended quality of results. CONCLUSIONS: A total QC plan with a risk-based SQC procedure provides a simpler alternative to an individualized QC plan. A Sigma-Metric SQC Run Size nomogram provides a practical tool for selecting appropriate control rules, numbers of control measurements, and run size (or frequency of SQC). Applications demonstrate the need for continued improvement of analytical performance of HbA1c laboratory methods.

Measuring Analytical Quality: Total Analytical Error Versus Measurement Uncertainty.

To characterize analytical quality of a laboratory test, common practice is to estimate Total Analytical Error (TAE) which includes both imprecision and trueness (bias). The metrologic approach is to determine Measurement Uncertainty (MU), which assumes bias can be eliminated, corrected, or ignored. Resolving the differences in these concepts and approaches is currently a global issue.

Rhetoric Versus Reality? Laboratory Surveys Show Actual Practice Differs Considerably from Proposed Models and Mandated Calculations.

The scientific debate on goals, measurement uncertainty, and individualized quality control plans has diverged significantly from the reality of laboratory operation. Academic articles promoting certain approaches are being ignored; laboratories may be in compliance with new regulations, mandates, and calculations, but most of them still adhere to traditional quality management practices. Despite a considerable effort to enforce measurement uncertainty and eliminate or discredit allowable total error, laboratories continue to use these older, more practical approaches for quality management.

Six Sigma Quality Management System and Design of Risk-based Statistical Quality Control.

Six sigma concepts provide a quality management system (QMS) with many useful tools for managing quality in medical laboratories. This Six Sigma QMS is driven by the quality required for the intended use of a test. The most useful form for this quality requirement is the allowable total error. Calculation of a sigma-metric provides the best predictor of risk for an analytical examination process, as well as a design parameter for selecting the statistical quality control (SQC) procedure necessary to detect medically important errors. Simple point estimates of sigma at medical decision concentrations are sufficient for laboratory applications.

Planning Risk-Based Statistical Quality Control Strategies: Graphical Tools to Support the New Clinical and Laboratory Standards Institute C24-Ed4 Guidance.

BACKGROUND: Clinical and Laboratory Standards Institute (CLSI)'s new guideline for statistical quality control (SQC; C24-Ed4) (CLSI C24-Ed4, 2016; Parvin CA, 2017) recommends the implementation of risk-based SQC strategies. Important changes from earlier editions include alignment of principles and concepts with the general patient risk model in CLSI EP23A (CLSI EP23A, 2011) and a recommendation for optimizing the frequency of SQC (number of patients included in a run, or run size) on the basis of the expected number of unreliable final patient results. The guideline outlines a planning process for risk-based SQC strategies and describes 2 applications for examination procedures that provide 9-σ and 4-σ quality. A serious limitation is that there are no practical tools to help laboratories verify the results of these examples or perform their own applications. METHODS: Power curves that characterize the rejection characteristics of SQC procedures were used to predict the risk of erroneous patient results based on Parvin's MaxE(Nuf) parameter (Clin Chem 2008;54:2049-54). Run size was calculated from MaxE(Nuf) and related to the probability of error detection for the critical systematic error (Pedc). RESULTS: A plot of run size vs Pedc was prepared to provide a simple nomogram for estimating run size for common single-rule and multirule SQC procedures with Ns of 2 and 4. CONCLUSIONS: The "traditional" SQC selection process that uses power function graphs to select control rules and the number of control measurements can be extended to determine SQC frequency by use of a run size nomogram. Such practical tools are needed for planning risk-based SQC strategies.

Utilizing global data to estimate analytical performance on the Sigma scale: A global comparative analysis of methods, instruments, and manufacturers through external quality assurance and proficiency testing programs.

OBJECTIVE: To assess the analytical performance of instruments and methods through external quality assessment and proficiency testing data on the Sigma scale. DESIGN AND METHODS: A representative report from five different EQA/PT programs around the world (2 US, 1 Canadian, 1 UK, and 1 Australasian) was accessed. The instrument group standard deviations were used as surrogate estimates of instrument imprecision. Performance specifications from the US CLIA proficiency testing criteria were used to establish a common quality goal. Then Sigma-metrics were calculated to grade the analytical performance. RESULTS: Different methods have different Sigma-metrics for each analyte reviewed. Summary Sigma-metrics estimate the percentage of the chemistry analytes that are expected to perform above Five Sigma, which is where optimized QC design can be implemented. The range of performance varies from 37% to 88%, exhibiting significant differentiation between instruments and manufacturers. Median Sigmas for the different manufacturers in three analytes (albumin, glucose, sodium) showed significant differentiation. CONCLUSIONS: Chemistry tests are not commodities. Quality varies significantly from manufacturer to manufacturer, instrument to instrument, and method to method. The Sigma-assessments from multiple EQA/PT programs provide more insight into the performance of methods and instruments than any single program by itself. It is possible to produce a ranking of performance by manufacturer, instrument and individual method. Laboratories seeking optimal instrumentation would do well to consult this data as part of their decision-making process. To confirm that these assessments are stable and reliable, a longer term study should be conducted that examines more results over a longer time period.

Quality control review: implementing a scientifically based quality control system.

This review focuses on statistical quality control in the context of a quality management system. It describes the use of a 'Sigma-metric' for validating the performance of a new examination procedure, developing a total quality control strategy, selecting a statistical quality control procedure and monitoring ongoing quality on the sigma scale. Acceptable method performance is a prerequisite to the design and implementation of statistical quality control procedures. Statistical quality control can only monitor performance, and when properly designed, alert analysts to the presence of additional errors that occur because of unstable performance. A new statistical quality control planning tool, called 'Westgard Sigma Rules,' provides a simple and quick way for selecting control rules and the number of control measurements needed to detect medically important errors. The concept of a quality control plan is described, along with alternative adaptations of a total quality control plan and a risk-based individualized quality control plan. Finally, the ongoing monitoring of analytic performance and test quality are discussed, including determination of measurement uncertainty from statistical quality control data collected under intermediate precision conditions and bias determined from proficiency testing/external quality assessment surveys. A new graphical tool, called the Sigma Quality Assessment Chart, is recommended for demonstrating the quality of current examination procedures on the sigma scale.

Assessing quality on the Sigma scale from proficiency testing and external quality assessment surveys.

BACKGROUND: There is a need to assess the quality being achieved for laboratory examinations that are being utilized to support evidence-based clinical guidelines. Application of Six Sigma concepts and metrics can provide an objective assessment of the current analytical quality of different examination procedures. METHODS: A "Sigma Proficiency Assessment Chart" can be constructed for data obtained from proficiency testing and external quality assessment surveys to evaluate the observed imprecision and bias of method subgroups and determine quality on the Sigma scale. RESULTS: Data for hemoglobin A1c (HbA1c) from a 2014 survey by the College of American Pathologists (CAP) demonstrates that approximately two-thirds of the examination subgroups provide only two-Sigma quality when evaluated against the CAP requirement of an allowable total error of 6.0%. The weighted averages were 1.46 Sigma for a survey sample with an assigned value of 6.49% Hb (average bias 2.31%, CV 2.87%), 1.45 Sigma at 6.97% Hb (average bias 2.29%, CV 2.81%), and 1.75 at 9.65% Hb (average bias 1.55%, CV 2.71%). Maximum biases for examination subgroups were 5.7%, 5.8%, and 4.1%, respectively. CONCLUSIONS: Assessment of quality on the Sigma scale provides evidence of the analytical performance that is being achieved relative to requirements for intended use and should be useful for identifying and prioritizing improvements that are needed in the analytical quality of laboratory examinations. In spite of global and national standardization programs, bias is still a critical limitation of current HbA1c examination procedures.

The quality of laboratory testing today: an assessment of sigma metrics for analytic quality using performance data from proficiency testing surveys and the CLIA criteria for acceptable performance.

To assess the analytic quality of laboratory testing in the United States, we obtained proficiency testing survey results from several national programs that comply with Clinical Laboratory Improvement Amendments (CLIA) regulations. We studied regulated tests (cholesterol, glucose, calcium, fibrinogen, and prothrombin time) and nonregulated tests (international normalized ratio [INR], glycohemoglobin, and prostate-specific antigen [PSA]). Quality was assessed on the sigma scale with a benchmark for minimum process performance of 3 sigma and a goal for world-class quality of 6 sigma. Based on the CLIA criteria for acceptable performance in proficiency testing (allowable total errors [TEa]), the national quality of cholesterol testing (TEa = 10%) estimated sigma values as 2.9 to 3.0; glucose (TEa = 10%), 2.9 to 3.3; calcium (TEa = 1.0 mg/dL), 2.8 to 3.0; prothrombin time (TEa = 15%), 1.8; INR (TEa = 20%), 2.4 to 3.5; fibrinogen (TEa = 20%), 1.8 to 3.2; glycohemoglobin (TEa = 10%), 1.9 to 2.6; and PSA (TEa = 10%), 1.2 to 1.8. The analytic quality of laboratory tests requires improvement in measurement performance and more intensive quality control monitoring than the CLIA minimum of 2 levels per day.