A framework for evaluating the frequency of external quality assessment challenges.

BACKGROUND AND AIMS: No clear rules about the optimal frequency of organizing External Quality Assessment (EQA) rounds exist. More frequent challenges will facilitate faster responses and more reliable statistics. Adding extra samples leads to extra information, but the correlation between results from different samples reduces the extra information from additional samples. MATERIALS AND METHODS: Data were used for ALT and Albumin from the RCPAQAP EQA scheme. Every two weeks, laboratories analysed two samples. Correlation between results of different samples was calculated to determine the power of distinguishing poorly from well-performing laboratories. The power was compared to hypothetical cases of no correlation and one-sample-per-week to estimate the number of samples negated due to correlation. RESULTS: Correlation leads to negation of a number of samples, but not more than 50% of samples were negated. The number of negated samples was positively related with the correlation between EQA results. CONCLUSIONS: The proposed framework provides a quantitative evaluation of the impact of adding more EQA rounds or samples. A correlation exists and is higher for analyses performed closer in time, but the examples shown here did not show a detrimental effect on correctly evaluating laboratories.

NGS for (Hemato-) Oncology in Belgium: Evaluation of Laboratory Performance and Feasibility of a National External Quality Assessment Program.

Next-generation sequencing (NGS) is being integrated into routine clinical practice in the field of (hemato-) oncology to search for variants with diagnostic, prognostic, or therapeutic value at potentially low allelic frequencies. The complex sequencing workflows used require careful validation and continuous quality control. Participation in external quality assessments (EQA) helps laboratories evaluate their performance and guarantee the validity of tests results with the ultimate goal of ensuring high-quality patient care. Here, we describe three benchmarking trials performed during the period 2017-2018 aiming firstly at establishing the state-of-the-art and secondly setting up a NGS-specific EQA program at the national level in the field of clinical (hemato-) oncology in Belgium. DNA samples derived from cell line mixes and artificially mutated cell lines, designed to carry variants of clinical relevance occurring in solid tumors, hematological malignancies, and BRCA1/BRCA2 genes, were sent to Belgian human genetics, anatomic pathology, and clinical biology laboratories, to be processed following routine practices, together with surveys covering technical aspects of the NGS workflows. Despite the wide variety of platforms and workflows currently applied in routine clinical practice, performance was satisfactory, since participating laboratories identified the targeted variants with success rates ranging between 93.06% and 97.63% depending on the benchmark, and few false negative or repeatability issues were identified. However, variant reporting and interpretation varied, underlining the need for further standardization. Our approach showcases the feasibility of developing and implementing EQA for routine clinical practice in the field of (hemato-) oncology, while highlighting the challenges faced.

A critical view at the ISO 13528 and IUPAC's harmonized protocol approach for proficiency testing for homogeneity assessment for quantitative variables.

To ensure a correct interpretation of Proficiency Testing (PT) or External Quality Assessment (EQA) results, sample batches need to be homogeneous. The importance of homogeneity testing has been underlined by ISO 13528 and the IUPAC harmonized protocol for proficiency testing. They require that the batch heterogeneity should be smaller than 0.3σpt, with σpt the standard deviation for proficiency assessment according to ISO/IEC 17043:2010 and provide procedures for assessing the batch heterogeneity. In this study, a critical appraisal of these procedures is given. Firstly, an assessment is made of the influence of the criterion of 0.3σpt on laboratory evaluation. Secondly, the assessment procedure of the batch heterogeneity is evaluated, with an emphasis on the calculation and the evaluation of the measured batch standard deviation. It was found that the 0.3σpt criterion may be loosened without a detrimental effect on laboratory evaluation. In addition, it was shown that a flaw in the calculation procedure forces the batch standard deviation to be 0 with a probability of 21.4% for batch standard deviations of 0.3σpt and repeatability of 0.5σpt. The hypothesis test for accepting a batch exhibits a high probability of accepting the batch, even for batch standard deviations that are a multiple of 0.3σpt. To conclude, this study shows that the criteria for homogeneity check of PT items by ISO 13528 and the IUPAC protocol do not assure batch homogeneity. They may be a combination of a criterion that is too strict and a permissive assessment. Alternative assessment procedures should be followed.

Alternative Sample-Homogeneity Test for Quantitative and Qualitative Proficiency Testing Schemes.

Proficiency Testing (PT) External Quality Assessment (EQA) schemes are designed to ascertain the ability of individual laboratories to perform satisfactorily with respect to their peer laboratories or to limits imposed by external sources. Observed deviation of a laboratory result for a PT sample must be entirely attributed to the laboratory and not to the PT provider. To minimize the probability that deviations could be attributed to the PT provider, sample homogeneity should be assured. It is generally required that for quantitative parameters, the standard deviation among PT units should be calculated on the basis of duplicate measurements of at least 10 samples chosen at random, and the standard deviation among PT units should not exceed 0.3 times the standard deviation used to evaluate laboratories. Because this approach has important drawbacks, an alternative procedure is proposed by applying the theory of acceptance sampling to the assessment of sample heterogeneity for both quantitative and qualitative data and deriving acceptance limits on the basis of minimizing the probability of falsely evaluating laboratories. For obtaining acceptance limits for quantitative parameters, a distinction is made between laboratory evaluation using fixed limits on the one hand and laboratory evaluation using limits that are based on the variability of the reported results on the other hand. Sequential tests are proposed to evaluate sample heterogeneity by means of a comparison with the χ2 distribution. For qualitative parameters, acceptance-sampling plans are proposed that are based on minimizing the joint probability of rejecting batches that have a satisfactory amount of defective samples and accepting batches unnecessarily. The approach for quantitative parameters is applied on samples for a PT scheme of ethanol quantification and for qualitative parameters on the presence of monoblasts in a blood smear. It was found that five samples could already be enough to prove that the batch was homogeneous for quantitative parameters, although more than 20 samples were needed to prove homogeneity for qualitative parameters. This study describes a direct relation among the objective of an PT round, the criteria for evaluating the results, and the sample heterogeneity. When samples are effectively homogeneous, less measurements are needed than current practices require. A drawback of the proposed approach is that the number of samples to be tested is not known beforehand, and good knowledge of the analytical variability is crucial. The formulas to be applied are relatively simple. Despite the drawbacks, the proposed approach is generally applicable for both quantitative and qualitative data.

Demystifying EQA statistics and reports.

Reports act as an important feedback tool in External Quality Assessment (EQA). Their main role is to score laboratories for their performance in an EQA round. The most common scores that apply to quantitative data are Q- and Z-scores. To calculate these scores, EQA providers need to have an assigned value and standard deviation for the sample. Both assigned values and standard deviations can be derived chemically or statistically. When derived statistically, different anomalies against the normal distribution of the data have to be handled. Various procedures for evaluating laboratories are able to handle these anomalies. Formal tests and graphical representation techniques are discussed and suggestions are given to help choosing between the different evaluations techniques. In order to obtain reliable estimates for calculating performance scores, a satisfactory number of data is needed. There is no general agreement about the minimal number that is needed. A solution for very small numbers is proposed by changing the limits of evaluation.Apart from analyte- and sample-specific laboratory evaluation, supplementary information can be obtained by combining results for different analytes and samples. Various techniques are overviewed. It is shown that combining results leads to supplementary information, not only for quantitative, but also for qualitative and semi-quantitative analytes.

A new approach to define acceptance limits for hematology in external quality assessment schemes.

BACKGROUND: A study performed in 2007 comparing the evaluation procedures used in European external quality assessment schemes (EQAS) for hemoglobin and leukocyte concentrations showed that acceptance criteria vary widely. For this reason, the Hematology working group from the European Organisation for External Quality Assurance Providers in Laboratory Medicine (EQALM) decided to perform a statistical study with the aim of establishing appropriate acceptance limits (AL) allowing harmonization between the evaluation procedures of European EQAS organizers. METHODS: Eight EQAS organizers from seven European countries provided their hematology survey results from 2010 to 2012 for red blood cells (RBC), hemoglobin, hematocrit, mean corpuscular volume (MCV), white blood cells (WBC), platelets and reticulocytes. More than 440,000 data were collected. The relation between the absolute value of the relative differences between reported EQA results and their corresponding assigned value (U-scores) was modeled by means of an adaptation of Thompson's "characteristic function". Quantile regression was used to investigate the percentiles of the U-scores for each target concentration range. For deriving AL, focus was mainly on the upper percentiles (90th, 95th and 99th). RESULTS: For RBC, hemoglobin, hematocrit and MCV, no relation was found between the U-scores and the target concentrations for any of the percentiles. For WBC, platelets and reticulocytes, a relation with the target concentrations was found and concentration-dependent ALs were determined. CONCLUSIONS: The approach enabled to determine state of the art-based ALs, that were concentration-dependent when necessary and usable by various EQA providers. It could also easily be applied to other domains.

Influence of dabigatran and rivaroxaban on routine coagulation assays. A nationwide Belgian survey.

The Belgian national External Quality Assessment Scheme performed a nationwide survey using lyophilised plasma samples spiked with dabigatran or rivaroxaban to demonstrate to the Belgian clinical laboratories how these drugs affect their routine coagulation assays prothrombin time (PT), activated partial thromboplastin time (aPTT), fibrinogen and antithrombin. Virtually all Belgian laboratories performing routine coagulation testing (189/192) participated in the survey. Both, dabigatran and rivaroxaban significantly prolonged the PT and aPTT in a concentration- and reagent-dependent manner. PT reagents were more influenced by rivaroxaban than by dabigatran and aPTT reagents more influenced by dabigatran than by rivaroxaban. Among PT reagents, Neoplastin R® was the most sensitive to rivaroxaban and Innovin® and Thromborel S® the least sensitive. Converting PT results to INR only increased the variability between reagents. Among aPTT reagents, Actin FSL® was the least sensitive to dabigatran while the other aPTT reagents showed slightly higher sensitivities. The presence of dabigatran led to falsely reduced fibrinogen concentrations when measured with a low thrombin concentration reagent. The presence of dabigatran caused an overestimation of the antithrombin level when measured with a thrombin-based activity assay and the presence of rivaroxaban an overestimation of the antithrombin level when measured with a FXa-based assay. Instrument-related differences were found for all tested parameters. In conclusion, this paper provides detailed information on the effect of dabigatran and rivaroxaban on routine coagulation assays as performed with a large number of reagent/instrument combinations.