The usefulness of calibration verification and linearity surveys in predicting acceptable performance in graded proficiency tests.

OBJECTIVE: To assess the association between performance on graded chemistry surveys and evaluation of linearity and calibration in Linearity surveys. DESIGN: Data from Linearity Surveys (LN series) and from routine comprehensive College of American Pathologists chemistry surveys (all series) were used to evaluate the hypothesis that laboratories with nonlinear or univerified calibration would have a greater likelihood of unacceptable performance on comprehensive chemistry surveys. RESULTS: This study found that acceptable calibration verification evaluation is significantly related to acceptable rates for most analytes, including albumin, calcium, chloride, glucose, iron, magnesium, sodium, total bilirubin, uric acid, high-density lipoprotein cholesterol, triglycerides, alkaline phosphatase, alanine and aspartate aminotransferase, digoxin, gentamicin, phenobarbital, procainamide, and thyroxine. CONCLUSION: There is a consistent and strong relationship between calibration verification problems in the Linearity Surveys and failure rates in the College of American Pathologists chemistry surveys. Laboratories with poor calibration evaluations on Linearity Surveys have higher unacceptable rates on proficiency tests. Individual laboratories who were rated linear and whose calibration was verified by Linearity Surveys have lower unacceptable rates.

Reference values and participant means as targets in proficiency testing.

Regulatory agencies will be placing greater reliance on the results of proficiency testing (PT) for determining licensing status of individual laboratories. This has caused a renewed interest in the process used by proficiency test providers for determining the correct result for each proficiency test challenge. Three general techniques have been proposed and would be allowed under the regulations: (1) participant-derived targets (means from a specific group of participants); (2) targets from a set of "reference" laboratories, from either a subset of participants or a selected group of experts using a rigorous experimental protocol; or (3) definitive values from a single source. The College of American Pathologists (CAP) has used all three techniques at different stages and for different purposes in the history of the CAP Surveys programs. This article reviews the early experiences of the CAP, the evolution of the current protocol, and activities to monitor the accuracy of target values. Participant-derived "consensus" results have been shown to be highly accurate and reliable. The CAP uses definitive methods at the National Institute for Standards and Technology and certified reference materials to monitor the validity of targets used in proficiency testing.

Alternative statistical techniques to evaluate linearity.

With the increase in demand for linearity assessment, a wide variety of analysis techniques have been advocated. There is no consensus on optimal techniques. This article reviews different approaches that have been advocated by the College of American Pathologists, by the National Committee for Clinical Laboratory Standards, and by manufacturers of diagnostic methods and controls. This is not a review of all statistical techniques proposed for linearity assessment. I discuss four basic categories of methods, with the first and simplest being visual review. The second general approach includes the conventional statistical techniques based on least-squares regression; this includes an equation for the line and statistical tests for linearity and for curvature. The third general approach involves comparison of slopes for line segments, or "deltas." This approximates the visual assessment of linearity. The fourth approach involves comparing observed and expected values, with some allowance for differences. Comparisons can be made for observed single results, means of replicates, or components of variance. Allowances for error can be internal goals or goals that are recommended for clinical usefulness. Each approach has strengths in different aspects of linearity assessment, including intuitive appeal, statistical rigor, and objectivity. Similarly, each approach has limitations that are not always obvious.

Proficiency testing of enzymes. Charting the way toward standardization.

Laboratories participating in the College of American Pathologists Enzyme Survey (ES) and Comprehensive Chemistry Survey used diverse methods for the same analyte, resulting in a considerable range of values for the commonly performed enzyme measurements. Nevertheless, with the techniques developed for the ES, both the short-term (within-mailing) and long-term (between-mailings) coefficients of variation (CVs) can be determined. The ten-year experience of the ES has shown improvement in the short-term CV; however, long-term stability of enzyme testing requires more effort on the part of the instrument and reagent suppliers and participating laboratories. A reference material with an International Federation of Clinical Chemistry-established aspartate aminotransferase value, National Bureau of Standards RM 8430, is now available and was sent to three large peer groups as part of a special study. Correction of the results to the RM 8430 aspartate aminotransferase value resulted in reducing the range of data from peers using the duPont aca but not from those using the American Monitor KDA or Technicon SMAC. Based on our experience with the ES, goals of 5% for the short-term CV and 10% for the long-term CV are proposed; they are achievable by most laboratories and meet medical needs for biochemical screening. Fixed criteria for the evaluation of enzyme results appear to be appropriate given the way most enzyme data are used clinically.

A nonparametric analysis method for evaluation of survey results. Application to ligand assay survey.

A nonparametric analysis method that does not depend on gaussian data distributions is proposed for evaluation of proficiency survey results. The proposed method also provides a mechanism for processing results qualitatively reported as "less than" or "greater than" pre-established limits of quantitation. The evaluation limits for acceptability are calculated to include the central 95% of all test results from all methods. Systematic differences in the levels of analyte concentrations caused by differences in analytical measurement systems are normalized by mathematically transforming the test results from each analytic method by dividing them by their method-specific group median. This nonparametric method was compared with the traditional mean +/- 2 SD limits using data collected for digoxin, free thyroxine, and quantitative chorionic gonadotropin measurement in the College of American Pathologists Ligand Assay Survey. The nonparametric method more accurately classified 5% of the results as "unacceptable." When more than 2.5% of the test results for a method were designated as "less than," no lower limit was used for evaluation and only 2.5% of results were classified as "unacceptably high." The current College of American Pathologists procedure favors analytical methods with larger coefficients of variation by setting wider limits of acceptability, while the proposed procedure favors methods with smaller coefficients of variation.

Multiprogram characterization of laboratory bias, precision, and total error. Proposal for improved assessment with shared external and expanded internal (regional) quality control pools.

We establish that, for the analytes aspartate aminotransferase, glucose, phosphorus, and potassium, there is significant correlation between laboratory performance as determined by College of American Pathologists-sponsored external (Surveys) and expanded internal (regional) quality control (Quality Assurance Service) programs. However, relatively low parametric and nonparametric correlation coefficients and significant departure of linear regression slopes from unity reflect major differences in the calculated parameters of absolute bias, precision, and total error obtained through internal and external quality control. Significantly better performance in both Surveys and the Quality Assurance Service was documented for laboratories participating in the College of American Pathologists Laboratory Accreditation Program. Multiple descriptors of laboratory quality, as indicated here, are superior in describing laboratory performance to proficiency testing alone. As a bridge between external and internal quality control, shared pools of quality control materials are described for use as unknown Survey challenges and subsequent distribution for regional quality control. Such programs, which can be configured to serve thousands of laboratories, could offer cost savings, better quality assurance, and improved characterization of laboratory performance by minimizing interprogram differences in control matrix and method classification and providing greater reliability of target values.

A statistical procedure for measuring and evaluating performance in interlaboratory comparison programs.

There are scientific and regulatory needs to measure individual laboratory performance on a series of challenges, for single analytes and for all analytes in a particular discipline. Because these needs must be met with a very limited amount of information, optimal statistics should be used to measure performance. Since punitive action could result from poor performance, there should be precise quantitative goals that can be measured directly with the performance statistic. Finally, it is important to limit the likelihood of falsely penalizing a laboratory, since the large majority of laboratories are not in need of regulatory action. A statistic is described that measures individual performance on quantitative interlaboratory proficiency tests. This statistic is based on actual error relative to the amount of error that is tolerable. It can be accumulated over several challenges (specimens tested as unknowns) to give an estimate of a participant's performance level for that analyte. It can also be accumulated across analytes to give scores for Survey mailings or for accumulated performance. Because the statistic measures error, it contains more information than does the percentage of acceptable results, and therefore has greater power to detect poor performance. The distribution of the statistic is described and tested for validity. Then, a procedure is presented to evaluate laboratories relative to a performance goal. The entire procedure is then tested with recent College of American Pathologists Chemistry Survey data.

Determination of reference ranges for serum enzymes via a large interlaboratory survey.

We obtained enzyme data on normal individuals in conjunction with a large interlaboratory enzyme survey. For the 12 largest peer groups using unique methods, we found a simple relationship between the upper reference limits and the laboratories' results obtained from human-enzyme-supplemented survey serum. A conversion to a common base made possible the merging of data on the normal individuals and interconversion of results by diverse methods. We determined the upper reference limits for serum lactate dehydrogenase, alkaline phosphatase, aspartate aminotransferase, alanine aminotransferase, and creatine kinase for approximately 8000 adult men and women believed to be in good health. Using a technique described here, we believe that the results can be transformed to user-specific units, and that the large data base can be applied to the many diverse enzyme methods in current use. With these data, enzyme survey participants will be able to estimate appropriate reference intervals for their particular method.

Survey of serum enzyme analyses. Human tissues as a source of enzymes.

The College of American Pathologists' enzyme survey permits the evaluation of the quality of enzyme analyses in clinical laboratories. Animal source enzymes for lactate dehydrogenase, aspartate aminotransferase, alanine aminotransferase, and creatine kinase are satisfactory as supplements for enzyme survey serum, since they give results equivalent to human source enzymes. This finding makes the interconversion of results by different methods possible, and the estimation of the upper limit of normal based on the results obtained for the enzyme survey serum.

The relationship of instrument parameters to performance within a survey peer group.

Using simulators of transmission imaging, an interlaboratory survey assessed the discriminatory performance of 86 subscribers, each of whom imaged a liver phantom in anterior and right lateral projections. Analysis was by receiver operating characteristic (ROC) with Az, the area under the ROC curve, used as a measure of accuracy unconfounded by decision bias. Az values were then defined as the dependent variable in a statistical model that related performance to several instrument design and operating parameters. Six of 14 postulated parameters explained approximately half of observed subscriber variability. These were: year of camera manufacture or upgrade, number of photomultiplier tubes, collimator type, total counts collected, use of a Co-57 disk source for imaging the phantom, and computer processing of the image. The findings confirm previous inferences drawn from controlled intralaboratory experimentation, but hitherto unsubstantiated by clinical imaging data.

Source of analytic variation in ligand assay: a study based on data from the 1976 and 1981 Basic Ligand Assay Surveys of the College of American Pathologists.

Six paired specimens distributed to laboratories in 1981 at approximately three-month intervals and four paired specimens distributed to laboratories in 1976 at intervals of three to nine months for analysis by the College of American Pathologists Survey form the basis for this study. Twelve of 37 (35%) of pool-analyte-technic combinations yielded significantly changed mean values in 1976, while 34 of 196 (17%) pool-analyte-technic combinations yielded significantly changed values in 1981. Probable instability in thyroxine and folate was demonstrated in the 1981 control pools. Precision generally continued to improve from 1976 to 1981. The poorest precision now is observed in the analysis of peptide hormones. A majority of the observed analytic variation during 1981 for most analytes relates to extralaboratory factors. Improvement in performance is largely dependent on intermanufacturer standardization of procedures and long-term maintenance of equivalence of the results of kit procedures by each manufacturer.