An investigation of new toxicity test method performance in validation studies: 1. Toxicity test methods that have predictive capacity no greater than chance.

An approach commonly used to measure new toxicity test method (NTM) performance in validation studies is to divide toxicity results into positive and negative classifications, and the identify true positive (TP), true negative (TN), false positive (FP) and false negative (FN) results. After this step is completed, the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are calculated. Although these statistics are widely used and often the only statistics used to assess the performance of toxicity test methods, there is little specific guidance in the validation literature on what values for these statistics indicate adequate performance. The purpose of this study was to begin developing data-based answers to this question by characterizing the CPS obtained from an NTM whose data have a completely random association with a reference test method (RTM). Determining the CPS of this worst-case scenario is useful because it provides a lower baseline from which the performance of an NTM can be judged in future validation studies. It also provides an indication of relationships in the CPS that help identify random or near-random relationships in the data. The results from this study of randomly associated tests show that the values obtained for the statistics vary significantly depending on the cut-offs chosen, that high values can be obtained for individual statistics, and that the different measures cannot be considered independently when evaluating the performance of an NTM. When the association between results of an NTM and RTM is random the sum of the complementary pairs of statistics (sensitivity + specificity, NPV + PPV) is approximately 1, and the prevalence (i.e., the proportion of toxic chemicals in the population of chemicals) and PPV are equal. Given that combinations of high sensitivity-low specificity or low specificity-high sensitivity (i.e., the sum of the sensitivity and specificity equal to approximately 1) indicate lack of predictive capacity, an NTM having these performance characteristics should be considered no better for predicting toxicity than by chance alone.

An investigation of new toxicity test method performance in validation studies: 2. Comparison of three measures of toxicity test performance.

An area that requires further research is how best to measure test method performance in validation studies and how to set criteria that should be used to judge the adequacy of this performance. The studies reported here were designed to begin an investigation of these questions. Computer simulations were used to generate data sets similar to those that might be obtained from a large validation study. These data were then analysed using three procedures including determination of the 95% prediction interval (PI), calculation of Pearson's correlation coefficient and calculation of the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The results of this work suggest that of the three approaches examined, quantitative measurements with calculation of the 95% PI provide the most information to allow discrimination between the performance of several different NTMs. The results also suggest that dividing data sets into positive and negative toxicity classifications followed by the calculation of CPS leads to considerable information loss. This loss of information may be so significant that it is not possible in certain circumstances to distinguish between NTMs that are adequate and those that are not.

An investigation of new toxicity test method performance in validation studies: 3. Sensitivity and specificity are not independent of prevalence or distribution of toxicity.

Often, the only measures of toxicity test performance provided in validation studies are the contingent probability statistics (CPS) sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Sensitivity and specificity are generally used in preference to NPV and PPV since NPV and PPV are assumed to vary with changes in prevalence while sensitivity and specificity are assumed to be independent of changes in prevalence. The purpose of the studies reported here was to test whether or not sensitivity and specificity are actually independent of changes in prevalence. Results derived from these studies indicate that sensitivity and specificity vary significantly depending on the prevalence of toxic substances in the set of chemicals being tested. This means sensitivity and specificity should not always be considered constant indicators of toxicity test performance.

Prediction of eye irritation from organic chemicals using membrane-interaction QSAR analysis.

Eye irritation potency of a compound or mixture has traditionally been evaluated using the Draize rabbit-eye test (Draize et al., 1944). In order to aid predictions of eye irritation and to explore possible corresponding mechanisms of eye irritation, a methodology termed "membrane-interaction QSAR analysis" (MI-QSAR) has been developed (Kulkarni and Hopfinger 1999). A set of Draize eye-irritation data established by the European Center for Ecotoxicology and Toxicology of Chemicals (ECETOC) (Bagley et al., 1992) was used as a structurally diverse training set in an MI-QSAR analysis. Significant QSAR models were constructed based primarily upon aqueous solvation-free energy of the solute and the strength of solute binding to a model phospholipid (DMPC) monolayer. The results demonstrate that inclusion of parameters to model membrane interactions of potentially irritating chemicals provides significantly better predictions of eye irritation for structurally diverse compounds than does modeling based solely on physiochemical properties of chemicals. The specific MI-QSAR models reported here are, in fact, close to the upper limit in both significance and robustness that can be expected for the variability inherent to the eye-irritation scores of the ECETOC training set. The MI-QSAR models can be used with high reliability to classify compounds of low- and high-predicted eye irritation scores. Thus, the models offer the opportunity to reduce animal testing for compounds predicted to fall into these two extreme eye-irritation score sets. The MI-QSAR paradigm may also be applicable to other toxicological endpoints, such as skin irritation, where interactions with cellular membranes are likely.

Eye Irritation Testing: The Way Forward. The Report and Recommendations of ECVAM Workshop 34.

This is the report of the thirty-fourth of a series of workshops organised by the European Centre for the Validation of Alternative Methods (ECVAM). ECVAM's main goal, as defined in 1993 by its Scientific Advisory Committee, is to promote the scientific and regulatory acceptance of alternative methods which are of importance to the biosciences and which reduce, refine or replace the use of laboratory animals. One of the first priorities set by ECVAM was the implementation of procedures which would enable it to become well-informed about the state-of-the-art of non-animal test development and validation, and the potential for the possible incorporation of alternative tests into regulatory procedures. It was decided that this would be best achieved by the organisation of ECVAM workshops on specific topics, at which small groups of invited experts would review the current status of various types of in vitro tests and their potential uses, and make recommendations about the best ways forward (1). The workshop on Eye Irritation Testing: The Way Forward was held in Egham, UK, on 15-17 June 1998, under the chairmanship of Michael Balls (ECVAM, Italy). The workshop had two aims, the first of which was to review some of the previous multi-laboratory validation studies on alternatives to the Draize eye test and assess why many promising alternative methods were not successful in these studies. The second aim was to discuss strategies for making progress toward the short-term reduction, refinement, and eventual replacement, of the Draize test, including: a new approach to the validation of in vitro tests for eye irritancy, based on the use of reference standards, which promises to overcome some of the problems encountered in previous studies; the use of stepwise testing strategies which reduce and refine the use of animals in eye irritation testing; the use of multivariate and other statistical techniques for the further analysis of data generated in previous validation studies; and a programme of research aimed at understanding the underlying mechanisms of eye irritation.

Workshop overview: scientific and regulatory challenges for the reduction, refinement, and replacement of animals in toxicity testing.

Public concern for animal welfare has been expressed through legislative control of animal use for experimental purposes since the first legislation was introduced in 1876 in the United Kingdom. Legislative control of animal use has been introduced in virtually every developed country, with major initiatives in Europe (1986) and the United States (1966 and 1985). Advances in scientific thinking resulted in the development of the concept of the three Rs--refinement, reduction, and replacement--by Russell and Burch in 1959. The field has expanded substantially since, with specialist scientific journals dedicated to alternatives, World Congresses organized to discuss the scientific and philosophical issues, and European and U.S. validation organizations being launched. Current scientific attention is focused on validation of alternative methods. The underlying scientific principles of chemical toxicity are complicated and insufficiently understood for alternative methods for all toxicity endpoints of importance in protecting human health to be available. Important lessons have been learned about how to validate methods, including the need to have prediction models available before the validation is undertaken, the need to understand the variability of the animal-based data which is to be used as the validation standard, and the need to have well-managed validation programs. Future progress will depend on the development of novel methods, which can now be validated through international collaborative efforts.

Validation of alternative methods for toxicity testing.

Before nonanimal toxicity tests may be officially accepted by regulatory agencies, it is generally agreed that the validity of the new methods must be demonstrated in an independent, scientifically sound validation program. Validation has been defined as the demonstration of the reliability and relevance of a test method for a particular purpose. This paper provides a brief review of the development of the theoretical aspects of the validation process and updates current thinking about objectively testing the performance of an alternative method in a validation study. Validation of alternative methods for eye irritation testing is a specific example illustrating important concepts. Although discussion focuses on the validation of alternative methods intended to replace current in vivo toxicity tests, the procedures can be used to assess the performance of alternative methods intended for other uses.

Investigation of ingredient interactions in cosmetic formulations using isolated bovine corneas.

The purpose of this paper is to report on use of a modified bovine cornea opacity and permeability assay (BCOP) to test the effects of several cosmetic formulations on eye-derived tissue in vitro. The results from these studies suggest that a BCOP protocol using prolonged exposure and repeated treatments may be useful for screening the eye effects of cosmetic formulations. Further work will be required, however, before the model is ready for formal validation. This series of experiments also provides an example of where the toxicity of one ingredient was significantly changed by its interaction with other ingredients in a mixture. As it was not possible to predict the highly reactive nature of the formulation in vitro based on an evaluation of ingredient toxicity data alone, this case illustrates the importance of obtaining adequate safety testing data on innovative mixtures of cosmetic ingredients before human exposure is allowed.

Report on the COLIPA Workshop on Mechanisms of Eye Irritation.

This report summarises the discussions of a workshop sponsored by the European Cosmetics, Toiletries and Perfumery Association (COLIPA). The workshop discussed the state-of-the-art of eye irritancy testing, and made recommendations as to the best ways in which to validate alternatives to the Draize eye irritation test. The importance of understanding the mechanisms of eye irritation, particularly when attempting to improve in vitro prediction of in vivo eye irritancy, was also emphasised.

Perspectives on alternatives to the eye irritation test: industry, public interest, government.

Government mandates are requiring serious consideration of alternatives to animal testing. For eye irritation testing, many non-whole animal alternatives exist that now need to be assessed as to their validity in replacing the animal model. The best promise for identifying useful alternatives comes from using both statistical and biological factors to evaluate results from formal validation studies. Industry submissions of side-by-side animal and alternative test results are also important. Empirical test results should be scrutinized first; mechanistic studies should follow, as needed. Co-operation is required by all parties to develop internationally harmonized test protocols and hazard classification systems.

A summary report of the COLIPA international validation study on alternatives to the draize rabbit eye irritation test.

The principal goal of this study was to determine whether the results from a set of selected currently available alternative methods as used by cosmetics companies are valid for predicting the eye irritation potential of cosmetics formulations and ingredients and, as a consequence, could be valid replacements for the Draize eye irritation test. For the first time in a validation study, prediction models (PMs) that convert the in vitro data from an assay to a prediction of eye irritation were developed for each alternative method before the study began. The PM is an unequivocal description of the relationship between the in vitro and the in vivo data and allows an objective assessment of the reliability and relevance of the alternative methods. In this study, 10 alternative methods were evaluated using 55 test substances selected as representative of substances commonly used in the cosmetics industry (23 ingredients and 32 formulations). Twenty of the single ingredients were common to the European Commission/British Home Office (EC/HO) eye irritation validation study (Balls et al., 1995b). The test substances were coded and supplied to the participating laboratories. The results were collected centrally and analysed independently, using statistical methods that had been agreed before the testing phase began. Each alternative method was then evaluated for reliability and relevance in assessing eye irritation potential. Using the criteria of both reliability and relevance as defined in the study, the preliminary results indicate that none of the alternative methods evaluated could be confirmed as a valid replacement for the Draize eye irritation test across the full irritation scale. However, three alternative methods-the fluorescein leakage test, the red blood cell assay (classification model) and the tissue equivalent assay-each satisfied one criterion of reliability or relevance. Further investigation of the decoded data from this study to explore more fully the relationship between the in vitro data and the in vivo data is recommended. Such a review may allow the development of new prediction models to be tested in a subsequent validation study.

Validation of alternative methods for toxicity testing.

Many studies have been conducted in order to assess the validity of alternative methods as replacements forin vivo toxicity tests. The purpose of this review is to build on what has been learned in the course of this work by presenting a practical process that can be used to conduct future validation programmes. The important role of a clearly stated prediction model, which defines how to use the results from an alternative method to predict anin vivo toxicity endpoint, has been introduced. Computer simulations have been used to demonstrate that data-based guidance can be developed to assist in judging the performance of alternative methods assessed in a validation study. Additionally, statistical procedures have been used in order to provide guidance on choosing the appropriate number of reference test substances and number of participating laboratories to include in a validation study. The validation of alternative methods for eye irritation testing is used as a specific example to illustrate important concepts. Although the focus of the discussion is on the validation of alternative methods intended to replace currentin vivo tests, the procedures can be used to assess the performance of alternative methods intended for other uses. This review will be particularly useful to those who require a practical guide for conducting a validation study and to those who must assess the results of such programmes.

Alternatives in cosmetics testing.

This paper represents a summary of presentations made during a round-table discussion at the ECVAM Opening Symposium. After introductory comments on the cosmetic industry's use of alternative methods in the safety assessment process, the use of alternative methods by L'Oréal and by the Japanese cosmetic industry is outlined, current validation studies in Japan are noted, and the involvement of COLIPA, the European Cosmetic, Toiletry and Perfumery Association, in promoting the use of alternative methods is discussed. Two final sections deal with the effect of data variability on the performance of alternative methods in validation studies and on the integrated use of quantitative structure-activity relationship (QSAR) analysis with other approaches in the safety assessment process.

The EC/HO international validation study on alternatives to the draize eye irritation test.

This is the final report of the Management Team for a European Commission/British Home Office (EC/HO) validation study on alternatives to the Draize eye irritation test. The principal goal of the study was to establish whether one or more of nine non-animal tests could be used to replace the Draize test for all severely irritating materials (or those belonging to specific classes) or the animal test completely for chemicals with or without regard to chemical class. Sixty chemicals were independently selected, coded and supplied, then the data obtained in 37 laboratories were analysed independently. The results of comparisons between 27 alternative test index scores and the Modified Maximum Average Scores (MMASs) obtained in the Draize eye test were compared. Tables of results showing Pearson's product moment correlation coefficients and Spearman's rank coefficients for each laboratory are provided, and correlation matrices of alternative test index scores among the different groups of laboratories are shown for each endpoint. Scatterplots are provided, in which the alternative test scores obtained by the lead laboratories for the nine tests are plotted against the MMAS for the full set of chemicals and 12 surfactants. It is concluded that, with the possible exception of predicting the irritancy of surfactants, none of the nine tests met any of the four performance targets. Possible reasons for this outcome are discussed.

Reducing the number of rabbits in the low-volume eye test.

Although the Draize eye irritation test has provided important and useful information for eye safety assessments, considerable effort has been directed toward refining the assay procedure, reducing the number of animals used, and replacing this assay with alternative methods. The low-volume eye test (LVET) is a refinement of the Draize eye irritation test that uses 1/10 the volume of test substance placed directly on the cornea. The level and duration of eye irritation in the LVET are less than those in the Draize procedure, which means that it is a less stressful test. Furthermore, LVETs are more predictive of human response. Statistical studies have been conducted to determine the effects of reducing the number of animals used in the Draize test. These results suggested that a three-animal test would provide essentially the same information as the six-animal test. A similar analysis has not been performed on results from the LVET. Accordingly, the present study was undertaken to evaluate previously existing LVET data to determine if the number of animals used in a LVET can be decreased as has been shown for the Draize test. The results of the analysis are consistent with the findings of earlier evaluations of classical Draize data. Three-animal subsets from 119 six-animal LVETs provided the correct classification greater than 92% of the time for three different classification schemes. Furthermore, the discrepancies between the three-animal subsets and the six-animal maximum average score tended to be smaller than those observed for the Draize test.(ABSTRACT TRUNCATED AT 250 WORDS)

An evaluation of five potential alternatives in vitro to the rabbit eye irritation test in vivo.

Five alternative techniques, each of which had been successfully used by one of the participating companies, were evaluated in the assessment of the eye-irritation potential of 32 samples. The 32 samples included chemical ingredients and preparations from household cleaning product, personal care, and cosmetic categories. Historical data from rabbit eye irritation tests in vivo existed for each sample; it was therefore not necessary to carry out any tests in vivo as part of this evaluation exercise. The five alternative methods used were the silicon microphysiometer test, the Microtox test, the neutral red uptake assay, the chorioallantoic membrane vascular assay (CAMVA) and the hen egg test-chorioallantoic membrane assay (HETCAM). Three of the assays were conducted in two laboratories, allowing an interlaboratory comparison of performance to be made. The CAMVA assay was carried out on 10-day-old as well as on 14-day-old fertile eggs. Correlations between the data sets in vivo and in vitro were determined for the five assays. The results demonstrated that for the materials tested, all of the assays show some promise as alternative methods to the rabbit eye test in vivo in the prediction of eye irritation, and that the reproducibility of results of those techniques carried out in two laboratories was very good. The results from 14-day and 10-day CAMVA assays were virtually identical. It is recommended that a larger-scale validation exercise be carried out to demonstrate the ultimate usefulness of these alternative procedures in the safety evaluation process.

Evaluation of seven in vitro alternatives for ocular safety testing.

Seven in vitro assays were evaluated to determine if any were useful as screening procedures in ocular safety assessment. Seventeen test materials (chemicals, household cleaners, hand soaps, dishwashing liquids, shampoos, and liquid laundry detergents) were tested in each assay. In vivo ocular irritation scores for the materials were obtained from existing rabbit low volume eye test (LVET) data. The seven assays evaluated included the silicon microphysiometer (SM), luminescent bacteria toxicity test (LBT), neutral red assay (NR), total protein assay (TP), Tetrahymena thermophila motility assay (TTMA), bovine eye/chorioallantoic membrane assay (BE/CAM), and the EYTEX system (ETS). For the seventeen materials used in this study there was a significant correlation between the in vivo irritant potential and in vitro data for all the tests except the EYTEX System (SM, r = -0.87; LBT, r = -0.91; NR, r = -0.85; TTMA, r = 0.78; TP, r = -0.86; ETS, r = 0.29). The irritation classifications provided by the BE/CAM also did not correspond with the actual in vivo irritancy potential of the test materials. The result of this study suggested it may be possible to classify materials into broad irritancy categories with some of the assays. This would allow their use as screens prior to limited in vivo confirmation in the ocular safety assessment process.

Testing ocular irritancy in vitro with the silicon microphysiometer.

The silicon microphysiometer, an instrument based on the light-addressable potentiometric sensor, was evaluated as an in vitro alternative for assessing ocular irritancy potential. It indirectly and non-invasively measures cell metabolism by determining the rate of acid metabolite production from cells, in this case human epidermal keratinocytes, placed inside the microphysiometer chamber. The 17 materials used for the evaluation included bar soaps, a liquid hand soap, shampoos, dishwashing liquids, laundry detergents, a fabric softener and several single chemicals. All materials tested were in liquid form. The in vivo irritancy potential of the materials was obtained from historical data using the rabbit low-volume eye test. There was a positive correlation between the in vivo irritancy potential of the test materials and the concentration of test material that decreased the acidification rate of cells by 50% (MRD(50); r = 0.86, P < 0.0001). Preliminary studies suggest other endpoints obtainable from the system may also provide useful information for making ocular safety assessments. Because the method is non-invasive, it is possible to determine whether cells recover from a treatment with the test material. The metabolic rate of the cells also increases at sub-inhibitory concentrations of some of the test materials. Because of the good correlation between the in vivo and in vitro data, the ease with which test materials can be applied to the system, and the multiple endpoints available from the system, it holds great potential as a useful in vitro alternative for ocular safety testing.

Measurement of intracellular fluorescence of human monocytes relative to oxidative metabolism.

Human monocytes (MN) produce O2- and H2O2 when stimulated by agonists. Dichlorofluorescin diacetate (DCFH-DA) has been used as a substrate for measuring intracellular oxidant production in neutrophils. DCFH-DA is hydrolyzed by esterases to dichlorofluorescin (DCFH), which is trapped within the cell. This nonfluorescent molecule is then oxidized to fluorescent dichlorofluorescin (DCF) by action of cellular oxidants. DCFH-DA can not be appreciably oxidized to a fluorescent state without prior hydrolysis. We have examined the utility of DCFH-DA for the assessment of monocyte oxidative responses. The levels of intracellular fluorescence measured by flow cytometry were considerably less than expected from reported levels of O2--production or chemiluminescence assays. Compared with neutrophils, monocytes produced minimal increases in DCF fluorescence after stimulation with phorbol myristate acetate as measured by flow cytometry, but both cell types showed increases in fluorescence when bulk cell suspensions were measured by spectrofluorometry. To determine the intracellular location of the DCFH, bulk fluorescence measurements were made on both whole and sonicated cell preparations. When intact mononuclear cells were preloaded with DCFH-DA, then sonicated and oxidized with added excess H2O2, the increase in fluorescence was only 30% of the fluorescence of mononuclear cell sonicates to which DCFH-DA was added and oxidized in a similar manner. These results suggest that a portion of the DCFH-DA incorporated by intact cells, is not susceptible to oxidation by the added H2O2. Addition of NaOH to induce hydrolysis of any residual DCFH-DA in the sonicates of DCFH-DA-loaded intact mononuclear cells resulted in a further increase in fluorescence upon addition H2O2, suggesting that a significant portion of the DCFH-DA was not hydrolyzed despite ample uptake of this dye by these cells. In contrast, no further increase in fluorescence was observed in sonicates of DCFH-DA-loaded intact neutrophils, suggesting complete hydrolysis of all incorporated DCFH-DA to DCFH. When monocytes were allowed to phagocytose DCFH-DA-loaded Staphylococcus aureus, intracellular fluorescence was measurable by flow cytometry, indicating intracellular oxidation of the fluorochromes. We therefore propose that in monocytes the mechanism of intracellular processing of these fluorochromes differs from that in neutrophils owing to differences in intracellular localization of fluorochromes, site of oxidant production, and/or accessibility of the DCFH-DA to esterolysis.