Avoiding a reproducibility crisis in regulatory toxicology-on the fundamental role of ring trials.

The ongoing transition from chemical hazard and risk assessment based on animal studies to assessment relying mostly on non-animal data, requires a multitude of novel experimental methods, and this means that guidance on the validation and standardisation of test methods intended for international applicability and acceptance, needs to be updated. These so-called new approach methodologies (NAMs) must be applicable to the chemical regulatory domain and provide reliable data which are relevant to hazard and risk assessment. Confidence in and use of NAMs will depend on their reliability and relevance, and both are thoroughly assessed by validation. Validation is, however, a time- and resource-demanding process. As updates on validation guidance are conducted, the valuable components must be kept: Reliable data are and will remain fundamental. In 2016, the scientific community was made aware of the general crisis in scientific reproducibility-validated methods must not fall into this. In this commentary, we emphasize the central importance of ring trials in the validation of experimental methods. Ring trials are sometimes considered to be a major hold-up with little value added to the validation. Here, we clarify that ring trials are indispensable to demonstrate the robustness and reproducibility of a new method. Further, that methods do fail in method transfer and ring trials due to different stumbling blocks, but these provide learnings to ensure the robustness of new methods. At the same time, we identify what it would take to perform ring trials more efficiently, and how ring trials fit into the much-needed update to the guidance on the validation of NAMs.

Report of the First ONTOX Stakeholder Network Meeting: Digging Under the Surface of ONTOX Together With the Stakeholders.

The first Stakeholder Network Meeting of the EU Horizon 2020-funded ONTOX project was held on 13-14 March 2023, in Brussels, Belgium. The discussion centred around identifying specific challenges, barriers and drivers in relation to the implementation of non-animal new approach methodologies (NAMs) and probabilistic risk assessment (PRA), in order to help address the issues and rank them according to their associated level of difficulty. ONTOX aims to advance the assessment of chemical risk to humans, without the use of animal testing, by developing non-animal NAMs and PRA in line with 21st century toxicity testing principles. Stakeholder groups (regulatory authorities, companies, academia, non-governmental organisations) were identified and invited to participate in a meeting and a survey, by which their current position in relation to the implementation of NAMs and PRA was ascertained, as well as specific challenges and drivers highlighted. The survey analysis revealed areas of agreement and disagreement among stakeholders on topics such as capacity building, sustainability, regulatory acceptance, validation of adverse outcome pathways, acceptance of artificial intelligence (AI) in risk assessment, and guaranteeing consumer safety. The stakeholder network meeting resulted in the identification of barriers, drivers and specific challenges that need to be addressed. Breakout groups discussed topics such as hazard versus risk assessment, future reliance on AI and machine learning, regulatory requirements for industry and sustainability of the ONTOX Hub platform. The outputs from these discussions provided insights for overcoming barriers and leveraging drivers for implementing NAMs and PRA. It was concluded that there is a continued need for stakeholder engagement, including the organisation of a 'hackathon' to tackle challenges, to ensure the successful implementation of NAMs and PRA in chemical risk assessment.

Retrospective evaluation of the eye irritation potential of agrochemical formulations.

Multiple in vitro eye irritation methods have been developed and adopted as OECD health effects test guidelines. However, for predicting the ocular irritation/damage potential of agrochemical formulations there is an applicability domain knowledge gap for most of the methods. To overcome this gap, a retrospective evaluation of 192 agrochemical formulations with in vivo (OECD TG 405) and in vitro (OECD TG 437, 438, and/or 492) data was conducted to determine if the in vitro methods could accurately assign United Nations Globally Harmonized System for Classification and Labelling of Chemicals (GHS) eye irritation hazard classifications. In addition, for each formulation the eye irritation classification was derived from the classification of the contained hazardous ingredients and their respective concentration in the product using the GHS concentration threshold (CT) approach. The results herein suggest that the three in vitro methods and the GHS CT approach were highly predictive of formulations that would not require GHS classification for eye irritation. Given most agrochemical formulations fall into this category, methods that accurately identify non-classified agrochemical formulations could significantly reduce the use of animals for this endpoint.

Critical assessment of the endocrine potential of Linalool and Linalyl acetate: proactive testing strategy assessing estrogenic and androgenic activity of Lavender oil main components.

The acyclic linear monoterpenes Linalool (Lin) and Linalyl acetate (LinAc) occur in nature as major constituents of various essential oils such as lavender oils. A potential endocrine activity of these compounds was discussed in literature including premature thelarche and prepubertal gynecomastia due to lavender product use. This study aims to follow-up on these critical findings reported by testing Lin and LinAc in several studies in line with current guidance and regulatory framework. No relevant anti-/ER and AR-mediated activity was observed in recombinant yeast cell-based screening tests and guideline reporter gene in vitro assays in mammalian cells. Findings in the screening test suggested an anti-androgenic activity, which could not be confirmed in the respective mammalian cell guideline assay. Mechanistic guideline in vivo studies (Uterotrophic and Hershberger assays) with Lin did not show significant dose related changes in estrogen or androgen sensitive organ weights and a guideline reproductive toxicity screening study did not reveal evident effects on sex steroid hormone sensitive organ weights, associated histopathological findings and altered sperm parameters. Estrous cycling and mating/fertility indices were not affected and no evident Lin-related steroid hormone dependent effects were found in the offspring. Overall, the initial concerns from literature were not confirmed. Findings in the yeast screening test were aberrant from follow-up guideline in vitro and in vivo studies, which underlines the need to apply careful interpretation of single in vitro test results to support a respective line of evidence and to establish a biologically plausible link to an adverse outcome.

Plant extracts, polymers and new approach methods: Practical experience with skin sensitization assessment.

Over the last decade, research into methodologies to identify skin sensitization hazards has led to the adoption of several non-animal methods as OECD test guidelines. However, predictive accuracy beyond the chemical domains of the individual validation studies remains largely untested. In the present study, skin sensitization test results from in vitro and in chemico methods for 12 plant extracts and 15 polymeric materials are reported and compared to available in vivo skin sensitization data. Eight plant extracts were tested in the DPRA and h-CLAT, with the 2 out of 3 approach resulting in a balanced accuracy of 50%. The balanced accuracy for the 11 plant extracts assessed in the SENS-IS was 88%. Excluding 5 polymers inconclusive in vitro, the remainder, assessed using the 2 out of 3 approach, resulted in 63% balanced accuracy. The SENS-IS method, excluding one polymeric material due to technical inapplicability, showed 68% balanced accuracy. Although based on limited numbers, the results presented here indicate that some substance subgroups may not be in the applicability domains of the method used and careful analysis is required before positive or negative results can be accepted.

Accounting for Precision Uncertainty of Toxicity Testing: Methods to Define Borderline Ranges and Implications for Hazard Assessment of Chemicals.

For hazard classifications of chemicals, continuous data from animal- or nonanimal testing methods are often dichotomized into binary positive/negative outcomes by defining classification thresholds (CT). Experimental data are, however, subject to biological and technical variability. Each test method's precision is limited resulting in uncertainty of the positive/negative outcome if the experimental result is close to the CT. Borderline ranges (BR) around the CT were suggested, which represent ranges in which the study result is ambiguous, that is, positive or negative results are equally likely. The BR reflects a method's precision uncertainty. This article explores and compares different approaches to quantify the BR. Besides using the pooled standard deviation, we determine the BR by means of the median absolute deviation (MAD), with a sequential combination of both methods, and by using nonparametric bootstrapping. Furthermore, we quantify the BR for different hazardous effects, including nonanimal tests for skin corrosion, eye irritation, skin irritation, and skin sensitization as well as for an animal test on skin sensitization (the local lymph node assay, LLNA). Additionally, for one method (direct peptide reactivity assay) the BR was determined experimentally and compared to calculated BRs. Our results demonstrate that (i) the precision of the methods is determining the size of their BRs, (ii) there is no "perfect" method to derive a BR, alas, (iii) a consensus on BR is needed to account for the limited precision of testing methods.

A triangular approach for the validation of new approach methods for skin sensitization.

The availability of reference data is a key requirement for the development of new approach methods (NAM), i.e., in vitro, in chemico and in silico methods and integrated approaches, like defined approaches (DA), which combine these data sources. Reference data are of even greater importance for regulatory acceptance. In contrast to most other adverse effects, human skin sensitization data on many chemicals are available, next to data from animal studies, such as the local lymph node assay (LLNA). Skin sensitization NAM data can therefore be compared to different reference datasets. Recent publications and validation at the OECD focused on human and LLNA reference data. The "2 out of 3" DA (2o3 DA) is the first DA for skin sensitization solely based on experimental data from validated tests and was recently adopted as an OECD test guideline. Here we review the predictivity of the 2o3 DA on multiple human and LLNA reference datasets. Concomitantly, we compare the predictivity of the LLNA for human data within the same datasets. Comparing predictivity of methods not only bilaterally (NAM or DA vs. animal method) but including human data in a triangle "NAM data ­ animal data ­ human data" offers a comprehensive assessment of the NAM's and DA's predictivity. In all these assessments, the 2o3 DA was superior to the LLNA in predicting human skin sensitization hazard. This highlights the importance of a holistic view of reference data instead of limiting validation of NAMs and DAs to data from a single animal test only.

Human-Derived In Vitro Models Used for Skin Toxicity Testing Under REACh.

In regulatory toxicology, in vivo studies are still prevailing, and human-derived in vitro models are mostly used in testing for local toxicity to the skin and the eyes. A single in vitro model may be limited to address one or few molecular or cellular events leading to adverse outcomes. Hence, in many instances their regulatory use involves the combination of several in vitro models to assess the hazard potential of test substance. A so-called defined approach combines different testing methods and a 'data interpretation procedure' to obtain a comprehensive overall assessment which is used for the regulatory hazard classification of the test substance.Validation is a prerequisite of regulatory acceptance of new testing methods: This chapter provides an overview of the method development from an experimental method to a test guideline via application of GIVIMP (good in vitro method practice), standardization, validation to the regulatory adoption as an OECD test guidelines. Quandaries associated with the validation towards reference data from in vivo animal studies with limited accuracy and limited human relevance are discussed, as well as uncertainty and limitations arising from restricted applicability and technical and biological variance of the in vitro methods.This chapter provides an overview of human-derived in vitro models currently adopted as OECD test guidelines: From the first skin corrosion tests utilizing reconstructed human epidermis models (RhE), to models to test for skin irritation, phototoxicity, eye irritation, and skin sensitization. The latter is using a battery of different methods and defined approaches which are still under discussion for their regulatory adoption. They will be a vanguard of future applications of human-derived models in regulatory toxicology. RhEs for testing of genotoxicity and of dermal penetration and absorption, have been developed, underwent validation studies and may soon be adopted for regulatory use; these are included in this chapter.

Predictivity of the kinetic direct peptide reactivity assay (kDPRA) for sensitizer potency assessment and GHS subclassification

Several in vitro OECD test guidelines address key events 1-3 of the adverse outcome pathway for skin sensitization, but none are validated for sensitizer potency assessment. The reaction of sensitizing molecules with skin proteins is the molecular initiating event and appears to be rate-limiting, as chemical reactivity strongly correlates with sensitizer potency. The kinetic direct peptide reactivity assay (kDPRA), a modification of the DPRA (OECD TG 442C), allows derivation of rate constants of the depletion of the cysteine-containing model peptide upon reaction with the test item. Its reproducibility was demonstrated in an inter-laboratory study. Here, we present a database of rate constants, expressed as log kmax, for 180 chemicals to define the prediction threshold to identify strong sensitizers (classified as GHS 1A). A threshold of log kmax -2 offers a balanced accuracy of 85% for predicting GHS 1A sensitizers according to the local lymph node assay. The kDPRA is proposed as a stand-alone assay for identification of GHS 1A sensitizers among chemicals identified as sensitizers by other tests or defined approaches. It may also be used for the prediction of sensitizer potency on a continuous scale, ideally in combination with continuous parameters from other in vitro assays. We show how the rate constant could be combined with read-outs of other in vitro assays in a defined approach. A decision model based on log kmax alone has, however, a high predictivity and can be used as stand-alone model for identification of GHS 1A sensitizers among chemicals predicted as sensitizers.

Replacing the refinement for skin sensitization testing: Considerations to the implementation of adverse outcome pathway (AOP)-based defined approaches (DA) in OECD guidelines.

While single non-animal methods have been adopted in OECD test guidelines, combinations of methods (so called defined approaches, DA) are not. Hardly any animal study can be replaced by a single non-animal method, rather DA are needed. The OECD published the Adverse Outcome Pathway (AOP) on skin sensitization in 2012 and is currently discussing the implementation of DA into a guideline. Obviously, it takes thorough considerations and evaluations to validate such DA. Currently we see four preconditions for a proper and expedient implementation of DA in a guideline: (i) The reference data should be selected to allow meaningful evaluations and must not replicate the limitations of the murine local lymph node assay (LLNA) (ii) Methods and prediction models should be validated before they are used in an OECD-adopted DA, (iii) An OECD-adopted DA should follow the respective AOP and (iv) acknowledge regulatory needs and successful toxicological practice. These points still need to be considered in the current discussion at the OECD. A guideline for skin sensitization DA is setting the scene for regulatory acceptance of all new approaches (for any toxicological endpoint) in the future. In this commentary, we are expounding these preconditions to allow a scientifically valid and sustainable application of modern (no-animal) approaches in regulatory toxicology.

The kinetic direct peptide reactivity assay (kDPRA): Intra- and inter-laboratory reproducibility in a seven-laboratory ring trial

While the skin sensitization hazard of substances can be identified using non-animal methods, the classification of potency into UN GHS sub-categories 1A and 1B remains challenging. The kinetic direct peptide reactivity assay (kDPRA) is a modification of the DPRA wherein the reaction kinetics of a test substance towards a synthetic cysteine-containing peptide are evaluated. For this purpose, several concentrations of the test substance are incubated with the synthetic peptide for several incubation times. The reaction is stopped by addition of monobromobimane, which forms a fluorescent complex with the free cysteine of the model peptide. The relative remaining non-depleted amount of peptide is determined. Kinetic rate constants are derived from the depletion vs concentration and time matrix and used to distinguish between UN GHS sub-category 1A sensitizers and test substances in sub-category 1B/not classified test substances. In this study, we present a ring trial of the kDPRA with 24 blind-coded test substances in seven laboratories. The intra- and inter-laboratory reproducibility were 96% and 88%, respectively (both for differentiating GHS Cat 1A sensitizers from GHS Cat 1B/not classified). Following an independent peer review, the kDPRA was considered to be acceptable for the identification of GHS Cat 1A skin sensitizers. Besides GHS Cat 1A identification, the kDPRA can be used as part of a defined approach(es) with a quantitative data integration procedure for skin sensitization potency assessment. For this aim, next to reproducibility of classification, the quantitative reproducibility and variability of the rate constants were quantified in this study.

A review of substances found positive in 1 of 3 in vitro tests for skin sensitization.

There has been significant progress in recent years in the development and application of alternative methods for assessing the skin sensitization potential of chemicals. The pathways involved in skin sensitization have been described in an OECD adverse outcome pathway (AOP). To date, a single non-animal test method is not sufficient to address this AOP so numerous approaches involving the use of 2 or more assays are being evaluated for their performance. The 2 out of 3 approach is a simple approach that has demonstrated very good sensitivity, specificity and overall accuracy numbers for predicting the skin sensitization potential of chemicals. Chemicals with at least two positive results in tests addressing Key events 1-3 are predicted sensitizers, while chemicals with none or only one positive outcome are predicted non-sensitizers. In this report we have thoroughly reviewed the discordant results of 29 chemicals with 1 out of 3 positive results to understand better what led to the results observed and how this information might impact our hazard assessments of these chemicals. We initially categorized each chemical using a weight of evidence approach as positive, negative or indeterminate based on review of available human and animal data as well as what skin sensitization alerts were triggered using two versions of OECD Toolbox and DEREK Nexus. We determined that 4 of the 29 chemicals should be classified as indeterminate and not included in analysis of method performance based on insufficient, borderline and/or conflicting data to confidently categorized the chemicals as allergens or non-allergens. Of the 29 chemicals included in this analysis, 17 were classified as negative and would be correctly identified using a 2 out of 3 approach while 8 chemicals were classified as positive in vivo and would be false-negative with this approach. For some of these chemicals, the outcomes observed can be explained by in vitro borderline results (13 chemicals) or in some instances there is mechanistic understanding of why a chemical is positive or negative in a particular assay (9 chemicals). Thus, when comparing the performance of different defined approaches, one should attempt to only include chemicals which demonstrate clear evidence to be categorize as allergens or non-allergens. Finally, when interpreting the results obtained for an individual unknown chemical it is critical that the in vitro skin sensitization data is reviewed critically and there is a good understanding of the variance and applicability domain limitations for each assay being used.

GHS additivity formula: can it predict the acute systemic toxicity of agrochemical formulations that contain acutely toxic ingredients?

In vivo acute systemic testing is a regulatory requirement for agrochemical formulations. GHS specifies an alternative computational approach (GHS additivity formula) for calculating the acute toxicity of mixtures. We collected acute systemic toxicity data from formulations that contained one of several acutely-toxic active ingredients. The resulting acute data set includes 210 formulations tested for oral toxicity, 128 formulations tested for inhalation toxicity and 31 formulations tested for dermal toxicity. The GHS additivity formula was applied to each of these formulations and compared with the experimental in vivo result. In the acute oral assay, the GHS additivity formula misclassified 110 formulations using the GHS classification criteria (48% accuracy) and 119 formulations using the USEPA classification criteria (43% accuracy). With acute inhalation, the GHS additivity formula misclassified 50 formulations using the GHS classification criteria (61% accuracy) and 34 formulations using the USEPA classification criteria (73% accuracy). For acute dermal toxicity, the GHS additivity formula misclassified 16 formulations using the GHS classification criteria (48% accuracy) and 20 formulations using the USEPA classification criteria (36% accuracy). This data indicates the acute systemic toxicity of many formulations is not the sum of the ingredients' toxicity (additivity); but rather, ingredients in a formulation can interact to result in lower or higher toxicity than predicted by the GHS additivity formula.

Prediction of skin sensitization potency sub-categories using peptide reactivity data.

While the skin sensitization hazard of substances can already be identified using non-animal methods, the classification of potency sub-categories GHS-1A and 1B is still challenging. Potency can be measured by the dose at which an effect is observed; since the protein-adduct formation is determining the dose of the allergen in the skin, peptide reactivity was used to assess the potency. The Direct Peptide Reactivity Assay (DPRA; one concentration and reaction-time) did not sufficiently discriminate between sub-categories 1A and 1B (56% accuracy compared to LLNA data, n=124). An extended protocol termed 'quantitative DPRA' (three concentrations and one reaction-time), discriminated sub-categories GHS 1A and 1B with an accuracy of 81% or 57% compared to LLNA (n=36) or human (n=14) data, respectively. The analysis of the Cys-adducts was already sufficient; additional analysis of Lys-adducts did not improve the predictivity. An additional modification, the 'kinetic DPRA' (several concentrations and reaction-times) was used to approximate the rate constant of Cys-peptide-adduct formation. 35 of 38 substances were correctly assigned to the potency sub-categories (LLNA data), and the predictivity for 14 human data was equally high. These results warrant the kinetic DPRA for further validation in order to fully replace in vivo testing for assessing skin sensitization including potency sub-classification.

Regulatory accepted but out of domain: In vitro skin irritation tests for agrochemical formulations.

Several in vitro methods have gained regulatory acceptance for the prediction of skin irritation and corrosion. However, the test guidelines for the majority of in vitro methods do not state whether they are applicable to agrochemical formulations. Hence, we would like to share the results from our routine skin corrosion and irritation testing of agrochemical formulations in which both in vitro (according to OECD TG 431 and OECD TG 439) and in vivo (according to OECD TG 404) tests were conducted as regulatory requirements. The in vitro skin irritation test did not correlate well with the CLP classification by in vivo results (44% sensitivity, 60% specificity, and 54% accuracy, based on 65 data pairs). This indicates a lack of applicability of the current protocol of the in vitro skin irritation test for agrochemical formulations. The data set did not contain formulations which were skin corrosive in vivo and hence its applicability could not be assessed. The correlation of in vitro skin corrosion testing to formulations which were not corrosive in vivo was, however, high (95% specificity based on 81 data pairs).

Lacking applicability of in vitro eye irritation methods to identify seriously eye irritating agrochemical formulations: Results of bovine cornea opacity and permeability assay, isolated chicken eye test and the EpiOcular™ ET-50 method to classify according to UN GHS.

In vitro methods have gained regulatory acceptance for the prediction of serious eye damage (UN GHS Cat 1). However, the majority of in vitro methods do not state whether they are applicable to agrochemical formulations. This manuscript presents a study of up to 27 agrochemical formulations tested in three in vitro assays (three versions of the bovine corneal opacity and permeability test (BCOP, OECD TG 437) assay, the isolated chicken eye test (ICE, OECD TG 438) and the EpiOcular™ ET-50 assay). The results were compared with already-available in vivo data. In the BCOP only one of the four, one of five in the ICE and six of eleven tested formulations in the EpiOcular™ ET-50 Neat Protocol resulted in the correct UN GHS Cat 1 prediction. Overpredictions occurred in all assays. These data indicate a lack of applicability of the three in vitro methods to reliably predict UN GHS Cat 1 of agrochemical formulations. In order to ensure animal-free identification of seriously eye damaging agrochemical formulations testing protocols and/or prediction models need to be modified or classification rules should be tailored to in vitro testing rather than using in vivo Draize data as a standard.