A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E).

BACKGROUND: Observational epidemiologic studies provide critical data for the evaluation of the potential effects of environmental, occupational and behavioural exposures on human health. Systematic reviews of these studies play a key role in informing policy and practice. Systematic reviews should incorporate assessments of the risk of bias in results of the included studies. OBJECTIVE: To develop a new tool, Risk Of Bias In Non-randomized Studies - of Exposures (ROBINS-E) to assess risk of bias in estimates from cohort studies of the causal effect of an exposure on an outcome. METHODS AND RESULTS: ROBINS-E was developed by a large group of researchers from diverse research and public health disciplines through a series of working groups, in-person meetings and pilot testing phases. The tool aims to assess the risk of bias in a specific result (exposure effect estimate) from an individual observational study that examines the effect of an exposure on an outcome. A series of preliminary considerations informs the core ROBINS-E assessment, including details of the result being assessed and the causal effect being estimated. The assessment addresses bias within seven domains, through a series of 'signalling questions'. Domain-level judgements about risk of bias are derived from the answers to these questions, then combined to produce an overall risk of bias judgement for the result, together with judgements about the direction of bias. CONCLUSION: ROBINS-E provides a standardized framework for examining potential biases in results from cohort studies. Future work will produce variants of the tool for other epidemiologic study designs (e.g. case-control studies). We believe that ROBINS-E represents an important development in the integration of exposure assessment, evidence synthesis and causal inference.

Systematic evidence map (SEM) template: Report format and methods used for the US EPA Integrated Risk Information System (IRIS) program, Provisional Peer Reviewed Toxicity Value (PPRTV) program, and other "fit for purpose" literature-based human health analyses.

BACKGROUND: Systematic evidence maps (SEMs) are gaining visibility in environmental health for their utility to serve as problem formulation tools and assist in decision-making, especially for priority setting. SEMs are now routinely prepared as part of the assessment development process for the US Environmental Protection Agency (EPA) Integrated Risk Information System (IRIS) and Provisional Peer Reviewed Toxicity Value (PPRTV) assessments. SEMs can also be prepared to explore the available literature for an individual chemical or groups of chemicals of emerging interest. OBJECTIVES: This document describes the typical methods used to produce SEMs for the IRIS and PPRTV Programs, as well as "fit for purpose" applications using a variety of examples drawn from existing analyses. It is intended to serve as an example base template that can be adapted as needed for the specific SEM. The presented methods include workflows intended to facilitate rapid production. The Populations, Exposures, Comparators and Outcomes (PECO) criteria are typically kept broad to identify mammalian animal bioassay and epidemiological studies that could be informative for human hazard identification. In addition, a variety of supplemental content is tracked, e.g., studies presenting information on in vitro model systems, non-mammalian model systems, exposure-level-only studies in humans, pharmacokinetic models, and absorption, distribution, metabolism, and excretion (ADME). The availability of New Approach Methods (NAMs) evidence is also tracked (e.g., high throughput, transcriptomic, in silico, etc.). Genotoxicity studies may be considered as PECO relevant or supplemental material, depending on the topic and context of the review. Standard systematic review practices (e.g., two independent reviewers per record) and specialized software applications are used to search and screen the literature and may include the use of machine learning software. Mammalian bioassay and epidemiological studies that meet the PECO criteria after full-text review are briefly summarized using structured web-based extraction forms with respect to study design and health system(s) assessed. Extracted data is available in interactive visual formats and can be downloaded in open access formats. Methods for conducting study evaluation are also presented which is conducted on a case-by-case basis, depending on the usage of the SEM.

Use of systematic evidence maps within the US environmental protection agency (EPA) integrated risk information system (IRIS) program: Advancements to date and looking ahead.

Systematic evidence maps (SEMs) are increasingly used to inform decision-making and risk management priority-setting and to serve as problem formulation tools to refine the focus of questions that get addressed in full systematic reviews. Within the U.S. Environmental Protection Agency (EPA) Office of Research and Development (ORD) Integrated Risk Information System (IRIS), SEMs have been used to inform data gaps, determine the need for updated assessments, inform assessment priorities, and inform development of study evaluation considerations, among other uses. Increased utilization of SEMs across the environmental health field has the potential to increase transparency and efficiency for data gathering, problem formulation, read-across, and evidence surveillance. Use of the SEM templates published in the companion text (Thayer et al.) can promote harmonization in the environmental health community and create more opportunities for sharing extracted content.

Integrated Risk Information System (IRIS) response to "Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools".

"Assessing risk of bias in human environmental epidemiology studies using three tools: different conclusions from different tools," a recent publication in this journal, applied the study evaluation approach developed by the U.S. Environmental Protection Agency's Integrated Risk Information System (IRIS), as well as other approaches, to a set of studies examining polybrominated diphenyl ethers (PBDEs) and neurodevelopment. They concluded that use of the IRIS approach resulted in exclusion of studies, which would lead to hazard conclusions based on an incomplete body of evidence. As scientists in the IRIS program, we support the comparison of approaches to improve systematic review methods for environmental exposures; however, we believe the IRIS approach was misrepresented. In this letter, we demonstrate that the ratings attributed to the IRIS approach were not consistent with our own application of the tool. We also clarify the use of studies rated as "low confidence" and the use of an overall study confidence rating in our systematic reviews. In conclusion, the IRIS study evaluation approach is a transparent method to inform certainty in our evidence synthesis decisions and ensures consistency in the development of IRIS health assessments.

Challenges and recommendations on the conduct of systematic reviews of observational epidemiologic studies in environmental and occupational health.

Systematic reviews are powerful tools for drawing causal inference for evidence-based decision-making. Published systematic reviews and meta-analyses of environmental and occupational epidemiology studies have increased dramatically in recent years; however, the quality and utility of published reviews are variable. Most methodologies were adapted from clinical epidemiology and have not been adequately modified to evaluate and integrate evidence from observational epidemiology studies assessing environmental and occupational hazards, especially in evaluating the quality of exposure assessments. Although many reviews conduct a systematic and transparent assessment for the potential for bias, they are often deficient in subsequently integrating across a body of evidence. A cohesive review considers the impact of the direction and magnitude of potential biases on the results, systematically evaluates important scientific issues such as study sensitivity and effect modifiers, identifies how different studies complement each other, and assesses other potential sources of heterogeneity. Given these challenges of conducting informative systematic reviews of observational studies, we provide a series of specific recommendations based on practical examples for cohesive evidence integration to reach an overall conclusion on a body of evidence to better support policy making in public health.

Development of outcome-specific criteria for study evaluation in systematic reviews of epidemiology studies.

INTRODUCTION AND OBJECTIVE: Systematic review tools that provide guidance on evaluating epidemiology studies are receiving increasing attention and support because their application facilitates improved quality of the review, consistency across reviewers, and transparency for readers. The U.S. Environmental Protection Agency's Integrated Risk Information System (IRIS) Program has developed an approach for systematic review of evidence of health effects from chemical exposures that includes structured approaches for literature search and screening, study evaluation, data extraction, and evidence synthesis and integration. This approach recognizes the need for developing outcome-specific criteria for study evaluation. Because studies are assessed at the outcome level, a study could be considered high quality for one investigated outcome, and low quality for another, due to differences in the outcome measures, analytic strategies, how relevant a certain bias is to the outcome, and how the exposure measure relates to the outcome. The objective of this paper is to illustrate the need for outcome-specific criteria in study evaluation or risk of bias evaluation, describe the process we used to develop the criteria, and summarize the resulting criteria. METHODS: We used a process of expert consultation to develop several sets of outcome-specific criteria to guide study reviewers, improve consistency, and ensure consideration of critical issues specific to the outcomes. The criteria were developed using the following domains: outcome assessment, exposure measurement (specifically timing of exposure in relation to outcome; other exposure measurement issues would be addressed in exposure-specific criteria), participant selection, confounding, analysis, and sensitivity (the study's ability to detect a true effect or hazard). RESULTS: We discuss the application of this process to pregnancy-related outcomes (preterm birth, spontaneous abortion), other reproductive-related outcomes (male reproductive hormones, sperm parameters, time to pregnancy, pubertal development), chronic disease (diabetes, insulin resistance), and acute or episodic conditions (asthma, allergies), and provide examples of the criteria developed. For each outcome the most influential methodological considerations are highlighted including biological sample collection and quality control, sensitivity and specificity of ascertainment tools, optimal timing for recruitment into the study (e.g., preconception, specific trimesters), the etiologically relevant window for exposure assessments, and important potential confounders. CONCLUSIONS: Outcome-specific criteria are an important part of a systematic review and will facilitate study evaluations by epidemiologists with experience in evaluating studies using systematic review methods who may not have extensive discipline-specific experience in the outcomes being reviewed.

Phthalate exposure and female reproductive and developmental outcomes: a systematic review of the human epidemiological evidence.

OBJECTIVE: We performed a systematic review of the epidemiology literature to identify the female reproductive and developmental effects associated with phthalate exposure. DATA SOURCES AND STUDY ELIGIBILITY CRITERIA: Six phthalates were included in the review: di(2-ethylhexyl) phthalate (DEHP), diisononyl phthalate (DINP), dibutyl phthalate (DBP), diisobutyl phthalate (DIBP), butyl benzyl phthalate (BBP), and diethyl phthalate (DEP). The initial literature search (of PubMed, Web of Science, and Toxline) included all studies of female reproductive and developmental effects in humans, and outcomes were selected for full systematic review based on data availability. STUDY EVALUATION AND SYNTHESIS METHODS: For each outcome, studies were evaluated using criteria defined a priori for risk of bias and sensitivity by two reviewers using a domain-based approach. Evidence was synthesized by outcome and phthalate and strength of evidence was summarized using a structured framework. RESULTS: The primary outcomes reviewed here are (number of included/excluded studies in parentheses): pubertal development (5/13), time to pregnancy (3/4), preterm birth (8/12), and spontaneous abortion (5/0). Among these outcomes, preterm birth had moderate evidence of a positive association with phthalate exposure (specifically DEHP, DBP, and DEP). Exposure levels for BBP, DIBP, and DINP were generally lower than for the phthalates with an observed effect, which may partially explain the difference due to lower sensitivity. Other phthalate/outcome combinations were considered to have slight or indeterminate evidence of an association. CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS: Overall, these results support that some phthalates may be associated with higher odds of preterm birth in humans, though there is some remaining inconsistency. More evidence is needed on the mechanism and relevant exposure window for this association. The views expressed are those of the authors and do not necessarily represent the views or policies of the U.S. EPA.

The evaluation of inhalation studies for exposure quality: A case study with formaldehyde.

The inherent complexity of generating and monitoring a test article in an inhalation chamber can make inhalation toxicity testing challenging. Poor study design, human error, and electrical and mechanical problems can adversely affect an inhalation exposure and undermine a study's results. We have developed a process for evaluating seven key elements of exposure quality in inhalation chamber studies: 1) test article characterization, 2) generation method, 3) chamber sampling and analytical method, 4) chamber concentrations, 5) particle size characteristics, 6) chamber type, and 7) controls. For each study evaluated, exposure deficiencies are documented, and a study is given an overall rating (Robust, Adequate, or Poor) for the quality of its exposure characterization and documentation. In combination with the systematic consideration of experimental features other than exposure, these ratings can inform the utility of a study for use in hazard identification and/or exposure-response analysis. Exposure quality evaluations of 204 formaldehyde inhalation studies are presented as a case study. Of these, 34% were rated Robust because they had comprehensive exposure documentation and no serious deficiencies in the key elements of exposure quality. Another 19% of studies with minor uncertainties or limitations were rated Adequate. Conversely, 47% of the studies were rated Poor due to multiple serious exposure deficiencies. This formaldehyde case study illustrates the need to carefully consider the exposure quality of inhalation toxicity studies when their results are used to support hazard and risk assessments.

Epidemiology: a foundation of environmental decision making.

Many epidemiologic studies are designed so they can be drawn upon to provide scientific evidence for evaluating hazards of environmental exposures, conducting quantitative assessments of risk, and informing decisions designed to reduce or eliminate harmful exposures. However, experimental animal studies are often relied upon for environmental and public health policy making despite the expanding body of observational epidemiologic studies that could inform the relationship between actual, as opposed to controlled, exposures and health effects. This paper provides historical examples of how epidemiology has informed decisions at the U.S. Environmental Protection Agency, discusses some challenges with using epidemiology to inform decision making, and highlights advances in the field that may help address these challenges and further the use of epidemiologic studies moving forward.

Study sensitivity: Evaluating the ability to detect effects in systematic reviews of chemical exposures.

A critical step in systematic reviews of potential health hazards is the structured evaluation of the strengths and weaknesses of the included studies; risk of bias is a term often used to represent this process, specifically with respect to the evaluation of systematic errors that can lead to inaccurate (biased) results (i.e. focusing on internal validity). Systematic review methods developed in the clinical medicine arena have been adapted for use in evaluating environmental health hazards; this expansion raises questions about the scope of risk of bias tools and the extent to which they capture the elements that can affect the interpretation of results from environmental and occupational epidemiology studies and in vivo animal toxicology studies, (the studies typically available for assessment of risk of chemicals). One such element, described here as "sensitivity", is a measure of the ability of a study to detect a true effect or hazard. This concept is similar to the concept of the sensitivity of an assay; an insensitive study may fail to show a difference that truly exists, leading to a false conclusion of no effect. Factors relating to study sensitivity should be evaluated in a systematic manner with the same rigor as the evaluation of other elements within a risk of bias framework. We discuss the importance of this component for the interpretation of individual studies, examine approaches proposed or in use to address it, and describe how it relates to other evaluation components. The evaluation domains contained within a risk of bias tool can include, or can be modified to include, some features relating to study sensitivity; the explicit inclusion of these sensitivity criteria with the same rigor and at the same stage of study evaluation as other bias-related criteria can improve the evaluation process. In some cases, these and other features may be better addressed through a separate sensitivity domain. The combined evaluation of risk of bias and sensitivity can be used to identify the most informative studies, to evaluate the confidence of the findings from individual studies and to identify those study elements that may help to explain heterogeneity across the body of literature.

Human health effects of tetrachloroethylene: key findings and scientific issues.

BACKGROUND: The U.S. Environmental Protection Agency (EPA) completed a toxicological review of tetrachloroethylene (perchloroethylene, PCE) in February 2012 in support of the Integrated Risk Information System (IRIS). OBJECTIVES: We reviewed key findings and scientific issues regarding the human health effects of PCE described in the U.S. EPA's Toxicological Review of Tetrachloroethylene (Perchloroethylene). METHODS: The updated assessment of PCE synthesized and characterized a substantial database of epidemiological, experimental animal, and mechanistic studies. Key scientific issues were addressed through modeling of PCE toxicokinetics, synthesis of evidence from neurological studies, and analyses of toxicokinetic, mechanistic, and other factors (tumor latency, severity, and background rate) in interpreting experimental animal cancer findings. Considerations in evaluating epidemiological studies included the quality (e.g., specificity) of the exposure assessment methods and other essential design features, and the potential for alternative explanations for observed associations (e.g., bias or confounding). DISCUSSION: Toxicokinetic modeling aided in characterizing the complex metabolism and multiple metabolites that contribute to PCE toxicity. The exposure assessment approach-a key evaluation factor for epidemiological studies of bladder cancer, non-Hodgkin lymphoma, and multiple myeloma-provided suggestive evidence of carcinogenicity. Bioassay data provided conclusive evidence of carcinogenicity in experimental animals. Neurotoxicity was identified as a sensitive noncancer health effect, occurring at low exposures: a conclusion supported by multiple studies. Evidence was integrated from human, experimental animal, and mechanistic data sets in assessing adverse health effects of PCE. CONCLUSIONS: PCE is likely to be carcinogenic to humans. Neurotoxicity is a sensitive adverse health effect of PCE.

Summary and findings of the EPA and CDC symposium on air pollution exposure and health.

The U.S. Environmental Protection Agency (EPA) and the U.S. Centers for Disease Control (CDC) co-organized a symposium on "Air Pollution Exposure and Health" at Research Triangle Park, North Carolina on September 19-20, 2006. The symposium brought together health and environmental scientists to discuss the state of the science and the cross-jurisdictional and methodological challenges in conducting air pollution epidemiology, environmental public health tracking and accountability research. The symposium was held over 2 days and consisted of technical presentations and breakout group discussions on each of the three principal themes of this meeting: (1) monitoring and exposure modeling information, (2) health effects data and (3) linkage of air quality and health data for research, tracking and accountability. This paper summarizes the symposium presentations and the conclusions and recommendations developed during the meeting. The accompanying two papers, which appear in this issue of the Journal, provide more in-depth discussion of issues pertinent to obtaining and analyzing air pollution exposure and health information. The symposium succeeded in identifying areas where there are critical gaps of knowledge in existing air pollution exposure and health information and in discovering institutional or programmatic barriers, which impede accessing and linking disparate data sets. Several suggestions and recommendations emerged from this meeting, directed toward (1) improving the utility of air monitoring data for exposure quantification, (2) improving access to and the quality of health data, (3) studying emerging air quality and health issues, (4) exploring improved or novel methods for linking data and (5) developing partnerships, building capacity and facilitating interdisciplinary communication. The meeting was successful in promoting an interdisciplinary dialogue around these issues and in formulating strategies to support these recommended activities. Finally, this symposium subsequently led to strengthening and initiating new partnerships or interactions between the EPA, CDC, States, academia and the research community at large.

Use of health information in air pollution health research: past successes and emerging needs.

In September 2006, the US Environmental Protection Agency and the US Centers for Disease Control (CDC) co-organized a symposium on "Air Pollution Exposure and Health." The main objective of this symposium was to identify opportunities for improving the use of exposure and health information in future studies of air pollution health effects. This paper deals with the health information needs of such studies. We begin with a selected review of different types of health data and how they were used in previous epidemiologic studies of health effects of ambient particulate matter (PM). We then examine the current and emerging information needs of the environmental health community, dealing with PM and other air pollutants of health concern. We conclude that the past use of routinely collected health data proved to be essential for activities to protect public health, including the identification and evaluation of health hazards by air pollution research, setting standards for criteria pollutants, surveillance of health outcomes to identify incidence trends, and the more recent CDC environmental public health tracking program. Unfortunately, access to vital statistics records that have informed such pivotal research has recently been curtailed sharply, threatening the continuation of the type of research necessary to support future standard setting and research on emerging exposure and health problems (e.g. asthma, multiple sclerosis, diabetes, and others), as well as our ability to evaluate the efficacy of regulatory and other prevention activities. A comprehensive devoted effort, perhaps new legislation, will be needed to address the standardization, centralization, and sharing of data sets, as well as to harmonize the interpretation of confidentiality and privacy protections across jurisdictions. These actions, combined with assuring researchers and public health practitioners appropriate access to data for evaluation of environmental risks, will be essential for the achievement of our environmental health protection goals.

Changes in systolic blood pressure associated with lead in blood and bone.

BACKGROUND: Several studies have examined longitudinal associations of blood pressure change or hypertension incidence with lead concentration in blood or bone. It is not clear whether the observed associations reflect an immediate response to lead as a consequence of recent dose or rather are a persistent effect of cumulative dose over a lifetime. METHODS: We followed 575 subjects in a lead-exposed occupational cohort in South Korea between October 1997 and June 2001. We used generalized estimating equation models to evaluate blood pressure change between study visits in relation to tibia lead concentrations at each prior visit and concurrent changes in blood lead. The modeling strategy summarized the longitudinal association of blood pressure with cumulative lead dose or changes in recent lead dose. RESULTS: On average, participants were 41 years old at baseline and had worked 8.5 years in lead-exposed jobs. At baseline, the average +/- standard deviation for blood lead was 31.4 +/- 14.2 microg/dL, and for tibia lead, it was 38.4 +/- 42.9 microg/g bone mineral. Change in systolic blood pressure during the study was associated with concurrent blood lead change, with an average annual increase of 0.9 (95% confidence interval = 0.1 to 1.6) mm Hg for every 10-microg/dL increase in blood lead per year. CONCLUSION: The findings in this relatively young population of current and former lead workers suggest that systolic blood pressure responds to lead dose through acute pathways in addition to the effects of cumulative injury.

The longitudinal association of lead with blood pressure.

BACKGROUND: Several investigators have reported an association of blood lead or bone lead with increased blood pressure and hypertension, but questions remain concerning whether these effects are acute or chronic in nature. METHODS: In this longitudinal study, we evaluated the relation of lead, measured in blood and tibia, to changes in blood pressure between 1994 and 1998. We studied 496 current and former employees of a chemical-manufacturing facility in the eastern United States who had previous occupational exposure to inorganic and organic lead. Cohort members who provided three or four blood pressure measurements during the study were included. RESULTS: Mean age at baseline was 55.8 years with a mean of 18 years since last occupational exposure to lead. Blood lead at baseline averaged 4.6 microg/dL (standard deviation [SD] = 2.6) or 0.22 micromole/Liter (SD = 0.13). Tibia lead at year three averaged 14.7-microg/gm (SD = 9.4) bone mineral. Change in systolic blood pressure during the study was associated with lead dose, with an average annual increase of 0.64 mmHg (standard error [SE] = 0.25), 0.73 mmHg (SE = 0.26), and 0.61 mmHg (SE = 0.27) for every standard deviation increase in blood lead at baseline, tibia lead at year three, or peak past tibia lead, respectively. CONCLUSIONS: The results support an etiologic role for lead in the elevation of systolic blood pressure among adult males and are consistent with both acute and chronic modes of action.