Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design.

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting.

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Misapplied management makes matters worse: Spatially explicit control leverages biotic interactions to slow invasion.

A wide range of approaches has been used to manage the spread of invasive species, yet invaders continue to be a challenge to control. In some cases, management actions have no effect or may even inadvertently benefit the targeted invader. Here, we use the mid-20th century management of the Red Imported Fire Ant, Solenopsis invicta, in the US as a motivating case study to explore the conditions under which such wasted management effort may occur. Introduced in approximately 1940, the fire ant spread widely through the southeast US and became a problematic pest. Historically, fire ants were managed with broad-spectrum pesticides; unfortunately, these efforts were largely unsuccessful. One hypothesis suggests that, by also killing native ants, mass pesticide application reduced competitive burdens thereby enabling fire ants to invade more quickly than they would in the absence of management. We use a mechanistic competition model to demonstrate the landscape-level effects of such management. We explicitly model the extent and location of pesticide applications, showing that the same pesticide application can have a positive, neutral, or negative effect on the progress of an invasion, depending on where it is applied on the landscape with respect to the invasion front. When designing management, the target species is often considered alone; however, this work suggests that leveraging existing biotic interactions, specifically competition with native species, can increase the efficacy of management. Our model not only highlights the potential unintended consequences of ignoring biotic interactions, but also provides a framework for developing spatially explicit management strategies that take advantage of these biotic interactions to work smarter, not harder.

Ensemble2: Scenarios ensembling for communication and performance analysis.

Throughout the COVID-19 pandemic, scenario modeling played a crucial role in shaping the decision-making process of public health policies. Unlike forecasts, scenario projections rely on specific assumptions about the future that consider different plausible states-of-the-world that may or may not be realized and that depend on policy interventions, unpredictable changes in the epidemic outlook, etc. As a consequence, long-term scenario projections require different evaluation criteria than the ones used for traditional short-term epidemic forecasts. Here, we propose a novel ensemble procedure for assessing pandemic scenario projections using the results of the Scenario Modeling Hub (SMH) for COVID-19 in the United States (US). By defining a "scenario ensemble" for each model and the ensemble of models, termed "Ensemble2", we provide a synthesis of potential epidemic outcomes, which we use to assess projections' performance, bypassing the identification of the most plausible scenario. We find that overall the Ensemble2 models are well-calibrated and provide better performance than the scenario ensemble of individual models. The ensemble procedure accounts for the full range of plausible outcomes and highlights the importance of scenario design and effective communication. The scenario ensembling approach can be extended to any scenario design strategy, with potential refinements including weighting scenarios and allowing the ensembling process to evolve over time.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy.

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Elevated temperatures shift flower head height distributions and seed dispersal patterns in two invasive thistle species.

Climate change may significantly alter how organisms disperse, with implications for population spread and species management. Wind-dispersed plants have emerged as a useful study system for investigating how climate change affects dispersal, although studies modeling wind dispersal often assume propagules are released from a single point on an individual. This simplifying assumption, while useful, may misestimate dispersal. Here, we investigate the effects of climate change on dispersal distances and spread rates, examining how these quantities shift when accounting for all points of seed release on an individual. Using the wind-dispersed invasive thistles Carduus nutans and Carduus acanthoides, we quantify temperature-driven shifts in the distribution of flower head heights using a passive warming field experiment, and estimate how these shifts affect dispersal using the Wald analytical long-distance (WALD) model; for C. nutans, we use existing demographic data to simulate how these shifts affect population spread rates. We also compare dispersal distances for both warmed and ambient temperature plants, considering the entire distribution of flower head heights versus the common assumption of point-source seed release at the maximum height. For experimentally grown individuals, an ~0.6°C higher growing temperature increased mean and maximum flower head height by 14.1 cm (15.0%) and 14.0 cm (13.2%), respectively, in C. nutans and by 21.2 cm (26.6%) and 31.8 cm (36.7%), respectively, in C. acanthoides. Seeds from warmed individuals were more likely to exceed a given dispersal distance than those from their unwarmed counterparts; warmed C. nutans and C. acanthoides seeds were on average 1.36 and 1.71 times as likely, respectively, to travel 10 m or more in dispersal simulations, with this disparity increasing at longer dispersal distances. For C. nutans, increased growing temperatures boosted simulated rates of population spread by 42.2%, while assuming dispersal from a maximum height point source rather than the true distribution of flower head heights increased simulated spread by up to 28.5%. Our results not only demonstrate faster population spread under increased temperatures, but also have substantial implications for modeling such spread, as the common simplifying assumption of dispersal from a single maximum height source may substantially overestimate spread rates.

Climate warming increases insect-driven seed removal of two elaiosome-bearing invasive thistle species.

Ants and other insects are often a source of localized secondary dispersal for wind-dispersed plants and thus play an important ecological role in their spatial dynamics, but there is limited information on how climate change will affect such dispersal processes. Here, we use field experiments to investigate how climate warming affects seed removal, as this initiation of movement represents the first step in insect-driven secondary dispersal. Our results indicate that for the invasive thistles Carduus nutans and Carduus acanthoides, increased growing temperature influences seed attractiveness to insect dispersers, with seeds from maternal plants grown at temperatures 0.6°C above ambient removed by insect dispersers at higher rates than their unwarmed counterparts. We also observe that seed elaiosomes in these two species play an important role in dispersal, as seeds without elaiosomes were significantly less likely to be removed over the same period. Significant interactions between elaiosome presence/absence and warming treatment were also observed, though only for C. acanthoides, with the boost in seed removal from warming dampened when the elaiosome was present compared to when it was absent. These findings provide evidence that climate warming may alter aspects of dispersal such as seed removal by secondary dispersers, with potential ramifications for dispersal in future climates since seed-bearing plants around the world may be subject to increased growing temperatures, and many of these plant species bear elaiosomes and experience seed dispersal by insects.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design.

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Diversity loss from multiple interacting disturbances is regime-dependent.

Anthropogenic activities expose many ecosystems to multiple novel disturbances simultaneously. Despite this, how biodiversity responds to simultaneous disturbances remains unclear, with conflicting empirical results on their interactive effects. Here, we experimentally test how one disturbance (an invasive species) affects the diversity of a community over multiple levels of another disturbance regime (pulse mortality). Specifically, we invade stably coexisting bacterial communities under four different pulse frequencies, and compare their final resident diversity to uninvaded communities under the same pulse mortality regimes. Our experiment shows that the disturbances synergistically interact, such that the invader significantly reduces resident diversity at high pulse frequency, but not at low. This work therefore highlights the need to study simultaneous disturbance effects over multiple disturbance regimes as well as to carefully document unmanipulated disturbances, and may help explain the conflicting results seen in previous multiple-disturbance work.

Multiple models for outbreak decision support in the face of uncertainty.

Policymakers must make management decisions despite incomplete knowledge and conflicting model projections. Little guidance exists for the rapid, representative, and unbiased collection of policy-relevant scientific input from independent modeling teams. Integrating approaches from decision analysis, expert judgment, and model aggregation, we convened multiple modeling teams to evaluate COVID-19 reopening strategies for a mid-sized United States county early in the pandemic. Projections from seventeen distinct models were inconsistent in magnitude but highly consistent in ranking interventions. The 6-mo-ahead aggregate projections were well in line with observed outbreaks in mid-sized US counties. The aggregate results showed that up to half the population could be infected with full workplace reopening, while workplace restrictions reduced median cumulative infections by 82%. Rankings of interventions were consistent across public health objectives, but there was a strong trade-off between public health outcomes and duration of workplace closures, and no win-win intermediate reopening strategies were identified. Between-model variation was high; the aggregate results thus provide valuable risk quantification for decision making. This approach can be applied to the evaluation of management interventions in any setting where models are used to inform decision making. This case study demonstrated the utility of our approach and was one of several multimodel efforts that laid the groundwork for the COVID-19 Scenario Modeling Hub, which has provided multiple rounds of real-time scenario projections for situational awareness and decision making to the Centers for Disease Control and Prevention since December 2020.

Optimal spatial evaluation of a pro rata vaccine distribution rule for COVID-19.

The COVID-19 Vaccines Global Access (COVAX) is a World Health Organization (WHO) initiative that aims for an equitable access of COVID-19 vaccines. Despite potential heterogeneous infection levels across a country, countries receiving allotments of vaccines may follow WHO's allocation guidelines and distribute vaccines based on a jurisdictions' relative population size. Utilizing economic-epidemiological modeling, we benchmark the performance of this pro rata allocation rule by comparing it to an optimal one that minimizes the economic damages and expenditures over time, including a penalty representing the social costs of deviating from the pro rata strategy. The pro rata rule performs better when the duration of naturally- and vaccine-acquired immunity is short, when there is population mixing, when the supply of vaccine is high, and when there is minimal heterogeneity in demographics. Despite behavioral and epidemiological uncertainty diminishing the performance of the optimal allocation, it generally outperforms the pro rata vaccine distribution rule.

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology.

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Pulse and Press Disturbances Have Different Effects on Transient Community Dynamics.

AbstractDisturbances are important determinants of diversity, and the combination of their aspects (e.g., disturbance intensity, frequency) can result in complex diversity patterns. Here, we leverage an important approach to classifying disturbances in terms of temporal span to understand the implications for species coexistence: pulse disturbances are acute and discrete events, while press disturbances occur continuously through time. We incorporate the resultant mortality rates into a common framework involving disturbance frequency and intensity. Press disturbances can be encoded into models in two distinct ways, and we show that the appropriateness of each depends on the type of data available. Using this framework, we compare the effects of pulse versus press disturbance on both asymptotic and transient dynamics of a two-species Lotka-Volterra competition model to understand how they engage with equalizing mechanisms of coexistence. We show that press and pulse disturbances differ in transient behavior, though their asymptotic diversity patterns are similar. Our work shows that these differences depend on how the underlying disturbance aspects interact and that the two ways of characterizing press disturbances can lead to contrasting interpretations of disturbance-diversity relationships. Our work demonstrates how theoretical modeling can strategically guide and help the interpretation of empirical work.

Vote-processing rules for combining control recommendations from multiple models.

Mathematical modelling is used during disease outbreaks to compare control interventions. Using multiple models, the best method to combine model recommendations is unclear. Existing methods weight model projections, then rank control interventions using the combined projections, presuming model outputs are directly comparable. However, the way each model represents the epidemiological system will vary. We apply electoral vote-processing rules to combine model-generated rankings of interventions. Combining rankings of interventions, instead of combining model projections, avoids assuming that projections are comparable as all comparisons of projections are made within each model. We investigate four rules: First-past-the-post, Alternative Vote (AV), Coombs Method and Borda Count. We investigate rule sensitivity by including models that favour only one action or including those that rank interventions randomly. We investigate two case studies: the 2014 Ebola outbreak in West Africa (37 compartmental models) and a hypothetical foot-and-mouth disease outbreak in UK (four individual-based models). The Coombs Method was least susceptible to adding models that favoured a single action, Borda Count and AV were most susceptible to adding models that ranked interventions randomly. Each rule chose the same intervention as when ranking interventions by mean projections, suggesting that combining rankings provides similar recommendations with fewer assumptions about model comparability. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Cattle transport network predicts endemic and epidemic foot-and-mouth disease risk on farms in Turkey.

The structure of contact networks affects the likelihood of disease spread at the population scale and the risk of infection at any given node. Though this has been well characterized for both theoretical and empirical networks for the spread of epidemics on completely susceptible networks, the long-term impact of network structure on risk of infection with an endemic pathogen, where nodes can be infected more than once, has been less well characterized. Here, we analyze detailed records of the transportation of cattle among farms in Turkey to characterize the global and local attributes of the directed-weighted shipments network between 2007-2012. We then study the correlations between network properties and the likelihood of infection with, or exposure to, foot-and-mouth disease (FMD) over the same time period using recorded outbreaks. The shipments network shows a complex combination of features (local and global) that have not been previously reported in other networks of shipments; i.e. small-worldness, scale-freeness, modular structure, among others. We find that nodes that were either infected or at high risk of infection with FMD (within one link from an infected farm) had disproportionately higher degree, were more central (eigenvector centrality and coreness), and were more likely to be net recipients of shipments compared to those that were always more than 2 links away from an infected farm. High in-degree (i.e. many shipments received) was the best univariate predictor of infection. Low in-coreness (i.e. peripheral nodes) was the best univariate predictor of nodes always more than 2 links away from an infected farm. These results are robust across the three different serotypes of FMD observed in Turkey and during periods of low-endemic prevalence and high-prevalence outbreaks.

Whole community invasions and the integration of novel ecosystems.

The impact of invasion by a single non-native species on the function and structure of ecological communities can be significant, and the effects can become more drastic-and harder to predict-when multiple species invade as a group. Here we modify a dynamic Boolean model of plant-pollinator community assembly to consider the invasion of native communities by multiple invasive species that are selected either randomly or such that the invaders constitute a stable community. We show that, compared to random invasion, whole community invasion leads to final stable communities (where the initial process of species turnover has given way to a static or near-static set of species in the community) including both native and non-native species that are larger, more likely to retain native species, and which experience smaller changes to the topological measures of nestedness and connectance. We consider the relationship between the prevalence of mutualistic interactions among native and invasive species in the final stable communities and demonstrate that mutualistic interactions may act as a buffer against significant disruptions to the native community.

Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Delta variant and faltering vaccination.

In Spring 2021, the highly transmissible SARS-CoV-2 Delta variant began to cause increases in cases, hospitalizations, and deaths in parts of the United States. At the time, with slowed vaccination uptake, this novel variant was expected to increase the risk of pandemic resurgence in the US in summer and fall 2021. As part of the COVID-19 Scenario Modeling Hub, an ensemble of nine mechanistic models produced 6-month scenario projections for July-December 2021 for the United States. These projections estimated substantial resurgences of COVID-19 across the US resulting from the more transmissible Delta variant, projected to occur across most of the US, coinciding with school and business reopening. The scenarios revealed that reaching higher vaccine coverage in July-December 2021 reduced the size and duration of the projected resurgence substantially, with the expected impacts was largely concentrated in a subset of states with lower vaccination coverage. Despite accurate projection of COVID-19 surges occurring and timing, the magnitude was substantially underestimated 2021 by the models compared with the of the reported cases, hospitalizations, and deaths occurring during July-December, highlighting the continued challenges to predict the evolving COVID-19 pandemic. Vaccination uptake remains critical to limiting transmission and disease, particularly in states with lower vaccination coverage. Higher vaccination goals at the onset of the surge of the new variant were estimated to avert over 1.5 million cases and 21,000 deaths, although may have had even greater impacts, considering the underestimated resurgence magnitude from the model.

Pest management in future climates: Warming reduces physical weed management effectiveness.

Climate change alters many aspects of weed performance and may also alter the effectiveness of management practices to control pests. Despite this concern, entire categories of widely used management practices, such as physical control, remain understudied in this context. We conducted a field experiment growing the invasive pest musk thistle (Carduus nutans) at ambient and experimentally elevated temperatures. We tested mowing management strategies that varied in the timing of a single mowing event relative to thistles' stem elongation phenology and compared these with an unmowed control. Results from this experiment informed demographic models to project population growth rates for different warming/mowing scenarios. Compared to plants grown under ambient conditions, warmed thistles were more likely to survive the same mowing treatment, flowered earlier in the season, grew to taller heights, and produced more flowering capitula. Proportional reductions in plant height and capitulum production caused by mowing were smaller under warming. Warming did not change the relative ranking of mowing treatments; mowing late in the growing season (2 weeks after individuals first reached a height of 40 cm) was most effective at ambient temperatures and under warming. Warming caused significant increases in projected local population growth rate for all mowing treatments. For invasive musk thistle, warmed individuals outperformed individuals grown at ambient temperatures across all the mowing treatments we considered. Our results suggest that to achieve outcomes comparable to those attainable at today's temperatures, farmers will need to apply supplemental management, possibly including additional mowing effort or alternative practices such as chemical control. We recommend that scientists test management practices under experimental warming, where possible, and that managers monitor ongoing management to identify changes in effectiveness. Information about changes in managed weeds' mortality, fecundity, and phenology can then be used to make informed decisions in future climates.

Disturbance-mediated invasions are dependent on community resource abundance.

Disturbances can facilitate biological invasions, with the associated increase in resource availability being a proposed cause. Here, we experimentally tested the interactive effects of disturbance regime (different frequencies of biomass removal at equal intensities) and resource abundance on invasion success using a factorial design containing five disturbance frequencies and three resource levels. We invaded populations of the bacterium Pseudomonas fluorescens with two ecologically different invader morphotypes: a fast-growing "colonizer" type and a slower growing "competitor" type. As resident populations were altered by the treatments, we additionally tested their effect on invader success. Disturbance frequency and resource abundance interacted to affect the success of both invaders, but this interaction differed between the invader types. The success of the colonizer type was positively affected by disturbance under high resources but negatively under low. However, disturbance negatively affected the success of the competitor type under high resource abundance but not under low or medium. Resident population changes did not alter invader success beyond direct treatment effects. We therefore demonstrate that the same disturbance regime can either be beneficial or detrimental for an invader depending on both community resource abundance and its life history. These results may help to explain some of the inconsistencies found in the disturbance-invasion literature.