Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design.

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Potential impact of annual vaccination with reformulated COVID-19 vaccines: Lessons from the US COVID-19 scenario modeling hub.

BACKGROUND: Coronavirus Disease 2019 (COVID-19) continues to cause significant hospitalizations and deaths in the United States. Its continued burden and the impact of annually reformulated vaccines remain unclear. Here, we present projections of COVID-19 hospitalizations and deaths in the United States for the next 2 years under 2 plausible assumptions about immune escape (20% per year and 50% per year) and 3 possible CDC recommendations for the use of annually reformulated vaccines (no recommendation, vaccination for those aged 65 years and over, vaccination for all eligible age groups based on FDA approval). METHODS AND FINDINGS: The COVID-19 Scenario Modeling Hub solicited projections of COVID-19 hospitalization and deaths between April 15, 2023 and April 15, 2025 under 6 scenarios representing the intersection of considered levels of immune escape and vaccination. Annually reformulated vaccines are assumed to be 65% effective against symptomatic infection with strains circulating on June 15 of each year and to become available on September 1. Age- and state-specific coverage in recommended groups was assumed to match that seen for the first (fall 2021) COVID-19 booster. State and national projections from 8 modeling teams were ensembled to produce projections for each scenario and expected reductions in disease outcomes due to vaccination over the projection period. From April 15, 2023 to April 15, 2025, COVID-19 is projected to cause annual epidemics peaking November to January. In the most pessimistic scenario (high immune escape, no vaccination recommendation), we project 2.1 million (90% projection interval (PI) [1,438,000, 4,270,000]) hospitalizations and 209,000 (90% PI [139,000, 461,000]) deaths, exceeding pre-pandemic mortality of influenza and pneumonia. In high immune escape scenarios, vaccination of those aged 65+ results in 230,000 (95% confidence interval (CI) [104,000, 355,000]) fewer hospitalizations and 33,000 (95% CI [12,000, 54,000]) fewer deaths, while vaccination of all eligible individuals results in 431,000 (95% CI: 264,000-598,000) fewer hospitalizations and 49,000 (95% CI [29,000, 69,000]) fewer deaths. CONCLUSIONS: COVID-19 is projected to be a significant public health threat over the coming 2 years. Broad vaccination has the potential to substantially reduce the burden of this disease, saving tens of thousands of lives each year.

COVSIM: A stochastic agent-based COVID-19 SIMulation model for North Carolina.

We document the evolution and use of the stochastic agent-based COVID-19 simulation model (COVSIM) to study the impact of population behaviors and public health policy on disease spread within age, race/ethnicity, and urbanicity subpopulations in North Carolina. We detail the methodologies used to model the complexities of COVID-19, including multiple agent attributes (i.e., age, race/ethnicity, high-risk medical status), census tract-level interaction network, disease state network, agent behavior (i.e., masking, pharmaceutical intervention (PI) uptake, quarantine, mobility), and variants. We describe its uses outside of the COVID-19 Scenario Modeling Hub (CSMH), which has focused on the interplay of nonpharmaceutical and pharmaceutical interventions, equitability of vaccine distribution, and supporting local county decision-makers in North Carolina. This work has led to multiple publications and meetings with a variety of local stakeholders. When COVSIM joined the CSMH in January 2022, we found it was a sustainable way to support new COVID-19 challenges and allowed the group to focus on broader scientific questions. The CSMH has informed adaptions to our modeling approach, including redesigning our high-performance computing implementation.

Potential impact of annual vaccination with reformulated COVID-19 vaccines: lessons from the U.S. COVID-19 Scenario Modeling Hub.

Importance: COVID-19 continues to cause significant hospitalizations and deaths in the United States. Its continued burden and the impact of annually reformulated vaccines remain unclear. Objective: To project COVID-19 hospitalizations and deaths from April 2023-April 2025 under two plausible assumptions about immune escape (20% per year and 50% per year) and three possible CDC recommendations for the use of annually reformulated vaccines (no vaccine recommendation, vaccination for those aged 65+, vaccination for all eligible groups). Design: The COVID-19 Scenario Modeling Hub solicited projections of COVID-19 hospitalization and deaths between April 15, 2023-April 15, 2025 under six scenarios representing the intersection of considered levels of immune escape and vaccination. State and national projections from eight modeling teams were ensembled to produce projections for each scenario. Setting: The entire United States. Participants: None. Exposure: Annually reformulated vaccines assumed to be 65% effective against strains circulating on June 15 of each year and to become available on September 1. Age and state specific coverage in recommended groups was assumed to match that seen for the first (fall 2021) COVID-19 booster. Main outcomes and measures: Ensemble estimates of weekly and cumulative COVID-19 hospitalizations and deaths. Expected relative and absolute reductions in hospitalizations and deaths due to vaccination over the projection period. Results: From April 15, 2023-April 15, 2025, COVID-19 is projected to cause annual epidemics peaking November-January. In the most pessimistic scenario (high immune escape, no vaccination recommendation), we project 2.1 million (90% PI: 1,438,000-4,270,000) hospitalizations and 209,000 (90% PI: 139,000-461,000) deaths, exceeding pre-pandemic mortality of influenza and pneumonia. In high immune escape scenarios, vaccination of those aged 65+ results in 230,000 (95% CI: 104,000-355,000) fewer hospitalizations and 33,000 (95% CI: 12,000-54,000) fewer deaths, while vaccination of all eligible individuals results in 431,000 (95% CI: 264,000-598,000) fewer hospitalizations and 49,000 (95% CI: 29,000-69,000) fewer deaths. Conclusion and Relevance: COVID-19 is projected to be a significant public health threat over the coming two years. Broad vaccination has the potential to substantially reduce the burden of this disease.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty.

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design.

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub.

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Vaccinating children against COVID-19 is crucial to protect schools and communities.

To evaluate the joint impact of childhood vaccination rates and school masking policies on community transmission and severe outcomes due to COVID-19, we utilized a stochastic, agent-based simulation of North Carolina to test 24 health policy scenarios. In these scenarios, we varied the childhood (ages 5 to 19) vaccination rate relative to the adult's (ages 20 to 64) vaccination rate and the masking relaxation policies in schools. We measured the overall incidence of disease, COVID-19-related hospitalization, and mortality from 2021 July 1 to 2023 July 1. Our simulation estimates that removing all masks in schools in January 2022 could lead to a 31% to 45%, 23% to 35%, and 13% to 19% increase in cumulative infections for ages 5 to 9, 10 to 19, and the total population, respectively, depending on the childhood vaccination rate. Additionally, achieving a childhood vaccine uptake rate of 50% of adults could lead to a 31% to 39% reduction in peak hospitalizations overall masking scenarios compared with not vaccinating this group. Finally, our simulation estimates that increasing vaccination uptake for the entire eligible population can reduce peak hospitalizations in 2022 by an average of 83% and 87% across all masking scenarios compared to the scenarios where no children are vaccinated. Our simulation suggests that high vaccination uptake among both children and adults is necessary to mitigate the increase in infections from mask removal in schools and workplaces.

Can vaccine prioritization reduce disparities in COVID-19 burden for historically marginalized populations?

SARS-CoV-2 vaccination strategies were designed to reduce COVID-19 mortality, morbidity, and health inequities. To assess the impact of vaccination strategies on disparities in COVID-19 burden among historically marginalized populations (HMPs), e.g. Black race and Hispanic ethnicity, we used an agent-based simulation model, populated with census-tract data from North Carolina. We projected COVID-19 deaths, hospitalizations, and cases from 2020 July 1 to 2021 December 31, and estimated racial/ethnic disparities in COVID-19 outcomes. We modeled 2-stage vaccination prioritization scenarios applied to sub-groups including essential workers, older adults (65+), adults with high-risk health conditions, HMPs, or people in low-income tracts. Additionally, we estimated the effects of maximal uptake (100% for HMP vs. 100% for everyone), and distribution to only susceptible people. We found strategies prioritizing essential workers, then older adults led to the largest mortality and case reductions compared to no prioritization. Under baseline uptake scenarios, the age-adjusted mortality for HMPs was higher (e.g. 33.3%-34.1% higher for the Black population and 13.3%-17.0% for the Hispanic population) compared to the White population. The burden on HMPs decreased only when uptake was increased to 100% in HMPs; however, the Black population still had the highest relative mortality rate even when targeted distribution strategies were employed. If prioritization schemes were not paired with increased uptake in HMPs, disparities did not improve. The vaccination strategies publicly outlined were insufficient, exacerbating disparities between racial and ethnic groups. Strategies targeted to increase vaccine uptake among HMPs are needed to ensure equitable distribution and minimize disparities in outcomes.

Vaccinating Children Against COVID-19 is Essential Prior to the Removal of Non-Pharmaceutical Interventions.

OBJECTIVESS: To evaluate the joint impact of childhood vaccination rates and masking policies, in schools and workplaces, on community transmission and severe outcomes due to COVID-19. STUDY DESIGN: We utilized a stochastic, agent-based simulation of North Carolina, to evaluate the impact of 24 health policy decisions on overall incidence of disease, COVID-19 related hospitalization, and mortality from July 1, 2021-July 1, 2023. RESULTS: Universal mask removal in schools in January 2022 could lead to a 38.1-47%, 27.6-36.2%, and 15.9-19.7% increase in cumulative infections for ages 5-9, 10-19, and the total population, respectively, depending on the rate of vaccination of children relative to the adult population. Additionally, without increased vaccination uptake in the adult population, a 25% increase in child vaccination uptake from 50% to 75% uptake and from 75% to 100% uptake relative to the adult population, leads to a 22% and 18% or 28% and 33% decrease in peak hospitalizations in 2022 across scenarios when masks are removed either January 1st or March 8th 2022, respectively. Increasing vaccination uptake for the entire eligible population can reduce peak hospitalizations in 2022 by an average of 89% and 92% across all masking scenarios compared to the scenarios where no children are vaccinated. CONCLUSIONS: High vaccination uptake among both children and adults is necessary to mitigate the increase in infections from mask removal in schools and workplaces.

Mask Interventions in K12 Schools Can Also Reduce Community Transmission in Fall 2021.

The dominance of the COVID-19 Delta variant has renewed questions about the impact of K12 school policies, including the role of masks, on disease burden.1 A recent study showed masks and testing could reduce infections in students, but failed to address the impact on the community,2 while another showed masking is critical to slow disease spread in communities, but did not consider school openings under Delta.3 We project the impact of school-masking on the community, which can inform policy decisions, and support healthcare system planning. Our findings indicate that the implementation of masking policies in school settings can reduce additional infections post-school opening by 23-36% for fully-open schools, with an additional 11-13% reduction for hybrid schooling, depending on mask quality and fit. Masking policies and hybrid schooling can also reduce peak hospitalization need by 71% and result in the fewest additional deaths post-school opening. We show that given the current vaccination rates within the community, the best option for children and the general population is to employ consistent high-quality masking, and use social distancing where possible.

Association of Simulated COVID-19 Vaccination and Nonpharmaceutical Interventions With Infections, Hospitalizations, and Mortality.

Importance: Vaccination against SARS-CoV-2 has the potential to significantly reduce transmission and COVID-19 morbidity and mortality. The relative importance of vaccination strategies and nonpharmaceutical interventions (NPIs) is not well understood. Objective: To assess the association of simulated COVID-19 vaccine efficacy and coverage scenarios with and without NPIs with infections, hospitalizations, and deaths. Design, Setting, and Participants: An established agent-based decision analytical model was used to simulate COVID-19 transmission and progression from March 24, 2020, to September 23, 2021. The model simulated COVID-19 spread in North Carolina, a US state of 10.5 million people. A network of 1 017 720 agents was constructed from US Census data to represent the statewide population. Exposures: Scenarios of vaccine efficacy (50% and 90%), vaccine coverage (25%, 50%, and 75% at the end of a 6-month distribution period), and NPIs (reduced mobility, school closings, and use of face masks) maintained and removed during vaccine distribution. Main Outcomes and Measures: Risks of infection from the start of vaccine distribution and risk differences comparing scenarios. Outcome means and SDs were calculated across replications. Results: In the worst-case vaccination scenario (50% efficacy, 25% coverage), a mean (SD) of 2 231 134 (117 867) new infections occurred after vaccination began with NPIs removed, and a mean (SD) of 799 949 (60 279) new infections occurred with NPIs maintained during 11 months. In contrast, in the best-case scenario (90% efficacy, 75% coverage), a mean (SD) of 527 409 (40 637) new infections occurred with NPIs removed and a mean (SD) of 450 575 (32 716) new infections occurred with NPIs maintained. With NPIs removed, lower efficacy (50%) and higher coverage (75%) reduced infection risk by a greater magnitude than higher efficacy (90%) and lower coverage (25%) compared with the worst-case scenario (mean [SD] absolute risk reduction, 13% [1%] and 8% [1%], respectively). Conclusions and Relevance: Simulation outcomes suggest that removing NPIs while vaccines are distributed may result in substantial increases in infections, hospitalizations, and deaths. Furthermore, as NPIs are removed, higher vaccination coverage with less efficacious vaccines can contribute to a larger reduction in risk of SARS-CoV-2 infection compared with more efficacious vaccines at lower coverage. These findings highlight the need for well-resourced and coordinated efforts to achieve high vaccine coverage and continued adherence to NPIs before many prepandemic activities can be resumed.

High-Quality Masks Reduce COVID-19 Infections and Deaths in the US.

Objectives: To evaluate the effectiveness of widespread adoption of masks or face coverings to reduce community transmission of the SARS-CoV-2 virus that causes COVID-19. Methods: We created an agent-based stochastic network simulation using a variant of the standard SEIR dynamic infectious disease model. We considered a mask order that was initiated 3.5 months after the first confirmed COVID-19 case. We varied the likelihood of individuals wearing masks from 0-100% in steps of 20% (mask adherence) and considered 25% to 90% mask-related reduction in viral transmission (mask efficacy). Sensitivity analyses assessed early (by week 13) versus late (by week 42) adoption of masks and geographic differences in adherence (highest in urban and lowest in rural areas). Results: Introduction of mask use with 50% efficacy worn by 50% of individuals reduces the cumulative infection attack rate (IAR) by 27%, the peak prevalence by 49%, and population-wide mortality by 29%. If 90% of individuals wear 50% efficacious masks, this decreases IAR by 54%, peak prevalence by 75%, and population-wide mortality by 55%; similar improvements hold if 70% of individuals wear 75% efficacious masks. Late adoption reduces IAR and deaths by 18% or more compared to no adoption. Lower adoption in rural areas than urban would lead to rural areas having the highest IAR. Conclusions: Even after community transmission of SARS-CoV-2 has been established, adoption of mask-wearing by a majority of community-dwelling individuals can meaningfully reduce the number and outcome of COVID-19 infections over and above physical distancing interventions.

The Joint Impact of COVID-19 Vaccination and Non-Pharmaceutical Interventions on Infections, Hospitalizations, and Mortality: An Agent-Based Simulation.

Background: Vaccination against SARS-CoV-2 has the potential to significantly reduce transmission and morbidity and mortality due to COVID-19. This modeling study simulated the comparative and joint impact of COVID-19 vaccine efficacy and coverage with and without non-pharmaceutical interventions (NPIs) on total infections, hospitalizations, and deaths. Methods: An agent-based simulation model was employed to estimate incident SARS-CoV-2 infections and COVID-19-associated hospitalizations and deaths over 18 months for the State of North Carolina, a population of roughly 10.5 million. Vaccine efficacy of 50% and 90% and vaccine coverage of 25%, 50%, and 75% (at the end of a 6-month distribution period) were evaluated. Six vaccination scenarios were simulated with NPIs (i.e., reduced mobility, school closings, face mask usage) maintained and removed during the period of vaccine distribution. Results: In the worst-case vaccination scenario (50% efficacy and 25% coverage), 2,231,134 new SARS-CoV-2 infections occurred with NPIs removed and 799,949 infections with NPIs maintained. In contrast, in the best-case scenario (90% efficacy and 75% coverage), there were 450,575 new infections with NPIs maintained and 527,409 with NPIs removed. When NPIs were removed, lower efficacy (50%) and higher coverage (75%) reduced infection risk by a greater magnitude than higher efficacy (90%) and lower coverage (25%) compared to the worst-case scenario (absolute risk reduction 13% and 8%, respectively). Conclusion: Simulation results suggest that premature lifting of NPIs while vaccines are distributed may result in substantial increases in infections, hospitalizations, and deaths. Furthermore, as NPIs are removed, higher vaccination coverage with less efficacious vaccines can contribute to a larger reduction in risk of SARS-CoV-2 infection compared to more efficacious vaccines at lower coverage. Our findings highlight the need for well-resourced and coordinated efforts to achieve high vaccine coverage and continued adherence to NPIs before many pre-pandemic activities can be resumed.