Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naive baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance StatementThis paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.

How Timing of Stay-home Orders and Mobility Reductions Impacted First-Wave COVID-19 Deaths in US Counties

As SARS-CoV-2 transmission continues to evolve, understanding how location-specific variations in non-pharmaceutical interventions and behaviors contributed to disease transmission during the initial epidemic wave will be key for future control strategies. We offer a rigorous statistical analysis of the relative effectiveness of the timing of both official stay-at-home orders and population mobility reductions during the initial stage of the US epidemic. We use a Bayesian hierarchical regression to fit county-level mortality data from the first case on Jan 21 2020 through Apr 20 2020 and quantify associations between the timing of stay-at-home orders and population mobility with epidemic control. We find that among 882 counties with an early local epidemic, a 10-day delay in the enactment of stay-at-home orders would have been associated with 14,700 additional deaths by Apr 20 (95% credible interval: 9,100, 21,500), whereas shifting orders 10 days earlier would have been associated with nearly 15,700 fewer lives lost (95% credible interval: 11,350, 18,950). Analogous estimates are available for reductions in mobility--which typically occurred before stay-at-home orders--and are also stratified by county urbanicity, showing significant heterogeneity. Results underscore the importance of timely policy and behavioral action for early-stage epidemic control.

The impact of asymptomatic COVID-19 infections on future pandemic waves

Letter textThe prevalence of asymptomatic COVID-19 infections is largely unknown and may determine the course of future pandemic waves and the effectiveness of interventions. Using an epidemiological model fit to COVID-19 hospitalization counts from New York City, New York and Austin, Texas, we found that the undocumented attack rate in the first pandemic wave depends on the proportion of asymptomatic infections but not on the infectiousness of such individuals. Based on a recent report that 22.7% of New Yorkers are seropositive for SARS-CoV-2, we estimate that 56% (95% CI: 53-59%) of COVID-19 infections are asymptomatic. Given uncertainty in the case hospitalization rate, however, the asymptomatic proportion could be as low as 20% or as high as 80%. We find that at most 1.26% of the Austin population was infected by April 27, 2020 and conclude that immunity from undetected infections is unlikely to slow future pandemic spread in most US cities in the summer of 2020.