Aggregating probabilistic predictions of the safety, efficacy, and timing of a COVID-19 vaccine

Safe, efficacious vaccines were developed to reduce the transmission of SARS-CoV-2 during the COVID-19 pandemic. But in the middle of 2020, vaccine effectiveness, safety, and the timeline for when a vaccine would be approved and distributed to the public was uncertain. To support public health decision making, we solicited trained forecasters and experts in vaccinology and infectious disease to provide monthly probabilistic predictions from July to September of 2020 of the efficacy, safety, timing, and delivery of a COVID-19 vaccine. We found, that despite sparse historical data, a consensus--a combination of human judgment probabilistic predictions--can quantify the uncertainty in clinical significance and timing of a potential vaccine. The consensus underestimated how fast a therapy would show a survival benefit and the high efficacy of approved COVID-19 vaccines. However, the consensus did make an accurate prediction for when a vaccine would be approved by the FDA. Compared to individual forecasters, the consensus was consistently above the 50th percentile of the most accurate forecasts. A consensus is a fast and versatile method to build probabilistic predictions of a developing vaccine that is robust to poor individual predictions. Though experts and trained forecasters did underestimate the speed of development and the high efficacy of a SARS-CoV-2 vaccine, consensus predictions can improve situational awareness for public health officials and for the public make clearer the risks, rewards, and timing of a vaccine.

An expert judgment model to predict early stages of the COVID-19 outbreak in the United States

During early stages of the COVID-19 pandemic, forecasts provided actionable information about disease transmission to public health decision-makers. Between February and May 2020, experts in infectious disease modeling made weekly predictions about the impact of the pandemic in the U.S. We aggregated these predictions into consensus predictions. In March and April 2020, experts predicted that the number of COVID-19 related deaths in the U.S. by the end of 2020 would be in the range of 150,000 to 250,000, with scenarios of near 1m deaths considered plausible. The wide range of possible future outcomes underscored the uncertainty surrounding the outbreak's trajectory. Experts' predictions of measurable short-term outcomes had varying levels of accuracy over the surveys but showed appropriate levels of uncertainty when aggregated. An expert consensus model can provide important insight early on in an emerging global catastrophe.