THEMIS: A Framework for Cost-Benefit Analysis of COVID-19 Non-Pharmaceutical Interventions

Since December 2019, the world has been ravaged by the COVID-19 pandemic, with over 150 million confirmed cases and 3 million confirmed deaths worldwide. To combat the spread of COVID-19, governments have issued unprecedented non-pharmaceutical interventions (NPIs), ranging from mass gathering restrictions to complete lockdowns. Despite their proven effectiveness in reducing virus transmission, the policies often carry significant economic and humanitarian cost, ranging from unemployment to depression, PTSD, and anxiety. In this paper, we create a data-driven system dynamics framework, THEMIS, that allows us to compare the costs and benefits of a large class of NPIs in any geographical region across different cost dimensions. As a demonstration, we analyzed thousands of alternative policies across 5 countries (United States, Germany, Brazil, Singapore, Spain) and compared with the actual implemented policy. Our results show that moderate NPIs (such as restrictions on mass gatherings) usually produce the worst results, incurring significant cost while unable to sufficiently slow down the pandemic to prevent the virus from becoming endemic. Short but severe restrictions (complete lockdown for 4-5 weeks) generally produced the best results for developed countries, but only if the speed of reopening is slow enough to prevent a resurgence. Developing countries exhibited very different trade-off profiles from developed countries, and suggests that severe NPIs such as lockdowns might not be as suitable for developing countries in general.

National and subnational short-term forecasting of COVID-19 in Germany and Poland, early 2021

BackgroundDuring the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021. MethodsWe evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess forecast calibration. The presented work is part of a pre-registered evaluation study and covers the period from January through April 2021. ResultsWe find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods (i.e., combinations of different available forecasts) show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (alpha) variant in March 2021, prove challenging to predict. ConclusionsMulti-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance. Plain language summaryThe goal of this study is to assess the quality of forecasts of weekly case and death numbers of COVID-19 in Germany and Poland during the period of January through April 2021. We focus on real-time forecasts at time horizons of one and two weeks ahead created by fourteen independent teams. Forecasts are systematically evaluated taking uncertainty ranges of predictions into account. We find that combining different forecasts into ensembles can improve the quality of predictions, but especially case numbers proved very challenging to predict beyond quite short time windows. Additional data sources, in particular genetic sequencing data, may help to improve forecasts in the future.

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the US

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multi-model ensemble forecast that combined predictions from dozens of different research groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naive baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-week horizon 3-5 times larger than when predicting at a 1-week horizon. This project underscores the role that collaboration and active coordination between governmental public health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks. Significance StatementThis paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the US. Results show high variation in accuracy between and within stand-alone models, and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public health action.

Short-term forecasting of COVID-19 in Germany and Poland during the second wave - a preregistered study

We report insights from ten weeks of collaborative COVID-19 forecasting for Germany and Poland (12 October - 19 December 2020). The study period covers the onset of the second wave in both countries, with tightening non-pharmaceutical interventions (NPIs) and subsequently a decay (Poland) or plateau and renewed increase (Germany) in reported cases. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths. These were reported for lead times of one to four weeks, with evaluation focused on one- and two-week horizons, which are less affected by changing NPIs. Heterogeneity between forecasts was considerable both in terms of point predictions and forecast spread. Ensemble forecasts showed good relative performance, in particular in terms of coverage, but did not clearly dominate single-model predictions. The study was preregistered and will be followed up in future phases of the pandemic.

Optimizing Vaccine Allocation to Combat the COVID-19 Pandemic

The outbreak of COVID-19 has spurred extensive research worldwide to develop a vaccine. However, when a vaccine becomes available, limited production and distribution capabilities will likely lead to another challenge: who to prioritize for vaccination to mitigate the near-end impact of the pandemic? To tackle that question, this paper first expands a state-of-the-art epidemiological model, called DELPHI, to capture the effects of vaccinations and the variability in mortality rates across subpopulations. It then integrates this predictive model into a prescriptive model to optimize vaccine allocation, formulated as a bilinear, non-convex optimization model. To solve it, this paper proposes a coordinate descent algorithm that iterates between optimizing vaccine allocations and simulating the dynamics of the pandemic. We implement the model and algorithm using real-world data in the United States. All else equal, the optimized vaccine allocation prioritizes states with a large number of projected cases and sub-populations facing higher risks (e.g., older ones). Ultimately, the optimized vaccine allocation can reduce the death toll of the pandemic by an estimated 10-25%, or 10,000-20,000 deaths over a three-month period in the United States alone. Highlights- This paper formulates an optimization model for vaccine allocation in response to the COVID-19 pandemic. This model, referred to as DELPHI-V-OPT, integrates a predictive epidemiological model into a prescriptive model to support the allocation of vaccines across geographic regions (e.g., US states) and across risk classes (e.g., age groups). - This paper develops a scalable coordinate descent algorithm to solve the DELPHI-V-OPT model. The proposed algorithm converges effectively and in short computational times. Therefore, the proposed approach can be implemented efficiently, and allows extensive sensitivity analyses for scenario planning and policy analysis. - Computational results demonstrate that optimized vaccine allocation strategies can curb the death toll of the COVID-19 pandemic by an estimated at 10-25%, or 10,000-20,000 deaths over a three-month period in the United States alone. These results highlight the critical role of vaccine allocation to combat the COVID-19 pandemic, in addition to vaccine design and vaccine production.

Personalized Prescription of ACEI/ARBs for Hypertensive COVID-19 Patients

The COVID-19 pandemic has prompted an international effort to develop and repurpose medications and procedures to effectively combat the disease. Several groups have focused on the potential treatment utility of angiotensin-converting-enzyme inhibitors (ACEIs) and angiotensin-receptor blockers (ARBs) for hypertensive COVID-19 patients, with inconclusive evidence thus far. We couple electronic medical record (EMR) and registry data of 3,643 patients from Spain, Italy, Germany, Ecuador, and the US with a machine learning framework to personalize the prescription of ACEIs and ARBs to hypertensive COVID-19 patients. Our approach leverages clinical and demographic information to identify hospitalized individuals whose probability of mortality or morbidity can decrease by prescribing this class of drugs. In particular, the algorithm proposes increasing ACEI/ARBs prescriptions for patients with cardiovascular disease and decreasing prescriptions for those with low oxygen saturation at admission. We show that personalized recommendations can improve patient outcomes by 1.0% compared to the standard of care when applied to external populations. We develop an interactive interface for our algorithm, providing physicians with an actionable tool to easily assess treatment alternatives and inform clinical decisions. This work offers the first personalized recommendation system to accurately evaluate the efficacy and risks of prescribing ACEIs and ARBs to hypertensive COVID-19 patients. Highlights- This paper introduces a data-driven approach for personalizing the prescription of ACE inhibitors (ACEIs) and angiotensin-receptor blockers (ARBs) for hypertensive COVID-19 patients. - Leveraging an international cohort of more than 3,500 patients, we identify clinical and demographic characteristics that may affect the effectiveness of ACEIs/ARBs for COVID-19 patients, such as low oxygen saturation at admission. - We developed a user-friendly online application that is available to physicians to facilitate interpretation and communication of the results of the algorithm.

COVID-19 Mortality Risk Assessment: An International Multi-Center Study

BackgroundTimely identification of COVID-19 patients at high risk of mortality can significantly improve patient management and resource allocation within hospitals. This study seeks to develop and validate a data-driven personalized mortality risk calculator for hospitalized COVID-19 patients. MethodsDe-identified data was obtained for 3,927 COVID-19 positive patients from six independent centers, comprising 33 different hospitals. Demographic, clinical, and laboratory variables were collected at hospital admission. The COVID-19 Mortality Risk (CMR) tool was developed using the XGBoost algorithm to predict mortality. Its discrimination performance was subsequently evaluated on three validation cohorts. FindingsThe derivation cohort of 3,062 patients has an observed mortality rate of 26.84%. Increased age, decreased oxygen saturation ([≤] 93%), elevated levels of C-reactive protein ([≥] 130 mg/L), blood urea nitrogen ([≥] 18 mg/dL), and blood creatinine ([≥] 1.2 mg/dL) were identified as primary risk factors, validating clinical findings. The model obtains out-of-sample AUCs of 0.90 (95% CI, 0.87-0.94) on the derivation cohort. In the validation cohorts, the model obtains AUCs of 0.92 (95% CI, 0.88-0.95) on Seville patients, 0.87 (95% CI, 0.84-0.91) on Hellenic COVID-19 Study Group patients, and 0.81 (95% CI, 0.76-0.85) on Hartford Hospital patients. The CMR tool is available as an online application at covidanalytics.io/mortality_calculator and is currently in clinical use. InterpretationThe CMR model leverages machine learning to generate accurate mortality predictions using commonly available clinical features. This is the first risk score trained and validated on a cohort of COVID-19 patients from Europe and the United States. Research in contextO_ST_ABSEvidence before this studyC_ST_ABSWe searched PubMed, BioRxiv, MedRxiv, arXiv, and SSRN for peer-reviewed articles, preprints, and research reports in English from inception to March 25th, 2020 focusing on disease severity and mortality risk scores for patients that had been infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Earlier investigations showed promise at predicting COVID-19 disease severity using data at admission. However, existing work was limited by its data scope, either relying on a single center with rich clinical information or broader cohort with sparse clinical information. No analysis has leveraged Electronic Health Records data from an international multi-center cohort from both Europe and the United States. Added value of this studyWe present the first multi-center COVID-19 mortality risk study that uses Electronic Health Records data from 3,062 patients across four different countries, including Greece, Italy, Spain, and the United States, encompassing 33 hospitals. We employed state-of-the-art machine learning techniques to develop a personalized COVID-19 mortality risk (CMR) score for hospitalized patients upon admission based on clinical features including vitals, lab results, and comorbidities. The model validates clinical findings of mortality risk factors and exhibits strong performance, with AUCs ranging from 0.81 to 0.92 across external validation cohorts. The model identifies increased age as a primary mortality predictor, consistent with observed disease trends and subsequent public health guidelines. Additionally, among the vital and lab values collected at admission, decreased oxygen saturation ([≤] 93%) and elevated levels of C-reactive protein ([≥] 130 mg/L), blood urea nitrogen ([≥] 18 mg/dL), blood creatinine ([≥] 1.2 mg/dL), and blood glucose ([≥]180 mg/dL) are highlighted as key biomarkers of mortality risk. These findings corroborate previous studies that link COVID-19 severity to hypoxemia, impaired kidney function, and diabetes. These features are also consistent with risk factors used in severity risk scores for related respiratory conditions such as community-acquired pneumonia. Implications of all the available evidenceOur work presents the development and validation of a personalized mortality risk score. We take a data-driven approach to derive insights from Electronic Health Records data spanning Europe and the United States. While many existing papers on COVID-19 clinical characteristics and risk factors are based on Chinese hospital data, the similarities in our findings suggest consistency in the disease characteristics across international cohorts. Additionally, our machine learning model offers a novel approach to understanding the disease and its risk factors. By creating a single comprehensive risk score that integrates various admission data components, the calculator offers a streamlined way of evaluating COVID-19 patients upon admission to augment clinical expertise. The CMR model provides a valuable clinical decision support tool for patient triage and care management, improving risk estimation early within admission, that can significantly affect the daily practice of physicians.

From predictions to prescriptions: A data-drivenresponse to COVID-19

The COVID-19 pandemic has created unprecedented challenges worldwide. Strained healthcare providers make difficult decisions on patient triage, treatment and care management on a daily basis. Policy makers have imposed social distancing measures to slow the disease, at a steep economic price. We design analytical tools to support these decisions and combat the pandemic. Specifically, we propose a comprehensive data-driven approach to understand the clinical characteristics of COVID-19, predict its mortality, forecast its evolution, and ultimately alleviate its impact. By leveraging cohort-level clinical data, patient-level hospital data, and census-level epidemiological data, we develop an integrated four-step approach, combining descriptive, predictive and prescriptive analytics. First, we aggregate hundreds of clinical studies into the most comprehensive database on COVID-19 to paint a new macroscopic picture of the disease. Second, we build personalized calculators to predict the risk of infection and mortality as a function of demographics, symptoms, comorbidities, and lab values. Third, we develop a novel epidemiological model to project the pandemics spread and inform social distancing policies. Fourth, we propose an optimization model to reallocate ventilators and alleviate shortages. Our results have been used at the clinical level by several hospitals to triage patients, guide care management, plan ICU capacity, and re-distribute ventilators. At the policy level, they are currently supporting safe back-to-work policies at a major institution and equitable vaccine distribution planning at a major pharmaceutical company, and have been integrated into the US Center for Disease Controls pandemic forecast. Significance StatementIn the midst of the COVID-19 pandemic, healthcare providers and policy makers are wrestling with unprecedented challenges. How to treat COVID-19 patients with equipment shortages? How to allocate resources to combat the disease? How to plan for the next stages of the pandemic? We present a data-driven approach to tackle these challenges. We gather comprehensive data from various sources, including clinical studies, electronic medical records, and census reports. We develop algorithms to understand the disease, predict its mortality, forecast its spread, inform social distancing policies, and re-distribute critical equipment. These algorithms provide decision support tools that have been deployed on our publicly available website, and are actively used by hospitals, companies, and policy makers around the globe.

Forecasting COVID-19 and Analyzing the Effect of Government Interventions

One key question in the ongoing COVID-19 pandemic is understanding the impact of government interventions, and when society can return to normal. To this end, we develop DELPHI, a novel epidemiological model that captures the effect of under-detection and government intervention. We applied DELPHI across 167 geographical areas since early April, and recorded 6% and 11% two-week out-of-sample Median Absolute Percentage Error on cases and deaths respectively. Furthermore, DELPHI successfully predicted the large-scale epidemics in many areas months before, including US, UK and Russia. Using our flexible formulation of government intervention in DELPHI, we are able to understand how government interventions impacted the pandemics spread. In particular, DELPHI predicts that in absence of any interventions, over 14 million individuals would have perished by May 17th, while 280,000 current deaths could have been avoided if interventions around the world started one week earlier. Furthermore, we find mass gathering restrictions and school closings on average reduced infection rates the most, at 29.9 {+/-} 6.9% and 17.3 {+/-} 6.7%, respectively. The most stringent policy, stay-at-home, on average reduced the infection rate by 74.4 {+/-} 3.7% from baseline across countries that implemented it. We also illustrate how DELPHI can be extended to provide insights on reopening societies under different policies.