Assessing the accuracy of California county level COVID-19 hospitalization forecasts to inform public policy decision making.

BACKGROUND: The COVID-19 pandemic has highlighted the role of infectious disease forecasting in informing public policy. However, significant barriers remain for effectively linking infectious disease forecasts to public health decision making, including a lack of model validation. Forecasting model performance and accuracy should be evaluated retrospectively to understand under which conditions models were reliable and could be improved in the future. METHODS: Using archived forecasts from the California Department of Public Health's California COVID Assessment Tool ( https://calcat.covid19.ca.gov/cacovidmodels/ ), we compared how well different forecasting models predicted COVID-19 hospitalization census across California counties and regions during periods of Alpha, Delta, and Omicron variant predominance. RESULTS: Based on mean absolute error estimates, forecasting models had variable performance across counties and through time. When accounting for model availability across counties and dates, some individual models performed consistently better than the ensemble model, but model rankings still differed across counties. Local transmission trends, variant prevalence, and county population size were informative predictors for determining which model performed best for a given county based on a random forest classification analysis. Overall, the ensemble model performed worse in less populous counties, in part because of fewer model contributors in these locations. CONCLUSIONS: Ensemble model predictions could be improved by incorporating geographic heterogeneity in model coverage and performance. Consistency in model reporting and improved model validation can strengthen the role of infectious disease forecasting in real-time public health decision making.

COVIDNearTerm: A simple method to forecast COVID-19 hospitalizations.

Introduction: COVID-19 has caused tremendous death and suffering since it first emerged in 2019. Soon after its emergence, models were developed to help predict the course of various disease metrics, and these models have been relied upon to help guide public health policy. Methods: Here we present a method called COVIDNearTerm to "forecast" hospitalizations in the short term, two to four weeks from the time of prediction. COVIDNearTerm is based on an autoregressive model and utilizes a parametric bootstrap approach to make predictions. It is easy to use as it requires only previous hospitalization data, and there is an open-source R package that implements the algorithm. We evaluated COVIDNearTerm on San Francisco Bay Area hospitalizations and compared it to models from the California COVID Assessment Tool (CalCAT). Results: We found that COVIDNearTerm predictions were more accurate than the CalCAT ensemble predictions for all comparisons and any CalCAT component for a majority of comparisons. For instance, at the county level our 14-day hospitalization median absolute percentage errors ranged from 16 to 36%. For those same comparisons, the CalCAT ensemble errors were between 30 and 59%. Conclusion: COVIDNearTerm is a simple and useful tool for predicting near-term COVID-19 hospitalizations.

Impacts of prescription drug monitoring program policy changes and county opioid safety coalitions on prescribing and overdose outcomes in California, 2015-2018.

In 2015, California received funding to implement the Prescription Drug Overdose Prevention Initiative, a 4-year program to reduce deaths involving prescription opioids by 1) leveraging improvements to California's prescription drug monitoring program (PDMP) (i.e., mandatory PDMP registration for prescribers and pharmacists), and 2) supporting county opioid safety coalitions. We used statewide data from 2011 to 2018 to evaluate the Initiative's impact on opioid prescribing and overdose rates. Prescribing data were obtained from California's PDMP; fatal and non-fatal overdose data were obtained from the California Department of Public Health. Outcomes were monthly opioid prescribing rates and opioid overdose rates, modeled using generalized linear mixed models. Exposures were mandatory PDMP registration, presence of county coalitions, and Initiative support for county coalitions. Mandatory PDMP registration was associated with a 25% decrease (95%CI, 0.71-0.79) in opioid prescribing rates after 24 months. Having a county coalition was associated with a 2% decrease (95%CI, 0.96-0.99) in the opioid prescribing rate; receiving Initiative support was associated with an additional 2% decrease (95%CI, 0.97-0.98). Mandatory PDMP registration and county coalitions were associated with a 35% decrease (95%CI, 0.43-0.97) and a 21% decrease (95% CI, 0.70-0.90), respectively in prescription opioid overdose deaths. Both interventions were also associated with significantly fewer deaths involving any opioid but had no significant association with non-fatal overdose rates. Findings add to the knowledge available to guide policy to prevent high-risk prescribing and opioid overdoses. While further study is needed, coalitions and mandatory PDMP registration may be important components in such efforts.