COVID RADAR app: Description and validation of population surveillance of symptoms and behavior in relation to COVID-19.

BACKGROUND: Monitoring of symptoms and behavior may enable prediction of emerging COVID-19 hotspots. The COVID Radar smartphone app, active in the Netherlands, allows users to self-report symptoms, social distancing behaviors, and COVID-19 status daily. The objective of this study is to describe the validation of the COVID Radar. METHODS: COVID Radar users are asked to complete a daily questionnaire consisting of 20 questions assessing their symptoms, social distancing behavior, and COVID-19 status. We describe the internal and external validation of symptoms, behavior, and both user-reported COVID-19 status and state-reported COVID-19 case numbers. RESULTS: Since April 2nd, 2020, over 6 million observations from over 250,000 users have been collected using the COVID Radar app. Almost 2,000 users reported having tested positive for SARS-CoV-2. Amongst users testing positive for SARS-CoV-2, the proportion of observations reporting symptoms was higher than that of the cohort as a whole in the week prior to a positive SARS-CoV-2 test. Likewise, users who tested positive for SARS-CoV-2 showed above average risk social-distancing behavior. Per-capita user-reported SARS-CoV-2 positive tests closely matched government-reported per-capita case counts in provinces with high user engagement. DISCUSSION: The COVID Radar app allows voluntarily self-reporting of COVID-19 related symptoms and social distancing behaviors. Symptoms and risk behavior increase prior to a positive SARS-CoV-2 test, and user-reported case counts match closely with nationally-reported case counts in regions with high user engagement. These results suggest the COVID Radar may be a valid instrument for future surveillance and potential predictive analytics to identify emerging hotspots.

Real-time imputation of missing predictor values improved the application of prediction models in daily practice.

OBJECTIVES: In clinical practice, many prediction models cannot be used when predictor values are missing. We, therefore, propose and evaluate methods for real-time imputation. STUDY DESIGN AND SETTING: We describe (i) mean imputation (where missing values are replaced by the sample mean), (ii) joint modeling imputation (JMI, where we use a multivariate normal approximation to generate patient-specific imputations), and (iii) conditional modeling imputation (CMI, where a multivariable imputation model is derived for each predictor from a population). We compared these methods in a case study evaluating the root mean squared error (RMSE) and coverage of the 95% confidence intervals (i.e., the proportion of confidence intervals that contain the true predictor value) of imputed predictor values. RESULTS: -RMSE was lowest when adopting JMI or CMI, although imputation of individual predictors did not always lead to substantial improvements as compared to mean imputation. JMI and CMI appeared particularly useful when the values of multiple predictors of the model were missing. Coverage reached the nominal level (i.e., 95%) for both CMI and JMI. CONCLUSION: Multiple imputations using either CMI or JMI is recommended when dealing with missing predictor values in real-time settings.

Real-time imputation of missing predictor values in clinical practice.

Aims: Use of prediction models is widely recommended by clinical guidelines, but usually requires complete information on all predictors, which is not always available in daily practice. We aim to describe two methods for real-time handling of missing predictor values when using prediction models in practice. Methods and results: We compare the widely used method of mean imputation (M-imp) to a method that personalizes the imputations by taking advantage of the observed patient characteristics. These characteristics may include both prediction model variables and other characteristics (auxiliary variables). The method was implemented using imputation from a joint multivariate normal model of the patient characteristics (joint modelling imputation; JMI). Data from two different cardiovascular cohorts with cardiovascular predictors and outcome were used to evaluate the real-time imputation methods. We quantified the prediction model's overall performance [mean squared error (MSE) of linear predictor], discrimination (c-index), calibration (intercept and slope), and net benefit (decision curve analysis). When compared with mean imputation, JMI substantially improved the MSE (0.10 vs. 0.13), c-index (0.70 vs. 0.68), and calibration (calibration-in-the-large: 0.04 vs. 0.06; calibration slope: 1.01 vs. 0.92), especially when incorporating auxiliary variables. When the imputation method was based on an external cohort, calibration deteriorated, but discrimination remained similar. Conclusions: We recommend JMI with auxiliary variables for real-time imputation of missing values, and to update imputation models when implementing them in new settings or (sub)populations.