Estimating the Effects of Non-Pharmaceutical Interventions and Population Mobility on Daily COVID-19 Cases: Evidence from Ontario.

This study uses coronavirus disease 2019 (COVID-19) case counts and Google mobility data for 12 of Ontario's largest Public Health Units from Spring 2020 until the end of January 2021 to evaluate the effects of non-pharmaceutical interventions (NPIs; policy restrictions on business operations and social gatherings) and population mobility on daily cases. Instrumental variables (IV) estimation is used to account for potential simultaneity bias, because both daily COVID-19 cases and NPIs are dependent on lagged case numbers. IV estimates based on differences in lag lengths to infer causal estimates imply that the implementation of stricter NPIs and indoor mask mandates are associated with reductions in COVID-19 cases. Moreover, estimates based on Google mobility data suggest that increases in workplace attendance are correlated with higher case counts. Finally, from October 2020 to January 2021, daily Ontario forecasts from Box-Jenkins time-series models are more accurate than official forecasts and forecasts from a susceptible-infected-removed epidemiology model.

Cette étude cherche à évaluer les effets des interventions non pharmaceutiques (INPs; restrictions sur les activités commerciales et rassemblements sociaux) et de la mobilité de la population sur le nombre de cas d'infection par jour, en utilisant les nombres de cas d'infection par la maladie à coronavirus 2019 (COVID-19) et les données de mobilité de Google pour 12 des plus grands Bureaux de Santé publique de l'Ontario entre le printemps 2020 et la fin janvier 2021. La méthode des variables instrumentales (VI) permet de rendre compte d'un biais potentiel de simultanéité puisque les taux quotidiens de COVID-19 et les INPs dépendent, tous les deux, du nombre de cas décalés. Les estimations par les VI basées sur les différences de durée des décalages d'ajustement pour inférer des estimations causales impliquent que de plus strictes INPs et le port obligatoire du masque dans les endroits fermés sont associés à une réduction de cas d'infection. Par ailleurs, Les estimations basées sur les données de mobilité de Google montrent que la présence accrue sur le lieu du travail est corrélée avec un plus grand nombre de cas d'infection. Finalement, d'octobre 2020 à Janvier 2021, les prévisions faites à partir de modèles de Box-Jenkins en série chronologique s'avèrent plus précises que les prévisions officielles et que celles utilisant le modèle épidémiologique susceptible ­ infecté ­ retiré.

An overview and critique of the use of cumulative sum methods with surgical learning curve data.

Cumulative sum (CUSUM) plots and methods have wide-ranging applications in healthcare. We review and discuss some issues related to the analysis of surgical learning curve (LC) data with a focus on three types of CUSUM statistical approaches. The underlying assumptions, benefits, and weaknesses of each approach are given. Our primary conclusion is that two types of CUSUM methods are useful in providing visual aids, but are subject to overinterpretation due to the lack of well-defined decision rules and performance metrics. The third type is based on plotting the CUSUM of the differences between observations and their average value. We show that this commonly applied retrospective method is frequently interpreted incorrectly and is thus unhelpful in the LC application. Curve-fitting methods are more suitable for meeting many of the goals associated with the study of surgical LCs.

Using telematics data to find risky driver behaviour.

Usage-based insurance schemes provide new opportunities for insurers to accurately price and manage risk. These schemes have the potential to better identify risky drivers which not only allows insurance companies to better price their products but it allows drivers to modify their behaviour to make roads safer and driving more efficient. However, for Usage-based insurance products, we need to better understand how driver behaviours influence the risk of a crash or an insurance claim. In this article, we present our analysis of automotive telematics data from over 28 million trips. We use a case control methodology to study the relationship between crash drivers and crash-free drivers and introduce an innovative method for determining control (crash-free) drivers. We fit a logistic regression model to our data and found that speeding was the most important driver behaviour linking driver behaviour to crash risk.

The k conditional nearest neighbor algorithm for classification and class probability estimation.

The k nearest neighbor (kNN) approach is a simple and effective nonparametric algorithm for classification. One of the drawbacks of kNN is that the method can only give coarse estimates of class probabilities, particularly for low values of k. To avoid this drawback, we propose a new nonparametric classification method based on nearest neighbors conditional on each class: the proposed approach calculates the distance between a new instance and the kth nearest neighbor from each class, estimates posterior probabilities of class memberships using the distances, and assigns the instance to the class with the largest posterior. We prove that the proposed approach converges to the Bayes classifier as the size of the training data increases. Further, we extend the proposed approach to an ensemble method. Experiments on benchmark data sets show that both the proposed approach and the ensemble version of the proposed approach on average outperform kNN, weighted kNN, probabilistic kNN and two similar algorithms (LMkNN and MLM-kHNN) in terms of the error rate. A simulation shows that kCNN may be useful for estimating posterior probabilities when the class distributions overlap.

Nearest labelset using double distances for multi-label classification.

Multi-label classification is a type of supervised learning where an instance may belong to multiple labels simultaneously. Predicting each label independently has been criticized for not exploiting any correlation between labels. In this article we propose a novel approach, Nearest Labelset using Double Distances (NLDD), that predicts the labelset observed in the training data that minimizes a weighted sum of the distances in both the feature space and the label space to the new instance. The weights specify the relative tradeoff between the two distances. The weights are estimated from a binomial regression of the number of misclassified labels as a function of the two distances. Model parameters are estimated by maximum likelihood. NLDD only considers labelsets observed in the training data, thus implicitly taking into account label dependencies. Experiments on benchmark multi-label data sets show that the proposed method on average outperforms other well-known approaches in terms of 0/1 loss, and multi-label accuracy and ranks second on the F-measure (after a method called ECC) and on Hamming loss (after a method called RF-PCT).

Comparing and Monitoring Risk-Adjusted Hospital Performance Measures: A Weighted Estimating Equations Approach.

Background. There is a great deal of interest in evaluating hospital performance in order to monitor and improve health care quality. Increasingly, risk-adjusted performance measures are available to the public and statistical approaches for estimating these measures are considered. Some methods in use currently are based on 3-year aggregates of data since a small number of cases may lead to imprecise estimates and make it hard for stakeholders to detect differences across hospitals over time. However, if quality changes over time, a measure based on these data is a biased estimate of present performance. Methods. We present an alternative approach (weighted estimating equations [WEE]) for combining historical data in estimation that regulates the tradeoff between bias and precision in the measure of present performance. The WEE approach uses all available historical data through estimating functions that down-weight past data. Results. We compare the WEE approach to two current practices using a realistic dataset of the mortality of patients following an elective percutaneous coronary intervention procedure in New York State who meet certain criteria. The width of the uncertainty interval in the realistic example is up to 65% smaller and the difference is more pronounced for hospitals with a small number of cases. Conclusions. The advantage of this approach extends from the example dataset to other datasets. The WEE approach uses all available data rather than data from an arbitrary 3-year window. The effect of borrowing strength from historical data is a more precise estimate of present performance than current practices. Its advantages are important for the comparison of other aspects of medical performance, including surgical or medical practitioner performance.

Comparing heteroscedastic measurement systems with the probability of agreement.

Deciding whether two measurement systems agree well enough to be used interchangeably is important in medical and clinical contexts. Recently, the probability of agreement was proposed as an alternative to comparison techniques such as correlation, regression, and the limits of agreement approach, when the systems' measurement errors are homoscedastic. However, in medical and clinical contexts, it is common for measurement variability to increase proportionally with the magnitude of measurement. In this article, we extend the probability of agreement analysis to accommodate heteroscedastic measurement errors, demonstrating the versatility of this simple metric. We illustrate its use with two examples: one involving the comparison of blood pressure measurement devices, and the other involving the comparison of serum cholesterol assays.

Assessing agreement between two measurement systems: An alternative to the limits of agreement approach.

The comparison of two measurement systems is important in medical and other contexts. A common goal is to decide if a new measurement system agrees suitably with an existing one, and hence whether the two can be used interchangeably. Various methods for assessing interchangeability are available, the most popular being the limits of agreement approach due to Bland and Altman. In this article, we review the challenges of this technique and propose a model-based framework for comparing measurement systems that overcomes those challenges. The proposal is based on a simple metric, the probability of agreement, and a corresponding plot which can be used to summarize the agreement between two measurement systems. We also make recommendations for a study design that facilitates accurate and precise estimation of the probability of agreement.

Debate: what is the best method to monitor surgical performance?

BACKGROUND: There is considerable recent interest in the monitoring of individual surgeon or hospital surgical outcomes. If one aggregates data over time and assesses performance with a funnel plot, then the detection of any process deterioration or improvement could be delayed. The variable life adjusted display (VLAD) is widely used for monitoring on a case-by-case basis, but we show that use of the risk-adjusted Bernoulli cumulative sum (RA-CUSUM) chart leads to much better performance. DISCUSSION: We use simulation to illustrate that the RA-CUSUM chart has better performance than the VLAD in detecting changes in the rates of adverse events. We recommend the RA-CUSUM approach over the VLAD approach for monitoring surgical performance. If the VLAD is used, we recommend running the RA-CUSUM chart in the background to generate signals that the process performance has changed.

Monitoring risk-adjusted medical outcomes allowing for changes over time.

We consider the problem of monitoring and comparing medical outcomes, such as surgical performance, over time. Performance is subject to change due to a variety of reasons including patient heterogeneity, learning, deteriorating skills due to aging, etc. For instance, we expect inexperienced surgeons to improve their skills with practice. We propose a graphical method to monitor surgical performance that incorporates risk adjustment to account for patient heterogeneity. The procedure gives more weight to recent outcomes and down-weights the influence of outcomes further in the past. The chart is clinically interpretable as it plots an estimate of the failure rate for a "standard" patient. The chart also includes a measure of uncertainty in this estimate. We can implement the method using historical data or start from scratch. As the monitoring proceeds, we can base the estimated failure rate on a known risk model or use the observed outcomes to update the risk model as time passes. We illustrate the proposed method with an example from cardiac surgery.

Assessing the effect of estimation error on risk-adjusted CUSUM chart performance.

BACKGROUND: Risk-adjusted control charts have become popular for monitoring processes that involve the management and treatment of patients in hospitals or other healthcare institutions. However, to date, the effect of estimation error on risk-adjusted control charts has not been studied. METHODS: We studied the effect of estimation error on risk-adjusted binary cumulative sum (CUSUM) performance using actual and simulated data on patients undergoing coronary artery bypass surgery and assessed for mortality up to 30 days post-surgery. The effect of estimation error was indicated by the variability of the 'true' average run lengths (ARLs) obtained using repeated sampling of the observed data under various realistic scenarios. RESULTS: Results showed that estimation error can have a substantial effect on risk-adjusted CUSUM chart performance in terms of variation of true ARLs. Moreover, the performance was highly dependent on the number of events used to derive the control chart parameters and the specified ARL for an in-control process (ARL(0)). However, the results suggest that it is the uncertainty in the overall adverse event rate that is the main component of estimation error. CONCLUSIONS: When designing a control chart, the effect of estimation error could be taken into account by generating a number of bootstrap samples of the available Phase I data and then determining the control limit needed to obtain an ARL(0) of a pre-specified level 95% of the time. If limited Phase I data are available, it may be advisable to continue to update model parameters even after prospective patient monitoring is implemented.

Monitoring binary outcomes using risk-adjusted charts: a comparative study.

Monitoring binary outcomes when evaluating health care performance has recently become common. Classical statistical methodologies such as cumulative sum (CUSUM) charts have been refined and used for this purpose. For instance, the risk-adjusted CUSUM chart (RA-CUSUM) for monitoring binary outcomes was proposed for monitoring 30-day mortality following cardiac surgery. The RA-CUSUM inherits optimality properties of the original CUSUM charts in the sense of signaling early when there is change. However, although the RA-CUSUM is a powerful monitoring tool, it will always eventually signal a change with probability 1 even when there is no real change. In other words, the probability of a type I error for the RA-CUSUM is 1. It also turns out that, because of the skewed distribution of the run lengths of the RA-CUSUM, the median is often well below the mean, and as a consequence more than half of all its false alarms occur before the designed average run length. In addition, when the change to be detected occurs at a later time in the series of observations being monitored, the rate of false alarms increases, and the RA-CUSUM may not be appropriate. Therefore, if the price of false alarms is high, it is preferable to use methods that control the rate of false alarms. In this paper, we propose alternative sequential curtailed and risk-adjusted charts that control the type I error rate in the context of monitoring 30-day mortality following cardiac surgery. We explore the merits of each of these methodologies in terms of average run lengths as well as in terms of type I error probabilities, and we compare them to the RA-CUSUM chart. We illustrate the methodologies by using data on monitoring performance of seven surgeons from a medical center.

Monitoring radiation use in cardiac fluoroscopy imaging procedures.

PURPOSE: Timely identification of systematic changes in radiation delivery of an imaging system can lead to a reduction in risk for the patients involved. However, existing quality assurance programs involving the routine testing of equipment performance using phantoms are limited in their ability to effectively carry out this task. To address this issue, the authors propose the implementation of an ongoing monitoring process that utilizes procedural data to identify unexpected large or small radiation exposures for individual patients, as well as to detect persistent changes in the radiation output of imaging platforms. METHODS: Data used in this study were obtained from records routinely collected during procedures performed in the cardiac catheterization imaging facility at St. Andrew's War Memorial Hospital, Brisbane, Australia, over the period January 2008-March 2010. A two stage monitoring process employing individual and exponentially weighted moving average (EWMA) control charts was developed and used to identify unexpectedly high or low radiation exposure levels for individual patients, as well as detect persistent changes in the radiation output delivered by the imaging systems. To increase sensitivity of the charts, we account for variation in dose area product (DAP) values due to other measured factors (patient weight, fluoroscopy time, and digital acquisition frame count) using multiple linear regression. Control charts are then constructed using the residual values from this linear regression. The proposed monitoring process was evaluated using simulation to model the performance of the process under known conditions. RESULTS: Retrospective application of this technique to actual clinical data identified a number of cases in which the DAP result could be considered unexpected. Most of these, upon review, were attributed to data entry errors. The charts monitoring the overall system radiation output trends demonstrated changes in equipment performance associated with relocation of the equipment to a new department. When tested under simulated conditions, the EWMA chart was capable of detecting a sustained 15% increase in average radiation output within 60 cases (<1 month of operation), while a 33% increase would be signaled within 20 cases. CONCLUSIONS: This technique offers a valuable enhancement to existing quality assurance programs in radiology that rely upon the testing of equipment radiation output at discrete time frames to ensure performance security.

Detecting the start of an influenza outbreak using exponentially weighted moving average charts.

BACKGROUND: Influenza viruses cause seasonal outbreaks in temperate climates, usually during winter and early spring, and are endemic in tropical climates. The severity and length of influenza outbreaks vary from year to year. Quick and reliable detection of the start of an outbreak is needed to promote public health measures. METHODS: We propose the use of an exponentially weighted moving average (EWMA) control chart of laboratory confirmed influenza counts to detect the start and end of influenza outbreaks. RESULTS: The chart is shown to provide timely signals in an example application with seven years of data from Victoria, Australia. CONCLUSIONS: The EWMA control chart could be applied in other applications to quickly detect influenza outbreaks.

Risk-adjusted survival time monitoring with an updating exponentially weighted moving average (EWMA) control chart.

Monitoring medical outcomes is desirable to help quickly detect performance changes. Previous applications have focused mostly on binary outcomes, such as 30-day mortality after surgery. However, in many applications the survival time data are routinely collected. In this paper, we propose an updating exponentially weighted moving average (EWMA) control chart to monitor risk-adjusted survival times. The updating EWMA (uEWMA) operates in a continuous time; hence, the scores for each patient always reflect the most up-to-date information. The uEWMA can be implemented based on a variety of survival-time models and can be set up to provide an ongoing estimate of a clinically interpretable average patient score. The efficiency of the uEWMA is shown to compare favorably with the competing methods.

Optimal two-stage reliability studies.

The intraclass correlation is often used to assess the reliability of a measurement system. There is a considerable literature devoted to optimizing the standard assessment plan in which a number of subjects are measured repeatedly. We propose a two-stage investigation, here called a leveraged plan (LP), where in Stage I, we measure a number of subjects once. Then in Stage II, we select a subset of subjects with extreme initial measurements for repeated measurement. For a fixed total number of measurements, we show that the optimal LP provides a more precise estimate of the intraclass correlation coefficient than does the optimal standard plan (SP). We provide a table for finding the optimal LP given the true intraclass correlation and a specified precision for the estimate. For a fixed total number of measurements N, a nearly optimal LP makes roughly N/2 measurements in Stage I and then selects roughly N/6 extreme subjects to re-measure thrice each in Stage II. We also compare optimal leveraged with optimal SPs when there is a limit on the number of times each subject can be re-measured.

Risk-adjusted analysis of early mortality after ruptured abdominal aortic aneurysm repair.

PURPOSE: Ruptured abdominal aortic aneurysms (RAAAs) continue to result in early mortality in up to 50% of patients. Additionally, it remains difficult to compare outcomes given the variability in patient comorbidities and presentation. The purpose of this study was to describe an instrument that permits the prospective analysis of outcomes after RAAA repair while adjusting for the variability in preoperative risk. METHODS: Consecutive patients undergoing attempted open RAAA repair over a 5-year period (1999 to 2003) at our center were reviewed. Thirty-day or in-hospital mortality was the main outcome variable. Preoperative mortality risk was estimated for each patient by using a validated modification of the POSSUM scoring system (V-POSSUM). A risk-adjusted cumulative sum method (RA-CUSUM) was used to compare observed versus predicted outcomes by assigning a risk-adjusted score, based on log-likelihood ratios, to each patient. These scores were sequentially plotted with preset control limits to allow for "signaling" when results were substantially different from expected (doubling or halving of odds ratios). RESULTS: A total of 136 patients were reviewed, with an early mortality rate of 45.6%. V-POSSUM scores were accurate in predicting mortality for the entire cohort, with an observed-to-predicted mortality ratio of 0.92 (P = .80). Each patient's risk-adjusted score was plotted sequentially. In one segment of the resulting plot, the graph adopted a negative slope and crossed the lower control limit, indicating improved results compared with predicted. CONCLUSIONS: V-POSSUM scores in this series accurately predicted early mortality after RAAA surgery. The RA-CUSUM method allows for the prospective evaluation of outcomes, while taking into account patient variability. In the current study, this resulted in the identification of a series of patients who had improved outcomes compared with predicted.

Risk-adjusted analysis of outcomes following elective open abdominal aortic aneurysm repair.

The purpose of this study was to describe a method to analyze outcomes following open abdominal aortic aneurysm (AAA) repair while considering the variability in patients' preoperative risk. Consecutive patients undergoing elective open infrarenal AAA repair during a 4-year period (2000-2003) were reviewed. Thirty-day or in-hospital mortality was the major outcome variable. Preoperative mortality risk was estimated for each patient using a validated scoring system that considers age, renal dysfunction, and coronary artery and cerebrovascular disease. A risk-adjusted cumulative sum method was used to compare observed versus predicted outcomes by assigning a risk-adjusted score, based on log-likelihood ratios, to each patient. These cumulative scores were sequentially plotted with preset control limits to allow for "signaling" when results were substantially different than expected (doubling or halving of odds ratios). Four hundred and sixty-three patients were studied with an overall early mortality rate of 4.5% (n = 21). Patients were allocated to three different preoperative risk groups (low, n = 89; medium, n = 160; high, n = 214) according to a medical comorbidity-based scoring system. Predicted (P) and observed (O) mortality rates for each group were as follows: low, 2.4% (P) and 2.2% (O); medium, 4.1% (P) and 4.4% (O); high, 9.3% (P) and 5.6% (O). The resulting risk-adjusted scores for each patient were plotted sequentially. This plot was flat for the first year and then adopted a negative slope crossing the lower control limit after 266 patients, indicating improved results compared to those expected. This coincided with the adoption of routine intraoperative cell saver use in our practice. This form of analysis allows for the prospective evaluation of results while considering patient-mix variabilities.

Monitoring the evolutionary process of quality: risk-adjusted charting to track outcomes in intensive care.

OBJECTIVE: To present graphical procedures for prospectively monitoring outcomes in the intensive care unit. DESIGN: Observational study: risk-adjusted control chart analysis of a case series. SETTING: Tertiary referral adult intensive care unit: Princess Alexandra Hospital, Brisbane, Australia. PATIENTS: A total of 3398 intensive care unit admissions from January 1, 1995, to January 1, 1998. CONCLUSIONS: Risk-adjusted process control charting procedures for continuous monitoring of intensive care unit outcomes are proposed as quality management tools. A modified Shewhart p chart and cumulative sum process control chart, using the Acute Physiology and Chronic Health Evaluation III model mortality prediction for risk adjustment, are presented. The risk-adjusted p chart summarizes performance at arbitrary intervals and plots observed against predicted mortality rate to detect large changes in risk-adjusted mortality. The risk-adjusted cumulative sum procedure is a likelihood-based scoring method that adjusts for estimated risk of death, accumulating evidence from outcomes of all previous patients. It formally tests the hypothesis of a change in the odds of death. In this application, we detected a decrease from above to predicted risk-adjusted mortality. This was temporally related to increased senior staffing levels and enhanced ongoing multidisciplinary review of practice, quality improvement, and educational activities. Formulas and analyses are provided as appendices.