in silico surveillance: evaluating outbreak detection with simulation models.

BACKGROUND: Detecting outbreaks is a crucial task for public health officials, yet gaps remain in the systematic evaluation of outbreak detection protocols. The authors' objectives were to design, implement, and test a flexible methodology for generating detailed synthetic surveillance data that provides realistic geographical and temporal clustering of cases and use to evaluate outbreak detection protocols. METHODS: A detailed representation of the Boston area was constructed, based on data about individuals, locations, and activity patterns. Influenza-like illness (ILI) transmission was simulated, producing 100 years of in silico ILI data. Six different surveillance systems were designed and developed using gathered cases from the simulated disease data. Performance was measured by inserting test outbreaks into the surveillance streams and analyzing the likelihood and timeliness of detection. RESULTS: Detection of outbreaks varied from 21% to 95%. Increased coverage did not linearly improve detection probability for all surveillance systems. Relaxing the decision threshold for signaling outbreaks greatly increased false-positives, improved outbreak detection slightly, and led to earlier outbreak detection. CONCLUSIONS: Geographical distribution can be more important than coverage level. Detailed simulations of infectious disease transmission can be configured to represent nearly any conceivable scenario. They are a powerful tool for evaluating the performance of surveillance systems and methods used for outbreak detection.

Gastrointestinal disease outbreak detection using multiple data streams from electronic medical records.

BACKGROUND: Passive reporting and laboratory testing delays may limit gastrointestinal (GI) disease outbreak detection. Healthcare systems routinely collect clinical data in electronic medical records (EMRs) that could be used for surveillance. This study's primary objective was to identify data streams from EMRs that may perform well for GI outbreak detection. METHODS: Zip code-specific daily episode counts in 2009 were generated for 22 syndromic and laboratory-based data streams from Kaiser Permanente Northern California EMRs, covering 3.3 million members. Data streams included outpatient and inpatient diagnosis codes, antidiarrheal medication dispensings, stool culture orders, and positive microbiology tests for six GI pathogens. Prospective daily surveillance was mimicked using the space-time permutation scan statistic in single and multi-stream analyses, and space-time clusters were identified. Serotype relatedness was assessed for isolates in two Salmonella clusters. RESULTS: Potential outbreaks included a cluster of 18 stool cultures ordered over 5 days in one zip code and a Salmonella cluster in three zip codes over 9 days, in which at least five of six cases had the same rare serotype. In all, 28 potential outbreaks were identified using single stream analyses, with signals in outpatient diagnosis codes most common. Multi-stream analyses identified additional potential outbreaks and in one example, improved the timeliness of detection. CONCLUSIONS: GI disease-related data streams can be used to identify potential outbreaks when generated from EMRs with extensive regional coverage. This process can supplement traditional GI outbreak reports to health departments, which frequently consist of outbreaks in well-defined settings (e.g., day care centers and restaurants) with no laboratory-confirmed pathogen. Data streams most promising for surveillance included microbiology test results, stool culture orders, and outpatient diagnoses. In particular, clusters of microbiology tests positive for specific pathogens could be identified in EMRs and used to prioritize further testing at state health departments, potentially improving outbreak detection.

Gumbel based p-value approximations for spatial scan statistics.

BACKGROUND: The spatial and space-time scan statistics are commonly applied for the detection of geographical disease clusters. Monte Carlo hypothesis testing is typically used to test whether the geographical clusters are statistically significant as there is no known way to calculate the null distribution analytically. In Monte Carlo hypothesis testing, simulated random data are generated multiple times under the null hypothesis, and the p-value is r/(R + 1), where R is the number of simulated random replicates of the data and r is the rank of the test statistic from the real data compared to the same test statistics calculated from each of the random data sets. A drawback to this powerful technique is that each additional digit of p-value precision requires ten times as many replicated datasets, and the additional processing can lead to excessive run times. RESULTS: We propose a new method for obtaining more precise p-values with a given number of replicates. The collection of test statistics from the random replicates is used to estimate the true distribution of the test statistic under the null hypothesis by fitting a continuous distribution to these observations. The choice of distribution is critical, and for the spatial and space-time scan statistics, the extreme value Gumbel distribution performs very well while the gamma, normal and lognormal distributions perform poorly. From the fitted Gumbel distribution, we show that it is possible to estimate the analytical p-value with great precision even when the test statistic is far out in the tail beyond any of the test statistics observed in the simulated replicates. In addition, Gumbel-based rejection probabilities have smaller variability than Monte Carlo-based rejection probabilities, suggesting that the proposed approach may result in greater power than the true Monte Carlo hypothesis test for a given number of replicates. CONCLUSIONS: For large data sets, it is often advantageous to replace computer intensive Monte Carlo hypothesis testing with this new method of fitting a Gumbel distribution to random data sets generated under the null, in order to reduce computation time and obtain much more precise p-values and slightly higher statistical power.

Use of outcomes to evaluate surveillance systems for bioterrorist attacks.

BACKGROUND: Syndromic surveillance systems can potentially be used to detect a bioterrorist attack earlier than traditional surveillance, by virtue of their near real-time analysis of relevant data. Receiver operator characteristic (ROC) curve analysis using the area under the curve (AUC) as a comparison metric has been recommended as a practical evaluation tool for syndromic surveillance systems, yet traditional ROC curves do not account for timeliness of detection or subsequent time-dependent health outcomes. METHODS: Using a decision-analytic approach, we predicted outcomes, measured in lives, quality adjusted life years (QALYs), and costs, for a series of simulated bioterrorist attacks. We then evaluated seven detection algorithms applied to syndromic surveillance data using outcomes-weighted ROC curves compared to simple ROC curves and timeliness-weighted ROC curves. We performed sensitivity analyses by varying the model inputs between best and worst case scenarios and by applying different methods of AUC calculation. RESULTS: The decision analytic model results indicate that if a surveillance system was successful in detecting an attack, and measures were immediately taken to deliver treatment to the population, the lives, QALYs and dollars lost could be reduced considerably. The ROC curve analysis shows that the incorporation of outcomes into the evaluation metric has an important effect on the apparent performance of the surveillance systems. The relative order of performance is also heavily dependent on the choice of AUC calculation method. CONCLUSIONS: This study demonstrates the importance of accounting for mortality, morbidity and costs in the evaluation of syndromic surveillance systems. Incorporating these outcomes into the ROC curve analysis allows for more accurate identification of the optimal method for signaling a possible bioterrorist attack. In addition, the parameters used to construct an ROC curve should be given careful consideration.

Use of well-child visits in high-deductible health plans.

OBJECTIVE: To examine how enrollment in high-deductible health plans (HDHPs) affects use of well-child visits relative to traditional plans, when preventive care is exempt from the deductible. STUDY DESIGN: Pre-post comparison between groups. METHODS: We selected children aged <18 years enrolled in a large Massachusetts health plan through employers offering only 1 type of plan. Children were in traditional plans for a 12-month baseline period between 2001 and 2004, then were either switched by a decision of the parent's employer to an HDHP or kept in the traditional plan (controls) for a 12-month follow-up period. Preventive and other office visits were exempt from the deductible and subject to copayments, as in traditional plans. The primary outcome was whether the child received well-child visits recommended for the 12-month period. Using generalized linear mixed models, we compared the change in receipt of recommended well-child visits between baseline and follow-up for the HDHP group relative to controls. RESULTS: We identified 1598 children who were switched to HDHPs and 10,093 controls. Between baseline and follow-up, the mean proportion of recommended well-child visits received by HDHP children decreased slightly from 0.846 to 0.841, and from 0.861 to 0.855 for controls. In adjusted models, there was no significant difference in the change in probability that recommended well-child visits were received by HDHP children compared with controls (P = .69). CONCLUSIONS: Receipt of recommended well-child visits did not change for children switching from traditional plans to HDHPs that exempt preventive care from the deductible.

Assessing the utility of public health surveillance using specificity, sensitivity, and lives saved.

In modern surveillance of public health, data may be reported in a timely fashion and include spatial data on cases in addition to the time of their occurrence. This has lead to many recent developments in statistical methods to detect events of public health importance. However, there has been relatively little work about how to compare such methods. One powerful rationale for performing surveillance is earlier detection of events of public health significance; previous evaluation tools have focused on metrics that include the timeliness of detection in addition to sensitivity and specificity. However, such metrics have not accounted for the number of persons affected by the events. We re-examine the rationale for this surveillance and conclude that earlier detection is preferred because it can prevent additional morbidity and mortality. On the basis this observation, we propose evaluating the number of cases prevented by each detection method, and include this information in assessing the value of different detection methods. Using this approach incorporates more information about the events and the detection and provides a sound basis for making decisions about which detection methods to employ.

A SaTScan macro accessory for cartography (SMAC) package implemented with SAS software.

BACKGROUND: SaTScan is a software program written to implement the scan statistic; it can be used to find clusters in space and/or time. It must often be run multiple times per day when doing disease surveillance. Running SaTScan frequently via its graphical user interface can be cumbersome, and the output can be difficult to visualize. RESULTS: The SaTScan Macro Accessory for Cartography (SMAC) package consists of four SAS macros and was designed as an easier way to run SaTScan multiple times and add graphical output. The package contains individual macros which allow the user to make the necessary input files for SaTScan, run SaTScan, and create graphical output all from within SAS software. The macros can also be combined to do this all in one step. CONCLUSION: The SMAC package can make SaTScan easier to use and can make the output more informative.

Assessing surveillance using sensitivity, specificity and timeliness.

Monitoring ongoing processes of illness to detect sudden changes is an important aspect of practical epidemiology and medicine more generally. Most commonly, the monitoring has been restricted to a unidimensional stream of data over time. In such situations, analytic results from the industrial process monitoring have suggested optimal approaches to monitor the data streams. Data streams including spatial location as well as temporal sequence are becoming available. Monitoring methods that incorporate spatial data may prove superior to those that ignore it. However, analytically, optimal methods for spatial surveillance data may not exist. In the present article, we introduce and discuss evaluation metrics that can be used to compare the performance of statistical methods of surveillance. Our general approach is to generalize receiver operating characteristic (ROC) curves to incorporate the time of detection in addition to the usual test characteristics of sensitivity and specificity. In addition to weighting ordinary ROC curves by two measures of timeliness, we describe three three-dimensional generalizations of ROC curves that result in timeliness-ROC surfaces. Working in the context of surveillance of cases of disease to detect a sudden outbreak, we demonstrate these in an artificial example and in a previously described simulation context and show how the metrics differ. We also discuss the differences and under which circumstances one might prefer a given method.

Simulated anthrax attacks and syndromic surveillance.

We measured sensitivity and timeliness of a syndromic surveillance system to detect bioterrorism events. A hypothetical anthrax release was modeled by using zip code population data, mall customer surveys, and membership information from HealthPartners Medical Group, which covers 9% of a metropolitan area population in Minnesota. For each infection level, 1,000 releases were simulated. Timing of increases in use of medical care was based on data from the Sverdlovsk, Russia, anthrax release. Cases from the simulated outbreak were added to actual respiratory visits recorded for those dates in HealthPartners Medical Group data. Analysis was done by using the space-time scan statistic. We evaluated the proportion of attacks detected at different attack rates and timeliness to detection. Timeliness and completeness of detection of events varied by rate of infection. First detection of events ranged from days 3 to 6. Similar modeling may be possible with other surveillance systems and should be a part of their evaluation.