Public Health Emergency Preparedness System Evaluation Criteria and Performance Metrics: A Review of Contributions of the CDC-Funded Preparedness and Emergency Response Research Centers.

OBJECTIVES: The US Centers for Disease Control and Prevention (CDC)-funded Preparedness and Emergency Response Research Centers (PERRCs) conducted research from 2008 to 2015 aimed to improve the complex public health emergency preparedness and response (PHEPR) system. This paper summarizes PERRC studies that addressed the development and assessment of criteria for evaluating PHEPR and metrics for measuring their efficiency and effectiveness. METHODS: We reviewed 171 PERRC publications indexed in PubMed between 2009 and 2016. These publications derived from 34 PERRC research projects. We identified publications that addressed the development or assessment of criteria and metrics pertaining to PHEPR systems and describe the evaluation methods used and tools developed, the system domains evaluated, and the metrics developed or assessed. RESULTS: We identified 29 publications from 12 of the 34 PERRC projects that addressed PHEPR system evaluation criteria and metrics. We grouped each study into 1 of 3 system domains, based on the metrics developed or assessed: (1) organizational characteristics (n = 9), (2) emergency response performance (n = 12), and (3) workforce capacity or capability (n = 8). These studies addressed PHEPR system activities including responses to the 2009 H1N1 pandemic and the 2011 tsunami, as well as emergency exercise performance, situational awareness, and workforce willingness to respond. Both PHEPR system process and outcome metrics were developed or assessed by PERRC studies. CONCLUSIONS: PERRC researchers developed and evaluated a range of PHEPR system evaluation criteria and metrics that should be considered by system partners interested in assessing the efficiency and effectiveness of their activities. Nonetheless, the monitoring and measurement problem in PHEPR is far from solved. Lack of standard measures that are readily obtained or computed at local levels remains a challenge for the public health preparedness field. (Disaster Med Public Health Preparedness. 2019;13:626-638).

Public Health Surveillance Systems: Recent Advances in Their Use and Evaluation.

Surveillance is critical for improving population health. Public health surveillance systems generate information that drives action, and the data must be of sufficient quality and with a resolution and timeliness that matches objectives. In the context of scientific advances in public health surveillance, changing health care and public health environments, and rapidly evolving technologies, the aim of this article is to review public health surveillance systems. We consider their current use to increase the efficiency and effectiveness of the public health system, the role of system stakeholders, the analysis and interpretation of surveillance data, approaches to system monitoring and evaluation, and opportunities for future advances in terms of increased scientific rigor, outcomes-focused research, and health informatics.

Coding of Electronic Laboratory Reports for Biosurveillance, Selected United States Hospitals, 2011.

Objective Electronic laboratory reporting has been promoted as a public health priority. The Office of the U.S. National Coordinator for Health Information Technology has endorsed two coding systems: Logical Observation Identifiers Names and Codes (LOINC) for laboratory test orders and Systemized Nomenclature of Medicine-Clinical Terms (SNOMED CT) for test results. Materials and Methods We examined LOINC and SNOMED CT code use in electronic laboratory data reported in 2011 by 63 non-federal hospitals to BioSense electronic syndromic surveillance system. We analyzed the frequencies, characteristics, and code concepts of test orders and results. Results A total of 14,028,774 laboratory test orders or results were reported. No test orders used SNOMED CT codes. To describe test orders, 77% used a LOINC code, 17% had no value, and 6% had a non-informative value, "OTH". Thirty-three percent (33%) of test results had missing or non-informative codes. For test results with at least one informative value, 91.8% had only LOINC codes, 0.7% had only SNOMED codes, and 7.4% had both. Of 108 SNOMED CT codes reported without LOINC codes, 45% could be matched to at least one LOINC code. Conclusion Missing or non-informative codes comprised almost a quarter of laboratory test orders and a third of test results reported to BioSense by non-federal hospitals. Use of LOINC codes for laboratory test results was more common than use of SNOMED CT. Complete and standardized coding could improve the usefulness of laboratory data for public health surveillance and response.

High priority preparedness research and its support.

The US Federal Government has considerable interest in supporting research into preparedness. Because of the diverse nature of possible threats and the responsibilities of different agencies, a number of different programs have been developed. Perspectives from representatives from 3 of the leading agencies; the Department of Homeland Security, the Centers from Disease Control and Prevention, and the National Institutes of Health, are described herein.

Application of change point analysis to daily influenza-like illness emergency department visits.

BACKGROUND: The utility of healthcare utilization data from US emergency departments (EDs) for rapid monitoring of changes in influenza-like illness (ILI) activity was highlighted during the recent influenza A (H1N1) pandemic. Monitoring has tended to rely on detection algorithms, such as the Early Aberration Reporting System (EARS), which are limited in their ability to detect subtle changes and identify disease trends. OBJECTIVE: To evaluate a complementary approach, change point analysis (CPA), for detecting changes in the incidence of ED visits due to ILI. METHODOLOGY AND PRINCIPAL FINDINGS: Data collected through the Distribute project (isdsdistribute.org), which aggregates data on ED visits for ILI from over 50 syndromic surveillance systems operated by state or local public health departments were used. The performance was compared of the cumulative sum (CUSUM) CPA method in combination with EARS and the performance of three CPA methods (CUSUM, structural change model and Bayesian) in detecting change points in daily time-series data from four contiguous US states participating in the Distribute network. Simulation data were generated to assess the impact of autocorrelation inherent in these time-series data on CPA performance. The CUSUM CPA method was robust in detecting change points with respect to autocorrelation in time-series data (coverage rates at 90% when -0.2≤ρ≤0.2 and 80% when -0.5≤ρ≤0.5). During the 2008-9 season, 21 change points were detected and ILI trends increased significantly after 12 of these change points and decreased nine times. In the 2009-10 flu season, we detected 11 change points and ILI trends increased significantly after two of these change points and decreased nine times. Using CPA combined with EARS to analyze automatically daily ED-based ILI data, a significant increase was detected of 3% in ILI on April 27, 2009, followed by multiple anomalies in the ensuing days, suggesting the onset of the H1N1 pandemic in the four contiguous states. CONCLUSIONS AND SIGNIFICANCE: As a complementary approach to EARS and other aberration detection methods, the CPA method can be used as a tool to detect subtle changes in time-series data more effectively and determine the moving direction (ie, up, down, or stable) in ILI trends between change points. The combined use of EARS and CPA might greatly improve the accuracy of outbreak detection in syndromic surveillance systems.

Application of the time-series approach to assess the temporal trend of racial disparity in chlamydia prevalence in the US National Job Training Program.

The authors applied a time-series approach to assess the temporal trend of racial disparity in chlamydia prevalence between young, socioeconomically disadvantaged blacks and whites entering the US National Job Training Program. Racial disparity was defined as the arithmetic difference between age group-, specimen type-, and region of residence-standardized chlamydia prevalences in blacks and whites. A regression with autoregressive moving average errors model was employed to adjust for serial correlation. Data from 46,849 women (2006-2008) and 136,892 men (2004-2008) were analyzed. Racial disparity significantly decreased among women (by an average of 0.122% per 2-month interval; P < 0.05) but not among men (-0.010%, P = 0.57). Chlamydia prevalence significantly declined for black women (-0.139% per 2-month interval; P = 0.004), black men (-0.045%, P < 0.001), and white men (-0.035%, P = 0.002) but not for white women (-0.028%, P = 0.413). Despite the decreases among black women and black men, the black-white disparities remained high for both sexes; in 2008, the racial disparity was 8.1% (95% confidence interval: 6.8, 9.3) for women and 9.0% (95% confidence interval: 8.4, 9.6) for men. These findings suggest that current chlamydia control efforts may be reaching young black men and women but need to be scaled up or modified to address the excess risk among blacks.

The next public health revolution: public health information fusion and social networks.

Social, political, and economic disruptions caused by natural and human-caused public health emergencies have catalyzed public health efforts to expand the scope of biosurveillance and increase the timeliness, quality, and comprehensiveness of disease detection, alerting, response, and prediction. Unfortunately, efforts to acquire, render, and visualize the diversity of health intelligence information are hindered by its wide distribution across disparate fields, multiple levels of government, and the complex interagency environment. Achieving this new level of situation awareness within public health will require a fundamental cultural shift in methods of acquiring, analyzing, and disseminating information. The notion of information "fusion" may provide opportunities to expand data access, analysis, and information exchange to better inform public health action.

Traditional sexually transmitted disease prevention and control strategies: tailoring for African American communities.

African Americans carry the largest disease burden for bacterial sexually transmitted diseases (STDs) in the United States. These infections can have a devastating impact on sexual and reproductive health if they are not diagnosed and treated. Traditionally, public health efforts to prevent and control bacterial STDs have been through surveillance, clinical services, partner management, and behavioral intervention strategies. However, the persistence of disparities in STDs indicates that these strategies are not achieving sufficient impact in African American communities. It may be that factors such as limited access, acceptability, appropriateness, and affordability of services reduce the efficacy of these strategies for African American communities. In this article we describe the STD prevention strategies and highlight the challenges and implications of these strategies in addressing disparities in African American communities.

Data suppression strategies used during surveillance data release by sexually transmitted disease prevention programs.

OBJECTIVES: This study investigates the data suppression or statistical disclosure limitation (DL) practices used during surveillance data Release by sexually transmitted disease (STD) prevention programs. METHODS: We classified DL strategies from a Web-based data query system that collected data from state health departments. We tested mean STD incidence Rates in states that used data suppression versus those that did not. RESULTS: Five types of DL were identified: no suppression (n = 15), numerator-only (n = 10), denominator-only (n = 6), demographic-only (n = 7), and mixed strategies (n = 12). Twenty-two states (62%) used data suppression strategies differently through time. Chlamydia, gonorrhea, and syphilis Rates were higher in the nonsuppression states than those of the suppression states (P = .03, P = .008, P = .009, Respectively). CONCLUSIONS: Cell suppression is the preferred method of DL used by STD prevention programs. More Research is needed to evaluate the effectiveness of this strategy as a means of balancing the public health utility of the data tables and the protection of confidentiality.

PHSkb: a knowledgebase to support notifiable disease surveillance.

BACKGROUND: Notifiable disease surveillance in the United States is predominantly a passive process that is often limited by poor timeliness and low sensitivity. Interoperable tools are needed that interact more seamlessly with existing clinical and laboratory data to improve notifiable disease surveillance. DESCRIPTION: The Public Health Surveillance Knowledgebase (PHSkb) is a computer database designed to provide quick, easy access to domain knowledge regarding notifiable diseases and conditions in the United States. The database was developed using Protégé ontology and knowledgebase editing software. Data regarding the notifiable disease domain were collected via a comprehensive review of state health department websites and integrated with other information used to support the National Notifiable Diseases Surveillance System (NNDSS). Domain concepts were harmonized, wherever possible, to existing vocabulary standards. The knowledgebase can be used: 1) as the basis for a controlled vocabulary of reportable conditions needed for data aggregation in public health surveillance systems; 2) to provide queriable domain knowledge for public health surveillance partners; 3) to facilitate more automated case detection and surveillance decision support as a reusable component in an architecture for intelligent clinical, laboratory, and public health surveillance information systems. CONCLUSIONS: The PHSkb provides an extensible, interoperable system architecture component to support notifiable disease surveillance. Further development and testing of this resource is needed.

Evaluation of reporting timeliness of public health surveillance systems for infectious diseases.

BACKGROUND: Timeliness is a key performance measure of public health surveillance systems. Timeliness can vary by disease, intended use of the data, and public health system level. Studies were reviewed to describe methods used to evaluate timeliness and the reporting timeliness of National Notifiable Diseases Surveillance System (NNDSS) data was evaluated to determine if this system could support timely notification and state response to multistate outbreaks. METHODS: Published papers that quantitatively measured timeliness of infectious disease surveillance systems operating in the U.S. were reviewed. Median reporting timeliness lags were computed for selected nationally notifiable infectious diseases based on a state-assigned week number and various date types. The percentage of cases reported within the estimated incubation periods for each disease was also computed. RESULTS: Few studies have published quantitative measures of reporting timeliness; these studies do not evaluate timeliness in a standard manner. When timeliness of NNDSS data was evaluated, the median national reporting delay, based on date of disease onset, ranged from 12 days for meningococcal disease to 40 days for pertussis. Diseases with the longer incubation periods tended to have a higher percentage of cases reported within its incubation period. For acute hepatitis A virus infection, which had the longest incubation period of the diseases studied, more than 60% of cases were reported within one incubation period for each date type reported. For cryptosporidiosis, Escherichia coli O157:H7 infection, meningococcal disease, salmonellosis, and shigellosis, less than 40% of cases were reported within one incubation period for each reported date type. CONCLUSION: Published evaluations of infectious disease surveillance reporting timeliness are few in number and are not comparable. A more standardized approach for evaluating and describing surveillance system timeliness should be considered; a recommended methodology is presented. Our analysis of NNDSS reporting timeliness indicated that among the conditions evaluated (except for acute hepatitis A infection), the long reporting lag and the variability across states limits the usefulness of NNDSS data and aberration detection analysis of those data for identification of and timely response to multistate outbreaks. Further evaluation of the factors that contribute to NNDSS reporting timeliness is warranted.

Endemic, notifiable bioterrorism-related diseases, United States, 1992-1999.

Little information is available in the United States regarding the incidence and distribution of diseases caused by critical microbiologic agents with the potential for use in acts of terrorism. We describe disease-specific, demographic, geographic, and seasonal distribution of selected bioterrorism-related conditions (anthrax, botulism, brucellosis, cholera, plague, tularemia, and viral encephalitides) reported to the National Notifiable Diseases Surveillance System in 1992 to 1999. Tularemia and brucellosis were the most frequently reported diseases. Anthrax, plague, western equine encephalitis, and eastern equine encephalitis were rare. Higher incidence rates for cholera and plague were noted in the western United States and for tularemia in the central United States. Overall, the incidence of conditions caused by these critical agents in the United States is low. Individual case reports should be considered sentinel events. For potential bioterrorism-related conditions that are endemic and have low incidence, the use of nontraditional surveillance methods and complementary data sources may enhance our ability to rapidly detect changes in disease incidence.

Introduction of a hierarchy to LOINC to facilitate public health reporting.

Public health reporting of laboratory results requires unambiguous identification of the test performed and the result observed. Some laboratories are currently using Logical Observation Identifier Names and Codes (LOINC) for the electronic reporting of laboratory tests and their results to public health departments. Initial use revealed inconsistent identification and use of LOINC concepts by laboratories and public health agencies and an inability to systematically extend, for public health use, the tables when adding new concepts. We applied simple, logical rules to existing LOINC concepts to facilitate the creation of a hierarchy of concepts and to allow the identification and specification of appropriate terms for public health reporting and subsequent data aggregation. The hierarchy also allows the systematic addition of new concepts further supporting public health reporting. Application of the hierarchy is illustrated by using all laboratory LOINC concepts assigned to the subset of microbiology test types (CLASS MICRO).

Completeness of notifiable infectious disease reporting in the United States: an analytical literature review.

Despite state and local laws requiring medical providers to report notifiable infectious diseases to public health authorities, reporting is believed to be incomplete. Through means of an analytical literature review, the authors synthesize current knowledge on the completeness of disease reporting and identify factors associated with reporting completeness. The review was limited to published studies, conducted in the United States between 1970 and 1999, that quantitatively assessed infectious disease reporting completeness. Thirty-three studies met the inclusion criteria. Reporting completeness, expressed between 0% and 100%, was treated as the dependent outcome variable in statistical analysis; disease, study location, time period, study design, and study size were treated as independent variables. Fifty-six distinct measures of reporting completeness were identified for 21 diseases. Reporting completeness varied from 9% to 99% and was most strongly associated with the disease being reported. The mean reporting completeness for acquired immunodeficiency syndrome, sexually transmitted diseases, and tuberculosis as a group was significantly higher (79%) than for all other diseases combined (49%) (p < 0.01).