Evaluation of the effectiveness of Washington State's digital COVID-19 exposure notification system over one pandemic year.

Introduction: Digital exposure notifications are a novel public health intervention used during the COVID-19 pandemic to alert users of possible COVID-19 exposure. We seek to quantify the effectiveness of Washington State's digital exposure notification system, WA Notify, as measured by the number of COVID-19 cases averted during a 1-year period. Methods: While maintaining individuals' privacy, WA Notify collected data that could be used to evaluate the system's effectiveness. This article uses these and other data and builds on a previous model to estimate the number of cases averted by WA Notify. Novel estimates of some model parameters are possible because of improvements in the quality and breadth of data reported by WA Notify. Results: We estimate that WA Notify averted 64,000 (sensitivity analysis: 35,000-92,000) COVID-19 cases in Washington State during the study period from 1 March 2021 to 28 February 2022. During this period, there were an estimated 1,089,000 exposure notifications generated and 155,000 cases reported to WA Notify. During the last 78 days of the study period, the median estimated number of daily active users was 1,740,000. Discussion: We believe WA Notify reduced the impact of the COVID-19 pandemic in Washington State and that similar systems could reduce the impact of future communicable disease outbreaks.

Use of Digital COVID-19 Exposure Notifications at a Large Gathering: Survey Analysis of Public Health Conference Attendees.

BACKGROUND: WA Notify was Washington State's smartphone-based COVID-19 digital exposure notification (EN) tool, which was used to help limit the spread of COVID-19 between November 30, 2020, and May 11, 2023. Following the 2022 Washington State Public Health Association Annual Conference, attendees who had WA Notify activated began receiving ENs alerting them to a possible COVID-19 exposure during the conference. A survey was emailed to all conference attendees to measure WA Notify adoption, mechanisms through which attendees received ENs, and self-reported engagement in protective behaviors postexposure. OBJECTIVE: This study aimed to learn more about the experiences of WA Notify adopters and nonadopters who may have been exposed to COVID-19 at a large group gathering. METHODS: A web-based survey administered through REDCap (Research Electronic Data Capture; Vanderbilt University) was sent to all attendees of the Washington State Public Health Association conference. Self-reported demographic information and characteristics of respondents were summarized. Regression models were used to estimate relative risks to compare WA Notify adoption and testing behaviors between groups. RESULTS: Of the 464 total registered attendees who were sent the survey, 205 (44%) responses were received; 201 eligible attendees were included in this analysis. Of those, 149 (74%) respondents reported having WA Notify activated on their phones at the time of the conference. Among respondents with WA Notify activated, 54% (n=77) reported learning of their potential exposure from a WA Notify EN. Respondents who reported that they did not have WA Notify activated and learned of their potential exposure via the event-wide email from conference organizers were 39% less likely to test for COVID-19 compared to respondents with WA Notify activated who learned of their potential exposure from the email (relative risk 0.61, 95% CI 0.40-0.93; P=.02), and this gap was even larger when compared to respondents who learned of their exposure from a WA Notify EN. The most commonly cited reason for not having WA Notify activated was privacy concerns (n=17, 35%), followed by not wanting to receive ENs (n=6, 12%) and being unaware of WA Notify (n=5, 10%). CONCLUSIONS: Digital EN systems are an important tool to directly and anonymously notify close contacts of potential exposures and provide guidance on the next steps in a timely manner. Given the privacy concerns, there is still a need for increasing transparency surrounding EN technology to increase uptake by the public if this technology were to be used in the future to slow the spread of communicable diseases.

Trading-off privacy and utility: the Washington State experience assessing the performance of a public health digital exposure notification system for coronavirus disease 2019.

OBJECTIVE: Digital exposure notifications (DEN) systems were an emergency response to the coronavirus disease 2019 (COVID-19) pandemic, harnessing smartphone-based technology to enhance conventional pandemic response strategies such as contact tracing. We identify and describe performance measurement constructs relevant to the implementation of DEN tools: (1) reach (number of users enrolled in the intervention); (2) engagement (utilization of the intervention); and (3) effectiveness in preventing transmissions of COVID-19 (impact of the intervention). We also describe WA State's experience utilizing these constructs to design data-driven evaluation approaches. METHODS: We conducted an environmental scan of DEN documentation and relevant publications. Participation in multidisciplinary collaborative environments facilitated shared learning. Compilation of available data sources and their relevance to implementation and operation workflows were synthesized to develop implementation evaluation constructs. RESULTS: We identified 8 useful performance indicators within reach, engagement, and effectiveness constructs. DISCUSSION: We use implementation science to frame the evaluation of DEN tools by linking the theoretical constructs with the metrics available in the underlying disparate, deidentified, and aggregate data infrastructure. Our challenges in developing meaningful metrics include limited data science competencies in public health, validation of analytic methodologies in the complex and evolving pandemic environment, and the lack of integration with the public health infrastructure. CONCLUSION: Continued collaboration and multidisciplinary consensus activities can improve the utility of DEN tools for future public health emergencies.

Engagement in Protective Behaviors by Digital Exposure Notification Users During the COVID-19 Pandemic, Washington State, January-June 2021.

OBJECTIVES: Smartphone-based digital exposure notification (EN) tools were introduced during the COVID-19 pandemic to supplement strained case investigation and contact tracing efforts. We examined the influence of an EN tool implemented in Washington State, WA Notify, on user engagement in behaviors that protect against COVID-19 transmission. METHODS: From January 25 through June 30, 2021, we administered 2 surveys to WA Notify users who received notification of a possible COVID-19 exposure. The initial survey, sent when users received a notification, focused on intent to engage in protective behaviors. The follow-up survey captured data on self-reported actual engagement in protective behaviors and contact by a public health contact tracer. RESULTS: Of 1507 WA Notify users who completed the initial survey, 40.1% (n = 604) reported intending to seek COVID-19 testing and 67.1% (n = 1011) intended to watch for COVID-19 symptoms. Of 407 respondents to the follow-up survey, 57.5% (n = 234) reported getting tested and 84.3% (n = 343) reported watching for COVID-19 symptoms. Approximately 84% (n = 1266) of respondents to the initial survey received a notification from WA Notify before being reached by public health contact tracers; on follow-up, 42.5% (n = 173) of respondents reported never being contacted by public health. CONCLUSIONS: Our findings suggest that WA Notify users may initiate protective behaviors earlier than nonusers who will not know of an exposure until notified by public health or by a known contact. Digital EN tools may be a valuable addition to existing public health outbreak investigation and response activities.

WA Notify: the planning and implementation of a Bluetooth exposure notification tool for COVID-19 pandemic response in Washington State.

Bluetooth exposure notification tools for mobile phones have emerged as one way to support public health contact tracing and mitigate the spread of COVID-19. Many states have launched their own versions of these tools. Washington State's exposure notification tool, WA Notify, became available on November 30, 2020, following a one-month Seattle campus pilot at the University of Washington. By the end of April 2021, 25% of the state's population had activated WA Notify, one of the highest adoption rates in the country. Washington State's formation of an Exposure Notification Advisory Committee, early pilot testing, and use of the EN Express system framework were all important factors in its adoption. Continuous monitoring and willingness to make early adjustments such as switching to automated texting of verification codes have also been important for improving the tool's value. Evaluation work is ongoing to determine and quantify WA Notify's effectiveness, timeliness, and accessibility.

Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial.

PURPOSE: Although patient-reported cancer symptoms and quality-of-life issues (SQLIs) have been promoted as essential to a comprehensive assessment, efficient and efficacious methods have not been widely tested in clinical settings. The purpose of this trial was to determine the effect of the Electronic Self-Report Assessment-Cancer (ESRA-C) on the likelihood of SQLIs discussed between clinicians and patients with cancer in ambulatory clinic visits. Secondary objectives included comparison of visit duration between groups and usefulness of the ESRA-C as reported by clinicians. PATIENTS AND METHODS: This randomized controlled trial was conducted in 660 patients with various cancer diagnoses and stages at two institutions of a comprehensive cancer center. Patient-reported SQLIs were automatically displayed on a graphical summary and provided to the clinical team before an on-treatment visit (n = 327); in the control group, no summary was provided (n = 333). SQLIs were scored for level of severity or distress. One on-treatment clinic visit was audio recorded for each participant and then scored for discussion of each SQLI. We hypothesized that problematic SQLIs would be discussed more often when the intervention was delivered to the clinicians. RESULTS: The likelihood of SQLIs being discussed differed by randomized group and depended on whether an SQLI was first reported as problematic (P = .032). Clinic visits were similar with regard to duration between groups, and clinicians reported the summary as useful. CONCLUSION: The ESRA-C is the first electronic self-report application to increase discussion of SQLIs in a US randomized clinical trial.

Time-tradeoff utilities for identifying and evaluating a minimum data set for time-critical biosurveillance.

BACKGROUND: Researchers and policy makers are interested in identifying, implementing, and evaluating a national minimum data set for biosurveillance. However, work remains to be done to establish methods for measuring the value of such data. PURPOSE: The purpose of this article is to establish and evaluate a method for measuring the utility of biosurveillance data. METHOD: The authors derive an expected utility model in which the value of data may be determined by trading data relevance for time delay in receiving data. In a sample of 23 disease surveillance practitioners, the authors test if such tradeoffs are sensitive to the types of data elements involved (chief complaint v. emergency department [ED] log of visit) and proportional changes to the time horizon needed for receiving data (24 v. 48 h). In addition, they evaluate the logical error rate: the proportion of responses that scored less relevant data as having higher utility. RESULTS: Utilities of chief complaints were significantly higher than ED log of visit, F(1, 21)= 5.60, P < 0.05, suggesting the method is sensitive. Further utilities did not depend on time horizon used in the exercise, F(1, 21) = 0.00, P = ns. Of 92 time tradeoffs elicited, there were 5 logical errors (i.e., 5% logical error rate). CONCLUSIONS: In this article, the authors establish a time-tradeoff exercise for valuing biosurveillance data. Empirically, the method shows initial promise for evaluating a minimum data set for biosurveillance. Future applications of this approach may prove useful in disease surveillance planning and evaluation.

Evaluation of a mass dispensing exercise in a Cities Readiness Initiative setting.

PURPOSE: The effectiveness of a point of dispensing (POD) used in a mass dispensing exercise was evaluated. METHODS: Public Health-Seattle & King County (PHSKC), in conjunction with the University of Washington, conducted a functional exercise of mass dispensing plans to test the effectiveness of a POD. Specifically, the organization and maintenance of patient flow, staffing model, signage, and dissemination of public information were evaluated. A data collection application using cellular telephones was used to record time and patient flow data to evaluate patient flow. Questionnaires distributed to staff and volunteer patients at the end of the exercise obtained feedback regarding the setup, organization, and operations of the POD. RESULTS: The POD was operational for approximately 68 minutes. The majority of POD staff reported feeling comfortable with their specific job tasks and duties within 15 minutes of opening the doors to the POD to the public. Staff questionnaires also revealed a high level of self-reported confidence in their ability and in the ability of their colleagues to perform the job-specific responsibilities required of them or respond to this hypothetical event. The majority of volunteer patients found the signs helpful and easy to follow and the check-in form easy to complete. Despite efforts to provide patients with oral and written information about the medications being dispensed, only 80% indicated that they knew how to take the medication, and only 73% reported understanding the medication instructions for all the individuals for whom they picked up medication. CONCLUSION: The majority of volunteer patients and staff who participated in a functional exercise of mass dispensing plans found the POD to be effective. Time-flow analysis provided preliminary estimates of the total amount of time needed to complete the dispensing process for each head of household.

ActiveOptions: leveraging existing knowledge and usability testing to develop a physical activity program website for older adults.

ActiveOptions (http://www.activeoptions.org) is a multi-agency effort to help people remain healthy as they age by providing Web access to information about senior-friendly exercise programs. This resource is currently available and in use in many location across the United States. This paper focuses on the user interface to the site. It synthesizes existing knowledge related to creating an effective interface for this population, and describes the process we used which included a heuristic evaluation and usability testing.

An automated surveillance and analysis tool for online STI/HIV behaviors.

As part of a preliminary investigation, proof-of-concept software was developed to automatically retrieve, process, and classify online behaviors related to STI/HIV transmission. 300 online sexual solicitations were automatically collected, processed and characterized by the software using a shallow natural language processor. The results of the software analysis were compared with expert manual coding for agreement. The study demonstrated the feasibility of an automated online behavioral surveillance system.

Electronic Self-report Assessment--Cancer (ESRA-C): Working towards an integrated survey system.

The Clinical Informatics Research Group and Biobehavioral Nursing and Health Systems at the University of Washington are working with interdisciplinary teams to improve patient care and tracking of patient-reported symptoms and outcomes by creating an extensible web-based survey and intervention platform. The findings and cumulative experience from these processes have led to incremental improvements and variations in each new implementation of the platform. This paper presents progress in the first year of a three-year NIH study entitled Electronic Self Report Assessment--Cancer (ESRA-C). The project's goals are to enhance and evaluate the web-based computerized patient self-reporting and assessment system at the Seattle Cancer Care Alliance. Preliminary work and lessons learned in the modification of the platform and enhancements to the system will be described.

Multidisciplinary eHealth survey evaluation methods.

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.

Mobile eHealth interventions for obesity: a timely opportunity to leverage convergence trends.

Obesity is often cited as the most prevalent chronic health condition and highest priority public health problem in the United States. There is a limited but growing body of evidence suggesting that mobile eHealth behavioral interventions, if properly designed, may be effective in promoting and sustaining successful weight loss and weight maintenance behavior changes. This paper reviews the current literature on the successes and failures of public health, provider-administered, and self-managed behavioral health interventions for weight loss. The prevailing theories of health behavior change are discussed from the perspective of how this knowledge can serve as an evidence base to inform the design of mobile eHealth weight loss interventions. Tailored informational interventions, which, in recent years, have proven to be the most effective form of conventional health behavior intervention for weight loss, are discussed. Lessons learned from the success of conventional tailored informational interventions and the early successes of desktop computer-assisted self-help weight management interventions are presented, as are design principles suggested by Social Cognitive Theory and the Social Marketing Model. Relevant computing and communications technology convergence trends are also discussed. The recent trends in rapid advancement, convergence, and public adoption of Web-enabled cellular telephone and wireless personal digital assistant (PDA) devices provide timely opportunities to deliver the mass customization capabilities, reach, and interactivity required for the development, administration, and adoption of effective population-level eHealth tailored informational interventions for obesity.

Design and implementation of Cell-PREVEN: a real-time surveillance system for adverse events using cell phones in Peru.

With more clinical trials involving evaluations of new drugs or vaccines, monitoring for early detection of adverse events is essential. The overall goal of this study was to develop an interactive-computer system using cell phones for real-time collection and transmission of adverse events related to metronidazole administration among female sex workers (FSW) in Peru. We developed an application for cell phones in Spanish, called Cell-PREVEN, based on a system from Voxiva Inc. We used cell phones to enter data collected by interviewers from FSW in three communities. Information was stored in an online database, where it could be immediately accessed worldwide and exported over a secure Internet connection. E-mail and text messages sent to mobile devices alerted key personnel to selected symptoms. This pilot project has demonstrated that it is feasible to develop a public-health-surveillance system based on cell phones to collect data in real-time in Peru (http://www.prevenperu.org).

Collection and integration of clinical data for surveillance.

OBJECTIVE: The syndromic surveillance project at Public Health-Seattle & King County incorporates several data sources, including emergency department and primary care visit data collected and normalized through an automated mechanism. We describe significant changes made in this "second generation" of our system to improve data quality while complying with privacy and state public health reporting regulations. METHODS/RESULTS: The system uses de-identified visit and patient numbers to assure data accuracy, while shielding patient identity. Presently, we have 124,000 basic visit records (used to generate stratified denominators), and 29,000 surveillance records, from four emergency departments and a primary care clinic network. The system is capable of producing syndrome-clustered data sets for analysis. DISCUSSION: We have incorporated data collection techniques such as automated querying, report parsing, and HL7 electronic data interchange. We are expanding the system to include greater population coverage, and developing an understanding how to implement data collections more rapidly at individual hospital sites, as well as how best to prepare the data for analysis.

Computerized symptom and quality-of-life assessment for patients with cancer part I: development and pilot testing.

PURPOSE/OBJECTIVES: To develop and test an innovative computerized symptom and quality-of-life (QOL) assessment for patients with cancer who are evaluated for and treated with radiation therapy. DESIGN: Descriptive, longitudinal prototype development and cross-sectional clinical data. SETTING: Department of radiation oncology in an urban, academic medical center. SAMPLE: 101 outpatients who were evaluated for radiation therapy, able to communicate in English (or through one of many interpreters available at the University of Washington), and competent to understand the study information and give informed consent. Six clinicians caring for the patients in the sample were enrolled. METHODS: Iterative prototype development was conducted using a standing focus group of clinicians. The software was developed based on survey markup language and implemented in a wireless, Web-based format. Patient participants completed the computerized assessment prior to consultation with the radiation physician. Graphical output pages with flagged areas of symptom distress or troublesome QOL issues were made available to consulting physicians and nurses. MAIN RESEARCH VARIABLES: Pain intensity, symptoms, QOL, and demographics. INSTRUMENTS: Computerized versions of a 0 to 10 Pain Intensity Numerical Scale (PINS), Symptom Distress Scale, and Short Form-8. FINDINGS: Focus group recommendations included clinician priorities of brevity, flexibility, and simplicity for both input interface and output and that the assessment output contain color graphic display. Patient participants included 45 women and 56 men with a mean age of 52.7 years (SD = 13.8). Fewer than half of the participants (40%) reported using a computer on a regular basis (weekly or daily). Completion time averaged 7.8 minutes (SD = 3.7). Moderate to high levels of distress were reported more often for fatigue, pain, and emotional issues than for other symptoms or concerns. CONCLUSIONS: Computerized assessment of cancer symptoms and QOL is technically possible and feasible in an ambulatory cancer clinic. A wireless, Web-based system facilitates access to results and data entry and retrieval. The symptom and QOL profiles of these patients new to radiation therapy were comparable to other samples of outpatients with cancer. IMPLICATIONS FOR NURSING: The ability to capture an easily interpreted illustration of a patients symptom and QOL experience in less than 10 minutes is a potentially useful adjunct to traditional face-to-face interviewing. Ultimately, electronic patient-generated data could produce automated red flags directed to the most appropriate clinicians (e.g., nurse, pain specialist, social worker, nutritionist) for further evaluation. Such system enhancement could greatly facilitate oncology nurses coordination role in caring for complex patients with cancer.

Syndromic surveillance using automated collection of computerized discharge diagnoses.

The Syndromic Surveillance Information Collection (SSIC) system aims to facilitate early detection of bioterrorism attacks (with such agents as anthrax, brucellosis, plague, Q fever, tularemia, smallpox, viral encephalitides, hemorrhagic fever, botulism toxins, staphylococcal enterotoxin B, etc.) and early detection of naturally occurring disease outbreaks, including large foodborne disease outbreaks, emerging infections, and pandemic influenza. This is accomplished using automated data collection of visit-level discharge diagnoses from heterogeneous clinical information systems, integrating those data into a common XML (Extensible Markup Language) form, and monitoring the results to detect unusual patterns of illness in the population. The system, operational since January 2001, collects, integrates, and displays data from three emergency department and urgent care (ED/UC) departments and nine primary care clinics by automatically mining data from the information systems of those facilities. With continued development, this system will constitute the foundation of a population-based surveillance system that will facilitate targeted investigation of clinical syndromes under surveillance and allow early detection of unusual clusters of illness compatible with bioterrorism or disease outbreaks.

A proposed ontology for online healthcare surveys.

This paper results from the research efforts of the Clinical Informatics Research Group in building a generalized system for online survey implementation. Key to the success of any generalized survey system is a standard ontology for the differing components of any survey, particularly those sought to be implemented online, over the World Wide Web. In this paper, we introduce the need for generalized survey authoring tools, discuss our methods for elucidating the different components present in many healthcare instruments and classifying them as per existing standards, and later present our proposed ontology for online surveys in the healthcare domain. This is followed by a more detailed description of the different question types mentioned in this ontology. Finally, we compare some general purpose authoring systems currently available to determine their flexibility in representing these disparate question types (www.cirg.washington.edu/SuML).

Diagnoses, syndromes, and diseases: a knowledge representation problem.

Despite their widespread use, the terms "syndrome", "disease" and "diagnosis" are sometimes utilized improperly and ambiguously, compounding the complexities of medical knowledge representation. The definitions and illustrative examples provided here will be useful for developers of diagnostic expert systems. Description of the Problem Representing medical knowledge is a highly complex endeavor. The improper use of the terms "syndrome", "disease" and their relations to "diagnosis" is one of the difficulties with which medical informaticians must deal, especially when developing expert systems to support diagnoses. Although ubiquitous in medical and lay discourse, the term "disease" has no unambiguous, generally accepted definition. How-ever, most of those using this term allow themselves the comfortable delusion that everyone knows what it means. Only sparse and fragmented literature could be found regarding this issue.