Clinical Decision Support for Surgery: A Mixed Methods Study on Design and Implementation Perspectives From Urologists.

OBJECTIVE: To assess urologist attitudes toward clinical decision support (CDS) embedded into the electronic health record (EHR) and define design needs to facilitate implementation and impact. With recent advances in big data and artificial intelligence (AI), enthusiasm for personalized, data-driven tools to improve surgical decision-making has grown, but the impact of current tools remains limited. METHODS: A sequential explanatory mixed methods study from 2019 to 2020 was performed. First, survey responses from the 2019 American Urological Association Annual Census evaluated attitudes toward an automatic CDS tool that would display risk/benefit data. This was followed by the purposeful sampling of 25 urologists and qualitative interviews assessing perspectives on CDS impact and design needs. Bivariable, multivariable, and coding-based thematic analysis were applied and integrated. RESULTS: Among a weighted sample of 12,366 practicing urologists, the majority agreed CDS would help decision-making (70.9%, 95% CI 68.7%-73.2%), aid patient counseling (78.5%, 95% CI 76.5%-80.5%), save time (58.1%, 95% CI 55.7%-60.5%), and improve patient outcomes (42.9%, 95% CI 40.5%-45.4%). More years in practice was negatively associated with agreement (P <.001). Urologists described how CDS could bolster evidence-based care, personalized medicine, resource utilization, and patient experience. They also identified multiple implementation barriers and provided suggestions on form, functionality, and visual design to improve usefulness and ease of use. CONCLUSION: Urologists have favorable attitudes toward the potential for clinical decision support in the EHR. Smart design will be critical to ensure effective implementation and impact.

Using Counterfactuals to Improve Causal Inferences From Visualizations.

Traditional approaches to data visualization have often focused on comparing different subsets of data, and this is reflected in the many techniques developed and evaluated over the years for visual comparison. Similarly, common workflows for exploratory visualization are built upon the idea of users interactively applying various filter and grouping mechanisms in search of new insights. This paradigm has proven effective at helping users identify correlations between variables that can inform thinking and decision-making. However, recent studies show that consumers of visualizations often draw causal conclusions even when not supported by the data. Motivated by these observations, this article highlights recent advances from a growing community of researchers exploring methods that aim to directly support visual causal inference. However, many of these approaches have their own limitations, which limit their use in many real-world scenarios. This article, therefore, also outlines a set of key open challenges and corresponding priorities for new research to advance the state of the art in visual causal inference.

Electronic Health Record Use and Perceptions among Urologic Surgeons.

OBJECTIVE: Electronic health records (EHRs) have become widely adopted with increasing emphasis on improving care delivery. Improvements in surgery may be limited by specialty-specific issues that impact EHR usability and engagement. Accordingly, we examined EHR use and perceptions in urology, a diverse surgical specialty. METHODS: We conducted a national, sequential explanatory mixed methods study. Through the 2019 American Urological Association Census, we surveyed urologic surgeons on EHR use and perceptions and then identified associated characteristics through bivariable and multivariable analyses. Using purposeful sampling, we interviewed 25 urologists and applied coding-based thematic analysis, which was then integrated with survey findings. RESULTS: Among 2,159 practicing urologic surgeons, 2,081 (96.4%) reported using an EHR. In the weighted sample (n = 12,366), over 90% used the EHR for charting, viewing results, and order entry with most using information exchange functions (59.0-79.6%). In contrast, only 35.8% felt the EHR increases clinical efficiency, whereas 43.1% agreed it improves patient care, which related thematically to information management, administrative burden, patient safety, and patient-surgeon interaction. Quantitatively and qualitatively, use and perceptions differed by years in practice and practice type with more use and better perceptions among more recent entrants into the urologic workforce and those in academic/multispecialty practices, who may have earlier EHR exposure, better infrastructure, and more support. CONCLUSION: Despite wide and substantive usage, EHRs engender mixed feelings, especially among longer-practicing surgeons and those in lower-resourced settings (e.g., smaller and private practices). Beyond reducing administrative burden and simplifying information management, efforts to improve care delivery through the EHR should focus on surgeon engagement, particularly in the community, to boost implementation and user experience.

A Design Space for Surfacing Content Recommendations in Visual Analytic Platforms.

Recommendation algorithms have been leveraged in various ways within visualization systems to assist users as they perform of a range of information tasks. One common focus for these techniques has been the recommendation of content, rather than visual form, as a means to assist users in the identification of information that is relevant to their task context. A wide variety of techniques have been proposed to address this general problem, with a range of design choices in how these solutions surface relevant information to users. This paper reviews the state-of-the-art in how visualization systems surface recommended content to users during users' visual analysis; introduces a four-dimensional design space for visual content recommendation based on a characterization of prior work; and discusses key observations regarding common patterns and future research opportunities.

Improving Visualization Interpretation Using Counterfactuals.

Complex, high-dimensional data is used in a wide range of domains to explore problems and make decisions. Analysis of high-dimensional data, however, is vulnerable to the hidden influence of confounding variables, especially as users apply ad hoc filtering operations to visualize only specific subsets of an entire dataset. Thus, visual data-driven analysis can mislead users and encourage mistaken assumptions about causality or the strength of relationships between features. This work introduces a novel visual approach designed to reveal the presence of confounding variables via counterfactual possibilities during visual data analysis. It is implemented in CoFact, an interactive visualization prototype that determines and visualizes counterfactual subsets to better support user exploration of feature relationships. Using publicly available datasets, we conducted a controlled user study to demonstrate the effectiveness of our approach; the results indicate that users exposed to counterfactual visualizations formed more careful judgments about feature-to-outcome relationships.

Survey on Visual Analysis of Event Sequence Data.

Event sequence data record series of discrete events in the time order of occurrence. They are commonly observed in a variety of applications ranging from electronic health records to network logs, with the characteristics of large-scale, high-dimensional and heterogeneous. This high complexity of event sequence data makes it difficult for analysts to manually explore and find patterns, resulting in ever-increasing needs for computational and perceptual aids from visual analytics techniques to extract and communicate insights from event sequence datasets. In this paper, we review the state-of-the-art visual analytics approaches, characterize them with our proposed design space, and categorize them based on analytical tasks and applications. From our review of relevant literature, we have also identified several remaining research challenges and future research opportunities.

Interpretable Anomaly Detection in Event Sequences via Sequence Matching and Visual Comparison.

Anomaly detection is a common analytical task that aims to identify rare cases that differ from the typical cases that make up the majority of a dataset. When analyzing event sequence data, the task of anomaly detection can be complex because the sequential and temporal nature of such data results in diverse definitions and flexible forms of anomalies. This, in turn, increases the difficulty in interpreting detected anomalies. In this article, we propose a visual analytic approach for detecting anomalous sequences in an event sequence dataset via an unsupervised anomaly detection algorithm based on Variational AutoEncoders. We further compare the anomalous sequences with their reconstructions and with the normal sequences through a sequence matching algorithm to identify event anomalies. A visual analytics system is developed to support interactive exploration and interpretations of anomalies through novel visualization designs that facilitate the comparison between anomalous sequences and normal sequences. Finally, we quantitatively evaluate the performance of our anomaly detection algorithm, demonstrate the effectiveness of our system through case studies, and report feedback collected from study participants.

Enabling Longitudinal Exploratory Analysis of Clinical COVID Data.

As the COVID-19 pandemic continues to impact the world, data is being gathered and analyzed to better understand the disease. Recognizing the potential for visual analytics technologies to support exploratory analysis and hypothesis generation from longitudinal clinical data, a team of collaborators worked to apply existing event sequence visual analytics technologies to a longitudinal clinical data from a cohort of 998 patients with high rates of COVID-19 infection. This paper describes the initial steps toward this goal, including: (1) the data transformation and processing work required to prepare the data for visual analysis, (2) initial findings and observations, and (3) qualitative feedback and lessons learned which highlight key features as well as limitations to address in future work.

Analysing EHR navigation patterns and digital workflows among physicians during ICU pre-rounds.

BACKGROUND: Some physicians in intensive care units (ICUs) report that electronic health records (EHRs) can be cumbersome and disruptive to workflow. There are significant gaps in our understanding of the physician-EHR interaction. OBJECTIVE: To better understand how clinicians use the EHR for chart review during ICU pre-rounds through the characterisation and description of screen navigation pathways and workflow patterns. METHOD: We conducted a live, direct observational study of six physician trainees performing electronic chart review during daily pre-rounds in the 30-bed medical ICU at a large academic medical centre in the Southeastern United States. A tailored checklist was used by observers for data collection. RESULTS: We observed 52 distinct live patient chart review encounters, capturing a total of 2.7 hours of pre-rounding chart review activity by six individual physicians. Physicians reviewed an average of 8.7 patients (range = 5-12), spending a mean of 3:05 minutes per patient (range = 1:34-5:18). On average, physicians visited 6.3 (±3.1) total EHR screens per patient (range = 1-16). Four unique screens were viewed most commonly, accounting for over half (52.7%) of all screen visits: results review (17.9%), summary/overview (13.0%), flowsheet (12.7%), and the chart review tab (9.1%). Navigation pathways were highly variable, but several common screen transition patterns emerged across users. Average interrater reliability for the paired EHR observation was 80.0%. CONCLUSION: We observed the physician-EHR interaction during ICU pre-rounds to be brief and highly focused. Although we observed a high degree of "information sprawl" in physicians' digital navigation, we also identified common launch points for electronic chart review, key high-traffic screens and common screen transition patterns. IMPLICATIONS: From the study findings, we suggest recommendations towards improved EHR design.

Visual Causality Analysis of Event Sequence Data.

Causality is crucial to understanding the mechanisms behind complex systems and making decisions that lead to intended outcomes. Event sequence data is widely collected from many real-world processes, such as electronic health records, web clickstreams, and financial transactions, which transmit a great deal of information reflecting the causal relations among event types. Unfortunately, recovering causalities from observational event sequences is challenging, as the heterogeneous and high-dimensional event variables are often connected to rather complex underlying event excitation mechanisms that are hard to infer from limited observations. Many existing automated causal analysis techniques suffer from poor explainability and fail to include an adequate amount of human knowledge. In this paper, we introduce a visual analytics method for recovering causalities in event sequence data. We extend the Granger causality analysis algorithm on Hawkes processes to incorporate user feedback into causal model refinement. The visualization system includes an interactive causal analysis framework that supports bottom-up causal exploration, iterative causal verification and refinement, and causal comparison through a set of novel visualizations and interactions. We report two forms of evaluation: a quantitative evaluation of the model improvements resulting from the user-feedback mechanism, and a qualitative evaluation through case studies in different application domains to demonstrate the usefulness of the system.

Selection-Bias-Corrected Visualization via Dynamic Reweighting.

The collection and visual analysis of large-scale data from complex systems, such as electronic health records or clickstream data, has become increasingly common across a wide range of industries. This type of retrospective visual analysis, however, is prone to a variety of selection bias effects, especially for high-dimensional data where only a subset of dimensions is visualized at any given time. The risk of selection bias is even higher when analysts dynamically apply filters or perform grouping operations during ad hoc analyses. These bias effects threaten the validity and generalizability of insights discovered during visual analysis as the basis for decision making. Past work has focused on bias transparency, helping users understand when selection bias may have occurred. However, countering the effects of selection bias via bias mitigation is typically left for the user to accomplish as a separate process. Dynamic reweighting (DR) is a novel computational approach to selection bias mitigation that helps users craft bias-corrected visualizations. This paper describes the DR workflow, introduces key DR visualization designs, and presents statistical methods that support the DR process. Use cases from the medical domain, as well as findings from domain expert user interviews, are also reported.

A rapidly deployed, interactive, online visualization system to support fatality management during the coronavirus disease 2019 (COVID-19) pandemic.

OBJECTIVE: To create an online visualization to support fatality management in North Carolina. MATERIALS AND METHODS: A web application aggregates online datasets for coronavirus disease 2019 (COVID-19) infection rates and morgue utilization. The data are visualized through an interactive, online dashboard. RESULTS: The web application was shared with state and local public health officials across North Carolina. Users could adjust interactive maps and other statistical charts to view live reports of metrics at multiple aggregation levels (eg, county or region). The application also provides access to detailed tabular data for individual facilities. DISCUSSION: Stakeholders found this tool helpful for providing situational awareness of capacity, hotspots, and utilization fluctuations. Timely reporting of facility and county data were key, and future work can help streamline the data collection process. There is potential to generalize the technology to other use cases. CONCLUSIONS: This dashboard facilitates fatality management by visualizing county and regional aggregate statistics in North Carolina.

Selection Bias Tracking and Detailed Subset Comparison for High-Dimensional Data.

The collection of large, complex datasets has become common across a wide variety of domains. Visual analytics tools increasingly play a key role in exploring and answering complex questions about these large datasets. However, many visualizations are not designed to concurrently visualize the large number of dimensions present in complex datasets (e.g. tens of thousands of distinct codes in an electronic health record system). This fact, combined with the ability of many visual analytics systems to enable rapid, ad-hoc specification of groups, or cohorts, of individuals based on a small subset of visualized dimensions, leads to the possibility of introducing selection bias-when the user creates a cohort based on a specified set of dimensions, differences across many other unseen dimensions may also be introduced. These unintended side effects may result in the cohort no longer being representative of the larger population intended to be studied, which can negatively affect the validity of subsequent analyses. We present techniques for selection bias tracking and visualization that can be incorporated into high-dimensional exploratory visual analytics systems, with a focus on medical data with existing data hierarchies. These techniques include: (1) tree-based cohort provenance and visualization, including a user-specified baseline cohort that all other cohorts are compared against, and visual encoding of cohort "drift", which indicates where selection bias may have occurred, and (2) a set of visualizations, including a novel icicle-plot based visualization, to compare in detail the per-dimension differences between the baseline and a user-specified focus cohort. These techniques are integrated into a medical temporal event sequence visual analytics tool. We present example use cases and report findings from domain expert user interviews.

Visual Analysis of High-Dimensional Event Sequence Data via Dynamic Hierarchical Aggregation.

Temporal event data are collected across a broad range of domains, and a variety of visual analytics techniques have been developed to empower analysts working with this form of data. These techniques generally display aggregate statistics computed over sets of event sequences that share common patterns. Such techniques are often hindered, however, by the high-dimensionality of many real-world event sequence datasets which can prevent effective aggregation. A common coping strategy for this challenge is to group event types together prior to visualization, as a pre-process, so that each group can be represented within an analysis as a single event type. However, computing these event groupings as a pre-process also places significant constraints on the analysis. This paper presents a new visual analytics approach for dynamic hierarchical dimension aggregation. The approach leverages a predefined hierarchy of dimensions to computationally quantify the informativeness, with respect to a measure of interest, of alternative levels of grouping within the hierarchy at runtime. This information is then interactively visualized, enabling users to dynamically explore the hierarchy to select the most appropriate level of grouping to use at any individual step within an analysis. Key contributions include an algorithm for interactively determining the most informative set of event groupings for a specific analysis context, and a scented scatter-plus-focus visualization design with an optimization-based layout algorithm that supports interactive hierarchical exploration of alternative event type groupings. We apply these techniques to high-dimensional event sequence data from the medical domain and report findings from domain expert interviews.

Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation.

OBJECTIVE: This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. METHODS: A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. RESULTS: After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. DISCUSSION: When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. CONCLUSION: Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Health and Fitness Apps for Hands-Free Voice-Activated Assistants: Content Analysis.

BACKGROUND: Hands-free voice-activated assistants and their associated devices have recently gained popularity with the release of commercial products, including Amazon Alexa and Google Assistant. Voice-activated assistants have many potential use cases in healthcare including education, health tracking and monitoring, and assistance with locating health providers. However, little is known about the types of health and fitness apps available for voice-activated assistants as it is an emerging market. OBJECTIVE: This review aimed to examine the characteristics of health and fitness apps for commercially available, hands-free voice-activated assistants, including Amazon Alexa and Google Assistant. METHODS: Amazon Alexa Skills Store and Google Assistant app were searched to find voice-activated assistant apps designated by vendors as health and fitness apps. Information was extracted for each app including name, description, vendor, vendor rating, user reviews and ratings, cost, developer and security policies, and the ability to pair with a smartphone app and website and device. Using a codebook, two reviewers independently coded each app using the vendor's descriptions and the app name into one or more health and fitness, intended age group, and target audience categories. A third reviewer adjudicated coding disagreements until consensus was reached. Descriptive statistics were used to summarize app characteristics. RESULTS: Overall, 309 apps were reviewed; health education apps (87) were the most commonly occurring, followed by fitness and training (72), nutrition (33), brain training and games (31), and health monitoring (25). Diet and calorie tracking apps were infrequent. Apps were mostly targeted towards adults and general audiences with few specifically geared towards patients, caregivers, or medical professionals. Most apps were free to enable or use and 18.1% (56/309) could be paired with a smartphone app and website and device; 30.7% (95/309) of vendors provided privacy policies; and 22.3% (69/309) provided terms of use. The majority (36/42, 85.7%) of Amazon Alexa apps were rated by the vendor as mature or guidance suggested, which were geared towards adults only. When there was a user rating available, apps had a wide range of ratings from 1 to 5 stars with a mean of 2.97. Google Assistant apps did not have user reviews available, whereas most of Amazon Alexa apps had at least 1-9 reviews available. CONCLUSIONS: The emerging market of health and fitness apps for voice-activated assistants is still nascent and mainly focused on health education and fitness. Voice-activated assistant apps had a wide range of content areas but many published in the health and fitness categories did not actually have a clear health or fitness focus. This may, in part, be due to Amazon and Google policies, which place restrictions on the delivery of care or direct recording of health data. As in the mobile app market, the content and functionalities may evolve to meet growing demands for self-monitoring and disease management.

Assessing the Status Quo of EHR Accessibility, Usability, and Knowledge Dissemination.

AIM: This study was performed to better characterize accessibility to electronic health records (EHRs) among informatics professionals in various roles, settings, and organizations across the United States and internationally. BACKGROUND: The EHR landscape has evolved significantly in recent years, though challenges remain in key areas such as usability. While patient access to electronic health information has gained more attention, levels of access among informatics professionals, including those conducting usability research, have not been well described in the literature. Ironically, many informatics professionals whose aim is to improve EHR design have restrictions on EHR access or publication, which interfere with broad dissemination of findings in areas of usability research. METHODS: To quantify the limitations on EHR access and publication rights, we conducted a survey of informatics professionals from a broad spectrum of roles including practicing clinicians, researchers, administrators, and members of industry. Results were analyzed and levels of EHR access were stratified by role, organizational affiliation, geographic region, EHR type, and restrictions with regard to publishing results of usability testing, including screenshots. RESULTS: 126 respondents completed the survey, representing all major geographic regions in the United States. 71.5 percent of participants reported some level of EHR access, while 13 percent reported no access whatsoever. Rates of no-access were higher among faculty members and researchers (19 percent). Among faculty members and researchers, 72 percent could access the EHR for usability and/or research purposes, but, of those, fewer than 1 in 3 could freely publish screenshots with results of usability testing and half could not publish such data at all. Across users from all roles, only 21 percent reported the ability to publish screenshots freely without restrictions. CONCLUSIONS: This study offers insight into current patterns of EHR accessibility among informatics professionals, highlighting restrictions that limit dissemination of usability research and testing. Further conversations and shared responsibility among the various stakeholders in industry, government, health care organizations, and informatics professionals are vital to continued EHR optimization.

Visual Progression Analysis of Event Sequence Data.

Event sequence data is common to a broad range of application domains, from security to health care to scholarly communication. This form of data captures information about the progression of events for an individual entity (e.g., a computer network device; a patient; an author) in the form of a series of time-stamped observations. Moreover, each event is associated with an event type (e.g., a computer login attempt, or a hospital discharge). Analyses of event sequence data have been shown to help reveal important temporal patterns, such as clinical paths resulting in improved outcomes, or an understanding of common career trajectories for scholars. Moreover, recent research has demonstrated a variety of techniques designed to overcome methodological challenges such as large volumes of data and high dimensionality. However, the effective identification and analysis of latent stages of progression, which can allow for variation within different but similarly evolving event sequences, remain a significant challenge with important real-world motivations. In this paper, we propose an unsupervised stage analysis algorithm to identify semantically meaningful progression stages as well as the critical events which help define those stages. The algorithm follows three key steps: (1) event representation estimation, (2) event sequence warping and alignment, and (3) sequence segmentation. We also present a novel visualization system, ET2, which interactively illustrates the results of the stage analysis algorithm to help reveal evolution patterns across stages. Finally, we report three forms of evaluation for ET2: (1) case studies with two real-world datasets, (2) interviews with domain expert users, and (3) a performance evaluation on the progression analysis algorithm and the visualization design.

EventThread: Visual Summarization and Stage Analysis of Event Sequence Data.

Event sequence data such as electronic health records, a person's academic records, or car service records, are ordered series of events which have occurred over a period of time. Analyzing collections of event sequences can reveal common or semantically important sequential patterns. For example, event sequence analysis might reveal frequently used care plans for treating a disease, typical publishing patterns of professors, and the patterns of service that result in a well-maintained car. It is challenging, however, to visually explore large numbers of event sequences, or sequences with large numbers of event types. Existing methods focus on extracting explicitly matching patterns of events using statistical analysis to create stages of event progression over time. However, these methods fail to capture latent clusters of similar but not identical evolutions of event sequences. In this paper, we introduce a novel visualization system named EventThread which clusters event sequences into threads based on tensor analysis and visualizes the latent stage categories and evolution patterns by interactively grouping the threads by similarity into time-specific clusters. We demonstrate the effectiveness of EventThread through usage scenarios in three different application domains and via interviews with an expert user.

RCLens: Interactive Rare Category Exploration and Identification.

Rare category identification is an important task in many application domains, ranging from network security, to financial fraud detection, to personalized medicine. These are all applications which require the discovery and characterization of sets of rare but structurally-similar data entities which are obscured within a larger but structurally different dataset. This paper introduces RCLens, a visual analytics system designed to support user-guided rare category exploration and identification. RCLens adopts a novel active learning-based algorithm to iteratively identify more accurate rare categories in response to user-provided feedback. The algorithm is tightly integrated with an interactive visualization-based interface which supports a novel and effective workflow for rare category identification. This paper (1) defines RCLens' underlying active-learning algorithm; (2) describes the visualization and interaction designs, including a discussion of how the designs support user-guided rare category identification; and (3) presents results from an evaluation demonstrating RCLens' ability to support the rare category identification process.