Evidence-Based Health Informatics as the Foundation for the COVID-19 Response: A Joint Call for Action.

BACKGROUND: As a major public health crisis, the novel coronavirus disease 2019 (COVID-19) pandemic demonstrates the urgent need for safe, effective, and evidence-based implementations of digital health. The urgency stems from the frequent tendency to focus attention on seemingly high promising digital health interventions despite being poorly validated in times of crisis. AIM: In this paper, we describe a joint call for action to use and leverage evidence-based health informatics as the foundation for the COVID-19 response and public health interventions. Tangible examples are provided for how the working groups and special interest groups of the International Medical Informatics Association (IMIA) are helping to build an evidence-based response to this crisis. METHODS: Leaders of working and special interest groups of the IMIA, a total of 26 groups, were contacted via e-mail to provide a summary of the scientific-based efforts taken to combat COVID-19 pandemic and participate in the discussion toward the creation of this manuscript. A total of 13 groups participated in this manuscript. RESULTS: Various efforts were exerted by members of IMIA including (1) developing evidence-based guidelines for the design and deployment of digital health solutions during COVID-19; (2) surveying clinical informaticians internationally about key digital solutions deployed to combat COVID-19 and the challenges faced when implementing and using them; and (3) offering necessary resources for clinicians about the use of digital tools in clinical practice, education, and research during COVID-19. DISCUSSION: Rigor and evidence need to be taken into consideration when designing, implementing, and using digital tools to combat COVID-19 to avoid delays and unforeseen negative consequences. It is paramount to employ a multidisciplinary approach for the development and implementation of digital health tools that have been rapidly deployed in response to the pandemic bearing in mind human factors, ethics, data privacy, and the diversity of context at the local, national, and international levels. The training and capacity building of front-line workers is crucial and must be linked to a clear strategy for evaluation of ongoing experiences.

The Value of Teledermoscopy to the Expertise of General Practitioners Diagnosing Skin Disorders Based on ICD-10 Coding.

Early recognition of skin cancer is vital to enhance patient outcomes. Teledermoscopy (TDsc), a telemedicine service, supports general practitioners (GPs) in gaining fast access to dermatologists' feedback to detect skin cancer. This study aimed to assess if GPs gain expertise in diagnosing skin disorders after continued use of TDsc, based on diagnosis classification by the International Statistical Classification of Diseases and Related Health Problems (ICD-10). A retrospective study was conducted on TDsc consultations sent by GPs to teledermatologists in the Netherlands (July 2015 - June 2018). GP sensitivity and confirmed cases in diagnosing skin disorders slightly increased over time. However, the total positive predictive value showed a decrease. In three years, 43 melanomas were diagnosed by the TD for which the GP did not provide a (correct) pre-diagnose. Though GPs appear to improve their expertise in skin disorder detection after continued TDsc use, TDsc remains imperative to early melanoma detection.

Is Teledermoscopy Improving General Practitioner Skin Cancer Care?

Skin cancer incidences have tripled in the Netherlands for the last twenty years and are expected to increase even more in the coming years. Teledermoscopy (TDsc) is implemented in Dutch practice to support and enhance early skin cancer detection by general practitioners (GPs) through remote consultation with dermatologists. This study assesses the effect of TDsc consultation on the quality and efficiency of skin cancer care in the primary setting by analyzing 10,184 TDsc consultations.

Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: A mixed method usability approach.

INTRODUCTION: Cumbersome electronic patient record (EPR) interfaces may complicate data-entry in clinical practice. Completeness of data entered in the EPR determines, among other things, the value of computerized clinical decision support (CCDS). Quantitative usability evaluations can provide insight into mismatches between the system design model of data entry and users' data entry behavior, but not into the underlying causes for these mismatches. Mixed method usability evaluation studies may provide these insights, and thus support generating redesign recommendations for improving an EPR system's data entry interface. AIM: To improve the usability of the data entry interface of an EPR system with CCDS in the field of cardiac rehabilitation (CR), and additionally, to assess the value of a mixed method usability approach in this context. METHODS: Seven CR professionals performed a think-aloud usability evaluation both before (beta-version) and after the redesign of the system. Observed usability problems from both evaluations were analyzed and categorized using Zhang et al.'s heuristic principles of good interface design. We combined the think-aloud usability evaluation of the system's beta-version with the measurement of a new usability construct: users' deviations in action sequence from the system's predefined data entry order sequence. Recommendations for redesign were implemented. We assessed whether the redesign improved CR professionals' (1) task efficacy (with respect to the completeness of data they collected), and (2) task efficiency (with respect to the average number of mouse clicks they needed to complete data entry subtasks). RESULTS: With the system's beta version, 40% of health care professionals' navigation actions through the system deviated from the predefined next system action. The causes for these deviations as revealed by the think-aloud method mostly concerned mismatches between the system design model for data entry action sequences and users expectations of these action sequences, based on their paper-based daily routines. This caused non completion of data entry tasks (31% of main tasks completed), and more navigation actions than minimally required (146% of the minimum required). In the redesigned system the data entry navigational structure was organized in a flexible way around an overview screen to better mimic users' paper-based daily routines of collecting patient data. This redesign resulted in an increased number of completed main tasks (70%) and a decrease in navigation actions (133% of the minimum required). The think-aloud usability evaluation of the redesigned system showed that remaining problems concerned flexibility (e.g., lack of customization options) and consistency (mainly with layout and position of items on the screen). CONCLUSION: The mixed method usability evaluation was supportive in revealing the magnitude and causes of mismatches between the system design model of data-entry with users' data entry behavior. However, as both task efficacy and efficiency were still not optimal with the redesigned EPR, we advise to perform a cognitive analysis on end users' mental processes and behavior patterns in daily work processes specifically during the requirements analysis phase of development of interactive healthcare information systems.

The value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool.

OBJECTIVE: To compare the performance of the Concurrent (CTA) and Retrospective (RTA) Think Aloud method and to assess their value in a formative usability evaluation of an Intensive Care Registry-physician data query tool designed to support ICU quality improvement processes. METHODS: Sixteen representative intensive care physicians participated in the usability evaluation study. Subjects were allocated to either the CTA or RTA method by a matched randomized design. Each subject performed six usability-testing tasks of varying complexity in the query tool in a real-working context. Methods were compared with regard to number and type of problems detected. Verbal protocols of CTA and RTA were analyzed in depth to assess differences in verbal output. Standardized measures were applied to assess thoroughness in usability problem detection weighted per problem severity level and method overall effectiveness in detecting usability problems with regard to the time subjects spent per method. RESULTS: The usability evaluation of the data query tool revealed a total of 43 unique usability problems that the intensive care physicians encountered. CTA detected unique usability problems with regard to graphics/symbols, navigation issues, error messages, and the organization of information on the query tool's screens. RTA detected unique issues concerning system match with subjects' language and applied terminology. The in-depth verbal protocol analysis of CTA provided information on intensive care physicians' query design strategies. Overall, CTA performed significantly better than RTA in detecting usability problems. CTA usability problem detection effectiveness was 0.80 vs. 0.62 (p<0.05) respectively, with an average difference of 42% less time spent per subject compared to RTA. In addition, CTA was more thorough in detecting usability problems of a moderate (0.85 vs. 0.7) and severe nature (0.71 vs. 0.57). CONCLUSION: In this study, the CTA is more effective in usability-problem detection and provided clarification of intensive care physician query design strategies to inform redesign of the query tool. However, CTA does not outperform RTA. The RTA additionally elucidated unique usability problems and new user requirements. Based on the results of this study, we recommend the use of CTA in formative usability evaluation studies of health information technology. However, we recommend further research on the application of RTA in usability studies with regard to user expertise and experience when focusing on user profile customized (re)design.

Formative usability evaluation of a web-based insulin self-titration system: preliminary results.

We developed a web-based system supporting patients in insulin self-titration and their caregivers in monitoring patients' self-management activities. Since usability flaws could cause user attrition and compromise patient safety, we evaluated the system's usability prior to its implementation in practice. Two pairs of researchers conducted cognitive walkthrough sessions and identified 81 unique usability problems, including four with a potential impact on patient safety. Usability evaluations could reveal many usability problems and allow solving the problems while avoiding user attrition and potential adverse patient events.

Pre-post evaluation of physicians' satisfaction with a redesigned electronic medical record system.

Physicians' acceptance of Electronic Medical Record Systems (EMRs) is closely related to their usability. Knowledge about end-users' opinions on usability of an EMR system may contribute to planning for the next phase of the usability cycle of the system. A demand for integration of new functionalities, such as computerized order entry and an electronic patient status led to redesign of our EMR system, which had been in use for over 8 years at the Academic Medical Center of Amsterdam. The aim of this study was to understand whether the redesigned EMR system was an improvement of the earlier EMR and which system aspects accounted for user satisfaction and which did not. We conducted a formative pre- and post usability evaluation of our former and redesigned EMR system. For the assessment of both system versions' usability, we distributed two standardized usability questionnaires among 150 clinicians who routinely had used the older EMR system and had been working with its newer version for 6 weeks. Though overall user satisfaction was relatively high for both EMR systems, screen layout and interaction structure proved less easy to work with in the newer EMR system. The new EMR system however was more appreciated because of its enhanced functionality, capabilities and likeable user-interface. The results point to a number of actions that might be useful in future usability improvement efforts of our EMR system and other EMRs.

Usability studies on interactive health information systems; where do we stand?

This paper discusses the preliminary results of a systematic review of the literature on applied usability studies of health information systems in the period 1990 to 2006. Abstracts were included when they described an evaluation of the usability of a health information system. To gain insight into usability methods applied and their properties we constructed a framework to analyze the studies. The framework includes objectives, designs, number of participants, user-profiles, settings, medical domain, and type of health information systems evaluated. Fifty-two Papers were included in the review. Findings show that from 2002 an increasing trend can be observed of publication of usability studies. Most studies discuss summative usability results on working systems thereby focusing on systems' adoption problems. Formative usability studies lack a uniform way to describe how study results contributed to the system's iterative development cycle.

The significance of a usability evaluation of an emerging laboratory order entry system.

OBJECTIVES: To assess the usability of an emerging POE system, OM/Lab, for the computer-supported ordering of laboratory tests. We were more specifically interested in the relation of the usability problems detected in the user testing sessions with the order behaviour in terms of efficiency and errors in ordering. METHODS: A cognitive walkthrough of the OM/Lab system was conducted by two analysts using four real-life scenarios for ordering laboratory tests, which were reviewed for comprehensiveness by an expert clinician. Thereafter, the OM/Lab system was evaluated on its usability in testing sessions with seven potential end-users of the system performing these same four scenarios. The results of these end-user testing sessions were used to analyze the effect of usability flaws on the quality of ordering in terms of omissions, errors in orders and cancelled orders. RESULTS: The analyses revealed a total of 33 usability problems, of which 25 problems were revealed both by the cognitive walkthrough and in the end-user sessions. These 25 usability problems indeed led to inefficient order behaviour, omissions and errors in orders and even to cancelled orders. DISCUSSION: Our results revealed that the OM/Lab system suffered from a high number of usability flaws. The interface design flaws were, among other things, related to misallocation of buttons on the screen, incomprehensibility of button labels and feedback containing no relevant information to the user about the cause of errors made and consequences of a user's action. Additionally, our user test session results indicated that the OM/Lab system also suffered from user interaction problems of a more socio-technical nature. These sessions revealed, among other things, that the more specific action sequences to be executed within the ordering model of the OM/Lab system did not correspond to the daily working routines of end-users and that the grouping of laboratory tests within clusters did not match the paper-based order forms. The seemingly negative effects of these usability flaws on the quality of ordering and the inefficiency of work processes during the pilot implementation finally lead to withdrawal of the OM/Lab system from clinical practice. Though the system implementation failed, our usability study brought research on (re)designing and evaluating clinical computer applications at a higher status in our institution. It is now recognized that usability evaluation studies that will support good quality of clinical practice are highly important.

Usability evaluation of a laboratory order entry system: cognitive walkthrough and think aloud combined.

Though designing physician order entry (POE) systems has been a highly discussed research topic in the last decade, it seems that engineering POE systems that truly optimise the quality and efficiency of ordering is still a challenge. This study addresses a usability evaluation of an emerging POE for the electronically ordering of laboratory tests. By applying two complementary cognitive approaches, specifically the cognitive walkthrough and think aloud method with seven potential end users of the system, we analyzed usability problems in the prototype system and their potential effect on the quality of orders in terms of efficiency and errors in ordering. The cognitive walkthrough provided a coding scheme that was used to analyse in more detail usability errors encountered during the think aloud tests with the seven end users. The analyses revealed a total of 33 usability problems, which indeed led to inefficiency, omissions in ordering and even to cancelled orders. Most of these usability problems referred to incomprehensiveness of required actions by the user and incomprehensiveness of text used in the system. Next to the discussion of the reasons for these usability problems, the surplus value of the think aloud method as supplementary to the cognitive walkthrough in evaluating physician order entry systems is outlined in this paper.