A community-of-practice-based evaluation methodology for knowledge intensive computational methods and its application to multimorbidity decision support.

OBJECTIVE: The study has dual objectives. Our first objective (1) is to develop a community-of-practice-based evaluation methodology for knowledge-intensive computational methods. We target a whitebox analysis of the computational methods to gain insight on their functional features and inner workings. In more detail, we aim to answer evaluation questions on (i) support offered by computational methods for functional features within the application domain; and (ii) in-depth characterizations of the underlying computational processes, models, data and knowledge of the computational methods. Our second objective (2) involves applying the evaluation methodology to answer questions (i) and (ii) for knowledge-intensive clinical decision support (CDS) methods, which operationalize clinical knowledge as computer interpretable guidelines (CIG); we focus on multimorbidity CIG-based clinical decision support (MGCDS) methods that target multimorbidity treatment plans. MATERIALS AND METHODS: Our methodology directly involves the research community of practice in (a) identifying functional features within the application domain; (b) defining exemplar case studies covering these features; and (c) solving the case studies using their developed computational methods-research groups detail their solutions and functional feature support in solution reports. Next, the study authors (d) perform a qualitative analysis of the solution reports, identifying and characterizing common themes (or dimensions) among the computational methods. This methodology is well suited to perform whitebox analysis, as it directly involves the respective developers in studying inner workings and feature support of computational methods. Moreover, the established evaluation parameters (e.g., features, case studies, themes) constitute a re-usable benchmark framework, which can be used to evaluate new computational methods as they are developed. We applied our community-of-practice-based evaluation methodology on MGCDS methods. RESULTS: Six research groups submitted comprehensive solution reports for the exemplar case studies. Solutions for two of these case studies were reported by all groups. We identified four evaluation dimensions: detection of adverse interactions, management strategy representation, implementation paradigms, and human-in-the-loop support. Based on our whitebox analysis, we present answers to the evaluation questions (i) and (ii) for MGCDS methods. DISCUSSION: The proposed evaluation methodology includes features of illuminative and comparison-based approaches; focusing on understanding rather than judging/scoring or identifying gaps in current methods. It involves answering evaluation questions with direct involvement of the research community of practice, who participate in setting up evaluation parameters and solving exemplar case studies. Our methodology was successfully applied to evaluate six MGCDS knowledge-intensive computational methods. We established that, while the evaluated methods provide a multifaceted set of solutions with different benefits and drawbacks, no single MGCDS method currently provides a comprehensive solution for MGCDS. CONCLUSION: We posit that our evaluation methodology, applied here to gain new insights into MGCDS, can be used to assess other types of knowledge-intensive computational methods and answer other types of evaluation questions. Our case studies can be accessed at our GitHub repository (https://github.com/william-vw/MGCDS).

Supporting physicians in the coordination of distributed execution of CIGs to treat comorbid patients.

Clinical Practice Guidelines (CPGs) encode the "best" medical practices to treat patients affected by a specific disease and are widely used in the medical practice. Starting from the '90s', several Computer-Interpretable Guideline (CIG) systems have been devised to provide physicians with CPG-based decision support. CPGs (and CIGs) are devoted to provide evidence-based recommendations for one specific disease. In order to support the treatment of patients affected by multiple diseases (i.e., comorbid patients), challenging additional tasks have to be addressed, such as (i) the detection of the interactions between CIG actions, (ii) their management, and, finally, (iii) the "merge" or conciliation of the CIGs. Several CIG approaches have been recently extended in order to face (at least one of) such challenging problems, and one of them is GLARE. However, besides the solutions to tasks (i)-(iii) above, the "run-time" support to physicians treating a comorbid patient requires additional capabilities, to support the distribution of the management of interactions and of the execution of CIGs among different physicians. In this paper, we propose a general framework, based on GLARE and GLARE-SSCPM, to provide such additional capabilities.

A machine learning approach for predicting high risk hospitalized patients with COVID-19 SARS-Cov-2.

BACKGROUND: This study aimed to explore whether explainable Artificial Intelligence methods can be fruitfully used to improve the medical management of patients suffering from complex diseases, and in particular to predict the death risk in hospitalized patients with SARS-Cov-2 based on admission data. METHODS: This work is based on an observational ambispective study that comprised patients older than 18 years with a positive SARS-Cov-2 diagnosis that were admitted to the hospital Azienda Ospedaliera "SS Antonio e Biagio e Cesare Arrigo", Alessandria, Italy from February, 24 2020 to May, 31 2021, and that completed the disease treatment inside this structure. The patients'medical history, demographic, epidemiologic and clinical data were collected from the electronic medical records system and paper based medical records, entered and managed by the Clinical Study Coordinators using the REDCap electronic data capture tool patient chart. The dataset was used to train and to evaluate predictive ML models. RESULTS: We overall trained, analysed and evaluated 19 predictive models (both supervised and unsupervised) on data from 824 patients described by 43 features. We focused our attention on models that provide an explanation that is understandable and directly usable by domain experts, and compared the results against other classical machine learning approaches. Among the former, JRIP showed the best performance in 10-fold cross validation, and the best average performance in a further validation test using a different patient dataset from the beginning of the third COVID-19 wave. Moreover, JRIP showed comparable performances with other approaches that do not provide a clear and/or understandable explanation. CONCLUSIONS: The ML supervised models showed to correctly discern between low-risk and high-risk patients, even when the medical disease context is complex and the list of features is limited to information available at admission time. Furthermore, the models demonstrated to reasonably perform on a dataset from the third COVID-19 wave that was not used in the training phase. Overall, these results are remarkable: (i) from a medical point of view, these models evaluate good predictions despite the possible differences entitled with different care protocols and the possible influence of other viral variants (i.e. delta variant); (ii) from the organizational point of view, they could be used to optimize the management of health-care path at the admission time.

Developing the Digital Healthcare Workforce in Italy: The SIBIM Experience.

The digital healthcare workforce is usually composed of two major types of professionals: the healthcare workers, who are the users of eHealth, and the health informatics developers, who are usually computer scientists, biomedical engineers, or other technical experts. Health informatics educators have the responsibility to develop the appropriate skills for both, acting within their specific curricula. Here we present the experience of the Italian Society of Biomedical Informatics (SIBIM) and show that, whereas the technical curricula are widely covered with a large range of topics, the eHealth education in medical curricula is often limited to simple bioengineering and informatics skills, thus suggesting that eHealth associations and organizations at the national level should focus their efforts towards increasing the level of eHealth contents in medical schools.

Supporting the distributed execution of clinical guidelines by multiple agents.

Clinical guidelines (GLs) are widely adopted in order to improve the quality of patient care, and to optimize it. To achieve such goals, their application on a specific patient usually requires the interventions of different agents, with different roles (e.g., physician, nurse), abilities (e.g., specialist in the treatment of alcohol-related problems) and contexts (e.g., many chronic patients may be treated at home). Additionally, the responsibility of the application of a guideline to a patient is usually retained by a physician, but delegation of responsibility (of the whole guideline, or of a part of it) is often used\required (e.g., delegation to a specialist), as well as the possibility, for a responsible, to select the executor of an action (e.g., a physician may retain the responsibility of an action, but delegate to a nurse its execution). To manage such phenomena, proper support to agent interaction and communication must be provided, providing agents with facilities for (1) treatment continuity (2) contextualization, (3) responsibility assignment and delegation (4) check of agent "appropriateness". In this paper we extend GLARE, a computerized GL management system, to support such needs. We illustrate our approach by means of a practical case study.

META-GLARE: A meta-system for defining your own computer interpretable guideline system-Architecture and acquisition.

CONTEXT: Several different computer-assisted management systems of computer interpretable guidelines (CIGs) have been developed by the Artificial Intelligence in Medicine community. Each CIG system is characterized by a specific formalism to represent CIGs, and usually provides a manager to acquire, consult and execute them. Though there are several commonalities between most formalisms in the literature, each formalism has its own peculiarities. OBJECTIVE: The goal of our work is to provide a flexible support to the extension or definition of CIGs formalisms, and of their acquisition and execution engines. Instead of defining "yet another CIG formalism and its manager", we propose META-GLARE (META Guideline Acquisition, Representation, and Execution), a "meta"-system to define new CIG systems. METHOD AND MATERIALS: In this paper, META-GLARE, a meta-system to define new CIG systems, is presented. We try to capture the commonalities among current CIG approaches, by providing (i) a general manager for the acquisition, consultation and execution of hierarchical graphs (representing the control flow of actions in CIGs), parameterized over the types of nodes and of arcs constituting it, and (ii) a library of different elementary components of guidelines nodes (actions) and arcs, in which each type definition involves the specification of how objects of this type can be acquired, consulted and executed. We provide generality and flexibility, by allowing free aggregations of such elementary components to define new primitive node and arc types. RESULTS: We have drawn several experiments, in which we have used META-GLARE to build a CIG system (Experiment 1 in Section 8), or to extend it (Experiments 2 and 3). Such experiments show that META-GLARE provides a useful and easy-to-use support to such tasks. For instance, re-building the Guideline Acquisition, Representation, and Execution (GLARE) system using META-GLARE required less than one day (Experiment 1). CONCLUSIONS: META-GLARE is a meta-system for CIGs supporting fast prototyping. Since META-GLARE provides acquisition and execution engines that are parametric over the specific CIG formalism, it supports easy update and construction of CIG systems.

META-GLARE: a shell for CIG systems.

In the last twenty years, many different approaches to deal with Computer-Interpretable clinical Guidelines (CIGs) have been developed, each one proposing its own representation formalism (mostly based on the Task-Network Model) execution engine. We propose META-GLARE a shell for easily defining new CIG systems. Using META-GLARE, CIG system designers can easily define their own systems (basically by defining their representation language), with a minimal programming effort. META-GLARE is thus a flexible and powerful vehicle for research about CIGs, since it supports easy and fast prototyping of new CIG systems.

Supporting a distributed execution of clinical guidelines.

Clinical guidelines (GL) play an important role in medical practice: the one of optimizing the quality of patient care on the basis of evidence based medicine. In order to achieve this goal, the interaction between different agents, who cooperate in the execution of the same GL, is a crucial issue. As a matter of fact, in many cases (e.g. in chronic disorders) the GL execution requires that patient treatment is not performed/completed in the hospital, but is continued in different contexts (e.g. at home, or in the general practitioner's ambulatory), under the responsibility of different agents. In this situation, the correct interaction and communication between the agents themselves is critical for the quality of care, and human resources coordination is a key issue to be addressed by the managers of the involved healthcare services. In this paper we describe how GLARE (Guideline Acquisition, Representation, and Execution), a computerized GL management system, has been extended in order to support such a need. In particular, we have provided: (i) an extension to GL actions representation languages, (ii) proper scheduling and (iii) querying services. By means of these enhancements we aimed at guaranteeing (1) treatment continuity and (2) responsibility assignment support in the various steps of a coordinated and distributed patient care process. We illustrate our approach by means of a practical case study.

Managing proposals and evaluations of updates to medical knowledge: theory and applications.

The process of keeping up-to-date the medical knowledge stored in relational databases is of paramount importance. Since quality and reliability of medical knowledge are essential, in many cases physicians' proposals of updates must undergo experts' evaluation before possibly becoming effective. However, until now no theoretical framework has been provided in order to cope with this phenomenon in a principled and non-ad hoc way. Indeed, such a framework is important not only in the medical domain, but in all Wikipedia-like contexts in which evaluation of update proposals is required. In this paper we propose GPVM (General Proposal Vetting Model), a general model to cope with update proposal⧹evaluation in relational databases. GPVM extends the current theory of temporal relational databases and, in particular, BCDM - Bitemporal Conceptual Data Model - "consensus" model, providing a new data model, new operations to propose and accept⧹reject updates, and new algebraic operators to query proposals. The properties of GPVM are also studied. In particular, GPVM is a consistent extension of BCDM and it is reducible to it. These properties ensure consistency with most relational temporal database frameworks, facilitating implementation on top of current frameworks and interoperability with previous approaches.

An implicit approach to deal with periodically repeated medical data.

CONTEXT: Temporal information plays a crucial role in medicine, so that in medical informatics there is an increasing awareness that suitable database approaches are needed to store and support it. Specifically, a great amount of clinical data (e.g., therapeutic data) are periodically repeated. Although an explicit treatment is possible in most cases, it causes severe storage and disk I/O problems. OBJECTIVE: In this paper, we propose an innovative approach to cope with periodic relational medical data in an implicit way. METHODS: We propose a new data model, representing periodic data in a compact (implicit) way, which is a consistent extension of TSQL2 consensus approach. Then, we identify some important types of temporal queries, and present query answering algorithms to answer them. Finally, we also run experiments to evaluate our approach. RESULTS: The experiments show that our approach outperforms current explicit approaches, especially as regard disk I/O. CONCLUSION: We have provided an implicit approach to periodic data with is a consistent extension of TSQL2 (and which is thus grant interoperable with it), and we have experimentally proven that it outperforms current explicit approaches.

Exceptions handling within GLARE clinical guideline framework.

Computerized clinical guidelines (CIGs) are widely adopted in order to assist practitioner and patient decision making. However, a main problem in their adoption is the fact that, during guidelines executions on specific patients, unpredictable facts and conditions (henceforth called exceptions) may occur. A proper and immediate treatment of such exception is necessary, but most current software systems coping with CIGs do not support it. In this paper, we describe how the GLARE system has been extended to deal with exceptions in CIGs.

Supporting human interaction and human resources coordination in distributed clinical guidelines.

Clinical guidelines (GL) play an important role in medical practice: the one of optimizing the quality of patient care on the basis of the best and most recent evidence based medicine. In order to achieve this goal, the interaction between different actors, who cooperate in the execution of the same GL, is a crucial issue. As a matter of fact, in many cases (e.g. in chronic disease treatment) the GL execution requires that patient treatment is not performed/completed in the hospital, but is continued in different contexts (e.g. at home, or in the general practitioner's ambulatory), under the responsibility of different actors. In this situation, the correct interaction and communication between the actors themselves is critical for the quality of care, and human resources coordination is a key issue to be addressed by the managers of the involved healthcare service. In this paper we describe how computerized GL management can be extended in order to support such needs, and we illustrate our approach by means of a practical case study.

Towards an implicit treatment of periodically-repeated medical data.

Temporal information plays a crucial role in medicine, so that in Medical Informatics there is an increasing awareness that suitable database approaches are needed to store and support it. Specifically, a great amount of clinical data (e.g., therapeutic data) are periodically repeated. Although an explicit treatment is possible in most cases, it causes severe storage and disk I/O problems. In this paper, we propose an innovative approach to cope with periodic medical data in an implicit way. We propose a new data model, representing periodic data in a compact (implicit) way, which is a consistent extension of TSQL2 consensus approach. Then, we identify some important types of temporal queries, and present query answering algorithms to answer them. We also sketch a temporal relational algebra for our approach. Finally, we show experimentally that our approach outperforms current explicit approaches.

Adopting model checking techniques for clinical guidelines verification.

OBJECTIVES: Clinical guidelines (GLs) are assuming a major role in the medical area, in order to grant the quality of the medical assistance and to optimize medical treatments within healthcare organizations. The verification of properties of the GL (e.g., the verification of GL correctness with respect to several criteria) is a demanding task, which may be enhanced through the adoption of advanced Artificial Intelligence techniques. In this paper, we propose a general and flexible approach to address such a task. METHODS AND MATERIALS: Our approach to GL verification is based on the integration of a computerized GL management system with a model-checker. We propose a general methodology, and we instantiate it by loosely coupling GLARE, our system for acquiring, representing and executing GLs, with the model-checker SPIN. RESULTS: We have carried out an in-depth analysis of the types of properties that can be effectively verified using our approach, and we have completed an overview of the usefulness of the verification task at the different stages of the GL life-cycle. In particular, experimentation on a GL for ischemic stroke has shown that the automatic verification of properties in the model checking approach is able to discover inconsistencies in the GL that cannot be detected in advance by hand. CONCLUSION: Our approach thus represents a further step in the direction of general and flexible automated GL verification, which also meets usability requirements.

Applying artificial intelligence to clinical guidelines: the GLARE approach.

We present GLARE, a domain-independent system for acquiring, representing and executing clinical guidelines (GL). GLARE is characterized by the adoption of Artificial Intelligence (AI) techniques in the definition and implementation of the system. First of all, a high-level and user-friendly knowledge representation language has been designed. Second, a user-friendly acquisition tool, which provides expert physicians with various forms of help, has been implemented. Third, a tool for executing GL on a specific patient has been made available. At all the levels above, advanced AI techniques have been exploited, in order to enhance flexibility and user-friendliness and to provide decision support. Specifically, this chapter focuses on the methods we have developed in order to cope with (i) automatic resource-based adaptation of GL, (ii) representation and reasoning about temporal constraints in GL, (iii) decision making support, and (iv) model-based verification. We stress that, although we have devised such techniques within the GLARE project, they are mostly system-independent, so that they might be applied to other guideline management systems.

Automatic checking of the correctness of clinical guidelines in GLARE.

Representing clinical guidelines is a very complex knowledge-representation task, requiring a lot of expertise and efforts. Nevertheless, guideline representations often contain several kinds of errors. Therefore, checking the well-formedness and correctness of a guideline representation is an important task, which can be drastically improved with the adoption of computer programs. In this paper, we discuss the advanced facilities provided by the GLARE system to assist physicians to produce correct representations of clinical guidelines.

Supporting therapy selection in computerized clinical guidelines by means of decision theory.

Supporting therapy selection is a fundamental task for a system for the computerized management of clinical guidelines (GL). The goal is particularly critical when no alternative is really better than the others, from a strictly clinical viewpoint. In these cases, decision theory appears to be a very suitable means to provide advice. In this paper, we describe how algorithms for calculating utility, and for evaluating the optimal policy, can be exploited to fit the GL management context.

Automatic treatment of temporal issues in clinical guidelines in the GLARE system.

Temporal constraints play a fundamental role in clinical guidelines. For example, temporal indeterminacy, constraints about duration, delays between actions and periodic repetitions of actions are essential in order to cope with clinical therapies. This paper proposes a computer-based approach in order to deal with temporal constraints in clinical guidelines. Specifically, it provides the possibility to represent such constraints and reason with them (i.e., perform inferences in the form of constraint propagation). We first propose a temporal representation formalism and two constraint propagation algorithms operating on it, and then we show how they can be exploited in order to provide clinical guideline systems with different temporal facilities. Our approach offers several advantages: for example, during the guideline acquisition phase, it enables to represent temporal constraints and to check their consistency; during the execution phase, it allows the physician to check the consistency between action execution-times and the constraints in the guidelines, and to provide query answering and temporal simulation facilities (e.g., when choosing among alternative paths in a guideline).

Extending temporal databases to deal with telic/atelic medical data.

OBJECTIVE: In this paper, we aim at defining a general-purpose data model and query language coping with both "telic" and "atelic" medical data. BACKGROUND: In the area of Medical Informatics, there is an increasing realization that temporal information plays a crucial role, so that suitable database models and query languages are needed to store and support it. However, despite the wide range of approaches in the area, in this paper we show that a relevant class of medical data cannot be properly dealt with. METHODOLOGY: We first show that data models based on the "point-based" semantics, which is (implicitly or explicitly) assumed by the totality of temporal database approaches, have several limitations when dealing with "telic" data. We then propose a new model (based on the "interval-based" semantics) to cope with such data, and extend the query language accordingly. RESULTS: We propose a new three-sorted model and a query language to properly deal with both "telic" and "atelic" medical data (as well as non-temporal data). Our query language is flexible, since it allows one to switch from "atelic" to "telic" data, and vice versa. CONCLUSION: In this paper, we demonstrate the feasibility of a database approach copying with both telic and atelic data as needed in several (medical) applications.

Towards a comprehensive treatment of repetitions, periodicity and temporal constraints in clinical guidelines.

OBJECTIVE: In this paper, we define a principled approach to represent temporal constraints in clinical guidelines and to reason (i.e., perform inferences in the form of constraint propagation) on them. We consider different types of constraints, including composite and repeated actions, and propose different types of temporal functionalities (e.g., temporal consistency checking). BACKGROUND: Constraints about actions, durations, delays and periodic repetitions of actions are an intrinsic part of most clinical guidelines. Although several approaches provide expressive temporal formalisms, only few of them deal with the related temporal reasoning issues. METHODOLOGY: We first propose a temporal representation formalism and two temporal reasoning algorithms. Then, we consider the trade-off between the expressiveness of the formalism and the computational complexity of the algorithms, in order to devise a correct, complete and tractable approach. Finally, we show how the algorithms can be exploited to provide clinical guideline systems with different types of temporal facilities. RESULTS: Our approach offers several advantages. During the guideline acquisition phase, it enables to represent temporal constraints, and to check their consistency. In the execution phase, it checks the consistency between the execution times of the actions and the constraints in the guidelines, and provides query answering and simulation facilities.