A multinational study on artificial intelligence adoption: Clinical implementers' perspectives.

BACKGROUND: Despite substantial progress in AI research for healthcare, translating research achievements to AI systems in clinical settings is challenging and, in many cases, unsatisfactory. As a result, many AI investments have stalled at the prototype level, never reaching clinical settings. OBJECTIVE: To improve the chances of future AI implementation projects succeeding, we analyzed the experiences of clinical AI system implementers to better understand the challenges and success factors in their implementations. METHODS: Thirty-seven implementers of clinical AI from European and North and South American countries were interviewed. Semi-structured interviews were transcribed and analyzed qualitatively with the framework method, identifying the success factors and the reasons for challenges as well as documenting proposals from implementers to improve AI adoption in clinical settings. RESULTS: We gathered the implementers' requirements for facilitating AI adoption in the clinical setting. The main findings include 1) the lesser importance of AI explainability in favor of proper clinical validation studies, 2) the need to actively involve clinical practitioners, and not only clinical researchers, in the inception of AI research projects, 3) the need for better information structures and processes to manage data access and the ethical approval of AI projects, 4) the need for better support for regulatory compliance and avoidance of duplications in data management approval bodies, 5) the need to increase both clinicians' and citizens' literacy as respects the benefits and limitations of AI, and 6) the need for better funding schemes to support the implementation, embedding, and validation of AI in the clinical workflow, beyond pilots. CONCLUSION: Participants in the interviews are positive about the future of AI in clinical settings. At the same time, they proposenumerous measures to transfer research advancesinto implementations that will benefit healthcare personnel. Transferring AI research into benefits for healthcare workers and patients requires adjustments in regulations, data access procedures, education, funding schemes, and validation of AI systems.

Cluster and trajectory analysis of motivation in an emergency remote programming course.

Emergency remote teaching is a temporary change in the way education occurs, whereby an educational system unexpectedly becomes entirely remote. This article analyzes the motivation of students undertaking a university course over one semester of emergency remote teaching in the context of the COVID-19 pandemic. University students undertaking a programming course were surveyed three times during one semester, about motivation and COVID concern. This work explores which student motivation profiles existed, how motivation evolved, and whether concern about the pandemic was a factor affecting motivation throughout the course. The most adaptive profile was highly motivated, more prepared and less frustrated by the conditions of the course. However, this cluster experienced the highest levels of COVID-19 concern. The least adaptive cluster behaved as a mirror image of the most adaptive cluster. Clear differences were found between the clusters that showed the most and least concern about COVID-19.

ProDeM: A Process-Oriented Delphi Method for systematic asynchronous and consensual surgical process modelling.

Surgical process models support improving healthcare provision by facilitating communication and reasoning about processes in the medical domain. Modelling surgical processes is challenging as it requires integrating information that might be fragmented, scattered, and not process-oriented. These challenges can be faced by involving healthcare domain experts during process modelling. This paper presents ProDeM: a novel Process-Oriented Delphi Method for the systematic, asynchronous, and consensual modelling of surgical processes. ProDeM is an adaptable and flexible method that acknowledges that: (i) domain experts have busy calendars and might be geographically dispersed, and (ii) various elements of the process model need to be assessed to ensure model quality. The contribution of the paper is twofold as it outlines ProDeM, but also demonstrates its operationalisation in the context of a well-known surgical process. Besides showing the method's feasibility in practice, we also present an evaluation of the method by the experts involved in the demonstration.

The Sequence of Steps: A Key Concept Missing in Surgical Training-A Systematic Review and Recommendations to Include It.

BACKGROUND: Surgical procedures have an inherent feature, which is the sequence of steps. Moreover, studies have shown variability in surgeons' performances, which is valuable to expose residents to different ways to perform a procedure. However, it is unclear how to include the sequence of steps in training programs. METHODS: We conducted a systematic review, including studies reporting explicit teaching of a standard sequence of steps, where assessment considered adherence to a standard sequence, and where faculty or students at any level participated. We searched for articles on PubMed, EMBASE, CINAHL, Web of Science, and Google Scholar databases. RESULTS: We selected nine articles that met the inclusion criteria. The main strategy to teach the sequence was to use videos to demonstrate the procedure. The simulation was the main strategy to assess the learning of the sequence of steps. Non-standardized scoring protocols and written tests with variable validity evidence were the instruments used to assess the learning, and were focused on adherence to a standard sequence and the omission of steps. CONCLUSIONS: Teaching and learning assessment of a standard sequence of steps is scarcely reported in procedural skills training literature. More research is needed to evaluate whether the new strategies to teach and assess the order of steps work. We recommend the use of Surgical Process Models and Surgical Data Science to incorporate the sequence of steps when teaching and assessing procedural skills.

Building Process-Oriented Data Science Solutions for Real-World Healthcare.

The COVID-19 pandemic has highlighted some of the opportunities, problems and barriers facing the application of Artificial Intelligence to the medical domain. It is becoming increasingly important to determine how Artificial Intelligence will help healthcare providers understand and improve the daily practice of medicine. As a part of the Artificial Intelligence research field, the Process-Oriented Data Science community has been active in the analysis of this situation and in identifying current challenges and available solutions. We have identified a need to integrate the best efforts made by the community to ensure that promised improvements to care processes can be achieved in real healthcare. In this paper, we argue that it is necessary to provide appropriate tools to support medical experts and that frequent, interactive communication between medical experts and data miners is needed to co-create solutions. Process-Oriented Data Science, and specifically concrete techniques such as Process Mining, can offer an easy to manage set of tools for developing understandable and explainable Artificial Intelligence solutions. Process Mining offers tools, methods and a data driven approach that can involve medical experts in the process of co-discovering real-world evidence in an interactive way. It is time for Process-Oriented Data scientists to collaborate more closely with healthcare professionals to provide and build useful, understandable solutions that answer practical questions in daily practice. With a shared vision, we should be better prepared to meet the complex challenges that will shape the future of healthcare.

Process-oriented metrics to provide feedback and assess the performance of students who are learning surgical procedures: The percutaneous dilatational tracheostomy case.

PURPOSE: Assessing competency in surgical procedures is key for instructors to distinguish whether a resident is qualified to perform them on patients. Currently, assessment techniques do not always focus on providing feedback about the order in which the activities need to be performed. In this research, using a Process Mining approach, process-oriented metrics are proposed to assess the training of residents in a Percutaneous Dilatational Tracheostomy (PDT) simulator, identifying the critical points in the execution of the surgical process. MATERIALS AND METHODS: A reference process model of the procedure was defined, and video recordings of student training sessions in the PDT simulator were collected and tagged to generate event logs. Three process-oriented metrics were proposed to assess the performance of the residents in training. RESULTS: Although the students were proficient in classic metrics, they did not reach the optimum in process-oriented metrics. Only in 25% of the stages the optimum was achieved in the last session. In these stages, the four more challenging activities were also identified, which account for 32% of the process-oriented metrics errors. CONCLUSIONS: Process-oriented metrics offer a new perspective on surgical procedures performance, providing a more granular perspective, which enables a more specific and actionable feedback for both students and instructors.

Process mining for healthcare: Characteristics and challenges.

Process mining techniques can be used to analyse business processes using the data logged during their execution. These techniques are leveraged in a wide range of domains, including healthcare, where it focuses mainly on the analysis of diagnostic, treatment, and organisational processes. Despite the huge amount of data generated in hospitals by staff and machinery involved in healthcare processes, there is no evidence of a systematic uptake of process mining beyond targeted case studies in a research context. When developing and using process mining in healthcare, distinguishing characteristics of healthcare processes such as their variability and patient-centred focus require targeted attention. Against this background, the Process-Oriented Data Science in Healthcare Alliance has been established to propagate the research and application of techniques targeting the data-driven improvement of healthcare processes. This paper, an initiative of the alliance, presents the distinguishing characteristics of the healthcare domain that need to be considered to successfully use process mining, as well as open challenges that need to be addressed by the community in the future.

Development of a comprehensive Percutaneous Dilatational Tracheostomy process model for procedural training: A Delphi-based experts consensus.

BACKGROUND: Deconstructing a complex procedure improves skills learning, but no model has covered all relevant Percutaneous Dilatational Tracheostomy (PDT) procedural aspects. Moreover, the heterogeneity of techniques described may hinder trainees' competency acquisition. Our objective was to develop a PDT model for procedural training that includes a comprehensive step-by-step design. METHODS: Procedural descriptions were retrieved after a structured search in medical databases. Activities were extracted and the adherence to McKinley's dimensions of procedural competence was analyzed. We developed a comprehensive PDT model, which was further validated through a Delphi-based consensus of Spanish-speaking international experts. RESULTS: The 14 descriptions retrieved for analysis presented a median [interquartile range] of 18 [11-22] steps, covering 3 [2-4] of McKinley's dimensions. The Delphi panel's first model included all McKinley's dimensions, and was answered by 25 experts from nine countries, ending in the second round. The final model included 59 activities divided into six stages (51 from the initial model and eight proposed by experts) and performed by two operators (bronchoscopy and tracheostomy). CONCLUSIONS: We have presented a PDT model that includes necessary competence dimensions to be considered complete. The model was validated by an experts' consensus, allowing to improve procedural training to promote safer patient care.

Simulation-Based Mastery Learning of Bronchoscopy-Guided Percutaneous Dilatational Tracheostomy: Competency Acquisition and Skills Transfer to a Cadaveric Model.

INTRODUCTION: Although simulation-based training has demonstrated improvement of procedural skills and clinical outcomes in different procedures, there are no published training protocols for bronchoscopy-guided percutaneous dilatational tracheostomy (BG-PDT). The objective of this study was to assess the acquisition of BG-PDT procedural competency with a simulation-based mastery learning training program, and skills transfer into cadaveric models. METHODS: Using a prospective interventional design, 8 trainees naive to the procedure were trained in a simulation-based mastery learning BG-PDT program. Students were assessed using a multimodal approach, including blind global rating scale (GRS) scores of video-recorded executions, total procedural time, and hand-motion tracking-derived parameters. The BG-PDT mastery was defined as proficient tracheostomy (successful procedural performance, with less than 3 puncture attempts, and no complications) with GRS scores higher than 21 points (of 25). After mastery was achieved in the simulator, residents performed 1 BG-PDT execution in a cadaveric model. RESULTS: Compared with baseline, in the final training session, residents presented a higher procedural proficiency (0% vs. 100%, P < 0.001), with higher GRS scores [8 (6-8) vs. 25 (24-25), P = 0.01] performed in less time [563 (408-600) vs. 246 (214-267), P = 0.01] and with higher movement economy. Procedural skills were further transferred to the cadaveric model. CONCLUSIONS: Residents successfully acquired BG-PDT procedural skills with a simulation-based mastery learning training program, and skills were effectively transferred to a cadaveric model. This easily replicable program is the first simulation-based BG-PDT training experience reported in the literature, enhancing safe competency acquisition, to further improve patient care.

Mapping the Patient's Journey in Healthcare through Process Mining.

Nowadays, assessing and improving customer experience has become a priority, and has emerged as a key differentiator for business and organizations worldwide. A customer journey (CJ) is a strategic tool, a map of the steps customers follow when engaging with a company or organization to obtain a product or service. The increase of the need to obtain knowledge about customers' perceptions and feelings when interacting with participants, touchpoints, and channels through different stages of the customer life cycle. This study aims to describe the application of process mining techniques in healthcare as a tool to asses customer journeys. The appropriateness of the approach presented is illustrated through a case study of a key healthcare process. Results depict how a healthcare process can be mapped through the CJ components, and its analysis can serve to understand and improve the patient's experience.

What Role Can Process Mining Play in Recurrent Clinical Guidelines Issues? A Position Paper.

In the age of Evidence-Based Medicine, Clinical Guidelines (CGs) are recognized to be an indispensable tool to support physicians in their daily clinical practice. Medical Informatics is expected to play a relevant role in facilitating diffusion and adoption of CGs. However, the past pioneering approaches, often fragmented in many disciplines, did not lead to solutions that are actually exploited in hospitals. Process Mining for Healthcare (PM4HC) is an emerging discipline gaining the interest of healthcare experts, and seems able to deal with many important issues in representing CGs. In this position paper, we briefly describe the story and the state-of-the-art of CGs, and the efforts and results of the past approaches of medical informatics. Then, we describe PM4HC, and we answer questions like how can PM4HC cope with this challenge? Which role does PM4HC play and which rules should be employed for the PM4HC scientific community?

Process-Oriented Instrument and Taxonomy for Teaching Surgical Procedures in Medical Training: The Ultrasound-Guided Insertion of Central Venous Catheter.

Procedural training is relevant for physicians who perform surgical procedures. In the medical education field, instructors who teach surgical procedures need to understand how their students are learning to give them feedback and assess them objectively. The sequence of steps of surgical procedures is an aspect rarely considered in medical education, and state-of-the-art tools for giving feedback and assessing students do not focus on this perspective. Process Mining can help to include this perspective in this field since it has recently been used successfully in some applications. However, these previous developments are more centred on students than on instructors. This paper presents the use of Process Mining to fill this gap, generating a taxonomy of activities and a process-oriented instrument. We evaluated both tools with instructors who teach central venous catheter insertion. The results show that the instructors found both tools useful to provide objective feedback and objective assessment. We concluded that the instructors understood the information provided by the instrument since it provides helpful information to understand students' performance regarding the sequence of steps followed.

Delphi Method to Achieve Clinical Consensus for a BPMN Representation of the Central Venous Access Placement for Training Purposes.

Proper teaching of the technical skills necessary to perform a medical procedure begins with its breakdown into its constituent steps. Currently available methodologies require substantial resources and their results may be biased. Therefore, it is difficult to generate the necessary breakdown capable of supporting a procedural curriculum. The aim of our work was to breakdown the steps required for ultrasound guided Central Venous Catheter (CVC) placement and represent this procedure graphically using the standard BPMN notation. METHODS: We performed the first breakdown based on the activities defined in validated evaluation checklists, which were then graphically represented in BPMN. In order to establish clinical consensus, we used the Delphi method by conducting an online survey in which experts were asked to score the suitability of the proposed activities and eventually propose new activities. RESULTS: Surveys were answered by 13 experts from three medical specialties and eight different institutions in two rounds. The final model included 28 activities proposed in the initial model and four new activities proposed by the experts; seven activities from the initial model were excluded. CONCLUSIONS: The proposed methodology proved to be simple and effective, generating a graphic representation to represent activities, decision points, and alternative paths. This approach is complementary to more classical representations for the development of a solid knowledge base that allows the standardization of medical procedures for teaching purposes.

Recommendations for enhancing the usability and understandability of process mining in healthcare.

Healthcare organizations are confronted with challenges including the contention between tightening budgets and increased care needs. In the light of these challenges, they are becoming increasingly aware of the need to improve their processes to ensure quality of care for patients. To identify process improvement opportunities, a thorough process analysis is required, which can be based on real-life process execution data captured by health information systems. Process mining is a research field that focuses on the development of techniques to extract process-related insights from process execution data, providing valuable and previously unknown information to instigate evidence-based process improvement in healthcare. However, despite the potential of process mining, its uptake in healthcare organizations outside case studies in a research context is rather limited. This observation was the starting point for an international brainstorm seminar. Based on the seminar's outcomes and with the ambition to stimulate a more widespread use of process mining in healthcare, this paper formulates recommendations to enhance the usability and understandability of process mining in healthcare. These recommendations are mainly targeted towards process mining researchers and the community to consider when developing a new research agenda for process mining in healthcare. Moreover, a limited number of recommendations are directed towards healthcare organizations and health information systems vendors, when shaping an environment to enable the continuous use of process mining.

Control-flow analysis of procedural skills competencies in medical training through process mining.

BACKGROUND: Procedural skills are key to good clinical results, and training in them involves a significant amount of resources. Control-flow analysis (ie, the order in which a process is performed) can provide new information for those who train and plan procedural training. This study outlines the steps required for control-flow analysis using process mining techniques in training in an ultrasound-guided internal jugular central venous catheter placement using a simulation. METHODS: A reference process model was defined through a Delphi study, and execution data (event logs) were collected from video recordings from pretraining (PRE), post-training (POST) and expert (EXP) procedure executions. The analysis was performed to outline differences between the model and executions. We analysed rework (activity repetition), alignment-based fitness (conformance with the ideal model) and trace alignment analysis (visual ordering pattern similarities). RESULTS: Expert executions do not present repetition of activities (rework). The POST rework is lower than the PRE, concentrated in the steps of the venous puncture and guidewire placement. The adjustment to the ideal model measure as alignment-based fitness, expressed as a median (25th-75th percentile) of PRE 0.74 (0.68-0.78) is less than POST 0.82 (0.76-0.86) and EXP 0.87 (0.82-0.87). There are no significant differences between POST and EXP. The graphic analysis of alignment and executions shows a progressive increase in order from PRE to EXP executions. CONCLUSION: Process mining analysis is able to pinpoint more difficult steps, assess the concordance between reference mode and executions, and identify control-flow patterns in procedural training courses.

Toward Value-Based Healthcare through Interactive Process Mining in Emergency Rooms: The Stroke Case.

The application of Value-based Healthcare requires not only the identification of key processes in the clinical domain but also an adequate analysis of the value chain delivered to the patient. Data Science and Big Data approaches are technologies that enable the creation of accurate systems that model reality. However, classical Data Mining techniques are presented by professionals as black boxes. This evokes a lack of trust in those techniques in the medical domain. Process Mining technologies are human-understandable Data Science tools that can fill this gap to support the application of Value-Based Healthcare in real domains. The aim of this paper is to perform an analysis of the ways in which Process Mining techniques can support health professionals in the application of Value-Based Technologies. For this purpose, we explored these techniques by analyzing emergency processes and applying the critical timing of Stroke treatment and a Question-Driven methodology. To demonstrate the possibilities of Process Mining in the characterization of the emergency process, we used a real log with 9046 emergency episodes from 2145 stroke patients that occurred from January 2010 to June 2017. Our results demonstrate how Process Mining technology can highlight the differences between the flow of stroke patients compared with that of other patients in an emergency. Further, we show that support for health professionals can be provided by improving their understanding of these techniques and enhancing the quality of care.

Process-Oriented Feedback through Process Mining for Surgical Procedures in Medical Training: The Ultrasound-Guided Central Venous Catheter Placement Case.

Developing high levels of competence in the execution of surgical procedures through training is a key factor for obtaining good clinical results in healthcare. To improve the effectiveness of the training, it is advisable to provide feedback to each student tailored to how the student has performed the procedure on each occasion. Current state-of-the-art feedback is based on Checklists and Global Rating Scales, which indicate whether all process steps have been carried out and the quality of each execution step. However, there is a process perspective that is not captured successfully by these instruments, e.g., steps performed, but in an undesired order, group of activities that are repeated an unnecessary number of times, or an excessive transition time between two consecutive steps. In this research, we propose a novel use of process mining techniques to effectively identify desired and undesired process patterns regarding rework, the order in which activities are performed, and time performance, in order to complement the tailored feedback for surgical procedures using a process perspective. The proposed approach was applied to analyze a real case of ultrasound-guided Central Venous Catheter placement training. It was quantitatively and qualitatively validated that the students who participated in the training program perceived the process-oriented feedback they received as favorable for their learning.

Performance Analysis of Emergency Room Episodes Through Process Mining.

The performance analysis of Emergency Room episodes is aimed at providing decision makers with knowledge that allows them to decrease waiting times, reduce patient congestion, and improve the quality of care provided. In this case study, Process Mining is used to determine which activities, sub-processes, interactions, and characteristics of episodes explain why some episodes have a longer duration. The employed method and the results obtained are described in detail to serve as a guide for future performance analysis in this domain. It was discovered that the main cause of the increment in the episode duration is the occurrence of a loop between the Examination and Treatment sub-processes. It was also found out that as the episode severity increases, the number of repetitions of the Examination-Treatment loop increases as well. Moreover, the episodes in which this loop is more common are those that lead to Hospitalization as discharge destination. These findings might help to reduce the occurrence of this loop, in turn lowering the episode duration and, consequently, providing faster attention to more patients.