The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

How the reliability and validity of the patient classification system can be ensured in daily nursing work? A follow-up study.

AIMS AND OBJECTIVES: This study aimed to determine the reliability and validity of the RAFAELA patient classification system (PCS) for qualified and efficient nurses. BACKGROUND: The number of patients per nurse or diagnosis-based determination of nursing workload are imprecise measures that do not consider the variation in patients' care needs. Ensuring the reliability and validity of the RAFAELA is important for the efficient allocation of nursing resources. METHODS: In this study, we investigated how the maintenance (parallel classification measurement and professional assessment of optimal nursing care intensity level measurement) of the RAFAELA was done with 9 years of follow-up data. The results were analysed using quantitative methods supplemented with qualitative audit descriptions. The STROBE checklist was used. RESULTS: The RAFAELA was used continuously in 44 units (40%). The length of use of the RAFAELA influenced the success of parallel classification measurements. Six per cent of units passed parallel classification measurement over 75% after 1-3 years' use, 42% after 4-6 years and 83% after 7-9 years. Among the units that used the RAFAELA PCS continuously, only four (9%) passed the professional assessment of optimal nursing care intensity level measurement. CONCLUSIONS: This study shows that ensuring the reliability and validity of the use of the RAFAELA is laborious, requires several years of use and continuous investments in nurses' skills and motivation. RELEVANCE TO CLINICAL PRACTICE: Qualified use of PCS is challenging, and organisations should invest to maintenance, training, support and user motivation. Each patient should be classified comprehensively, and nursing resources should be calculated correctly. In addition, utilisation of the nursing intensity level should be maximised. CLINICAL TRIAL REGISTRATION NUMBER: Kuopio University Hospital organisation permit number 73/2014. PATIENT OR PUBLIC CONTRIBUTION: Information regarding individual patients or nurses was not available to the researchers. All materials are in the form of summary tables.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications.

OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

The Safe and Effective Use of Shared Data Underpinned by Stakeholder Engagement and Evaluation Practice.

OBJECTIVES: The paper draws attention to: i) key considerations involving the confidentiality, privacy, and security of shared data; and ii) the requirements needed to build collaborative arrangements encompassing all stakeholders with the goal of ensuring safe, secure, and quality use of shared data. METHOD: A narrative review of existing research and policy approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Care and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: The technological ability to merge, link, re-use, and exchange data has outpaced the establishment of policies, procedures, and processes to monitor the ethics and legality of shared use of data. Questions remain about how to guarantee the security of shared data, and how to establish and maintain public trust across large-scale shared data enterprises. This paper identifies the importance of data governance frameworks (incorporating engagement with all stakeholders) to underpin the management of the ethics and legality of shared data use. The paper also provides some key considerations for the establishment of national approaches and measures to monitor compliance with best practice. CONCLUSION: Data sharing endeavours can help to underpin new collaborative models of health care which provide shared information, engagement, and accountability amongst all stakeholders. We believe that commitment to rigorous evaluation and stakeholder engagement will be critical to delivering health data benefits and the establishment of collaborative models of health care into the future.

How to Teach Health IT Evaluation: Recommendations for Health IT Evaluation Courses.

Systematic health IT evaluation studies are needed to ensure system quality and safety and to provide the basis for evidence-based health informatics. Well-trained health informatics specialists are required to guarantee that health IT evaluation studies are conducted in accordance with robust standards. Also, policy makers and managers need to appreciate how good evidence is obtained by scientific process and used as an essential justification for policy decisions. In a consensus-based approach with over 80 experts in health IT evaluation, recommendations for the structure, scope and content of health IT evaluation courses on the master or postgraduate level have been developed, supported by a structured analysis of available courses and of available literature. The recommendations comprise 15 mandatory topics and 15 optional topics for a health IT evaluation course.

Green Turning Brown - Domain Engineering for Social and Health Services in Finland.

Being able to design information systems to an untouched domain, without the burden of existing information systems, especially legacy systems, is often seen as a dream of most information system professionals. Uncharted domains are anyway scarce, and often such greenfield projects turn into brownfield projects, also to projects where existing structures severely constrain the development of new systems. In this article we discuss the concepts of greenfield and brownfield domain engineering and software development, and reflect their possible messages to the re-engineering of the Finnish health- and social care ecosystem currently under way. In our fieldwork we could identify a lot of need and wish for greenfield domain engineering in the Finnish health and social services delivery. As well we found a lot of brownfield elements inhibiting change. Our proposal for the future is a ecosystem approach, where new and established elements could live together in a self-governed balance.

Hospital mortality and optimality of nursing workload: A study on the predictive validity of the RAFAELA Nursing Intensity and Staffing system.

BACKGROUND: Patient classification systems have been developed to manage workloads by estimating the need for nursing resources through the identification and quantification of individual patients' care needs. There is in use a diverse variety of patient classification systems. Most of them lack validity and reliability testing and evidence of the relationship to nursing outcomes. OBJECTIVE: Predictive validity of the RAFAELA system was tested by examining whether hospital mortality can be predicted by the optimality of nursing workload. METHODS: In this cross-sectional retrospective observational study, monthly mortality statistics and reports of daily registrations from the RAFAELA system were gathered from 34 inpatient units of two acute care hospitals in 2012 and 2013 (n=732). The association of hospital mortality with the chosen predictors (hospital, average daily patient to nurse ratio, average daily nursing workload and average daily workload optimality) was examined by negative binomial regression analyses. RESULTS: Compared to the incidence rate of death in the months of overstaffing when average daily nursing workload was below the optimal level, the incidence rate was nearly fivefold when average daily nursing workload was at the optimal level (IRR 4.79, 95% CI 1.57-14.67, p=0.006) and 13-fold in the months of understaffing when average daily nursing workload was above the optimal level (IRR 12.97, 95% CI 2.86-58.88, p=0.001). CONCLUSIONS: Hospital mortality can be predicted by the RAFAELA system. This study rendered additional confirmation for the predictive validity of this patient classification system. In future, larger studies with a wider variety of nurse sensitive outcomes and multiple risk adjustments are needed. Future research should also focus on other important criteria for an adequate nursing workforce management tool such as simplicity, efficiency and acceptability.

Quality of Health IT Evaluations.

Health IT evaluation studies have often been found to be of limited quality. To address this problem, several guidelines and frameworks have been developed as tools to support improvement of the quality of evaluation studies. In this contribution, we review available guidelines and then present the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) in more detail. GEP-HI is a comprehensive guideline which supports especially planning and execution of a health IT evaluation study. The GEP-HI guideline helps to overcome the quality problems related to weak study planning and methodological study design. We also discuss application of GEP-HI on an evaluation project and discuss the need to publish systematically following the recognised publication guidelines. Finally we discuss the future trend on multi-method evaluation approaches.

Privacy-related context information for ubiquitous health.

BACKGROUND: Ubiquitous health has been defined as a dynamic network of interconnected systems. A system is composed of one or more information systems, their stakeholders, and the environment. These systems offer health services to individuals and thus implement ubiquitous computing. Privacy is the key challenge for ubiquitous health because of autonomous processing, rich contextual metadata, lack of predefined trust among participants, and the business objectives. Additionally, regulations and policies of stakeholders may be unknown to the individual. Context-sensitive privacy policies are needed to regulate information processing. OBJECTIVE: Our goal was to analyze privacy-related context information and to define the corresponding components and their properties that support privacy management in ubiquitous health. These properties should describe the privacy issues of information processing. With components and their properties, individuals can define context-aware privacy policies and set their privacy preferences that can change in different information-processing situations. METHODS: Scenarios and user stories are used to analyze typical activities in ubiquitous health to identify main actors, goals, tasks, and stakeholders. Context arises from an activity and, therefore, we can determine different situations, services, and systems to identify properties for privacy-related context information in information-processing situations. RESULTS: Privacy-related context information components are situation, environment, individual, information technology system, service, and stakeholder. Combining our analyses and previously identified characteristics of ubiquitous health, more detailed properties for the components are defined. Properties define explicitly what context information for different components is needed to create context-aware privacy policies that can control, limit, and constrain information processing. With properties, we can define, for example, how data can be processed or how components are regulated or in what kind of environment data can be processed. CONCLUSIONS: This study added to the vision of ubiquitous health by analyzing information processing from the viewpoint of an individual's privacy. We learned that health and wellness-related activities may happen in several environments and situations with multiple stakeholders, services, and systems. We have provided new knowledge regarding privacy-related context information and corresponding components by analyzing typical activities in ubiquitous health. With the identified components and their properties, individuals can define their personal preferences on information processing based on situational information, and privacy services can capture privacy-related context of the information-processing situation.

Trust-based information system architecture for personal wellness.

Modern eHealth, ubiquitous health and personal wellness systems take place in an unsecure and ubiquitous information space where no predefined trust occurs. This paper presents novel information model and an architecture for trust based privacy management of personal health and wellness information in ubiquitous environment. The architecture enables a person to calculate a dynamic and context-aware trust value for each service provider, and using it to design personal privacy policies for trustworthy use of health and wellness services. For trust calculation a novel set of measurable context-aware and health information-sensitive attributes is developed. The architecture enables a person to manage his or her privacy in ubiquitous environment by formulating context-aware and service provider specific policies. Focus groups and information modelling was used for developing a wellness information model. System analysis method based on sequential steps that enable to combine results of analysis of privacy and trust concerns and the selection of trust and privacy services was used for development of the information system architecture. Its services (e.g. trust calculation, decision support, policy management and policy binding services) and developed attributes enable a person to define situation-aware policies that regulate the way his or her wellness and health information is processed.

Feasibility analysis of the privacy attributes of the personal wellness information model.

A feasibility analysis has been performed to study the applicability of privacy attributes with a developed wellness information model. Information privacy concerns specifically access to individually identifiable personal information and one's ability to control information about oneself. We carried out a user scenario walk-through of the privacy attributes related to the wellness components. The walk-through showed a need to relate self-regulating privacy policies to the pervasive context so that during various trust-building processes, a person is aware and can control the use, disclosure and even secondary use of his personal, private wellness information.

Trust information-based privacy architecture for ubiquitous health.

BACKGROUND: Ubiquitous health is defined as a dynamic network of interconnected systems that offers health services independent of time and location to a data subject (DS). The network takes place in open and unsecure information space. It is created and managed by the DS who sets rules that regulate the way personal health information is collected and used. Compared to health care, it is impossible in ubiquitous health to assume the existence of a priori trust between the DS and service providers and to produce privacy using static security services. In ubiquitous health features, business goals and regulations systems followed often remain unknown. Furthermore, health care-specific regulations do not rule the ways health data is processed and shared. To be successful, ubiquitous health requires novel privacy architecture. OBJECTIVE: The goal of this study was to develop a privacy management architecture that helps the DS to create and dynamically manage the network and to maintain information privacy. The architecture should enable the DS to dynamically define service and system-specific rules that regulate the way subject data is processed. The architecture should provide to the DS reliable trust information about systems and assist in the formulation of privacy policies. Furthermore, the architecture should give feedback upon how systems follow the policies of DS and offer protection against privacy and trust threats existing in ubiquitous environments. METHODS: A sequential method that combines methodologies used in system theory, systems engineering, requirement analysis, and system design was used in the study. In the first phase, principles, trust and privacy models, and viewpoints were selected. Thereafter, functional requirements and services were developed on the basis of a careful analysis of existing research published in journals and conference proceedings. Based on principles, models, and requirements, architectural components and their interconnections were developed using system analysis. RESULTS: The architecture mimics the way humans use trust information in decision making, and enables the DS to design system-specific privacy policies using computational trust information that is based on systems' measured features. The trust attributes that were developed describe the level systems for support awareness and transparency, and how they follow general and domain-specific regulations and laws. The monitoring component of the architecture offers dynamic feedback concerning how the system enforces the polices of DS. CONCLUSIONS: The privacy management architecture developed in this study enables the DS to dynamically manage information privacy in ubiquitous health and to define individual policies for all systems considering their trust value and corresponding attributes. The DS can also set policies for secondary use and reuse of health information. The architecture offers protection against privacy threats existing in ubiquitous environments. Although the architecture is targeted to ubiquitous health, it can easily be modified to other ubiquitous applications.

A conceptual framework and principles for trusted pervasive health.

BACKGROUND: Ubiquitous computing technology, sensor networks, wireless communication and the latest developments of the Internet have enabled the rise of a new concept-pervasive health-which takes place in an open, unsecure, and highly dynamic environment (ie, in the information space). To be successful, pervasive health requires implementable principles for privacy and trustworthiness. OBJECTIVE: This research has two interconnected objectives. The first is to define pervasive health as a system and to understand its trust and privacy challenges. The second goal is to build a conceptual model for pervasive health and use it to develop principles and policies which can make pervasive health trustworthy. METHODS: In this study, a five-step system analysis method is used. Pervasive health is defined using a metaphor of digital bubbles. A conceptual framework model focused on trustworthiness and privacy is then developed for pervasive health. On that model, principles and rules for trusted information management in pervasive health are defined. RESULTS: In the first phase of this study, a new definition of pervasive health was created. Using this model, differences between pervasive health and health care are stated. Reviewed publications demonstrate that the widely used principles of predefined and static trust cannot guarantee trustworthiness and privacy in pervasive health. Instead, such an environment requires personal dynamic and context-aware policies, awareness, and transparency. A conceptual framework model focused on information processing in pervasive health is developed. Using features of pervasive health and relations from the framework model, new principles for trusted pervasive health have been developed. The principles propose that personal health data should be under control of the data subject. The person shall have the right to verify the level of trust of any system which collects or processes his or her health information. Principles require that any stakeholder or system collecting or processing health data must support transparency and shall publish its trust and privacy attributes and even its domain specific policies. CONCLUSIONS: The developed principles enable trustworthiness and guarantee privacy in pervasive health. The implementation of principles requires new infrastructural services such as trust verification and policy conflict resolution. After implementation, the accuracy and usability of principles should be analyzed.

Applying GEP-HI for the planning of an evaluation study: a case study walkthrough (workshop).

A Good evaluation practice in Health Informatics (GEP-HI) Evaluation Practice guideline has been developed through a consensus making process. The guideline lists a set of 60 issues that are relevant for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover the phases of an evaluation study: Study exploration, first study design, operationalization of methods, detailed study design, execution and finalization of an evaluation study. In this seminar we walk through a case study to present how to plan a health information system evaluation study applying the good evaluation practice guideline.

Evaluation of the national nursing model and four nursing documentation systems in Finland--lessons learned and directions for the future.

OBJECTIVES: To evaluate the feasibility of the national nursing model and usability of four widely used nursing documentation systems and to study their usefulness in multi-professional collaboration and information exchange. METHODS: Qualitative usability study methods were used, including the use of scenario walkthroughs, contextual inquiries, thematic interviews and inspection-based expert reviews in the users' clinical contexts. RESULTS: The nursing process model was shown to be feasible in nursing practice but the national nursing classification was considered too detailed, multi-layered and difficult to use and understand. The four evaluated nursing documentation systems had many usability problems which resulted in them being difficult to use and produced extra documentation workload. Generally, electronic nursing documentation improves patients' and health professionals' legal protection and makes nursing care more transparent; however, the documentation systems did not provide good support for multi-professional care and information exchange. CONCLUSIONS: Nursing models should comply better with nursing practices and support nurses in patient care and interventions. An essential improvement in practice would be the use of specific templates that are easy to apply in specific situations with homogeneous patient groups. Collaborative care aspects and better utilization of information require that the nursing model is designed to support not just documentation but also information utilization.

The national nursing model from the nursing practice perspective - results from a finnish evaluation study.

The objective of this study was to evaluate the feasibility of the national nursing model in Finland. The feasibility evaluation was carried out with nurses using interviews and patient case scenarios in primary, specialized and private healthcare. The nursing process model showed to be feasible in nursing practice but the current national nursing classification (FinCC) was considered to be too detailed, multi-layered and difficult to understand and use. Overall, electronic nursing documentation improves the legal protection of patients and health professionals and makes nursing care transparent, but the nursing documentation systems do not support multi-professional care or information exchange. This study resulted in that the nursing model should conform better to nursing practices and support better nurses in their care interventions. An essential improvement for nursing practice would be specific templates that are easy to apply in specific situations with homogenous patient groups.

Guideline for good evaluation practice in health informatics (GEP-HI).

OBJECTIVE: Development of a good practice guideline to plan and perform scientifically robust evaluation studies in health informatics. METHODS: Issues to be addressed in evaluation studies were identified and guidance drafted based on the evaluation literature and on experiences by key players. Successive drafts of the guideline were discussed in several rounds by an increasing number of experts during conferences and by e-mail. At a fairly early point the guideline was put up for comments on the web. RESULTS: Sixty issues were identified that are of potential relevance for planning, implementation and execution of an evaluation study in the health informatics domain. These issues cover all phases of an evaluation study: Preliminary outline, study design, operationalization of methods, project planning, execution and completion of the evaluation study. Issues of risk management and project control as well as reporting and publication of the evaluation results are also addressed. CONCLUSION: A comprehensive list of issues is presented as a guideline for good evaluation practice in health informatics (GEP-HI). The strengths and weaknesses of the guideline are discussed. Application of this guideline will support better handling of an evaluation study, potentially leading to a higher quality of evaluation studies. This guideline is an important step towards building stronger evidence and thus to progress towards evidence-based health informatics.

Next steps in evaluation and evidence - from generic to context-related.

INTRODUCTION: E-health systems are increasingly important and widespread, but their selection and implementation are still frequently based on belief, rather than scientific evidence, and adverse effects are not systematically addressed. Progress is being made in promoting generic evaluation methodologies as a source of scientific evidence, but effort is now needed to consider methods for special situations. METHOD: Review of five evaluation contexts - national e-health plans, telemedicine, Health Informatics 3.0, usability and economics. CONCLUSION: Identification of requirements for approaches to be developed in these five settings.

Framework model and principles for trusted information sharing in pervasive health.

Trustfulness (i.e. health and wellness information is processed ethically, and privacy is guaranteed) is one of the cornerstones for future Personal Health Systems, ubiquitous healthcare and pervasive health. Trust in today's healthcare is organizational, static and predefined. Pervasive health takes place in an open and untrusted information space where person's lifelong health and wellness information together with contextual data are dynamically collected and used by many stakeholders. This generates new threats that do not exist in today's eHealth systems. Our analysis shows that the way security and trust are implemented in today's healthcare cannot guarantee information autonomy and trustfulness in pervasive health. Based on a framework model of pervasive health and risks analysis of ubiquitous information space, we have formulated principles which enable trusted information sharing in pervasive health. Principles imply that the data subject should have the right to dynamically verify trust and to control the use of her health information, as well as the right to set situation based context-aware personal policies. Data collectors and processors have responsibilities including transparency of information processing, and openness of interests, policies and environmental features. Our principles create a base for successful management of privacy and information autonomy in pervasive health. They also imply that it is necessary to create new data models for personal health information and new architectures which support situation depending trust and privacy management.