Evaluating health information systems-related errors using the human, organization, process, technology-fit (HOPT-fit) framework.

Complex socio-technical health information systems (HIS) issues can create new error risks. Therefore, we evaluated the management of HIS-related errors using the proposed human, organization, process, and technology-fit framework to identify the lessons learned. Qualitative case study methodology through observation, interview, and document analysis was conducted at a 1000-bed Japanese specialist teaching hospital. Effective management of HIS-related errors was attributable to many socio-technical factors including continuous improvement, safety culture, strong management and leadership, effective communication, preventive and corrective mechanisms, an incident reporting system, and closed feedback loops. Enablers of medication errors include system sophistication and process factors like workarounds, variance, clinical workload, slips and mistakes, and miscommunication. The case management effectiveness in handling the HIS-related errors can guide other clinical settings. The potential of HIS to minimize errors can be achieved through continual, systematic, and structured evaluation. The case study validated the applicability of the proposed evaluation framework that can be applied flexibly according to study contexts to inform HIS stakeholders in decision-making. The comprehensive and specific measures of the proposed framework and approach can be a useful guide for evaluating complex HIS-related errors. Leaner and fitter socio-technical components of HIS can yield safer system use.

Towards an evaluation framework for medical device-integrated electronic medical record.

INTRODUCTION: Medical device (MD)-integrated (I) electronic medical record (EMR) (MDI-EMR) poses cyber threats that undermine patient safety, and thus, they require effective control mechanisms. We reviewed the related literature, including existing EMR and MD risk assessment approaches, to identify MDI-EMR comprehensive evaluation dimensions and measures. AREAS COVERED: We searched multiple databases, including PubMed, Web of Knowledge, Scopus, ACM, Embase, IEEE and Ingenta. We explored various evaluation aspects of MD and EMR to gain a better understanding of their complex integration. We reviewed numerous risk management and assessment frameworks related to MD and EMR security aspects and mitigation controls and then identified their common evaluation aspects. Our review indicated that previous evaluation frameworks assessed MD and EMR independently. To address this gap, we proposed an evaluation framework based on the sociotechnical dimensions of health information systems and risk assessment approaches for MDs to evaluate MDI-EMR integratively. EXPERT OPINION: The emergence of MDI-EMR cyber threats requires appropriate evaluation tools to ensure the safe development and application of MDI-EMR. Consequently, our proposed framework will continue to evolve through subsequent validations and refinements. This process aims to establish its applicability in informing stakeholders of the safety level and assessing its effectiveness in mitigating risks for future improvements.

Unintended consequences of pharmacy information systems: A case study.

BACKGROUND: Pharmacy information systems (PhIS) can cause medication errors that pharmacists may overlook due to their increased workload and lack of understanding of maintaining information quality. This study seeks to identify factors influencing unintended consequences of PhIS and how they affect the information quality, which can pose a risk to patient safety. MATERIALS AND METHODS: This qualitative, explanatory case study evaluated PhIS in ambulatory pharmacies in a hospital and a clinic. Data were collected through observations, interviews, and document analysis. We applied the socio-technical interactive analysis (ISTA) framework to investigate the socio-technical interactions of pharmacy information systems that lead to unintended consequences. We then adopted the human-organization-process-technology-fit (HOPT-fit) framework to identify their contributing and dominant factors, misfits, and mitigation measures. RESULTS: We identified 28 unintended consequences of PhIS, their key contributing factors, and their interrelations with the systems. The primary causes of unintended consequences include system rigidity and complexity, unclear knowledge, understanding, skills, and purpose of using the system, use of hybrid paper and electronic documentation, unclear and confusing transitions, additions and duplication of tasks and roles in the workflow, and time pressure, causing cognitive overload and workarounds. Recommended mitigating mechanisms include human factor principles in system design, data quality improvement for PhIS in terms of effective use of workspace, training, PhIS master data management, and communication by standardizing workarounds. CONCLUSION: Threats to information quality emerge in PhIS because of its poor design, a failure to coordinate its functions and clinical tasks, and pharmacists' lack of understanding of the system use. Therefore, safe system design, fostering awareness in maintaining the information quality of PhIS and cultivating its safe use in organizations is essential to ensure patient safety. The proposed evaluation approach facilitates the evaluator to identify complex socio-technical interactions and unintended consequences factors, impact, and mitigation mechanisms.

Evaluating Unintended Consequences in Health Information Systems.

The use of health information systems (HIS) and complex sociotechnical interactions can generate dangerous unintended consequences (UC). The evaluation of such interactions can provide an understanding of the root causes of UC. This paper reviews the interactions that lead to UC and its contributing factors.

A Lean Approach to Evaluating Prescribing Errors in Medication Reconciliation.

We evaluated medication reconciliation processes of a qualitative case study at a 1000-bed public hospital. Lean tools were applied to identify factors contributing to prescribing errors and propose process improvement. Errors were attributed to the prescriber's skills, high workload, staff shortage, poor user attitude and rigid system function. Continuous evaluation of medication reconciliation efficiency is imperative to identify and mitigate errors and increase patient safety.

Evaluating the alert appropriateness of clinical decision support systems in supporting clinical workflow.

The overwhelming number of medication alerts generated by clinical decision support systems (CDSS) has led to inappropriate alert overrides, which may lead to unintended patient harm. This review highlights the factors affecting the alert appropriateness of CDSS and barriers to the fit of CDSS alert with clinical workflow. A literature review was conducted to identify features and functions pertinent to CDSS alert appropriateness using the five rights of CDSS. Moreover, a process improvement method, namely, Lean, was used as a tool to optimise clinical workflows, and the appropriate design for CDSS alert using a human automation interaction (HAI) model was recommended. Evaluating the appropriateness of CDSS alert and its impact on workflow provided insights into how alerts can be designed and triggered effectively to support clinical workflow. The application of Lean methods and tools to analyse alert efficiencies in supporting workflow in this study provides an in-depth understanding of alert-workflow fit problems and their root cause, which is required for improving CDSS design. The application of the HAI model is recommended in the design of CDSS alerts to support various levels and stages of alert automations, namely, information acquisition and analysis, decision action and action implementation.

A Socio-Technical and Lean Approach Towards a Framework for Health Information Systems-Induced Error.

The evaluation of Health Information Systems (HIS)-induced medication errors is crucial in efforts to understand its cause, impact and mitigation measures when trying to minimize errors and increase patient safety. A review of evaluation studies on HIS-induced medication errors was carried out, which indicated the need to further structure complex socio technical aspects of the subject. In order to satisfy this requirement, a new framework was introduced for the evaluation of HIS-induced error management in clinical settings. The proposed HO(P)T-fit framework (Human, Organization, Process and Technology-fit) was developed after critically appraising existing findings in HIS related evaluation studies. It also builds on previous models related to HIS evaluation, in particular, the HOT-fit (Human, Organization, Process and Technology-fit) framework, error model, business process management, Lean method, and medication workflow. HOPT-fit incorporates the concept of fit between the four factors. The framework has the potential to be used as a tool to conduct a structured, systematic, and comprehensive HIS evaluation.

A case study evaluation of a Critical Care Information System adoption using the socio-technical and fit approach.

BACKGROUND: Clinical information systems have long been used in intensive care units but reports on their adoption and benefits are limited. This study evaluated a Critical Care Information System implementation. METHODS: A case study summative evaluation was conducted, employing observation, interview, and document analysis in operating theatres and 16-bed adult intensive care units in a 400-bed Malaysian tertiary referral centre from the perspectives of users (nurses and physicians), management, and information technology staff. System implementation, factors influencing adoption, fit between these factors, and the impact of the Critical Care Information System were evaluated after eight months of operation. RESULTS: Positive influences on system adoption were associated with technical factors, including system ease of use, usefulness, and information relevancy; human factors, particularly user attitude; and organisational factors, namely clinical process-technology alignment and champions. Organisational factors such as planning, project management, training, technology support, turnover rate, clinical workload, and communication were barriers to system implementation and use. Recommendations to improve the current system problems were discussed. Most nursing staff positively perceived the system's reduction of documentation and data access time, giving them more time with patients. System acceptance varied among doctors. System use also had positive impacts on timesaving, data quality, and clinical workflow. CONCLUSIONS: Critical Care Information Systems is crucial and has great potentials in enhancing and delivering critical care. However, the case study findings showed that the system faced complex challenges and was underutilised despite its potential. The role of socio-technical factors and their fit in realizing the potential of Critical Care Information Systems requires continuous, in-depth evaluation and stakeholder understanding and acknowledgement. The comprehensive and specific evaluation measures of the Human-Organisation-Technology Fit framework can flexibly evaluate Critical Care Information Systems.

Understanding waste for lean health information systems: a preliminary review.

Despite the rapid application of the Lean method in healthcare, its study in IT environments, particularly in Health Information Systems (HIS), is still limited primarily by a lack of waste identification. This paper aims to review the literature to provide an insight into the nature of waste in HIS from the perspective of Lean management. Eight waste frameworks within the context of healthcare and information technology were reviewed. Based on the review, it was found that all the seven waste categories from the manufacturing sector also exist in both the healthcare and IT domains. However, the nature of the waste varied depending on the processes of the domains. A number of additional waste categories were also identified. The findings reveal that the traditional waste model can be adapted to identify waste in both the healthcare and IT sectors.

ICDTag: a prototype for a web-based system for organizing physician-written blog posts using a hybrid taxonomy-folksonomy approach.

BACKGROUND: Medical blogs have emerged as new media, extending to a wider range of medical audiences, including health professionals and patients to share health-related information. However, extraction of quality health-related information from medical blogs is challenging primarily because these blogs lack systematic methods to organize their posts. Medical blogs can be categorized according to their author into (1) physician-written blogs, (2) nurse-written blogs, and (3) patient-written blogs. This study focuses on how to organize physician-written blog posts that discuss disease-related issues and how to extract quality information from these posts. OBJECTIVE: The goal of this study was to create and implement a prototype for a Web-based system, called ICDTag, based on a hybrid taxonomy-folksonomy approach that follows a combination of a taxonomy classification schemes and user-generated tags to organize physician-written blog posts and extract information from these posts. METHODS: First, the design specifications for the Web-based system were identified. This system included two modules: (1) a blogging module that was implemented as one or more blogs, and (2) an aggregator module that aggregated posts from different blogs into an aggregator website. We then developed a prototype for this system in which the blogging module included two blogs, the cardiology blog and the gastroenterology blog. To analyze the usage patterns of the prototype, we conducted an experiment with data provided by cardiologists and gastroenterologists. Next, we conducted two evaluation types: (1) an evaluation of the ICDTag blog, in which the browsing functionalities of the blogging module were evaluated from the end-user's perspective using an online questionnaire, and (2) an evaluation of information quality, in which the quality of the content on the aggregator website was assessed from the perspective of medical experts using an emailed questionnaire. RESULTS: Participants of this experiment included 23 cardiologists and 24 gastroenterologists. Positive evaluations on the main functions and the organization of information on the ICDTag blogs were given by 18 of the participants via an online questionnaire. These results supported our hypothesis that the use of a taxonomy-folksonomy structure has significant potential to improve the organization of information in physician-written blogs. The quality of the content on the aggregator website was assessed by 3 cardiology experts and 3 gastroenterology experts via an email questionnaire. The results of this questionnaire demonstrated that the experts considered the aggregated tags and categories semantically related to the posts' content. CONCLUSIONS: This study demonstrated that applying the hybrid taxonomy-folksonomy approach to physician-written blogs that discuss disease-related issues has valuable potential to make these blogs a more organized and systematic medium and supports the extraction of quality information from their posts. Thus, it is worthwhile to develop more mature systems that make use of the hybrid approach to organize posts in physician-written blogs.

Towards an evaluation framework for information quality management (IQM) practices for health information systems--evaluation criteria for effective IQM practices.

RATIONALE, AIMS AND OBJECTIVES: Poor information quality (IQ) must be understood as a business problem rather than systems problem. In health care organization, what is required is an effective quality management that continuously manages and reviews the factors influencing IQ in health information systems (HIS) so as to achieve the desired outcomes. Hence, in order to understand the issues of information quality management (IQM) practices in health care organizations, a more holistic evaluation study should be undertaken to investigate the IQM practices in health care organizations. It is the aim of this paper to identify the significant evaluation criteria that influence the production of good IQ in HIS. METHODS: Six selected frameworks and best practices both from health informatics and information systems literature have been reviewed to identify the evaluation criteria from the perspective of human, organizational and technological factors. RESULTS: From the review, it was found that human and organization factors are of greater significance in influencing HIS IQ. Our review depicts that there is still shortage in finding a comprehensive IQM evaluation framework. Thus, the criteria from the frameworks reviewed can be used in combination for more comprehensive evaluation criteria. Integrated IQM evaluation criteria for HIS are then proposed in this study. CONCLUSIONS: Poor IQ is the result of complex interdependency within sociotechnical factors in health care organization and lack of formal and structured IQM practices. Thus, a feedback mechanism such as evaluation is needed to understand the issues in depth in the future.

Evaluation of the clinical process in a critical care information system using the Lean method: a case study.

BACKGROUND: There are numerous applications for Health Information Systems (HIS) that support specific tasks in the clinical workflow. The Lean method has been used increasingly to optimize clinical workflows, by removing waste and shortening the delivery cycle time. There are a limited number of studies on Lean applications related to HIS. Therefore, we applied the Lean method to evaluate the clinical processes related to HIS, in order to evaluate its efficiency in removing waste and optimizing the process flow. This paper presents the evaluation findings of these clinical processes, with regards to a critical care information system (CCIS), known as IntelliVue Clinical Information Portfolio (ICIP), and recommends solutions to the problems that were identified during the study. METHODS: We conducted a case study under actual clinical settings, to investigate how the Lean method can be used to improve the clinical process. We used observations, interviews, and document analysis, to achieve our stated goal. We also applied two tools from the Lean methodology, namely the Value Stream Mapping and the A3 problem-solving tools. We used eVSM software to plot the Value Stream Map and A3 reports. RESULTS: We identified a number of problems related to inefficiency and waste in the clinical process, and proposed an improved process model. CONCLUSIONS: The case study findings show that the Value Stream Mapping and the A3 reports can be used as tools to identify waste and integrate the process steps more efficiently. We also proposed a standardized and improved clinical process model and suggested an integrated information system that combines database and software applications to reduce waste and data redundancy.

Investigating evaluation frameworks for health information systems.

BACKGROUND AND PURPOSE: Evaluation of health information systems (HIS) enables the assessment of the extent to which HIS are fulfilling their objectives in supporting the services of healthcare delivery. This paper presents an overview of evaluation in health informatics and information systems. METHODS: Literature review on discourses, dimensions and methods of HIS and IS evaluation. A critical appraisal of selected HIS and IS evaluation frameworks is undertaken in order to identify HIS evaluation dimensions and measures. The frameworks are compared based on their inclusion of human, organizational and technological factors. RESULTS: We found that an increasing number of evaluation studies deal with two distinct trends of HIS: one considers human and organizational issues and the other is concerned with the employment of a subjectivist approach. Our review indicates that current evaluation methods complement each other in that they evaluate different aspects of HIS and they can be improved upon. CONCLUSIONS: Evaluation is complex; it is easy to measure many things but not necessarily the right ones. Nevertheless, it is possible to consider, a HIS evaluation framework with more comprehensive and specific measures that would incorporate technological, human and organizational issues to facilitate HIS evaluation.

An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit).

BACKGROUND AND PURPOSE: The realization of Health Information Systems (HIS) requires rigorous evaluation that addresses technology, human and organization issues. Our review indicates that current evaluation methods evaluate different aspects of HIS and they can be improved upon. A new evaluation framework, human, organization and technology-fit (HOT-fit) was developed after having conducted a critical appraisal of the findings of existing HIS evaluation studies. HOT-fit builds on previous models of IS evaluation--in particular, the IS Success Model and the IT-Organization Fit Model. This paper introduces the new framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational fit. METHODS: Literature review on HIS and IS evaluation studies and pilot testing of developed framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary care organization in the UK. The case study was conducted through observation, interview and document analysis. RESULTS: The main findings show that having the right user attitude and skills base together with good leadership, IT-friendly environment and good communication can have positive influence on the system adoption. CONCLUSIONS: Comprehensive, specific evaluation factors, dimensions and measures in the new framework (HOT-fit) are applicable in HIS evaluation. The use of such a framework is argued to be useful not only for comprehensive evaluation of the particular FIS system under investigation, but potentially also for any Health Information System in general.

Health information systems adoption: findings from a systematic review.

Earlier evaluation studies on Health Information Systems (HIS) adoption have highlighted a large number of adoption problems that were attributed to the lack of fit between technology, human and organisation factors. Lessons can be learned from these evaluation studies by identifying the most important factors of HIS adoption. In order to study the adoption issue, a qualitative systematic review has been performed using a recently introduced framework, known as HOT-fit (Human, Organisation and Technology fit). The paper identifies and highlights the following critical adoption factors: technology (ease of use, system usefulness, system flexibility, time efficiency, information accessibility and relevancy); human (user training, user perception, user roles, user skills, clarity of system purpose, user involvement); organisation (leadership and support, clinical process, user involvement, internal communication, inter organisational system, as well as the fit between them. The findings can be used to guide future system development and inform relevant decision making.

Health information systems evaluation: a focus on clinical decision supports system.

In a review of selected literature on Health Information Systems (HIS) evaluation, a specific focus on Clinical Decision Support Systems (CDSS) is taken because of their relative popularity. This paper discusses the issues and problems of CDSS evaluation such as methods, adoption and barriers. The limited use and evaluation of CDSS are still debated. Clinical evaluations of CDSS performed in the actual clinical settings may provide better understanding of their adoption, particularly in the diagnostic function. New HIS evaluation frameworks that incorporate technological, human and organisational issues may be useful to complement existing ones.