Improving Evaluation to Address the Unintended Consequences of Health Information Technology:. a Position Paper from the Working Group on Technology Assessment & Quality Development.

BACKGROUND AND OBJECTIVES: With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT. METHODS: Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation. CONCLUSION: Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a proactive and evidence-based approach to detecting and addressing the unintended effects of health IT.

Discussion of "Biomedical informatics: we are what we publish".

This article is part of a For-Discussion-Section of Methods of Information in Medicine about the paper "Biomedical Informatics: We Are What We Publish", written by Peter L. Elkin, Steven H. Brown, and Graham Wright. It is introduced by an editorial. This article contains the combined commentaries invited to independently comment on the Elkin et al. paper. In subsequent issues the discussion can continue through letters to the editor.

STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration.

BACKGROUND: Improving the quality of reporting of evaluation studies in health informatics is an important requirement towards the vision of evidence-based health informatics. The STARE-HI - Statement on Reporting of Evaluation Studies in health informatics, published in 2009, provides guidelines on the elements to be contained in an evaluation study report. OBJECTIVES: To elaborate on and provide a rationale for the principles of STARE-HI and to guide authors and readers of evaluation studies in health informatics by providing explanatory examples of reporting. METHODS: A group of methodologists, researchers and editors prepared the present elaboration of the STARE-HI statement and selected examples from the literature. RESULTS: The 35 STARE-HI items to be addressed in evaluation papers describing health informatics interventions are discussed one by one and each is extended with examples and elaborations. CONCLUSION: The STARE-HI statement and this elaboration document should be helpful resources to improve reporting of both quantitative and qualitative evaluation studies. Evaluation manuscripts adhering to the principles will enable readers of such papers to better place the studies in a proper context and judge their validity and generalizability, and thus in turn optimize the exploitation of the evidence contained therein. LIMITATIONS: This paper is based on experiences of a group of editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the details of these principles has to evolve as a function of their use in practice.

Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL.

OBJECTIVES: To present the importance of Evidence-based Health Informatics (EBHI) and the ethical imperative of this approach; to highlight the work of the IMIA Working Group on Technology Assessment and Quality Improvement and the EFMI Working Group on Assessment of Health Information Systems; and to introduce the further important evaluation and evidence aspects being addressed. METHODS: Reviews of IMIA, EFMA and other initiatives, together with literature reviews on evaluation methods and on published systematic reviews. RESULTS: Presentation of the rationale for the health informatics domain to adopt a scientific approach by assessing impact, avoiding harm, and empirically demonstrating benefit and best use; reporting of the origins and rationale of the IMIA- and EQUATOR-endorsed Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) and of the IMIA WG's Guideline for Good Evaluation Practice in Health Informatics (GEP-HI); presentation of other initiatives for objective evaluation; and outlining of further work in hand on usability and indicators; together with the case for development of relevant evaluation methods in newer applications such as telemedicine. The focus is on scientific evaluation as a reliable source of evidence, and on structured presentation of results to enable easy retrieval of evidence. CONCLUSIONS: EBHI is feasible, necessary for efficiency and safety, and ethically essential. Given the significant impact of health informatics on health systems, care delivery and personal health, it is vital that cultures change to insist on evidence-based policies and investment, and that emergent global moves for this are supported.

Remote monitoring and follow-up of cardiovascular implantable electronic devices in the Netherlands : An expert consensus report of the Netherlands Society of Cardiology.

Remote monitoring of cardiac implanted electronic devices (CIED: pacemaker, cardiac resynchronisation therapy device and implantable cardioverter defibrillator) has been developed for technical control and follow-up using transtelephonic data transmission. In addition, automatic or patient-triggered alerts are sent to the cardiologist or allied professional who can respond if necessary with various interventions. The advantage of remote monitoring appears obvious in impending CIED failures and suspected symptoms but is less likely in routine follow-up of CIED. For this follow-up the indications, quality of care, cost-effectiveneness and patient satisfaction have to be determined before remote CIED monitoring can be applied in daily practice. Nevertheless remote CIED monitoring is expanding rapidly in the Netherlands without professional agreements about methodology, responsibilities of all the parties involved and that of the device patient, and reimbursement. The purpose of this consensus document on remote CIED monitoring and follow-up is to lay the base for a nationwide, uniform implementation in the Netherlands. This report describes the technical communication, current indications, benefits and limitations of remote CIED monitoring and follow-up, the role of the patient and device manufacturer, and costs and reimbursement. The view of cardiology experts and of other disciplines in conjunction with literature was incorporated in a preliminary series of recommendations. In addition, an overview of the questions related to remote CIED monitoring that need to be answered is given. This consensus document can be used for future guidelines for the Dutch profession.

Systematic prioritization of the STARE-HI reporting items. An application to short conference papers on health informatics evaluation.

BACKGROUND: We previously devised and published a guideline for reporting health informatics evaluation studies named STARE-HI, which is formally endorsed by IMIA and EFMI. OBJECTIVE: To develop a prioritization framework of ranked reporting items to assist authors when reporting health informatics evaluation studies in space restricted conference papers and to apply this prioritization framework to measure the quality of recent health informatics conference papers on evaluation studies. METHOD: We deconstructed the STARE-HI guideline to identify reporting items. We invited a total of 111 authors of health informatics evaluation studies, reviewers and editors of health Informatics conference proceedings to score those reporting items on a scale ranging from "0 - not necessary in a conference paper" through to "10 - essential in a conference paper" by a web-based survey. From the responses we derived a mean priority score. All evaluation papers published in proceedings of MIE2006, Medinfo2007, MIE2008 and AMIA2008 were rated on these items by two reviewers. From these ratings a priority adjusted completeness score was computed for each paper. RESULTS: We identified 104 reporting items from the STARE-HI guideline. The response rate for the survey was 59% (66 out of 111). The most important reporting items (mean score ≥9) were "Interpret the data and give an answer to the study question - (in Discussion)", "Whether it is a laboratory, simulation or field study - (in Methods-study design)" and "Description of the outcome measure/evaluation criteria - (in Methods-study design)". Per reporting area the statistically more significant important reporting items were distinguished from less important ones. Four reporting items had a mean score ≤6. The mean priority adjusted completeness of evaluation papers of recent health informatics conferences was 48% (range 14-78%). CONCLUSION: We produced a ranked list of reporting items from STARE-HI according to their prioritized relevance for inclusion in space-limited conference papers. The priority adjusted completeness scores demonstrated room for improvement for the analyzed conference papers. We believe that this prioritization framework is an aid to improving the quality and utility of conference papers on health informatics evaluation studies.

Biomedical informatics--a confluence of disciplines?

BACKGROUND: Biomedical informatics is a broad discipline that borrows many methods and techniques from other disciplines. OBJECTIVE: To reflect a) on the character of biomedical informatics and to determine whether it is multi-disciplinary or inter-disciplinary; b) on the question whether biomedical informatics is more than the sum of its supporting disciplines and c) on the position of biomedical informatics with respect to related disciplines. METHOD: Inviting an international group of experts in biomedical informatics and related disciplines on the occasion of the 50th anniversary of Methods of Information in Medicine to present their viewpoints. RESULTS AND CONCLUSIONS: This paper contains the reflections of a number of the invited experts on the character of biomedical informatics. Most of the authors agree that biomedical informatics is an interdisciplinary field of study where researchers with different scientific backgrounds alone or in combination carry out research. Biomedical informatics is a very broad scientific field and still expanding, yet comprised of a constructive aspect (designing and building systems). One author expressed that the essence of biomedical informatics, as opposed to related disciplines, lies in the modelling of the biomedical content. Interdisciplinarity also has consequences for education. Maintaining rigid disciplinary structures does not allow for sufficient adaptability to capitalize on important trends nor to leverage the influences these trends may have on biomedical informatics. It is therefore important for students to become aware of research findings in related disciplines. In this respect, it was also noted that the fact that many scientific fields use different languages and that the research findings are stored in separate bibliographic databases makes it possible that potentially connected findings will never be linked, despite the fact that these findings were published. Bridges between the sciences are needed for the success of biomedical informatics.

Hospital and health information systems - Current perspectives. Contribution of the IMIA Health Information Systems Working Group.

OBJECTIVE: To celebrate over 30 years of health information systems' (HIS) evolution by bringing together pioneers in the field, members of the next generation of leaders, and government officials from several developing nations in Africa to discuss the past, present, and future of HISs. METHODS: Participants gathered in Le Franschhoek, South Africa for a 2 1/2 day working conference consisting of scientific presentations followed by several concurrent breakout sessions. A small writing group prepared draft statements representing their positions on various topics of discussion which were circulated and revised by the entire group. RESULTS: Many new tools, techniques and technologies were described and discussed in great detail. Interestingly, all of the key themes identified in the first HIS meeting held over 30 years ago are still of vital importance today: Patient Centered design, Clinical User Support, Real-time Education, Human-computer Factors and Measuring Clinical User Performance, Meaningful use. CONCLUSIONS: As we continue to work to develop next-generation HISs, we must remember the lessons of the past as we strive to develop the solutions for tomorrow.

Health Informatics 3.0 and other increasingly dispersed technologies require even greater trust: promoting safe evidence-based health informatics. Contribution of the IMIA Working Group on Technology Assessment & Quality Development in Health Informatics.

BACKGROUND: Health informatics is generally less committed to a scientific evidence-based approach than any other area of health science, which is an unsound position. Introducing the new Web 3.0 paradigms into health IT applications can unleash a further great potential, able to integrate and distribute data from multiple sources. The counter side is that it makes the user and the patient evermore dependent on the 'black box' of the system, and the re-use of the data remote from the author and initial context. Thus anticipatory consideration of uses, and proactive analysis of evidence of effects, are imperative, as only when a clinical technology can be proven to be trustworthy and safe should it be implemented widely - as is the case with other health technologies. OBJECTIVES: To argue for promoting evidence-based health informatics as systems become more powerful and pro-active yet more dispersed and remote; and evaluation as the means of generating the necessary scientific evidence base. To present ongoing IMIA and EFMI initiatives in this field. METHODS: Critical overview of recent developments in health informatics evaluation, alongside the precedents of other health technologies, summarising current initiatives and the new challenges presented by Health Informatics 3.0. RESULTS: Web 3.0 should be taken as an opportunity to move health informatics from being largely unaccountable to one of being an ethical and responsible science-based domain. Recent and planned activities of the EFMI and IMIA working groups have significantly progressed key initiatives. CONCLUSIONS: Concurrent with the emergence of Web 3.0 as a means of new-generation diffuse health information systems comes an increasing need for an evidence-based culture in health informatics.

On using references as evidence.

BACKGROUND: One has to start to consider references as a kind of evidence similar to measured data and observations. Moreover, proper referencing adds to the credibility of a paper and gives appropriate credits to work of others. Experience of reviewers and editors indicate that there is room for improvement in the way that our scientific evidence base is referred to. OBJECTIVE: To provide guidelines for referencing in medical informatics publications. METHOD: The authors have collected examples of poor as well as good referencing as encountered in manuscripts submitted for review as well as in published literature. RESULTS AND CONCLUSION: A set of 12 rules for proper referencing was derived. The rationale for each rule is provided and examples of rule violations and proper use of the rule are presented.

STARE-HI -statement on reporting of evaluation studies in health informatics.

OBJECTIVE: Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS: An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers and as authors of systematic reviews , taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail. RESULTS: A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion. CONCLUSION: A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of (both quantitative and qualitative) evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS: This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Mapping the domain of medical informatics.

OBJECTIVES: The domain of medical informatics (MI) is not well defined. It covers a wide range of research topics. Our objective is to characterize the field of MI by means of the scientific literature in this domain. METHODS: We used titles and abstracts from MEDLINE records of papers published between July 1993 and July 2008, and extracted uni-, bi- and trigrams as features. Starting with the ISI category of medical informatics, we applied a semi-automated procedure to identify the set of journals and proceedings pertaining to MI. A clustering algorithm was subsequently applied to the articles from this set of publications. RESULTS: MI literature can be divided into three subdomains: 1) the organization, application, and evaluation of health information systems, 2) medical knowledge representation, and 3) signal and data analysis. Over the last fifteen years, the field has remained relatively stable, although most journals have shifted their focus somewhat. CONCLUSIONS: We identified the scientific literature pertaining to the field of MI, and the main areas of research. We were able to show trends in the field, and the positioning of different journals within this field.

Impact of CPOE on mortality rates--contradictory findings, important messages.

OBJECTIVE: To analyze the seemingly contradictory results of the Han study (Pediatrics 2005) and the Del Beccaro study (Pediatrics 2006), both analyzing the effect of CPOE systems on mortality rates in pediatric intensive care settings. METHODS: Seven CPOE system experts from the United States and Europe comment on these papers. RESULTS: The two studies are not contradictory, but almost non-comparable due to differences in design and implementation. They demonstrate the range of outcomes that can be obtained from introducing informatics applications in complex health care settings. Implementing informatics applications is a sociotechnical activity, which often depends more on the organizational context than on a specific technology. As health informaticians, we must not only learn from failures, but also avoid both uncritical scepticism that may arise from drawing overly general conclusions from one negative trial, as much as uncritical optimism from limited successful ones. CONCLUSION: The commentaries emphasize the need to promote systematic studies for assessing the socio-technical factors that influence the introduction of increasingly sophisticated informatics applications within complex organizations. The emergence of evidence-based health informatics will be based both on evaluation guidelines and implementation guidelines, both of which increase the chances of successful implementation. In addition, well-educated health informaticians are needed to manage and guide the implementation processes.

Triangulation applied to Jan H. van Bemmel.

OBJECTIVE: To describe the person of Jan H. van Bemmel from different points of view. METHOD: Triangulation. RESULTS AND CONCLUSIONS: Jan H. van Bemmel successfully contributed to research and education in medical informatics. He inspired a lot of people in The Netherlands and internationally.

Evaluation and implementation: a call for action.

OBJECTIVES: To raise awareness for actions that are urgently needed to accompany the large scale implementations of ICT in Health Care that are currently taking place in many countries around the world. METHODS: An analysis of a few studies that have recently been described in the literature guided by recent suggestions for research and development of evaluation of health ICT. RESULTS AND CONCLUSION: Six specific recommendations for action are specified: Development of good implementation practice, Development of an experience base of implementation of ICT in health care, Setting up a surveillance system for unintended effects, Build an evidence base of best evaluation practice, Developing guidelines for proper reporting of evaluation studies, Education of clinicians and decision makers.

Factors influencing success and failure of health informatics systems--a pilot Delphi study.

OBJECTIVES: The aim is to gain information on factors influencing success and failure for Health Informatics applications from a group of medical informaticians. METHODS: Based on the presentations at a special topic conference on success and failure in Health ICT and analysis of the proceedings, we conducted a Delphi study on success and failure aspects. RESULTS: A total of 110 success factors and 27 failure criteria were identified, distributed on categories like functional, organizational, behavioral, technical, managerial, political, cultural, legal, strategy, economy, education and user acceptance. These factors and criteria were rated for six different system types. Unanimously it was agreed that "collaboration and co-operation" and "setting goals and courses" are "essential for the success" of clinical systems, and "user acceptance" for educational systems. Similarly, the score "essential in order to avoid a failure" were given unanimously on clinical systems for "response rate and other performance measures" and on administrative systems for "not understanding the organizational context" with "not understanding or foreseeing the extent to which the new IT-system affects the organization, its structure and/or work procedures" as the highest scoring sub-item. CONCLUSIONS: All success factors and failure criteria were considered relevant by the Delphi expert panel. There is no small set of relevant factors or indicators, but success or failure of a Health ICT depends on a large set of issues. Further, clinical systems and decision support systems depend on more factors than other systems.

Diagnostic reference frames for seizures: A validation study.

INTRODUCTION: We developed structured descriptions of signs and symptoms for specific seizure types (called Diagnostic Reference Frames-DRFs-by us) that can serve as a frame of reference in the process of classifying patients with epileptic seizures. In this study the validity of the DRFs for clinical use is evaluated and described. MATERIAL AND METHODS: In this study we use a decision support system based on the DRFs and using Bayes's rule for the validation of the DRFs. Patient's manifestations are entered in the decision support system and by successively applying Bayes's rule posterior probabilities are calculated. The DRFs with the highest posterior probability gives an indication of the classification of the seizure. The validation of the DRFs was performed by comparing the seizure type with the highest posterior probability with the classification of experienced epileptologists on a series of test cases with known epileptic seizures. In this way we assessed the accuracy of the DRFs in classifying patients with epileptic seizures. RESULTS: We included sixty-six patients in this efficacy study. The patients and/or their relatives described the manifestations occurring during a seizure. Sixty cases (91%) were correctly classified using the decision support system. DISCUSSION: The accuracy of 91 % indicates that the knowledge encoded in the DRFs for the included seizure types is valid. The next step is to test the DRFs in a clinical setting to evaluate the applicability in daily practice.

Three-Layer Model for the design of a Protocol Support System.

OBJECTIVE: The aim of the PropeR project is to investigate the impact of Active Computerized Protocol Support (ACPS) on daily care processes in different settings (home care and hospital care). ACPS consists of an active Protocol Support System (PSS) that is linked to an Electronic Patient Record system. The aim of this paper is to describe how we have taken the organizational and social aspects into account in the hospital setting and the consequences of this approach for the design of the PSS. METHODS: Socio-technical approaches have been applied. Observations and interviews with various health care providers were performed at the hematology and oncology department of the University Hospital Maastricht. Ten extensive sessions with a specialist physician and research nurse took place to further elaborate a study protocol and to discuss how it is integrated in daily practice. The knowledge editor component of Gaston was used to build a computer interpretable version of the selected protocol. RESULTS AND CONCLUSIONS: To support the representation of a study protocol integrated in routine clinical care, a Three-Layer Model was developed. This model distinguishes the protocol description, local adaptations to the protocol and communication as three separate layers. These layers have been incorporated into the knowledge acquisition tool Gaston. The Three-Layer Model makes easy updating possible, and also supports transferability of computerized (study) protocols to other organizations.

An approach to knowledge base construction based on expert opinions.

OBJECTIVES: To describe, validate and demonstrate an approach for knowledge base construction based on expert opinions. METHODS: A knowledge base containing the frequency of occurrence of manifestations in epileptic seizures is constructed based on information provided by neurologists/epileptologists. The reliability of the responses is determined with the inter-rater intraclass correlation coefficient (ICC). If the ICC is not large enough the Spearman-Brown prophecy formula can be used to predict the number of additional experts. We propose a method to assess whether an additional expert provides information consistent with the already acquired data as well as a method to detect experts with deviating opinions. The power of the first method was determined. RESULTS: Data were collected for five seizure types. The ICCs determined from the responses for the various seizure types after inclusion of the additional experts was in all cases almost equal to 0.9, the target value. Yet one expert with diverging opinions concerning the frequency of occurrence of manifestations for different seizure types could be identified. Excluding this participant improved the reliability of the data. The power of the methods was good (> or =0.75). CONCLUSIONS: It is shown that human experts can provide reliable information about the frequency of occurrence of manifestations in epileptic seizures. In addition, the described approach correctly identified neurologists/epileptologists with both consistent and diverging opinions about the frequency of occurrence of manifestations in a number of seizure types.