Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis.

Objectives: To set the scientific context and then suggest principles for an evidence-based approach to secondary uses of clinical data, covering both evaluation of the secondary uses of data and evaluation of health systems and services based upon secondary uses of data. Method: Working Group review of selected literature and policy approaches. Results: We present important considerations in the evaluation of secondary uses of clinical data from the angles of governance and trust, theory, semantics, and policy. We make the case for a multi-level and multi-factorial approach to the evaluation of secondary uses of clinical data and describe a methodological framework for best practice. We emphasise the importance of evaluating the governance of secondary uses of health data in maintaining trust, which is essential for such uses. We also offer examples of the re-use of routine health data to demonstrate how it can support evaluation of clinical performance and optimize health IT system design. Conclusions: Great expectations are resting upon "Big Data" and innovative analytics. However, to build and maintain public trust, improve data reliability, and assure the validity of analytic inferences, there must be independent and transparent evaluation. A mature and evidence-based approach needs not merely data science, but must be guided by the broader concerns of applied health informatics.

Improving Evaluation to Address the Unintended Consequences of Health Information Technology:. a Position Paper from the Working Group on Technology Assessment & Quality Development.

BACKGROUND AND OBJECTIVES: With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT. METHODS: Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation. CONCLUSION: Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a proactive and evidence-based approach to detecting and addressing the unintended effects of health IT.

Patient Portals as a Means of Information and Communication Technology Support to Patient- Centric Care Coordination - the Missing Evidence and the Challenges of Evaluation. A joint contribution of IMIA WG EVAL and EFMI WG EVAL.

OBJECTIVES: To review the potential contribution of Information and Communication Technology (ICT) to enable patient-centric and coordinated care, and in particular to explore the role of patient portals as a developing ICT tool, to assess the available evidence, and to describe the evaluation challenges. METHODS: Reviews of IMIA, EFMI, and other initiatives, together with literature reviews. RESULTS: We present the progression from care coordination to care integration, and from patient-centric to person-centric approaches. We describe the different roles of ICT as an enabler of the effective presentation of information as and when needed. We focus on the patient's role as a co-producer of health as well as the focus and purpose of care. We discuss the need for changing organisational processes as well as the current mixed evidence regarding patient portals as a logical tool, and the reasons for this dichotomy, together with the evaluation principles supported by theoretical frameworks so as to yield robust evidence. CONCLUSIONS: There is expressed commitment to coordinated care and to putting the patient in the centre. However to achieve this, new interactive patient portals will be needed to enable peer communication by all stakeholders including patients and professionals. Few portals capable of this exist to date. The evaluation of these portals as enablers of system change, rather than as simple windows into electronic records, is at an early stage and novel evaluation approaches are needed.

Human Factors in the Large: Experiences from Denmark, Finland and Canada in Moving Towards Regional and National Evaluations of Health Information System Usability. Contribution of the IMIA Human Factors Working Group.

OBJECTIVES: The objective of this paper is to explore approaches to understanding the usability of health information systems at regional and national levels. METHODS: Several different methods are discussed in case studies from Denmark, Finland and Canada. They range from small scale qualitative studies involving usability testing of systems to larger scale national level questionnaire studies aimed at assessing the use and usability of health information systems by entire groups of health professionals. RESULTS: It was found that regional and national usability studies can complement smaller scale usability studies, and that they are needed in order to understand larger trends regarding system usability. Despite adoption of EHRs, many health professionals rate the usability of the systems as low. A range of usability issues have been noted when data is collected on a large scale through use of widely distributed questionnaires and websites designed to monitor user perceptions of usability. CONCLUSION: As health information systems are deployed on a widespread basis, studies that examine systems used regionally or nationally are required. In addition, collection of large scale data on the usability of specific IT products is needed in order to complement smaller scale studies of specific systems.

Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL.

OBJECTIVES: To present the importance of Evidence-based Health Informatics (EBHI) and the ethical imperative of this approach; to highlight the work of the IMIA Working Group on Technology Assessment and Quality Improvement and the EFMI Working Group on Assessment of Health Information Systems; and to introduce the further important evaluation and evidence aspects being addressed. METHODS: Reviews of IMIA, EFMA and other initiatives, together with literature reviews on evaluation methods and on published systematic reviews. RESULTS: Presentation of the rationale for the health informatics domain to adopt a scientific approach by assessing impact, avoiding harm, and empirically demonstrating benefit and best use; reporting of the origins and rationale of the IMIA- and EQUATOR-endorsed Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) and of the IMIA WG's Guideline for Good Evaluation Practice in Health Informatics (GEP-HI); presentation of other initiatives for objective evaluation; and outlining of further work in hand on usability and indicators; together with the case for development of relevant evaluation methods in newer applications such as telemedicine. The focus is on scientific evaluation as a reliable source of evidence, and on structured presentation of results to enable easy retrieval of evidence. CONCLUSIONS: EBHI is feasible, necessary for efficiency and safety, and ethically essential. Given the significant impact of health informatics on health systems, care delivery and personal health, it is vital that cultures change to insist on evidence-based policies and investment, and that emergent global moves for this are supported.