ABSTRACT
BACKGROUND: Summative eHealth evaluations frequently lack quality, which affects the generalizability of the evidence, and its use in practice and further research. To guarantee quality, a number of activities are recommended in the guidelines for evaluation planning. This study aimed to examine a case of an eHealth evaluation planning in a multi-national and interdisciplinary setting and to provide recommendations for eHealth evaluation planning guidelines. METHODS: An empirical eHealth evaluation process was developed through a case study. The empirical process was compared with selected guidelines for eHealth evaluation planning using a pattern-matching technique. RESULTS: Planning in the interdisciplinary and multi-national team demanded extensive negotiation and alignment to support the future use of the evidence created. The evaluation planning guidelines did not provide specific strategies for different set-ups of the evaluation teams. Further, they did not address important aspects of quality evaluation, such as feasibility analysis of the outcome measures and data collection, monitoring of data quality, and consideration of the methods and measures employed in similar evaluations. CONCLUSIONS: Activities to prevent quality problems need to be incorporated in the guidelines for evaluation planning. Additionally, evaluators could benefit from guidance in evaluation planning related to the different set-ups of the evaluation teams.
Subject(s)
Telemedicine , HumansABSTRACT
BACKGROUND: Until now, the use of technology in health care was driven mostly by the assumptions about the benefits of electronic health (eHealth) rather than its evidence. It is noticeable that the magnitude of evidence of effectiveness and efficiency of eHealth is not proportionate to the number of interventions that are regularly conducted. Reliable evidence generated through comprehensive evaluation of eHealth interventions may accelerate the growth of eHealth for long-term successful implementation and help to experience eHealth benefits in an enhanced way. OBJECTIVE: This study aimed to understand how the evidence of effectiveness and efficiency of eHealth can be generated through evaluation. Hence, we aim to discern (1) how evaluation is conducted in distinct eHealth intervention phases, (2) the aspects of effectiveness and efficiency that are typically evaluated during eHealth interventions, and (3) how eHealth interventions are evaluated in practice. METHODS: A systematic literature review was conducted to explore the evaluation methods for eHealth interventions. Preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines were followed. We searched Google Scholar and Scopus for the published papers that addressed the evaluation of eHealth or described an eHealth intervention study. A qualitative analysis of the selected papers was conducted in several steps. RESULTS: We intended to see how the process of evaluation unfolds in distinct phases of an eHealth intervention. We revealed that in practice and in several conceptual papers, evaluation is performed at the end of the intervention. There are some studies that discuss the importance of conducting evaluation throughout the intervention; however, in practice, we found no case study that followed this. For our second research question, we discovered aspects of efficiency and effectiveness that are proposed to be assessed during interventions. The aspects that were recurrent in the conceptual papers include clinical, human and social, organizational, technological, cost, ethical and legal, and transferability. However, the case studies reviewed only evaluate the clinical and human and social aspects. At the end of the paper, we discussed a novel approach to look into the evaluation. Our intention was to stir up a discussion around this approach with the hope that it might be able to gather evidence in a comprehensive and credible way. CONCLUSIONS: The importance of evidence in eHealth has not been discussed as rigorously as have the diverse evaluation approaches and evaluation frameworks. Further research directed toward evidence-based evaluation can not only improve the quality of intervention studies but also facilitate successful long-term implementation of eHealth in general. We conclude that the development of more robust and comprehensive evaluation of eHealth studies or an improved validation of evaluation methods could ease the transferability of results among similar studies. Thus, the resources can be used for supplementary research in eHealth.
Subject(s)
Quality of Health Care/trends , Telemedicine/methods , Adult , Aged , Female , Humans , Male , Middle Aged , Retrospective Studies , Young AdultABSTRACT
Objetivo: Identificar los estudios clínicos controlados (ECC), los estudios clínicos aleatorizados (RCT) y metaanálisis publicados en las revistas de salud de Colombia, en el período comprendido entre 1948 y 1998. Material y métodos: se ubicaron 255 revistas del listado suministrado por la Hemeroteca del Instituto Colombiano para la Educación Superior (ICFES), se seleccionaron 85 a nivel nacional y 42 que correspondían a la ciudad de Santafé de Bogotá que cumplían con los siguientes requisitos: 1) ser llevado a cabo en humanos; 2) comparar dos o más intervenciones entre sí; 3) asignación aleatoria de las intervenciones; 4) diseño prospectivo en la planificación de las intervenciones y asignación de los sujetos. Se utilizó la metodología "Guía para la búsqueda manual de ensayos clínicos controlados" propuesta por el Centro Cochrane Español. Resultados: en el período comprendido entre octubre del 98 y octubre del 99 se revisaron 22 publicaciones de la ciudad de Santafé de Bogotá, encontrándose un total de 151 estudios distribuidos así: 133 CCT, 16 RCT y dos metaanálisis, ocupando los primeros lugares: Revista Colombiana de Anestesiología, Revista Colombiana de Ginecología y Obstetricia, Acta de Otorrinolaringología y Cirugía de Cabeza y Cuello, Revista de la Sociedad Colombiana de Pediatría y Acta Médica Colombiana. Los temas más publicados fueron: ginecología y obstetricia 12 ensayos (sin clasificar), trauma seis ensayos, infectología cinco ensayos (sin clasificar) y patología osteomuscular tres ensayos. Conclusiones: la identificación de publicaciones con validez metodológica permite evaluar la calidad de la investigación y de las publicaciones en nuestro medio, además de comparar si corresponden a prioridades nuestras. Es necesario completar el estudio incluyendo el resto de las publicaciones identificadas en las diferentes ciudades del país.
