Planning a holistic summative eHealth evaluation in an interdisciplinary and multi-national setting: a case study and propositions for guideline development.

BACKGROUND: Summative eHealth evaluations frequently lack quality, which affects the generalizability of the evidence, and its use in practice and further research. To guarantee quality, a number of activities are recommended in the guidelines for evaluation planning. This study aimed to examine a case of an eHealth evaluation planning in a multi-national and interdisciplinary setting and to provide recommendations for eHealth evaluation planning guidelines. METHODS: An empirical eHealth evaluation process was developed through a case study. The empirical process was compared with selected guidelines for eHealth evaluation planning using a pattern-matching technique. RESULTS: Planning in the interdisciplinary and multi-national team demanded extensive negotiation and alignment to support the future use of the evidence created. The evaluation planning guidelines did not provide specific strategies for different set-ups of the evaluation teams. Further, they did not address important aspects of quality evaluation, such as feasibility analysis of the outcome measures and data collection, monitoring of data quality, and consideration of the methods and measures employed in similar evaluations. CONCLUSIONS: Activities to prevent quality problems need to be incorporated in the guidelines for evaluation planning. Additionally, evaluators could benefit from guidance in evaluation planning related to the different set-ups of the evaluation teams.

Identifying the Value of an eHealth Intervention Aimed at Cognitive Impairments: Observational Study in Different Contexts and Service Models.

BACKGROUND: Value is one of the central concepts in health care, but it is vague within the field of summative eHealth evaluations. Moreover, the role of context in explaining the value is underexplored, and there is no explicit framework guiding the evaluation of the value of eHealth interventions. Hence, different studies conceptualize and operationalize value in different ways, ranging from measuring outcomes such as clinical efficacy or behavior change of patients or professionals to measuring the perceptions of various stakeholders or in economic terms. OBJECTIVE: The objective of our study is to identify contextual factors that determine similarities and differences in the value of an eHealth intervention between two contexts. We also aim to reflect on and contribute to the discussion about the specification, assessment, and relativity of the "value" concept in the evaluation of eHealth interventions. METHODS: The study concerned a 6-month eHealth intervention targeted at elderly patients (n=107) diagnosed with cognitive impairment in Italy and Sweden. The intervention introduced a case manager role and an eHealth platform to provide remote monitoring and coaching services to the patients. A model for evaluating the value of eHealth interventions was designed as monetary and nonmonetary benefits and sacrifices, based on the value conceptualizations in eHealth and marketing literature. The data was collected using the Mini-Mental State Examination (MMSE), the clock drawing test, and the 5-level EQ-5D (EQ-5D-5L). Semistructured interviews were conducted with patients and health care professionals. Monetary data was collected from the health care and technology providers. RESULTS: The value of an eHealth intervention applied to similar types of populations but differed in different contexts. In Sweden, patients improved cognitive performance (MMSE mean 0.85, SD 1.62, P<.001), reduced anxiety (EQ-5D-5L mean 0.16, SD 0.54, P=.046), perceived their health better (EQ-5D-5L VAS scale mean 2.6, SD 9.7, P=.035), and both patients and health care professionals were satisfied with the care. However, the Swedish service model demonstrated an increased cost, higher workload for health care professionals, and the intervention was not cost-efficient. In Italy, the patients were satisfied with the care received, and the health care professionals felt empowered and had an acceptable workload. Moreover, the intervention was cost-effective. However, clinical efficacy and quality of life improvements have not been observed. We identified 6 factors that influence the value of eHealth intervention in a particular context: (1) service delivery design of the intervention (process of delivery), (2) organizational setup of the intervention (ie, organizational structure and professionals involved), (3) cost of different treatments, (4) hourly rates of staff for delivering the intervention, (5) lifestyle habits of the population (eg, how physically active they were in their daily life and if they were living alone or with family), and (6) local preferences on the quality of patient care. CONCLUSIONS: Value in the assessments of eHealth interventions need to be considered beyond economic terms, perceptions, or behavior changes. To obtain a holistic view of the value created, it needs to be operationalized into monetary and nonmonetary outcomes, categorizing these into benefits and sacrifices.

Exploring the Use of Evidence From the Development and Evaluation of an Electronic Health (eHealth) Trial: Case Study.

BACKGROUND: Evidence-based practice refers to building clinical decisions on credible research evidence, professional experience, and patient preferences. However, there is a growing concern that evidence in the context of electronic health (eHealth) is not sufficiently used when forming policies and practice of health care. In this context, using evaluation and research evidence in clinical or policy decisions dominates the discourse. However, the use of additional types of evidence, such as professional experience, is underexplored. Moreover, there might be other ways of using evidence than in clinical or policy decisions. OBJECTIVE: This study aimed to analyze how different types of evidence (such as evaluation outcomes [including patient preferences], professional experiences, and existing scientific evidence from other research) obtained within the development and evaluation of an eHealth trial are used by diverse stakeholders. An additional aim was to identify barriers to the use of evidence and ways to support its use. METHODS: This study was built on a case of an eHealth trial funded by the European Union. The project included 4 care centers, 2 research and development companies that provided the web-based physical exercise program and an activity monitoring device, and 2 science institutions. The qualitative data collection included 9 semistructured interviews conducted 8 months after the evaluation was concluded. The data analysis concerned (1) activities and decisions that were made based on evidence after the project ended, (2) evidence used for those activities and decisions, (3) in what way the evidence was used, and (4) barriers to the use of evidence. RESULTS: Evidence generated from eHealth trials can be used by various stakeholders for decisions regarding clinical integration of eHealth solutions, policy making, scientific publishing, research funding applications, eHealth technology, and teaching. Evaluation evidence has less value than professional experiences to local decision making regarding eHealth integration into clinical practice. Professional experiences constitute the evidence that is valuable to the highest variety of activities and decisions in relation to eHealth trials. When using existing scientific evidence related to eHealth trials, it is important to consider contextual relevance, such as location or disease. To support the use of evidence, it is suggested to create possibilities for health care professionals to gain experience, assess a few rather than a large number of variables, and design for shorter iterative cycles of evaluation. CONCLUSIONS: Initiatives to support and standardize evidence-based practice in the context of eHealth should consider the complexities in how the evidence is used in order to achieve better uptake of evidence in practice. However, one should be aware that the assumption of fact-based decision making in organizations is misleading. In order to create better chances that the evidence produced would be used, this should be addressed through the design of eHealth trials.

Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice.

BACKGROUND: Application of standards is a way to increase quality in an evaluation study. However, standards are used insufficiently in eHealth evaluation, affecting the generalization of the knowledge generated. This study aimed to explore how standards are used in a practical setting of an eHealth evaluation, and to identify the factors that can hinder their use. METHODS: The data were collected in a multi-national and interdisciplinary eHealth evaluation study targeted at the elderly people suffering from mild cognitive impairment and mild dementia. The study was carried out in four countries and funded by the European Union. The collected data included meeting minutes (n = 8) and e-mail correspondence (n = 261) between partners. The chronological sequence of events related to the use of standards was established. Subsequently, the hindering factors related to the use of standards were identified from the sequence. RESULTS: The use of four standards was described, reflecting a variety of related processes or barriers that emerge during eHealth evaluation. The processes differed based on the type of the standard. Specifically, evaluation frameworks were found to be conceptual standards and they were easy to agree upon, while standardized metrics were more tangible and their use triggered negotiations. Further, the following factors hindered the use of standards in eHealth evaluations: (1) inadequacy of a standard to address a target population or a disease, (2) insufficient resources to use a standard, (3) lack of experience in using a standard, and (4) lack of validation of a standard in a particular location. CONCLUSIONS: Standardization initiatives in eHealth evaluation provide a blueprint for evaluation, but their practical application is problematic. The practical circumstances of an evaluation study can cause deviations in the standards, thus producing heterogeneity in the evaluation methodologies.