SEEP-CI: A Structured Economic Evaluation Process for Complex Health System Interventions.

The economic evaluation of health system interventions is challenging, and methods guidance on how to respond to these challenges is lacking. The REACHOUT consortium developed and evaluated complex interventions for community health program quality improvement in six countries in Africa and Asia. Reflecting on the challenges we faced in conducting an economic evaluation alongside REACHOUT, we developed a Structured Economic Evaluation Process for Complex Health System Interventions (SEEP-CI). The SEEP-CI aims to establish the threshold effect size that would justify investment in a complex intervention, and provide an assessment to a decision-maker of how likely it is that the intervention can achieve this impact. We illustrate how the SEEP-CI could have been applied to REACHOUT to identify outcomes where the intervention might have impact and causal mechanisms, through which that impact might occur, guide data collection by focusing on proximal outcomes most likely to illustrate the effectiveness of the intervention, identify the size of health gain required to justify investment in the intervention, and indicate the assumptions required to accept that such health gains are credible. Further research is required to determine the feasibility and acceptability of the SEEP-CI, and the contexts in which it could be used.

'Do you trust those data?'-a mixed-methods study assessing the quality of data reported by community health workers in Kenya and Malawi.

High-quality data are essential to monitor and evaluate community health worker (CHW) programmes in low- and middle-income countries striving towards universal health coverage. This mixed-methods study was conducted in two purposively selected districts in Kenya (where volunteers collect data) and two in Malawi (where health surveillance assistants are a paid cadre). We calculated data verification ratios to quantify reporting consistency for selected health indicators over 3 months across 339 registers and 72 summary reports. These indicators are related to antenatal care, skilled delivery, immunization, growth monitoring and nutrition in Kenya; new cases, danger signs, drug stock-outs and under-five mortality in Malawi. We used qualitative methods to explore perceptions of data quality with 52 CHWs in Kenya, 83 CHWs in Malawi and 36 key informants. We analysed these data using a framework approach assisted by NVivo11. We found that only 15% of data were reported consistently between CHWs and their supervisors in both contexts. We found remarkable similarities in our qualitative data in Kenya and Malawi. Barriers to data quality mirrored those previously reported elsewhere including unavailability of data collection and reporting tools; inadequate training and supervision; lack of quality control mechanisms; and inadequate register completion. In addition, we found that CHWs experienced tensions at the interface between the formal health system and the communities they served, mediated by the social and cultural expectations of their role. These issues affected data quality in both contexts with reports of difficulties in negotiating gender norms leading to skipping sensitive questions when completing registers; fabrication of data; lack of trust in the data; and limited use of data for decision-making. While routine systems need strengthening, these more nuanced issues also need addressing. This is backed up by our finding of the high value placed on supportive supervision as an enabler of data quality.