How to evaluate the implementation of complex health programmes in low-income settings: the approach of the Gavi Full Country Evaluations.

Vaccination, like most other public health services, relies on a complex package of intervention components, functioning systems and committed actors to achieve universal coverage. Despite significant investment in immunization programmes, national coverage trends have slowed and equity gaps have grown. This paper describes the design and implementation of the Gavi Full Country Evaluations, a multi-country, prospective, mixed-methods approach whose goal was to monitor and evaluate processes, inputs, outputs and outcomes of immunization programmes in Bangladesh, Mozambique, Uganda and Zambia. We implemented the Full Country Evaluations from 2013 to 2018 with the goal of identifying the drivers of immunization programme improvement to support programme implementation and increase equitable immunization coverage. The framework supported methodological and paradigmatic flexibility to respond to a broad range of evaluation and implementation research questions at global, national and cross-country levels, but was primarily underpinned by a focus on evaluating processes and identifying the root causes of implementation breakdowns. Process evaluation was driven by theories of change for each Gavi funding stream (e.g. Health Systems Strengthening) or activity, ranging from global policy development to district-level programme implementation. Mixing of methods increased in relevance and rigour over time as we learned to build multiple methods into increasingly tailored evaluation questions. Evaluation teams in country-based research institutes increasingly strengthened their level of embeddedness with immunization programmes as the emphasis shifted over time to focus more heavily on the use of findings for programme learning and adaptation. Based on our experiences implementing this approach, we recommend it for the evaluation of other complex interventions, health programmes or development assistance.

Evaluating Global Health Partnerships: A Case Study of a Gavi HPV Vaccine Application Process in Uganda.

BACKGROUND: Global health partnerships have grown rapidly in number and scope, yet there has been less emphasis on their evaluation. Gavi, the Vaccine Alliance, is one such public-private partnership; in Gavi-eligible countries partnerships are dynamic networks of immunization actors who work together to support all stages and aspects of Gavi support. This paper describes a conceptual framework - the partnership framework - and analytic approach for evaluating the perceptions of partnerships' added value as well as the results from an application to one case in Uganda. METHODS: We used a mixed-methods case study design embedded in the Gavi Full Country Evaluations (FCE) to test the partnership framework on Uganda's human papillomavirus (HPV) vaccine application partnership. Data from document review, interviews, and social network surveys enabled the testing of the relationships between partnership framework domains (context, structure, practices, performance, and outcomes). Topic guides were based on the framework domains and network surveys identified working together relationships, professional trust, and perceptions of the effectiveness, efficiency, and legitimacy of the partnership's role in this process. RESULTS: Data from seven in-depth interviews, 11 network surveys and document review were analyzed according to the partnership framework, confirming relationships between the framework domains. Trust was an important contributor to the perceived effectiveness of the process. The network was structured around the EPI program, who was considered the leader of this process. While the structure and composition of the network was largely viewed as supporting an effective and legitimate process, the absence of the Ministry of Education (MoE) may have had downstream consequences if this study's results had not been shared with the Ministry of Health (MoH) and acted upon. The partnership was not perceived to have increased the efficiency of the process, perhaps as a result of unclear or absent guidelines around roles and responsibilities. CONCLUSION: The health and functioning of global health partnerships can be evaluated using the framework and approach presented here. Network theory and methods added value to the conceptual and analytic processes and we recommend applying this approach to other global health partnerships to ensure that they are meeting the complex challenges they were designed to address.

An updated methodology to review developing-country vaccine manufacturer viability.

In 1997, Milstien, Batson, and Meaney published "A Systematic Method for Evaluating the Potential Viability of Local Vaccine Producers." The paper identified characteristics of successful vaccine manufacturers and developed a viability framework to evaluate their performance. This paper revisits the original study after two decades to determine the ability of the framework to predict manufacturer success. By reconstructing much of the original dataset and conducting in-depth interviews, the authors developed informed views on the continued viability of manufacturers in low- and middle-income country markets. Considering the marked changes in the market and technology landscape since 1997, the authors find the viability framework to be predictive and a useful lens through which to evaluate manufacturer success or failure. Of particular interest is how incumbent and potentially new developing-country vaccine manufacturers enter and sustain production in competitive international markets and how they integrate (or fail to integrate) new technology into the production process. Ultimately, most manufacturers will need to meet global quality standards to be viable. As governments and donors consider investments in vaccine producers, the updated viability factors will be a useful tool in evaluating the prospects of manufacturers over the mid to long term. The paper emphasizes that while up-front investments are important, other critical factors-including investments in a national regulatory authority, manufacturer independence, and ability to adapt and adopt new technology-are necessary to ensure viability.