Lessons Learned From Implementing Prospective, Multicountry Mixed-Methods Evaluations for Gavi and the Global Fund.

INTRODUCTION: As global health programs have become increasingly complex, corresponding evaluations must be designed to assess the full complexity of these programs. Gavi and the Global Fund have commissioned 2 such evaluations to assess the full spectrum of their investments using a prospective mixed-methods approach. We aim to describe lessons learned from implementing these evaluations. METHODS: This article presents a synthesis of lessons learned based on the Gavi and Global Fund prospective mixed-methods evaluations, with each evaluation considered a case study. The lessons are based on the evaluation team's experience from over 7 years (2013-2020) implementing these evaluations. The Centers for Disease Control and Prevention Framework for Evaluation in Public Health was used to ground the identification of lessons learned. RESULTS: We identified 5 lessons learned that build on existing evaluation best practices and include a mix of practical and conceptual considerations. The lessons cover the importance of (1) including an inception phase to engage stakeholders and inform a relevant, useful evaluation design; (2) aligning on the degree to which the evaluation is embedded in the program implementation; (3) monitoring programmatic, organizational, or contextual changes and adapting the evaluation accordingly; (4) hiring evaluators with mixed-methods expertise and using tools and approaches that facilitate mixing methods; and (5) contextualizing recommendations and clearly communicating their underlying strength of evidence. CONCLUSION: Global health initiatives, particularly those leveraging complex interventions, should consider embedding evaluations to understand how and why the programs are working. These initiatives can learn from the lessons presented here to inform the design and implementation of such evaluations.

Partnership and Participation-A Social Network Analysis of the 2017 Global Fund Application Process in the Democratic Republic of the Congo and Uganda.

Background: The Global Fund to Fight AIDS, Tuberculosis and Malaria was founded in 2002 as a public-private partnership between governments, the private sector, civil society, and populations affected by the three diseases. A key principle of the Global Fund is country ownership in accessing funding through "engagement of in-country stakeholders, including key and vulnerable populations, communities, and civil society." Research documenting whether diverse stakeholders are actually engaged and on how stakeholder engagement affects processes and outcomes of grant applications is limited. Objective: To examine representation during the 2017 Global Fund application process in the Democratic Republic of the Congo (DRC) and Uganda and the benefits and drawbacks of partnership to the process. Methods: We developed a mixed-methods social network survey to measure network structure and assess perceptions of how working together in partnership with other individuals/organizations affected perceived effectiveness, efficiency, and country ownership of the application process. Surveys were administered from December 2017-May 2018, initially to a set of central actors, followed by any individuals named during the surveys (up to 10) as collaborators. Network analyses were conducted using R. Findings: Collaborators spanning many organizations and expertise areas contributed to the 2017 applications (DRC: 152 nodes, 237 ties; Uganda: 118 nodes, 241 ties). Participation from NGOs and civil society representatives was relatively strong, with most of their ties being to different organization types, Uganda (63%), and DRC (67%), highlighting their collaborative efforts across the network. Overall, the perceived benefits of partnership were high, including very strong ratings for effectiveness in both countries. Perceived drawbacks of partnership were minimal; however, less than half of respondents thought partnership helped reduce transaction costs or financial costs, suggesting an inclusive and participatory process may come with short-term efficiency tradeoffs. Conclusions: Social network analysis can be useful for identifying who is included and excluded from the process, which can support efforts to ensure stronger, more meaningful engagement in future Global Fund application processes.

Bringing a health systems modelling approach to complex evaluations: multicountry applications in HIV, TB and malaria.

INTRODUCTION: Understanding how to deliver interventions more effectively is a growing emphasis in Global Health. Simultaneously, health system strengthening is a key component to improving delivery. As a result, it is challenging to evaluate programme implementation while reflecting real-world complexity. We present our experience in using a health systems modelling approach as part of a mixed-methods evaluation and describe applications of these models. METHODS: We developed a framework for how health systems translate financial inputs into health outcomes, with in-country and international experts. We collated available data to measure framework indicators and developed models for malaria in Democratic Republic of the Congo (DRC), and tuberculosis in Guatemala and Senegal using Bayesian structural equation modelling. We conducted several postmodelling analyses: measuring efficiency, assessing bottlenecks, understanding mediation, analysing the cascade of care and measuring subnational effectiveness. RESULTS: The DRC model indicated a strong relationship between shipment of commodities and utilisation thereof. In Guatemala, the strongest model coefficients were more evenly distributed. Results in Senegal varied most, but pathways related to community care had the strongest relationships. In DRC, we used model results to estimate the end-to-end cost of delivering commodities. In Guatemala, we used model results to identify potential bottlenecks and understand mediation. In Senegal, we used model results to identify potential weak links in the cascade of care, and explore subnationally. CONCLUSION: This study demonstrates a complementary modelling approach to traditional evaluation methods. Although these models have limitations, they can be applied in a variety of ways to gain greater insight into implementation and functioning of health service delivery.

Getting to FP2020: Harnessing the private sector to increase modern contraceptive access and choice in Ethiopia, Nigeria, and DRC.

BACKGROUND: An estimated 214 million women have unmet need for family planning in developing regions. Improved utilization of the private sector is key to achieving universal access to a range of safe and effective modern contraceptive methods stipulated by FP2020 and SDG commitments. Until now, a lack of market data has limited understanding of the private sector's role in increasing contraceptive coverage and choice. METHODS: In 2015, the FPwatch Project conducted representative outlet surveys in Ethiopia, Nigeria, and DRC using a full census approach in selected administrative areas. Every public and private sector outlet with the potential to sell or distribute modern contraceptives was approached. In outlets with modern contraceptives, product audits and provider interviews assessed contraceptive market composition, availability, and price. FINDINGS: Excluding general retailers, 96% of potential outlets in Ethiopia, 55% in Nigeria, and 41% in DRC had modern contraceptive methods available. In Ethiopia, 41% of modern contraceptive stocking outlets were in the private sector compared with approximately 80% in Nigeria and DRC where drug shops were dominant. Ninety-five percent of private sector outlets in Ethiopia had modern contraceptive methods available; 37% had three or more methods. In Nigeria and DRC, only 54% and 42% of private sector outlets stocked modern contraceptives with 5% and 4% stocking three or more methods, respectively. High prices in Nigeria and DRC create barriers to consumer access and choice. DISCUSSION: There is a missed opportunity to provide modern contraception through the private sector, particularly drug shops. Subsidies and interventions, like social marketing and social franchising, could leverage the private sector's role in increasing access to a range of contraceptives. Achieving global FP2020 commitments depends on the expansion of national contraceptive policies that promote greater partnership and cooperation with the private sector and improvement of decisions around funding streams of countries with large populations and high unmet need like Ethiopia, Nigeria, and DRC.

The malaria testing and treatment market in Kinshasa, Democratic Republic of the Congo, 2013.

BACKGROUND: The Democratic Republic of Congo (DRC) is one of the two most leading contributors to the global burden of disease due to malaria. This paper describes the malaria testing and treatment market in the nation's capital province of Kinshasa, including availability of malaria testing and treatment and relative anti-malarial market share for the public and private sector. METHODS: A malaria medicine outlet survey was conducted in Kinshasa province in 2013. Stratified multi-staged sampling was used to select areas for the survey. Within sampled areas, all outlets with the potential to sell or distribute anti-malarials in the public and private sector were screened for eligibility. Among outlets with anti-malarials or malaria rapid diagnostic tests (RDT) in stock, a full audit of all available products was conducted. Information collected included product information (e.g. active ingredients, brand name), amount reportedly distributed to patients in the past week, and retail price. RESULTS: In total, 3364 outlets were screened for inclusion across Kinshasa and 1118 outlets were eligible for the study. Among all screened outlets in the private sector only about one in ten (12.1%) were stocking quality-assured Artemisinin-based Combination Therapy (ACT) medicines. Among all screened public sector facilities, 24.5% had both confirmatory testing and quality-assured ACT available, and 20.2% had sulfadoxine-pyrimethamine (SP) available for intermittent preventive therapy during pregnancy (IPTp). The private sector distributed the majority of anti-malarials in Kinshasa (96.7%), typically through drug stores (89.1% of the total anti-malarial market). Non-artemisinin therapies were the most commonly distributed anti-malarial (50.1% of the total market), followed by non quality-assured ACT medicines (38.5%). The median price of an adult quality-assured ACT was $6.59, and more expensive than non quality-assured ACT ($3.71) and SP ($0.44). Confirmatory testing was largely not available in the private sector (1.1%). CONCLUSIONS: While the vast majority of anti-malarial medicines distributed to patients in Kinshasa province are sold within the private sector, availability of malaria testing and appropriate treatment for malaria is alarmingly low. There is a critical need to improve access to confirmatory testing and quality-assured ACT in the private sector. Widespread availability and distribution of non quality-assured ACT and non-artemisinin therapies must be addressed to ensure effective malaria case management.