Using Novel Multimethod Evaluation Approaches to Understand Complex Food System Interventions: Insights from a Supply Chain Intervention Intended to Improve Nutrition.

Background: A "food system" approach to improve diet quality by intervening within areas such as food supply chains is gaining prominence. However, evidence of such interventions' impact, and understanding of appropriate methods to evaluate them, is lacking. Objectives: We present an impact evaluation of an intervention that aimed to increase consumption of nutritious foods by supporting food-producing firms in Kenya. In doing so, we demonstrate how multiple methods, including those from other disciplines, can be used to evaluate a complex food systems intervention. Methods: Four methods focused on food-producing firms and their management, including a survey of intervention participants (n = 83 individuals), a "laboratory-in-the-field" experiment (n = 83 individuals), baseline/endline data on firm performance (n = 71 firms), and semistructured interviews (n = 19 firms). Three methods focused on consumers in neighborhoods targeted by a supported firm: a randomized field experiment tested effects of making a supported product exhaustively available on consumers' purchases and consumption (n = 1295 consumers); 3 discrete choice experiments (n = 1295 consumers) tested factors influencing consumers' willingness to pay for foods with relevant characteristics. Results: Among firms, we saw suggestive evidence of increased networking and business relationships, while laboratory-in-the-field experiments indicated the intervention might foster cooperation among participants. Qualitative interviews suggested that the intervention enabled firms to increase production, improve management, increase revenues, and lower costs. Baseline/endline data confirmed a positive effect only on the launch of new products and hiring workers. In the field experiment, consumption of the supported product increased in areas where it was made available relative to a control group, but this did not increase overall consumption of the food type or dietary diversity. Conclusions: Results showed positive signs of the intervention improving firm-level outcomes but limited impact on consumers' diet quality. The evaluation also demonstrates how diverse methods can be used to evaluate complex interventions.

Effectiveness of the Eggs Make Kids demand-creation campaign at improving household availability of eggs and egg consumption by young children in Nigeria: A quasi-experimental study.

Using a quasi-experimental design, our study aimed to determine the effectiveness of the 'Eggs Make Kids Sharp & Bright and Strong & Active' demand-creation campaign in Nigeria. The intervention arm received emotionally compelling radio and television advertisements about eggs, and was exposed to promotional activities and advertising about eggs at points of purchase, schools and health facilities; the comparison arm received no intervention. Children 6-59 months of age (intervention: n = 1359; comparison: n = 1485) were assessed 14 months apart. Intent-to-treat analyses with analysis of covariance method assessed the impact of the intervention on caregivers' behaviour towards eggs, caregivers' willingness to pay for eggs, availability of eggs in households, and consumption of eggs by children 6-59 months of age. Analyses were adjusted for possible confounders and perceived effects of COVID-19 on finances and food consumption. Compared to the comparison arm, the intervention arm showed a greater prevalence of household egg acquisition (odds ratio = 1.34, p < 0.0001), and larger improvements in caregiver self-efficacy (ß = 0.242, p = 0.004) and intent to feed eggs to children (ß = 0.080, p = 0.021). No effects were found on children's egg consumption or caregivers' reported willingness to pay for eggs. The lack of impact on child egg consumption despite increased acquisition of eggs and caregiver self-efficacy suggests that other barriers to child consumption may exist. Additional research should further investigate factors that may influence intrahousehold distribution of eggs and whether these may also influence other nutritious foods.

Determinants of egg consumption by infants and young children in Ethiopia.

OBJECTIVE: To identify determinants of egg consumption in infants and young children aged 6-23·9 months in Ethiopia. DESIGN AND SETTING: Data used were from the cross-sectional baseline survey of an egg campaign in Ethiopia implemented by the Global Alliance for Improved Nutrition. PARTICIPANTS: Children aged 6-23·9 months (n 453) were sampled. Data on socio-demographic characteristics, economic resources, caregiver's behaviour, child health and feeding practices, and egg consumption in the last 7 d were collected using interviewer-administered questionnaires. Multivariable ordinal logistic regression was used to examine the association between explanatory variables and egg consumption in the last 7 d. RESULTS: About half of children (53·4 %) did not consume eggs in the last 7 d. The odds of children consuming eggs were 4·33 (P < 0·002) times higher when their caregivers had some college education compared with no education. Wealth was positively (OR, 1·13, P = 0·029) and household food insecurity was negatively (OR, 0·96, P = 0·117) associated with child egg consumption. Purchasing eggs (OR, 9·73, P < 0·001) and caregiver's positive behavioural determinants (OR, 1·37, P = 0·005) were associated with child egg consumption. The associations of socio-demographic characteristics and economic resources with egg consumption provide evidence of partial mediation through caregiver behaviour and child health. CONCLUSIONS: About half of children aged 6-23·9 months consumed eggs. Availability of eggs in households, mainly through purchase, was strongly associated with egg consumption. Education of caregivers and household heads and economic resources were associated with egg consumption and may operate through caregiver behaviour.

A Nutrition Behavior Change Program Moderately Improves Minimum Diet Diversity and Handwashing Behaviors Among Tea Workers in Assam and Tamil Nadu, India.

Many workers in global supply chains remain nutritionally vulnerable despite the income they earn. The Seeds of Prosperity (SOP) program was implemented in Tamil Nadu and Assam, India, for tea supply chain workers (estate workers, small holder farmers, and farm workers). The aim was to enhance demand for diverse and nutritious foods and improve practices related to handwashing. The program used a behavior change communication approach wherein participants received weekly 1-hour group sessions with messaging on dietary diversity for 5 weeks and handwashing for 4 weeks. An impact evaluation was conducted to estimate changes in reported dietary and hygiene knowledge and behaviors among women. The study used a longitudinal quasi-experimental design in a subsample of program participants at baseline and post-intervention among both intervention and comparison. There was a small but significant increase in mean dietary diversity (DD) for all 4 worker groups (ranging from DD score changes of 0.3 to 0.7; P < .05) and in the proportion of women meeting the minimum dietary diversity in 2 of the 4 groups. Similarly, a significant increase in the mean number of handwashing moments was observed in 2 of the worker groups. An increase in home garden use was observed in 1 of the 4 worker groups. While the SOP program resulted in improvements in dietary diversity, most tea farming women still do not achieve minimum dietary diversity. Nutritious food access may be an important constraint to further improvement.

Can Double Fortification of Salt with Iron and Iodine Reduce Anemia, Iron Deficiency Anemia, Iron Deficiency, Iodine Deficiency, and Functional Outcomes? Evidence of Efficacy, Effectiveness, and Safety.

BACKGROUND: Anemia, iron deficiency, and iodine deficiency are problems of important public health concern in many parts of the world, with consequences for the health, development, and work capacity of populations. Several countries are beginning to implement double fortified salt (DFS) programs to simultaneously address iodine and iron deficiencies. OBJECTIVE: Our objective was to summarize the evidence for efficacy and effectiveness of DFS on the full range of status and functional outcomes and across different implementation and evaluation designs essential to successful interventions. METHODS: We conducted a systematic review and meta-analysis of published and gray literature examining the effects of DFS on nutritional status, cognition, work productivity, development, and morbidity of all population groups. We searched for articles in Medline, Embase, CINAHL, Cochrane Central Register, and ProQuest for randomized trials, quasi-randomized trials, and program effectiveness evaluations. RESULTS: A total of 22 studies (N individuals = 52,758) were included. Efficacy studies indicated a significant overall positive effect on hemoglobin concentration [standardized mean difference (95% CI): 0.33 (0.18, 0.48)], ferritin [0.42 (0.08, 0.76)], anemia [risk ratio (95% CI): 0.80 (0.70, 0.92)], and iron deficiency anemia [0.36 (0.24, 0.55)]. Effects on urinary iodine concentration were not significantly different between DFS and iodized salt. The impact on functional outcomes was mixed. Only 2 effectiveness studies were identified. They reported programmatic challenges including low coverage, suboptimal DFS quality, and storage constraints. CONCLUSIONS: Given the biological benefits of DFS across several populations in efficacy research, additional evaluations of robust DFS programs delivered at scale, which consider effective implementation and measure appropriate biomarkers, are needed.

Cost-effectiveness of a market-based home fortification of food with micronutrient powder programme in Bangladesh.

OBJECTIVE: We estimated the cost-effectiveness of home fortification with micronutrient powder delivered in a sales-based programme in reducing the prevalence of Fe deficiency anaemia among children 6-59 months in Bangladesh. DESIGN: Cross-sectional interviews with local and central-level programme staff and document reviews were conducted. Using an activity-based costing approach, we estimated start-up and implementation costs of the programme. The incremental cost per anaemia case averted and disability-adjusted life years (DALY) averted were estimated by comparing the home fortification programme and no intervention scenarios. SETTING: The home fortification programme was implemented in 164 upazilas (sub-districts) in Bangladesh. PARTICIPANTS: Caregivers of child 6-59 months and BRAC staff members including community health workers were the participants for this study. RESULTS: The home fortification programme had an estimated total start-up cost of 35·46 million BDT (456 thousand USD) and implementation cost of 1111·63 million BDT (14·12 million USD). The incremental cost per Fe deficiency anaemia case averted and per DALY averted was estimated to be 1749 BDT (22·2 USD) and 12 558 BDT (159·3 USD), respectively. Considering per capita gross domestic product (1516·5 USD) as the cost-effectiveness threshold, the home fortification programme was highly cost-effective. The programme coverage and costs for nutritional counselling of the beneficiary were influential parameters for cost per DALY averted in the one-way sensitivity analysis. CONCLUSIONS: The market-based home fortification programme was a highly cost-effective mechanism for delivering micronutrients to a large number of children in Bangladesh. The policymakers should consider funding and sustaining large-scale sales-based micronutrient home fortification efforts assuming the clear population-level need and potential to benefit persists.

Replication of influential studies on biomedical, social, behavioural and structural interventions for HIV prevention and treatment.

Replication is an important tool to promote high quality research and ensure policy makers can rely on studies in making guidelines or funding programs. By ensuring influential studies are replicable we provide assurance that the policies based on these studies are well-founded and the conclusions and recommendations are robust-to different estimation models or different choices. In this paper, we argue that replication is not only useful but necessary to ensure that an author's choice in how to analyse data is not the only factor that determines whether an intervention is effective or not. We also show that while most research is done well and provides robust results, small differences can lead to different interpretations and these differences need to be acknowledged. This special issue highlights 5 such replication studies, which are replications of influential studies on biomedical, social, behavioural and structural interventions for HIV prevention and treatment. We reflect on their findings. Four out of five studies, which conduct push button replication and pure replication, were able to reproduce the results of the original studies with minor differences, mainly due to minor typographical errors or rounding differences. The analysis of the measurement and estimation analyses conducted in these five studies reveals that the original results are not very robust to alternative analytical approaches, especially when these results rely on a small number of observations. In these cases, the original results are weakened. Furthermore, in contrast to the original papers, two of the five included replication studies conducted a theory of change analysis-to explore how or why the interventions work (or do not) not just whether the intervention works or not. These two analyses indicate that the estimated impacts of the interventions are drawn from few mediators. In addition, they demonstrate that, in some cases, a lack of effect may be related to lack of adequate exposure to the intervention rather than inefficacy of the intervention per se. However, overall, the included replication studies show that the results presented in the original papers are trustworthy and robust, especially when based on larger sample sizes. Replication studies can not only verify the results of a study, they can also provide additional insights on the published results, such as how and why an intervention was effective or less effective than expected. They can thus be a tool to inform the research community and/ or policymakers about whether and how interventions could be adopted, which need to be tested further, and which should be discontinued because of their ineffectiveness. Thus, publishing these replication studies in peer-reviewed journals makes the work public and publicized. The work advances knowledge, and publication should be encouraged, as it is for other types of research.

Treatment of HIV among tuberculosis patients: A replication study of timing of antiretroviral therapy for HIV-1-associated tuberculosis.

Co-diagnosis of HIV and tuberculosis presents a treatment dilemma. Starting both treatments at the same time can cause a flood of immune response called immune reconstitution inflammatory syndrome (IRIS) which can be lethal. But, how long to delay HIV treatment is less understood. In 2011, based on the conclusions of three separate studies, WHO recommended starting HIV treatment earlier for those with later HIV disease progression. This paper conducts a replication study of one of the three studies, by Havlir and colleagues. Using their publicly available data, we were able to replicate most of the results presented in the original paper. In our measurement and estimation analyses we use different estimation techniques to assess the robustness of the results. We find that adjusting for loss to follow-up does not affect the main results of the paper. However, an ANCOVA estimation and an instrumental variable model weaken the main result of the paper of better outcomes with early HIV treatment only for those who are sicker, reducing significance from the 5% to the 10% level. A change-point analysis also detects no changes in effect by timing of HIV treatment initiation or different thresholds of CD4 count for the primary outcome. This result suggests that the choice of start time for HIV treatment initiation should be based on other factors including potential drug interactions, overlapping side effects, a high pill burden and severity of illness rather than CD4 threshold and preset timeframes. While we caution against overgeneralizing, the result of this replication is aligned with more recent studies that show no evidence that early initiation of HIV treatment reduces mortality for any patients.

Quasi-experimental study designs series-paper 7: assessing the assumptions.

Quasi-experimental designs are gaining popularity in epidemiology and health systems research-in particular for the evaluation of health care practice, programs, and policy-because they allow strong causal inferences without randomized controlled experiments. We describe the concepts underlying five important quasi-experimental designs: Instrumental Variables, Regression Discontinuity, Interrupted Time Series, Fixed Effects, and Difference-in-Differences designs. We illustrate each of the designs with an example from health research. We then describe the assumptions required for each of the designs to ensure valid causal inference and discuss the tests available to examine the assumptions.

Quasi-experimental study designs series-paper 6: risk of bias assessment.

OBJECTIVES: Rigorous and transparent bias assessment is a core component of high-quality systematic reviews. We assess modifications to existing risk of bias approaches to incorporate rigorous quasi-experimental approaches with selection on unobservables. These are nonrandomized studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments, and regression-discontinuity designs. STUDY DESIGN AND SETTING: We review existing risk of bias tools. Drawing on these tools, we present domains of bias and suggest directions for evaluation questions. RESULTS: The review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of these designs. The paper then presents an approach to evaluating the internal validity of quasi-experiments with selection on unobservables. CONCLUSION: We conclude that tools for nonrandomized studies of interventions need to be further developed to incorporate evaluation questions for quasi-experiments with selection on unobservables.

Strength of Evidence on Demand Creation for Voluntary Medical Male Circumcision From 7 Impact Evaluations in Southern and Eastern Africa.

BACKGROUND: Seven new impact evaluations of pilot programs for increasing the demand for voluntary medical male circumcision (VMMC) provide evidence of what works and what does not. The study findings suggest that financial compensation designed to relieve the opportunity or transportation costs from undergoing the procedure can increase the uptake of VMMC. There is also evidence that programs using peer influence can be effective, although so far only sports-based programs demonstrate a strong effect. We explore the strength of evidence in each of these 7 studies to better interpret the findings for policy making. METHODS: We perform a risk of bias assessment and conduct power calculations using actual values for each of the 7 studies. RESULTS: Three of the 7 studies have a medium risk of bias, whereas the other 4 have a low risk of bias. All but 2 of the studies have adequate power to detect meaningful effects. In the 2 with insufficient power, the estimated effects are large but statistically insignificant. CONCLUSION: The positive evidence that financial incentives presented as compensation for opportunity costs to men seeking and obtaining VMMC can increase uptake comes from strong studies, which have high power and low to medium risk of bias. The positive evidence that a comprehensive sports-based program for young men can increase uptake also comes from a strong study. The strength of the studies further validates these findings.

A review of the evidence of harm from self-tests.

Although HIV self-testing may overcome some barriers to HIV testing, various stakeholders have expressed concerns that HIV self-testing may lead to unintended harm, including psychological, social and medical harm. Recognizing that similar concerns were raised in the past for some other self-tests, we conduct a review of the literature on a set of self-tests that share some characteristics with HIV self-tests to determine whether there is any evidence of harm. We find that although the potential for harm is discussed in the literature on self-tests, there is very little evidence that such harm occurs.

The impact of social action funds on child health in a conflict affected country: evidence from Angola.

Although recent evidence shows significant and long-lasting detrimental effects of armed conflict on child health, there is lack of studies rigorously assessing the effectiveness of different social and economic development interventions aiming to mitigate the impact of armed conflict on child health. In order to fill this knowledge gap, this study assesses the impact of health projects and water, sanitation, and waste management interventions financed by the Angola Social Action Fund (ASAF) from 1994 to 2001 on child health. I use data from Inquérito aos Agregados Familiares sobre Despesas e Receitas 2000/2001(IDR 2001), a household survey on expenditures and incomes conducted between February 2000 and February 2001 in Angola. IDR 2001 uses a stratified sampling design in which 12 households were surveyed in a random fashion in each aldeia (village) in rural areas and bairro (neighborhood) in urban areas. Using propensity score matching, a fixed effects model, and propensity-based weighted regression, I find that ASAF leads to a statistically significant increase of the height-for-age Z-scores (HAZ) by 0.335 standard deviations of children less than 5 years. This finding is robust to different implementations of the propensity score model specification and when conducting the sensitivity analysis of hidden bias. The main result that emerges from an analysis of heterogeneous effects shows that ASAF has no impact on children living in war displaced households. Despite many challenges faced by conflict affected countries, social funds which are one the key instruments of the World Bank used to promote development at the local level can be used to mitigate the impact of armed conflict on child health. For children living in war displaced households, specific interventions should be designed to mitigate the impact of armed conflict.