How do we design and evaluate health system strengthening? Collaborative development of a set of health system process goals.

Strong health systems are widely recognized as a key requirement for improving health outcomes and also for ensuring that health systems are equitable, resilient and responsive to population needs. However, the related term Health Systems Strengthening (HSS) remains unclear and contested, and this creates challenges for how HSS can be monitored and evaluated. A previous review argued for the need to rethink evaluation methods for HSS to examine systemic effects of HSS investments. In line with that recommendation, this article describes the work of the HSS Evaluation Collaborative (HSSEC) in the development of a framework and tool to guide HSS monitoring, evaluation and learning by national and global actors. It was developed based on a rapid review of the literature and iterative expert consultation, with the aim of going beyond a focus on the building blocks of health systems and on health system outputs or health outcomes to think about the features that constitute a strong health system. As a result, we developed a list of 22 health system process goals which represent desirable attributes for health systems. The health system process goals (or rather, progress towards them) are influenced by positive and negative, intended and unintended effects of HSS interventions. Finally, we illustrate how the health system process goals can be operationalised for prospective and retrospective HSS monitoring, evaluation and learning, and how they also have the potential to be used for opening a space for participatory, inclusive policy dialogue about HSS.

Health system strengthening-Reflections on its meaning, assessment, and our state of knowledge.

Comprehensive reviews of health system strengthening (HSS) interventions are rare, partly because of lack of clarity on definitions of the term but also the potentially huge scale of the evidence. We reflect on the process of undertaking such an evidence review recently, drawing out suggestions on definitions of HSS and approaches to assessment, as well as summarising some key conclusions from the current evidence base. The key elements of a clear definition include, in our view, consideration of scope (with effects cutting across building blocks in practice, even if not in intervention design, and also tackling more than one disease), scale (having national reach and cutting across levels of the system), sustainability (effects being sustained over time and addressing systemic blockages), and effects (impacting on health outcomes, equity, financial risk protection, and responsiveness). We also argue that agreeing a framework for design and evaluation of HSS is urgent. Most HSS interventions have theories of change relating to specific system blocks, but more work is needed on capturing their spillover effects and their contribution to meeting overarching health system process goals. We make some initial suggestions about such goals, to reflect the features that characterise a "strong health system." We highlight that current findings on "what works" are just indicative, given the limitations and biases in what has been studied and how, and argue that there is need to rethink evaluation methods for HSS beyond finite interventions and narrow outcomes. Clearer concepts, frameworks, and methods can support more coherent HSS investment.

Health worker perspectives on user fee removal in Zambia.

BACKGROUND: User fees for primary care services were removed in rural districts in Zambia in 2006. Experience from other countries has suggested that health workers play a key role in determining the success of a fee removal policy, but also find the implementation of such a policy challenging. The policy was introduced against a backdrop of a major shortage in qualified health staff. METHODS: As part of a larger study on the experience and effect of user fee removal in Zambia, a number of case studies at the facility level were conducted. As part of these, quantitative and qualitative data were collected to evaluate health workers' satisfaction and experiences in charging and non-charging facilities. RESULTS: Our findings show that health-care workers have mixed feelings about the policy change and its consequences. We found some evidence that personnel motivation was higher in non-charging facilities compared to facilities still charging. Yet it is unclear whether this effect was due to differences in the user fee policy or to the fact that a lot of staff interviewed in non-charging facilities were working in mission facilities, where we found a significantly higher motivation. Health workers expressed satisfaction with an apparent increase in the number of patients visiting the facilities and the removal of a deterring factor for many needy patients, but also complained about an increased workload. Furthermore, working conditions were said to have worsened, which staff felt was linked to the absence of additional resources to deal with the increased demand or replace the loss of revenue generated by fees. CONCLUSION: These findings highlight the need to pay attention to supply-side measures when removing demand-side barriers such as user fees and in particular to be concerned about the burden that increased demand can place on already over-stretched health workers.

Why performance-based contracting failed in Uganda--an "open-box" evaluation of a complex health system intervention.

Performance-based contracting (PBC) is a tool that links rewards to attainment of measurable performance targets. Significant problems remain in the methods used to evaluate this tool. The primary focus of evaluations on the effects of PBC (black-box) and less attention to how these effects arise (open-box) generates suboptimal policy learning. A black-box impact evaluation of PBC pilot by the Development Research Group of the World Bank (DRG) and the Ministry of Health (MOH) concluded that PBC was ineffective. This paper reports a theory-based case study intended to clarify how and why PBC failed to achieve its objectives. To explain the observed PBC implementation and responses of participants, this case study employed two related theories i.e. complex adaptive system and expectancy theory respectively. A prospective study trailed the implementation of PBC (2003-2006) while collecting experiences of participants at district and hospital levels. Significant problems were encountered in the implementation of PBC that reflected its inadequate design. As problems were encountered, hasty adaptations resulted in a de facto intervention distinct from the one implied at the design stage. For example, inadequate time was allowed for the selection of service targets by the health centres yet they got 'locked-in' to these poor choices. The learning curve and workload among performance auditors weakened the validity of audit results. Above all, financial shortfalls led to delays, short-cuts and uncertainty about the size and payment of bonuses. The lesson for those intending to implement similar interventions is that PBC should not be attempted 'on the cheap'. It requires a plan to boost local institutional and technical capacities of implementers. It also requires careful consideration of the responses of multiple actors - both insiders and outsiders to the intended change process. Given the costs and complexity of PBC implementation, strengthening conventional approaches that are better attuned to low income contexts (financing resource inputs and systems management) remains a viable policy option towards improving health service delivery.

Assessing the effects of removing user fees in Zambia and Niger.

OBJECTIVES: This study aims to overcome some of the limitations of previous studies investigating the effects of fee removal, by looking at heterogeneity of effects within countries and over time, as well as the existence of spill-over effects on groups not targeted by the policy change. METHODS: Using routine district health services data before and after recent abolitions of user charges in Zambia and Niger, we examine the effects of the policy change on the use of health services by different groups and over time, using an interrupted timeseries design. RESULTS: Removing user fees for primary health care services in rural districts in Zambia and for children over five years old in Niger increased use of services by the targeted groups. The impact of the policy change differed widely across districts, ranging from +12% and +194% in Niger to -39% and +108% in Zambia. Eighteen months after the policy change, some of these effects had been eroded. There was evidence that abolishing user fees can both have positive and negative spillover effects. CONCLUSION: These results highlight the importance of paying attention to implementation challenges and monitoring the effects of policy reforms which are often more mixed and complicated that they appear. The comparison of these reforms in two countries also sheds light on the potentially different ways in which free care can be used as a tool to improve access.

Constraints to implementing the Essential Health Package in Malawi.

Increasingly seen as a useful tool of health policy, Essential or Minimal Health Packages direct resources to interventions that aim to address the local burden of disease and be cost-effective. Less attention has been paid to the delivery mechanisms for such interventions. This study aimed to assess the degree to which the Essential Health Package (EHP) in Malawi was available to its population and what health system constraints impeded its full implementation. The first phase of this study comprised a survey of all facilities in three districts including interviews with all managers and clinical staff. In the second and third phase, results were discussed with District Health Management Teams and national level stakeholders, respectively, including representatives of the Ministry of Health, Central Medical Stores, donors and NGOs. The EHP in Malawi is focussing on the local burden of disease; however, key constraints to its successful implementation included a widespread shortage of staff due to vacancies but also caused by frequent trainings and meetings (only 48% of expected man days of clinical staff were available; training and meetings represented 57% of all absences in health centres). Despite the training, the percentage of health workers aware of vital diagnostic and therapeutic approaches to EHP conditions was weak. Another major constraint was shortages of vital drugs at all levels of facilities (e.g. Cotrimoxazole was sufficiently available to treat the average number of patients in only 27% of health centres). Although a few health workers noted some improvement in infrastructure and working conditions, they still considered them to be widely inadequate. In Malawi, as in similar resource poor countries, greater attention needs to be given to the health system constraints to delivering health care. Removal of these constraints should receive priority over the considerable focus on the development and implementation of essential packages of interventions.

The impact of user fees on access to health services in low- and middle-income countries.

BACKGROUND: Following an international push for financing reforms, many low- and middle-income countries introduced user fees to raise additional revenue for health systems. User fees are charges levied at the point of use and are supposed to help reduce 'frivolous' consumption of health services, increase quality of services available and, as a result, increase utilisation of services. OBJECTIVES: To assess the effectiveness of introducing, removing or changing user fees to improve access to care in low-and middle-income countries SEARCH STRATEGY: We searched 25 international databases, including the Cochrane Effective Practice and Organisation of Care (EPOC) Group's Trials Register, CENTRAL, MEDLINE and EMBASE. We also searched the websites and online resources of international agencies, organisations and universities to find relevant grey literature. We conducted the original searches between November 2005 and April 2006 and the updated search in CENTRAL (DVD-ROM 2011, Issue 1); MEDLINE In-Process & Other Non-Indexed Citations, Ovid (January 25, 2011); MEDLINE, Ovid (1948 to January Week 2 2011); EMBASE, Ovid (1980 to 2011 Week 03) and EconLit, CSA Illumina (1969 - present) on the 26th of January 2011. SELECTION CRITERIA: We included randomised controlled trials, interrupted time-series studies and controlled before-and-after studies that reported an objective measure of at least one of the following outcomes: healthcare utilisation, health expenditures, or health outcomes. DATA COLLECTION AND ANALYSIS: We re-analysed studies with longitudinal data. We computed price elasticities of demand for health services in controlled before-and-after studies as a standardised measure. Due to the diversity of contexts and outcome measures, we did not perform meta-analysis. Instead, we undertook a narrative summary of evidence. MAIN RESULTS: We included 16 studies out of the 243 identified. Most of the included studies showed methodological weaknesses that hamper the strength and reliability of their findings. When fees were introduced or increased, we found the use of health services decreased significantly in most studies. Two studies found increases in health service use when quality improvements were introduced at the same time as user fees. However, these studies have a high risk of bias. We found no evidence of effects on health outcomes or health expenditure. AUTHORS' CONCLUSIONS: The review suggests that reducing or removing user fees increases the utilisation of certain healthcare services. However, emerging evidence suggests that such a change may have unintended consequences on utilisation of preventive services and service quality. The review also found that introducing or increasing fees can have a negative impact on health services utilisation, although some evidence suggests that when implemented with quality improvements these interventions could be beneficial. Most of the included studies suffered from important methodological weaknesses. More rigorous research is needed to inform debates on the desirability and effects of user fees.

The impact of contracting out on health outcomes and use of health services in low and middle-income countries.

BACKGROUND: Recent literature on the lack of efficiency and acceptability of publicly provided health services has led to an interest in the use of partnerships with the private sector to deliver public services. OBJECTIVES: To assess the effectiveness of contracting out healthcare services in improving access to care in low and middle-income countries and, where possible, health outcomes. SEARCH STRATEGY: We searched a wide range of international databases, including the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE and EMBASE, in addition to development studies and economic databases. We also searched the websites and online resources of numerous international agencies, organisations and universities to find relevant grey literature. The original searches were conducted between November 2005 and April 2006. An updated search in MEDLINE was carried out in May 2009. SELECTION CRITERIA: Contracting out health services is defined as the provision of healthcare services on behalf of the government by non-state providers. Studies had to include an objective measure of at least one of the following outcomes: health care utilisation, health expenditure, health outcomes or equity outcomes. Studies also needed to use one of the following study designs: randomised controlled trial, non-randomised controlled trial, interrupted time series analysis or controlled before and after study. DATA COLLECTION AND ANALYSIS: We made an attempt to present results from the different studies in a systematic way, however due to the diversity of sources, contexts and methods used, we undertook a narrative synthesis. MAIN RESULTS: Three studies met our inclusion criteria (one after re-analysis of data). These studies suggest that contracting out services to non-state providers can increase access and utilisation of health services. One study found a reduction in out-of-pocket expenditures and improvement in some health outcomes. However, methodological weaknesses and particularities of the reported programme settings limit the strength and generalisability of their conclusions. AUTHORS' CONCLUSIONS: Three studies suggest that contracting out may be an appropriate response to scale up service delivery in particular settings, such as post-conflict or fragile states. Evidence was not presented on whether this approach was more effective than making a similar investment in the public sector, as there was not an exact control available in any of the settings. In addition, the introduction of non-state providers into some settings and not others also brings many potentially confounding variables, such as the presence of additional management expertise or expatriate doctors, which may improve drug supply or increase utilisation.

The impact of conditional cash transfers on health outcomes and use of health services in low and middle income countries.

BACKGROUND: Conditional cash transfers (CCT) provide monetary transfers to households on the condition that they comply with some pre-defined requirements. CCT programmes have been justified on the grounds that demand-side subsidies are necessary to address inequities in access to health and social services for poor people. In the past decade they have become increasingly popular, particularly in middle income countries in Latin America. OBJECTIVES: To assess the effectiveness of CCT in improving access to care and health outcomes, in particular for poorer populations in low and middle income countries. SEARCH STRATEGY: We searched a wide range of international databases, including the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE and EMBASE, in addition to development studies and economic databases. We also searched the websites and online resources of numerous international agencies, organisations and universities to find relevant grey literature. The original searches were conducted between November 2005 and April 2006. An updated search in MEDLINE was carried out in May 2009. SELECTION CRITERIA: CCT were defined as monetary transfers made to households on the condition that they comply with some pre-determined requirements in relation to health care. Studies had to include an objective measure of at least one of the following outcomes: health care utilisation, health expenditure, health outcomes or equity outcomes. Eligible study designs were: randomised controlled trial, interrupted time series analysis, or controlled before-after study of the impact of health financing policies following criteria used by the Cochrane Effective Practice and Organisation of Care Group. DATA COLLECTION AND ANALYSIS: We performed qualitative analysis of the evidence. MAIN RESULTS: We included ten papers reporting results from six intervention studies. Overall, design quality and analysis limited the risks of bias. Several CCT programmes provided strong evidence of a positive impact on the use of health services, nutritional status and health outcomes, respectively assessed by anthropometric measurements and self-reported episodes of illness. It is hard to attribute these positive effects to the cash incentives specifically because other components may also contribute. Several studies provide evidence of positive impacts on the uptake of preventive services by children and pregnant women. We found no evidence about effects on health care expenditure. AUTHORS' CONCLUSIONS: Conditional cash transfer programmes have been the subject of some well-designed evaluations, which strongly suggest that they could be an effective approach to improving access to preventive services. Their replicability under different conditions - particularly in more deprived settings - is still unclear because they depend on effective primary health care and mechanisms to disburse payments. Further rigorous evaluative research is needed, particularly where CCTs are being introduced in low income countries, for example in Sub-Saharan Africa or South Asia.

Availability of essential medicines in Ethiopia: an efficiency-equity trade-off?

OBJECTIVE: To investigate the availability and cost of essential medicines in health centres in rural Ethiopia, and to explore if the fee waiver system protects patients from having to pay for medicines. METHODS: The study took place in five health centres in rural Ethiopia. Availability and price of selected key essential medicines was established in the budget and special pharmacy of the health centre, as well as private outlets. Information on availability and cost of prescribed drugs was obtained through patient exit-interviews. RESULTS: Availability based of essential drugs at facility level was 91% based on a list of selected drugs vs. 84% based on prescriptions filled. However, less than half the prescribed drugs were obtained from the budget pharmacy, and one in six patients was forced to purchase drugs in the private sector, where drugs are roughly twice as expensive. The waiver system did not safeguard against having to pay for medicines. CONCLUSION: A revolving drug fund system in Ethiopia seems to improve availability of medicines, and can improve affordability by protecting people from purchasing drugs in the private sector. However, it may result in a parallel system, whereby the poor cannot access drugs if these are not available in the budget pharmacy. Equity is a concern in the absence of an adequate mechanism to protect the poor from catastrophic health expenditure.

Performance-based payment: some reflections on the discourse, evidence and unanswered questions.

Performance-based payment (PBP) is increasingly advocated as a way to improve the performance of health systems in low-income countries. This study conducted a systematic review of the current literature on this topic and found that while it is a popular term, there was little consensus about the meaning or the use of the concept of PBP. Significant weaknesses in the current evidence base on the success of PBP initiatives were also found. The literature would be strengthened by multi-disciplinary case studies that present both the advantages and disadvantages of PBP, influential factors for success, and more details about the projects from which this evidence is drawn. Where possible, data from control facilities where PBP is not being implemented would be an important addition. This paper suggests a further agenda for research, including assessing optimal conditions for implementation of PBP schemes in less developed health systems, the impact of adopting measures of performance as targets, and the requirements for monitoring PBP adequately.

Changing roles and responses of health care workers in HIV treatment and care.

A key limiting factor in the scale up and sustainability of HIV care and treatment programmes is the global shortage of trained health care workers. This paper discusses why it is important to move beyond conceptualising health care workers simply as 'inputs' in the delivery of HIV treatment and care, and to also consider their roles as partners and agents in the process of health care. It suggests a framework for thinking about their roles and responses in HIV care, considers the current evidence base, and concludes by identifying key areas for future research on health care workers' responses in HIV treatment and care in low and middle income settings.

The impact of user fees on health service utilization in low- and middle-income countries: how strong is the evidence?

OBJECTIVE: To assess the effects of user charges on the uptake of health services in low- and middle-income countries. METHODS: A systematic search of 25 social science, economics and health literature databases and other sources was performed to identify and appraise studies on the effects of introducing, removing, increasing or reducing user charges on the uptake of various health services in low- and middle-income countries. Only experimental or quasi-experimental study designs were considered: cluster randomized controlled trials (C-RCT), controlled " before and after" (CBA) studies and interrupted time series (ITS) studies. Papers were assessed in which the effect of the intervention was measured in terms of changes in service utilization (including equity outcomes), household expenditure or health outcomes. FINDINGS: Sixteen studies were included: five CBA, two C-RCT and nine ITS. Only studies reporting effects on health service utilization, sometimes across socioeconomic groups, were identified. Removing or reducing user fees was found to increase the utilization of curative services and perhaps preventive services as well, but may have negatively impacted service quality. Introducing or increasing fees reduced the utilization of some curative services, although quality improvements may have helped maintain utilization in some cases. When fees were either introduced or removed, the impact was immediate and abrupt. Studies did not adequately show whether such an increase or reduction in utilization was sustained over the longer term. In addition, most of the studies were given low-quality ratings based on criteria adapted from those of the Cochrane Collaboration's Effective Practice and Organisation of Care group. CONCLUSIONS: There is a need for more high-quality research examining the effects of changes in user fees for health services in low- and middle-income countries.

Conditional cash transfers for improving uptake of health interventions in low- and middle-income countries: a systematic review.

CONTEXT: Cash transfers conditional on certain behaviors, intended to provide access to social services, have been introduced in several developing countries. The effectiveness of these strategies in different contexts has not previously been the subject of a systematic review. OBJECTIVE: To assess the effectiveness of conditional monetary transfers in improving access to and use of health services, as well as improving health outcomes, in low- and middle-income countries. DATA SOURCES: Relevant publications were identified via electronic medical and social science databases from inception to April 2006 (PubMED, EMBASE, POPLINE, CAB Direct, Healthcare Management Information Consortium, WHOLIS (World Health Organization Library Database), African Healthline, International Bibliography of the Social Sciences (IBSS), Eldis, British Library for Development Studies (BLDS), ID21, Journal Storage (Jstor), Inter-Science, ScienceDirect, Internet Documents in Economics Access Service (Research Papers in Economics) (IDEAS[Repec]), Latin American and Caribbean Health Sciences Literature (LILACS), MEDCARIB, Virtual Library in Health (ADOLEC), Pan American Health Organization (PAHO), FRANCIS, The Cochrane Central Register of Controlled Trials, the Database of Abstracts of Reviews of Effectiveness, and the Effective Practice and Organization of Care Group (EPOC) Register. Reference lists of relevant papers and "gray" literature resources were also searched. STUDY SELECTION: To be included, a paper had to meet study design criteria (randomized controlled trial, interrupted time series analysis, and controlled before and after study) and include a measure of at least 1 of the following outcomes: health care utilization, health expenditure, or health outcomes. Twenty-eight papers were retrieved for assessment and 10 were included in this review. DATA EXTRACTION: Methodological details and outcomes were extracted by 2 reviewers who independently assessed the quality of the papers. RESULTS: Overall, the evidence suggests that conditional cash transfer programs are effective in increasing the use of preventive services and sometimes improving health status. CONCLUSIONS: Further research is needed to clarify the cost effectiveness of conditional cash transfer programs and better understand which components play a critical role. The potential success and desirability of such programs in low-income settings, with more limited health system capacity, also deserves more investigation.

Trust and health worker performance: exploring a conceptual framework using South African evidence.

Two relationships of particular importance to health care provision are those between patient and provider, and health worker and employer. This paper presents an analytical framework that establishes the key dimensions of trust within these relationships, and suggests how they may combine in influencing health system responsiveness. The paper then explores the relevance of the framework by using it to analyse case studies of primary care providers in South Africa. The analysis suggests that respectful treatment is the central demand of primary care service users, in terms of positive attitudes/behaviours, thoroughness, and technical competence, as well as institutions that support fair treatment. It is argued that such treatment is necessary for, and integral to, patient-provider trust. The findings also suggest that the notion of workplace trust (combining trust in colleagues, supervisor and employing organisation) has relevance to provider experiences of their workplaces, and so can provide important insights for strengthening management. Nonetheless, given the limitations of this preliminary analysis, further research is needed to develop the notion of workplace trust and to test what role it has, along with that of provider-community relations, in influencing health worker performance.

Contracts in the real world: case studies from Southern Africa.

The desirability of using the private sector to deliver public services is widely debated internationally. Understanding the nature of contracts that initiate and govern such public-private partnerships, and the extent to which they can define the performance of private providers, is key in addressing the questions that underlie this debate. Such understanding has to be gained through better knowledge of all the influences upon contractual relationships. Environmental and institutional factors have been highlighted as one set of influences in need of more attention. This paper presents case studies of three contracts for primary care services in Southern Africa. It reports aspects of the institutional and environmental context in which they operate, and reflects on the nature of publicly financed primary care as a service to be contracted out. An urban-based private sector contract for a sub-set of primary care services was found to operate very differently from rural-based public sector contracts, which attempted to provide broader coverage. The latter contracts were more loosely defined and operated in a more relational manner. Important environmental influences on incomplete contractual relationships explored here are the nature of the market, scope of services, management capacity and involvement of a public purchaser. The paper illustrates some of the practical challenges for low- and middle-income countries in pursuing a policy of contracting with private providers for public primary care services, and particularly highlights the difficulties of deciding how to divide up responsibility between the public and private sectors and yet maintain a comprehensive service delivery system.

Health financing to promote access in low income settings-how much do we know?

In this article we outline research since 1995 on the impact of various financing strategies on access to health services or health outcomes in low income countries. The limited evidence available suggests, in general, that user fees deterred utilisation. Prepayment or insurance schemes offered potential for improving access, but are very limited in scope. Conditional cash payments showed promise for improving uptake of interventions, but could also create a perverse incentive. The largely African origin of the reports of user fees, and the evidence from Latin America on conditional cash transfers, demonstrate the importance of the context in which studies are done. There is a need for improved quality of research in this area. Larger scale, upfront funding for evaluation of health financing initiatives is necessary to ensure an evidence base that corresponds to the importance of this issue for achieving development goals.