Predictors of health worker performance after Integrated Management of Childhood Illness training in Benin: a cohort study.

BACKGROUND: Correct treatment of potentially life-threatening illnesses (PLTIs) in children under 5 years, such as malaria, pneumonia, and diarrhea, can substantially reduce mortality. The Integrated Management of Childhood Illness (IMCI) strategy has been shown to improve treatment of child illnesses, but multiple studies have shown that gaps in health worker performance remain after training. To better understand factors related to health worker performance, we analyzed 9,330 patient consultations in Benin from 2001-2002, after training one of the first cohorts of 32 health workers in IMCI. METHODS: With data abstracted from patient registers specially designed for IMCI-trained health workers, we examined associations between health facility-, health worker-, and patient-level factors and 10 case-management outcomes for PLTIs. RESULTS: Altogether, 63.6% of children received treatment for all their PLTIs in accordance with IMCI guidelines, and 77.8% received life-saving treatment (i.e., clinically effective treatment, even if not exactly in accordance with IMCI guidelines). Performance of individual health workers varied greatly, from 15-88% of patients treated correctly, on average. Multivariate regression analyses identified several factors that might have influenced case-management quality, many outside a manager's direct control. Younger health workers significantly outperformed older ones, and infants received better care than older children. Children with danger signs, those with more complex illnesses, and those with anemia received worse care. Health worker supervision was associated with improved performance for some outcomes. CONCLUSIONS: A variety of factors, some outside the direct control of program managers, can influence health worker practices. An understanding of these influences can help inform the development of strategies to improve performance.

Evaluating health worker performance in Benin using the simulated client method with real children.

BACKGROUND: The simulated client (SC) method for evaluating health worker performance utilizes surveyors who pose as patients to make surreptitious observations during consultations. Compared to conspicuous observation (CO) by surveyors, which is commonly done in developing countries, SC data better reflect usual health worker practices. This information is important because CO can cause performance to be better than usual. Despite this advantage of SCs, the method's full potential has not been realized for evaluating performance for pediatric illnesses because real children have not been utilized as SCs. Previous SC studies used scenarios of ill children that were not actually brought to health workers. During a trial that evaluated a quality improvement intervention in Benin (the Integrated Management of Childhood Illness [IMCI] strategy), we conducted an SC survey with adult caretakers as surveyors and real children to evaluate the feasibility of this approach and used the results to assess the validity of CO. METHODS: We conducted an SC survey and a CO survey (one right after the other) of health workers in the same 55 health facilities. A detailed description of the SC survey process was produced. Results of the two surveys were compared for 27 performance indicators using logistic regression modeling. RESULTS: SC and CO surveyors observed 54 and 185 consultations, respectively. No serious problems occurred during the SC survey. Performance levels measured by CO were moderately higher than those measured by SCs (median CO - SC difference = 16.4 percentage-points). Survey differences were sometimes much greater for IMCI-trained health workers (median difference = 29.7 percentage-points) than for workers without IMCI training (median difference = 3.1 percentage-points). CONCLUSION: SC surveys can be done safely with real children if appropriate precautions are taken. CO can introduce moderately large positive biases, and these biases might be greater for health workers exposed to quality improvement interventions. TRIAL NUMBER: http://clinicaltrials.gov Identifier NCT00510679.

Trends in health worker performance after implementing the Integrated Management of Childhood Illness strategy in Benin.

OBJECTIVE: Training health workers to use Integrated Management of Childhood Illness (IMCI) guidelines can improve care for ill children in outpatient settings in developing countries. However, even after IMCI training, important performance gaps exist. One potential reason is that the effect of training can rapidly wane. Our aim was to determine if the performance of IMCI-trained health workers deteriorated over 3 years. METHODS: We studied two departments in Benin. First, we performed a record review of 32 IMCI-trained health workers during the first year of IMCI implementation (2001-2002). Second, we analysed data from cross-sectional health facility surveys from 2001 to 2004 that represented the entire study area. Primary outcomes were the proportion of children under 5 years old with potentially life-threatening illnesses who received either recommended or adequate treatment, and among all children, an index of overall guideline adherence. Secondary outcomes reflected the treatment of individual diseases. Outcomes were calculated monthly, and time trends were evaluated with regression modelling. RESULTS: The record review included 9393 consultations, and the surveys included 411 consultations performed by 105 health workers. For both data sources, performance trends were essentially flat for nearly all outcomes. Absolute levels of performance revealed substantial performance gaps. CONCLUSIONS: We found no evidence that performance declined over 3 years after IMCI training. However, important performance gaps found immediately after IMCI training persisted and should be addressed.

Adoption of rapid diagnostic tests for the diagnosis of malaria, a preliminary analysis of the Global Fund program data, 2005 to 2010.

INTRODUCTION: The World Health Organization Guidelines for the Treatment of Malaria, in 2006 and 2010, recommend parasitological confirmation of malaria before commencing treatment. Although microscopy has been the mainstay of malaria diagnostics, the magnitude of diagnostic scale up required to follow the Guidelines suggests that rapid diagnostic tests (RDTs) will be a large component. This study analyzes the adoption of rapid diagnostic testing in malaria programs supported by the Global Fund to fight AIDS, Tuberculosis and Malaria (Global Fund), the leading international funder of malaria control globally. METHODS AND FINDINGS: We analyzed, for the period 2005 to 2010, Global Fund programmatic data for 81 countries on the quantity of RDTs planned; actual quantities of RDTs and artemisinin-based combination treatments (ACTs) procured in 2009 and 2010; RDT-related activities including RDTs distributed, RDTs used, total diagnostic tests including RDTs and microscopy performed, health facilities equipped with RDTs; personnel trained to perform rapid diagnostic malaria test; and grant budgets allocated to malaria diagnosis. In 2010, diagnosis accounted for 5.2% of malaria grant budget. From 2005 to 2010, the procurement plans include148 million RDTs through 96 malaria grants in 81 countries. Around 115 million parasitological tests, including RDTs, had reportedly been performed from 2005 to 2010. Over this period, 123,132 health facilities were equipped with RDTs and 137,140 health personnel had been trained to perform RDT examinations. In 2009 and 2010, 41 million RDTs and 136 million ACTs were purchased. The ratio of procured RDTs to ACTs was 0.26 in 2009 and 0.34 in 2010. CONCLUSIONS/SIGNIFICANCE: Global Fund financing has enabled 81 malaria-endemic countries to adopt WHO guidelines by investing in RDTs for malaria diagnosis, thereby helping improve case management of acute febrile illness in children. However, roll-out of parasitological diagnosis lags behind the roll-out of ACT-based treatment, and will require prioritization of investments.

Indicators measuring the performance of malaria programs supported by the global fund in Asia, progress and the way forward.

INTRODUCTION: In 2010, the Global Fund provided more than 75% of external international financing for malaria control. The Global Fund uses performance based funding in the grants it finances. This paper analyses the indicators used to measure the performance of Global Fund supported malaria grants in Asia. METHODS: Indicators used in the performance frameworks for all Global Fund supported malaria grants in Asia were retrieved from grant database and grouped into impact, outcome, output and input categories and categorized by service delivery areas. Indicators of each group were compared over rounds. Indicators used in performance frameworks were compared with internationally adopted indicators included in the Monitoring and Evaluation Toolkit developed by the Global Fund and international technical agencies. RESULTS: Between 2002 and 2010, 1,434 indicators were included in the performance frameworks of the 48 malaria grants awarded in Asia, including 229 impact and 227 outcome indicators, 437 output and 541 input indicators, with an average of 29.9 indicators per grant. The proportion of impact and outcome indicators increased over rounds, with that of input indicators declining from 44.1% in Round 1 to 22.7% in Round 9. CONCLUSIONS: Input indicators, which have predominated the performance frameworks of the Global Fund supported malaria programs in Asia have declined between Rounds 1 and 9. However, increased alignment with internationally adopted indicators included in the Monitoring and Evaluation Toolkit is needed to improve the validity of reported results.

Impact of a malaria-control project in Benin that included the integrated management of childhood illness strategy.

OBJECTIVES: To estimate the impact of the Integrated Management of Childhood Illness (IMCI) strategy on early-childhood mortality, we evaluated a malaria-control project in Benin that implemented IMCI and promoted insecticide-treated nets (ITNs). METHODS: We conducted a before-and-after intervention study that included a nonrandomized comparison group. We used the preceding birth technique to measure early-childhood mortality (risk of dying before age 30 months), and we used health facility surveys and household surveys to measure process indicators. RESULTS: Most process indicators improved in the area covered by the intervention. Notably, because ITNs were also promoted in the comparison area children's ITN use increased by about 20 percentage points in both areas. Regarding early-childhood mortality, the trend from baseline (1999-2001) to follow-up (2002-2004) for the intervention area (13.0% decrease; P < .001) was 14.1% (P < .001) lower than was the trend for the comparison area (1.3% increase; P = .46). CONCLUSIONS: Mortality decreased in the intervention area after IMCI and ITN promotion. ITN use increased similarly in both study areas, so the mortality impact of ITNs in the 2 areas might have canceled each other out. Thus, the mortality reduction could have been primarily attributable to IMCI's effect on health care quality and care-seeking.

Rapid scale-up of long-lasting insecticide-treated bed nets through integration into the national immunization program during child health week in Togo, 2004.

In December 2004, Togo was the first country to conduct a nationwide free insecticide-treated net (ITN) distribution as part of its National Integrated Child Health Campaign. Community-based cross-sectional surveys were conducted one and nine months post-campaign as part of a multidisciplinary evaluation of the nationwide distribution of ITNs to children 9-59 months of age to evaluate ITN ownership, equity, and use. Our results demonstrated that at one month post-campaign, 93.1% of all eligible children received an ITN. Household ITN ownership and equity increased significantly post-campaign. Nine months post-campaign, 78.6% of households with a child eligible to participate in the campaign retained at least one campaign net. Use by eligible children was 43.5% at one month post-campaign (during the dry season) and 52.9% at nine months post-campaign (during the rainy season). Household ownership of at least one ITN increased from 8.0% pre-campaign to 62.5% one month post-campaign. Together, these findings demonstrate that in this setting, increased household ITN ownership, equity, and retention can be achieved on a national scale through free ITN distribution during an integrated campaign.

The rise and fall of supervision in a project designed to strengthen supervision of Integrated Management of Childhood Illness in Benin.

OBJECTIVE: In developing countries, supervision is a widely recognized strategy for improving health worker performance; and anecdotally, maintaining regular, high-quality supervision is difficult. However, remarkably little research has explored in depth why supervision is so challenging. METHODS: In the context of a trial to improve health worker adherence to Integrated Management of Childhood Illness (IMCI) guidelines and strengthen supervision in southeastern Benin, we used record reviews, focus group discussions, key informant interviews, and cross-sectional surveys to examine the supervision process. FINDINGS: Initially, little IMCI supervision occurred. The frequency increased substantially after implementing a series of workshops, but then deteriorated. Quantitative and qualitative data revealed obstacles to supervision at multiple levels of the health system. Based on supervisors' opinions, the main problems were: poor coordination; inadequate management skills and ineffective management teams; a lack of motivation; problems related to decentralization; health workers sometimes resisting IMCI implementation; and less priority given to IMCI supervision because of incentives for non-supervision activities, a lack of leadership, and an expectation of integrated supervision. To this list, based on our observations, we add: the increasing supervision workload, time required for non-supervision activities, project interventions not always being implemented as planned, and the loss of particularly effective supervisors. In terms of correctly completing steps of the supervision process, the quality of supervision was generally good. CONCLUSIONS: Managers should monitor supervision, understand the evolving influences on supervision, and use their resources and authority to both promote supervision and remove impediments to supervision. Support from leaders can be crucial, thus donors and politicians should help make supervision a true priority. As with front-line clinicians, supervisors are health workers who need support. We emphasize the importance of research to identify effective and affordable strategies for improving supervision frequency and quality. (ClinicalTrials.gov number NCT00510679).

Improving pneumonia case-management in Benin: a randomized trial of a multi-faceted intervention to support health worker adherence to Integrated Management of Childhood Illness guidelines.

BACKGROUND: Pneumonia is a leading cause of death among children under five years of age. The Integrated Management of Childhood Illness strategy can improve the quality of care for pneumonia and other common illnesses in developing countries, but adherence to these guidelines could be improved. We evaluated an intervention in Benin to support health worker adherence to the guidelines after training, focusing on pneumonia case management. METHODS: We conducted a randomized trial. After a health facility survey in 1999 to assess health care quality before Integrated Management of Childhood Illness training, health workers received training plus either study supports (job aids, non-financial incentives and supervision of workers and supervisors) or "usual" supports. Follow-up surveys were conducted in 2001, 2002 and 2004. Outcomes were indicators of health care quality for Integrated Management-defined pneumonia. Further analyses included a graphical pathway analysis and multivariable logistic regression modelling to identify factors influencing case-management quality. RESULTS: We observed 301 consultations of children with non-severe pneumonia that were performed by 128 health workers in 88 public and private health facilities. Although outcomes improved in both intervention and control groups, we found no statistically significant difference between groups. However, training proceeded slowly, and low-quality care from untrained health workers diluted intervention effects. Per-protocol analyses suggested that health workers with training plus study supports performed better than those with training plus usual supports (20.4 and 19.2 percentage-point improvements for recommended treatment [p=0.08] and "recommended or adequate" treatment [p=0.01], respectively). Both groups tended to perform better than untrained health workers. Analyses of treatment errors revealed that incomplete assessment and difficulties processing clinical findings led to missed pneumonia diagnoses, and missed diagnoses led to inadequate treatment. Increased supervision frequency was associated with better care (odds ratio for recommended treatment=2.1 [95% confidence interval: 1.13.9] per additional supervisory visit). CONCLUSION: Integrated Management of Childhood Illness training was useful, but insufficient, to achieve high-quality pneumonia case management. Our study supports led to additional improvements, although large gaps in performance still remained. A simple graphical pathway analysis can identify specific, common errors that health workers make in the case-management process; this information could be used to target quality improvement activities, such as supervision (ClinicalTrials.gov number NCT00510679).

A multifaceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin.

OBJECTIVES: We evaluated an intervention to support health workers after training in Integrated Management of Childhood Illness (IMCI), a strategy that can improve outcomes for children in developing countries by encouraging workers' use of evidence-based guidelines for managing the leading causes of child mortality. METHODS: We conducted a randomized trial in Benin. We administered a survey in 1999 to assess health care quality before IMCI training. Health workers then received training plus either study supports (job aids, nonfinancial incentives, and supervision of workers and supervisors) or usual supports. Follow-up surveys conducted in 2001 to 2004 assessed recommended treatment, recommended or adequate treatment, and an index of overall guideline adherence. RESULTS: We analyzed 1244 consultations. Performance improved in both intervention and control groups, with no significant differences between groups. However, training proceeded slowly, and low-quality care from health workers without IMCI training diluted intervention effects. Per-protocol analyses revealed that workers with IMCI training plus study supports provided better care than did those with training plus usual supports (27.3 percentage-point difference for recommended treatment; P < .05), and both groups outperformed untrained workers. CONCLUSIONS: IMCI training was useful but insufficient. Relatively inexpensive supports can lead to additional improvements.

Insecticide-treated net ownership and usage in Niger after a nationwide integrated campaign.

OBJECTIVES: In December 2005 and March 2006, Niger conducted nationwide integrated campaigns to distribute polio vaccine and long lasting insecticide-treated nets (LLINs) to children <5 years of age. We evaluated the campaign effectiveness, net retention, insecticide-treated net (ITN) ownership, and usage. METHODS: Two nationwide cross-sectional surveys in January 2006 (dry season) and September 2006 (rainy season), using a stratified two-stage cluster sampling design. We mapped selected communities, selected households by simple random sampling, and administered questionnaires by interviewers using personal digital assistants. RESULTS: The first survey showed that ITN ownership in all households was 6.3% prior to the campaign, increasing to 65.1% after the campaign in the second survey. The second survey also showed that 73.4% of households with children <5 received an LLIN and that 97.7% of households that received > or = one LLIN retained it. The wealth equity ratio for ITN ownership in households with children <5 increased from 0.17 prior to the campaign to 0.79 afterward. During the dry season, 15.4% of all children <5 and 11.3% of pregnant women slept under an ITN, while during rainy season, 55.5% of children <5 and 48.2% of pregnant women slept under an ITN. CONCLUSIONS: Free distribution during the integrated campaign rapidly increased ITN ownership and decreased inequities between those in the highest and lowest wealth quintiles. Retention of ITNs was very high, and usage was high during malaria transmission season. However, ITN ownership and usage by vulnerable groups continues to fall short of RBM targets, and additional strategies are needed to increase ownership and usage.

Viewpoint: evaluating the impact of malaria control efforts on mortality in sub-Saharan Africa.

OBJECTIVE: To describe an approach for evaluating the impact of malaria control efforts on malaria-associated mortality in sub-Saharan Africa, where disease-specific mortality trends usually cannot be measured directly and most malaria deaths occur among young children. METHODS: Methods for evaluating changes in malaria-associated mortality are examined; advantages and disadvantages are presented. RESULTS: All methods require a plausibility argument-i.e., an assumption that mortality reductions can be attributed to programmatic efforts if improvements are found in steps of the causal pathway between intervention scale-up and mortality trends. As different methods provide complementary information, they can be used together. We recommend following trends in the coverage of malaria control interventions, other factors influencing childhood mortality, malaria-associated morbidity (especially anaemia), and all-cause childhood mortality. This approach reflects decreases in malaria's direct and indirect mortality burden and can be examined in nearly all countries. Adding other information can strengthen the plausibility argument: trends in indicators of malaria transmission, information from demographic surveillance systems and sentinel sites where malaria diagnostics are systematically used, and verbal autopsies linked to representative household surveys. Health facility data on malaria deaths have well-recognized limitations; however, in specific circumstances, they could produce reliable trends. Model-based predictions can help describe changes in malaria-specific burden and assist with program management and advocacy. CONCLUSIONS: Despite challenges, efforts to reduce malaria-associated mortality in Africa can be evaluated with trends in malaria intervention coverage and all-cause childhood mortality. Where there are resources and interest, complementary data on malaria morbidity and malaria-specific mortality could be added.

Care takers' recall of Integrated Management of Childhood Illness counselling messages in Benin.

A key goal of the Integrated Management of Childhood Illness (IMCI) strategy is to improve the management of childhood illness at health facilities. IMCIguidelines contain many counselling messages, and as it is not known how well caretakers recall these messages, we studied caretakers' recall of IMCI messages when given under ideal conditions. At a clinic in Benin, a study clinician performed counselling and confirmed caretakers'comprehension of all messages. Caretakers were randomly assigned to be interviewed either immediately after the consultation or a day later. Recall was assessed with general and focused open-ended questions. Recall was assessed for 55 caretakers, 29.1% of whom were literate. Caretakers received 3-75 messages (mean = 38.7). The mean percentage of messages recalled was 89.7% immediately after the consultation and 81.9% one day later. These results support IMCI's recommendation that health workers should verify caretakers' comprehension by asking caretakers to repeat counselling messages during consultations.

Risk and protective factors for two types of error in the treatment of children with fever at outpatient health facilities in Benin.

BACKGROUND: In developing countries, health workers often do not follow clinical practice guidelines. However, few studies have examined why different types of errors occur. METHODS: We analysed a sample of consultations of children with non-severe malaria (defined as fever without signs of severe illness) from a health facility survey conducted in Ouémé Département, Benin. Treatment was defined as correct (recommended antimalarial), a minor error (non-recommended antimalarial), or a major error (no antimalarial). RESULTS: In all, 85 health workers and 289 children were studied. In a multivariate logistic regression analysis, the following factors were significantly associated with major errors: treatment by a physician (adjusted odds ratio [aOR] = 13.57, 95% CI: 1.45-126.75), child's age <12 months (aOR = 3.41, 95% CI: 1.15-10.07), and child's temperature (aOR = 0.58 per degrees C, 95% CI: 0.34-0.97). Factors significantly associated with minor errors were: child's temperature (aOR = 1.43 per degrees C, 95% CI: 1.07-1.92), electricity at the health facility (aOR = 3.10, 95% CI: 1.05-9.17), >/=1 supervision visit in the past 6 months (aOR = 0.33, 95% CI: 0.14-0.77), fever treatment wall chart in the consultation room (aOR = 0.29, 95% CI: 0.12-0.73), and number of non-fever chief complaints (aOR = 0.67 per complaint, 95% CI: 0.48-0.93). In-service training in malaria treatment was not significantly associated with either error type. CONCLUSIONS: Many factors may influence health worker performance, and factors such as pre-service training may influence performance in unexpected ways. Identifying different errors and analysing them separately can help reveal potential causes that may be masked by combining errors into a single category.