Development and feasibility testing of an artificially intelligent chatbot to answer immunization-related queries of caregivers in Pakistan: A mixed-methods study.

BACKGROUND: Gaps in information access impede immunization uptake, especially in low-resource settings where cutting-edge and innovative digital interventions are limited given the digital inequity. Our objective was to develop an Artificially Intelligent (AI) chatbot to respond to caregiver's immunization-related queries in Pakistan and investigate its feasibility and acceptability in a low-resource, low-literacy setting. METHODS: We developed Bablibot (Babybot), a local language immunization chatbot, using Natural Language Processing (NLP) and Machine Learning (ML) technologies with Human in the Loop feature. We evaluated the bot through a sequential mixed-methods study. We enrolled caregivers visiting the 12 selected immunization centers for routine childhood vaccines. Additional caregivers were reached through targeted text message communication. We assessed Bablibot's feasibility and acceptability by tracking user engagement and technological metrics, and through thematic analysis of in-depth interviews with 20 caregivers. FINDINGS: Between March 9, 2020, and April 15, 2021, 2,202 caregivers were enrolled in the study, of which, 677 (30.7%) interacted with Bablibot (users). Bablibot responded to 1,877 messages through 874 conversations. Conversation topics included vaccination due dates (32.4%; 283/874), side-effect management (15.7%;137/874), or delaying vaccination due to child's illness or COVID-lockdown (16.8%;147/874). Over 90% (277/307) of responses to text-based exit surveys indicated satisfaction with Bablibot. Qualitative analysis showed caregivers appreciated Bablibot's usefulness and provided feedback for further improvement of the system. CONCLUSION: Our results demonstrate the feasibility and acceptability of local-language NLP chatbots in providing real-time immunization information in low-resource settings. Text-based chatbots canminimize the workload on helpline operators, in addition to instantaneously resolving caregiver queries that otherwise lead to delay or default.

Using geographic information system to track children and optimize immunization coverage and equity in Karachi, Pakistan.

BACKGROUND: Despite the potential of geospatial technologies to track and monitor coverage, they are underutilized for guiding immunization program strategy and implementation, especially in low-and-middle-income countries. We conducted geospatial analysis to explore the geographic and temporal trends of immunization coverage, and examined the pattern of immunization service access (outreach and facility based) by children. METHODOLOGY: We extracted data to analyze coverage rates across different dimensions (by enrolment year, birth year and vaccination year) from 2018 till 2020 in Karachi, Pakistan using the Sindh Electronic Immunization Registry (SEIR). We conducted geospatial analysis to assess variation in coverage rates of BCG, Pentavalent (Penta)-1, Penta-3, and Measles-1 vaccines using Government targets. We also analyzed the proportion of children receiving their routine vaccinations at fixed centers and outreach and examined whether children received vaccinations at the same or multiple immunization centers. RESULTS: A total of 1,298,555 children were born, enrolled or vaccinated from 2018 till 2020. At the district level, analysis by enrollment and birth year showed coverage increased between 2018 and 2019 and declined in 2020, while analysis by vaccination year showed consistent increase in coverage. However, micro-geographic analysis revealed pockets where coverage persistently declined. Notably 27/168, 39/168 and 3/156 Union councils showed consistently declining coverage when analyzing by enrollment, birth and vaccination year respectively. More than half (52.2%, 678,280/1,298,555) of the children received all their vaccinations exclusively through fixed centers and, 71.7% (499,391/696,701) received all vaccinations from the same centers. CONCLUSION: Despite overall improving vaccination coverage between 2018 and 2020, certain geographic areas have consistently declining coverage rates, which is detrimental for equity. Making immunization inequities visible through geospatial analysis is the first step to ensure resources are allocated optimally. Our study provides impetus for immunization programs to develop and invest in geospatial technologies, harnessing its potential for improved coverage and equity.

Immunization Gender Inequity in Pakistan: An Analysis of 6.2 Million Children Born from 2019 to 2022 and Enrolled in the Sindh Electronic Immunization Registry.

Gender-based inequities in immunization impede the universal coverage of childhood vaccines. Leveraging data from the Government of Sindh's Electronic Immunization Registry (SEIR), we estimated inequalities in immunization for males and females from the 2019-2022 birth cohorts in Pakistan. We computed male-to-female (M:F) and gender inequality ratios (GIR) Tfor enrollment, vaccine coverage, and timeliness. We also explored the inequities by maternal literacy, geographic location, mode of vaccination delivery, and gender of vaccinators. Between 1 January 2019, and 31 December 2022, 6,235,305 children were enrolled in the SEIR, 52.2% males and 47.8% females. We observed a median M:F ratio of 1.03 at enrollment and at Penta-1, Penta-3, and Measles-1 vaccinations, indicating more males were enrolled in the immunization system than females. Once enrolled, a median GIR of 1.00 indicated similar coverage for females and males over time; however, females experienced a delay in their vaccination timeliness. Low maternal education; residing in remote-rural, rural, and slum regions; and receiving vaccines at fixed sites, as compared to outreach, were associated with fewer females being vaccinated, as compared to males. Our findings suggeste the need to tailor and implement gender-sensitive policies and strategies for improving equity in immunization, especially in vulnerable geographies with persistently high inequalities.

Evaluation of a Mobile-Based Immunization Decision Support System for Scheduling Age-Appropriate Vaccine Schedules for Children Younger Than 2 Years in Pakistan and Bangladesh: Lessons From a Multisite, Mixed Methods Study.

BACKGROUND: Missed opportunities for vaccination (MOVs), that is, when children interact with the health system but fail to receive age-eligible vaccines, pose a crucial challenge for equitable and universal immunization coverage. Inaccurate interpretations of complex catch-up schedules by health workers contribute to MOVs. OBJECTIVE: We assessed the feasibility of a mobile-based immunization decision support system (iDSS) to automatically construct age-appropriate vaccination schedules for children and to prevent MOVs. METHODS: A sequential exploratory mixed methods study was conducted at 6 immunization centers in Pakistan and Bangladesh. An android-based iDSS that is packaged in the form of an application programming interface constructed age-appropriate immunization schedules for eligible children. The diagnostic accuracy of the iDSS was measured by comparing the schedules constructed by the iDSS with the gold standard of evaluation (World Health Organization-recommended Expanded Programme on Immunization schedule constructed by a vaccines expert). Preliminary estimates were collected on the number of MOVs among visiting children (caused by inaccurate vaccination scheduling by vaccinators) that could be reduced through iDSS by comparing the manual schedules constructed by vaccinators with the gold standard. Finally, the vaccinators' understanding, perceived usability, and acceptability of the iDSS were determined through interviews with key informants. RESULTS: From July 5, 2019, to April 11, 2020, a total of 6241 immunization visits were recorded from 4613 eligible children. Data were collected for 17,961 immunization doses for all antigens. The iDSS correctly scheduled 99.8% (17,932/17,961) of all age-appropriate immunization doses compared with the gold standard. In comparison, vaccinators correctly scheduled 96.8% (17,378/17,961) of all immunization doses. A total of 3.2% (583/17,961) of all due doses (across antigens) were missed in age-eligible children by the vaccinators across both countries. Vaccinators reported positively on the usefulness of iDSS, as well as the understanding and benefits of the technology. CONCLUSIONS: This study demonstrated the feasibility of a mobile-based iDSS to accurately construct age-appropriate vaccination schedules for children aged 0 to 23 months across multicountry and low- and middle-income country settings, and underscores its potential to increase immunization coverage and timeliness by eliminating MOVs.

Assessment of vaccination service delivery and quality: a cross-sectional survey of over 1300 health facilities from 29 districts in Sindh, Pakistan conducted between 2017-18.

BACKGROUND: Routine childhood immunization coverage in Pakistan remains sub-par, in part, due to suboptimal utilization of existing vaccination services. Quality of vaccine delivery can affect both supply and demand for immunization, but data for immunization center quality in Pakistan is sparse and in Sindh province in Southern Pakistan, no comprehensive health facility assessment has ever been conducted at a provincial level. We assessed health facilities, specifically immunization centers, and their associated health workers throughout the province to summarize quality of immunization centers.  METHODS: An exhaustive list of health facilities obtained from Sindh's provincial government was included in our analysis, comprising a total of 1396 public, private, and public-private health facilities. We adapted a health facility and health worker assessment survey developed by BASICS and EPI-Sindh to record indicators pertaining to health facility infrastructure, processes and human resources. Using expert panel ranking, we developed critical criteria (the presence of a cold box/refrigerator, vaccinator and vaccination equipment at the immunization center) to indicate the bare minimum items required by immunization centers to vaccinate children. We also categorized other infrastructure, process, and human resource items to determine high, low and moderate function requirements to ascertain quality. We evaluated presence of critical criteria, calculated scores for high, moderate and low function requirements, and displayed frequencies of infrastructure, process and human resource indicators for all immunization centers across Sindh. We analyzed results at the division level and utilized a two-sample independent clustered t-test to test differences in average function requirement scores between facilities that met critical criteria and those that did not. RESULTS: Out of the 1396 health facilities assessed across Sindh province from October 2017 to January 2018, 1236 (88.5%) were operational while 1209 (86.6%) offered vaccination services (immunization centers). Only 793 (65.6%; 793/1209) immunization centers met the critical criteria of having all the following items: vaccinator, a cold box or refrigerator and vaccine supplies. Of the 416 (34.4%; 416/1209) immunization centers that did not meet the critical criteria, most of the centers did not have a cold box or refrigerator (28.3%; 342/1209), followed by lack of vaccines (19.9%; 240/1209), and a vaccinator (13.0%; 157/1209). Of the 2153 healthcare workers interviewed, 1875 (87.1%) were vaccinators, of which 1745 (81.0%; 1745/2153) were male, and had an average of 12.4 years of schooling. A total of 1805 (96.3%; 1805/1875), 1655 (88.3%; 1655/1875) and 1387 (74.0%; 1387/1875) of the vaccinators were trained in vaccination, cold chain and inventory management respectively. CONCLUSION: One out of three immunization centers in Sindh lack the critical components essential for quality vaccination services. While the majority of health workers (>80%) were trained on vaccination and cold chain management, the proportion trained on inventory management was comparatively low. Our findings therefore suggest that suboptimal immunization center quality is partly due to inadequate infrastructure and inefficient processes contributed to an extent, by low levels of inventory management training among vaccinators. Our study presents critical research findings with high-impact policy implications for identifying and addressing gaps to improve vaccination uptake within a low-middle income country setting.

Prevalence, geographical distribution and factors associated with pentavalent vaccine zero dose status among children in Sindh, Pakistan: analysis of data from the 2017 and 2018 birth cohorts enrolled in the provincial electronic immunisation registry.

OBJECTIVES: To estimate the prevalence of zero dose children (who have not received any dose of pentavalent (diphtheria, tetanus, pertussis, Haemophilus influenzae type B and hepatitis B) vaccine by their first birthday) among those who interacted with the immunisation system in Sindh, Pakistan along with their sociodemographic characteristics and risk factors. DESIGN AND PARTICIPANTS: We conducted a descriptive analysis of child-level longitudinal immunisation records of 1 467 975 0-23 months children from the Sindh's Zindagi Mehfooz (Safe Life) Electronic Immunisation Registry (ZM-EIR), for the birth cohorts of 2017 and 2018. SETTING: Sindh province, Pakistan which has a population of 47.9 million people and an annual birth cohort of 1.7 million. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome measure was zero dose status among enrolled children. Logistic regression was performed to identify the risk factors associated with the zero dose status. RESULTS: Out of 1 467 975 children enrolled in the ZM-EIR in Sindh, 10.6% (154 881/1 467 975) were zero dose. There were sharp inequities across the 27 districts. Zero dose children had a lower proportion of hospital births (28.5% vs 34.0%; difference 5.5 percentage points (pp) (95% CI 5.26 to 5.74); p<0.001) and higher prevalence from slums (49.5% vs 42.3%; difference 7.2 pp (95% CI 6.93 to 7.46); p<0.001), compared with non-zero dose children. Children residing in urban compared with rural areas were at a higher risk (relative risk (RR): 1.20; p<0.001; 95% CI 1.18 to 1.22), while children with educated compared with uneducated mothers were at a lower risk of being zero dose (RR: 0.47-0.96; p<0.001; 95% CI 0.45 to 0.98). CONCLUSIONS: Despite interacting with the immunisation system, 1 out of 10 children enrolled in the ZM-EIR in Sindh were zero dose. It is crucial to monitor the prevalence of zero dose children and investigate their characteristics and risk factors to effectively reach and follow-up with them.

Using a low-cost, real-time electronic immunization registry in Pakistan to demonstrate utility of data for immunization programs and evidence-based decision making to achieve SDG-3: Insights from analysis of Big Data on vaccines.

BACKGROUND: Despite the proliferation of digital interventions such as Electronic Immunization Registries (EIR), currently, there is little evidence regarding the use of EIR data to improve immunization outcomes in resource-constrained settings. To achieve the Sustainable Development Goal (SDG) of ensuring healthy lives and well-being for all ages, particularly for newborns and children under the age of 5 (goal 3b), it is essential to generate and use quality data for evidence-based decision making to overcome barriers inherent in immunization systems. In Pakistan, only 66 % of children receive all basic vaccinations, and in Sindh province, the number is even lower at 49 %. In 2012, IRD developed and piloted Zindagi Mehfooz (Safe Life; ZM) ElR, an Android-based platform that records and analyses individual-level child data in real-time. In 2017 in collaboration with Expanded Programme for Immunization (EPI) Sindh, ZM was scaled-up across the entire Sindh province and is currently being used by 2521 government vaccinators in 1539 basic health facilities, serving >48 million population. OBJECTIVE: The study aims to demonstrate how big immunization data from the ZM-EIR is being leveraged in Sindh, Pakistan for actionable decision making via three use cases (a) improving performance management of vaccinators to increase geographical coverage, (b) quantifying the impact of provincial accelerated outreach activities, and (c) examining the impact of the COVID-19 pandemic on routine immunization coverage to help devise a tailored approach for future efforts. METHODS: From October 2017 to April 2020, more than 2.9 million children and 0.9 million women have been enrolled, and more than 22 million immunization events have been recorded in the ZM-EIR. We extracted de-identified data from ZM-EIR for January 1, 2019 - April 20, 2020, period. Given the needs of each use case, monthly and daily indicators on vaccinator performance (attendance and compliance), daily immunization visits, and the number of antigens administered were calculated. Geo-coordinate data of antigen administration was extracted and displayed on geographic maps using QGIS. All generated reports were shared at fixed frequency with various stakeholders, such as partners at EPI-Sindh, for utilization in decision making and informing policy. RESULT: Our use-cases demonstrate the use of EIR data for data-driven decision making. From January - December 2019, the monthly monitoring of program indicators helped increase the vaccinator attendance from 44% to 88%, while an 85 % increase in geographical coverage was observed in a polio-endemic super high-risk union council (SHRUC) in Karachi. The analysis of daily average antigens administered during accelerated outreach efforts (AOE) as compared to routine activities showed an increase in average daily Pentavalent-3, Measles-1, and Measles-2 vaccines administered by 103%, 154%, and 180% respectively. These findings helped decide to continue the accelerated effort in high-risk areas (compared to the entire province) rather than discontinuing the activity due to high costs. During COVID-19 lockdown, the daily average number of child immunizations reduced from 16,649 to 4335 per day, a decline of 74% compared to 6 months preceding COVID-19 lockdown. ZM-EIR data is currently helping to shape the planning and implementation of critical strategies to mitigate the impact of the COVID-19 pandemic. CONCLUSION: The big data for vaccines generated through EIRs is a powerful tool to monitor immunization work-force and ensure chronically missed communities are identified and covered through targeted strategies. Geospatial data availability and analysis is changing the way EPI review meetings occur with stakeholders, taking data-driven decisions for better planning and resource allocation. In the fight against COVID-19 pandemic, as governments gradually begin to shift from containing the outbreak to strategizing a plan for sustaining the essential health services, the countries that will emerge most successful are likely the ones who can best use technology and real-time data for targeted efforts.

Impact of COVID-19 pandemic response on uptake of routine immunizations in Sindh, Pakistan: An analysis of provincial electronic immunization registry data.

BACKGROUND: COVID-19 pandemic has affected routine immunization globally. Impact will likely be higher in low and middle-income countries with limited healthcare resources and fragile health systems. We quantified the impact, spatial heterogeneity, and determinants for childhood immunizations of 48 million population affected in the Sindh province of Pakistan. METHODS: We extracted individual immunization records from real-time provincial Electronic Immunization Registry from September 23, 2019, to July 11, 2020. Comparing baseline (6 months preceding the lockdown) and the COVID-19 lockdown period, we analyzed the impact on daily immunization coverage rate for each antigen by geographical area. We used multivariable logistic regression to explore the predictors associated with immunizations during the lockdown. RESULTS: There was a 52.5% decline in the daily average total number of vaccinations administered during lockdown compared to baseline. The highest decline was seen for Bacille Cal-mette Guérin (BCG) (40.6% (958/2360) immunization at fixed sites. Around 8438 children/day were missing immunization during the lockdown. Enrollments declined furthest in rural districts, urban sub-districts with large slums, and polio-endemic super high-risk sub-districts. Pentavalent-3 (penta-3) immunization rates were higher in infants born in hospitals (RR: 1.09; 95% CI: 1.04-1.15) and those with mothers having higher education (RR: 1.19-1.50; 95% CI: 1.13-1.65). Likelihood of penta-3 immunization was reduced by 5% for each week of delayed enrollment into the immunization program. CONCLUSION: One out of every two children in Sindh province has missed their routine vaccinations during the provincial COVID-19 lockdown. The pool of un-immunized children is expanding during lockdown, leaving them susceptible to vaccine-preventable diseases. There is a need for tailored interventions to promote immunization visits and safe service delivery. Higher maternal education, facility-based births, and early enrollment into the immunization program continue to show a positive association with immunization uptake, even during a challenging lockdown.

Using Predictive Analytics to Identify Children at High Risk of Defaulting From a Routine Immunization Program: Feasibility Study.

BACKGROUND: Despite the availability of free routine immunizations in low- and middle-income countries, many children are not completely vaccinated, vaccinated late for age, or drop out from the course of the immunization schedule. Without the technology to model and visualize risk of large datasets, vaccinators and policy makers are unable to identify target groups and individuals at high risk of dropping out; thus default rates remain high, preventing universal immunization coverage. Predictive analytics algorithm leverages artificial intelligence and uses statistical modeling, machine learning, and multidimensional data mining to accurately identify children who are most likely to delay or miss their follow-up immunization visits. OBJECTIVE: This study aimed to conduct feasibility testing and validation of a predictive analytics algorithm to identify the children who are likely to default on subsequent immunization visits for any vaccine included in the routine immunization schedule. METHODS: The algorithm was developed using 47,554 longitudinal immunization records, which were classified into the training and validation cohorts. Four machine learning models (random forest; recursive partitioning; support vector machines, SVMs; and C-forest) were used to generate the algorithm that predicts the likelihood of each child defaulting from the follow-up immunization visit. The following variables were used in the models as predictors of defaulting: gender of the child, language spoken at the child's house, place of residence of the child (town or city), enrollment vaccine, timeliness of vaccination, enrolling staff (vaccinator or others), date of birth (accurate or estimated), and age group of the child. The models were encapsulated in the predictive engine, which identified the most appropriate method to use in a given case. Each of the models was assessed in terms of accuracy, precision (positive predictive value), sensitivity, specificity and negative predictive value, and area under the curve (AUC). RESULTS: Out of 11,889 cases in the validation dataset, the random forest model correctly predicted 8994 cases, yielding 94.9% sensitivity and 54.9% specificity. The C-forest model, SVMs, and recursive partitioning models improved prediction by achieving 352, 376, and 389 correctly predicted cases, respectively, above the predictions made by the random forest model. All models had a C-statistic of 0.750 or above, whereas the highest statistic (AUC 0.791, 95% CI 0.784-0.798) was observed in the recursive partitioning algorithm. CONCLUSIONS: This feasibility study demonstrates that predictive analytics can accurately identify children who are at a higher risk for defaulting on follow-up immunization visits. Correct identification of potential defaulters opens a window for evidence-based targeted interventions in resource limited settings to achieve optimal immunization coverage and timeliness.

Feasibility of using global system for mobile communication (GSM)-based tracking for vaccinators to improve oral poliomyelitis vaccine campaign coverage in rural Pakistan.

Despite multiple rounds of immunization campaigns, it has not been possible to achieve optimum immunization coverage for poliovirus in Pakistan. Supplementary activities to improve coverage of immunization, such as door-to-door campaigns are constrained by several factors including inaccurate hand-drawn maps and a lack of means to objectively monitor field teams in real time, resulting in suboptimal vaccine coverage during campaigns. Global System for Mobile Communications (GSM) - based tracking of mobile subscriber identity modules (SIMs) of vaccinators provides a low-cost solution to identify missed areas and ensure effective immunization coverage. We conducted a pilot study to investigate the feasibility of using GSM technology to track vaccinators through observing indicators including acceptability, ease of implementation, costs and scalability as well as the likelihood of ownership by District Health Officials. The real-time location of the field teams was displayed on a GSM tracking web dashboard accessible by supervisors and managers for effective monitoring of workforce attendance including 'time in-time out', and discerning if all target areas - specifically remote and high-risk locations - had been reached. Direct access to this information by supervisors eliminated the possibility of data fudging and inaccurate reporting by workers regarding their mobility. The tracking cost per vaccinator was USD 0.26/month. Our study shows that GSM-based tracking is potentially a cost-efficient approach, results in better monitoring and accountability, is scalable and provides the potential for improved geographic coverage of health services.