Effectiveness and Feasibility of Using Local Medical University for Capacity Building to Improve Immunization Program in Ethiopia. Implementation Science Study

doi:10.21203/rs.2.19244/v1

Background: Supportive supervision is one of five essential components of the WHO’s Reaching Every District strategy, an operational approach to reach 90%, immunization coverage. However, it is not mostly practiced based on the standard schedule because of capacity and low number of staffs in the health system. When it is practiced after long interval, it focuses more on inspection and finding lacunas. Taking into account the 54 universities in Ethiopia, the fundamental idea in this implementation science research is to test the feasibility and effectiveness of a Capacity Building and Mentorship Program in Immunization (CBMPI) by medical and health science colleges aimed at supplementing the current approach.

Methods: This study applied a pre-post quasi experimental research design. The study included health workers of 30 health facilities (15 intervention and 15 control) followed for six months. To assess effectiveness on immunization coverage and a change in reaching every district strategy implementation difference in difference analysis was used to estimate the effect of the intervention implementation over time. Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework was used to evaluate the implementation process.

Results: The findings of the study indicated that the intervention package has a statistically significant effect (P=0.0001) in the overall implementation of Reaching Every District (RED) outcomes. The difference in difference analysis also indicated that health facilities in the intervention district have shown a higher Penta III coverage (17.4%) and complete vaccination coverage (16.6%) that is attributable to the intervention package. Similarly, knowledge and skills of health workers improved significantly (P<0.05) after the intervention in most of the performance measures. During the qualitative evaluation, the key informants mentioned that the CBMPI program was effective and the new approach was acceptable by health workers for integration in the routine EPI program.

Conclusions: In conclusion, capacity building and mentorship by well-trained personnel with the appropriate knowledge and skills have positive effects on immunization program indicators. Thus, it is recommended to do implementation study within the framework of national immunization program to facilitate policy adoption and readiness for routine use in large scale.

This work was supported by the Bill & Melinda Gates Foundation, Grant Number OPP1191298

Health Policy

CBMPI

Capacity Building

Immunization

Medical University

Vaccination

Ethiopia

Ethiopia has a decentralized three-tier health system comprising a primary, secondary and tertiary level of care which has hierarchical supervisory mechanisms at each level. Appropriate and timely supportive supervision conducted in regular basis is an essential element in the process of transforming the health sector. Hence, Ethiopia has invested heavily in health system strengthening which helped the country to remarkably achieve most of the MDG targets (1,2).

Despite major improvements in reducing under five mortality in Ethiopia, vaccine preventable diseases are still an important cause of deaths. Currently, under the Sustainable Development Goals (SDGs) Ethiopia plans to end preventable deaths of newborns and children by implementing different strategies. Strengthening routine immunization is one of the child survival strategies in Ethiopia (3).

Although the different immunization strategies adopted in Ethiopia has brought some improvement, the coverage level is still below the recommended 90% level for the sustained control of vaccine preventable deaths (VPDs) (1,2,4). According to Ethiopian DHS 2016 report, the complete immunization coverage was 38.5% at national level and 45.8% in Amhara region (5) which further endowed with substantial geographical and social inequities, high dropout rates and declining trend in some poor performing regions.

It is notable that a committed, confident and competent vaccination workforce is integral to ensuring quality immunization service with high vaccination coverage (6,7). A lack of experienced and quality human resources can easily jeopardize the success of any health program including an immunization program. Reports from developing countries indicate that health workers seem to be overwhelmingly concerned with only raising immunization coverage (8).

The quality of immunization services that is equally important for the achievement of the ultimate goal of EPI has been neglected (9,10). For this, Reach Every District (RED) approach was launched in 2003 that aims to build capacity of the districts and health facility level to address common obstacles to improving immunization services. This district planning and monitoring approach promoted by the World Health Organization is based on five operational approaches or components that all specifically aim at improving coverage in every district;

Planning and management of resources: better management of human and financial resources.
Reaching target populations: improving access to immunization services by all.
Linking services with communities: partnering with communities to promote and deliver services.
Supportive supervision: regular on-site teaching, feedback and follow-up with health staff.
Monitoring for action: using tools and providing feedback for continuous self- assessment and improvement (11–13).

Supportive supervision is one of five essential components of the WHO’s Reaching Every District strategy, an operational approach to reach 90% coverage (14). Globally evidences showed that supportive supervision provides critical support for the delivery of health services and has a positive impact on improving service quality and coverage (8,14–16). National experiences also indicated that the onsite training has the most lasting effect on the quality of health services including immunization program (17). However, one of the major lacunae observed in the implementation of the Immunization program leading to low coverage and compromised quality is lack of supportive supervision by the supervisors in the health system18).

Currently, supervision is mostly crisis-oriented and supervisory visits are generally carried out in case a need arises and the feedbacks are not problem-solving (16). Common barriers to effective supervision in developing countries include lack of supervisor’s skill and knowledge, lack of supervisor’s authority to reward or sanction performances and lack of resource and accountability in the health system (2).

Ethiopia currently has 54 universities distributed across the country having medical colleges with high level of expertise. In this research, there was no published literature on direct engagement of local medical and health science colleges in building the capacity of the health system staff through continuous supportive supervision practice to improve immunization programs in Ethiopia. Hence, this study aimed to assess the effectiveness of capacity building and mentorship intervention by medical and health science colleges through the development of direct and indirect capacity building, mentorship, and community involvement to reach to every child in each household.

General Objective: The goal of this study was to assess the effect of medical and health Science College’s mentorship and supervision (CBMP) interventions on improving the coverage and quality of immunization services, reaching every district (RED) strategy implementation and improvements in capacity of health workers in selected districts of North-West, Ethiopia.

Specific Objectives

To test the effectiveness of CBMP intervention in improving the immunization service coverage and the RED strategy implementation
To test the effectiveness of CBMP intervention in improving the immunization practice, knowledge and skills of immunization providers
To assess the feasibility and acceptability of the CBMP in immunization intervention in selected districts

Study setting: The study was conducted in Wogera and Dabat districts of Amhara region, North-West Ethiopia.
Study period: The study was conducted from October 1, 2018 up to May 30, 2019.
Study design and methods: The study employed a pre-post quasi experimental design. In addition; qualitative study design was used to evaluate the implementation process of the CBMPI program.
Study population: A cohort of health care providers responsible for the supervision and provision of immunization services were included from the selected 15 health facilities in both groups.
Intervention packages

Development of supportive supervision and mentorship guidelines and tools based on WHO endorsed checklists and guidelines.
Training supervisors on supportive supervision and mentoring based on updating supervisors on current policies, new immunization practices, techniques, and management skills. Training on how to coach, mentor, effectively communicate, and conduct performance planning.
Preparation of tools for supervision which include detailed instructions for conducting supervision and mentoring, namely, sequence for conducting supervision meeting, checklist for supervisory visit, self-assessment for providers, work planning action sheets, supervisor competencies, giving feedback, resolving conflict and conducting difficult conversations.
Making supervision more effective by nature and objectives of supervision, to make it more supportive and facilitative than giving it the sense of auditing and controlling which was being practiced before.

Sample size and study participants: Health workers working in the 30 health facilities (15 health facilities from intervention and 15 health facilities from control groups) were considered for this study. In addition, district health office head, health center heads, health workers, health extension workers, supervisors from the selected district health offices and medical college staffs who participated in the program were approached for the key informant interview.
Study procedures and instruments: In this quasi experimental study, we assessed the intervention effectiveness and implementation process of the CBMPI program. For effectiveness, we used reaching every district programmatic outcomes and immunization service delivery outcomes which were assessed at baseline and at the end of the intervention implementation. For the qualitative part, RE-AIM framework has been applied to evaluate the implementation process over time.

Data collection: Primary data sources for the evaluation of the interventions included: (1) Interviews with immunization providers and supervisors; (2) Document review (checklists, filed notes, project documents) and (3) District administrative healthcare data bases. Measurements were done at baseline and end line in both the intervention and control areas at an individual and organizational level.
Quantitative data collection: All the quantitative data were collected using a structured questionnaire. The data were collected by six independent data collectors in the pre-implementation and post-implementation phase of the study.
Qualitative: Semi-structured questionnaire was used for the key informant interviews took place during the data collection period. Interviews were facilitated through the use of interview guides prepared in English and were translated into the local language and field tested out of the study sites. Interviews were digitally recorded for subsequent analysis transcribed verbatim by an independent transcriptionist and verified for accuracy.
Document review: A standardized checklist was developed and validated to evaluate the immunization programs.

Data Analysis

To assess effectiveness on immunization coverage and a change in reaching every district (RED) strategy implementation difference in difference analysis (DID) was used to estimate the effect of the intervention implementation over time. RED implementation was quantified by calculating a RED implementation score for each of the RED components. For assessing effectiveness in improving capacity of health workers, chi square test was used. Stata 14 software was used for analysis and significance level of 0.05 was considered statistically significant.

For the qualitative component of the implementation analysis, we took a general inductive approach with the evaluation questions related to the RE-AIM framework directing the analysis of data. All qualitative data sources, including interview transcripts and documents, were compiled and analyzed thematically.

Data quality management

The quality of the study output was ensured through employing a piloted and validated data collection tools. Pilot test was done before the actual data collection period. We provided rigorous training for all study staff and data collectors. A standard operating procedure was developed for data collection tool, interview protocols and document review protocols. Continuous supervisions were made during data collection to maintain the integrity of the collected data. We also established appropriate record keeping, documentation system and reporting mechanisms.

Operational definitions

Full vaccination coverage: proportion of under 1-year old infants vaccinated with all recommended 11 vaccines under the immunization program of Ethiopia (1).
Supportive supervision: A process that promotes quality at all levels of the health system by strengthening relationships within the system, focusing on the identification and resolution of problems, and helping to optimize the allocation of resources promoting high standards, teamwork, and better two-way communication (18,19).

Ethics and dissemination
This study obtained ethical approval from University of Gondar Institutional Ethical Review Board Ref. No: O/V/P/RCS/05/781/2018. Interventional program approval was also be gained from the regional health bureau, the district health office, health centers and community health workers. Written informed consent was sought from study participants. The scientific findings of the study will be made available in reputable journals through publications and will be presented in scientific conferences.

Profile of health facilities

In this study a total of 30 health facilities were approached. Among those health facilities 15(50%) were from Wogera district receiving the intervention and 15(50%) were from Dabat District as control group. Finally, a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis [Table 1].

Profile of key informants

Key informant interviews were done on woreda health office head (1), CBMPI project supervisors (2), health center head (1), EPI focal persons in health centers (2) and health extension workers from health posts (2). Among the key informants 5 were males and 3 were females. The average age of the respondents was 32 years ranging from minimum of 28 years to maximum of 47 years. The respondents had on average 6 years’ experience in EPI related programs which ranged from 1 year to 14 years.

Implementation of individual indicators for RED Strategy

Table 2 shows the individual indicators and their weight factor on the various indicators per component indicator. In brief, each operational component of the RED strategy was evaluated by a number of individual indicators and a component indicator was calculated by summing the weights which were assigned to each individual indicator. The overall RED implementation score was determined by summing the five RED component indicators.

Among the individual indicators under planning management component of the RED strategy, presence of micro plan (P-value = 0.027), identification of high risk areas (P-value = 0.005) and having plan for outreach sessions (P-value = 0.004) have shown a statistically significant improvement in implementation after the introduction of the CBMPI intervention package [Table 2].

In terms of using data for action, presence of EPI monitoring chart (P-Value = 0.002 and presence of defaulter tracing system (p-value = 0.021) were significantly improved after implementation of the intervention. Similarly, regularity of supportive supervision (P-value = 0.016) and availability of checklist (P-value = 0.004) during supervision has been improved significantly by the CBMPI program [Table 2].

Pertaining to community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the intervention (P-Value = 0.027) [Table 2].

RED Strategy implementation score for component indicators

Table 3 shows the mean score of each component indicator of the RED strategy in the study area before and after the implementation of the intervention package. The findings showed that the mean score for all the five component indictors has increased after the implementation of the intervention package in the study area. The overall RED strategy implementation score was 2.71 (range: 0.86–4.1) during pre-intervention while the score has increased to 4.36 (range: 3.39–4.99) during post intervention [Table 3].

The use of data for action with mean score of 0.95 (range: 0.8–1) and community engagement with mean score of 0.94 (range: 0.66–1) component indictors had the highest score with better implementation after the introduction of the CBMPI project. On the other hand, planning management was relatively the least well implemented component indictor with mean score of 0.76 (range 0.4–1) [Table 3].

Effect of the CBMPI intervention on RED implementation strategy components

The findings of the study revealed that the CBMPI intervention package has a statistically significant effect on implementation of planning management, data use for action and out-reach components of the RED strategy. The difference in differences estimator is positive for planning and data use component which showed that health facilities in Wogera District receiving the intervention package have a statistically significant higher (0.3) implementation score than health facilities of Dabat district in the control group over time. The out-reach component of the RED implementation strategy was also better implemented in the intervention group with a statistically significant higher (0.5) implementation score (P-Value = 0.001). In general, the intervention package has a positive statically significant effect on over all implementation of RED strategy (P-Value = 0.0001) [Table 4].

Effect of the CBMPI intervention package on vaccination service delivery outcomes

The difference in differences estimator is positive for Penta III coverage and shows that health facilities in Wogera woreda receiving the intervention package have a statically significant higher (17.4%) Penta III coverage during 2010 to 2011 than health facilities of Dabat district in the control group over time. Similarly, the difference in differences estimator is positive for full vaccination coverage and shows that health facilities in Wogera woreda receiving the intervention package have a statically significant higher (16.6%) full vaccination coverage during 2010 to 2011 than health facilities of Dabat district in the control group over time. On the other hand, the intervention package reduced the Penta III dropout rate in the intervention group significantly. Accordingly, the Penta drop-out rate has reduced on average by 2.6% in the intervention group than the control group during 2010 to 2011 over time [Table 5].

Knowledge of health workers on cold chain management

In this study knowledge of health workers working in the immunization program of each health facility were asked about the correct methods of maintaining the cold chain system. Accordingly, after the implementation of the intervention package 46% of the respondents correctly explained the expected distance of ILR and DF from the wall (P-value = 0.011) which showed a statistically significant difference as compared with pre implementation [Table 6].

Similarly, the knowledge of health workers on correct temperature log maintenance (P-value = 0.39), appropriate cabinet temperature for ILRs (P-value = 0.016) and the correct placement of diluents in ILR (P-value = 0.011) were significantly improved after implementation of the CBMPI intervention package [Table 6].

Assessment of skills and practice of health workers during child immunization

Assessment of the skills and practice of health workers were assessed for five children at each health facility. Accordingly, provision of appropriate vaccines (P-value = 0.027), communication on key vaccination messages (p-value–0.005) and safe disposal of used syringes/needles (P-value = 0.004) were performed with correct procedures after the implementation of the CBMPI intervention package. While, administration of vaccines with the correct route (P-value = 0.141) and touching/ recapping of needles by vaccinators didn’t show a statistically significant difference after the implementation of the intervention package [Table 7].

Qualitative results using RE-AIM implementation science framework

The findings from the implementation science qualitative synthesis were presented using RE-AIM framework as follows.

REACH

All the eight key informants were willing to participate in the CBMPI program and explained that the CBMPI program is reachable to all health facilities. One of the health center head reported thatI believed that the CBMPI project will give us better support and improvement in our health facility’s immunization program which is actually reachable to our health posts and other health facilities.””

EFFECTIVENESS

Over all program effectiveness: In terms of effectiveness, all the key informants have reported that the CBMPI project has brought over all change in improving immunization program of the supervised health facilities. All the respondents mentioned that the CBMPIprogram had better support to solve previously existing basic problems in the EPI program of supervised health facilities. All of the key informants agreed that, capacity of health workers have been strengthened by the new CBMPI intervention package. One of the CBMPI program supervisor replied that “I believe that both health extension workers and health care workers gained new knowledge and skills as a result of our support for EPI”. More over a Health center EPI focal person supplemented that; "The CBMPI program brought significant change. There were changes in improving knowledge of health workers on cold chain system management especially in vaccine labelling management and ice pack use for vaccine carrier.

The qualitative findings also indicated that the use of data for action have been changed in supervised health facilities since the introduction of the new CBMPI program. One of the supervisor stated that “for Immunization, the EPI monitoring chart is vital and it was not updated. With our support, we let the health workers to use the EPI immunization chart and other data’s for decision making. For example, health workers can see the EPI monitoring chart to assess their performance and identify defaulters from the routine immunization schedules.” The Health center head also explained that; “With the support, our documentation has been improved including our registration on log books and as well on vaccine stock status management. With this we are being using our collected data for decision making especially the EPI monitoring chart.”

Majority of the respondents mentioned that the community engagement and communication with caregivers during vaccination sessions had been done routinely during the CBMPI program implementation by leveraging the existing health development army. One health extension worker mentioned that; “With the help of CBMPI program supervisors, we were strictly educating mothers on anticipated vaccination side effects so that they will not drop from the vaccination schedules.””

The CBMPI program has also brought remarkable change on the cold chain system. The cold chain management was improved significantly in fulfilling icepacks and foam pads in vaccine carriers during vaccine transportation at health post level. The health extension worker elaborated that; “Initially, we were using 2 icepacks for vaccine transport. The supervisors taught us to use the four icepacks in vaccine carrier. Since then, we are using four ice packs to transport vaccines from the health center every month. They also showed us how to arrange and manage vaccine to keep them potent. One of the supervisors also complimented that; "The cold chain was one of the major areas with gaps where we have been giving support.

In terms of fixed and outreach sessions, all the respondents agreed that there had been continuous support and change in the fixed and outreach EPI programs after the introduction of the CBMPI program.

ADOPTION

Regarding program acceptance, majority of the key informants mentioned that adopting the new intervention package in the routine immunization program was acceptable and the commitment of health facility staffs to accept the new program was encouraging. One of the supervisor mentioned that "the health workers were committed and supportive in implementing the CBMPI project. At the start some of health extension workers were believing that we went there for inspection, gradually after understanding the objective of the project, they had good attitudes for the program and we brought change in the EPI service delivery in collaborative manner.”

IMPLEMENTATION

All key informants explained that the CBMPI program has been implemented very well in a way that could bring change.Regarding the implementation of the intervention package as intended, one of the supervisors reported that “The mentorship was conducted very well. Mentorship brings change immediately. We gave them the activities to be done in one round has been improved for the next round.” TheWoreda health office supervisor also complimented that; "The supervision was by team and even we shared experiences among the team members.

Implementation of agreed action points: All respondents agreed that majority of the action points have been implemented by respective responsible persons.With regard to the implementation of agreed action points one supervisor reported that "Unless it is beyond their capacity, action points were implemented at health center and health post level based on their time schedules. We usually give responsibility based on their level and for those activities beyond health facility we give to woreda health office and followed their implementation.”” The woreda health office head also mentioned that; “Action points were prepared in each visits and we tried our level best to manage the problems encountered especially in fulfilling’s logistics. We also make follow-ups in the implementation of the action points by responsible bodies. Yes, majority of action points were implemented.”

Contributing factors for success of the program implementation: The overall coordination of the project and the support from health workers were reported as one of the contributing factors for the success of the CBMPI program implementation. One of the supervisor explained that "The CBMPI program was well organized and implemented as planned. Among the reasons for the program effectiveness, one supervisor reported that "The program was implemented continually and the health workers were supportive as well. In terms of outcome there was an increment in service coverage and decrease in drop-out rates.” The woreda supervisor also reported that: “It is effective. The first thing that makes it effective is that the University physicians and other staffs had continues supervision and support system with development of action plans and its implementation.” The woreda health office head also reported that; what makes the program successful is the coordination process of the implementation. Additionally, the supervisors had also the necessary skill and knowledge for the mentorship program.”

Challenges faced during program implementation: During program implementation there was no major challenge. Some of the challenges include transportation arrangement, security issues for the monthly mentorship activities and delay in fulfilling the requested logistics on time were also mentioned as a challenge.The woreda health office supervisor explained that; “As a gap the transportation arrangement should be corrected.” The woreda health office head also elaborated that; “The implementation was good except for some interruptions due to security reasons.” As a challenge, some logistics that has been requested by health posts were not fulfilled. One of the supervisors also added “Some problem noticed was on installation and maintenance of refrigerators for health posts on time as agreed on the action points.”

Recommendation to make this program better: The recommendations from key informants to make this program better in the future focused on collaborative activities and increasing the number of health facilities that has to be covered. The recommendations by the woreda health office head to make this program better. As to me it will be good to cover all out reach sessions. The content of the support is fine but frequency of the schedules should be routine to make it sustainable.” The health center head also reported that; "We have weekly learning sessions and at that session, we have discussed about the CBMPI project. In that session, the health workers have good attitude and are interested to work with the project. It is helpful and we want it to be continuous for the future.

MAINTENANCE

Integration and sustainability of the program: All the respondents emphasized that involvement of different stakeholders and government ownership are critical for the integration and sustainability of the program effectiveness. For the program maintenance one of the supervisor recommends that "for academician it is usually theoretical and they need to work in the community more in practical support that will bring change in their local communities. They should work in collaboration with program people and the community.” The health center head also supplemented that; “There should be nationally coordinated program to organize and work with other medical college across the country.” The woreda health office head also explained that; “We, as woreda health office are committed and collaborative to work in integration with all other stakeholders in the future. We have good relationship with the university. The University of Gondar has capacity and I suggest the university to continue the support by increasing the number of health facilities to be included in the project.”

National health programs often have a weak supervision system with traditional approaches that are more attuned towards fault-finding and mere inspection. While there is plenty of evidence on supervision, there is a lack of sufficient evidence that inform the role of medical science colleagues in improving the quality and coverage of immunization programs. The overarching goal of our proposed project is, therefore, to close the gap in this evidence required for enhanced immunization program related supportive supervisions.

In this study a total of 30 health facilities were included. Finally a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis. Two health facilities (one health center and one health post) were excluded from the intervention group due to o security reasons. During the intervention period we were able to train 25 supervisors from the medical college and the district health office who conducted 65 mentorship activities (86.7% completion) in a six month period reaching to 40 health workers in both health centers and health posts with a high rate of content fidelity measured monthly.

The findings of this study revealed that, the overall RED implementation status has increased after the introduction of the capacity building and mentorship program. This finding is consistent with other evaluations which showed that there are a number of elements of the RED strategy which are well implemented (10).

In African regions, planning and management of resources lacked details which hamper the effective implementation of RED strategies (20). The purpose of the RED strategy is to support countries so as to strengthen immunization systems to sustainably and equitably increase access to and use of vaccination services (12). This study found that among the individual indicators under planning management component of the RED strategy, presence of micro plan, identification of high risk areas and having plan for outreach sessions have shown a statistically significant improvement in implementation after the introduction of the CBMPI intervention package. Studies also indicated that immunization service delivery works best when health facilities make their own micro plans and used their local data for decision making (10, 13, 21). From the qualitative findings data for action, presence of EPI monitoring chart and presence of defaulter tracing system were reported as having better achievement after the introduction of the CBMPI program.

Similarly, conducting supportive supervision and availability of checklists during supervision has been improved significantly by the CBMPI program. In another studies, the supportive supervision component of the RED strategy was the best implemented component (10,18). Other studies indicated that supportive supervision may not have independent effects on improving the quality of immunization services (15). These could be attributed with the availability of essential logistics, supply chain management, human resource and financial resources that may complement the supportive supervision strategy in improving immunization service delivery.

Regarding out-reach related component of the RED strategy, there was a significant improvement in the number of out-reach sessions conducted in using a strategy to address hard to reach after introduction of the CBMPI project. From the qualitative findings, health workers from District and Health facility identified outreach sessions as a positive contributor to immunization service delivery and the number and quality of outreach sessions has been improved with the CBMPI program. Evidences also reported an increase in the number of outreach sessions planned and held since the introduction of RED strategy (20). In Ethiopia, outreach was often provided door to door to increase vaccination coverage (13).

Community leaders can be tremendously useful to health care providers, especially when it comes to improving vaccination coverage (22). Pertaining to community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the CBMPI program. Other studies also showed that community volunteers, typically assist with defaulter tracking, outreach and community education (20, 23).

The qualitative findings revealed that the community engagement after the introduction of the intervention package was not as such strong. This is mainly because of the fact that women’s development army is in place and there has been a lot of investment and empowerment in rural Ethiopia (18). Evidences also showed that local health managers in Africa must work strategically to improve health service capacity and effectiveness and increase the use of health services (23–25). Our paper also has demonstrated that supportive supervision has a significant role for overall immunization program strengthening.

The results of this study showed that penta III and full immunization coverage’s has shown a statically significant higher coverage that is attributable to the CBMPI intervention. A study from India indicated that capacity building through supportive supervision improves immunization coverage and also serves an efficient tool to strengthen the local health system (18). On the other hand, the intervention package reduced the Penta III dropout rate in the intervention group significantly. Other studies also demonstrated that capacity building interventions contributed to relative improvements in district-level service delivery outcomes (18). Other evidences also complimented that without an effective system to deliver routine immunization, coverage rates cannot be increased (23, 26).

Hence the integrated approach of supportive supervision done by medical colleges and the district health office was productive in terms of improving the immunization coverage. Studies and program evaluations also suggested that facilitating on-the-job mentoring, quality improvement, and problem-solving to make supervision more conducive to improve in health workers’ performance (19). The qualitative findings also supplemented that the program was very effective in increasing immunization service quality and coverage.

In low and middle income countries, health workers are essential for the delivery of health interventions. However, inadequate health-worker performance is a very widespread problem. If correctly done, supervision could be amechanism for providing professional development,improving health workers’ job satisfaction, andincreasing motivation (18). Interventions like training can increase the knowledge and self-reported practice of immunization providers (27). In this study knowledge of health workers has been improved through the CBMPI program in terms of correct temperature log maintenance, appropriate cabinet temperature for ILRs and the correct placement of diluents in ILR. Studies also showed that capacity building can play a role in improving knowledge of health workers working in remote sites (24,26,28).

Assessment of the skills and practice of health workers was done for five children at each health facility. Accordingly, provision of appropriate vaccines, communication on key vaccination messages and safe disposal of used syringes/needles were performed with correct procedures after the implementation of the CBMPI intervention package. From the qualitative findings, key informants reported that the CBMPI program had a major role in building the skills of health workers working under the EPI program. Supervision of CHWs could be strengthened by streamlining supervision protocols to focus less on report checking and more on problem solving and skills development (29). In a study in Odisha, those supervisors who had undergone the rigorously planned capacity building and demonstration exercise in the field were better able to transfer skills to the supervisees (15).

During the implementation of the project, distance to health facilities was one of the challenges that required mentors to spend a long time travelling, resulting in transportation becoming a common barrier. In many cases, mentors were required to share one vehicle with other people travelling for different reasons due to limited vehicle availability in order to decrease cost. Competing priorities also served as a challenge, as in some cases mentors were called to work on other works in the college. Security problems were also one of the challenges.

Limitations of the study: The sample size to observe a clinically significant difference couldn’t be calculated which might have affected the statistical power of the study. Because some of the data were obtained through interviews, the information collected may be susceptible to interview bias.

Even though supportive supervision is regarded by the WHO as one of the five essential components of the Reaching Every District strategy, its implementation has been hampered by lack of capacity in the health system. To fill this implementation gap, we designed and tested, Capacity building and mentorship (CBMP) by well-trained personnel with the appropriate knowledge and skills have positive effects on immunization program. The intervention package also improved the knowledge and skills of health workers. Moreover, the study shows that most of the RED strategy components are well implemented; however, some elements need further improvements. Thus, it is recommended to do implementation study of the program within the framework of national immunization programmatic setting to facilitate policy adoption and readiness for scale up for routine use in large scale.

EPI: Extended Program of Immunization

HEW: Health Extension Worker

CBMPI: Capacity Building and Mentorship Program for Immunization

MAQ: Maximum Access and Quality

MDG: Millennium Development Goal

RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance

RED: Reach Every District

SDG: Sustainable Development Goal

USAID: United States Agency for International Development

WHO: World Health Organization

Ethics approval and consent to participate: This study obtained ethical approval from University of Gondar Institutional Ethical Review Board Ref. No: O/V/P/RCS/05/781/2018. Written informed consent was sought from study participants.

Consent for publication: Not applicable

Availability of data and materials: The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request

Competing interests: None.

Funding: This work was supported by Bill and Melinda Gates Foundation [Grant number OPP1191298]

Authors’ contributions: AA, AT, ZM, BE and BT conceived the study and wrote the study protocol. All authors read and approved the final study protocol.

Acknowledgements: The authors would like to thank the University of Gondar, Bill and Melinda Gates Foundation. The authors also want to thank Professor Lulu Muhe of WHO and Dr. Kumie Alene for their support in the development of the study protocol.

FMOH. Ethiopia national expanded programme on immunization comprehensive multi-year plan, Federal Ministry of Health, 2010.
FMOH. Health Sector Transformation Plan (2015/16–2019/20). 2015;20(May):1–118.
UNDP. Sustainable Development Goals,2015.
Teka BA, Kassahun, Fiona Braka Thomas Karengera Kathleen, Aron Gallagher Nsubuga PA, Assefu YL. Factors contributing to routine immunization performance in Ethiopia, 2014. Pan Afr Med J. 2017;27(Supp 2):5.
CSA. Ethiopian Demographic and Health Survey. 2016;
Rowe AK, de Savigny D, Lanata CF, Victora CG. How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet (London, England). 2005 Sep;366(9490):1026–35.
JSI. Building routine immunization. Capacity, knowledge and skills (bricks). 2016;
Djibuti M, Gotsadze G, Zoidze A, Mataradze G, Esmail LC, Kohler J. The role of supportive supervision on immunization program outcome - A randomized field trial from Georgia. BMC Int Health Hum Rights. 2009;9(SUPPL.1):1–12.
Yemane Berhane and M Demissie. Cold Chain Status at Immunisation Centers in Ethiopia. East Afr Med J. 2000;77(9):476–9.
Mwanamwenge A, Masumbu P, Mwansa FD, Masaninga F, Mazaba ML, Songolo P, et al. Evaluation of the Implementation of the Reaching Every District Approach in Routine Immunisation in Lusaka District, Zambia. Med J Zambia. 2016;43(3):152–5.
WHO. Implementing the Reaching Every District Approach. A Guide for District Health Management Teams. 2008;
WHO. REACHING EVERY DISTRICT (RED).A guide to increasing coverage and equity in all communities in the African Region. 2017.
Ryman T, Macauley R, Taylor P, Shimp L, Wilkins K. Reaching every district (RED) approach to strengthen routine immunization services: evaluation in the African region, 2005. J Public Health (Bangkok). 2009;Vol. 32, N.
Mentey VK, Moduga RP. Supportive Supervision as a Tool to Improve and Monitor Progress the Quality of Immunization Services in India. Int J Pharmateutical Med Res. 2015;(April):14–7.
Som M, Panda B, Pati S, Nallala S, Anasuya A, Chauhan AS, et al. Effect of Supportive Supervision on Routine Immunization Service Delivery-A Randomized Post-Test Study in Odisha. Glob J Health Sci [Internet]. 2014;6(6):61–7. Available from: http://ccsenet.org/journal/index.php/gjhs/article/view/36291
Milstien JB, Tapia M, Sow SO, Keita L, Kotloff K. Strengthening immunization in a West African country: Mali. Educ Health (Abingdon). 2007 Nov;20(3):120.
USAID. Assessment of District Health Supervisory System. 2006;
Babu GR, Singh V V., Nandy S, Jana S, Sathyanarayana TN, Sadhana SM. Supportive supervision and immunization coverage: Evidence from India. Internet J Epidemiol. 2011;9(2):3–8.
Marquez L and Kean L. Making supervision supportive and sustainable: new approaches to old problems. Maximising Access Qual Pap. 2002;30(4):1–28.
WHO. In-Depth Evaluation of the REACHING EVERY DISTRICT APPROACH in the African Region. 2007;
USAID. Reaching Every Community Using Quality Improvement (REC-QI): Mapping to support routine immunization microplanning in Uganda. 2016;
J. Montgomery, P. Ganguly and B. Carlson. An evaluation of immunization services, using the reaching every district criteria, in two districts of Gujarat, India. Glob Heal Res Policy. 2018;
Justice J, Sequeira J. M., LaFond AK., Biellik R, Tarekegn GM., & Negussie H. Study of the Drivers of Routine Immunization System Performance in Ethiopia. Arlington, VA: JSI Research & Training Institute, Inc./ARISE Pro, 2012.
O’Connell M and Wonodia C. Routine Immunization Consultant Program in Nigeria: A Qualitative Review of a Country-Driven Management Approach for Health Systems Strengthening. Glob Helath Sci Pract. 2016;4(1).
Fetene N et al. The Ethiopian health extension program and variation in health systems performance: What matters? PLoS One [Internet]. 2016;11(5):1–19. Available from: http://dx.doi.org/10.1371/journal.pone.0156438
Chutima Suraratdecha, CBS Venkata Ramana, Satish Kaipilyawar, c JVG Krishnamurthy, c Srilatha Sivalenka, c Naveena Ambatipudi, c Sanjay Gandhi c KU& JC. Cost and effectiveness analysis of immunization service delivery support in Andhra Pradesh, India. 2008;
Oluwatosin VB and Ogundeji MO. Impact of training intervention on immunization providers’ knowledge and practice of routine immunization in Ibadan, south-western Nigeria: a primary health care experience. Pan Afr Med J. 2016;
Redick C, Sarah H and Anne L. The Current State of CHW Training Programs in Sub-Saharan Africa and South Asia: What We Know, What We Don’t Know, and What We Need to Do. 2014;
Timothy Roberton et al. Initial experiences and innovations in supervising community health workers for maternal, newborn, and child health in Morogoro region, Tanzania. Robert al Hum Resour Heal. 2015;13(19).

Table 1: Health facilities included in the analysis

Health facilities	Wogera District (Intervention )	Dabat District (Control )
Health center	4	3
Health Post	9	12
Total	13	15

Table 2: Component and individual indicators with their respective weight factor in the study

Component indicator	Individual indicator	Individual indicator weight	Level	Pre	Post	P value
Planning Management (n=5)	Having a micro-plan for immunization	0.2	Yes	7(53.9%)	12(92.3%)	0.027
	Having a micro-plan for immunization	0	No	6(46.1%)	1(7.7%)	0.027
	Having list of the communities with target population	0.2	Yes	5(38.5%)	9(69.2%)	0.116
	Having list of the communities with target population	0	No	8(61.5%)	4(30.8%)	0.116
	High risk areas identified	0.2	Yes	7 (53.8%)	13(100%)	0.005
	High risk areas identified	0	No	6(46.2%)	0(0%)	0.005
	Having a plan to conduct outreach sessions	0.2	Yes	5(38.4%)	12(92.3%)	0.004
	Having a plan to conduct outreach sessions	0	No	8(61.6%)	1(7.7%)	0.004
	Having a plan to conduct fixed vaccination sessions	0.2	Yes	3(23.1%)	4(30.8%)	0.658
	Having a plan to conduct fixed vaccination sessions	0	No	10(76.9%)	9(69.2%)	0.658
Use of data for action component n=5	Having an EPI monitoring chart	0.2	Yes	6(46.2%)	13(100%)	0.002
	Having an EPI monitoring chart	0	No	7(53.8)	0(05)	0.002
	Attending district-level immunization review meetings	0.2	Yes	12(92.3%)	11(84.6%)	0.539
	Attending district-level immunization review meetings	0	No	1(7.7%)	2(15.4%)	0.539
	Having a defaulter tracking system	0.2	Yes	6(46.2%)	13(100%)	0.021
	Having a defaulter tracking system	0	No	7(53.8%)	0(0%)	0.021
	Providing complete and timely reports to district	0.2	Yes	12(92.3%)	13(100%)	0.308
	Providing complete and timely reports to district	0	No	1(7.7%)	0(0%)	0.308
	Receiving feedback on monthly reports	0.2	Yes	10(76.9%)	12(92.3%)	0.277
	Receiving feedback on monthly reports	0	No	3(23.1%)	1(7.7%)	0.277
Supportive Supervision (N=2)	Regular supervisory visit received	0.5	Yes	5(38.5%)	11(84.6%)	0.016
	Regular supervisory visit received	0	No	8(61.5%)	2(15.4%)	0.016
	Checklist available for supervision	0.5	Yes	5(38.5%)	12(92.3%)	0.004
	Checklist available for supervision	0	No	8(61.5%)	1(7.7%)	0.004
Outreach (N=3)	Outreach visits conducted	0.33	Yes	8(61.5%)	11(84.6%)	0.185
	Outreach visits conducted	0	No	5(38.5%)	2(15.4%)	0.185
	Number of outreach more than 2 per month	0.33	Yes	3(23.1%)	9(69.2%)	0.018
	Number of outreach more than 2 per month	0	No	10(76.9%)	4(30.8%)	0.018
	Having a strategy to address hard to reach areas	0.33	Yes	8(61.5%)	13(100%)	0.013
	Having a strategy to address hard to reach areas	0	No	5(38.5%)	0(0%)	0.013
Community (N=3)	Presence of social mobilisation activities	0.33	Yes	12(92.3%)	13 (100%)	0.308
	Presence of social mobilisation activities	0	No	1(7.7%)	0(0%)	0.308
	Planned meetings with the local leaders to discuss routine immunization	0.33	Yes	7(53.8%)	12(92.3%)	0.027
		0	No	6(46.2%)	1(7.7%)	0.027
	3. Using village-level volunteers to mobilize beneficiaries	0.33	Yes	9(69.2%)	12(92.3%)	0.135
	3. Using village-level volunteers to mobilize beneficiaries	0	No	4(30.8%)	1(7.7%)	0.135

Table 3: RED implementation score for component indicators for RED strategy during pre and post intervention implementation

S.No.	Component indicator	Pre intervention			Post intervention
S.No.	Component indicator	Maximum	Minimum	Mean	Maximum	Minimum	Mean
1	Planning Management	1	0	0.42	1	0.4	0.76
2	Use of data for action	1	0.2	0.71	1	0.8	0.95
3	Supportive supervision	1	0	0.38	1	0	0.85
4	Outreach	1	0	0.48	1	0.33	0.84
5	Community engagement	1	0	0.71	1	0.66	0.94
6	Overall red implementation score	4.1	0.86	2.71	4.99	3.39	4.36

Table 4: The effect of the CBMPI intervention package on RED implementation strategy components

S.No.	Indicator	DID estimator *(InterventionTime)**	95% CI	P-Value
1	Planning management	0.314	[0.027-0.6003]	0.032
2	Data use for action	0.326	[0.117-0.534]	0.003
3	Supervision	0.061	[-0.374-0.497]	0.778
4	Out reach	0.511	[0.217-0.805]	0.001
5	Community	0.032	[-0.234-0.297]	0.812
6	Over all RED implementation	1.244	[0.507-1.981]	0.001

Table 5: The effect of the CBMPI intervention package on vaccination service delivery outcomes

S.No.	Indicators	DID estimator *(InterventionTime)**	95% CI	P-Value
1	Penta III coverage	17.4159	[2.037, 32.793]	0.027
2	Polio III coverage	17.31128	[1.921, 32.700]	0.028
3	Penta dropout coverage	-2.672821	[-4.398, -0.947]	0.003
4	Full vaccination coverage	16.63179	[1.33, 31.928]	0.034

Table 6: Knowledge of health workers pre and post intervention implementation

S.No.	Item	Correct answer	Pre	Post	Chi-Square, P-Value
1	Distance of ILR and DF from the wall	Yes	1(7.7%)	6(46.1%)	0.011
1	Distance of ILR and DF from the wall	No	12(92.3%)	7(53.9%)	0.011
2	Temperature log maintenance	Yes	6(46.1%)	11(84.6%)	0.039
2	Temperature log maintenance	No	7(53.9%)	2(15.4%)	0.039
3	Appropriate cabinet temperature for ILRs	Yes	5(38.5%)	11(84.6%)	0.016
3	Appropriate cabinet temperature for ILRs	No	8(61.6%)	2(15.4%)	0.016
4	How many hours before distribution, diluents be placed in ILR point	Yes	6(46.1%)	12(92.3%)	0.011
4		No	7(53.9%)	1(7.6%)	0.011
5	Correct manner of placing Ice packs inside DF	Yes	6(46.1%)	10(76.9%)	0.107
5	Correct manner of placing Ice packs inside DF	No	7(53.9%)	3(23.1%)	0.107

Table 7: Skills and practice of health workers during child immunization measured pre and post intervention implementation

Components	Correct procedure for all children	Intervention districts		P-Value
		Pre	Post
All the child received appropriate vaccines*	Yes	7(53.9%)	12(92.3%)	0.027
	No	6(46.1%)	1(7.7%)
The vaccinator communicate with the mother/care giver on the key messages	Yes	4(30.8%)	11(84.6%)	0.005
	No	9(69.2%)	2(15.4%)
The vaccinator touch or recap the needle	Yes	5(38.5%)	2(15.4%)	0.185
	No	8(61.5%)	11(84.6%)
Each vaccine administered using the correct route for the vaccine	Yes	11(84.6%)	13(100%)	0.141
	No	2(15.4%)	0(0%)
Used syringes/needles disposed of in safety boxes	Yes	5(38.5%)	12(92.3%)	0.004
	No	8(61.5%)	1(7.7%)

* Five children were observed during vaccination sessions

Effectiveness and Feasibility of Using Local Medical University for Capacity Building to Improve Immunization Program in Ethiopia. Implementation Science Study

Status:

Version 1

Abstract

1. Background

2. Objectives

3. Methods

4. Results