Effectiveness and Feasibility of Using Local Medical University for Capacity Building to Improve Immunization Program in Ethiopia. Implementation Science Study

Background: Supportive supervision is one of five essential components of the WHO’s Reaching Every District strategy, an operational approach to reach 90%, immunization coverage. However, it is not mostly practiced based on the standard schedule because of capacity and low number of staffs in the health system. When it is practiced after long interval, it focuses more on inspection and finding lacunas. Taking into account the 54 universities in Ethiopia, the fundamental idea in this implementation science research is to test the feasibility and effectiveness of a Capacity Building and Mentorship Program in Immunization (CBMPI) by medical and health science colleges aimed at supplementing the current approach. Methods: This study applied a pre-post quasi experimental research design. The study included health workers of 30 health facilities (15 intervention and 15 control) followed for six months. To assess effectiveness on immunization coverage and a change in reaching every district strategy implementation difference in difference analysis was used to estimate the effect of the intervention implementation over time. Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework was used to evaluate the implementation process. Results: The findings of the study indicated that the intervention package has a statistically significant effect (P=0.0001) in the overall implementation of Reaching Every District (RED) outcomes. The difference in difference analysis also indicated that health facilities in the intervention district have shown a higher Penta III coverage (17.4%) and complete vaccination coverage (16.6%) that is attributable to the intervention package. Similarly, knowledge and skills of health workers improved significantly (P<0.05) after the intervention in most of the performance measures. During the qualitative evaluation, the key informants mentioned that the CBMPI program was effective and the new approach

was acceptable by health workers for integration in the routine EPI program.
Conclusions: In conclusion, capacity building and mentorship by well-trained personnel with the appropriate knowledge and skills have positive effects on immunization program indicators. Thus, it is recommended to do implementation study within the framework of national immunization program to facilitate policy adoption and readiness for routine use in large scale. This work was supported by the Bill & Melinda Gates Foundation, Grant Number OPP1191298

Background
Ethiopia has a decentralized three-tier health system comprising a primary, secondary and tertiary level of care which has hierarchical supervisory mechanisms at each level.
Appropriate and timely supportive supervision conducted in regular basis is an essential element in the process of transforming the health sector. Hence, Ethiopia has invested heavily in health system strengthening which helped the country to remarkably achieve most of the MDG targets (1,2).
Despite major improvements in reducing under five mortality in Ethiopia, vaccine preventable diseases are still an important cause of deaths. Currently, under the Sustainable Development Goals (SDGs) Ethiopia plans to end preventable deaths of newborns and children by implementing different strategies. Strengthening routine immunization is one of the child survival strategies in Ethiopia (3).
Although the different immunization strategies adopted in Ethiopia has brought some improvement, the coverage level is still below the recommended 90% level for the sustained control of vaccine preventable deaths (VPDs) (1,2,4). According to Ethiopian DHS 2016 report, the complete immunization coverage was 38.5% at national level and 45.8% in Amhara region (5) which further endowed with substantial geographical and social inequities, high dropout rates and declining trend in some poor performing regions.
It is notable that a committed, confident and competent vaccination workforce is integral to ensuring quality immunization service with high vaccination coverage (6,7). A lack of experienced and quality human resources can easily jeopardize the success of any health program including an immunization program. Reports from developing countries indicate that health workers seem to be overwhelmingly concerned with only raising immunization coverage (8).
The quality of immunization services that is equally important for the achievement of the ultimate goal of EPI has been neglected (9,10). For this, Reach Every District (RED) approach was launched in 2003 that aims to build capacity of the districts and health facility level to address common obstacles to improving immunization services. This district planning and monitoring approach promoted by the World Health Organization is based on five operational approaches or components that all specifically aim at improving coverage in every district; Planning and management of resources: better management of human and financial resources.
Reaching target populations: improving access to immunization services by all.
Linking services with communities: partnering with communities to promote and deliver services.
Supportive supervision: regular on-site teaching, feedback and follow-up with health staff.
Monitoring for action: using tools and providing feedback for continuous self-assessment and improvement (11)(12)(13).
Supportive supervision is one of five essential components of the WHO's Reaching Every District strategy, an operational approach to reach 90% coverage (14). Globally evidences showed that supportive supervision provides critical support for the delivery of health services and has a positive impact on improving service quality and coverage (8,(14)(15)(16).
National experiences also indicated that the onsite training has the most lasting effect on the quality of health services including immunization program (17). However, one of the major lacunae observed in the implementation of the Immunization program leading to low coverage and compromised quality is lack of supportive supervision by the supervisors in the health system (18).
Currently, supervision is mostly crisis-oriented and supervisory visits are generally carried out in case a need arises and the feedbacks are not problem-solving (16). Common barriers to effective supervision in developing countries include lack of supervisor's skill and knowledge, lack of supervisor's authority to reward or sanction performances and lack of resource and accountability in the health system (2).
Ethiopia currently has 54 universities distributed across the country having medical colleges with high level of expertise. In this research, there was no published literature on direct engagement of local medical and health science colleges in building the capacity of the health system staff through continuous supportive supervision practice to improve immunization programs in Ethiopia. Hence, this study aimed to assess the effectiveness of capacity building and mentorship intervention by medical and health science colleges through the development of direct and indirect capacity building, mentorship, and community involvement to reach to every child in each household.

Objectives
2.1 General Objective: The goal of this study was to assess the effect of medical and health Science College's mentorship and supervision (CBMP) interventions on improving the coverage and quality of immunization services, reaching every district (RED) strategy implementation and improvements in capacity of health workers in selected districts of North-West, Ethiopia.

1.
To test the effectiveness of CBMP intervention in improving the immunization service coverage and the RED strategy implementation 2.
To test the effectiveness of CBMP intervention in improving the immunization practice, knowledge and skills of immunization providers 3.
To assess the feasibility and acceptability of the CBMP in immunization intervention

Study procedures and instruments
In this quasi experimental study, we assessed the intervention effectiveness and implementation process of the CBMPI program. For effectiveness, we used reaching every district programmatic outcomes and immunization service delivery outcomes which were assessed at baseline and at the end of the intervention implementation. For the qualitative part, RE-AIM framework has been applied to evaluate the implementation process over time.
Data collection: Primary data sources for the evaluation of the interventions included: (1) Interviews with immunization providers and supervisors; (2) Document review (checklists, filed notes, project documents) and (3) District administrative healthcare data bases.
Measurements were done at baseline and end line in both the intervention and control areas at an individual and organizational level.
Quantitative data collection: All the quantitative data were collected using a structured questionnaire. The data were collected by six independent data collectors in the preimplementation and post-implementation phase of the study.
Qualitative: Semi-structured questionnaire was used for the key informant interviews took place during the data collection period. Interviews were facilitated through the use of interview guides prepared in English and were translated into the local language and field tested out of the study sites. Interviews were digitally recorded for subsequent analysis transcribed verbatim by an independent transcriptionist and verified for accuracy.
Document review: A standardized checklist was developed and validated to evaluate the immunization programs.

Data Analysis
Analysis for Specific objective 1 and 2: Effectiveness To assess effectiveness on immunization coverage and a change in reaching every district (RED) strategy implementation difference in difference analysis (DID) was used to estimate the effect of the intervention implementation over time. RED implementation was quantified by calculating a RED implementation score for each of the RED components. For assessing effectiveness in improving capacity of health workers, chi square test was used. Stata 14 software was used for analysis and significance level of 0.05 was considered statistically significant.

Analysis for Specific objective 3: RE-AIM implementation analysis
For the qualitative component of the implementation analysis, we took a general inductive approach with the evaluation questions related to the RE-AIM framework directing the analysis of data. All qualitative data sources, including interview transcripts and documents, were compiled and analyzed thematically.

Data quality management
The quality of the study output was ensured through employing a piloted and validated data collection tools. Pilot test was done before the actual data collection period. We provided rigorous training for all study staff and data collectors. A standard operating procedure was developed for data collection tool, interview protocols and document review protocols. Continuous supervisions were made during data collection to maintain the integrity of the collected data. We also established appropriate record keeping, documentation system and reporting mechanisms.

Operational definitions
Full vaccination coverage: proportion of under 1-year old infants vaccinated with all recommended 11 vaccines under the immunization program of Ethiopia (1).
Supportive supervision: A process that promotes quality at all levels of the health system by strengthening relationships within the system, focusing on the identification and resolution of problems, and helping to optimize the allocation of resources promoting high standards, teamwork, and better two-way communication (18,19).

Ethics and dissemination
This study obtained ethical approval from University of Gondar Institutional Ethical Review Board Ref. No: O/V/P/RCS/05/781/2018. Interventional program approval was also be gained from the regional health bureau, the district health office, health centers and community health workers. Written informed consent was sought from study participants.
The scientific findings of the study will be made available in reputable journals through publications and will be presented in scientific conferences.

Profile of health facilities
In this study a total of 30 health facilities were approached. Among those health facilities 15(50%) were from Wogera district receiving the intervention and 15(50%) were from Dabat District as control group. Finally, a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis [ Table 1].

Profile of key informants
Key informant interviews were done on woreda health office head (1), CBMPI project supervisors (2), health center head (1), EPI focal persons in health centers (2) and health extension workers from health posts (2). Among the key informants 5 were males and 3 were females. The average age of the respondents was 32 years ranging from minimum of 28 years to maximum of 47 years. The respondents had on average 6 years' experience in EPI related programs which ranged from 1 year to 14 years. Table 2 Table 2].

Implementation of individual indicators for RED Strategy
In terms of using data for action, presence of EPI monitoring chart (P-Value = 0.002 and presence of defaulter tracing system (p-value = 0.021) were significantly improved after implementation of the intervention. Similarly, regularity of supportive supervision (Pvalue = 0.016) and availability of checklist (P-value = 0.004) during supervision has been improved significantly by the CBMPI program [ Table 2].
Pertaining to community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the intervention (P-Value = 0.027) [ Table 2].   Table 3].

Effect of the CBMPI intervention on RED implementation strategy components
The findings of the study revealed that the CBMPI intervention package has a statistically significant effect on implementation of planning management, data use for action and out-  Table 4].  In this study knowledge of health workers working in the immunization program of each health facility were asked about the correct methods of maintaining the cold chain system. Accordingly, after the implementation of the intervention package 46% of the respondents correctly explained the expected distance of ILR and DF from the wall (Pvalue = 0.011) which showed a statistically significant difference as compared with pre implementation [ Table 6]. Similarly, the knowledge of health workers on correct temperature log maintenance (Pvalue = 0.39), appropriate cabinet temperature for ILRs (P-value = 0.016) and the correct placement of diluents in ILR (P-value = 0.011) were significantly improved after implementation of the CBMPI intervention package [ Table 6].
Assessment of skills and practice of health workers during child immunization Assessment of the skills and practice of health workers were assessed for five children at  Table 7]. Table 7 Skills and practice of health workers during child immunization measured pre and post intervention Qualitative results using RE-AIM implementation science framework The findings from the implementation science qualitative synthesis were presented using RE-AIM framework as follows.

REACH
All the eight key informants were willing to participate in the CBMPI program and explained that the CBMPI program is reachable to all health facilities. One of the health center head reported that "I believed that the CBMPI project will give us better support and improvement in our health facility's immunization program which is actually reachable to our health posts and other health facilities."

EFFECTIVENESS Over all program effectiveness
In terms of effectiveness, all the key informants have reported that the CBMPI project has brought over all change in improving immunization program of the supervised health facilities. All the respondents mentioned that the CBMPI program had better support to solve previously existing basic problems in the EPI program of supervised health facilities.
All of the key informants agreed that, capacity of health workers have been strengthened by the new CBMPI intervention package. One of the CBMPI program supervisor replied that "I believe that both health extension workers and health care workers gained new knowledge and skills as a result of our support for EPI". More over a Health center EPI focal person supplemented that; "The CBMPI program brought significant change. There were changes in improving knowledge of health workers on cold chain system management especially in vaccine labelling management and ice pack use for vaccine carrier.
The qualitative findings also indicated that the use of data for action have been changed in supervised health facilities since the introduction of the new CBMPI program. One of the supervisor stated that "for Immunization, the EPI monitoring chart is vital and it was not updated. With our support, we let the health workers to use the EPI immunization chart and other data's for decision making. For example, health workers can see the EPI monitoring chart to assess their performance and identify defaulters from the routine immunization schedules." The Health center head also explained that; "With the support, our documentation has been improved including our registration on log books and as well on vaccine stock status management. With this we are being using our collected data for decision making especially the EPI monitoring chart." Majority of the respondents mentioned that the community engagement and communication with caregivers during vaccination sessions had been done routinely during the CBMPI program implementation by leveraging the existing health development army. One health extension worker mentioned that; "With the help of CBMPI program supervisors, we were strictly educating mothers on anticipated vaccination side effects so that they will not drop from the vaccination schedules." The CBMPI program has also brought remarkable change on the cold chain system. The cold chain management was improved significantly in fulfilling icepacks and foam pads in vaccine carriers during vaccine transportation at health post level. The health extension worker elaborated that; "Initially, we were using 2 icepacks for vaccine transport. The supervisors taught us to use the four icepacks in vaccine carrier. Since then, we are using four ice packs to transport vaccines from the health center every month. They also showed us how to arrange and manage vaccine to keep them potent. One of the supervisors also complimented that; "The cold chain was one of the major areas with gaps where we have been giving support.
In terms of fixed and outreach sessions, all the respondents agreed that there had been continuous support and change in the fixed and outreach EPI programs after the introduction of the CBMPI program.

ADOPTION
Regarding program acceptance, majority of the key informants mentioned that adopting the new intervention package in the routine immunization program was acceptable and the commitment of health facility staffs to accept the new program was encouraging. One of the supervisor mentioned that "the health workers were committed and supportive in implementing the CBMPI project. At the start some of health extension workers were believing that we went there for inspection, gradually after understanding the objective of the project, they had good attitudes for the program and we brought change in the EPI service delivery in collaborative manner."

IMPLEMENTATION
All key informants explained that the CBMPI program has been implemented very well in a way that could bring change. Regarding the implementation of the intervention package as intended, one of the supervisors reported that "The mentorship was conducted very well. Mentorship brings change immediately. We gave them the activities to be done in one round has been improved for the next round." The Woreda health office supervisor also complimented that; "The supervision was by team and even we shared experiences among the team members.

Implementation of agreed action points
All respondents agreed that majority of the action points have been implemented by respective responsible persons. With regard to the implementation of agreed action points one supervisor reported that "Unless it is beyond their capacity, action points were implemented at health center and health post level based on their time schedules. We usually give responsibility based on their level and for those activities beyond health facility we give to woreda health office and followed their implementation." The woreda health office head also mentioned that; "Action points were prepared in each visits and we tried our level best to manage the problems encountered especially in fulfilling's logistics.
We also make follow-ups in the implementation of the action points by responsible bodies.
Yes, majority of action points were implemented." Contributing factors for success of the program implementation: The overall coordination of the project and the support from health workers were reported as one of the contributing factors for the success of the CBMPI program implementation.
One of the supervisor explained that "The CBMPI program was well organized and implemented as planned. Among the reasons for the program effectiveness, one supervisor reported that "The program was implemented continually and the health workers were supportive as well. In terms of outcome there was an increment in service coverage and decrease in drop-out rates." The woreda supervisor also reported that: "It is effective. The first thing that makes it effective is that the University physicians and other staffs had continues supervision and support system with development of action plans and its implementation." The woreda health office head also reported that; what makes the program successful is the coordination process of the implementation. Additionally, the supervisors had also the necessary skill and knowledge for the mentorship program."

Challenges faced during program implementation
During program implementation there was no major challenge. Some of the challenges include transportation arrangement, security issues for the monthly mentorship activities and delay in fulfilling the requested logistics on time were also mentioned as a challenge.
The woreda health office supervisor explained that; "As a gap the transportation arrangement should be corrected." The woreda health office head also elaborated that; "The implementation was good except for some interruptions due to security reasons." As a challenge, some logistics that has been requested by health posts were not fulfilled. One of the supervisors also added "Some problem noticed was on installation and maintenance of refrigerators for health posts on time as agreed on the action points." Recommendation to make this program better The recommendations from key informants to make this program better in the future focused on collaborative activities and increasing the number of health facilities that has to be covered. The recommendations by the woreda health office head to make this program better. As to me it will be good to cover all out reach sessions. The content of the support is fine but frequency of the schedules should be routine to make it sustainable." The health center head also reported that; "We have weekly learning sessions and at that session, we have discussed about the CBMPI project. In that session, the health workers have good attitude and are interested to work with the project. It is helpful and we want it to be continuous for the future.

MAINTENANCE
Integration and sustainability of the program All the respondents emphasized that involvement of different stakeholders and government ownership are critical for the integration and sustainability of the program effectiveness. For the program maintenance one of the supervisor recommends that "for academician it is usually theoretical and they need to work in the community more in practical support that will bring change in their local communities. They should work in collaboration with program people and the community." The health center head also supplemented that; "There should be nationally coordinated program to organize and work with other medical college across the country." The woreda health office head also explained that; "We, as woreda health office are committed and collaborative to work in integration with all other stakeholders in the future. We have good relationship with the university. The University of Gondar has capacity and I suggest the university to continue the support by increasing the number of health facilities to be included in the project."

Discussions
National health programs often have a weak supervision system with traditional approaches that are more attuned towards fault-finding and mere inspection. While there is plenty of evidence on supervision, there is a lack of sufficient evidence that inform the role of medical science colleagues in improving the quality and coverage of immunization programs. The overarching goal of our proposed project is, therefore, to close the gap in this evidence required for enhanced immunization program related supportive supervisions.
In this study a total of 30 health facilities were included. Finally a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis.
Two health facilities (one health center and one health post) were excluded from the intervention group due to o security reasons. During the intervention period we were able to train 25 supervisors from the medical college and the district health office who conducted 65 mentorship activities (86.7% completion) in a six month period reaching to 40 health workers in both health centers and health posts with a high rate of content fidelity measured monthly.
The findings of this study revealed that, the overall RED implementation status has increased after the introduction of the capacity building and mentorship program. This finding is consistent with other evaluations which showed that there are a number of elements of the RED strategy which are well implemented (10).
In African regions, planning and management of resources lacked details which hamper the effective implementation of RED strategies (20). The purpose of the RED strategy is to support countries so as to strengthen immunization systems to sustainably and equitably increase access to and use of vaccination services (12). This study found that among the individual indicators under planning management component of the RED strategy, presence of micro plan, identification of high risk areas and having plan for outreach sessions have shown a statistically significant improvement in implementation after the introduction of the CBMPI intervention package. Studies also indicated that immunization service delivery works best when health facilities make their own micro plans and used their local data for decision making (10,13,21). From the qualitative findings data for action, presence of EPI monitoring chart and presence of defaulter tracing system were reported as having better achievement after the introduction of the CBMPI program.
Similarly, conducting supportive supervision and availability of checklists during supervision has been improved significantly by the CBMPI program. In another studies, the supportive supervision component of the RED strategy was the best implemented component (10,18). Other studies indicated that supportive supervision may not have independent effects on improving the quality of immunization services (15). These could be attributed with the availability of essential logistics, supply chain management, human resource and financial resources that may complement the supportive supervision strategy in improving immunization service delivery.
Regarding out-reach related component of the RED strategy, there was a significant improvement in the number of out-reach sessions conducted in using a strategy to address hard to reach after introduction of the CBMPI project. From the qualitative findings, health workers from District and Health facility identified outreach sessions as a positive contributor to immunization service delivery and the number and quality of outreach sessions has been improved with the CBMPI program. Evidences also reported an increase in the number of outreach sessions planned and held since the introduction of RED strategy (20). In Ethiopia, outreach was often provided door to door to increase vaccination coverage (13).
Community leaders can be tremendously useful to health care providers, especially when it comes to improving vaccination coverage (22). Pertaining to community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the CBMPI program. Other studies also showed that community volunteers, typically assist with defaulter tracking, outreach and community education (20, 23).
The qualitative findings revealed that the community engagement after the introduction of the intervention package was not as such strong. This is mainly because of the fact that women's development army is in place and there has been a lot of investment and empowerment in rural Ethiopia (18). Evidences also showed that local health managers in Africa must work strategically to improve health service capacity and effectiveness and increase the use of health services (23)(24)(25). Our paper also has demonstrated that supportive supervision has a significant role for overall immunization program strengthening.
The results of this study showed that penta III and full immunization coverage's has shown a statically significant higher coverage that is attributable to the CBMPI intervention. A study from India indicated that capacity building through supportive supervision improves immunization coverage and also serves an efficient tool to strengthen the local health system (18). On the other hand, the intervention package reduced the Penta III dropout rate in the intervention group significantly. Other studies also demonstrated that capacity building interventions contributed to relative improvements in district-level service delivery outcomes (18). Other evidences also complimented that without an effective system to deliver routine immunization, coverage rates cannot be increased (23,26).
Hence the integrated approach of supportive supervision done by medical colleges and the district health office was productive in terms of improving the immunization coverage.
Studies and program evaluations also suggested that facilitating on-the-job mentoring, quality improvement, and problem-solving to make supervision more conducive to improve in health workers' performance (19). The qualitative findings also supplemented that the program was very effective in increasing immunization service quality and coverage.
In low and middle income countries, health workers are essential for the delivery of health interventions. However, inadequate health-worker performance is a very widespread problem. If correctly done, supervision could be a mechanism for providing professional development, improving health workers' job satisfaction, and increasing motivation (18).
Interventions like training can increase the knowledge and self-reported practice of immunization providers (27). In this study knowledge of health workers has been improved through the CBMPI program in terms of correct temperature log maintenance, appropriate cabinet temperature for ILRs and the correct placement of diluents in ILR. Studies also showed that capacity building can play a role in improving knowledge of health workers working in remote sites (24,26,28).
Assessment of the skills and practice of health workers was done for five children at each health facility. Accordingly, provision of appropriate vaccines, communication on key vaccination messages and safe disposal of used syringes/needles were performed with correct procedures after the implementation of the CBMPI intervention package. From the qualitative findings, key informants reported that the CBMPI program had a major role in building the skills of health workers working under the EPI program. Supervision of CHWs could be strengthened by streamlining supervision protocols to focus less on report checking and more on problem solving and skills development (29). In a study in Odisha, those supervisors who had undergone the rigorously planned capacity building and demonstration exercise in the field were better able to transfer skills to the supervisees (15).
During the implementation of the project, distance to health facilities was one of the challenges that required mentors to spend a long time travelling, resulting in transportation becoming a common barrier. In many cases, mentors were required to share one vehicle with other people travelling for different reasons due to limited vehicle availability in order to decrease cost. Competing priorities also served as a challenge, as in some cases mentors were called to work on other works in the college. Security problems were also one of the challenges.

Limitations of the study
The sample size to observe a clinically significant difference couldn't be calculated which might have affected the statistical power of the study. Because some of the data were obtained through interviews, the information collected may be susceptible to interview bias.

Conclusion And Recommendations
Even though supportive supervision is regarded by the WHO as one of the five essential components of the Reaching Every District strategy, its implementation has been hampered by lack of capacity in the health system. To fill this implementation gap, we designed and tested, Capacity building and mentorship (CBMP) by well-trained personnel with the appropriate knowledge and skills have positive effects on immunization program.
The intervention package also improved the knowledge and skills of health workers.
Moreover, the study shows that most of the RED strategy components are well implemented; however, some elements need further improvements. Thus, it is