Profile of health facilities
In this study a total of 30 health facilities were approached. Among those health facilities 15(50%) were from Wogera district receiving the intervention and 15(50%) were from Dabat District as control group. Finally, a total of 28 facilities (13 from the intervention group and 15 from the control group) were included for analysis [Table 1].
Table 1
Health facilities included in the analysis
Health facilities | Wogera District (Intervention ) | Dabat District (Control ) |
Health center | 4 | 3 |
Health Post | 9 | 12 |
Total | 13 | 15 |
Profile of key informants
Key informant interviews were done on woreda health office head (1), CBMPI project supervisors (2), health center head (1), EPI focal persons in health centers (2) and health extension workers from health posts (2). Among the key informants 5 were males and 3 were females. The average age of the respondents was 32 years ranging from minimum of 28 years to maximum of 47 years. The respondents had on average 6 years’ experience in EPI related programs which ranged from 1 year to 14 years.
Implementation of individual indicators for RED Strategy
Table 2 shows the individual indicators and their weight factor on the various indicators per component indicator. In brief, each operational component of the RED strategy was evaluated by a number of individual indicators and a component indicator was calculated by summing the weights which were assigned to each individual indicator. The overall RED implementation score was determined by summing the five RED component indicators.
Table 2
Component and individual indicators with their respective weight factor in the study
Component indicator | Individual indicator | Individual indicator weight | Level | Pre | Post | P value |
Planning Management (n = 5) | 1. Having a micro-plan for immunization | 0.2 | Yes | 7(53.9%) | 12(92.3%) | 0.027 |
0 | No | 6(46.1%) | 1(7.7%) |
2. Having list of the communities with target population | 0.2 | Yes | 5(38.5%) | 9(69.2%) | 0.116 |
0 | No | 8(61.5%) | 4(30.8%) |
4. High risk areas identified | 0.2 | Yes | 7 (53.8%) | 13(100%) | 0.005 |
0 | No | 6(46.2%) | 0(0%) |
5. Having a plan to conduct outreach sessions | 0.2 | Yes | 5(38.4%) | 12(92.3%) | 0.004 |
0 | No | 8(61.6%) | 1(7.7%) |
6. Having a plan to conduct fixed vaccination sessions | 0.2 | Yes | 3(23.1%) | 4(30.8%) | 0.658 |
0 | No | 10(76.9%) | 9(69.2%) |
Use of data for action component n = 5 | 1. Having an EPI monitoring chart | 0.2 | Yes | 6(46.2%) | 13(100%) | 0.002 |
0 | No | 7(53.8) | 0(05) |
3. Attending district-level immunization review meetings | 0.2 | Yes | 12(92.3%) | 11(84.6%) | 0.539 |
0 | No | 1(7.7%) | 2(15.4%) |
5. Having a defaulter tracking system | 0.2 | Yes | 6(46.2%) | 13(100%) | 0.021 |
0 | No | 7(53.8%) | 0(0%) |
7. Providing complete and timely reports to district | 0.2 | Yes | 12(92.3%) | 13(100%) | 0.308 |
0 | No | 1(7.7%) | 0(0%) |
9. Receiving feedback on monthly reports | 0.2 | Yes | 10(76.9%) | 12(92.3%) | 0.277 |
0 | No | 3(23.1%) | 1(7.7%) |
Supportive Supervision (N = 2) | 1. Regular supervisory visit received | 0.5 | Yes | 5(38.5%) | 11(84.6%) | 0.016 |
0 | No | 8(61.5%) | 2(15.4%) |
3. Checklist available for supervision | 0.5 | Yes | 5(38.5%) | 12(92.3%) | 0.004 |
0 | No | 8(61.5%) | 1(7.7%) |
Outreach (N = 3) | 1. Outreach visits conducted | 0.33 | Yes | 8(61.5%) | 11(84.6%) | 0.185 |
0 | No | 5(38.5%) | 2(15.4%) |
3. Number of outreach more than 2 per month | 0.33 | Yes | 3(23.1%) | 9(69.2%) | 0.018 |
0 | No | 10(76.9%) | 4(30.8%) |
5. Having a strategy to address hard to reach areas | 0.33 | Yes | 8(61.5%) | 13(100%) | 0.013 |
0 | No | 5(38.5%) | 0(0%) |
Community (N = 3) | 1. Presence of social mobilisation activities | 0.33 | Yes | 12(92.3%) | 13 (100%) | 0.308 |
0 | No | 1(7.7%) | 0(0%) |
3. Planned meetings with the local leaders to discuss routine immunization | 0.33 | Yes | 7(53.8%) | 12(92.3%) | 0.027 |
0 | No | 6(46.2%) | 1(7.7%) |
3. Using village-level volunteers to mobilize beneficiaries | 0.33 | Yes | 9(69.2%) | 12(92.3%) | 0.135 |
0 | No | 4(30.8%) | 1(7.7%) |
Among the individual indicators under planning management component of the RED strategy, presence of micro plan (P-value = 0.027), identification of high risk areas (P-value = 0.005) and having plan for outreach sessions (P-value = 0.004) have shown a statistically significant improvement in implementation after the introduction of the CBMPI intervention package [Table 2].
In terms of using data for action, presence of EPI monitoring chart (P-Value = 0.002 and presence of defaulter tracing system (p-value = 0.021) were significantly improved after implementation of the intervention. Similarly, regularity of supportive supervision (P-value = 0.016) and availability of checklist (P-value = 0.004) during supervision has been improved significantly by the CBMPI program [Table 2].
Pertaining to community engagement component of the RED strategies, there was a statistically significant improvement in the intervention group in having planned meetings with the local leaders to discuss routine immunization activities after the introduction of the intervention (P-Value = 0.027) [Table 2].
RED Strategy implementation score for component indicators
Table 3 shows the mean score of each component indicator of the RED strategy in the study area before and after the implementation of the intervention package. The findings showed that the mean score for all the five component indictors has increased after the implementation of the intervention package in the study area. The overall RED strategy implementation score was 2.71 (range: 0.86–4.1) during pre-intervention while the score has increased to 4.36 (range: 3.39–4.99) during post intervention [Table 3].
Table 3
RED implementation score for component indicators for RED strategy during pre and post intervention implementation
S.No. | Component indicator | Pre intervention | Post intervention |
Maximum | Minimum | Mean | Maximum | Minimum | Mean |
1 | Planning Management | 1 | 0 | 0.42 | 1 | 0.4 | 0.76 |
2 | Use of data for action | 1 | 0.2 | 0.71 | 1 | 0.8 | 0.95 |
3 | Supportive supervision | 1 | 0 | 0.38 | 1 | 0 | 0.85 |
4 | Outreach | 1 | 0 | 0.48 | 1 | 0.33 | 0.84 |
5 | Community engagement | 1 | 0 | 0.71 | 1 | 0.66 | 0.94 |
6 | Overall red implementation score | 4.1 | 0.86 | 2.71 | 4.99 | 3.39 | 4.36 |
The use of data for action with mean score of 0.95 (range: 0.8-1) and community engagement with mean score of 0.94 (range: 0.66-1) component indictors had the highest score with better implementation after the introduction of the CBMPI project. On the other hand, planning management was relatively the least well implemented component indictor with mean score of 0.76 (range 0.4-1) [Table 3].
Effect of the CBMPI intervention on RED implementation strategy components
The findings of the study revealed that the CBMPI intervention package has a statistically significant effect on implementation of planning management, data use for action and out-reach components of the RED strategy. The difference in differences estimator is positive for planning and data use component which showed that health facilities in Wogera District receiving the intervention package have a statistically significant higher (0.3) implementation score than health facilities of Dabat district in the control group over time. The out-reach component of the RED implementation strategy was also better implemented in the intervention group with a statistically significant higher (0.5) implementation score (P-Value = 0.001). In general, the intervention package has a positive statically significant effect on over all implementation of RED strategy (P-Value = 0.0001) [Table 4].
Table 4
The effect of the CBMPI intervention package on RED implementation strategy components
S.No. | Indicator | DID estimator (Intervention*Time) | 95% CI | P-Value |
1 | Planning management | 0.314 | [0.027–0.6003] | 0.032 |
2 | Data use for action | 0.326 | [0.117–0.534] | 0.003 |
3 | Supervision | 0.061 | [-0.374-0.497] | 0.778 |
4 | Out reach | 0.511 | [0.217–0.805] | 0.001 |
5 | Community | 0.032 | [-0.234-0.297] | 0.812 |
6 | Over all RED implementation | 1.244 | [0.507–1.981] | 0.001 |
Effect of the CBMPI intervention package on vaccination service delivery outcomes
The difference in differences estimator is positive for Penta III coverage and shows that health facilities in Wogera woreda receiving the intervention package have a statically significant higher (17.4%) Penta III coverage during 2010 to 2011 than health facilities of Dabat district in the control group over time. Similarly, the difference in differences estimator is positive for full vaccination coverage and shows that health facilities in Wogera woreda receiving the intervention package have a statically significant higher (16.6%) full vaccination coverage during 2010 to 2011 than health facilities of Dabat district in the control group over time. On the other hand, the intervention package reduced the Penta III dropout rate in the intervention group significantly. Accordingly, the Penta drop-out rate has reduced on average by 2.6% in the intervention group than the control group during 2010 to 2011 over time [Table 5].
Table 5
The effect of the CBMPI intervention package on vaccination service delivery outcomes
S.No. | Indicators | DID estimator (Intervention*Time) | 95% CI | P-Value |
1 | Penta III coverage | 17.4159 | [2.037, 32.793] | 0.027 |
2 | Polio III coverage | 17.31128 | [1.921, 32.700] | 0.028 |
3 | Penta dropout coverage | -2.672821 | [-4.398, -0.947] | 0.003 |
4 | Full vaccination coverage | 16.63179 | [1.33, 31.928] | 0.034 |
Knowledge of health workers on cold chain management
In this study knowledge of health workers working in the immunization program of each health facility were asked about the correct methods of maintaining the cold chain system. Accordingly, after the implementation of the intervention package 46% of the respondents correctly explained the expected distance of ILR and DF from the wall (P-value = 0.011) which showed a statistically significant difference as compared with pre implementation [Table 6].
Table 6
Knowledge of health workers pre and post intervention implementation
S.No. | Item | Correct answer | Pre | Post | Chi-Square, P-Value |
1 | Distance of ILR and DF from the wall | Yes | 1(7.7%) | 6(46.1%) | 0.011 |
No | 12(92.3%) | 7(53.9%) |
2 | Temperature log maintenance | Yes | 6(46.1%) | 11(84.6%) | 0.039 |
No | 7(53.9%) | 2(15.4%) |
3 | Appropriate cabinet temperature for ILRs | Yes | 5(38.5%) | 11(84.6%) | 0.016 |
No | 8(61.6%) | 2(15.4%) |
4 | How many hours before distribution, diluents be placed in ILR point | Yes | 6(46.1%) | 12(92.3%) | 0.011 |
No | 7(53.9%) | 1(7.6%) |
5 | Correct manner of placing Ice packs inside DF | Yes | 6(46.1%) | 10(76.9%) | 0.107 |
No | 7(53.9%) | 3(23.1%) |
Similarly, the knowledge of health workers on correct temperature log maintenance (P-value = 0.39), appropriate cabinet temperature for ILRs (P-value = 0.016) and the correct placement of diluents in ILR (P-value = 0.011) were significantly improved after implementation of the CBMPI intervention package [Table 6].
Assessment of skills and practice of health workers during child immunization
Assessment of the skills and practice of health workers were assessed for five children at each health facility. Accordingly, provision of appropriate vaccines (P-value = 0.027), communication on key vaccination messages (p-value-0.005) and safe disposal of used syringes/needles (P-value = 0.004) were performed with correct procedures after the implementation of the CBMPI intervention package. While, administration of vaccines with the correct route (P-value = 0.141) and touching/ recapping of needles by vaccinators didn’t show a statistically significant difference after the implementation of the intervention package [Table 7].
Table 7
Skills and practice of health workers during child immunization measured pre and post intervention implementation
Components | Correct procedure for all children | Intervention districts | P-Value |
Pre | Post |
All the child received appropriate vaccines* | Yes | 7(53.9%) | 12(92.3%) | 0.027 |
No | 6(46.1%) | 1(7.7%) |
The vaccinator communicate with the mother/care giver on the key messages | Yes | 4(30.8%) | 11(84.6%) | 0.005 |
No | 9(69.2%) | 2(15.4%) |
The vaccinator touch or recap the needle | Yes | 5(38.5%) | 2(15.4%) | 0.185 |
No | 8(61.5%) | 11(84.6%) |
Each vaccine administered using the correct route for the vaccine | Yes | 11(84.6%) | 13(100%) | 0.141 |
No | 2(15.4%) | 0(0%) |
Used syringes/needles disposed of in safety boxes | Yes | 5(38.5%) | 12(92.3%) | 0.004 |
No | 8(61.5%) | 1(7.7%) |
* Five children were observed during vaccination sessions |
Qualitative results using RE-AIM implementation science framework
The findings from the implementation science qualitative synthesis were presented using RE-AIM framework as follows.
REACH
All the eight key informants were willing to participate in the CBMPI program and explained that the CBMPI program is reachable to all health facilities. One of the health center head reported that “I believed that the CBMPI project will give us better support and improvement in our health facility’s immunization program which is actually reachable to our health posts and other health facilities.”
EFFECTIVENESS
The qualitative findings also indicated that the use of data for action have been changed in supervised health facilities since the introduction of the new CBMPI program. One of the supervisor stated that “for Immunization, the EPI monitoring chart is vital and it was not updated. With our support, we let the health workers to use the EPI immunization chart and other data’s for decision making. For example, health workers can see the EPI monitoring chart to assess their performance and identify defaulters from the routine immunization schedules.” The Health center head also explained that; “With the support, our documentation has been improved including our registration on log books and as well on vaccine stock status management. With this we are being using our collected data for decision making especially the EPI monitoring chart.”
Majority of the respondents mentioned that the community engagement and communication with caregivers during vaccination sessions had been done routinely during the CBMPI program implementation by leveraging the existing health development army. One health extension worker mentioned that; “With the help of CBMPI program supervisors, we were strictly educating mothers on anticipated vaccination side effects so that they will not drop from the vaccination schedules.”
The CBMPI program has also brought remarkable change on the cold chain system. The cold chain management was improved significantly in fulfilling icepacks and foam pads in vaccine carriers during vaccine transportation at health post level. The health extension worker elaborated that; “Initially, we were using 2 icepacks for vaccine transport. The supervisors taught us to use the four icepacks in vaccine carrier. Since then, we are using four ice packs to transport vaccines from the health center every month. They also showed us how to arrange and manage vaccine to keep them potent. One of the supervisors also complimented that; “The cold chain was one of the major areas with gaps where we have been giving support.
In terms of fixed and outreach sessions, all the respondents agreed that there had been continuous support and change in the fixed and outreach EPI programs after the introduction of the CBMPI program.
ADOPTION
Regarding program acceptance, majority of the key informants mentioned that adopting the new intervention package in the routine immunization program was acceptable and the commitment of health facility staffs to accept the new program was encouraging. One of the supervisor mentioned that “the health workers were committed and supportive in implementing the CBMPI project. At the start some of health extension workers were believing that we went there for inspection, gradually after understanding the objective of the project, they had good attitudes for the program and we brought change in the EPI service delivery in collaborative manner.”
IMPLEMENTATION
All key informants explained that the CBMPI program has been implemented very well in a way that could bring change. Regarding the implementation of the intervention package as intended, one of the supervisors reported that “The mentorship was conducted very well. Mentorship brings change immediately. We gave them the activities to be done in one round has been improved for the next round.” The Woreda health office supervisor also complimented that; “The supervision was by team and even we shared experiences among the team members.
Contributing factors for success of the program implementation:
The overall coordination of the project and the support from health workers were reported as one of the contributing factors for the success of the CBMPI program implementation. One of the supervisor explained that “The CBMPI program was well organized and implemented as planned. Among the reasons for the program effectiveness, one supervisor reported that “The program was implemented continually and the health workers were supportive as well. In terms of outcome there was an increment in service coverage and decrease in drop-out rates.” The woreda supervisor also reported that: “It is effective. The first thing that makes it effective is that the University physicians and other staffs had continues supervision and support system with development of action plans and its implementation.” The woreda health office head also reported that; what makes the program successful is the coordination process of the implementation. Additionally, the supervisors had also the necessary skill and knowledge for the mentorship program.”
MAINTENANCE