A mixed-methods evaluation of a cluster randomized controlled trial on simulation-based low-dose, high-frequency plus mobile mentoring training versus traditional group-based training of health workers in Ebonyi and Kogi States, Nigeria

Background: There are few studies from low- and middle-income countries on learning outcomes among health workers who have been trained on day of birth care using onsite, simulation-based, low-dose, high frequency (LDHF) training plus mobile (m) mentoring. The aim of this study was to compare their knowledge and skills competencies with those of health workers trained using the traditional offsite, group-based training (TRAD) approach in Kogi and Ebonyi states, Nigeria, over a 12-month period. Methods: We conducted a prospective cluster randomized controlled trial, enrolling 299 health workers in 60 health facilities in Kogi and Ebonyi states, randomized to either LDHF/m-mentoring (intervention, n=30 facilities) or traditional group-based training (TRAD, n=30 facilities) control arm. Health workers in both arms received basic emergency obstetric and newborn care training with simulated practice using anatomic models and role-plays. The control arm participants were trained offsite while the intervention arm were trained onsite where they work. Mentorship was done through telephone calls and reminder text messages. The multiple choice questions and objective structured clinical examinations mean scores were compared; p-value <0.05 was considered statistically significant. Qualitative data were collected and analyzed along themes of interest. Results: The mean knowledge scores between the two arms at months 3 and 12 post-training were equally high; no statistically significant differences. Both arms showed improvements in composite scores for assessed BEmONC clinical skills from around 30% at baseline to 75% and above at endline (p <0.05). Overall, the observed improvement and retention of skills was higher in intervention arm compared to the control arm at 12 months post-training, (p<0.05). Some LDHF/m-mentoring approach trainees reported that mentors’ support improved their acquisition and maintenance of knowledge and skills, which may have led to reductions in maternal and newborn deaths in their facilities. Conclusion: The LDHF/m-mentoring intervention is more effective than TRAD approach in improving health workers’ skills acquisition and retention. Health care managers should have the option to select the LDHF/m-mentoring learning approach, depending on their country’s priorities or context, as birth (including care of the newborn), AMTSL, manual removal of placenta, bimanual compression of the uterus, abdominal aortic compression, pre-eclampsia/eclampsia management (p<0.05).

it ensures health workers remain in their place of work during training events thus less disruption to service delivery.

Background
Trainings of health workers in Nigeria commonly adopt the traditional lecture-based, offsite approach with a limited number participants from any health facility at a time. This training method can be expensive cost wise and in terms of interrupting service deliver when health workers are absent from their workstations for long duration [1][2][3][4]. The rising rates at which health workers are absent from work to attend offsite training is a concern for health facility managers [3,4]. Besides, the few numbers of health workers who can be trained in at a particular time thus only a few of them can benefit from the through these training updates and ultimately, the quality of care may not be improved to the desired level [2]. After the offsite training, the healthcare workers may be unable to practice new or updated skills because the entire team members have not been trained. Therefore, step-down training by those that were trained upon return to their place of work most often does not happen as intended due to lack of will and management support among other factors. This may present a barrier to the practice of new or updated skills by the health workers who received training.
Moreover when there are health workers turnover and attrition over time as well as movement to other sectors, the quality of care provided remains poor if they involved those who received training.
Consequently, maternal and newborn health outcomes have remained suboptimal despite trainings [2].
There are few studies from low-and middle-income countries (LMICs) on learning outcomes and facilitators and barriers to learning approaches on day-of-birth care among maternal and newborn health workers trained using onsite simulation-based low-dose, high-frequency (LDHF) plus mobile (m)-mentoring training approach compared to those trained by offsite traditional group-based training (TRAD). The LDHF/m-mentoring approach is described in the study protocol manuscript; it differ from the TRAD approach primarily by being onsite, more interactive, and team-oriented thus more suited to meet the needs of the trainees in various contextual situations [5]. The LDHF/m-mentoring approach also uses mobile phone mentorship calls and reminder text messages to the trainees.
There are low numbers and quality of skilled birth attendants in Nigeria. Therefore evidence-based approach is needed to train health workers to improve competencies and maternal/newborn outcomes [6,7]. Doctors, nurses and community health extension workers (CHEWs) are commonly trained on day of birth care. The CHEWs as a result of task sharing in Nigeria are also trained and posted to primary health centers with or without nurses. The training of CHEWs did not originally involve training on day of birth care at maternity. They should be able to initiate life-saving procedures if they are trained on basic emergency obstetric and newborn care (BEmONC) functions such as management of bleeding after birth and newborn resuscitation.
Considering the high maternal and newborn mortality and morbidity figures in Nigeria, there is need to train birth attendants on high-impact interventions using competency-based approaches to improve the quality of maternal and newborn care and reduce maternal and perinatal/neonatal mortality [8][9][10][11].
There is limited evidence from developing countries including Nigeria on facilitators, barriers, and effectiveness of simulation-based LDHF/m-mentoring learning approaches in improving maternal and newborn health, including day of birth care. In the few studies where the LDHF/m-mentoring approach has been tested, there sample sizes have been limited and methodological approaches have not been standardized [7]. it is therefore for research on capacity building of workers to incorporate in their designs culturally appropriate and effective techniques to improve the competencies health workers.
The studies should be adequately powered and use rigorous designs to answer the research questions [7].

Study Objectives
The aim of this study was to compare the knowledge and skills competencies of providers who attended training in LDHF/m-mentoring with TRAD approaches in improving maternal and newborn care on the day of birth in Ebonyi and Kogi states, Nigeria. The study's primary outcomes were increase in knowledge, clinical skills in BEmONC, and retention of clinical competency at 3-and 12months post-training. The secondary outcomes measured were facilitators of and barriers to the Our hypothesis was that LDHF/m-mentoring results in better knowledge and skills outcomes compared to the TRAD approach. The study was designed to answer three research questions:

1.
How do knowledge learning outcomes of skilled birth attendants (SBA) in Kogi and Ebonyi states who were exposed to simulation-based LDHF/m-mentoring training approach differ from those exposed to TRAD approaches over 12 months? 2.
How do skills learning outcomes of skilled birth attendants in Kogi and Ebonyi states who were exposed to simulation-based LDHF/m-mentoring training approaches differ from those exposed to TRAD approaches over 12 months?

3.
What are the facilitators of and barriers to the LDHF/m-mentoring training approach at individual and institutional levels?

Study design
This mixed-methods study used a prospective cluster randomized controlled trial design and qualitative methods that included focus group discussions (FGDs) and in-depth interviews (IDIs). The 60 health facilities were randomly selected and assigned to either the intervention arm or the control arm as described in the study protocol [5]. Trained assessors with a clinical background training administered questionnaires for knowledge test, observation checklists for the objective structured clinical examinations (OSCEs). Additionally, IDIs, and FGDs were used for qualitative data collection.
BEmONC training packages were adapted from previous studies and training documents [1,2]. The tools were pre-tested among 25 health workers from health facilities that were not part of the study.
Before being used, checklists for the assessed clinical skills were modified in accordance with the Nigeria context by the research team and data collectors. At the time of the study, MCSP did not support any interventions in these facilities that are likely to cause contamination. Other quality improvement interventions only happened at the non-study facilities at the time of the study.
The details of the study methodology, including training approaches in the two study arms have been extensively described below and in the study protocol [5]. In brief, for both study arms, independent assessors who were blinded to the training approaches used did skills and knowledge assessments.
The assessors underwent a 5-day clinical skills standardization training. Training of trainers was also done.

Eligibility criteria
Health facilities: selection of health facilities was based on a sampling frame of the 120 facilities supported by MCSP in the two states across all three geopolitical zones per state and included all three levels of health care delivery, primary (primary health care/private), secondary (general/missions), and tertiary.
Health workers: individual study participants were drawn from among those attending to women during labor and delivery in the selected health facilities in the two states. Participating health workers included doctors, nurses, midwives, and community health extension workers. Selected health workers were those who had spent at least six months in the health facility providing maternal and/or newborn care services and were available to participate in the trainings from the beginning to the end of the study (a 12-month period).

Randomization
The unit of randomization was the health facility. The 60 facilities were divided into nine strata by three geopolitical zones and by the level of health facility. Each stratum was randomized to either control or intervention arm using randomly permuted blocks in a ratio of 1:1 to achieve balance in the type of facility and location for each study arm. Health facilities were matched based on locations and levels of care prior to randomization and grouped as either receiving LDHF/m-mentoring or TRAD intervention. As there are, few skilled birth attendants in some health facilities, all health workers employed in the maternity or newborn units in such facilities who met the inclusion criteria were selected from the randomized health care facilities.

Training approach 1 and data collection: Simulation-based LDHF/m-mentoring training of participants -group 1 or intervention group
The training for the LDHF/m-mentoring arm was for the entire team of service providers available at the health facility, but only those who met the study inclusion criteria were assessed. After consent was obtained, the participants completed pre-training assessments consisting of MCQs and OSCEs done through use of anatomical models, to assess their baseline knowledge and skills in BEmONC.
The MCQ test comprised a set questions to test trainees' knowledge while OSCEs were used to assess

Sample size
The sample size was calculated as the average number of health workers who needed to be included from each of the 60 facilities (30 per study each arm) using a test of two proportions for a binary indicator of a present competency score. Competency was assumed at a score of 80% or higher on both the knowledge and skills assessments. We assumed that the proportion of competent providers would be 50% in the control group. The goal was to have at least 80% power to reject a null hypothesis that the proportions are equal across the two arms against the alternative hypothesis of a minimum effect size of 20 percentage point difference in proportions of competent providers. The test of hypothesis was performed at 0.05 level of significance. Sample size was calculated using PASS statistical software. Since some facility-level factors that are shared by the providers working in the same facility can influence competency level, we assumed 0.05 within facility correlation of competency levels across providers in each health facility. Assuming an average of four health workers sampled per health facility, a total of 240 providers across the two groups was needed so that there would be at least 80% power to detect a 20 percentage point difference in competence level between the two arms (assessment at 3 and 12 months) at 0.05 level of statistical significance.
The final target sample size was 300 (or 150 providers per study arm) to account for potential 20% drop-out at the time of post-training assessment. More providers were to be recruited from Kogi State since it has a larger population than Ebonyi. Study participants recruited from each facility included the maternity unit head, wherever possible, and two others to ensure that the other team members received the necessary support to practice the competencies.

Data analysis
The Qualitative data, notes, and transcripts were entered into Atlas-ti version 8 software. All data were collected in English since the service providers have above basic level of education. Content analysis of the discussions was undertaken to generate themes of interest along which the analysis was done.
The analytic process proceeded in two basic steps: using the thematic groups, emerging themes were summarized related to relevant specific objectives, and content was analyzed to compare the three different respondent groups across the two states.

Results
A total of 299 participants completed the study out of the original 323 providers who were randomized to different study arms; LDHF arm=172; TRAD arm=127 (Table 1 and Figure 1). Both arms had similar socio-demographic characteristics. Table 2 shows that the TRAD arm had better knowledge test scores compared to the LDHF/mmentoring arm at baseline and immediately after the training intervention in some thematic areas, such as AMTSL, essential newborn care, and neonatal resuscitation (p <0.05). However, at 3 and 12 months after training assessment, both arms are equal in knowledge acquisition and retention; no statistically significant differences were noted.

Comparison of Skills Competency Scores
Health workers in both arms showed improvement in overall pass rates in clinical skills competency, improving from around 30% at baseline to 75% and above at endline; difference-in-differences were

Provider Perspectives on the LDHF/m-mentoring approach
A total of 40 IDIs were conducted among 20 peer practice coordinators, 14 master mentors, four health facility managers, two ministry of health officers. In addition, six FGDs comprising eight to 10 participants each who were selected from among the trainees at 12 months post-training. During the qualitative data collection, some participants commented in favor of the LDHF approach to capacity building. Some noted that the support provided by master mentors via phone facilitated the approach. One of the mentors reported that the support provided helped solve challenges:

Discussion
In this cluster randomized control trial, the knowledge assessment scores increase from baseline to endline at 12 months between the study arms, but the acquisition and retention of BEmONC skills was different at 12 months with the LDHF/m-mentoring arm demonstrating better performance than the TRAD arm. This study contributes to the much needed evidence base for LDHF/m-mentoring training approach in LMICs, adding on the finding of related studies done in Uganda [1] and Ghana [2]. As part of the intervention package, LDHF/m-mentoring study arm participants received weekly SMS reminders and quizzes along with continuous skills practice on anatomical models, which helped, reinforce knowledge on specific clinical areas over time. In contrast, the TRAD arm participants, which represent the status-quo, did not have the extra support, which might explain the differences in skills performance at 12 months. Furthermore, information gathered qualitatively suggests that a key benefit of m-mentoring is the ability of trainees to call their mentors at the point of care to address uncommon complications such as placenta praevia. This close mentor-mentee relationship might also have helped the mentees to develop confidence in these crucial skills, which demonstrates a great potential of the LDHF/m-mentoring approach.
Overall, the LDHF arm had better skills performance at all the assessment points compared to the TRAD arm. Of note, there was an observed drop in skills competency at three months in both arms from the immediate post-training assessment. Some of the possible explanation for the drop in skills competency is that it takes time following intense training, before a practice has been established as routine. Another possible cause for the drop may have been that there was a health workers' strike that was ongoing just before the post-training assessments in the two states. Despite these factors, the LDHF/m-mentoring arm still performed better in all competencies at 3-and 12-months posttraining. It's encouraging to note that this observed better performance was in competences that are very crucial for the survival of the mother and the baby on the day of birth; these include, AMTSL, manual removal of placenta, neonatal resuscitation, loading dose of magnesium sulphate, and preeclampsia/eclampsia. The continued simulation practice may have helped in building and retaining these competencies.
Studies on BEmONC in-service training have demonstrated the value of simulation and continuous practice in improving knowledge and skills competency [1,2,7]. This has led to adaptation of various approaches to determine which ones lead to the greatest returns on investment [1][2][3][4][8][9][10][11]. Our findings are consistent with those of previous studies that reported TRAD offsite training may improve the trainees' competencies, but knowledge and skills may not be transferred to other co-workers at the facility, nor translated into practice or performance [12,13]. On the other hand, methods that involve repetitive, mentor-supported learning and regularly coordinated practice sessions, such as the onsite LDHF/m-mentoring training, result in better skills acquisition and retention over time [1,2,5,7,[14][15][16][17]. This claim has been supported by other rigorously, well-designed studies that showed that onsite LDHF/m-mentoring training approach, coupled with an efficient mentorship program, resulted in improved provider preparedness to deal with complications [1,2,15].
Findings from the Ghana study [2] that measured patient-level outcomes showed that the LDHF/mmentoring training approach was associated with a sustained decrease in facility-based newborn mortality and intrapartum stillbirths [2]. The limitations acknowledged by Gomez and colleagues in the Ghana study, such as the lack of a randomized control trial design and lack of concealment/information bias, have been addressed in our study using a cluster randomized controlled trial in a similar low-resource setting. Although our study did not measure patient-level outcomes, we assessed a crucial component of the day of birth, namely the competency of the service providers. Our study was adequately powered to produce generalizable conclusions on the comparative effectiveness of the LDHF/m-mentoring and TRAD approaches in the study setting and possibly in similar environments especially in LMICs. As reported by other studies, to be effective, training programs should be conducted as close as possible to the workplace and be competencybased [16][17][18].
The present study also adds to the growing yet still insufficient body of evidence in LMICs supporting the use of simulation, specifically for training in the management of obstetrics and newborn emergencies [19][20][21]. Skills that are not used very frequently, such as management of obstetric and newborn complications, are likely to be non-existent or lost due to a lack of use, and therefore the need to practice often. The appropriate dose or frequency of practice in order to reach competency level depends on many factors, which may include the personal attributes of service provider among other things. Simulation-based trainings provide participants with the opportunity to repeat practice sessions in a safe environment where they will not harm patients [1-2, 18, 21]. Using the workplace to practice skills is beneficial. The availability of the anatomical models for continuous onsite simulation practice sessions also helps to organize emergency drills to test the health systems response to lifethreatening complications.
Another important finding is that some of the enablers for the LDHF/m-mentoring approach are related to successful outcomes that were reported as health workers improved their skills, which motivated the participants to practice more and seek support from master mentors. On the other hand, some of the barriers to this approach include logistics, such as availability of space for the practice sections and labor strikes by health workers. It has been reported that inadequate mentorship should be addressed to improve quality of BEmONC [22]. To function maximally, health workers need to improve their competency and they need an enabling environment, including availability of supplies, equipment, appropriate guidelines and policies, and timely remuneration [23,24]. Our study findings corroborated those of previous studies in LMICs on the effectiveness of the LDHF training approach compared to the offsite traditional training approach to build health workers capacity [1,2].
This study had some limitations some of which have been reported through the qualitative data presented as challenges and barriers experienced. These include some operational-level logistical challenges in ensuring that the mentors always had the airtime in a timely manner in order to maintain seamless communication with the mentees. The was mitigated by the mentors being very understanding by using their resources as they awaited support provided through the study. Another limitation was the health workers' strike which occurred around month three post-training, which potentially affected the findings negatively at that assessment period. Finally, although the study team had lobbied the MOH not to transfer the participants to other non-study facilities for duration of the study, some staff were transferred. However, the proportion was very low; they were not included in the analysis.

Conclusion
This is one of the few studies from a middle-income country that has used a pragmatic design-a cluster randomized controlled trial and qualitative methods-to generate much needed information on learning outcomes of two approaches to health worker capacity building. Our study findings corroborate those of similar studies that used rigorous methodology, to support the proposition to a shift from traditional offsite group-based training to low-dose, high-frequency and onsite training as an effective way to build the capacity of maternal and newborn care providers. This shift can be explored in other clinical settings to improve the quality of care. The findings suggest that a low-dose, high frequency and simulation-and practice-based training approach, including mobile mentoring is likely to result in better skills competency outcomes and higher skills retention among health workers.
This study has shown promising results that can address health worker absenteeism due to frequent offsite trainings, which are a major concern for health facility managers in LMICs. Training staff onsite without taking them away from service delivery points is a strong reason for adoption of the LDHF/mmentoring approach.
The findings are expected to provide health facility managers and other decision makers with valuable information to guide resource allocation for maximum benefit and effectiveness in building the capacity of frontline health workers. The authors recommend that capacity building among healthcare workers, to the extent possible, should utilize the onsite LDHF/m-mentoring learning approach. Further evidence is needed to determine the value of m-mentoring in influencing learning outcomes and the cost-effectiveness of both training approaches as these were not explored in our study. Review Board #000007196, on August 23rd, 2016. The trial was retrospectively registered with the NIH-Sponsored ClinicalTrials.Gov registry on 24th August, 2017: NCT03269240. Verbal informed consent was obtained from study participants and confidentiality was assured. Verbal informed consent was used because the authors thought that asking health providers to review and sign forms might be considered risky because of the fear that if they performed poorly their supervisors may see their scores and trace it back to them.

Consent for publication
Not applicable Figure 1 Consort flowchart of enrolment of study participants and data analysis Trends in BEmONC skills mean composite scores by study arm over 12-month period Assessment of providers' BEmONC skills at 12 months post-training