Study Objectives
The aim of this study was to compare the knowledge and skills competencies of providers who attended training in LDHF/m-mentoring with TRAD approaches in improving maternal and newborn care on the day of birth in Ebonyi and Kogi states, Nigeria. The study’s primary outcomes were increase in knowledge, clinical skills in BEmONC, and retention of clinical competency at 3- and 12-months post-training. The secondary outcomes measured were facilitators of and barriers to the LDHF/m-mentoring training approach at individual and institutional levels.
Our hypothesis was that LDHF/m-mentoring results in better knowledge and skills outcomes compared to the TRAD approach. The study was designed to answer three research questions:
- How do knowledge learning outcomes of skilled birth attendants (SBA) in Kogi and Ebonyi states who were exposed to simulation-based LDHF/m-mentoring training approach differ from those exposed to TRAD approaches over 12 months?
- How do skills learning outcomes of skilled birth attendants in Kogi and Ebonyi states who were exposed to simulation-based LDHF/m-mentoring training approaches differ from those exposed to TRAD approaches over 12 months?
- What are the facilitators of and barriers to the LDHF/m-mentoring training approach at individual and institutional levels?
Study setting
Nigeria, with a population of 185.7 million as of 2017 (according to Nigeria Bureau of Statistics), has six geopolitical zones and 36 states plus the Federal Capital Territory (FCT). The study was conducted in two states: Ebonyi state has a population of 2.8 million and is in the South-East zone; Kogi state has a population of 4.3 million and is in the North-Central zone [5]. At the time of the study, the Maternal and Child Survival Program (MCSP), in partnership with the Federal Ministry of Health, Ebonyi and Kogi States Ministries of Health and Professional Associations supported 120 health facilities across the two states, of which 60 were selected to be part of the study, 30 in each state. The MCSP is a global, $560 million, 5-year cooperative agreement funded by the United States Agency for International Development (USAID) to introduce and support scale-up of high-impact health interventions among USAID’s 25 maternal and child health priority countries, as well as other countries.
Study design
This mixed-methods study used a prospective cluster randomized controlled trial design and qualitative methods that included focus group discussions (FGDs) and in-depth interviews (IDIs). The 60 health facilities were randomly selected and assigned to either the intervention arm or the control arm as described in the study protocol [5]. Trained assessors with a clinical background training administered questionnaires for knowledge test, observation checklists for the objective structured clinical examinations (OSCEs). Additionally, IDIs, and FGDs were used for qualitative data collection. BEmONC training packages were adapted from previous studies and training documents [1,2]. The tools were pre-tested among 25 health workers from health facilities that were not part of the study. Before being used, checklists for the assessed clinical skills were modified in accordance with the Nigeria context by the research team and data collectors. At the time of the study, MCSP did not support any interventions in these facilities that are likely to cause contamination. Other quality improvement interventions only happened at the non-study facilities at the time of the study.
The details of the study methodology, including training approaches in the two study arms have been extensively described below and in the study protocol [5]. In brief, for both study arms, independent assessors who were blinded to the training approaches used did skills and knowledge assessments. The assessors underwent a 5-day clinical skills standardization training. Training of trainers was also done.
Eligibility criteria
Health facilities: selection of health facilities was based on a sampling frame of the 120 facilities supported by MCSP in the two states across all three geopolitical zones per state and included all three levels of health care delivery, primary (primary health care/private), secondary (general/missions), and tertiary.
Health workers: individual study participants were drawn from among those attending to women during labor and delivery in the selected health facilities in the two states. Participating health workers included doctors, nurses, midwives, and community health extension workers. Selected health workers were those who had spent at least six months in the health facility providing maternal and/or newborn care services and were available to participate in the trainings from the beginning to the end of the study (a 12-month period).
Randomization
The unit of randomization was the health facility. The 60 facilities were divided into nine strata by three geopolitical zones and by the level of health facility. Each stratum was randomized to either control or intervention arm using randomly permuted blocks in a ratio of 1:1 to achieve balance in the type of facility and location for each study arm. Health facilities were matched based on locations and levels of care prior to randomization and grouped as either receiving LDHF/m-mentoring or TRAD intervention. As there are, few skilled birth attendants in some health facilities, all health workers employed in the maternity or newborn units in such facilities who met the inclusion criteria were selected from the randomized health care facilities.
Training approach 1 and data collection: Simulation-based LDHF/m-mentoring training of participants - group 1 or intervention group
The training for the LDHF/m-mentoring arm was for the entire team of service providers available at the health facility, but only those who met the study inclusion criteria were assessed. After consent was obtained, the participants completed pre-training assessments consisting of MCQs and OSCEs done through use of anatomical models, to assess their baseline knowledge and skills in BEmONC. The MCQ test comprised a set questions to test trainees’ knowledge while OSCEs were used to assess skills competency. The assessments tested their knowledge and skills on conduct of normal delivery, active management of the third stage of labor (AMTSL), neonatal resuscitation, case management of pre-eclampsia and eclampsia (PEE) and management of PPH (e.g. manual removal of placenta, internal bimanual uterine compression and compression of the abdominal aorta). The training was divided into two “low-dose” training courses of four days each, and included additional time for assessment as needed and was conducted at the health facilities using the adapted BEmONC training package repeated after 1 month. The training techniques were modified to focus more onpractice. Time spent on lectures was reduced and additional time spent on hands-on practice. Peer Practice Coordinators (PPCs) were selected from among the trainees. These are people who had shown exceptional interest on the course and had received technical update in LDHF including the use of session plans, case scenario and MamaNatalie/NeoNatalie models to conduct simulation practices. At the end of the four-day training, the participants underwent an immediate post-training assessment, which included MCQs and OSCEs. The questions answered correctly, and procedures done competently were scored over a total of 100%. A test score of ≥80% is accepted as level of competence. During the one-month intervals between training courses, health care providers had the opportunity to practice what they learned and reinforced their competencies through high-frequency simulation-based practices of 2–3 times per week. The PPCs completed the practice logs and submitted them to the study coordinator. In addition to the simulation exercises, all the trained providers in the LDHF arm participated in mobile Mentoring, , which consisted of receiving weekly reminder messages and quizzes on the topics reviewed via SMS.. Also, the PPCs received structured, monthly half-hour mentoring calls from a trainer/master mentor that provide remote support, answering questions, providing guidance and reinforcing key messages. Skills and knowledge assessments were done at three time points post-training (immediate post training day, at three months 12 months). Trainees’ satisfaction with the simulation-based LDHF/m-mentoring training approach will also be determined using satisfaction (quantitative) survey (5).
Qualitative data were collected through six focus group discussions (FGDs) comprising 8–10 participants per group who were purposively selected from among the trainees at 12 months. The FGDs focused on experiences and level of satisfaction of trainees with LDHF/m-mentoring training approach, the high-frequency practice sessions with simulators, mobile mentoring through SMS and quizzes, opinions about changes in clinical practice and outcomes and overall impressions of the LDHF/m-mentoring approach and what could be improved. Each FGD lasted 60–90 minutes. In-depth interviews were also conducted with PPCs and trainers at 12 months post-training by study staff. Additionally, IDIs were conducted among PPCs aimed at elucidating their insights and experience managing simulator practice sessions for their facility, interacting and working with the trainers/master mentors, m-mentoring, changes in clinical practice and outcomes, success and challenges and overall impression about LDHF approach. For the trainers, their thoughts and experience with the LDHF/m-mentoring training approach, successes and challenges with mobile mentoring and supporting peer practice coordinators, opinions on the effectiveness of m-mentoring, confidence in their role as trainers, successes and challenges in collecting data was sought. Each IDI lasted about 45 minutes and was audio recorded.
Training approach 2 and data collection: Traditional group training of participants – Group 2 or control group
The traditional training approach consisted of eight days of lectures and practice sessions on simulators, offsite the participants’ workplace, usually in a hotel. After obtaining consent, participants completed pre-training assessments consisting of MCQs and OSCEs; the latter were administered through use of anatomical models. . The same modules offered to the LDHF/m-mentoting arm were offered and assed in the TRAD arm, all during the eight days training period. At the end of the eight-day training, the participants underwent an immediate post-training assessment, which included MCQs and OSCEs. As described in study arm 1, all questions answered correctly and procedure done competently were scored out of 100%.
Sample size
The sample size was calculated as the average number of health workers who needed to be included from each of the 60 facilities (30 per study each arm) using a test of two proportions for a binary indicator of a present competency score. Competency was assumed at a score of 80% or higher on both the knowledge and skills assessments. We assumed that the proportion of competent providers would be 50% in the control group. The goal was to have at least 80% power to reject a null hypothesis that the proportions are equal across the two arms against the alternative hypothesis of a minimum effect size of 20 percentage point difference in proportions of competent providers. The test of hypothesis was performed at 0.05 level of significance. Sample size was calculated using PASS statistical software. Since some facility-level factors that are shared by the providers working in the same facility can influence competency level, we assumed 0.05 within facility correlation of competency levels across providers in each health facility. Assuming an average of four health workers sampled per health facility, a total of 240 providers across the two groups was needed so that there would be at least 80% power to detect a 20 percentage point difference in competence level between the two arms (assessment at 3 and 12 months) at 0.05 level of statistical significance. The final target sample size was 300 (or 150 providers per study arm) to account for potential 20% drop-out at the time of post-training assessment. More providers were to be recruited from Kogi State since it has a larger population than Ebonyi. Study participants recruited from each facility included the maternity unit head, wherever possible, and two others to ensure that the other team members received the necessary support to practice the competencies.
Data analysis
The primary outcome was health workers’ clinical competency level in selected BEmONC skills at three months post-training assessed through simulation. The e secondary outcome was retention in clinical competency level in BEmONC skills as assessed through OSCEs and multiple choice knowledge questionnaires scores at 12 months post-training. The results presented in this article are based on assessments three and 12 months after training. For both arms, composite scores were computed for infection prevention, conduct of normal delivery, active management of the third stage of labor (AMTSL), neonatal resuscitation, case management of pre-eclampsia and eclampsia with magnesium sulphate (MgSO4) only, and management of primary postpartum hemorrhage (e.g., manual removal of placenta, internal bimanual uterine compression, and compression of the abdominal aorta). The composite scores calculated for each participant identified questions answered correctly and procedures done competently. The authors have published the analytical methods in a previous paper [5].
Descriptive analysis: Counts of absolute numbers and simple proportions were used to describe categorical variables. Measures of central tendency (mean and median) as well as dispersion (range and standard deviation (SD)) were also determined. Percentage of those achieving the required competency level (≥80% post-training scores in multiple choice questionnaires and OSCE) in both study groups were computed.
Inferential analysis: A generalized linear model with robust facility-level variance was used to test the differences between arms on multiple choice knowledge questions and OSCE scores at 12 months using a group indicator as the main predictor in the model. Adjustment was made for health worker and facility-level characteristics that show imbalance between arms at baseline and can be strongly correlated with the outcome scores. A p-value <0.05 was considered statistically significant. In addition, we developed a longitudinal model that assessed the change in scores, aproxy for competency, over time post-training. This model is appropriate, since it accounts for correlations within a provider over time as well as within-facility correlations. Generalized linear population-average model estimates using generalized estimating equations with working exchangeable correlation structure were used.
Qualitative data, notes, and transcripts were entered into Atlas-ti version 8 software. All data were collected in English since the service providers have above basic level of education. Content analysis of the discussions was undertaken to generate themes of interest along which the analysis was done. The analytic process proceeded in two basic steps: using the thematic groups, emerging themes were summarized related to relevant specific objectives, and content was analyzed to compare the three different respondent groups across the two states.