Study Objectives
This study aimed to compare the knowledge and skills competencies of health workers in improving maternal and newborn day of birth care after the LDHF/m-mentoring versus the TRAD training approaches in Ebonyi and Kogi states, Nigeria. The primary outcomes were increase in knowledge, clinical skills, and retention of clinical competency at 3- and 12-months post-training. The secondary outcomes measured were facilitators of and barriers to the LDHF/m-mentoring training approach at individual and institutional levels.
Our hypothesis was that LDHF/m-mentoring results in better knowledge and skills outcomes compared to the TRAD approach.
Study setting
The study was conducted in Ebonyi and Kogi States, in Nigeria. Ebonyi State has a population of 2.8 million and is in the South-East zone. Kogi state has a population of 4.3 million and is in the North-Central zone. At the time of the study, the Maternal and Child Survival Program (MCSP), in partnership with the Federal Ministry of Health, Ebonyi and Kogi States Ministries of Health and Professional Associations supported 120 health facilities across the two states, of which 60 were selected to be part of the study, 30 in each state. The MCSP was a global, $560 million, 5-year cooperative agreement funded by the United States Agency for International Development (USAID) to introduce and support scale-up of high-impact health interventions among USAID’s 25 maternal and child health priority countries, as well as other countries.
Study design
This was a prospective cluster randomized controlled trial. It was a mixed- method study. Informed consent and ethical approval were obtained. The study was conducted between October 2016 and November 2017 in sixty health facilities that were randomly selected and assigned to either the intervention arm or the control arm. The authors have attempted a brief description of the methodology. The training approaches, eligibility criteria and randomization have been described in the published study protocol [11].
Training approach 1 and data collection: Simulation based LDHF/m-mentoring training of participants – group 1 or intervention group
The training for the LDHF arm was for the entire team of service providers available at the health facility, but only those who met the study inclusion criteria were assessed. The training was divided into two “low-dose” training courses of 4 days each, with additional time for assessment as needed and was conducted at the health facilities using the basic emergency obstetrics and newborn care (BEmONC) package. The BEmONC package include training on (i) administering parenteral antibiotics, (ii) administering uterogenic drugs for active management of the third stage of labour and prevention of postpartum haemorrhage, (iii) use of parenteral anticonvulsants for the management of pre-eclampsia/eclampsia, (iv) manual removal of placenta, (v) removal of retained products (e.g. manual vacuum extraction, dilatation, and curettage), (vi) performing assisted vaginal delivery (vii) performing basic neonatal resuscitation [12]. This was for an initial 4days with emphasis on normal uncomplicated cases and repeated after 1 month to emphasize more complicated skills (figure 1). The training techniques were modified to shift the emphasis to practice. This was a competency-based training that has been used in Jhpiego’s (affiliate of Johns Hopkins University) training with the aim of improving and retaining the skills of the health worker. The trainer practices the skills while the trainee observes. The trainee then practices using the anatomical models (manikins) while the trainer observes. At the end, the trainer debriefs and provides feedback to the trainee. The trainee repeats the practice until competency is achieved. The Peer Practice Coordinators (PPCs) received technical update in LDHF including the use of session plans, case scenario and MamaNatalie/NeoNatalie models to conduct simulation practices. During the one-month interval between training courses, health care workers had opportunity to practice what they learned and reinforce their competencies through high-frequency simulation-based practices of 2–3 times weekly. The time spent on lectures was reduced and time spent on hands-on practice was increased. The PPCs completed the practice log. In addition to the simulation exercises, all the trained health workers in the LDHF arm participated in mobile Mentoring (mMentoring), which consisted of receiving weekly reminder messages and quiz questions on the topics reviewed via SMS messaging. Also the PPCs received structured, monthly half-hour mentoring calls from a trainer/master mentor that provided remote support, answering questions, providing guidance and reinforcing key messages.
After oral consent was obtained, the participants took a pre-training assessment consisting of MCQs and OSCEs to assess their baseline knowledge and skills on BeMONC. The MCQ contained a set of multiple choice questions to test trainees’ knowledge. OSCE involved the use of checklists to evaluate trainees’ demonstration of clinical skills to ensure that each step is correctly and completely carried out. These tools were adapted from previous studies and validated by the researchers and trainers [6,7,12]. The assessments tested their knowledge and skills on conduct of normal delivery, active management of the third stage of labor (AMTSL), neonatal resuscitation, case management of pre-eclampsia and eclampsia (PEE) and management of PPH (e.g. manual removal of placenta, internal bimanual uterine compression and compression of the abdominal aorta). They also had an immediate post-training assessment which included MCQs and OSCEs. The questions answered correctly and procedure done competently was scored over a total of 100%. A test score of ≥80% was accepted as level of competence. The pre-training and immediate post-training assessments results were compared. Skills and knowledge assessments were done at three time points post-training (immediate post training day, at 3 months 12 months). Trainees’ satisfaction with the simulation-based LDHF/m-Mentoring training approach was determined using satisfaction (quantitative) survey. The scores at each assessment were collected using validated tools and recorded in real-time on android devises and sent to a central server after verification.
Qualitative data were collected through six focus group discussions (FGD) comprising 8-10 participants per group purposively selected from LDHF arm at 12 months. The participants were greeted and invited via telephone and given full study information including the aims of the study. All the invited respondents gave consent to participate in the study. The FGDs were used to collect data on experiences and satisfaction of trainees with the LDHF/m-mentoring training approach. The enquiry was based on the high-frequency practice sessions with simulators, mobile mentoring through short messages and quizzes, overall impressions of the LDHF/m-mentoring approach, and how it could be improved. The IDIs with PPCs collected data on their experience managing simulator practice sessions, interactions with trainers/master mentors, m-Mentoring, changes made in clinical practice after the training, success, challenges and their overall impression about LDHF/m-mentoring training approach. With regard to trainers, data were collected about their experiences with the LDHF/m-mentoring training approach, successes and challenges with mobile mentoring to support the PPCs, and the effectiveness of approach in building the capacity of the health workers. The data was collected at their workplaces. Only the interviewer and the respondents were present at a private place where the interviews were conducted. Interview guides were adapted from previous work and pretested. Repeated interviews were not carried out. Audio-recorders were used during interviews to help during recall. Each interview lasted for 45-60minutes. The qualitative data was collected until data saturation was reached. The interviews were transcribed. The Transcripts were not returned to participants for comments.
Training approach 2 and data collection: Traditional group training of participants – Group 2 or control group
The traditional training approach consisted of 8 days of lectures with practice sessions on simulators, outside the participants’ workplace, usually in a hotel (figure 2). The group had lectures and practice sessions on conduct of normal delivery, AMTSL, neonatal resuscitation, case management of PEE and management of PPH. The BEMONC package and clinical observation checklists were used during the training sessions as was done for the intervention arm. The group-based training approach involved training sessions consisting of 8 days of lectures and fewer practice sessions using manikins (MamaNatalie/NeoNatalie) offsite and not at the trainees’ place of work. Emphasis was not made on practice sessions at the health facilities when they returned. They did not have mMentoring.
Oral consent was obtained. The participants/trainees took pre-training assessments consisting of MCQs and OSCEs through use of manikins to assess their baseline knowledge and skills respectively. The assessments tested their knowledge and skills on conduct of normal delivery, AMTSL, neonatal resuscitation, case management of PEE and management of PPH (e.g. manual removal of placenta, internal bimanual uterine compression and compression of the abdominal aorta). At the end of the eight-day training, the participants had an immediate post-training assessment which included MCQs and OSCEs. As described for the intervention arm, the questions answered correctly and procedure done competently were scored out of a total of 100%. These were recorded in real-time on android devises and sent to a central server after verification. The assessments results were compared. Trainees’ satisfaction survey was also conducted.
Both study arms were trained and assessed by senior clinicians (mainly obstetricians, pediatricians and midwives) whose knowledge and skills were standardized. The tools were pre-tested among 25 health workers from health facilities that were not part of the study. Before being used, checklists for the assessed clinical skills were modified in accordance with the Nigeria context by the research team and data collectors. At the time of the training, MCSP did not support any interventions in these facilities that are likely to cause contamination. Other quality improvement interventions only happened at the non-study facilities at the time of the study.
The assessors were blinded to the groups the participants were assigned to as they conducted the assessments.
Eligibility criteria
The sixty health facilities were selected from a sampling frame of the 120 MCSP-supported health facilities in the two states located in three geopolitical zones where the project support. They represented all three levels of the healthcare system in Nigeria (Primary health care or PHC, secondary, tertiary). The health workers were drawn from among those working in labor and delivery sections of the participating health facilities in the two states. In addition, the health workers had spent at least six months in the health facility working in either maternal or newborn care section.
Randomization
The unit of randomization in this study was the health facility. The 60 facilities were matched based on locations and level in the health care system, then divided into nine strata taking into consideration the three geopolitical zones as well as the three levels of the health care system in Nigeria. Thereafter, each stratum was randomized to either intervention or control arm using randomly permuted blocks in a ratio of 1:1 so as to achieve balance in geographical location and types of health facilities in the two study arms. Randomization was done by a study team member. Those assessing outcomes were blinded to the training methodology used for the health facilities. Since there were at times less than three skilled birth attendants in some health facilities, all health workers employed in the maternity or newborn units in such facilities who met the inclusion criteria were selected from the randomized health care facilities.[5]
Sample size
The details on the computation of the sample size, including the assumptions are provided in the study protocol manuscript. [5]. Briefly, the required number of participants was computed as the average number of health workers to be included from each of the 60 facilities (30 per arm) using a test of two proportions. The percent of competent health workers was estimated at 50% in the control group. Study power was set at 80% to reject a null hypothesis that the proportions of competent health workers are equal in the two study arms against the alternative hypothesis of 20 percentage point difference in proportions of competent health workers between the two study arms - effect size. The significance level was set at 0.05. Sample size computation was done using PASS statistical software. The correlation of health workers competency in each health facility was assumed to be 0.05 given that some facility-level factors are shared by the health workers working together and influence how they perform certain tasks. We assumed there are about four health workers selected per facility, thus 240 health workers sampled across the two groups. An adjusted sample size of 300 participants (150 per study arm) was arrived at after factoring in potential 20% drop-out during follow up period after baseline. More health workers were recruited from Kogi State since it had a larger population than Ebonyi. Study participants recruited from each facility included the maternity unit head, wherever possible, and two others to ensure that the other team members received the necessary support to practice.
Data analysis
The primary outcome was health workers clinical competency in BEmONC skills at three months post-training assessed through simulation. The secondary outcome was retention in clinical competency level in BEmONC skills as determined through MCQs and OSCEs assessments at 12 months post-training. The results on assessments at three and 12 months after training were presented. For both arms, composite scores were computed for the infection prevention and the BEmONC functions, namely; skills in conducting normal delivery, active management of the third stage of labor (AMTSL), neonatal resuscitation, case management of pre-eclampsia and eclampsia with magnesium sulphate (MgSO4), and the management of primary postpartum hemorrhage. Percent scores were computed per participant based on either questions answered correctly or procedures done competently depending on the assessment tool used.
To describe categorical variables, counts and simple proportions were used. Mean and standard deviations as well as median and interquartile range were used to summarize data on continuous variables. Percentage of those achieving the required competency level of ≥80% post-training scores in MCQs and OSCEs were computed and compared across the two study arms. A generalized linear model was used to test the differences between arms on MCQ and OSCE scores at 12 months using a group indicator as the main predictor in the model. Furthermore, adjustments were made to account for facility-level and health worker characteristics that might have influenced the two study arms at baseline and could be strongly correlated with the outcome scores. A significance level of <0.05 was set for statistical significance. A longitudinal model that accounts for intra-provider correlations over time as well as within-facility correlations was developed. This model assessed the change in scores over time post-training. The model is appropriate, since it estimated using generalized estimating equations (GEE) with working exchangeable correlation structure.
In regards to the qualitative data, transcripts in Microsoft Word were imported into ATLAS.ti software (version 8.0) for content analysis. The codebook was developed by the two (2) qualitative researchers, the data analyst and a second coder, using a priori codes developed from the research aims and questions, and the interview guides adopted from previous studies. In order to enhance validity, we adopted a 1st tier triangulation (of researchers) and ensured a well-documented audit trail of materials and processes. To ensure reliability, refutational analysis, constant data comparison, comprehensive data use was done. Reliability checks were performed by each coder independently re-coding documents already coded by the other.