Study Design and Population
This mixed-methods program evaluation was conducted in Lagos, Oyo, and Niger States of Nigeria from January 2021 through October 2023. The quantitative component of the study consisted of an assessment of group-level changes in DMPA-SC SI service provision in 36 health facilities (12 per state). Qualitative in-depth interviews with providers were conducted to better understand providers’ experiences of the training and subsequent provision of contraceptive care and counseling, as well as to contextualize quantitative findings.
Within each state, study facilities were selected from within local government areas (LGAs) with low levels of insecurity where no DISC activities had been conducted, to ensure a valid pre-implementation baseline (Phase 0). Within eligible LGAs, study facilities were selected that offered the full range of contraceptive methods, had the highest reported contraceptive injectable service volumes in the prior 12 months, and had low but non-zero baseline DMPA-SC service provision and low SI provision, as reported in the Nigerian Health Management Information System (NHMIS). A total of 110 providers were trained across the 36 selected study facilities, with an average of 3 providers trained per facility.
The DISC intervention
The DISC intervention model was designed based on learnings from formative research, insight generation, and evidence-based prototyping with prospective SI clients and family planning (FP) providers that the project conducted throughout 2020–2021 [20]. The intervention package includes a provider-focused empathy-based training; community mobilization and awareness-raising; and post-training support (Fig. 1).
I.
Provider empathy-based training. The training directly addressed client drop-off during initiation of self-injection by strengthening provider skills to provide empathetic counseling, with the goal of increasing client confidence and addressing common concerns such as fear of the needle and pain. Training modules – which included tools such as checklists and job aids – address aspects of quality service provision including values clarification, informed choice, person-centered care, proficiency in self-injection, approaches to strengthening supportive supervision, and strategies to improve commodity forecasting and reporting. DISC’s approach to empathy training builds upon behavior change principles (Supplemental Material, Figure
S1), with content tailored in a way that addresses both providers’ motivations and capabilities and builds in practice components and supportive supervision and on-job mentorship to reinforce learnings (Supplemental Material, Figure S2). Trainings were conducted at study facilities and were co-facilitated by representatives from the Federal Ministry of Health, State Primary Healthcare Development Agency (state FP and M&E officers), and DISC-Nigeria project staff. Integrated within the training is a practicum which allowed trainees to visit a clinic where clients have been pre-mobilized to seek DMPA-SC services, so that each trainee was able to practice DMPA-SC SI counseling in a supervised setting. During the practicum, each trainee provided actual SI training to a client who has expressed interest in the method and provided consent to take part in a training exercise. The provider practicum was observed by one of the trainers, who used a checklist to note down any feedback during the training session. The trainer is instructed to only interrupt counseling if incorrect, incomplete, or harmful information is being passed along to the client—to ensure each client receives quality counseling. Trainers provided feedback to all trainees afterwards in a plenary session.
II. Community mobilization and awareness-raising. Commencing shortly before the training delivery and continuing concurrently, the DISC-Nigeria team led facility- and community-based events aiming to counter misinformation, educate community members about self-care and FP with the aid of a structured flipchart, and raise awareness about the available SI services at nearby health facilities participating in the study. Community mobilizers also aim to bring care to the community-level by organizing community outreach events with the presence of trained service providers at bustling locations where women frequently gather or work – such as health centers on clinic days, markets, women groups/trade associations, village squares, and religious centers. Awareness-raising activities included digital interventions such as interactive voice response and a digital companion (chatbot), made use of DISC’s #DiscoverYourPower (DYP) slogan and leveraged DISC’s campaign messages which had been co-designed with a creative agency. Concurrently, radio campaigns were underway in the three states, as were social media ad campaigns on Facebook. Additional detail about the DYP campaign and the program’s awareness-raising activities have been published elsewhere [21, 22].
III. Post-training support. The DISC empathy-based SI materials included a supportive supervision checklist to help supervisors assess resource availability and providers’ adherence to empathetic counseling practices. Post-training supportive supervision was provided jointly by RHCs, project staff and state SBC officers to ensure high quality service delivery, support for empathetic counseling skills and supply management, and accuracy of data capture. Following trainings, health facilities received ongoing on-site mentoring support to reinforce knowledge and skills learned during trainings. The onsite mentorship also helped cascade the training to 22 other providers who were not part of the original training. Each post-training visit included an evaluation of the competency of SI services, a review of community mobilization activities, data quality checks, and commodity availability tracking. Additionally, ongoing community mobilization activities were conducted in facilities’ catchment areas during follow-up visits.
Quantitative Methods
To evaluate the impact of the intervention on DMPA-SC service provision within study facilities, we used a single-group interrupted time series design that leveraged phased implementation. This design allows for detection of both immediate changes (measured as change in mean service provision at the facility-level in the first month of each implementation phase compared to the last month in the prior phase), as well as long-term changes (measured as changes in the monthly trend in average facility-level service provision relative to the monthly trend in the preceding phase).
The following four phases were defined for the purpose of the interrupted time series analysis:
Phase 0: Pre-intervention (January 2021 – June 2022). This phase captured baseline service delivery in the absence of DISC intervention. No DISC project activities were implemented in study facilities or the surrounding LGAs during this phase. Monthly contraceptive service statistics data were extracted from the NHMIS.
Phase 1: Data strengthening (July – September 2022). The aim of the data strengthening phase was two-fold: first, to ensure that pre-implementation data reported in study facilities were high quality and provide a valid pre-implementation estimate of service delivery; and second, to allow the evaluation approach to differentiate the impact of the intervention itself from improvements in data quality. To these aims, DISC held one-day-long SI data reporting trainings at the state level during the first week of July 2022, which were attended by at least one FP provider from each of the 36 facilities, the facility data reporting officer, and LGA monitoring and evaluation (M&E) officers who are responsible for data reporting into that national NHMIS. The objective of this training was to equip providers on how to correctly document data on DMPA-SC/SI for reporting into NHMIS. Trained providers subsequently cascaded the training to other providers in the study facilities.
Phase 2: Implementation (October – December 2022). DISC project implementation consisted of a combined supply- and demand-side approach, details of which are provided in the previous section.
Phase 3: Maintenance and Routine Monitoring (January 2022 – October 2023). The maintenance phase (through March 2023) included ongoing mentorship and supportive supervision, while during the routine monitoring phase (April through October 2023), mentorship ended and only routine monitoring activities continued.
For each phase of the implementation, activities were conducted in full (for the data strengthening training and the provider empathy-based training) or initiated and ongoing (for the community-based awareness and mobilization activities) in the first week of the respective phase. For this reason, assessing level change in the first month of implementation was deemed appropriate to capture short-term changes associated with program implementation activities.
We hypothesized that implementation of the empathy training intervention would result in increases in the level of DMPA-SC SI service provision, comparing Phase 1 to Phase 2. In the maintenance phase, we hypothesized that both levels and trends observed in Phase 3 would be maintained relative to Phase 2.
Data Sources
NHMIS data were extracted at the facility level for each month of interest during Phase 0; subsequent data for Phases 1 through 3 were extracted by DISC data entry clerks every month directly from facility family planning registers and submitted to project monitoring and evaluation (M&E) staff monthly via Excel reporting template for review and analysis. Per NHMIS reporting protocol, facilities which did not deliver or did not report on a specific method in a month left NHMIS data (used for Phase 0) on that method’s service delivery volumes missing. As a result, all missing facility-month observations for specific FP method service volumes were assumed to be zero (i.e., no method provision). It was not possible to differentiate missingness due to non-reporting from a true zero (no method provision). We observed few (n = 8) instances of missing data in Phases 1–3, all for facilities that reported non-missing data on service provision for at least 1 contraceptive method for that month. For that reason, we also treat these instances of missingness as zeros for consistency. To limit bias due to non-reporting, we excluded all NHMIS data available prior to January 2021 due to suspected incompleteness and quality issues with DMPA-SC reporting prior to this date.
Primary and Secondary Outcomes of Interest
Study facilities recorded the number of FP visits in which clients received DMPA-SC by mode of administration: for SI visits, the client self-injected under provider observation, while for PA visits the injectable was administered by the provider. Our three primary outcomes of interest were number of DMPA-SC SI visits; number of DMPA-SC PA visits; and number of total DMPA-SC visits (SI and PA combined).
To evaluate whether implementation was associated with shifts in overall FP service volumes and in the service delivery method mix, we conducted secondary analyses using analogous ITSA models on total FP, long-acting reversible contraception (LARC), and DMPA-IM visits. To model total FP visits, we summed FP visits for all methods except condoms, as data on condom distribution in unreliable and inconsistently reported across facilities. LARC visits were calculated by summing all intrauterine device (IUD) and implant service provision. All outcomes were modeled as service delivery totals aggregated at the facility-month level.
Statistical Analysis
We exploited the phased implementation to evaluate trends in DMPA-SC service provision at study facilities using a single-group interrupted time series analysis (ITSA). Models were fit using generalized estimating equation (GEE) models accounting for repeated observations at the facility level. We used linear models with observations at the facility-month-level, in which coefficients can be interpreted as mean differences in monthly facility-level counts (e.g., of DMPA-SC SI visits). Models were fit with an exchangeable working correlation structure and robust standard errors. We analyzed models that were both unadjusted and adjusted for season (in 3-month-long periods) and LGA using dummy variables. All estimates were generated using the xtitsa package in the statistical analysis software Stata [23].
Sensitivity and Exploratory Analyses
Because we were not able to conduct a complete audit of facility registers and client intake forms during Phase 0, pre-intervention data was extracted directly from NHMIS. The project team opted to use data collectors to extract monthly data directly from facility registers during Phase 1 and subsequent phases because of concerns around the quality of service provision data once it was aggregated into the NHMIS monthly summary forms. To assess robustness of our results to this shift to use of programmatic data, we conducted two sensitivity analyses: one in which we combined Phase 0 and 1 include a single pre-intervention phase; and a second in which we refit models using NHMIS data only. In an exploratory analysis, we estimated primary outcomes in State-specific models.
Qualitative Methods
In January 2023, we conducted in-depth interviews with 31 Reproductive Health Coordinators (RHCs) and front-line family planning service providers purposively selected in Niger, Lagos, and Oyo. Interviews were conducted to assess the acceptability and self-reported changes in attitudes and behaviors related to SI services among providers who had received the empathy training intervention.
Prior to selection of providers for the interviews, SI visit service data for the period between April and December 2022 were reviewed for all 36 DISC-supported evaluation facilities. Three performance categories were defined: facilities with increased, decreased, or stable SI service provision after the intervention. A subset of 18 health facilities were purposively selected across the performance categories to sample 24 providers (9 in Niger and Lagos each, and 6 in Lagos). A sample of 24 providers and 7 RHCs was expected to be sufficient to achieve the required theoretical saturation for generation of insights relevant for contextualizing the quantitative evaluation findings.
Trained research assistants conducted interviews in English using a semi-structured guide. Interviews were audio-recorded with verbal consent of the participating providers and transcribed verbatim under the supervision of the first and third authors [SO, OA].
All interview transcripts were reviewed for quality and uploaded into Dedoose, a cloud-based application for managing and analyzing qualitative data [24]. A hybrid approach of inductive and deductive coding was used to analyze the data [25]. All the transcripts were read in full by one co-author (JN) to gain deeper understanding of the interview scope and content and uncover emerging subcodes for documentation in the final coding structure. Each interview was reviewed line-by-line, and relevant segments and phrases labelled with respective codes and subcodes. Dedoose was then used to assess data patterns and identify recurrent themes. De-identified quotations from the participants are reported in this paper to characterize analyzed themes.
Ethics Statement
NHMIS data were collected at the facility-month level and did not include any individual patient identifiers. After consultation with the Nigerian Federal Ministry of Health, data in this manuscript are reported at the state level to protect the confidentiality of individual health facilities.
Permission was sought from the facility in-charge prior to provider eligibility screening. Healthcare providers provided verbal consent for eligibility screening. Eligible providers were read an informed consent form by the study enumerator. Informed consent for participation was documented via paper-based enumerator signature and date on an informed consent acknowledgment form. Providers did not receive any incentive or compensation for participation in the study.
All study procedures were approved by the Population Services International (PSI) Research Ethics Board and Nigerian National Health Research Ethics Committee (NHREC/01/01/2007-24/05/2021).