Study design
This impact evaluation was a two-arm difference-in-difference cluster randomized controlled trial conducted in Lusaka and Eastern provinces of Zambia. It assessed the change in proportion of HIV-exposed infants retained in care 12 months before and after the Umoyo MIP clinic was launched in 14 intervention sites compared to the change in retention in 14 control sites with the standard of care.
Study population
There were three main target populations from which the sample was drawn: health facilities, HEIs and HIV-positive mothers.
Facilities were eligible for the study if they were public sector health facilities caring for large numbers of HIV-positive pregnant women (i.e., an estimate of 10 HEIs born per month) and supported by the implementing partner for the study (the Center for Infectious Disease Research in Zambia [CIDRZ]) at the launch of Umoyo in February 2017. At that time, CIDRZ was supporting a total of 124 public health facilities in Lusaka, Western and Eastern provinces with the goal of expanding access to more efficacious PMTCT regimens and antiretroviral therapy (ART), and on early infant diagnosis for HEIs. CIDRZ was providing intensified on-site mentorship to facility staff on provision of integrated services to HIV patients in general. Additionally, health facility staff and community health workers were oriented and mentored on early infant diagnosis, family planning and how to strengthen documentation in the health facility registers. Although CIDRZ was mentoring facility staff on provision of integrated services to HIV patients, there was no specific intervention targeting mother-infant pairs in the PMTCT care. However, the community health workers were actively following up mother-infant pairs who missed scheduled appointments.
The HEI target populations for the study (the primary cohort) were those infants born to HIV-positive mothers within a four-month period (November 2015 through February 2016 for the pre-implementation period or November 2016 through February 2017 for the post-implementation period) and had their 6-week EID test as well as their first follow-up visit at the public health facilities. The final target population was HIV-positive mothers to participate in an exit survey, and mothers were eligible if they had attended the public health facilities for infant care (6 weeks to 2 years old) during the study period.
Intervention
The Umoyo MIP clinic is a designated clinic day for HIV-positive mothers and their HEIs where they receive routine child health care and routine maternal ART services including the following:
-
Enhanced, group-based sensitization and intensified Information Education and Counselling (IEC) for mothers;
-
Integrated services including HIV and TB screening for mothers and infants, provision of isoniazid TB prophylaxis for eligible mothers and infants; family planning for mothers, and immunizations and ART services for infants and;
-
Active defaulter tracing by lay counselors.
Services are provided to eligible MIPs on one or more days in the month, depending on the volume of infants that the health facility expects in the month. Facilities with relatively large catchment populations had more than one Umoyo MIP clinic day in a month – even up to four per month in one site. Regardless of the number of sessions a clinic held, each mother attended only one visit monthly. Since HIV-positive mothers attend the clinic together on the same day and participate in group-based education sessions, it was hoped that there would be an increased level of social support among mothers and reduced stigma or shame about one’s HIV-positive status.
Whereas the primary cohort for the study was a narrow cohort of infants in 2016 and 2017, the population eligible for the intervention was any infant born to an HIV positive mother all the way up to 24 months of age. In addition, MIPs can remain in the Umoyo MIP clinic until cessation of breastfeeding and/or the child is 24 months. Those infants who test HIV negative at 24 months are discharged from PMTCT care while those who seroconvert are transitioned to continue treatment at the normal ART clinic together with their mothers.
Before the intervention began, an orientation of staff in charge of facilities and the mother and child health (MCH) departments in the intervention facilities was conducted. The District PMTCT coordinators for all districts in the sample were also invited to this training to ensure district buy-in and ownership throughout the process. This constituted a two-day training, which included introducing them to the Umoyo model and also mentoring them on data quality. The orientation of the Umoyo MIP clinics was done by the Kitwe District PMTCT Coordinator and staff from health facilities in Kitwe that were implementing Umoyo using the human and financial resources provided by the public health system. After this general orientation, a follow-up, on-site orientation of all facility staff, including lay counselors, on how to implement Umoyo MIP clinics was conducted with the help of the CIDRZ team. Additionally, a nurse within MCH was identified to be the custodian of the list of defaulters and was responsible for communicating with the lay counsellors regarding any MIPs that needed follow up within the community. This on-site orientation marked the beginning of the first month of the Umoyo MIP clinic implementation. Therefore, the support of CIDRZ in the roll out of the intervention was to facilitate the on-site orientation of Umoyo MIP clinics in the intervention sites, monitor the implementation fidelity of intervention (at least once within a quarter), ensure that each facility has adequate supplies of requisite medical supplies needed for PMTCT service provision and mentorship on data quality.
Intervention sites were also provided with a job aid (Annex 1) to help the health facility staff with scheduling their monthly Umoyo MIP clinics. Intervention sites were also instructed to keep an attendance book (Annex 2) for the Umoyo program. At the control sites, the facilities continued to function on a business as usual basis providing services as per public facility standard of care.
Study Outcomes
The primary outcome of interest was the change in the proportion of HEIs who were retained in care at 12 months; this outcome combined HEIs with negative or unknown HIV status receiving a test at 12 months with HIV-positive infants attending the scheduled 12-month visit for ARV drug refills. Secondary outcomes included:
-
Change in the proportion of HEIs retained at 6 months;
-
Change in the proportion of HEIs with regular attendance as defined by retention in care at 6, 9 and 12 months of age;
-
Change in perceived social support of mothers with HEIs and;
-
Change in perceived stigma of mothers with HEIs.
To evaluate these outcomes, a cluster-randomized trial was conducted in 28 facilities. Each of the facilities were selected based on availability of PMTCT services and approximately 10 or more HIV-exposed infants born each month. The sample of HEIs was selected from the HEI registers based on the inclusion criteria in Table 1.
For the primary outcome, we report the a priori estimated effect size we could detect assuming 14 facilities in each of two arms and 30 MIPs per facility (pre- and post-). The following equation (20) with the specific parameters in Table 2 was used to estimate that we would be able to determine a 12% difference in the average change in the proportion of HEI who returned for their 12-month visit (alpha 0.05 and power of 80%).
(Due to technical limitations, the equation has been placed in the Supplementary Files section.)
For the mother interviews, mothers were approached during clinic days when PMTCT services were offered: women were eligible if they were the mother of an HEI born within 24 months of the survey date, and if the mothers were over the age of 18. Women and children who did not meet these criteria or those that did not attend the under-five clinic on the day the research team visited the facility were excluded.
Randomisation
Given the relatively small number of facilities per arm (N=14), a covariate constrained optimization technique was used to randomize intervention assignment and achieve balance in the two arms (21). Using estimates for annual expected HEI live births, the 2016 catchment population and districts, facilities were randomized using covariate-constrained randomization. The 28 sites were randomized 1,000 times and those iterations in which there was imbalance with an alpha of 0.1 on the aforementioned characteristics were removed. Among those remaining iterations where there was no imbalance detected at a threshold (F test beta = 0, alpha 0.1), an iteration of assignment was then selected at random.
Data collection
Three key sources were collected: patient registers, mother exit interviews, and facility-level questionnaires. In the original protocol, we had intended to gather data on monthly visits using the Septrin/Cotrimoxazole booklets (often called Septrin booklets in Zambia) and information on TB treatment using the Zambia National TB & Leprosy Control Program, IPT Register. However, we found that these booklets were not utilized in a standard way across all facilities. Thus, to minimize bias, the outcomes within this report rely on the HIV Exposed Infant (and mothers) Follow-Up Register only. We list our specific method of defining “retained” for each outcome by HIV status within the analysis section below.
For the mothers’ stigma and social support, we performed mother exit surveys. The tools were translated into local dialect and back translated by an independent contractor. The surveys were created as follows:
-
Social support: Nine questions were adapted from the Social Provisions Scale (22) to assess the degree of social support that women received from other women that attend the standard of care under-five clinic (pre-implementation) or the Umoyo MIP clinic (post-implementation). Adaptations reduced the scale to three items (“Disagree”, “Not sure”, and “Agree”) and included an option for “Don’t know, refused or don’t want to say”.
-
Stigma: 28 questions on stigma have been adapted from the HIV/AIDS stigma instrument (HASI-P) to include questions on internalized stigma and enacted stigma (e.g., verbal, fear of contagion, social isolation, and work place stigma)(23). We adapted the scale to limit to three (rather than four options), “Never”, “Once or twice”, and “Several times”; we also included an option for “Don’t know/refused/ not applicable”. Questions regarding HCW stigma were added and had a similar coding.
Facility level data was collected from surveys with key facility-level personnel to assess the general working environment in the facilities during the period of the study. The aspects relating to general working environment included frequency of stock-outs for key commodities relevant to the study outcomes, and whether sites received any additional support for PMTCT and/or paediatric HIV from implementing partners other than CIDRZ.
Data was collected at three points in time, with training of data collectors immediately prior to the start of each data collection (table 3).
Data collectors were hired specifically for this study and received three-day training. Direct supervision within facilities was conducted for all enumeration teams on a rolling basis and troubleshooting support was provided to all enumerators. All data was electronically entered using SurveyCTO at the point of collection except the mother interviews which were administered using paper tools. At the end of the data collection period, all the mothers’ questionnaires were entered twice by two different enumerators. Once the double data entry was complete, the discrepancies in entry were reconciled by checking with the entry in the hardcopy.
Analysis
Facility, child, and mother characteristics were compared to examine balance between the two arms before the intervention was implemented. For the facility characteristics, the self-reported values from the HCW were assumed to be facility-specific and the distribution (percentage and 99% confidence interval [CI]) was calculated. For the child and mother characteristics, an average estimate was calculated for each facility which was then pooled together per arm; to this end, the values, unless otherwise specified, are aggregate facility-level values. This analysis method for cRCTs produces a conservative estimate of difference when the facility sample size is less than 10 facilities per arm [6]. The difference between the two arms during the pre-implementation phase, as well as in the change from pre- to post-implementation was tested using unweighted t-tests.
Given that the evaluation was being implemented in a natural setting, monitoring of Umoyo exposure was not standardized or rigorously maintained, thus per protocol analysis was not carried out. All analysis reported are intent-to-treat.
For the outcomes, we categorized the eligible children from both the pre-implementation period and post-implementation period as retained according to the outcomes as listed in Table 4.
For each outcome above, the analysis was run in two methods: a facility-aggregate analysis as well as an individual-level method accounting for clustering for sensitivity analysis and varying number of eligible children per facility. For the first method, a facility-level proportion was calculated for both pre- and post-implementation, the difference between pre- and post-implementation in the proportion of children per facility was calculated, and an unweighted t-test with unequal variances was used to compare the differences between the two arms (20). For the sensitivity analysis, an individual-level logistic generalized estimating equation (GEE) with a binomial distribution and link logit was conducted with an indicator variable for time (pre- vs. post-), an indicator for whether the child came from an intervention facility (control vs. intervention), and a time by intervention interaction to estimate the program impact. The model also accounted for a potential difference during the pre-implementation phase between the arms; though differences were not statistically significant, we included a covariate for the average number of children per facility. To account for the multiple outcomes and increased risk of Type I error (incorrectly rejecting the null hypothesis), we adjusted the alpha to an alpha of 0.05/11 (for 11 outcomes) or 0.0045; thus, we report 99% confidence intervals (1-0.005) for our estimates, and all p-value testing is considered significant if below <0.0045. The GEE model included the facility-level as cluster with an exchangeable correlation structure.
Finally, we originally proposed to complete both weighted and unweighted t-test but focused primarily on unweighted t-tests with unequal variances and the individual-level model (GEE) which allows for varying population sizes per facility.
For mothers regarding social support and stigma outcomes all mothers received a score similar to the Holzmer 2007 work but accounting for the fact we included an ‘N/A’ response and had a lower maximum possible range (23). To this end, we gave all “Don’t Know, refused, or don’t want to say, or NA” responses a missing value. We then gave those who had the lowest answer on the item (e.g., disagree, or never occurred, or no) a zero and those with the next two levels of responses a one or two, accordingly. We then created a weighted score: all items for the specific topic were summed and the mother received a score by dividing this sum by the total number of non-missing items. Thus, the score always represented the highest level of either social support or stigma. Similar to the child outcomes, the analysis was run in two methods: a facility-aggregate analysis using unweighted t-tests and an individual-level linear GEE. For the multivariable linear GEE model, though differences were not statistically significant, we included covariates for the average number of interviews per facility, if the mother was over 40, if she were married, or had been on HIV treatment for a year.
All analyses were conducted assuming intent-to-treat for a conservative estimate as written in the original protocol. Given the lack of a robust and rigorous method to identify true exposure of the children, a per protocol analysis was not reported.