This impact evaluation was a two-arm difference-in-difference cluster randomized controlled trial conducted in Lusaka and Eastern provinces of Zambia. It assessed the change in proportion of HIV-exposed infants retained in care 12 months before and after the Umoyo MIP clinic was launched in 14 intervention sites compared to the change in retention in 14 control sites with the standard of care.
There were three main target populations from which the sample was drawn: health facilities, HEIs and HIV-positive mothers.
Facilities were eligible for the study if they were public sector health facilities caring for large numbers of HIV-positive pregnant women (i.e., an estimate of 10 HEIs born per month) and supported by the implementing partner for the study (the Center for Infectious Disease Research in Zambia [CIDRZ]) at the launch of Umoyo in February 2017. At that time, CIDRZ was supporting a total of 124 public health facilities in Lusaka, Western and Eastern provinces with the goal of expanding access to more efficacious PMTCT regimens and antiretroviral therapy (ART), and on early infant diagnosis for HEIs.
The HEI target populations for the study (the primary cohort) were those infants born to HIV-positive mothers within a four-month period (November 2015 through February 2016 for the pre-implementation period or November 2016 through February 2017 for the post-implementation period) and had their 6-week early infant diagnostic test as well as their first follow-up visit at the public health facilities.
The final target population was HIV-positive mothers to participate in an exit survey, and mothers were eligible if they had attended the public health facilities for infant care (6 weeks to 2 years old) during the study period.
A cluster-randomized control trial was conducted in a sample of 28 facilities. Each of the facilities were selected based on availability of PMTCT services and approximately 10 or more HIV-exposed infants born each month. The sample of HEIs was selected from the HEI registers based on the inclusion criteria in Table 1. All the criteria in the table need to be jointly met for a child to be considered eligible for enrolment in the study.
Table 1:Sample selection for children
For the primary cohort, we report the a priori estimated effect size we could detect assuming 14 facilities in each of two arms and 30 MIPs per facility (pre- and post-). The following equation (21) with the specific parameters in Table 2 was used to estimate that we would be able to determine a
12% difference in the average change in the proportion of HEI who returned for their 12-month visit (alpha 0.05 and power of 80%).
Table 2: Sample size calculation for primary outcome
For the mother interviews, mothers were approached during clinic days when PMTCT services were offered: women were eligible if they were the mother of an HEI born within 24 months of the survey date, and if the mothers were over the age of 18. Women and children who did not meet these criteria or those that did not attend the under-five clinic on the day the research team visited the facility were excluded.
CIDRZ was providing intensified on-site mentorship to facility staff on provision of integrated services to HIV patients in general. Additionally, health facility staff and community health workers were oriented and mentored on early infant diagnosis, family planning and how to strengthen documentation in the health facility registers. Although CIDRZ was mentoring facility staff on provision of integrated services to HIV patients, there was no specific intervention targeting mother-infant pairs in PMTCT care. However, the community health workers were actively following up mother-infant pairs who missed scheduled appointments.
The Umoyo MIP clinic is a designated clinic day for HIV-positive mothers and their HEIs where they receive routine child health care and routine maternal ART services including the following:
- Enhanced, group-based sensitization and intensified Information Education and Counselling for mothers;
- Integrated services including HIV and TB screening for mothers and infants, provision of isoniazid TB prophylaxis for eligible mothers and infants; family planning for mothers, and immunizations and ART services for infants and;
- Active defaulter tracing by lay counselors.
Services are provided to eligible MIPs on one or more days in the month, depending on the volume of infants that the health facility expects in the month. Facilities with relatively large catchment populations had more than one Umoyo MIP clinic day in a month—even up to four per month in one site. Regardless of the number of sessions a clinic held, each mother attended only one visit monthly.
Whereas the primary population for the study was a narrow cohort of infants in 2016 and 2017, the population eligible for the intervention was any infant born to an HIV positive mother all the way up to 24 months of age. In addition, MIPs can remain in the Umoyo MIP clinic until cessation of breastfeeding and/or the child is 24 months. Those infants who test HIV negative at 24 months are discharged from PMTCT care while those who seroconvert are transitioned to continue treatment at the normal ART clinic together with their mothers. However, only results on the primary cohort of infants are presented in this study.
Six weeks before the study began, a two day off-site orientation of at least two health facility staff in the treatment arm was conducted by staff from Kitwe District who were already implementing Umoyo within the public health system. A follow-up on-site (facility by facility) orientation of all health facility including lay counselors was conducted by the CIDRZ team. To ensure that data quality was maintained in both study arms, the CIDRZ team conducted data quality mentorship exercises in all facilities enrolled in the study. Additionally, across both study arms, a nurse within mother and child health department was identified to be the custodian of the list of defaulters and was responsible for communicating with the lay counsellors regarding any MIPs that needed follow up within the community. This on-site orientation marked the beginning of the first month of the Umoyo MIP clinic implementation. Throughout the intervention, CIDRZ role was to monitor the implementation fidelity of intervention (at least once within a quarter), ensure that each facility has adequate supplies of requisite medical supplies needed for PMTCT service provision and mentorship on data quality.
Intervention sites were also provided with a job aid (Annex 1) to help the health facility staff with scheduling their monthly Umoyo MIP clinics. Intervention sites were also instructed to keep an attendance book (Annex 2) for the Umoyo program. At the control sites, the facilities continued to function on a business as usual basis.
The primary outcome of interest was the change in the proportion of HEIs who were retained in care at 12 months; this outcome combined HEIs with negative or unknown HIV status receiving a test at 12 months with HIV-positive infants attending the scheduled 12-month visit for ARV drug refills. Secondary outcomes included:
- Change in the proportion of HEIs retained at 6 months;
- Change in the proportion of HEIs with regular attendance as defined by retention in care at 6, 9 and 12 months of age;
- Change in perceived social support of mothers with HEIs and;
- Change in perceived stigma of mothers with HEIs.
Given the relatively small number of facilities per arm (N = 14), a covariate constrained optimization technique was used to randomize intervention assignment and achieve balance in the two arms (22). Using estimates for annual expected HEI live births, the 2016 catchment population and districts, facilities were randomized using covariate-constrained randomization. The 28 sites were randomized 1,000 times and those iterations in which there was imbalance with an alpha of 0.1 on the aforementioned characteristics were removed. Among those remaining iterations where there was no imbalance detected at a threshold (F test beta = 0, alpha 0.1), an iteration of assignment was then selected at random.
Three key sources were collected: patient registers, mother exit interviews, and facility-level questionnaires. We also gathered data on monthly visits using the Septrine/Cotrimoxazole booklets (often called Septrine booklets in Zambia) and information on TB treatment using the Zambia National TB & Leprosy Control Program, IPT Register. However, we found that these booklets were not utilized in a standard way across all facilities. Thus, to minimize bias, the outcomes within this report rely on the HIV Exposed Infant (and mothers) Follow-Up Register only. We list our specific method of defining “retained” for each outcome by HIV status within the analysis section below.
For the mothers’ stigma and social support, we performed mother exit surveys using questionnaires validated in African countries with comparable HIV profiles to Zambia such as Swaziland, Lesotho Malawi, Tanzania, Uganda (23; 24). The tools were translated into local dialect and back translated by an independent contractor. The surveys were created as follows:
- Social support: Nine questions were adapted from the Social Provisions Scale (23) to assess the degree of social support that women received from other women that attend the standard of care under-five clinic (pre-implementation) or the Umoyo MIP clinic (post-implementation). Adaptations reduced the scale to three items (“Disagree”, “Not sure”, and “Agree”) and included an option for “Don’t know, refused or don’t want to say”.
- Stigma: 28 questions on stigma have been adapted from the HIV/AIDS stigma instrument (HASI-P) to include questions on internalized stigma and enacted stigma (e.g., verbal, fear of contagion, social isolation, and work place stigma) (24). We adapted the scale to limit to three (rather than four options), “Never”, “Once or twice”, and “Several times”; we also included an option for “Don’t know/refused/ not applicable”. Questions regarding HCW stigma were added and had a similar coding.
Facility level data was collected from surveys with key facility-level personnel to assess the general working environment in the facilities during the period of the study. The aspects relating to general working environment included frequency of stock-outs for key commodities relevant to the study outcomes, and whether sites received any additional support for PMTCT and/or paediatric HIV from implementing partners other than CIDRZ.
Data was collected at three points in time, with training of data collectors immediately prior to the start of each data collection (table 3).
Table 3: Schedule of data collection
Data collectors were hired specifically for this study and received three-day training. Direct supervision within facilities was conducted for all enumeration teams on a rolling basis and troubleshooting support was provided to all enumerators. All data was electronically entered using SurveyCTO at the point of collection except the mother interviews which were administered using paper tools. At the end of the data collection period, all the mothers’ questionnaires were entered twice by two different enumerators. Once the double data entry was complete, the discrepancies in entry were reconciled by checking with the entry in the hardcopy.
Facility, child, and mother characteristics were compared to examine balance between the two arms before the intervention was implemented. For the facility characteristics, the self-reported values from the HCW were assumed to be facility-specific and the distribution (percentage and 99% confidence interval [CI]) was calculated. For the child and mother characteristics, an average estimate was calculated for each facility which was then pooled together per arm; to this end, the values, unless otherwise specified, are aggregate facility-level values. This analysis method for cRCTs produces a conservative estimate of difference when the facility sample size is less than 10 facilities per arm . The difference between the two arms during the pre-implementation phase, as well as in the change from pre- to post-implementation was tested using unweighted t-tests with unequal variances.
Given that the evaluation was being implemented in a natural setting, monitoring of Umoyo exposure was not standardized or rigorously maintained, thus per protocol analysis was not carried out. All analysis reported are intent-to-treat.
For the outcomes, we categorized the eligible children from both the pre-implementation period and post-implementation period as retained according to the outcomes as listed in Table 4.
Table 4: Categorization for primary child outcomes
. For each outcome above, an individual-level logistic generalized estimating equation (GEE) with a binomial distribution and link logit was conducted as part of the sensitivity analysis which accounted for clustering and varying number of eligible children per facility. This regression included an indicator variable for time (pre- vs. post-), an indicator for whether the child came from an intervention facility (control vs. intervention), and a time by intervention interaction to estimate the program impact. The model also accounted for a potential difference during the pre-implementation phase between the arms; though differences were not statistically significant, we included a covariate for the average number of children per facility.
To account for the multiple outcomes and increased risk of Type I error (incorrectly rejecting the null hypothesis), we adjusted the alpha to an alpha of 0.05/11 (for 11 outcomes) or 0.0045; thus, we report 99% confidence intervals (1–0.005) for our estimates, and all p-value testing is considered significant if below <0.0045. The GEE model included the facility-level as cluster with an exchangeable correlation structure.
For mothers regarding social support and stigma outcomes all mothers received a score similar to the Holzmer 2007 work but accounting for the fact we included an ‘N/A’ response and had a lower maximum possible range (24). To this end, we gave all “Don’t Know, refused, or don’t want to say, or NA” responses a missing value. We then gave those who had the lowest answer on the item (e.g., disagree, or never occurred, or no) a zero and those with the next two levels of responses a one or two, accordingly. We then created a weighted score: all items for the specific topic were summed and the mother received a score by dividing this sum by the total number of non-missing items. Thus, the score always represented the highest level of either social support or stigma. Similar to the child outcomes, the analysis was run in two methods: a facility-aggregate analysis using unweighted t-tests and an individual-level linear GEE. For the multivariable linear GEE model, though differences were not statistically significant, we included covariates for the average number of interviews per facility, if the mother was over 40, if she were married, or had been on HIV treatment for a year.