This mixed-methods study involves the integration of quantitative and qualitative methods to address the research questions. The data sources to be used are outlined in Table 1. Qualitative (including focus group discussions, in-depth interviews, and observation) and quantitative (merged secondary data collected via routine monitoring and evaluation by Pact Tanzania, local implementing partners or LIPs, and Clowns Without Borders South Africa or CWBSA) methods will be used to explore the impact, acceptability, appropriateness, feasibility, fidelity, and cost of PLH-Teens. As randomization to intervention and control groups is not possible, the study will make the most of the routine service delivery data available. Analyzing this data will allow for a unique inquiry into the real-world implementation of a parenting program at scale.
Collaborators and Setting
The FAIR study is being conducted by the National Institute for Medical Research (NIMR) in Tanzania, the University of Oxford, CWBSA, and Pact Tanzania. The study will be conducted in eight districts of rural and semi-urban Tanzania: Kyela District Council (DC), Mbeya DC, Muleba DC, Shinyanga DC, Shinyanga Municipal Council (MC), Kahama Town Council (TC), Msalala DC, and Ushetu DC. PLH-Teens will be delivered by facilitators in schools by teachers and in communities by volunteers (compensated with an honorarium) (N=444) with Furaha program coaches (N=70) providing facilitators with ongoing supervision. Facilitators will deliver the program via the coordination of five LIPs - Humuliza, Tadepa, Integrated Rural Development Organisation, Caritas, and Tanzania Red Cross Society.
Study Participants
The study will collect primary data from 48 program coaches, 96 program facilitators, 58 Pact Tanzania and LIP staff, eight school principals, three CWBSA staff, 155 parents/caregivers, and 155 adolescents. The study will also collect anonymized secondary data from approximately 50,000 parent-child dyads (N=100,000), 444 program facilitators, 70 program coaches, and five LIP organizations. The inclusion criteria used to select study participants for primary and secondary data collection are outlined in Tables 2 and 3.
Study Recruitment and Informed Consent
For the collection of primary qualitative data, a combination of purposive and snowball sampling will be used in collaboration with Pact Tanzania and LIPs to identify potential participants in each of the eight districts for semi-structured interviews and focus group discussions (FGDs). If potential participants consent to their contact details being shared with the researchers, the participants will be contacted by email or phone to outline the scope of the interviews and FGDs prior to seeking informed consent. Alternatively, a researcher may be present during program training or another meeting to explain the study. Once potential participants have been identified, they will be invited to provide informed consent. Pact Tanzania staff will then provide potential participants with consent forms, assent forms (in the case of study participants under age 18), and the opportunity to discuss the study with researchers.
Oxford and NIMR researchers will not be involved in recruiting participants for the secondary data. Instead, Pact Tanzania and CWBSA will ask all program participants if they would like to participate in the research upon their enrolment in the Kizazi Kipya Project. Those who agree to participate in the study will be asked to sign consent forms, and assent forms in the case of participants under age 18.
Primary Qualitative Data Collection
The qualitative data collection methods include: individual semi-structured interviews; FGDs; structured observations of PLH-Teens group sessions conducted by facilitators and structured observations of facilitator supervision sessions conducted by coaches; analysis of policies, progress reports, and other documents anonymized and voluntarily provided by Pact Tanzania; and field notes taken by researchers during community of practice meetings with stakeholders.
Interviews and FGDs.
Individual interviews will be conducted with program coaches (N=16), program facilitators (N=16), LIP staff (N=8), school principals (N=8), and CWBSA staff (N=3). FGDs will be held with program coaches (N=32, 8/FGD), program facilitators (N=80, 10/FGD), parents/caregivers (N=80, 10/FGD), and adolescents (N=80, 10/FGD). All interviews (approximately 60-90 minutes) and FGDs (approximately 90-120 minutes) will be conducted in Kiswahili based on semi-structured interview guides (see Open Science Framework). The guides provide an outline of key topics and questions for the interviewers to ask study participants as well as leaves room to delve into pertinent issues that emerge during interviews and FGDs. All interviews and FGDs will be audio-recorded with the permission of the participants. Where a participant declines, permission will be sought for field notes to be taken instead. Interview and FGD participants will be provided with lunch and transportation to and from the meeting venues (approximately $10-15 USD). In cases where face-to-face interviews and FGDs are not possible, interviews will be conducted remotely via telephone. While the importance of confidentiality will be emphasized during FGDs, participants will be informed about how limited researchers are in their ability to enforce post-discussion adherence to confidentiality commitments made by FGD participants.
Session observations.
To better understand the implementation fidelity of PLH-Teens, researchers will conduct direct observations of program delivery and supervision sessions. Program participants and facilitators will be observed during program sessions (N=5 sessions; 150 participants) and facilitators and coaches will be observed during supervision sessions (N=5 sessions; 50 participants). The exact locations of the five program observations will be selected by the implementation team to take the variation and contextual factors of each district into consideration. A random selection of five coaching observations will be conducted in consultation with Pact Tanzania. Observations of program sessions and supervision sessions will follow structured observation guides (see Open Science Framework).
Document analysis.
The researchers will conduct content analyses of Pact Tanzania and CWBSA reports to identify implementation barriers and supports as well as to identify how PLH-Teens fits within the larger Kizazi Kipya Project. Formal requests will be sent to partner organizations seeking permission to review and analyze relevant documents, with sensitive information redacted before the documents are shared and analyzed.
Community of practice meetings.
Following program delivery, stakeholder engagement meetings will be held with government and non-government stakeholders involved in the implementation of PLH-Teens (N=2 session; 50 participants). In these meetings, stakeholders will be asked to provide an overview of their experiences, including challenges implementing the program and possible solutions to the challenges identified. These participatory community of practice meetings will be held in Dar es Salaam during which researchers will take field notes.
Qualitative data collection tools have been developed based on the EPIS framework and Proctor’s taxonomy. The interview, FGD, and observation guides cover relevant parts of the implementation process experienced by various participants (see Open Science Framework). For example, questions for facilitators focus on the implementation process since they are most familiar with implementation while questions for Pact managers emphasize exploration and sustainment. Each tool includes questions and probes to guide the conversation or observation as well as leaves room to delve into pertinent matters that emerge during conversations or observations. The guides for conversations with implementing staff cover topics including: program likes and dislikes; balancing program delivery with other commitments; perspectives on the training and ongoing supervision provided; anticipated impact on program beneficiaries; barriers and challenges to program implementation; potential improvements to the program and its delivery; impact of COVID-19; and appropriateness of delivering the program via the education system. The guides for conversations with participating family members are similarly wide ranging and covers topics including cultural relevance of the program; experience attending the program; program likes and dislikes; impact of COVID-19; barriers and supports to participation; impact of the program on their family; and perspectives on the quality of their facilitators.
Secondary Quantitative Data Collection
The study will analyze the following anonymized secondary process and outcome data from Pact Tanzania and CWBSA: pre-post surveys completed by parent/caregiver program participants (N=50,000); pre-post surveys completed by adolescent program participants (N=50,000); parent/caregiver and adolescent program attendance registers (N=100,000 participants); facilitator demographic questionnaires (N=444); coach demographic questionnaires (N=70); coach assessments of facilitators (N=444); CWBSA assessments of coaches (N=70); LIP organizational surveys (N=5); and implementation cost surveys (N=300).
Family outcome measures.
Pact Tanzania was provided with a set of process and outcome tools by CWBSA as part of the monitoring and evaluation technical support they provide to all implementing partners delivering PLH programs. CWBSA recommends and provides these tools because they are open-access and have been psychometrically tested in previous studies. Due to the large number of beneficiaries Pact Tanzania is reaching and their limited capacity to collect evaluation data, they are using abbreviated versions of the tools provided by CWBSA.
Demographic items.
The demographic information that will be collected includes parent/caregiver and adolescent age, gender, education level, economic status, food security, health insurance status, HIV status, and home-level risk factors of VAC (15 items).
Positive parenting.
An adapted version of the Alabama Parenting Questionnaire (APQ) (50) will be used to assess parent/caregiver- and child-reports on the frequency of specific parent/caregiver behaviors towards adolescents in the past month on a seven-point Likert scale (0 = never; 6 = always). The APQ measures parental involvement (3 items, e.g., “you/your caregiver get(s) involved in activities that your child/you like(s)”) and parental monitoring (3 items, e.g., “you/your child are/is left at home without adult supervision”) subscales. Items are summed to create a total positive parenting score as well as for each subscale.
Child behavior and mental health.
The Strengths and Difficulties Questionnaire (SDQ) (51) will be used to assess child behavior problems. The tool asks parents/caregivers and adolescents to indicate the frequency of specific child behaviors using a three-point Likert scale (1 = not true; 3 = very true). Pact Tanzania uses the SDQ Conduct Problems subscale to assess externalizing behavior (5 items, e.g., “I get/your child gets angry and often lose(s) my/their temper”) and the SDQ Emotional Problems subscale to assess internalizing behavior (5 items, e.g., “I am/your child is often unhappy, downhearted or tearful”). The items in each subscale are summed, with higher scores indicating more behavioral problems.
Child maltreatment.
The ISPCAN Child Abuse Screening Tools-Trial Version (ICAST-T) will be used to assess parent/caregiver- and child-reports on child maltreatment (4 items). The tool asks parents/caregivers and adolescents to indicate the frequency of emotional abuse (e.g., “shouting or screaming” and “saying mean things to upset,”) and physical abuse (e.g., “spanking, slapping, or hitting with a hand” and “discipline with an object like a stick or belt,”) over the past month using a nine-point Likert scale (0 = never; 8 = 8 or more times) (52). Items are summed to create a total child maltreatment score as well as a score for each subscale.
Acceptability of corporal punishment.
One item from the Multiple Indicator Cluster Survey (MICS) will be used to assess parents/caregivers and adolescent views on the acceptability of corporal punishment. This item asks respondents to indicate the extent to which they agree or disagree (1 = strongly disagree; 5 = strongly agree) with the statement: “In order to bring up, raise, or educate a child properly, a child needs to be physically punished.”
Parental depression.
Parental depression will be assessed using the Centre for Epidemiologic Studies Depression Scale (CES-D 10) (53). The tool asks parents/caregivers to respond to items related to how they have felt over the past seven days (3 items, e.g., “How often in the past week have you felt depressed?”). Responses are coded on a four-point Likert scale (1 = rarely or none of the time; 4 = most or all of the time). Items are summed with higher scores indicating higher levels of parental depression.
Parental support of education.
An adapted version of the Parental Support for School Scale (54) will be used to measure parent/caregiver- and adolescent-reports on the frequency of supportive behavior by parents/caregivers towards their children’s learning (e.g. “I/your caregiver support(s) my child’s/your schoolwork in any way that I/they can” and “I/your caregiver praise(s) my child/you for working hard at school”) using a five point Likert scale (1 = never; 5 = always). Items are summed to create a frequency score, with higher scores suggesting more parental support and value for school.
Economic strengthening.
The Family Financial Coping Scale (FFCS; 6 items) will be used to gain insight into the financial status of the participating families. The tool asks parents/caregivers to respond to items related to financial matters in the past month. These items include questions on whether parents/caregivers were worried about money, saved money, and ran out of money to buy certain items, such as two meals a day.
Intimate partner violence.
Parent/caregiver reports of intimate partner violence victimization and perpetration in the past month will be assessed using four items adapted from the Revised Conflict Tactics Scale Short Form (CTS2S; 8 items) (55). Items included in the tool ask about the frequency of physical assault (e.g., “my partner/I hit, push, shove, or slap me/my partner”) and psychological aggression (e.g., “my partner/I insult(s), shout(s), yell(s) or swear(s) at me/them”). Answers are coded using the same nine-point Likert scale as the ICAST (0 = never; 8 = 8 or more times). Items are summed, with higher scores indicating higher levels of victimization or perpetration of intimate partner violence.
School violence.
Child experience of school violence will be assessed using three items, one on bullying (“In the past 4 weeks, how often did you experience any bullying at school such as persistent name calling, threats of violence, or physical attacks?”), one on physical discipline from adults at school (“In the past 4 weeks, how often did a teacher or any other adult discipline you at school by hitting you with their hand or an object like a stick or belt?”), and one on verbal discipline from adults at the school (“In the past 4 weeks, how often did a teacher or other adult at your school discipline you by shouting, yelling, or screaming at you?”). These questions were designed by FAIR study researchers and will be coded using the same nine-point Likert scale as the ICAST (0 = never; 8 = 8 or more times). Items are summed with higher scores indicating higher levels of school violence victimization.
Other measures.
As part of their monitoring and evaluation of the broader Kizazi Kipya Project, Pact Tanzania collects a variety of information from all enrolled families: HIV status and HIV risk assessment; caregiver-child communication on sexual and reproductive health (about puberty and growth, safe sex practices and contraceptive methods, relationship with adults, sugar daddy/sugar mummy); food security; and other sociodemographic indicators such as wealth quintile and household size.
Cost measures.
Information about the time and resource costs of program set-up and implementation will be collected by Pact from facilitators, coaches, and LIP coordinators to determine how much program delivery costs at scale. Costing information will be collected using surveys which ask participants for retrospective estimates of the amount of time used or money expended on a program activity. The surveys were created based on resources provided by The Abdul Latif Jameel Poverty Action Lab. The collection of cost information will also include a review of program budgets, spending, and other data obtained from Pact Tanzania about the resources required to set-up and deliver the program.
Implementation process measures.
Pact Tanzania, LIPs, and CWBSA will collect data about parents/caregivers and adolescents (e.g., attendance), facilitators (e.g., demographic characteristics, fidelity), and coaches (e.g., demographic characteristics, fidelity) (see OSF page). The data will be used to understand the quality of program implementation, the factors that predict implementation outcomes, how implementation varies from context to context, and how implementation is associated with intervention outcomes. All of the implementation data will be linked to parent/caregiver and adolescent outcome data through the use of unique identifiers supplied by LIPs which will make it possible to link data from multiple sources. The data will be anonymized by the LIPs before it is shared with researchers.
Attendance. Attendance refers to the number of sessions attended by a program participant out of the total possible number of sessions offered to the participant. Attendance data will be collected by Pact Tanzania via attendance registers completed by facilitators each week. An overall attendance rate will be calculated for each parent/caregiver-child dyad.
Staff demographic data. Pact will collect demographic data on facilitators and coaches using an implementation staff questionnaire (Facilitator and Coach Profile Forms). The demographic data to be collected includes facilitator/coach age, gender, marital status, parental status, number and age of children, employment status, and educational level. The questionnaires will also assess facilitator/coach self-efficacy and their view on the acceptability of corporal punishment using the same MICS item administered to parents/caregivers and adolescents.
Facilitator competent adherence. Facilitator competent adherence is the skill with which a facilitator delivers intervention components and the strictness with which they follow the activities outlined in the programme manual (56, 57). Data on facilitator competent adherence will be collected by Pact coaches using the PLH-Facilitator Assessment Tool for Teens (PLH-FAT-T). The PLH-FAT-T is an observational assessment tool which will be administered by coaches based on either live observations or video recordings of facilitator group sessions. The PLH-FAT-T was developed by the study investigators and PLH program developers to assess the proficiency of program delivery by facilitators as a prerequisite to their certification. The items in the tool are grouped into two subscales based on the core activities and process skills required of facilitators as outlined in the PLH-Teens program manual. The assessment of core activities (22 items) requires coaches to rate the quality of facilitator delivery during home activity discussions (11 items, e.g., “identify specific challenges when shared by at least one parent”) and role-plays (11 items, e.g., “make sure everyone can see and hear the action in the role-play”). The assessment of process skills (28 items) requires coaches to rate the quality of facilitator use of modelling skills (5 items, e.g., “give positive, specific, and realistic instructions”), the Accept-Explore-Connect-Practice facilitation technique (8 items, e.g., “accept participant responses verbally by reflecting back what the participant says”), and collaborative leadership skills (15 items, e.g., “use open-ended questions during group discussions”). Each PLH-FAT-T item is rated on a three-point Likert scale ranging from zero to two (0= inadequate, 1= good, 2= excellent). By totaling the score from all items, an overall impression score is produced and represented as a percentage.
Coach competent adherence. Data on coach competent adherence will be collected by CWBSA staff using the PLH-Coach Assessment Tool (PLH-CAT) which is an observational assessment tool similar to the PLH-FAT-T. The PLH-CAT assesses the quality of coaching provided to facilitators based on either live observations or video recordings of coaching sessions. The tool includes an activity subscale which assesses a coach’s review of delivery highlights and challenges (12 items) and use of process skills similar to those assessed by the PLH-FAT-T (26 items). Each PLH-CAT item is rated on a three-point Likert scale ranging from zero to two (0= inadequate, 1= good, 2= excellent). By totaling the score from all items, an overall impression score is produced and represented as a percentage.
Organizational surveys. A short organizational survey has been developed to gather LIP characteristics from staff and to explore their observations about variations in program adoption and differences between the districts.
Data Analysis
Qualitative analyses.
Qualitative data will be transcribed verbatim and translated into English. Analysis will be conducted with the aid of NVIVO 12 qualitative analysis software. Multiple researchers will review the translated transcripts to generate a coding framework based on the research questions and a thorough reading of a sample of interview and FGD transcripts. Following the creation of the coding scheme, the data will be double coded to establish reliability among the researchers. Thereafter, data-driven coding will be used to identify concepts, relationships, and broad themes (thematic analysis). The findings will then be discussed by the research team to identify overarching themes and to select and extract data segments that represent the key themes and divergent viewpoints. Where appropriate, COREQ standards will be used when reporting qualitative data (58).
Quantitative analyses.
Quantitative data will be cleaned using Stata and analyzed in Stata and R, using methods such as correlation and regression analyses, as well as structural equation models. The frequencies and distribution of each variable will be examined to check for any implausible values as well as to select the appropriate analysis method (e.g., a suitable regression link function). When there are more than two items in a given scale, coefficients such as Cronbach Alphas or Omegas will be used to assess the item-level reliability of the measures. Where possible, mixed effect models will be utilized to account for nesting within parenting groups (59). Missing data will be addressed appropriately by considering the complete case observations as well as using full information maximum likelihood or multiple imputation, as appropriate (60, 61). Where relevant, TREND guidelines will be used when reporting quantitative results (62).
Research question 1.
The level of implementation of PLH-Teens delivery will be determined by analyzing data from family participation registers; facilitator assessments; coach assessments; structured observations of group sessions; individual interviews held with facilitators, coaches, and LIP staff; and focus group discussions held with adolescents, parents/caregivers, facilitators, and coaches. Attendance rates and attendance trends among parents/caregivers and adolescents, as well as variations in attendance, and program completion rates will be calculated based on the attendance registers to determine the extent of participation in PLH-Teens. The level of competent adherence with which facilitators deliver the program will be determined using the results from the Facilitator Assessment Tool assessments completed by coaches. To examine the reliability and validity of the Facilitator Assessment Tool results, a psychometric evaluation consisting of content validity (stakeholder perspectives from interviews and focus groups with facilitators, coaches, and CWBSA staff), intra-rater reliability (percentage agreements and intra-class correlations), inter-rater reliability (percentage agreements and intra-class correlations), internal consistency (Cronbach Alphas), construct validity (exploratory factor analyses), and predictive validity analyses will be performed. Similarly, the level of competent adherence with which coaches deliver facilitator supervision will be determined using the results from the Coach Assessment Tool assessments completed by CWBSA staff. Interviews, focus group discussions, and session observations will be used to expand upon and contextualize the findings regarding the demographic, attendance, facilitator competent adherence, and coach competent adherence data.
Research question 2.
Factors associated with the quality of implementation will be examined using the socio-demographic data from the Facilitator and Coach Profile Forms; LIP organizational characteristics surveys; individual interviews; focus group discussions; and structured observations of group sessions. Correlation and regression analyses will be used to examine the relationship between facilitator and coach competent adherence and their associations with family, facilitator, coach, and organizational characteristics. Interviews, focus group discussions, and session observations will be used to expand upon and contextualize the findings.
Research question 3.
A variety of data sources will be used to examine how implementation is associated with changes in VAC and family well-being. In particular, correlation and regression analyses will be used to look at whether pre-post changes in family outcomes are associated with family attendance, facilitator and coach competent adherence, and facilitator and coach characteristics, as well as LIP characteristics. Interviews, focus group discussions, and session observations will be used to expand upon and contextualize the findings.
Research question 4.
Participant and implementing staff perspectives on the acceptability, appropriateness, feasibility, benefits, and challenges of delivering PLH-Teens in their communities will be examined by analyzing the interviews, focus group discussions, and session observations with school principals, facilitators, coaches, LIP staff, CWBSA staff, adolescents, and parents/caregivers.
Research question 5.
Changes in VAC and participant well-being will be analyzed based on data gathered from parent/caregiver pre-post questionnaires, adolescent pre-post questionnaires, individual interviews, and focus group discussions. Multi-level models will be used to examine differences in pre- to post-intervention family-level outcomes and to compare differences in outcomes reported by both adolescents and parents/caregivers. Variation in the pre-post effects changes will be examined by participant baseline characteristics, and, if possible, by parenting group and LIP. The analyses will be similar to treatment-on-the-treated analyses since all participants included in the monitoring data would have engaged with the program to some extent. The levels of change reported by participants will be compared to the levels of change reported by the treatment and control groups in the randomized trial of the program in South Africa. Where possible, the reliability of the family survey items will also be examined using coefficients such as Cronbach Alphas or Omegas.
The findings from the interviews and focus group discussions will also be analyzed to explore participant perspectives on the impacts of the program on them and their families. The interviews and focus group discussions will also reveal what impact implementing volunteers and staff assess the program had and will have on themselves, participants, schools, and communities.
Research question 6.
The cost of delivering PLH-Teens at scale will be calculated using retrospective cost estimates provided by program facilitators, coaches, and LIP coordinators and costing data provided by Pact Tanzania. Average costs will also be calculated and summarized for each program component (e.g., facilitator training, group sessions, supervision), family (parent/caregiver-adolescent dyad), district, and facilitator type (community volunteer or teacher).
A summary of the data that will be analyzed to answer each of the FAIR study’s six research questions is shown in Table 4.