Six program managers were interviewed for data collection. Five of them were woman, and one was a man, with ages ranging from 38 to 52 years. Ten facilitators were also interviewed (three men and seven women), with ages ranging from 23 to 49 years. Interviews were conducted with one manager and two facilitators from each institution. Eighteen adolescents aged 10–14 years and 33 adults participated in the focus groups.
A total of 53 families were enrolled in 11 groups. One group had two families, one group had three families, eight groups had five families, and one group had 10 families (sessions were held in Google meetings in this group to include all the interested participants). Fifty-six guardians were registered (50 mothers, two grandmothers, and four fathers), and for three families, two guardians were present (mother and father). Fifty-seven adolescents aged 10–14 years participated in the study. In the workshops, there was an average participation of three families in the adolescent, parent, and family workshops. Researchers have noted that other family members (including fathers, grandparents, aunts, and younger and/or older children) often participate in sessions although they are not formally enrolled.
The results of the methods applied based on the implementation variables are presented below. Table 2 summarizes the qualitative results and data sources. Table 3 presents the main alterations suggested to address the difficulties encountered.
Table 2
Synthesis of the qualitative results and data source
| Topic | Subtopic (data source) | Results and Golden quotes |
| Fidelity /Core elements | Time for application of activities, proportion of activities performed, homework (fidelity forms) | Activities carried out in full, but with low fidelity in application |
| Core elements kept (observations and interviews of researchers) | Most core elements kept. One not kept: the virtual version did not allow the creation of a support group among the participants. “The proximity with the families augmented significantly during the program application” (Interview G1) “Our relationship with the parents was very different after the program” (Interview G1) |
| Sending videos and audios (observations) | Videos sent correctly; audios sent incompletely in general |
| Appropriateness | Accessibility (focus groups) | “If it wasn’t the online program I would not have been able to participate” (FG adults 5) “It’s impossible to organize myself to go every week to the institution” (FG adults 6) |
| Videos on WhatsApp (focus groups) | “It was good to be able to watch the videos from home” (FG adults 3) “I would sometimes watch the videos multiple times; I even sent them to some friends” (FG adults 3) |
| Confidentiality (researchers’ observations) | Workshop confidentiality not respected (presence of other family members in the same room during the workshops) |
| Feasibility | Internet access (researchers’ observations and interviews) | The course of the meetings much more impacted by the connection problems of the facilitators than of the participants. “When we had internet connection problems it was really difficult to continue the meeting” (Interview F3) |
| Presentation of materials through the camera (researchers’ observations) | Activities that require the facilitator's camera to be used to show something did not work |
| Institutional Organization (interviews) | “When we are doing the program, all of our other activities in the week are impacted by it” (Interview F5) “The ideal would be to have a team only dedicated to the application of the program” (Interview G4) |
| Too short a time for complex activities (researchers’ observations) | Workshops with many consecutive short activities presented greater difficulties in execution. |
| Facilitators’ training (interviews) | “The training is insufficient for the facilitators that have more difficulties” (Interview G1) “We need more experienced facilitators to be able to help the ones that are beginning” (Interview G5) |
| Activities | “Special effects” activity (researchers’ observation) | Facilitators suggesting that drug use is equivalent to having superpowers. |
| "Listing Small Household Tasks” activity (focus groups) | “He has to know that living in a house he must share obligations like everyone else and that shouldn’t be rewarded” (FG adults 2) |
| Activities: "Supporting Children's Dreams and Goals" and "How Your Parents Treated You” (researchers’ observations and focus groups) | Activities that lead many adults to relive difficult memories of their own childhoods. "We have our childhood traumas, I had many (...) we need to make a difference (...) when this was dealt with in the program, I saw myself in those stories and thought 'I need to change'" (FG adults 2) |
Table 3
| Category | Difficulty Encountered | Suggested Solution |
| Fidelity | Failure to form a support group among participants (Core element) | Organize informal in-person meetups for participants, such as coffee and snack breaks |
| Appropriateness | Too many videos sent between meetings 5 and 6, hindering effective utilization of materials | Remove some of the videos or schedule dedicated time to watch them at the beginning of the next meeting |
| Lack of confidentiality in the workshops | Highlight the importance of maintaining confidentiality during the facilitators' training |
| Feasibility | Internet connectivity issues, particularly for facilitators, significantly impacting the progress of the meetings | Make a reliable internet connection an absolute priority for participating institutions |
| Activities requiring the facilitator's camera to present materials are not functioning | Short-term: Verbally convey the phrases or describe the images to participants instead of showing them through the camera Medium-term: Include the materials (phrases, images) in the participants' workbook |
| Difficulty in organizing institutionally for program implementation, demanding excessive time from facilitators | Adapt the workload of professionals working as facilitators in the program to ensure they are not overwhelmed and do not negatively impact overall institutional functioning |
| Insufficient time for complex activities | Reorganize the workshops to have fewer activities, but allocate more time to each activity |
| Inadequate training to adequately equip facilitators | Include an opportunity for facilitators to interact with experienced facilitators responsible for the program during their training |
| Activities | "Special Effects" activity associates drugs with superpowers | Modify the example of superpowers in this activity or remove it completely |
| "Small Household Tasks" activity associates domestic chores with punishment | Remove domestic chores from the points system |
| "Supporting Children's Dreams and Goals" and "How Your Parents Treated You" activities trigger difficult childhood memories | Include a warning in the facilitator's manual about the potential for intense emotional reactions from participants during these activities, along with suggestions for facilitators on how to manage these situations |
3.1. Fidelity /core elements
The evaluation found low fidelity in the execution of several planned activities, and most core elements were retained in the online version. In 75% of the workshops, the facilitators considered that they had sufficient time to carry out all the proposed activities. The facilitators declared that they had carried out an average of 86% of the planned activities in each workshop: 90% in the children's workshop, 87% in the caregivers' workshop, and 79% in the families' workshops. However, the researchers’ observations revealed low fidelity in the application of certain activities.
In 79% of the groups, all videos were correctly sent following the script instructions. In contrast, audio clips were correctly sent in only 36% of the groups. Only 47% of adult participants completed the tasks they were supposed to complete throughout the meeting weeks, with a significant drop over the weeks (from 68% in the first week to 20% in the fifth week). The study also evaluated whether and how the program's core elements were affected by the online version. This research suggests that seven of the core elements of the program were maintained in the online version: themes and sequence of the meetings were respected (except for one activity called “Special Effects,” detailed later); participants engaged with the distribution of the videos on their cellphones weekly; duration, format, and content of the activities was respected; the facilitators believe that their link with the families facilitated improvement; the logistics of attending the meetings was respected; and the profile of the families was also maintained. However, one core element was negatively affected, that is, the online version did not facilitate the creation of a support group for participants.
3.2. Appropriateness
Confidentiality: The confidentiality of the adolescents' workshops appeared in the evaluation as a hindrance, but the accessibility and the sending of the videos via WhatsApp in the online version was perceived in a positive way by the participants. During the meetings, researchers observed difficulty in maintaining the confidentiality of the workshops as the adults were often in the same room as the adolescents. A suggestion for improvement is to highlight this shortcoming during the facilitators’ training, reminding them to watch out for this concern and whenever an adult is present, to kindly ask them to respect this rule and not be in the same room.
Accessibility: In the focus groups, the participants stated that the online version improved accessibility for those who would not have been able to guarantee face-to-face participation, given the difficulties some would have faced in traveling to the institution for meetings.
Video materials: In the virtual version, videos are sent gradually through the WhatsApp chat group to be watched throughout the week, and there is interaction with facilitators through message exchange. The adult participants positively perceived the distribution of content and interaction with facilitators over time. However, on one specific week (between meetings 5 and 6), the participants said that they had received too many videos and did not have time to watch all of them. We suggest that some of the videos could be removed or, in meeting six, facilitators might give participants time to watch the videos.
3.3. Feasibility
The feasibility of the program was affected by various factors, including low-quality internet connections, difficulties in presenting materials through cameras, internal organization problems within institutions, insufficient time for complex activities, and inadequate facilitator training provided by the MMFDH platform.
Internet access: Facilitators faced difficulties connecting to the internet and experienced poor video call quality. These issues disrupted the sessions, particularly in groups with few participants, where interruptions due to internet failure significantly affected group dynamics. To address this, it is recommended that institutions prioritize ensuring a high-quality internet connection for facilitators when implementing the online version of the program.
Presenting materials through the camera: The activities where the facilitator was supposed to present material led to recurring execution problems. The facilitators' cell phone cameras did not provide clear images, causing technical difficulties and impacting the activity's development and fidelity. In groups that avoided using the cell phone camera and instead read aloud the content that should be shown, fewer problems were encountered. In the short term, it is suggested to simplify these activities by conducting them through spoken phrases without attempting to show content through a cell phone camera. In the long term, modifying the adult and adolescent activity books to include necessary images and phrases for these activities is recommended.
Internal organization within institutions: Scheduling conflicts for facilitators arose, as their workload extended beyond the two-hour weekly sessions. They were also responsible for sending audios and videos, answering participants' questions, and assigning homework. This was a major challenge reported by managers and facilitators. To address this issue, it is recommended to provide institutions with an estimate of the time needed to implement the program and prioritize time management for facilitators during the application weeks. Additionally, establishing a dedicated team solely responsible for program implementation would be ideal for long-term continuity.
Insufficient time for complex activities: Many short activities were often completed incompletely or incorrectly when performed in sequence. In workshops with fewer planned activities, they were more thoroughly accomplished. It is suggested that the program's manual be revised by the MMFDH to maintain the main activities while reorganizing and combining several smaller activities into longer, cohesive ones whenever possible. This approach would provide clarity and enhance the effectiveness of the application.
Facilitator training: Managers and facilitators felt that the training offered by the MMFDH platform was insufficient for correct program implementation. Lack of interaction with a trainer or tutor to address specific questions and the absence of reports from experienced facilitators were highlighted as drawbacks. To improve training, it is recommended to review the facilitator training program and incorporate mentoring by experienced facilitators. If direct mentoring is not feasible, including videos that depict practical examples of errors in program implementation and demonstrate expected outcomes would be beneficial.
Addressing these feasibility challenges will contribute to the successful implementation and effectiveness of the program. By prioritizing internet access, revising activity organization, improving facilitator training, and ensuring effective internal organization within institutions, the program can overcome obstacles and deliver better outcomes.
3.4. Specific activities that presented problems
Four program activities require attention:
"Special effects" activity: In the fifth meeting of the adolescents' workshop, this activity aimed to teach them not to blindly trust television. However, facilitators in multiple groups changed the topic and compared it to using drugs with special powers. To address this, either change the example or completely revise the activity. Program managers suggested a hybrid version with face-to-face moments to facilitate support group formation.
"Listing domestic chores" activity: In the fourth meeting of the caregivers' workshop, the activity introduced a "point system" where adolescents gain or lose points based on their behavior and activities. Domestic chores were included as possible consequences of negative acts. However, some parents expressed discomfort with associating domestic activities with punishment. To address this, remove domestic activities from the point system or clearly explain their role and importance in the family routine.
"Supporting Children's Dreams and Goals" and "How Your Parents Treated You" activities: These activities occur in the first and fifth meetings of the caregivers' workshop. Participants reflect on their upbringing, their relationship with their parents, and what they would change to avoid repeating negative experiences with their children. Some participants expressed difficulty due to their past experiences. While considering the impact on mental health, the activities had a positive effect overall. Retain the activities but add a note to the facilitators' script about potential effects and provide more time. Ideally, offer support options like a Psychosocial Care Center (CAPS).
Addressing these problematic activities is crucial for improving the program's effectiveness and participant experience. By making appropriate revisions and providing support where needed, the program can better meet participants' needs and expectations.