Implementation Research for the Evaluation of the Child Health Education and Surveillance Tool Application

Community health agent programs and modern information and communication technology can greatly improve knowledge of healthy childrearing practices by caregivers in low resource settings, if implemented effectively. Improved knowledge by caregivers can lead to better sanitation, diet, and child development practices. A digital health tool (CHEST App) was developed and deployed in a community health agent program in the Amazon of Peru to improve community health agent performance and ultimately improve early childhood development in the communities. This study presents the results of and evaluation of the implementation and clinical outcomes of the program. METHODS The CHEST App intervention was evaluated using a Hybrid Type II evaluation study design. The effectiveness of the intervention was determined by conducted a paired t-test analysis to compare the mean differences in knowledge scores, hemoglobin levels, early childhood development (ECD) scores, and incidence of diarrhea. The evaluation of the implementation outcomes was conducted with a mixed method approach to identify the extent to which the intervention was successfully installed into the local CHA program. The results of the study are presented within the framework of the Implementation Research Logic Model. RESULTS The CHEST App intervention is associated with improvements in knowledge scores, hemoglobin levels, ECD scores, and decreased diarrhea. However, the evaluation could not isolate the effect of the intervention due to reduction in sample size from COVID-19 closures. The implementation of the CHEST App intervention was effective with high degrees of acceptability, adoption, and delity. Adoption and delity of the surveillance function of the CHEST App by program coordinators was not achieved. CONCLUSION The CHEST App intervention is a promising tool to improve the performance of CHAs during their home visits, to accomplish their objective of teaching caregivers healthy childrearing practices and improving child health and development in their communities. Trial Registration Trial registered on 11/29/2018 at ISRCTN43591826. describes how multiple innovative implementation science tools can be combined to guide implementation research, including the CFIR ERIC Matching Tool, Implementation Research Logic Model, Consolidated Advice on Reporting ECD Implementation Research (C.A.R.E.) guidelines, Hybrid Type II evaluation study design, Active Implementation Frameworks, Proctor’s outcomes, and others. of implementation, implementation strategies, and the implementation, service, and clinical outcomes. 26 The results of the study are presented within the framework of the IR Logic Model in the Discussion Section (Appendix 2). The implementation outcomes were selected based on the conceptual framework presented by Proctor, et. al., 2011. 39 The implementation outcomes serve as a precondition for attaining the desired changes in service outcomes and clinical outcomes. The implementation outcomes relevant to this study include, acceptability, adoption, delity, and cost. Other implementation outcomes were not measured because the intervention had not yet reached the point in its maturation to measure them with condence, including, feasibility, penetration, and sustainability. Acceptability was measured to identify the extent to which the implementation stakeholders (caregivers and providers) perceive the innovation to be satisfactory. 39 Both the CHAs and caregivers in the intervention group received a semi-structured interview to determine their level of acceptance of the CHEST App. The quantitative portion of the survey was analyzed and reported using descriptive statistics. The qualitative responses were analyzed by identifying a set of sub-themes from the responses then assessed to identify similarities and differences in perspectives of the participants. Key quotes that best reect the position of each actor group is reported. components App an open-ended questionnaire signicance increase at endline in cognitive scores, +1.029 (95% 1.342) t(55)=6.585, p<0.000) and in motor scores, +0.714 (95% 0.377 1.052) km, t(55)=4.24, p=0.001). A paired t-test on ECD scores by children in the intervention group at endline and scores by children in the control group at endline showed no statistically signicant difference in cognitive (p=0.526) nor motor scores (p=0.305).


Contributions To The Literature
The current study is exemplary for how implementation research methods and tools can be used to evaluate and report a pilot study, to improve learning and replicability.
The research presented here provides important generalizable knowledge about the feasibility and value of providing community health agents a digital tool to improve their capacity to teach caregivers healthy childrearing practices.
Animated videos, automated message selection, and the novelty of an electronic device are associated with increase knowledge acquisition by caregivers in rural Amazonian communities.
The study describes how multiple innovative implementation science tools can be combined to guide implementation research, including the CFIR ERIC Matching Tool, Implementation Research Logic Model, Consolidated Advice on Reporting ECD Implementation Research (C.A.R.E.) guidelines, Hybrid Type II evaluation study design, Active Implementation Frameworks, Proctor's outcomes, and others.

Background
Children around the world continue to suffer from ailments such as malnutrition and developmental delay due to unhealthy practices in the household. [1][2][3][4] A large portion of the ailments could be alleviated by the adoption of better sanitation practices, diet, and disease prevention, however, caregivers' knowledge of healthy practices can be limited due to poor education and health promotion efforts in impoverished settings. [5][6][7][8][9] Effective transmission of knowledge of healthy practices has been di cult to achieve at scale in rural communities, but when accomplished it can substantially improve child health and development. 10 The utilization of community health agents (CHAs) to conduct health promotion and education in their communities has been shown to be effective, though outcomes vary greatly. [11][12][13][14] The practice represents an evidence-based intervention that can improve child health outcomes, however, effective, consistent, and scalable implementation of CHAs programs have been elusive. 14,15 The utilization of information and communication technology (ICT) has also been shown to be effective at improving health behavior in the household. [16][17][18][19][20][21][22][23][24] However, effective implementation of ICT tools in rural settings has also been varied, and most populations in impoverished settings have not received the bene ts of knowledge transmission through modern technology. 25 Ineffective implementation of evidence-based interventions, such as CHA programs and ICT tools for behavior change, is partly due to an imprecise understanding of what has been done, what has worked, and what has failed. The implementation of innovations needs to be mapped, evaluated, and reported with su cient detail to support continuous learning and improvement. [26][27][28] The current study reports on the implementation and impact of an mHealth innovation to improve CHA performance in the Amazon of Peru. The implementation research was conducted to generate data on the success and failures of the implementation strategies to install the innovation and the feasibility of the intervention to function effectively in the rural setting.
The complete study protocol is described in the article, Westgard, et. al., 2018. 29 The speci c stages of implementation (based on the Active Implementation Frameworks 30,31 ) and implementation strategies that were used to install the intervention are described in the article, Westgard, et. al., 2020. 32 The intervention was implemented in the northern Amazon region of Peru, in the department of Loreto. In Loreto, 57% of children under 3 have anemia, 20% under 5 have chronic malnutrition, and 26.7% of children are developmentally delayed. 33,34 Caregivers' knowledge of healthy childrearing practices are limited as they transition from traditional practices to modern medicine. [5][6][7]9 In many rural communities in Peru, CHAs conduct home visits with new mothers to teach them health topics such as sanitation, diet, disease prevention, and early childhood development. However, the impact of the CHA programs has been hindered by poor performance of CHAs and a lack of effective educational material. The CHAs are often unequipped and undertrained to successfully transmit the information to the caregivers and convince them the importance of the behavior change. 15,35,36 To address the underperformance, the research team at the Peruvian research organization, Elementos, developed a tablet-based application and animated videos to support the CHAs in their task to teach caregivers key health messages and collect child health indicators during home visits. The innovative tool is titled, The Child Health Education and Surveillance Tool Application (The CHEST App). A video of the CHEST App and the animated videos to accompany the App can be viewed online. 37 The primary objectives of the CHEST App intervention are to improve early childhood development (ECD) scores, reduce anemia rates, and reduce chronic malnutrition rates. The theory of change of the intervention is displayed in Appendix 1.
The pilot study was interrupted by the COVID-19 pandemic, causing CHA program shutdowns and thus a signi cant loss of program participants and sample size. After 6 months of shutdowns, one community in the intervention group and one in the control group reactivated their CHA programs. The intervention group community continues to use the CHEST App in their program. The other municipalities maintain their CHA programs inactive but anticipate using the CHEST App when they reactive their programs. Due to the loss of program participants, the program evaluation methodology had to adapt to assess the effectiveness of the intervention with a smaller sample size. The adapted statistical analyses are described below.

Study Design
The implementation process and CHEST App intervention are evaluated using a Hybrid Type II evaluation study design. 38 In a Hybrid Type 2 study, the implementation process is evaluated alongside the intervention (the CHEST App) to determine what mechanisms are responsible for the change associated with the intervention. 38 The study took place in the Amazon of Peru, department of Loreto. The study includes a randomly selected intervention community and a control community. The communities were matched with propensity score matching and then randomly assigned to a group. The intervention group is the community of Indiana and the control group is the community of Tamshiyacu. Both communities are similar in size (8-10,000), access to health facility (Level I-2 in community), distance to province capital (1.5 hours in fast boat), rates of anemia (25-32%), and structure of CHA program in the community.
A baseline survey was conducted prior to implementation of the intervention and an endline survey was conducted 16 months following implementation, in the intervention and control communities. The research team conducted a consensus in each community to survey all children in the communities ages 6-36 months, regardless of if they were active in the CHA program. A detailed description of the selection process and study population is included in the published study protocol. 29 The effectiveness of the intervention was determined by measuring changes in knowledge scores, hemoglobin levels, early childhood development (ECD) scores, and incidence of diarrhea. To determine if the intervention and control communities were similar at baseline, a paired t-test was conducted on the mean differences of each intervention outcome. To assess changes in the intervention outcomes associated with the CHEST App, a paired t-test analysis was conducted to compare the mean differences in the intervention group at endline and baseline, and between the intervention group and control group at endline. A difference-in-difference analysis or analysis of covariance regression were not used due to the reduced sample size of the study.
The knowledge score represents the service outcome of the study, as described by the conceptual model by Proctor, et. al., 2011. 39 The service outcome of the study re ects the effectiveness of the CHEST App to improve CHA performance. If CHAs are effective at delivering their intended service, the caregivers will have greater knowledge pertaining to healthy childrearing practices. Improved knowledge acquisition is the intermediary variable that is expected to lead to improvements in the clinical outcomes (anemia, ECD scores, diarrhea). Knowledge scores were measured by an opened-end questionnaire with the CHAs and caregivers, at baseline and endline. The questionnaire was designed to give the participant ample opportunity to describe what they know about each question. The surveyors were trained to ask the participant the survey question, then probe the participant to provide further information. For example, the participant was asked, what are the bene ts of breast milk for a child, then follow-up with statements such as, "what other bene ts" and "have you heard of any other bene ts". Probes continued until the participate indicated that they do not know any further information. For each correct answer, the participant received a point. The points were totaled to provide the knowledge score for the participant. The questionnaire included 15 questions with a total possible score of 91. The topics included, nutrition, sanitation and hygiene, disease prevention, and early childhood development. ECD was evaluated with the Caregiver-Reported Early Development Instrument (CREDI), an instrument that utilizes caregiver-reported data to assess the ability of the child to perform age-related developmental activities. [40][41][42][43][44][45] The CREDI is conducted by asking the caregiver if the child is able to complete a list of age-appropriate activities that become progressively more di cult. The instrument was used to assess motor development and cognitive development. A developmental z-score is calculated for each development measure (motor and cognitive). The z-scores (+ or -) were used as the ECD score for the study. The CREDI has been validated as an effective instrument to evaluate ECD in 17 low-and middle-income countries. 45 Days with diarrhea by the child was measured because the educational material of the CHEST App presents information that is intended to improve sanitation practices of the caregiver. If the CHEST App is effective, it should improve sanitation practices and thus, decrease the incidence of infectious diseases, such as diarrhea. To measure the intermediary outcome, the caregivers are asked if their child has had diarrhea in the last month, and if so, how many days have they had diarrhea. Although the result is affected by recall bias by the caregiver, and thus, may not be precise, the gross estimation provides an effective measure to calculate the population average of each group.
The evaluation of the implementation outcomes was conducted with a mixed method approach to identify the extent to which the intervention was successfully incorporated and adopted into the local CHA program. The implementation research measures the activities of the program, as displayed in the Theory of Change of the CHEST Program (Appendix 1). The implementation process, including barriers and adaptations, is described in the publication by Westgard, et. al., 2020. 32 Reporting the implementation process was done to allow others (researchers and program implementers) to understand what activities took place during the implementation of the intervention that may have in uenced the success or failure of the intervention. 46 The implementation research is reported here following the C.A.R.E. guidelines (Consolidated Advice on Reporting ECD Implementation Research) to ensure the necessary information is included in the evaluation 47 The Implementation Research Logic Model (IR Logic Model) was used to design the evaluation process and specify the relationship between the determinants of implementation, implementation strategies, and the implementation, service, and clinical outcomes. 26 The results of the study are presented within the framework of the IR Logic Model in the Discussion Section (Appendix 2). The implementation outcomes were selected based on the conceptual framework presented by Proctor, et. al., 2011. 39 The implementation outcomes serve as a precondition for attaining the desired changes in service outcomes and clinical outcomes. The implementation outcomes relevant to this study include, acceptability, adoption, delity, and cost. Other implementation outcomes were not measured because the intervention had not yet reached the point in its maturation to measure them with con dence, including, feasibility, penetration, and sustainability.
Acceptability was measured to identify the extent to which the implementation stakeholders (caregivers and providers) perceive the innovation to be satisfactory. 39 Both the CHAs and caregivers in the intervention group received a semi-structured interview to determine their level of acceptance of the CHEST App. The quantitative portion of the survey was analyzed and reported using descriptive statistics. The qualitative responses were analyzed by identifying a set of sub-themes from the responses then assessed to identify similarities and differences in perspectives of the participants. Key quotes that best re ect the position of each actor group is reported.
Adoption was measured to determine the uptake of the innovation and continued use throughout the study period. Adoption of the CHEST App by the CHAs was measured by analyzing data that was collected with the CHEST App and then uploaded to the server. The number of home visits conducted with the CHEST App was compared to the number of home visits the CHAs were expected to complete each month to determine a percentage of use. The CHAs reported in the semi-structured interview the extent to which they used the CHEST App during their home visits and the components of the CHEST App they use with consistency. The caregivers also reported in their semi-structured interview the number of home visits they have received from a CHA that used an electronic tablet. The responses from the CHAs and caregivers are reported. The program coordinator of the CHA program received an open-ended questionnaire pertaining to the adoption of the CHEST App and the use of each of its components. The response to the questionnaire is summarized.
Implementation cost is reported to provide the nancial impact of the implementation effort. The costs include equipment, training, and supervisory services. The costs are broken down to provide a per CHA cost and the cost to implement the intervention with 20 CHAs.
Fidelity was measured to determine the degree to which the innovation was delivered as prescribed. The research team conducted observations of home visits with the CHEST App to determine if each component of the CHEST App was being used as intended. The observers had an observation checklist to mark the completion, or incompletion, of each component. The results of the checklist are reported as the measure of delity.
A SPIRIT Checklist was completed to ensure the manuscript satisfying research reporting guidelines. The SPIRIT Checklist is included in Appendix 3.

Implementation Outcomes
The results of the implementation research can be seen in the Implementation Research Logic Model in Appendix 2. The evaluation of the implementation outcomes included 41 caregivers and 6 CHAs in the intervention community.

Acceptability by Caregivers
Most caregivers (98%) that received home visits with the CHEST App expressed that they prefer the App than use of traditional methods (pen, paper, and ipcharts). 5% expressed that they had no preference and 3% expressed that they prefer traditional methods of home visits.
When caregivers were asked if they learned the health messages delivered during the home visits better with the CHEST App or without the CHEST App, 78% expressed that they learned better with the CHEST App, 13% expressed that they learned the same with or without it, and 3% (n=1) expressed that they learned better with traditional methods. When asked what they liked most about receiving home visits with the CHEST App, all caregivers answered that they most enjoyed the educational component of the App. Many (43%) speci cally mentioned the animated videos as a reason why they prefer the CHEST App to traditional methods. For example, a mother in the community of Indiana said, "I like that they show use the videos and how to feed the children. You learn better because you can see how to do it through the animations".
(Me gusta que nos hacen mirar los videos y la alimentación de los niños. Se aprende mejor porque se ven como se hace para aprender mediante dibujos) Acceptability by CHAs All CHAs expressed that they prefer to conduct the home visits with the CHEST App than their traditional methods (pen, paper, and ipcharts). When asked what they liked about using the App, the CHWs reported that they most liked: learning from the educational material, the animated videos, and the child health indicators displayed in red or blue. The animated videos were the most cited reason that they liked the CHEST App and how it helped them better conduct their home visits. A CHA in the community of Indiana said, "Yes, it (the App) has everything summarized and is faster. They like the videos. They laugh and understand more quickly. The child points. They like it a lot." (Si. Tiene todo resumido y mas rápido. Les gustan los videos. Se ríen. Aprenden más rápido y el niño apunta. Les gusta mucho) The CHAs expressed that the most di cult part about using the CHEST App was sending the data and registering the information in the tablet.

Adoption by CHAs
The CHAs were registering the health indicators of 140 children per month with the CHEST App, the same number of children they were assigned to visit. Adoption was con rmed to be 100%, de ned by being used by the CHAs during every scheduled home visit. After 18 months of use, the CHAs expressed that they continue to use the CHEST App during all their home visits. The caregivers veri ed adoption of the CHEST App by CHAs by reporting the number of home visits they have received and number of home visits with an electronic tablet. The results were similar (8.6 vs. 7.8), indicating that the CHAs usually use the CHEST App during their home visits. All the CHWs reported that they use all the App's functions when conducting a home visit (share health indicators with caregiver, register health indicator, scheduling, educational images, and animated videos).
Adoption of the CHEST App was hindered due to the cancelation of the CHA programs in several communities. The instability of the local CHA programs makes adoption and sustainability of the CHEST App di cult to measure because without the infrastructure of the CHA program the CHEST App cannot be utilized.

Adoption by CHA Program Coordinators
CHA Program Coordinators were tasked to upload and utilize the data collected with the CHEST App. Adoption of this practice was not achieved. The CHA Program Coordinators were not interested in uploading the data to the server. The task was completed during the pilot study, but only to satisfy the external research team at Elementos. Therefore, the research team is not able to monitor the data from the tablet unless they visit the community and upload the data from the tablets to the server. The Program Coordinator and local municipality continue to desire a paper-based list of results of the surveillance data and did not have a printer readily available to print the data from the tablets. The CHA Program Coordinator wrote the data displayed in the tablet on paper and submitted the paper report to the municipality. Thus, re ecting low adoption and poor delity of the surveillance function of the CHEST App.

Cost
The intervention was implemented at a scale of 20 CHAs in 4 communities. The cost of incorporating the CHEST App into the established CHA program included the following: providing an electronic tablet to each CHA ($120), training the CHAs on how to use the tablet ($184/CHA) and the cost of a Field Supervisor to visit the CHAs for continued training. The Field Supervisor can cover a greater number of CHAs (max 40), decreasing the cost per CHA. The cost of the Field Supervisor can also decrease after the rst 2-3 months as less visits and attention are needed. The incremental implementation costs are displayed in Table 1.

Fidelity
Fidelity of the CHEST App by the CHAs, as observed during home visits, remained high throughout the pilot. At 18 months, each intended activity associated the CHEST APP showed delity scores between 80-100%, as shown in Table 2.

Knowledge Scores
The average knowledge scores by the study population are shown in Table 3. The mean difference in knowledge scores in the intervention group and the control group at baseline is not statistically signi cant (p=0.136). A paired t-test on the mean difference of knowledge scores of caregivers in the intervention group at endline and those in the control group at endline displayed higher mean scores by those in the in the intervention group, a statistically signi cant increase of 8.29 (95% CI, 1.46 to 6.66) km, t(65)=3.12, p=0.0027. A paired t-test on the mean difference of knowledge scores in the intervention group at endline and scores at baseline displayed a statistically signi cant mean increase of 8.07 at endline (95% CI, 5.43 to 10.72) km, t(53)=6.12, p<0.000.

Anemia
The average hemoglobin levels and anemia rates are displayed in Table 4. Assessment of the two groups (intervention and control) displayed that there was no statistically signi cant difference in hemoglobin levels at baseline (mean difference=.14 gm/dl, p=0.46). A paired t-test on hemoglobin levels of the children in the intervention at endline and hemoglobin levels at baseline displayed a statistically signi cant mean increase of .47 (95% CI, 0.86 to 0.86) km, t(45)=2.46, p=0.0178. A paired t-test on hemoglobin levels of children in the intervention group at endline and hemoglobin levels of children in the control group at endling displayed no statistically signi cant mean difference (p=0.49).

Early Childhood Development
The average ECD z-scores for the children in the study are displayed in Table 5. The mean difference in motor and cognitive ECD scores in the intervention group and control group at baseline were not statistically signi cant (motor, p=0.75, cognitive, p=0.64). A paired t-test on ECD scores by children in the intervention group at endline and scores at baseline displayed a statistically signi cance increase at endline in cognitive scores, +1.029 (95% CI, 0.716 to 1.342) km, t(55)=6.585, p<0.000) and in motor scores, +0.714 (95% CI, 0.377 to 1.052) km, t(55)=4.24, p=0.001). A paired t-test on ECD scores by children in the intervention group at endline and scores by children in the control group at endline showed no statistically signi cant difference in cognitive (p=0.526) nor motor scores (p=0.305).

Diarrhea
The average number of days of diarrhea experienced by the children in the study is displayed in Table 6. The mean difference in number of days with diarrhea in the intervention group and the control group at baseline was not statistically signi cant (p=0.495). A paired t-test on the number of days of diarrhea by children in the intervention group at endline and the number of days by children in the control group at endline displayed a statistically signi cant decrease by those in the intervention group of 1.08 days, (95% CI, -1.94 to -0.21) km, t(66)=-2.49, p=0.0152. A paired t-test on the number of days of diarrhea by children in the intervention group at endline and number of days at baseline showed a decrease at endline of 0.52 days, however the mean difference was not statistically signi cant (p=0.1829).

Discussion
The conduct the implementation research, the research team rst assessed the key implementation determinants in the local context to identify potential barriers and facilitators to success. The determinants were based on the categorization from the Consolidated Framework for Implementation Research/CFIR; Damschroder et al., 2009. 49 Implementation strategies were then selected to reduce the barriers and leverage the facilitators in the local setting. To help determine which implementation strategies should be included, the research team used the CFIR ERIC Matching Tool. 50 The Matching Tool was used to help guide the choice of implementation strategies based on the potential barriers that were identi ed. The list of strategies produced by the Matching Tool is included in Appendix 4. The strategies that were used during the implementation process are listed in the IR Logic Model, ordered based on the results of the CFIR ERIC Matching Tool. Not all strategies produced by the Matching Tool were utilized, and thus not included in the IR Logic Model. For instance, funding strategies were not used due to a lack of receptibility by the local government to discuss changes to funding structures. For each implementation strategy, the research team implemented a set of discrete activities to operationalize the strategies. 51,52 Methods from Intervention Mapping (Implementation Mapping) were used to develop implementation strategies and evaluate the implementation outcomes. 53 The speci c implementation activities are described in Westgard, et al., 2020. 32 Finally, the implementation outcomes, service outcomes, and clinical outcomes were de ned, monitored, evaluated, and reported, as displayed in the Implementation Research Logic Model.
Although the intervention was disrupted by the COVID-19 pandemic, the CHA program that continued operations displayed promising results. The implementation outcomes suggest that the implementation strategies were affective at installing the intervention in the established CHA program, with high degrees of acceptability (100%), adoption, and delity (80-100%). Acceptability and adoption by the CHAs were greatly in uenced by the novelty of the CHEST App technology. The CHAs felt empowered from using the modern technology and more prepared to teach the caregivers due to the extensive educational material they had at hand. Fidelity rates of the CHEST App by the CHAs were consistently high, especially considering the dynamic nature of home visits. Registering the child health indicators with the CHEST app was the most likely to be omitted during the home visit. At times, home visit occurred when the caregiver had not been to a child health monitoring visit since the last home visit. The CHAs believed that they should not register the child health indicators when the child had not been to the health center since the last home visit. This led to missed opportunity to record cases of infections that may have occurred since the last home visit. The animated videos were also sometimes omitted, primarily because the videos are the last step of the home visit. If the home visit needed to be cut short, the videos were often the step that was omitted. The inconsistency of time and information available for each home visit made accomplishing perfect delity di cult.
Full adoption and delity by CHA Program Coordinators was not achieved because there was little desire to digitize the surveillance data. The coordinators and local municipality do not have the desire to rely on technology to track the health of children in their communities. Because the programs are not integrated into a larger, regional program, the number of children managed by the program remains relatively small. Also, the reporting requirements are minimal because the CHA program is only supported by their local municipality. The CHA program and municipality value the educational component of the CHEST App but do not value the digital surveillance component. The digital surveillance component will be more important for CHA programs that are integrated into a regional or national level program, which demands greater organization of data.
Outcomes related to sustainability and penetration need to be assessed after the communities have had more time with the CHEST App and following the end of support from the external research team from Elementos. Sustainability of the CHEST App intervention is greatly in uenced by the sustainability of the CHA programs. The local municipalities have been inconsistent in their support of their CHA programs. The three municipalities in the CHEST App pilot study each changed the CHA program during the 18-month trial. Two of the municipalities temporarily canceled the CHA program and one of the municipalities cut back the number of CHAs from 10 to 5, due to a lack of available funds. The National CHA program (Cuna Mas) 54 that also conducts home visits throughout the poorest regions of Peru is more stable, and thus a better alternative to incorporate the CHEST App. The implementation outcomes, service outcomes, and clinical outcomes from this pilot study are expected to be generalizable to the context of the National CHA program. Improvement in knowledge scores by the CHAs and caregivers re ects the immediate impact of the CHEST App. Interestingly, the CHAs and caregivers had received extensive training and numerous home visits before the baseline knowledge evaluation. In theory, they should have been knowledgeable about the material from their previous training and home visits by the CHAs. The CHEST App provided a signi cant incremental increase in knowledge transmitted to the CHAs and caregivers. The educational material, automated selection of health messages, animated videos, and improved selfe cacy of the CHAs are theorized to be the driving factors of increasing the knowledge of healthy childrearing practices by the caregivers.
The three clinical measures (anemia, ECD, and diarrhea) each showed signi cant improvement when compared to baseline measures or control group measures. Mean ECD scores (cognitive and motor) and mean hemoglobin levels were signi cantly improved 18 months after the implementation of the CHEST App intervention, when compared to baseline. The improvement occurred despite a near 5-month community shutdown due to COVID-19.
However, the difference in ECD scores and hemoglobin levels were not signi cantly different than those in the control group. This may be due to the reduced sample size in the intervention group.
The effectiveness-implementation hybrid type 2 study provided the following results: 1. The CHEST App intervention can be effectively incorporated into a CHA program with high degrees of acceptability, adoption, and delity. 2. Adoption and delity of the surveillance function of the CHEST App by program coordinators was not achieved. 3. The CHEST App intervention is associated with improvements in knowledge scores, hemoglobin levels, ECD scores, and decreased diarrhea, however, further study is needed to determine the effect on each.

Limitations
The reduced sample size of the study due to program shutdowns caused by COVID-19 diminished the ability of the study to detect changes to the clinical outcomes with a more rigorous statistical model. Initially, the recruitment of providers and study participants at baseline provided a su cient sample size to conduct a difference-in-difference analysis to evaluate the impact of the intervention. However, multiple CHA programs cancelled their services, thus reducing the number of providers and participants that received the intervention. The t-test analysis used in this study highlights the association between the intervention and changes in clinical outcomes that occurred, however, it does not indicate that the intervention caused the changes. An additional limitation was caused by the method to select who received a hemoglobin test. Hemoglobin levels were measured by the local health center and were not random nor a census of all children in the community. There may be selection bias of those that received the hemoglobin test and reported hemoglobin levels to the study. Selection bias may have also occurred due to the method that CHAs selected which children to visit. The CHAs were not able to visit all children in their community because the number of CHAs in the program was reduced due to funding constraints.
The study team is not aware of the method in which the program chose which children would receive visits. The anemia rates reported in the study during suggests that there was a selection bias present in the sampling because the anemia rates are signi cantly lower than the regional average. To accurately detect the effect of the CHEST App on anemia rates a more rigorous method of hemoglobin measurement will be needed.

Conclusions
As the Implementation Research Logic Model displays (Appendix 2), the implementation strategies were successful at installing the CHEST App into the CHA program, producing favorable implementation outcomes. The educational component of the CHEST App was implemented with high levels of acceptability, adoption, and delity. The surveillance component of the CHEST App was not adopted with consistency. The CHEST App led to improved knowledge acquisition by caregivers who received the intervention and for the CHAs that utilized it. The improved knowledge scores are theorized to contribute to the reduction of anemia rates and improved ECD scores by those who received the CHEST App. The CHEST App intervention is a promising tool to improve the performance of CHAs during their home visits, to accomplish their objective of teaching caregivers healthy childrearing practices and improving childhood health and development in their communities.

Consent for Publication
Not Applicable

Availability of Data and Material
The datasets generated and analyzed during the current study are available in the gshare repository, 10.6084/m9. gshare.13681423.

Competing Interests
The author declares that they have no competing interests  Asked caregiver what they understood from the health messages 100% Showed animated video to caregiver 80%