Development of a functional and psychosocial evaluation toolkit using mixed methodology in a community-based physical activity program for childhood cancer survivors

The evidence demonstrating the benefits of exercise and PA in patients and survivors of childhood cancer has been translated into a handful of community-based programs, such as the Pediatric cancer patients and survivors Engaging in Exercise for Recovery Program (PEER). To support the translation of research to practice, the next step in knowledge translation is to evaluate program effectiveness. An evaluation must consider the goals of the PEER program, feedback from key stakeholders, and logistics of this program. Thus, the purpose of this study was to develop an evaluation toolkit with an algorithm for the implementation of the PEER program. Semi-structured interviews were conducted with three different groups (stakeholders in pediatric oncology, PEER parents, and PEER participants). The interviews were transcribed and coded by two independent reviewers. Key themes extracted from the interviews were split into physical and psychosocial themes. The most reported psychosocial themes were quality of life (QOL), fatigue/energy levels, fun, and cs; and physical themes included motor skills, physical literacy, and physical activity levels. Tools were compiled into the evaluation based on key themes identified as well as logistics of PEER. An algorithm was developed to tailor the evaluation to participants based on age and mobility. To date, this is the first evaluation toolkit and algorithm developed for a specific community-based PA program, the PEER program. The next step in knowledge translation will be to implement the evaluation to assess feasibility and share the evaluation for adoption within other developing programs.


Pediatric oncology and physical activity
Advances in treatments and supportive care over the past 30 years have led to increased survival rates in children diagnosed with cancer [1]. However, therapy for childhood cancer is not without adverse effects. The potential acute and chronic complications of cancer and its treatment have been increasingly described, impacting physical and psychosocial wellbeing [2]. Increasing evidence supports the benefits of physical activity (PA) to aid in improving physical and psychosocial outcomes in pediatric oncology [3]. This supportive evidence has led to the development of a handful of community-based programs that move this research evidence to practice. One such program is the Pediatric cancer patients and survivors Engaging in Exercise for Recovery (PEER) program [4] 1 3 .

The PEER program
PEER is a community-based PA program developed by the Health and Wellness Lab at the University of children and youth affected by different types of cancer such as brain tumors, leukemia, solid tumors, and hematological diseases. This makes our population very heterogenic because side effects from treatment are very different and thus challenging to assess all of them with the same test. The main goal of the PEER program is to provide a safe and fun exercise program for children and adolescents [4]. The program is committed to encouraging the adoption of an active lifestyle, improving physical literacy, fitness, and quality of life (QOL) and reducing the negative side effects of cancer and its treatments [4] and will favor the reinsertion of children in the community.

Age 8-12
No mobility restricƟons To promote the continued growth of the PEER program and increase its viability as a key supportive care service during the pediatric oncology journey, evaluation of the potential physical and psychosocial benefits of the program is essential. An evaluation aligned with the goals of the PEER program may provide valuable feedback about the program's effectiveness and continue to be offered sustainably in the community. The PEER program evaluation of participant outcomes must include key stakeholders, including the health care professionals (HCP), families, program developers and administration operating the program [5].
Involving key stakeholders in the development of the evaluation toolkit is critical to identify key themes such that they can be aligned with the goals of the program. Given the community-based nature of the program, it is important to balance the collection of psychometrically sound data with a proposed evaluation that is logistically feasible for the PEER program.
The purpose of this study was to (a) conduct interviews to evaluate what stakeholders and users consider valuable and required components of a PEER evaluation toolbox [6] and (b) develop a PEER toolbox and algorithm for implementation based on the findings of the interviews and practicalities/goals of program delivery.

Methods
A scoping review compiled evaluation tools used to date in PA/exercise studies and programs in childhood cancer [6]. This review was used to guide interviews with key stakeholders to further understand the potential participant outcomes they want to see reported from the PEER program or value from PEER participation [9]. The results of both the review and the interviews guided the current work, the compilation of an evaluation toolkit and algorithm for implementation within the PEER program.

Participants
Interviewees were comprised of three categories. First, key stakeholders and health care providers in pediatric oncology (HCPs) at the Alberta Children's Hospital and within the PEER program, second parents of children using the PEER program and third, children's users of the PEER program. Potential HCP interviewees were approached based on their involvement in the PEER program, in addition to using snowball recruitment to approach HCPs in pediatric oncology who were not aware of the PEER program. Individuals interested in participating were contacted via email to set up an interview, and those who were unable to meet in person were invited to set up a phone interview. Second, parents in the PEER program (all age groups) were emailed by the PEER program developer/coordinator (CCV), inviting them to participate in a 10-15-min interview. Third, school-aged children and adolescents in the PEER program were invited to participate in an interview. Parents were asked to attend the interview with the children and adolescents. Participants were fully informed of the study requirements and provided written informed consent, and those under 18 years of age provided assent and parental consent was provided. The Health Research Ethics Board of Alberta (HREBA) Cancer Committee approved this study (HREBA.CC-17-0094).

Interview
Interviews lasted 10-30 min and were either at a location convenient to the participant (n = 17) or over the telephone (n = 1). Interviews were conducted with each participant individually, except for children in the PEER program, in which case his/her parent was present. The interviewer (JAS) gave a description of the purpose of the study, provided a list to HCPs, parents and children of potential evaluation tools from the scoping review [6] and provided time to answer any questions before the interview. The interviewer had a set of topics to guide the semi-structured interview (see Appendix 1). The interviews were customized to different age groups or key stakeholders in the program. For example, children were asked, "Why do you enjoy coming to the PEER program?" compared to parents who were asked, "What do you want your child to get out of the PEER program (i.e., specific areas you find they need improvement both physical and psychosocial aspects)?" The interviews were designed to understand why children and parents like the PEER program and what themes they value in the program. Interviews with HCPs were also designed to determine what aspects of the PEER program they valued. Thus, HCPs were asked more specific questions, such as "What do you believe are important physical and psychosocial themes to capture in the PEER program (i.e., QOL, physical literacy, motor proficiency, well-being)?" and the results of a scoping review performed prior to this study determined evaluation tools which were suggested to help guide/prompt the discussion.

Transcription
The interviews were recorded using a digital recorder and transcribed verbatim by the interviewer (JAS).

Analysis tool
Key themes and any logistical problems that arose from interviews were discussed between the interviewer (JAS) and second reviewer (CCV). Any disagreements among themes were discussed and resolved by a third reviewer (NCR). Consistent with the thematic analysis approach, the data coding and extraction was performed by two reviewers (JAS) and (CCV) [7]. The first step included familiarization with data where the independent reviewer (JAS) read and re-read the data and generated a list of key ideas. These key ideas were discussed with the second reviewer (CCV), which lead to phase 2, identification of initial outcomes [7]. These outcomes were identified by asking participants what outcomes were considered important for evaluation to the PEER program. Following this, the outcomes were grouped into broader themes, where they were continually refined and developed until both reviewers (JAS) and (CCV) agreed on the themes. A thematic map was created where the final set of themes were defined, which lead to the algorithm development. The subset of evaluations recommended for each child based on age and physical and cognitive abilities was determined from the scoping review [6] and interview thematic analysis, with consideration of the operational logistics of the PEER program.

Evaluation toolkit and algorithm development
The evaluation tools for the current work were selected based on three criteria: (a) results from the scoping review that allow us to gather information about assessments previously used in pediatric oncology and exercise research [9]; (b) the themes identified from the interviews, that allow us to understand what participants and stakeholders want to know from an evaluation; and (c) feedback from the PEER program developer/coordinator that allow us to understand if assessments selected fit within the logistics of delivery for the PEER program (cost, required personnel, duration, and objectives and fit heterogeneity of population). The goal is to implement the evaluation at 2-3 weeks after starting the PEER program (whenever the participant joins PEER, given rolling recruitment), and then again after 18 weeks and 36 weeks of participation. This allows for one evaluation at a minimum per semester (fall, winter, spring). An algorithm was developed to ensure participants received a systematic evaluation that was tailored to their needs. For important outcomes identified in the interviews that were not identified in the scoping review, evaluation tools were selected for the toolkit based on evaluation tools that closely aligned with the interviews that may not have been identified in pediatric cancer and exercise literature. These evaluation tools were selected based on feedback from a cancer and exercise expert.

Participants
A total of 18 participants were approached and interviewed (Table 1): 4 HCP, 3 PEER Admin, 7 parents, and 4 children. There were no individuals approached who declined to participate.

Interviews
Interviews resulted in physical/physiological and psychosocial themes, as outlined below.

Physical/physiological themes
The most reported physical themes for outcomes to assess in the PEER program as identified by parents, participants/ users and HCPs were motor skills (56%), physical literacy (50%), fatigue/energy levels (50%), physical activity levels (61%), and strength (28%; see Table 2). One HCP did not agree with having a formal evaluation of participant outcomes in the PEER program and thus did not identify any topics.

Psychosocial themes
The most common psychosocial themes for outcomes identified in the interviews were QOL (44%), fun (50%), and confidence levels (39%; see Table 2). Less commonly reported were mood, communication, social support, and coping mechanisms.

Proposed PEER program evaluation toolkit
An evaluation toolkit and algorithm to assess PEER program participants (see Appendix 2) was compiled. The evaluation tools were selected based on (1) the results of the scoping review, (2) the interviews, and (3) the logical restraints of the PEER program. The results of the scoping review indicated that the most reported patient outcomes were fatigue, functional mobility, well-being, QOL, and motor performance [6]. The thematic results from the interviews confirmed that the most valued/important themes to capture were (a) physical literacy, (b) motor performance, (c) developmental milestones, (d) QOL, (e) confidence, (f) PA levels, (g) exploring new activities, (h) fatigue/energy, and (i) fun (refer to Table 3).
For the motor performance assessment, defined as the skills required for movement patterns [8], several evaluation tools were found in the review [9][10][11][12]; however, none of the identified tools assessed physical literacy in pediatric oncology [6]. Therefore, the Bruininks-Oseretsky Test of Motor Proficiency-Second Edition (BOT2) was selected to assess children aged 4-18 years that display mobility limitations to provide a validated and extensive evaluation of motor performance [10]. The BOT2 was selected given the option to complete the short form assessment in 20 min and allowed us to assess children up to 18 years in the program [6,10]. The BOT2 has been widely used and validated and has age-based standard scores that allowed us to understand gross and fine motor developmental delays to adapt the PEER program to participant needs [9]. The BOT2 was specifically designed to assess motor proficiency for all children; however, it is often used to identify motor impairments; as such, we decided not to use it with all children in PEER [10]. Physical literacy is defined by the International Physical Literacy Association as "the motivation, confidence, physical competence, knowledge, and understanding to value and take responsibility for engagement in physical activities for life [13]." It is important to note that we did not find in our review of physical activity and pediatric oncology any tool that was used to assess physical literacy [14]. However, we found in the literature a physical literacy assessment, the  Physical Literacy Assessment for Youth Tools (PLAY) tools, developed by the Sport for Life Society [13]. PLAY-Tools have not yet been reported in the cancer and exercise literature [14]. However, recent research suggests that the PLAYTools are a valid and reliable measure of physical literacy in school-aged children [15]. This assessment tool is valid for kids between 7 and 18 years old. Thus, we select the PLAY tool to assess the physical literacy of children between 7 and 17 years old in the PEER program. The PLAYTools were developed for kids aged 7 and older because this is the stage of development where children develop these fundamental movement skills that are assessed in the PLAYTools. For those aged 2-6 years, a developmental milestone assessment was used to see if they are reaching sufficient gross motor milestones for their age [16]. The developmental milestones were selected to assess age-appropriate skill development which is required for the development of fundamental movement skills which will thus lead to improved physical literacy [17]. As such, the developmental milestone checklist allowed us to assess motor development in a relatively easy and quick manner [16]. Evaluation tools that assessed fatigue in the scoping review included the PedsQL Multidimensional Fatigue Scale [18] and The Fatigue Scale [19]. The PedsQL Multidimensional Fatigue Scale was selected for use in the PEER evaluation toolkit, given the age range of 2-18 and the scale has been widely used in pediatric oncology [18]. Strength was also mentioned as an outcome of interest; however, it was not included in the evaluation given this assessment does not reflect the main outcomes addressed in the PEER program. While strength was mentioned as an important outcome, it is not a priority for the PEER program and would not be less feasible to complete in the PEER session. For the assessment of QOL, a wide variety of QOL tests were reported in the scoping review including the CHQ: Child Health Questionnaire [20]; PedsQL Generic Core and Cancer Module [18]; and KINDL [21]. The PedsQL Generic Core was selected given this assessment tool is available for children 2-18 (both participant or parent-proxy measures) and has been widely used and validated in pediatric oncology [18]. This test can be used without cost for communitybased exercise programs. The results of the interviews also indicated an interest in assessing confidence to engage in PA

Free
No required qualifications and exploring new activities; however, the scoping review [6] did not provide any evaluation tools that assessed these criteria. Thus, the PLAYSelf and PLAYParent [14,15] were used for children aged 7 and up, as this physical literacy tool also captures confidence to engage in physical activity. The PLAYSelf and PLAYParent provide an understanding of factors that impact a child's ability to engage in physical activity, thus providing a better understanding of a child's physical literacy [14]. Given the relative ease with which this tool may be implemented, it was included in the evaluation toolkit. Lastly, a limitation of these assessments is the ability to read and write; for children unable to do this, a session observation assessment was determined to be a more valid and thus appropriate assessment tool for social interaction in the PEER program (see Appendix 2). The session observation assessment allows the evaluator to set goals for the child to complete in the session, specific to his/her needs. The session observation assessment was also used for children unable to physically complete the physical evaluation portion (see Appendix 3).

Algorithm
The evaluation tools were organized into an algorithm for implementation, considering the needs of the participants in the PEER program (i.e., age, ability to read/write, and mobility) and within the logistics of the PEER program (i.e., duration needed to perform each assessment). The evaluation was split into an online portion for completion of the questionnaires, and an in-session assessment for completion of the physical assessments. This makes the assessment of the outcomes feasible within the 1-h time constraints of the PEER program session (see Appendix 2). The evaluation tools were split into 4 age groups, based on the use of the assessments: (a) 2-4 years, (b) 5-7 years, (c) 8-12 years, and (d) 13-18 years.

Discussion
Early research has established the benefits of PA and exercise in pediatric oncology [3]. However, this has led to only limited translation of research to community-based PA programs [1]. Earlier evaluation work on the PEER program has determined its feasibility and safety [4]. To assess the effectiveness of the PEER program, an evaluation of the program's potential benefits is necessary. Given the communitybased nature of PEER, consideration of both the program logistics and participant characteristics (age, cancer type, cancer treatment, cognitive ability, and mobility) must be considered. The results of the scoping review and the interviews guided the development of an evaluation toolkit and algorithm for implementation within the PEER program [6]. The final evaluation toolkit was determined based on the scoping review, which reported on measures that have been used and validated in the literature, important outcomes identified by the interviewees, and evaluation tools that were practical for use in the PEER program. The final evaluation toolkit and algorithm (Appendix 2) have important research and clinical implications. Continued application of the toolkit in the PEER program will allow re-evaluation of the usability of the evaluation in PEER. First, from a research perspective, development and implementation of an evaluation toolkit can aid in understanding the effectiveness of community-based PA/exercise programming in childhood cancer. The toolkit provides a "starting point" to assess the PEER participants and corroborate the effectiveness of community-based programs, such as the PEER program. This allows for the necessary adaptations to the specific goals and logistics within a specific program to be considered. While most evaluation tools selected have been widely used and accepted in childhood cancer, some evaluation tools have yet to be reported in the literature (healthy or cancer-specific populations). For example, the PLAYTools have been developed by a panel of health and exercise experts and are a reliable and valid tool [15], but have yet to be reported in pediatric oncology. However, a recent article highlights the importance of advancing exercise in pediatric oncology and supporting children with cancer moving more [22]. The goal of the PEER program is to encourage and instill the enjoyment of physical activity, and thus, the PLAYTools aligned closely with the results of the interviews and the PEER goals.
From a clinical perspective, the evaluation toolkit can provide an objective assessment of key patient themes that can be provided to both the health care team and the families. Given the nature of community-based programs such as PEER, this clinical feedback is both valuable and extremely important in terms of patient themes and maintaining the ongoing value of the program. Ongoing evaluation within the PEER program specifically ensures that each participant receives an assessment tailored to their needs and that will assess potential benefits from participation. Participants, parents, and their HCPs will receive feedback from the evaluation, allowing tracking of progress/improvements over time. Regularly seeing the impact of the PEER program participation on their patients may facilitate referral from HCPs into the program. In addition, tracking how this information was used by patients/families and providers is important to understand where evaluation feedback is being used.

Limitations
There are a few important limitations to note regarding the evaluation tools selected. First, the evaluation toolkit and algorithm were specially designed for the PEER program and may not be applicable to all community-based PA/ exercise programs for childhood cancer. For example, the evaluation tools were selected in alignment with available resources at the PEER program and the goals of the program. Implementation of the toolkit to any specific program must consider that program's unique needs. Second, given the limited number of interviews conducted, the results may not represent the views of all HCPs and PEER program participants/families. Third, the algorithm does not account for individual limitations such as osteoporosis or peripheral neuropathy; thus, a trained professional is required to use discretion to ensure any individual alterations are made. The evaluation tool should be applied in a manner that provides valuable feedback to the participant/family and provides substantive evidence to the HCP with regard to the potential role of PA/exercise in pediatric oncology. Lastly, the evaluation tools were selected based on several criteria (psychometric properties, cost, duration, and personnel required), and thus, there is a challenge in selecting tools that meet all criteria. For example, the PLAYTools best suited the needs of the PEER program but have yet to be reported in childhood cancer. While we deemed the PLAYTools suitable for use, future research on physical literacy in childhood cancer may provide a more suitable measure.

Implications of research findings
Despite these limitations, there are also important strengths to be noted. To date, no literature has been published specifically evaluating community-based PA/exercise programs in pediatric oncology. The generated toolkit and algorithm may provide a framework for programs to utilize, thus enhancing our knowledge about the effectiveness of such programs. Lastly, consistent application of the toolkit may provide further feedback regarding the reliability and validity of the evaluation tools. The ongoing collection of data and re-evaluation is critical to the validity of this process and the toolkit for future development and use of community-based programs for young people impacted by cancer.
Author contribution Ms. Shank conducted the research project for her MSc thesis and wrote the manuscript. Dr. Chamorro-Viña assisted with data collection and provided feedback and guidance throughout. Dr. Guilcher and Dr. Schulte were mentors and provided feedback on Ms. Shank's thesis. Dr. Culos-Reed was Ms. Shank's MSc supervisor and provided feedback and mentorship throughout her graduate work.
Funding This research study was funded by the Canadian Institute of Health Research (CIHR) and the Alberta Children's Hospital Research Institute (ACHRI).
Data availability All identifying information was removed from data before manuscript preparation. Participants who were interested in study results will be provided with the final manuscript.
Code availability Not applicable.

Declarations
Ethics approval This research study was approved through the University of Calgary Research Ethics Board (HREBA.CC-17-0094).
Consent to participate All participants signed an informed consent or assent form. All participants were informed; they may withdraw from participation at any point.