Overview of the study
The aim of this mixed methods research is to implement universal, systematic family psychosocial risk screening with the PAT in English and Spanish to assure that all families of children newly diagnosed with cancer at the participating cancer centers are screened. The setting/context of the research is pediatric cancer programs in the United States. The approach, reflected in the three aims, is guided by the ISF.36 We selected specific implementation strategies from the Expert Recommendations for Implementing Change (ERIC) project 37 targeting implementation outcomes at three levels (patient/family, provider, institution).
Aim 1 corresponds to the first component of the ISF, Prevention Synthesis and Translation System. We will prepare pediatric cancer programs to implement the PAT by conducting qualitative semi-structured interviews with a diverse set of stakeholders (n = 19). Interviews will be focused on details of Strategy I (Training via webinar) and Strategy II (Training + TIER—Consultation Calls and Identification of a Champion), followed by broad questions about facilitators and barriers, and more specific questions about implementation strategies and resources needed for universal screening and care delivery to address health inequities. Major modifications are unlikely but rather we anticipate fine-tuning and adding components to improve penetration and health equity targets, acceptability and feasibility, adoption and sustainability.
The activities of Aim 2 correspond to the second component of the ISF, Prevention Support System. Support for those implementing innovation is necessary and must occur at multiple levels within the system – patients/families, providers, and institution. To implement the PAT in English and Spanish, 18 pediatric cancer programs, of varying sizes and with geographic distribution assuring representation of ethnic and racial minority families and families at socioeconomic risk, have agreed to participate. We will conduct a cluster randomized comparative effectiveness trial of the two implementation strategies across three cohorts stratified by size of site based on new patients per year.
Aim 3 activities correspond to the third component of the ISF, Prevention Delivery System, in the development and dissemination of the PAT Implementation Toolkit.
Stakeholder interviews to refine implementation strategies (Aim 1)
Qualitative interviews will be conducted with a national Implementation Team of diverse stakeholders, selected using purposive criterion-based sampling40 to represent different levels of the social ecology. The data from the interviews will inform components of the two implementation strategies and incorporate questions related to broader implementation (e.g. facilitators and barriers). The stakeholder interviews focus on ERIC implementation strategies - Use Advisory Boards and Workgroups, Prepare Patients/Consumers to be Active Participants, and Build a Coalition.
Participants. Participants (n = 19) are parent advocates, multidisciplinary healthcare providers, members/leaders of key pediatric oncology professional organizations, and leaders in the pediatric healthcare industry. In the event that additional perspectives are needed from the stakeholders to achieve saturation, we will identify additional individuals from the relevant type(s) of stakeholders, with the sample potentially expanding to 25.
Procedure. Each member of the Implementation Team will be interviewed by video conference using a theoretically driven and empirically-based semi-structured interview guide.41 The purpose of the audiotaped 60-minute interview will be to refine Strategies I and II, identify additional barriers or facilitators of implementation across the patient/family, provider, and institution levels, and ensure PAT implementation strategies address these barriers to and disparities in care.
Cluster randomized comparative effectiveness trial (Aim 2)
PAT implementation will be tested using a head-to-head randomized implementation (comparative effectiveness of two implementation strategies) trial initiated in three cohorts over three years. All 18 pediatric cancer programs invited to participate in the trial agreed to a 2-step randomization process stratified by size of site; sites will be randomized to one of three cohorts and then to strategy, comparing Strategy I Training with Strategy II TIER.
Selection of Sites. The following criteria were used in site selection:
- Provide staff and tablet computer for screening. Each site agreed to support PAT implementation by providing the staff person(s) screening and tablets/computer access. Each participating site will be provided with support for a portion of the site PI’s time and effort and a part-time research coordinator based on center size for purposes of the research study only (that is, research staff will not conduct screening).
- Center size. Because program size is related to the size of the psychosocial team and related psychosocial resources,6 we used PIPS-CSS data to stratify by size to obtain three equal and clinically relevant groups based on number of new pediatric cancer patients annually - Small (30-60), Medium (61-149) and Large (150+). We selected 6 sites for each of the three size categories;
- Psychosocial Staff. To assure that there are staff to conduct screening and act on the results of screening, selected sites are at or above the median for the size of their psychosocial team (number [full time equivalents] social workers + psychologists + psychiatrists + child life specialists)6 based on PIPS-CSS data;
- Diversity of population. Selection focused on centers in states with a high percentage of ethnic or racial minority families and/or families with socioeconomic disadvantage. CA, TX, FL, NY and NJ are states with a large Hispanic population.42 Among the 50 states, AL, NC, and VA have the most rapidly increasing Hispanic populations, and AL, LA, NC and VA have the largest percentage of African-Americans.43 The majority 13/18 (72%) of our sites are in these states. 12/18 sites are in states with >20% of children living in poverty.44 Other sites were selected for geographic balance and because they reflect a balance in race, ethnicity and SES.
Participants. Since screening will be integrated into the clinical services of the participating centers, it will be routine clinical care. Coded EHR data with minimal protected health information (PHI) will be collected for patients. Reflecting the importance of staff as critical to implementation,45 consented participants include the site PI and clinical staff who will screen at each site and, in Strategy II, the Champion.
Procedure. All sites, regardless of randomization to Strategy I or II, will participate in a 3-hour webinar at the beginning of their cohort. The training webinar will include all information necessary to understand, access and deliver the web-based PAT. Sites will be provided with a User Agreement, user manual, and access to the password protected site. The webinar will be based on our in-person training program and curriculum,35 modified to integrate feedback from the Implementation Team interviews in Aim 1. It will be professionally prepared to be engaging and effective as an educational approach with resource materials provided (Develop Educational Materials, Distribute Educational Materials).
Each site PI, all staff who will be directly involved in screening, and the research coordinator will participate. For sites randomized to Strategy II, the Champion will also attend. We will work with sites to identify the team that will participate in screening and the Champion (in TIER) (“individuals who dedicate themselves to supporting, marketing, and driving through an implementation, overcoming indifference or resistance that the intervention may encounter in an organization”),37 recognizing that these determinations will vary across programs.
After the PAT is completed online it is scored immediately and a summary of the score and clinical concerns identified is generated. Only coded data will be transmitted to the study data core. A master list for this data will be maintained at each respective study site and will not be shared with the core research team. We will provide support to sites in the technical aspects of the implementation related to using the web-based forms (Centralize Technical Assistance). If we identify any pattern of problems with technical aspects of implementation, we will communicate with sites promptly.
PAT Implementation Plan. During training, each site PI and staff involved in screening will work together to develop a specific implementation plan, describing who will implement the PAT, who will be screened, where results will be stored, how results will be communicated to families and to staff, and how results will be used (Develop a Formal Implementation Blueprint). For TIER, the Plan includes questions about the roles and responsibilities of the Champion and the focus of the Consultation Calls.
Strategy I. For sites randomized to Strategy I (Training), the webinar is the implementation condition. Sites will also receive technology support, as needed, throughout the one-year implementation period. These strategies correspond to implementation strategies: creating a structure for implementation including creating implementation teams and developing an implementation plan.37
Strategy II. Strategy II, Training + Implementation Expanded Resources (TIER), includes the same webinar and technical support as in Strategy I with the addition of two evidence-based resources that may improve implementation (as finalized from analysis of Aim 1 interviews). The Site PI and center staff conducting screening will participate in a monthly one-hour PAT Implementation Consultation Video Call (Provide Ongoing Consultation, Create a Learning Collaborative) with other TIER sites in that cohort. The group format of this strategy is intended to foster group problem-solving about common issues in implementation and provide peer support for those implementing the PAT. The data regarding implementation progress and challenges from the calls will be addressed in subsequent calls and will be considered in the development of the Toolkit. Second, sites will identify a Champion for PAT implementation (Identify and Prepare Champions). The Champion will advocate for PAT implementation and support staff in screening activities by serving as a resource to problem-solve and communicate with the broader clinical staff about screening and psychosocial risk. The Champion will most likely be a clinical leader who demonstrates enthusiasm and commitment to the goals of universal psychosocial risk screening.
Measurement/Outcomes (all sites). The measures assess outcomes across patient/family, provider, and institution levels (Table 1) are clearly operationalized, reproducible and useful for future studies.46 Each site PI and Coordinator will attend a one-day training on data collection procedures at the beginning of the year in which they participate in the trial. There will be two separate training sessions (Strategy I v II) to assure that staff will not be exposed to the other study arm.
At the Patient/Family level (Hyp 2.1) the site Coordinator will extract EHR data and send the coded data via REDCap to the Nemours Data Core. We expect that most (if not all) sites will have an EHR although data can be extracted from paper records if necessary. The following data for the English and Spanish versions of the PAT will be reported monthly: new patients meeting eligibility requirements as outlined on the PAT Implementation Plan (n); patients with documentation of PAT screening (n); patients with evidence of family feedback letter (n); demographic data on all eligible and all screened (race, ethnicity, zip code, insurance). As part of the PAT Implementation Plan, the Site PI and Coordinator will determine institution-specific ways to identify new patients and their demographic characteristics. Generally, across sites, data on newly diagnosed patients are easily accessible and tracked through tumor registries.
At the Provider level (Hyp 2.2) the Site PI and staff identified as screeners will complete the self-report measures prior to beginning the trial (T1), and at 6 (T2) and 12 (T3) months via REDCap. The total time for administration is less than 15 minutes. Feasibility and acceptability will be measured using The Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM) modified to refer to screening.47 We will use the Measure of Physician Engagement in Addressing Racial and Ethnic Health Care Disparities,48 the Satisfaction of Employees in Healthcare (SEHC) survey,49 and the Maslach Burnout Inventory (MBI).50
There are four outcomes at the Institution level (Hyp 2.3). We will assess adoption, or the intention of sites to use the PAT, by examining the site participation rate (ratio of sites that initiate the study/18 that agreed + acceptance rate of any substitution sites [see F6]). Second, we will evaluate the extent to which screening is perceived as an asset to the institution using the PAT Implementation Questionnaire (PIQ), 35 which generates a score for benefits and challenges. The PIQ will be completed by the site PI, the Screeners and the Champion (TIER), at T1, T2, and T3. Third, we will assess whether services are matched to the needs of families at each center, based on PAT PPPHM levels. EHR data will be extracted using the Psychosocial Services and Medical Treatment Checklist (PSMTC), adapted from our prior research.18 The data, aggregated at the site level, will be collected for each family at 30, 60 and 90 days after screening. The PSMTC records psychosocial and related care organized by discipline and the number of occurrences of each. It provides a sum of hospital based services and external referrals and the number and length of inpatient and outpatient visits, missed appointments and evidence of non-adherence or difficulties in delivering care, allowing determination of healthcare utilization or medical services delivered. Fourth, we will assess cost to understand the relationship among PAT risk level, psychosocial services provided, and aspects of medical care that may reflect unnecessary or inefficient services. The primary cost of psychosocial care is the cost of people – psychosocial providers. Therefore, we estimate direct costs associated with personnel rather than non-personnel, billing, and/or indirect costs. We will use the data from the PSMTC to estimate the amount of psychosocial staff time and national median hourly salary data for the major psychosocial provider professions.51,52,53 Because the care provided may be related to the intensity of medical treatment, the Intensity of Treatment Rating Scale (ITR-3)54 will be completed at the 90-day post-screening data collection time point: it uses EHR data (diagnosis, stage, treatment modality) to identify one of four levels of intensity of pediatric cancer treatment.
Fidelity. Whether the PAT was administered as intended will be assessed using the EHR data sent monthly from the sites. The coded data captured at Nemours will also allow us to confirm the completion of each PAT in its entirety. Missing data has not been a concern with the web-based PAT. The coordinator at Nemours will lead a separate group call with the coordinators at each of the sites on a monthly basis to problem-solve any concerns about extracting the EHR data, with minutes to document the discussion points. For sites in the TIER condition attendance on video calls and detailed minutes of the monthly discussions will be prepared. The Champion will complete a questionnaire at the beginning of the study year, and at 6 and 12 months to document the activities performed by the Champion.
Dissemination Toolkit (Aim 3).
To broadly disseminate the PAT, a theoretically and data driven web-based PAT Implementation Toolkit will be developed and will contain all the information, resources and tools necessary to implement the PAT successfully across centers of different sizes. Aim 3 activities correspond to Prevention Delivery System of the Interactive Systems Framework (ISF) and the final phase of implementation (Develop educational materials; Distribute educational materials; Purposely re-examine the implementation).37
Participants. There will be 20 participants, purposively sampled from Aim 1 stakeholders (n = 10) and from the sites that implemented the PAT in Aim 2 (n = 10 site PIs and Champions). Cognitive interviews will be conducted with this group of individuals to ascertain their interpretation and understanding of the preliminary PAT Implementation Toolkit, intending to identify any omissions in the Toolkit or modifications needed to components of the Toolkit. This sample size is consistent with PROMIS methods55 and the past experience of the research team.56 All of the Aim 1 stakeholders indicated agreement to participate in the Aim 3 interviews. To increase the likelihood that they will remain engaged with the study, we will send them a summary of study progress twice a year. If any resign or are unable to continue, they will be replaced, considering the overall composition of the sample and diversity of stakeholders.
Procedure. Subsequent to the final analysis of data from the comparative effectiveness trial, we will refine the preliminary framework for the PAT Implementation Toolkit website. Tentatively, the components of the Toolkit will include: modified PAT training webinar; technology support; PAT Implementation Plan; complete information to implement the PAT across centers of different sizes; materials specific to using the Spanish version of the PAT; suggestions for overcoming identified barriers and bolstering facilitators; summary of evidence for how screening can impact health disparities; frequently asked questions; and guide for sites in identifying resources to guide intervention based on PAT scores. The Toolkit will be completely web-based and easily accessible on all computer, tablet and smartphone platforms. We will identify options/procedures for integrating PAT into EHRs although we expect that sites may have integrated the PAT into EHRs by that time. We understand that technology and “state of the art” materials are evolving rapidly, and we will rely on best practices at that time to assure a user-friendly resource for broad dissemination of the PAT.
Cognitive interviews. A preliminary version of the PAT Implementation Toolkit will be presented to participants. Qualitative, think-aloud interviews57 will be conducted by video conference using a semi-structured interview guide. During this audio-recorded interview, providers “walk through” the web-based PAT Implementation Toolkit to identify areas for clarification and improvement. Interviews will be transcribed by a professional transcription service and uploaded to a qualitative data management program (Atlas.ti ©) for formal coding.
Dissemination. The Agency for Healthcare Research and Quality Publishing and Communications Guidelines (https://www.ahrq.gov/research/publications/pubcomguide/index.html) will guide dissemination. We will coordinate with Children’s Oncology Group (COG) to assure all COG sites in the United States are aware of and have access to the Toolkit and are offered support in terms of PAT implementation. Similarly, we will contact all sites in the PIPS-CSS study to inform and support their use of the Toolkit. As a site in the National Cancer Institute Community Oncology Research Program (NCORP), Nemours is in an ideal position to inform a broad network of community sites regarding PAT implementation and encourage inclusion of the PAT in Cancer Care Delivery Research studies.
Data analysis
Aim 1. The qualitative interviews will be electronically recorded and professionally transcribed. All data will be stored and managed on a secure research drive. Data will be inductively analyzed using Atlas.ti ©, and qualitative oriented content analytic strategies will be used to identify codes, categories, and then themes.58,59 The process of analysis will proceed as the investigators simultaneously collect information through interviews, read each interview as an individual case, consider the topics addressed in the interview guide, disassemble each interview through coding with preliminary codes, define and combine codes into categories and categories into themes, and consider data from each category across all cases.60
The investigative team will first independently code three interviews selected to represent different stakeholders (patient/family, providers, and institution) to define an initial set of codes and create a codebook. Three experienced study coordinators will be trained to manage and analyze qualitative data following recommendations for qualitative methodology61,62,63 and supervised (in-person) by an expert qualitative researcher. The investigators will independently code five interviews selected to represent different stakeholders to define an initial set of codes and create a codebook, which will guide the analysis. For the remaining interviews, each transcript will be coded independently by a two-person team. Each team will discuss each interview and reconcile any discrepancies until they reach at least 75% agreement. After this point, interviews will be coded independently but will still be reviewed by a team member and discussed if a discrepancy is found. The investigators will review all the coded transcripts and provide feedback to the coders. As coding continues, codes will be combined into categories. Finally, all data will be examined in each category to combine categories into themes. The themes will be translated into content of the PAT webinar for Strategy I and enhanced strategies included in Strategy II with attention to health disparities and barriers and facilitators at the patient/family, provider and institution levels. The rigor of the iterative analytic process will be guided by standards for qualitative research.63,64
Aim 2. PAT implementation will be tested using a head-to-head randomized implementation (comparative effectiveness of two implementation strategies) trial in three cohorts over three years. Randomization. First, all 18 sites will be randomized to one of three cohorts. Second, each cohort will be further randomized to one of the two strategy conditions. Randomization will stratify sites by size to maximize internal validity and statistical power. Sites will be randomized in a two step process using the Excel = RAND() function. The sequence will be stored in a password-protected electronic file. Sample size. The estimated sample of patients/families is based on the median number of new patients each year from the PIPS-CSS database: Small (n = 60), Medium (n = 92), Large (n = 339) multiplied by 6 sites at each size, a total of 2946, with 1473 families allocated to each strategy. With an anticipated Interclass Correlation Coefficient (ICC) 65 of .005, the effective sample size is 950 per arm, a sample size that is sufficient to detect a small effect between the two strategies given 80% power (alpha=.05, one sided). We project a Hispanic sample of 20% of the total (n=589). Based on our research,9 approximately 30% of Hispanic caregivers will be more acculturated/not literate in Spanish at the level necessary to complete the Spanish PAT, rendering a sample of 412. With an anticipated ICC of .005, the effective sample size is 224 which is sufficient to detect a medium effective between the English and Spanish versions given 80% power (alpha = .05, one sided). At the provider level, each of 18 sites has a PI and up to four people screening, and for TIER sites, a Champion. Therefore the staff sample ranges from a minimum of 45 ([18x2]+9) to a maximum of 99 ([18x5]+9), likely in the middle given the range of size of sites. At the institution level, 18 sites will be randomly assigned to one of two strategies (9 sites/group). These sample sizes are comparable to other implementation science studies.66
Data cleaning and missing data. All data will be reviewed for valid values/data entry errors, outliers, and extent and pattern of missing data. Descriptive statistics will be reviewed for all variables. Consistency and logic checks that constitute standard review/cleaning procedures will be applied. The multiple group analysis models will provide valid estimates of efficacy if the proportion of missing values is <10%. Analysis will be conducted at the patient/family, provider and institution levels. The possible effect of program size and cohort will be examined and controlled if needed.
H2.1. ANOVA will compare the effectiveness of the two implementation strategies on penetration and health equity. The outcomes are proportions: families screened/families eligible, families provided feedback/families screened, ethnic minority families screened/ethnic minority families eligible, low SES families screened/low SES families eligible. ICCs among the clusters will be calculated and used to adjust for the cluster effect.67,68
H2.2. Three sets of outcome variables - perception of implementation, engagement in addressing health disparities, and burnout/job satisfaction - will be tested. A two group analysis with the framework of Structural Equation Modeling (SEM) will be conducted to compare the effectiveness of the two strategies.69,70 To test the effect of time (T1, T2, T3), we will conduct latent growth curve analysis;71,72,73 it is expected that provider outcomes will improve over time. Analyses will be conducted using Mplus 5.074 with ML estimation for outcome variables that meet the distribution assumptions, and with WLSMV estimation for outcome variables that do not. Potential mediating effects of favorable perception of implementation on provider job satisfaction and burnout will be examined using mediation models. TIER is expected to be associated with more favorable perceptions of implementation, which in turn will lead to less burnout and higher job satisfaction.
H2.3. At the institution level adoption of the PAT will be measured by a ratio of sites that initiate implementation/sites that agreed (H2.3a). If substitutions are necessary, sites that are newly invited will be added to the denominator and adoption calculated by total acceptances/total invited. ANOVA will be conducted to compare the effectiveness of the two strategies. For H2.3b sustainability (PIQ perceptions of implementation benefits and challenges), ANOVA will be conducted to compare the effectiveness of the two strategies on benefits and challenges. For H2.3c we are interested in the extent to which psychosocial care is matched to need and the extent to which intensive medical and psychosocial services are delivered for those most in need. Because this research is innovative, a cost-effectiveness threshold or criterion to which to compare costs of screening with these two implementation strategies has not been established, necessitating our consideration of valued outcomes in the psychosocial screening literature and resources available. Thus, we will use a data analytic approach that we used previously.18 ANOVA will be conducted to test whether psychosocial care is matched with levels of psychosocial risks, resulting in a 2 (Strategy) x 3 (psychosocial risks: clinical, targeted, universal) design on equitable distribution of services and costs of care. It is expected that the 3 PPPHM levels will be related to number and costs of services provided as measured on the PSMTC, with least at Universal and most at Clinical. It is further expected that TIER will result in a better match between level of risk and services provided. Scores from the PSMTC will be derived and mapped onto the levels of the PPPHM. Additional analyses will be conducted to compare English speaking and Spanish speaking families, different ethnicity, race, and SES and insurance status on the outcome variables for H2.3c.
Aim 3
Following Knafl and colleagues’56 protocol for the analysis of cognitive interview data, we will summarize the data by item and then aggregate the results across participants to reflect potential problems with Toolkit components and to identify Toolkit components that are clear and supportive of effective implementation. To ensure rigor, we will systematically analyze and then summarize the interview data following a formal coding scheme. Subsequently, we will develop a cognitive interviewing outcome report, which will include a description of the number and type of participants and interviews completed, a description of the specific procedures used in the interviews and the interview guide, and a written summary of feedback on each of the Toolkit components.56 This report will be distributed to the study team, which will then meet to discuss the issues identified. Decisions will be documented in a tracking matrix.75 Based on this feedback, with Aim 2 data and theoretical frameworks, we will revise the PAT Implementation Toolkit and finalize the website.