Background and project objectives
Each year, more than 20,000 young adults between the ages of 18 and 30 are diagnosed with cancer[25]; many of them do not receive services to meet the range of needs they experience during and after cancer treatment[26–30]. Young adults’ unmet needs result in negative outcomes, including higher distress[27, 28], poorer health-related quality of life[31], and higher physical symptom burden[26]. Despite the complexity and scope of their needs, young adults often do not use potentially beneficial services/resources, even when access is not an issue[32–34]. This disconnect between young adult needs and their use of existing services/resources suggests the need for a care coordination model that (1) effectively assesses young adults’ multifaceted, age-specific, individual, and dynamic needs, and (2) uses that information to efficiently connect them to services/resources.
A substantial step toward this care coordination model was the development of the first multidimensional measure of unmet needs designed specifically for adolescents and young adults: the Cancer Needs Questionnaire - Young People (CNQ-YP)[35, 36]. However, limitations to the usability and usefulness of patient-reported outcome measures like the CNQ-YP (e.g., length; wording ambiguity; redundancy or missing content; lack of connection between identified needs and follow-up actions) have frustrated their real-world implementation and effect on patient outcomes[37, 38]. In this project, we used UCD to redesign the CNQ-YP to optimize its usability and usefulness and prepare for its implementation in the University of North Carolina’s Lineberger Comprehensive Cancer Center (hereafter, Lineberger). Our UCD process (Table 2, Fig. 2) produced the Needs Assessment and Service Bridge (NA-SB), a care coordination intervention for young adults with cancer. All procedures were approved by the University of North Carolina’s Institutional Review Board.
Table 2
Data Collection Summary
UCD Aim | Method | Deliverable |
Review and refine EBP prototype (the thread) | Usability Testing • Young adult Survey • Cognitive interviews with young adults • Concept mapping with providers/staff | Evidence of the usability and usefulness of the CNQ-YP |
Identify user and contextual requirements (the fabric) | Ethnographic contextual inquiry • Guided tours with young adults and providers/staff from Lineberger • Semi-structured interviews with providers/staff from external organizations | User and contextual requirements for NA-SB’s design and implementation |
Design EBP and implementation strategy prototypes based on user and contextual requirements (thread + fabric + needle) | Design team Workshops • Workshop #1 • Workshop #2 | NA-SB prototypes and anticipated implementation strategies needed |
RESULT | NA-SB + compilation of relevant implementation strategies |
Multidisciplinary design team
In dissemination and implementation, stakeholder engagement is often limited or superficial[39–41]. In contrast, UCD demands an active and iterative approach to engagement, often with the same group of users reviewing prototypes at multiple time points[42]. Thus, at the beginning of the project, we convened an NA-SB design team comprised of key stakeholder groups. Throughout the project, the investigator team presented prototypes and other information to the design team and, based on their interactions with prototypes and collaborative discussion, made iterative improvements to NA-SB design and implementation strategies.
Design team members included researchers in cancer care delivery, patient-reported outcomes, UCD, and implementation science (n = 4) and prospective NA-SB users, including Lineberger clinical partners (oncologist; social worker/director of Lineberger’s young adult program [n = 2]), and young adult representatives (n = 5) nominated by clinical partners. Nominees were primarily individuals who had previously expressed interest in research or advocacy activities related to young adult cancer and thus, would be more likely to consider the extensive and ongoing participation that joining the design team would entail.
To recruit young adult representatives for the design team, clinical partners connected young adults via email to the project lead (EH). EH provided them with materials including a project summary, a breakdown of their expected role and time commitment, and a brief summary of UCD. EH then met with each young adult interested in participating to discuss the project and develop rapport, then met with them all together to build group rapport. Young adult representatives received a one-time $150 incentive for participation.
Young adult representatives included a racially and ethnically diverse group of 5 men and women. They represented different timepoints in their cancer trajectory, with one in active treatment, one in maintenance treatment, and the others in the survivorship phase. In addition to their lived experience with cancer, young adult representatives brought other areas of content expertise to the project, including instructional design and mental health service delivery.
(1) Review and refine prototypes (optimize the thread)
Overview
We conducted three rounds of usability testing to examine user interactions with the CNQ-YP: (1) an online survey assessing young adults’ needs and preferences for a needs assessment using the CNQ-YP as a prototype for them to react to; (2) cognitive interviews[43] with young adults to triangulate survey data with in-depth evidence of their perceptions of the CNQ-YP’s usability and usefulness; and (3) concept mapping[44] exercises focused on usefulness, in which young adult program providers mapped CNQ-YP-identified needs onto services/resources to address the needs.
Young adult survey
Objectives. To identify missing content, streamline redundant or low-priority content, and other usability and usefulness concerns.
Instrument. The survey instrument (Additional File 1) included three sections: (1) study information, consent, and demographic items (i.e., age, gender, clinical characteristics, social support, educational/vocational status, health insurance status), (2) the CNQ-YP in its original form, and (3) items assessing respondents’ perception of the CNQ-YP. To assess general attitudes towards the tool, we used items from three Likert-type measures of feasibility, acceptability, and appropriateness[45]. We assessed usefulness through two Likert-type items asking (1) the extent to which respondents thought the CNQ-YP accurately captured their needs, and (2) the likelihood that they would use services or resources offered to them based on indicated needs. For each of these measures, we qualitatively probed respondents on usability and usefulness issues driving their concerns with the tool’s feasibility, acceptability, or appropriateness.
Sample and Recruitment. To be included in the survey, we required participants (n = 100) to be age 18–30 and have been diagnosed with cancer prior to survey administration. To ensure young adult participant diversity (race, ethnicity, age, geographic region, setting of care, etc.), we recruited through key contacts (i.e., leaders of young adult programs and advocacy groups in the United States identified by our clinical partners), social media (i.e., a series of Twitter messages shared by tagging relevant groups and hashtags), and our design team.
Procedure. We administered the survey through a secure online platform, Qualtrics (Provo, UT). On average, the survey took 15 minutes to complete.
Analysis. We used descriptive statistics for respondents’ demographics, needs reported on the CNQ-YP tool, and perceptions of the CNQ-YP. To identify emergent themes regarding the CNQ-YP’s usability and usefulness in free-text responses, we used template analysis[46].
Cognitive interviews
Objective. To triangulate survey data on CNQ-YP usability and usefulness through nuanced understanding of content, wording, or comprehension concerns.
Interview guide. With input from the design team, we developed the cognitive interview guide to encourage participants to “think aloud” as they read and reflected on the CNQ-YP itemset and probe them to comment on topics such as item content and wording, response options, format, length, comprehensiveness, repetitiveness, etc. (Additional File 2).
Sample and recruitment. To promote generalizability of cognitive interviews, we purposively sampled from among survey participants individuals who varied across select demographic characteristics (i.e., age, race, gender). We recruited young adults (n = 5) until we reached thematic saturation, i.e., when subsequent interviews did not generate new information regarding CNQ-YP’s usability or usefulness.
Procedure. EH conducted one-hour cognitive interviews (n = 5) via Zoom, a video-conferencing platform. Interviews were audio-recorded. EH navigated the CNQ-YP through the Zoom screen-share function, soliciting participants’ input on each item. At the end of each interview, EH summarized her takeaways with interviewees for the purposes of member checking[47].
Analysis. We inductively identified themes, noting concerns related to the CNQ-YP’s usability and usefulness. We then created a table organizing participants’ concerns within each of the identified themes for presentation to the design team during our first workshop (described later).
Concept mapping
Objective. To group needs assessed by the CNQ-YP by services/resources expected to address those needs.
Instrument. The design team approved changes to CNQ-YP content based on survey and cognitive interview results. We pre-loaded the resulting list of young adult needs into an online secure platform called Concept Systems Global Max © (CSGM). CSGM included two concept mapping exercises: (1) sorting an electronic deck of cards, each containing a young adult need, into like categories (i.e., “follow-up domains”) that could be addressed by the same service/resource, or other follow-up action (e.g., needs related to depression and anxiety might be grouped together as potentially addressable by referral to a mental health professional); and (2) rating needs on Likert-type response scales in terms of two key pragmatic properties: importance (i.e., severity of consequences if that need goes unmet) and actionability (i.e., likelihood that need can be met through a service or resource)[48].
Sample and recruitment. Concept mapping participants included cancer program providers (e.g., oncologists, nurses, and social workers) and staff (e.g., program managers and administrators) - i.e., the prospective NA-SB user group expected to have the most knowledge about service and resource delivery for this population. Recruitment through the key contacts established during survey recruitment was intended to achieve the minimum sample size of n = 15 needed for concept mapping analyses[49].
Procedure. Participants accessed the web-based concept mapping exercises through emailed links to the project in CSGM. The exercises took approximately 30 minutes to complete.
Analysis. CSGM used hierarchical cluster analysis to characterize how participants grouped needs, creating several potential cluster maps based on proximity among needs, where proximal needs were more frequently grouped together as triggering the same follow-up action than distal ones, and ‘go-zone graphs’, in which needs are displayed as points on a quadrant in terms of their relative importance and actionability. Young adult design team representatives vetted results.
(2) Identify user and contextual requirements (understand and prepare the fabric)
Overview
To gather detailed information about prospective users and the context for NA-SB implementation, we used ethnography (i.e., guided tours and semi-structured interviews), a promising yet underused method for implementation research[50]. By documenting naturally-occurring user tasks and interactions among patients and providers through in-depth observation, ethnography provides rich data on implementation context[51, 52]. Ethnographic methods are relevant to UCD because they offer more nuanced understanding of users and context than traditional questionnaires or interviews, including novel insights on user tasks, attitudes, and interactions with their environment[17, 21, 53]. We used a novel UCD user and contextual factor framework[21] to guide data collection and analysis, combining ethnography and contextual inquiry. Data from this phase informed context modifications and identification of minimally necessary implementation strategies. Additional File 4 includes the Standards for Reporting Qualitative Research (SRQR) checklist adhered to for these data collection activities.
Guided tours
Objective. To capture contextual elements beyond just those which users can verbalize, including details and motivations that have become habitual or implicit to the tasks they perform[54].
Instrument. To promote the flexibility required for guided tours[55, 56], we identified potential questions based on four domains of Maguire et al.’s typology of user and contextual factors to consider in UCD from which we could choose: (1) user characteristics, (2) user tasks, (3) physical and technical environment, and (4) organizational environment[21] (Additional File 3).
Sample and recruitment. To capture the perspective of potential NA-SB implementers, we conducted guided tours with our clinical partners at Lineberger (n = 2). To capture the patient perspective, we conducted guided tours with young adults ages 18–30 receiving inpatient or outpatient care at Lineberger (n = 10). Our clinical partners at Lineberger facilitated the recruitment of young adults for guided tours by distributing a recruitment flyer and connecting EH via email to those interested.
Procedure. EH conducted four-hour guided tours with clinical partners as they completed clinical, administrative and other duties, asking questions about their tasks and thoughts. EH followed young adults and accompanying family members from the moment they entered the hospital for their outpatient appointments until the moment they exited, asking them questions as they interacted with their environment and healthcare professionals, while attempting to minimize participant disruptions. For inpatient guided tours, EH spent two hours with young adults receiving inpatient care. EH took extensive field notes and audio-recorded portions of the guided tours for which only consenting parties were present. We offered young adult participants a $50 participation incentive.
Analysis. We used template analysis, identifying a priori themes based on Maguire’s constructs and allowing for identification of additional themes[46]. Two study authors independently coded excerpts from guided tour field notes and interview transcriptions per Maguire constructs. For each domain, they collaboratively synthesized user and contextual factors and created a “translation table”[57], which translated factors into their implications for NA-SB design and implementation. For example, providers reported the importance of integrating new tools into the electronic medical record; we translated this into the requirement that NA-SB interface with Lineberger’s electronic medical record. All requirements were vetted and prioritized by the design team during the second workshop (see description below).
Semi-structured interviews
Objective. To review findings from guided tours with external users and identify any areas of divergence or additional needs or contextual features, thus promoting generalizability of findings.
Interview guide. With input from the design team, we developed a semi-structured interview guide based on Maguire’s typology[21] and guided tour findings.
Sample. We conducted semi-structured interviews with the leaders of young adult programs and advocacy groups who had previously facilitated survey and concept mapping recruitment: program managers (n = 2), nurse navigators (n = 2), and patient navigators (n = 2) serving primarily young adults.
Procedure. EH conducted one-hour semi-structured telephone interviews. At the end of each interview, EH summarized major takeaways for member checking[47]. We audio-recorded and transcribed interviews verbatim.
Analysis. We analyzed interview data using template analysis[46].
(3) Design prototypes based on user and contextual requirements (thread the needle)
Overview
Through two three-hour workshops, the design team collaboratively redesigned the CNQ-YP (i.e., the EBP) with usability and usefulness in mind and redesigned Lineberger care processes (i.e., the context) to facilitate the tool’s implementation and usefulness in routine care. This prototyping process – which relied on visual cues to digest user data with multiple user groups – represents a novel method for coproduction in implementation science. It resulted in NA-SB and a compilation of implementation strategies, each informed by context and designed to account for the other’s characteristics.
Design Team Workshop #1
After the survey, cognitive interviews, and concept mapping, we convened the NA-SB design team for a workshop in which we selected the most interpretable concept mapping cluster map (i.e., the cluster map with the highest face validity) and considered eliminating from the CNQ-YP needs that were relatively unimportant and not actionable. For example, the design team eliminated items assessing whether patients perceived their treatment staff as “approachable” or “friendly” based on concept mapping participants’ low ratings of these items’ actionability in terms of service or resource provision. The refinement of CNQ-YP items was further informed by usability and usefulness data from the survey and cognitive interviews. For example, the original CNQ-YP features similar items with different lookback periods (e.g., “since my diagnosis”, “in the last month”); survey and cognitive interview participants felt these multiple reference periods were confusing and redundant given the tool’s goal of addressing current needs through service provision. Thus, the design team anchored all items to needs present at the time of CNQ-YP completion and eliminated redundant content. In other cases, items were added based on user feedback on missing content (e.g., sexual health). Usability data also informed changes to the tool’s display, instructions, and sequencing.
After grouping high-priority needs by follow-up domains, the design team identified services/resources at Lineberger which corresponded to each follow-up domain. We also anticipated implementation strategies needed to facilitate this kind of multidisciplinary service provision. The meeting resulted in a redesigned patient-reported outcome measure for use in NA-SB in which priority needs were grouped based on services/resources available at Lineberger to address those needs.
Design Team Workshop #2
After soliciting user and contextual data through guided tours and interviews, we convened the design team for workshop #2 during which we presented the ethnography translation table (developed from the guided tours and semi-structured interviews) alongside the NA-SB measure produced during design team workshop #1. This allowed the design team to anticipate context modifications and necessary implementation strategies with respect to the redesigned tool. For example, the content of the tool might necessitate its administration by certain providers (e.g., social workers); staffing and workflow patterns may need modification to facilitate administration by social workers. Through popular UCD methods, ‘personas’ (i.e., generic representations of key user groups to convey users' needs to the design team,) and ‘scenarios of use’ (i.e., specific examples of how users, context, and NA-SB might interact)[21], we collaboratively determined the most salient requirements for NA-SB delivery, resolved discrepant requirements, and specified who will deliver the needs assessment, when, how often, and the materials and procedure that will be used to do so. Additionally, we used this workshop to co-design the bridge between identified needs and care delivery—i.e., making explicit the follow-up actions needed for each patient-reported need. Finally, we walked through Maguire’s four domains to discuss potential remaining barriers to NA-SB-context fit and identify appropriate implementation strategies to address them. For example, some important user and contextual requirements could not be addressed through specification of NA-SB delivery (e.g., leadership buy-in); these were areas in which the application of strategies was deemed important for future implementation.