Throughout iTP3, data from each year was used for continuous quality improvement (CQI) and aided in updating the program model. Results are formatted to show the evolution of the project based on CQI processes.
Traditionally-funded innovators (cohorts 1-3)
Seventeen Innovators were funded across 2016-2019, using a traditional funding model. Each year, Innovators submitted an application for continued funding based on their progress and the viability of their projects. Four Innovators were engaged for the entire 4-year time period, 2 for 3 years, 2 for 2 years, and 9 for a 1-year period.
Innovator programs represented a variety of priority populations including ethnic and racial minority youth; older teens; pregnant and parenting teens; lesbian, gay, bisexual, transgender, queer and questioning (LGBTQ+) youth; youth in foster care; youth with intellectual and/or developmental disabilities; depressed teens; homeless and runaway youth; and pre-service teachers. Programs also targeted ecological levels of intervention including interpersonal relationships, organizational interventions, community-based-intervention, and individual behaviors. As was expected in the initial funding of innovative interventions, not all programs were fully developed for testing. Rather some only made it through the initial stages of exploration or program development. See Table 3 for a full description of Cohort 1 Innovators.
Cohort 1. Consisting of 15 funded organizations, Cohort 1 Innovators were interviewed regarding their program and their experience with iTP3. All Innovators had experience with TPP and most had previous experience with program development. They used this experience to address a gap identified from past programming, apply lessons learned from past programming to their iTP3 program, and apply or adopt a TPP program for a new priority population.
Additionally, all Innovators considered themselves to be innovative thinkers. Innovators planned to push innovation further through engaging partners, working with iTP3 and CBA providers, and utilizing their organization’s embedded culture of innovation. Long-term, they hoped to have a program ready for a large-scale trial, improve adolescent health, have a new program component, and/or have a program that could be implemented in other communities.
Despite experience with both TPP and program development, Innovators identified seven primary challenges to their iTP3 program. These challenges were related to their 1) priority population, 2) ability to develop and disseminate a program, 3) organization, 4) advisory board, 5) partners, 6) ability to replicate the program, and 7) ability to plan for sustainability.
Cohort 2. Cohort 2 included eight total Innovators; six returning from year two and two new. Interviews assessed the program planning and implementation process, innovation, and successes and challenges experienced by Innovators. Overall, Innovators felt their core definition of innovation in TPP stayed largely the same, but expanded. Innovators reflected their overall long-term goals remained the same as indicated in their initial applications; however, their process of reaching the end goal changed in some cases. Innovators felt they made progress toward reaching their goals, but more work was needed. This is reflective of this iterative process and push for HCD and systems thinking that was a core aspect of iTP3.
There was recognition that innovation is a process rather than an outcome. Innovators recognized that innovation required flexibility and openness to making changes, as well as collaboration with others. In addition to reflecting on their particular programs, Innovators expressed appreciation for the flexibility of the funding model as the ability to make changes and iterate programs is crucial for development of an innovative approach. This funding allowed project teams a unique opportunity to focus on development and truly meet the needs of their priority population, as opposed to being held to strict, unchangeable metrics and ideas initially proposed.
In year four, seven Innovators, all funded the previous year, continued another year of program development. All seven of these Innovators were interviewed, once again. Key findings built upon findings from previous interview iterations. Innovators felt their definition of innovation and how their program was innovative had not changed, but rather solidified throughout the past year. They were still largely working toward the same goal(s) identified in their initial interviews in year two or three (depending on Innovator), but in some cases had expanded their goal to incorporate new elements and reflect lessons learned and/or priority population feedback. Continuing Innovators largely identified HCD and systems thinking as useful and were able to incorporate it broadly into their program development, but several felt the concepts would have been more useful if introduced during their first funding cycle (before the concepts became the focus of iTP3 CBA). Overall, these Innovators appreciated the flexibility of this funding model and the opportunity to develop an innovative program through a mechanism that allowed for iteration based on feedback and results. They felt iTP3 and innovation funding in general was important to advance the field through reaching underserved populations, fostering cross-sector collaborations, community relevance, and the opportunity to look at problems in new ways and develop creative solutions.
Post funding follow-up. In the final project year, six previously funded Innovators who were funded over multiple cohorts were contacted for follow-up interviews. All six who were contacted agreed to be interviewed. The interviews focused on the status of their program post funding, and overall perceptions of iTP3. At the time of the interview all interviewees reported they were able to continue development and expansion of the program. Four reported they completed a pilot of their program and two were ready for rigorous testing. Following iTP3 funding, Innovators sought external funding and disseminated their work through peer reviewed publications, dissemination through a website, community presentations, conference presentations, and internal organization conferences.
Innovators were asked specifically how iTP3 helped in program development. Innovators reported that administratively, iTP3 helped with keeping the program on track. However, beyond administrative functioning, Innovators found that iTP3 made a contribution to their program development through sharing HCD and systems thinking technical assistance, funding, guiding the development and iteration processes, encouraging or requiring engagement of stakeholders, and facilitating sharing or collaboration amongst Innovators.
As the years of iTP3 progressed, iTP3 had a greater emphasis on both HCD and systems thinking as technical assistance for Innovators. Most Innovators indicated they planned to use HCD or systems thinking in the future, while another indicated they hoped to use the techniques but were not as certain. When talking about HCD, most highlighted specific tools and use of the process of design including continuous testing and iteration, and the development of a program that meets the needs of the priority population. For systems thinking, Innovators identified benefits of systems thinking including getting multiple perspectives and stakeholders, looking at the big picture, creating a systems map, and identifying leverage points. Additionally, some Innovators identified specific applications of systems thinking within their work.
Program model iteration one
Following Cohort 1 and 2, there was an observed need to explore new program development and funding models. For this reason, the first iteration of design funded Innovators (referred to as cohort 3) was funded concurrent to cohort 2 of the traditionally-funded Innovators. The purpose of this new iteration was to give organizations an opportunity to develop a new idea through an intensive HCD training process. This design process was further iterated in subsequent years.
Design funded innovators
Cohorts 3 through 5 (2017-2020), consisted of three separate models for facilitating innovation through intensive HCD training. Cohort 3 was considered a pilot year for the design funding model, as it overlapped with the traditional model. The design funding model provided intensive CBA in HCD, in an effort to facilitate the process of innovation among program developers.
Cohort 3: Design bootcamp cohort. Cohort 3, a Design Bootcamp, was the first iteration of the design funding model. The Design Bootcamp was created to both teach Innovators the principals of HCD and facilitate HCD in a four-day period, resulting in an innovative program. In total, two Innovators were funded as a result of the Design Bootcamp.
The two programs funded through the Design Bootcamp both targeted rural populations in the same community through community-based (out of school) interventions. However, the interventions were quite different. One intervention entailed adapting an existing HIV/AIDS curriculum for pregnancy prevention, using a community health worker model. The other focused on the creation of a sexual and reproductive health (SRH) game for middle school students. See Table 4 for an overview of Cohort 3 programs.
Program model iteration two
Throughout the process, it was quickly evident that requiring a full week for design work was an unrealistic expectation for many professionals. For this reason, the iTP3 team developed four- and two-day HCD workshops to accommodate different schedules through a Design Bootcamp model. To further address this challenge, iTP3 transitioned to a Design Sprint model in which full-day design sessions were separated by several weeks. Additionally, the time between sessions was intended to allow for organizations to further develop their idea and gather feedback between sessions.
Cohort 4: Design sprint cohort. Cohort 4, the Design Sprint model, utilized much of the same methodology as the Design Bootcamp but was divided in multiple one-day workshops over the course of several months. Organizations were supported to bring together teams of people within their local communities to participate in the Design Sprints. Distinct programs resulted from the Design Sprints in four very different communities. The resulting programs focused on training the staff of youth serving organizations, a music-based program for young males in juvenile justice settings, adult caregivers and adolescents through SRH videos, and creation of a guide for policy development for communities and schools. See Table 5 for an overview of Cohort 4 programs.
Interviews. Interviews were conducted with representatives from all four Cohort 4 organizations. Cohort 4 Innovators felt that HCD and systems thinking were useful in their program development and appreciated the opportunity to bring together a diverse group with varying perspectives. Because of the fluidity and intentional flexibility of this model, each Design Sprint team participated in different activities and a process tailored to their community. For this reason, it was more difficult to identify distinct themes across Innovators. However, these Innovators recognized the need to involve stakeholders and gain priority population input, and felt that the skills gained through the Design Sprint process were not only valuable in this project, but also transferrable to other projects and could be used throughout their broader organizations.
After action reports. After action reports, completed by the iTP3 team, documented design activities completed and notes on the design session including process, engagement, reflections, pivot points, and lessons learned. In reviewing the reports, the iTP3 team recognized a challenge in traveling to communities to conduct Design Sprint sessions, and the desire for community members to be equipped to lead design activities and use HCD strategies in other aspects of their work. The team decided it would be more feasible to train community members to be design leaders that conduct Design Sprints with stakeholders in their community, rather than iTP3 traveling to communities and leading design sessions. This would increase the number of communities served and sustain the use of HCD beyond iTP3-funded projects.
Program model iteration three
As iTP3 approached its final year, the team added a focus on sustainability of not just programs, but also equipping individuals at the local level to use HCD and systems thinking methodologies beyond this project. For this reason, Cohort 5 shifted to a Design Cadre model in which individuals were trained to lead design sessions within their organizations or communities. Additionally, this model of training facilitators lessened the travel burden on iTP3 team members as the iTP3 team did not have to travel repeatedly to all sites to facilitate design sessions.
Cohort 5: Design cadre cohort. The final iteration of Design Funded Innovators made the transition to empowering individuals to lead the design process within their own communities rather than relying on the external facilitation of the iTP3 team. This served to increase program design capacity and sustainability within communities. Eleven individuals were funded to participate in the Design Cadre to receive in person and virtual training, and then lead a Design Sprint within their organization or community. In total, ten organizations were represented in the design process as two individuals represented one organization. Following Design Sprint sessions conducted in communities, Cadre participants were asked to present or ‘pitch’ their initial program idea to the iTP3 project team. After the initial presentations, ten individuals (nine organizations) were invited to further iterate their idea and prepare a full pitch to be presented for a chance at receiving funding to pursue implementation of their program idea. Based on the final pitches, eight organizations were provided funding in varying amounts to continue program development activities.
Of the nine programs that were pitched, the programs focused on nine separate priority populations. Examples of priority populations identified for these programs include homeless and runaway male youth, School Based Health Center Youth Action Councils, health educators, and youth of varying age ranges. Settings represented rural and urban environments, community-based settings, school-based settings, and virtual settings. The program ideas were formulated and proposed, but not fully developed and executed due to the format of the cohort, timeline barriers imposed by the onset of COVID-19, and community-based restrictions enforced soon after the programs were funded for development. See Table 6 for an overview of Cohort 5 Innovators.
Interviews. Interviews were conducted with 10 of the 11 Cadre participants in Cohort 5. Interviews focused on the participant’s perceptions of the design process, the resulting program, innovation, and other relevant domains. After the Cadre experience, most felt prepared to lead design sessions to facilitate innovative program development in their communities with half of participants making recommendations for additional training needs to help with full program development. All participants considered the in-person group trainings to be the most beneficial part of the experience. Within the in-person group trainings, participants identified experiencing the design methods, practicing facilitation, and one-on-one coaching as the most valuable aspects.
Cadre participants were asked to share the most important characteristics for innovation. The most common response was the need to have appropriate people involved in the design process including relevant stakeholders and end users, and ensuring that those involved are committed to the process. Participants shared that having the appropriate time to dedicate to the process and capacity to conduct the sessions was very important from a logistical perspective. Support for the process was also highlighted by participants particularly noting the importance of having buy-in from leadership, teams and the overall organization. Additionally, broad knowledge of design skills for both the facilitator and team was noted as important, as well as other individual characteristics including being willing to try something new and fail, having flexibility to change course, and being able to step back and listen.