Blending Participatory Action Synthesis and Meta-ethnography: An Innovative Approach to Evaluating Complex Community Health Transformation

doi:10.21203/rs.3.rs-99033/v1

Download PDF

Research

Blending Participatory Action Synthesis and Meta-ethnography: An Innovative Approach to Evaluating Complex Community Health Transformation

https://doi.org/10.21203/rs.3.rs-99033/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 17 Nov, 2021

Read the published version in Journal of Participatory Research Methods →

Version 1

posted

You are reading this latest preprint version

Background: Community health improvement processes are diverse and complex, and evaluation methods to gain generalizable knowledge across community are limited by available data, and the need for deep contextual knowledge.

Methods: This article describes an innovative participatory approach to evaluation of a community transformation initiative involving up to eighteen communication nationwide. The approach blends two qualitative research synthesis methods: participatory action synthesis and meta-ethnography and applies them to the pragmatic evaluation of a program in real-life settings.

Results: In this article, we present the justification for and details about the evaluation process. Four cycles of synthesis and engagement resulted in development of concepts to describe community actions for transformation.

Conclusion: The approach presented here will be useful to both researchers and practitioners interested in evaluating community-based health and well-being initiatives and other complex interventions conducted in complex settings.

Health Economics & Outcomes Research

Health Policy

Complex Systems

Participatory Research

Adaptation

Evaluation Methodology

Few systematic approaches exist for the evaluation of implementation of multi-site community improvement programs. Community contexts are heterogenous and the interventions and implementation strategies are context specific.
Data for evaluation is routine implementation process data collected and owned by communities.
We describe an innovative approach that adapts two qualitative research methods to support a community-led data synthesis process to identify generalizable themes underlying community improvement efforts.
Integration of implementation science with participatory methods is an advancement of the field.

Addressing Complexity in Public Health

Despite the trillions of dollars in healthcare spending, the achievement of population health outcomes in the US have stagnated, and inequities in well-being continue to persist [1]. Solutions to these problems cannot come only from healthcare, but require collaboration across multiple sectors (e.g. social services, criminal justice, education, economic development) with an explicit and intentional focus on equity and justice [2].

In their call to action to embrace a new approach to public health, dubbed Public Health 3.0, DeSalvo et al. [3] describe the future role of public health leaders as public health strategists who can lead cross-sector collaborative efforts to address social determinants of health. This requires the focus of public health efforts shift from being community-placed (i.e. situated in communities but with services owned and delivered in a fragmented manner by public health agencies and healthcare institutions) to being community-based (i.e. facilitating the development and planning of integrated solutions led by diverse community coalitions focused on local priorities, local context and local innovation). As the country struggles to overcome and build from COVID-19, the need for these approaches will have even greater urgency.

This shift to community-based strategies increases the complexity of planning, implementation, and evaluation of community health programs in two ways. First, these community-led interventions will need to have multiple interacting components at various levels of the socio-ecological model [4]. Second, the settings in which these interventions are implemented are complex and heterogeneous, with the path between interventions and outcomes being subject to context-dependent characteristics, non-linear feedback loops and emergent, unpredictable behavior [5]. Therefore, the development and implementation of community-based interventions cannot simply be mechanical replications of generic evidence-based approaches but must utilize an approach referred to by Greenhalgh and Papoutsi [6] as “complexity-informed.” This approach recognizes that successful implementation of interventions in a complex system requires the identification of how multiple community-specific influencers and their interactions affect the results and the development of a plan to address any unanticipated consequences that might arise as a result of the introduction of the interventions. For example, as communities begin to reopen after the COVID-19 pandemic begins to subside, there will be no single formula. While broad guidelines may be issued at the federal or state level, communities will need to find solutions that work for their own particular circumstances.

A complexity-informed approach to program implementation requires the introduction of solutions through a series of sequential experiments, evaluating how the system responds and making any context-appropriate adaptations based on the findings. In an article about decision-making in complex systems, Snowden [7] describes this approach as “probe-sense-respond”, which uses small tests (probes) led by communities to “sense” the system before any response can be formulated. Snowden contrasts this approach with the typical process of top-down decision making adopted by leaders which he calls “sense-analyze-respond,” where a common formulaic approach to data collection (e.g. a needs assessment) and analysis (to identify gaps) is used to make decisions about interventions and programs priorities across a large number of contexts.

The Need for New Evaluation Methods

A complexity-informed approach to program implementation needs the appropriate research and evaluation methods that allow for insights from iterative cycles of learning and action to be incorporated into evaluation. These methods and the data they require are different from those typically used for outcome evaluation. As noted by DeSalvo et al. [3], they need to utilize granular, actionable data that provides rich qualitative and quantitative information about the process of implementation and include stakeholders living in the community in the process of meaning making. It is enormously resource intensive to create systems to collect this data, especially because the need for data may change as the implementation progresses. Much of the data available therefore are likely to be routine project implementation data collected opportunistically in the field. The data are often incomplete and are referred to in the literature as Flawed, Uncertain, Proximate, and Sparse (FUPS) [8]. Generating useful information to create generalizable knowledge in complex systems requires careful triangulation of FUPS data from multiple sources.

At this time, few systematic approaches exist for practical community-led evaluation using multi-source, imperfect real-world data. In this paper, we demonstrate the adaptation of two traditional qualitative research methods to help communities gain insights about the mechanisms that facilitate change in their complex systems. We describe how these methods were used to evaluate a community-based system transformation initiative called Spreading Community Accelerators through Learning and Evaluation (SCALE).

Background – The SCALE Initiative

SCALE was a multi-year Robert Wood Johnson Foundation funded national initiative to build the capacity of community coalitions to achieve long-term improvements in community health, well-being, and equity. The initiative was a signature initiative of the broader 100 Million Healthier Lives (100MLives) movement and was led by the Institute for Healthcare Improvement (IHI), the industry leader in the use of Quality Improvement (QI) methods to achieve better outcomes in health and healthcare. SCALE was implemented based on an active partnership between the implementation team (IHI, partners, and faculty), and communities coalitions. Between eighteen and twenty-two communities participated during the duration of the project.

SCALE Theory of Change

The SCALE theory of change is shown in Fig. 1 [9]. The theory links the achievement of healthy, equitable communities to the acquisition and everyday practice of a core set of skills, strategies, and tools known collectively as the “Community of Solutions Framework.” These included skills in personal leadership, in developing deep and lasting intra- and intercommunity relationships and in the use of systematic QI methods. The theory of change posits that when these skills are acquired and become part of everyday practice, the way in which members of community coalitions relate to each other begins to shift and an environment is created where communities can effectively engage in complexity informed improvement to find innovative and lasting solutions to their health and wellbeing challenges.

SCALE Communities

Twenty-two diverse community coalitions nationwide were selected through a competitive process to participate in SCALE from 2015–2019. Community improvement initiatives undertaken by these coalitions spanned a wide range of topics including food security, adverse childhood events, safe neighborhoods, building youth leadership, women’s mental health, racism, and equity. The communities were trained in the Community of Solutions Framework through eight multi-day face to face events called CHILAs (Community Health Improvement Leadership Academies) and were assigned coaches to help them apply these skills to their individual improvement initiatives. In the later stages of SCALE, which included 18 communities, the coalitions were required to spread these skills to other communities within the region.

SCALE is an example of a complex intervention implemented across multiple complex settings. It consists of a diverse set of improvement interventions carried out in a variety of contexts by highly heterogeneous community coalitions with shifting leadership, staff transitions and competing priorities. An evaluation team consisting of faculty and graduate students from the Universities of North and South Carolina was embedded within SCALE to assist the communities through the activities specified in the theory of change. The evaluation team, in partnership with IHI, developed data collection systems that the communities could use to document their progress, evaluate their results, and make adjustments and improvements as needed.

At the end of SCALE, the evaluation team led a collaborative effort to learn about the process of community transformation across SCALE, using the data collected by the communities during the implementation process. The overall project documentation was of variable quality and completeness and the language and data were highly specific to each community context and improvement goal. To make sense of these data to draw generalizable conclusions about the achievement of SCALE, the evaluation team needed a method with two key components: (a) an engagement component that invited SCALE stakeholders to bring their knowledge and perspectives to fill data gaps and to provide community specific interpretations of their data and (b) a synthesis component that aggregated these insights into broader generalizable themes and concepts. In the absence of any existing methodology to do this, the evaluation team developed a novel approach adapting a combination of two existing qualitative research methods: meta-ethnography and participatory action synthesis. We describe the details of this approach in this paper. Results of the SCALE evaluation using this approach are currently under preparation as a separate manuscript [10].

Evaluation Questions

Evaluation questions were developed through an iterative collaborative process involving the funder, IHI, and select community members who volunteered to participate in the evaluation. Questions were solicited online, sorted and streamlined by the evaluation team and sent back for the next iteration. Multiple cycles of online feedback were used to craft a final set of overarching questions important to all stakeholders. They were:

What are the most common pathways that communities have followed in their transformation journey?
What are the common and unique knowledge, capabilities, practices, and relationships that the communities have used?
What are the mechanisms that have brought about change?

Evaluation Approach

Our evaluation approach is a new method involving an amalgam of meta-ethnography and participatory action synthesis. Meta-ethnography, pioneered by Noblit and Hare [11], is a structured seven-step method for synthesizing findings from a small number of ethnographic studies to create new interpretations. The primary focus is to use convergent and oppositional themes across the studies to craft higher-level interpretations that reveal new levels of meaning linking the studies. Meta-ethnography is part of a set of interpretive methodologies called qualitative research synthesis [12] that expand the approach to qualitative studies beyond just ethnography. Overall, these approaches seek to create higher order meaning by synthesizing the learning from lower level data.

Participatory action synthesis [13] arose in response to critique that interpretive methods such as meta-synthesis are limited in their ability to construct new knowledge because they do not utilize the contextual experience of researchers to bring new and creative interpretations beyond what is present in the data. Participatory action synthesis adopts a social constructivist lens to the meaning making process, and explicitly acknowledges the need to ground the data within the contexts and values of the settings in which the data was collected. It recognizes synthesis as a communal process where the stakeholders bring their own social, political, and cultural experiences to the table. Additionally, it posits that these experiences could result in the collective creation of new interpretations of the data that offer greater insights than what can be achieved through a passive process of identifying convergent and divergent themes. During the synthesis process, each participant brings his or her own personal worldview or research locus to the data, and a team interpretation is negotiated through open discussion of the intersecting narratives emerging from these worldviews. Iterative cycles of joint meaning making and integration of new themes with the data deepens the collaborative process.

Both meta-ethnography and participatory action synthesis were developed as research methods, and the guidelines for their use are targeted at researchers [14]. For researchers, the epistemological and ontological underpinnings of these synthesis methods are important because they determine the participants and the types of data that should be included to answer a particular research question. In their introduction to participatory action synthesis, Wimpenny and Savin-Baden [13] require the data to be to be situated within an established qualitative research theory, and caution against poor quality data. Moreover, they emphasize the importance of the explicit articulation of each team member’s research position (i.e. the “reflexive stance” of an individual, which is intrinsically tied with the team member’s role), so that the negotiation about what to include and exclude in the communal interpretations can be equitable and transparent.

Our goal was to adapt these methods for use by practitioners to reinforce heterogeneous program implementation data of variable quality and completion, and to collaboratively synthesize these data to create generalizable knowledge about mechanisms of community transformation. To do so, our evaluation approach blended the social-constructivist aspect of participatory action synthesis with the interpretive aspect of meta-ethnography. We relaxed some of the theoretical constraints mentioned above (e.g. the need for established research theory, and the requirement of high-quality data), but remained faithful to the key principles underlying these methods: communal meaning making, valuing contextual experience, encouraging open discussion and negotiation, and systematic integration of data from multiple sources.

Specifically, we used the iterative reflection action cycles of the participatory action synthesis process to engage key SCALE stakeholders in reflecting on, validating and interpreting community specific data. We then used the steps of the meta-ethnography approach to create progressively higher order concepts and themes, and to develop lines of argument about the process of community transformation. Figure 2 shows our hybrid approach that integrates the two methods. It involves two cycles of engagement resulting in communal meaning-making, followed by two cycles of synthesis and interpretation with a smaller team of evaluation team members with input from select community representatives. All cycles maintain integrity with the data, and all interpretations are validated back with the communities creating iterative cycles of learning and action appropriate to analyzing complex systems.

SCALE stakeholders

There were three key stakeholder groups who participated in the synthesis process. Implementation team stakeholders included IHI staff and coaches responsible for developing SCALE training and tools and supporting community coalitions on their use. Community stakeholders were representatives of community coalitions. Some of them volunteered to participate on the synthesis teams and were actively involved in the meaning making process; others reviewed the data relevant to their communities and provided inputs on their accuracy and validity. There was at least one stakeholder from each SCALE community involved in the process. Evaluation team stakeholders planned and facilitated the synthesis sessions.

SCALE data sources

Data used for the evaluation fell into two main categories: routine data on the progress of SCALE activities, and data collected by the evaluation team. Descriptions of the available data sources and the SCALE activities they aligned with are shown in Table 1.

Engagement Cycles

As shown in Fig. 2, the first two cycles focused on incorporating the context and experience of community members in the analysis process.

Cycle 1: Community level data validation by subgroup: The objective of the first cycle was to use community input to improve the quality of implementation data. To engage community coalition stakeholders in their areas of interest, four subgroups were created, each focusing on one key aspect of the SCALE Theory of Change: Community of Solutions skills, Quality Improvement methods, Spreading and Scaling-up Practices, and Engaging People with Lived Experience. Coalition members were invited to participate in any subgroup they chose, and participation in the process as a whole was voluntary. Each subgroup met virtually for 90 minutes biweekly for 26 weeks, facilitated by an evaluation team member, who reviewed the data, developed a data summary for each community, assessed data quality and identified data gaps. In the subgroup meetings, the communities provided additional details to close data gaps. Summaries were updated after each session and recorded in collaborative software worksheets.

Cycle 2: Cross community synthesis by subgroup: During the second cycle, the subgroup structure was continued to ensure the engagement of community members whose inputs were critical. The subgroups consolidated their data across the communities. For each subgroup, evaluation and implementation team members reviewed the summaries from the first cycle to identify similar and discordant ideas about the key learnings from each subgroup across the communities. These were organized, interpreted, expressed as themes and documented in the collaborative worksheets. In the 90-minute synthesis sessions, the themes were discussed among all stakeholders, and adjustments were made as necessary. Throughout this cycle, stakeholders who developed themes revisited the original data to ensure integrity to the data was being maintained.

In their description of meta-ethnography, Noblit and Hare recommended Schutz’ [15] characterization of first, second and third order constructs to assist with the synthesis. First order constructs are supposed to represent participant views and beliefs. Researchers using published studies have found this challenging, because the studies often document interpretations and not original voices [16]. This was not an issue for us because the participants were directly involved in these cycles, and the generated themes can therefore be considered first-order constructs of the data.

Synthesis Cycles

The final two cycles extended the community-driven work of the first two cycles into higher levels of synthesis.

Cycle 3: Community level synthesis by theme: The subgroup structure adopted for the first two cycles required a significant investment of community and evaluation team time, since four separate meetings needed to be conducted every week. For the next two cycles, a consolidated synthesis team was formed, consisting of a few interested community stakeholders and implementation and evaluation team members. Led by the evaluation team, the synthesis team developed a community-level picture of the transformation process by assembling relevant themes from Cycle 2 into community specific interpretations called concepts. A non-technical definition was developed for each concept and community. Evaluation team members set up calls with community stakeholders who were not part of the synthesis team to validate and fine tune the interpretation of each community’s concepts while ensuring that the changes still maintained fidelity to the data from which the themes were derived. The outputs of this cycle were second order constructs of the data, defined in meta-ethnographic research as the study authors’ interpretation of their data. In our case, these were the interpretations of the synthesis team based on their reflections of the community level data.

Cycle 4: Cross community consolidation of concepts: The fourth cycle of synthesis was conducted entirely by the evaluation team. This synthesis was focused on integrating the second order constructs to develop eight overarching concepts that combine the common community-level concepts into cross-community mechanisms that were adopted by all communities in their transformation efforts. Detailed definitions were created for each overarching concept. Lines of argument, reflecting pathways through which these concepts interacted to facilitate community transformation were created and visually depicted using concept maps and causal loop diagrams [17]. A detailed explanation of these diagrams is presented elsewhere [10]. The overarching concepts are shown in Table 4.

Outputs from the synthesis process – an illustrative example

A summary of the four cycles of synthesis and engagement are presented in Table 2. Table 3 provides an example of outputs of the synthesis process for two communities and one subgroup: Engaging People with Lived Experience (PLE). In Cycle 1, we have shown snippets of data from three program implementation sources: (a) the project management system where program goals are set and progress is measured; (b) transcripts from community calls with their coaches; and (c) an interview with the implementation team during a routine project milestone call. These and data from other sources were reviewed by the PLE subgroup for each community and elaborations and descriptive summaries were generated as shown in Table 3. These summaries highlighted the most salient parts of the data that could be used to answer the research questions. Community stakeholders reviewed the summaries and provided feedback. In Cycle 2, the summaries were aggregated across communities, and themes emerging from various communities were documented. Similar and discordant ideas in the summaries were listed, sorted, and organized. Themes were produced for each subgroup and Table 3 shows examples of themes pertaining to PLE.

In Cycle 3, the themes for each subgroup were compared to the interpretations of community data, and a community concept (second-order construct), which is an overall narrative of each community’s transformation journey, was generated. The PLE component of the community concept for the two selected communities in our example is shown under the “Cycle 3” heading in Table 3. These represent the collective actions and efforts taken by each community to engage PLE, and what this meant for the community’s transformation efforts. Finally, the community concepts were analyzed for common overarching patterns, and the definition of the overall concept related to PLE across all communities is shown under the “Cycle 4” heading. All overarching concepts and their definitions are shown in Table 4.

The approach described in this paper to engage stakeholders to interpret and synthesize community data reflects the increasing realization among public health researchers and practitioners that evaluation of complex community programs requires a shift from narrowly defined effectiveness studies that maximize internal validity to a focus on generalizability and external validity, while recognizing the heterogeneity of local contexts. This requires a balance between identifying features of the program that are transferrable across locations and local adaptations that matter [18]. At this point, because of the separation that exists between community practitioners or program implementers and researchers or evaluators, this dual goal can only be accomplished by partnering across multiple stakeholders via the kind of process described in this paper. This brings together those able to interpret local data with those who are able to guide the identification of generalizable themes that transcend context.

These collaborations are difficult for several reasons. First, because of funding streams and training they are usually researcher or evaluator led, and without careful planning and communication, could easily create the impression that local participation is only solicited to serve an external research agenda. Second, even if the objective of the collaboration is not for research but for joint learning about what works, the emphasis by researchers or evaluators on rigorous validation of the data and a process of critical inquiry can be experienced as judgmental by some stakeholders, resulting in a barrier to the open communication needed for joint meaning making.

In the SCALE evaluation, the evaluation team had the benefit of having worked with the communities and the implementation team for four years, initially in a formative evaluation role that supported communities within real time through their process of implementation [19]. Evaluation team members had personal relationships with the community stakeholders, and, therefore, a degree of trust had been established over time. Additionally, a lot of careful planning and preparation went into both the design and the execution of the process to ensure that the locus of the evaluation was centered around the communities and that their ownership and knowledge of their data was recognized and appreciated.

In order to forestall any perception that the data of some communities were favored over others, our approach explicitly designed the first two cycles of synthesis to create subgroups that focused on the common components of the theory of change across communities. Community specific interpretations were not undertaken until the third cycle by which time most stakeholders were comfortable with the synthesis process, and thematic consensus on the subgroups data had been achieved. This design also made it possible to begin conversations about generalizability early, since the second cycle focused on cross community themes.

On the execution side, each cycle was preceded by communication sessions to explain the goals for the cycle, and community members were encouraged to provide suggestions about the methodology. Each synthesis meeting began with an update of progress to date to enable members who had not attended previous meetings to be caught up. All templates used for synthesis were created using collaborative group software, were extensively tested with community and implementation stakeholders and were editable by everyone irrespective of whether they participated in the synthesis meetings. Recognizing the wide variation in evaluation expertise across the stakeholder groups, roles for community members were clearly established and agreed upon so that each team member could bring their own area of expertise into the synthesis process. Stakeholders were informed of the time commitment needed in advance, but competing priorities made it difficult for several of them to continue in the group meetings, necessitating follow-up conversations that consumed additional time and resources. Overall, the evaluation team significantly underestimated the time and labor required to manage the synthesis cycles to assure adequate data quality, community input and overall engagement.

Some of the extensive effort needed was due to the intrinsic heterogeneity across the SCALE communities that required piecing together disparate data to elicit both community specific and generalizable insights. The varying quality and heterogeneity of the data and the need to shore up gaps in data through supplementary collection was another major factor that contributed to additional time and effort. Even though each SCALE community was tackling a different health topic, they followed a common theory of change and were all provided with common tools for data collection. For a variety of reasons beyond the scope of this paper, some of the data collection tools were felt by the communities to be burdensome and not valuable, and, therefore, overall community compliance with the data collection recommendations was poor.

To address these considerations, future attempts at participatory evaluation of complex community or public health interventions would benefit from promoting the salience of these approaches during implementation and not just after. As mentioned previously, complex interventions require an iterative testing process; building the capacity of program implementers or community coalitions to develop systems for routine cross-sector data collection, and to conduct synthesis cycles within their communities or program sites in partnership with community members, could encourage improved data collection and use during implementation. Further, this approach could also prepare the data required to answer broader evaluation questions about external validity and generalizability at the end of project, without the need to start from scratch. Moreover, since a key goal of the participatory synthesis is to encourage meaning making by those who are closest to the data, building evaluation capability within community coalitions enables them to engage their community members in evaluating the effects of transformation efforts such as SCALE. Adopting this “empowerment evaluation” approach ensures both that the power of interpretation is situated within communities and that high-quality data is available for both local and higher order decision making [20].

One aspect of the plan of action under the Healthy People 2030 framework, approved by the Department of Health and Human Services in 2018, is to provide accurate, timely and accessible data to enable action to address regions with poor health or at high risk. Another aspect is to provide tools to the public and other stakeholders to evaluate progress related to the achievement of health and wellbeing. Data and tools are a good start, but are inanimate without people embedded in communities who have the ability to collaboratively gain insights from the data about how to improve. This requires processes for systematic data analysis and for bringing diverse community members together to enrich the analysis with contextual interpretations so that both community specific and generalizable change ideas can be created. In this paper, we have described an innovative evaluation methodology that achieves both these aims by blending two qualitative research methods: participatory action synthesis and meta-ethnography and adapting them for use in practice. Our efforts resulted both in an identification of improvement pathways followed by individual communities and general principles of community transformation. This methodology has promise for learning how to advance the aims of large-scale health and well-being initiatives domestically and globally. Researchers and practitioners interested in using this approach should be willing to take the time and put in the effort, should be clear and transparent about the process, should assign roles to all stakeholders to encourage participation and shared ownership, and should co-design and test data synthesis tools prior to implementation.

100MLives

100 Million Healthier Lives

CHILA

Community Health Improvement Leadership Academies

FUPS

Flawed, Uncertain, Proximate, and Sparse

IHI

Institute for Healthcare Improvement

PLE

People with Lived Experience

Quality Improvement

SCALE

Spreading Community Accelerators through Learning and Evaluation

Ethics approval and consent to participate: Not applicable

Consent for publication: Not applicable

Availability of data and materials: Available from the corresponding author on request.

Competing interests: The authors declare that they have no competing interests.

Funding: This study was funded by the Robert Wood Johnson Foundation (RWJF) [grant numbers 74438, 76406]. RWJF had no role in the design of the study and collection, analysis, and interpretation of data or in writing and preparing the manuscript.

Authors contributions: KR led the synthesis of data, developed initial drafts and readied manuscript for publication. TC, RSR, JC, BSC, and PH were involved in data collection and contributed to the synthesis of data. CEC assisted in data synthesis. RR developed evaluation design, led data synthesis, developed the manuscript structure, and revised manuscript drafts. All authors reviewed, provided comments, and approved final manuscript.

Acknowledgements: Not applicable

Tikkanen R, Abrams MK. U.S. Health Care from a Global Perspective, 2019: Higher Spending, Worse Outcomes? Commonwealth Fund. 2020. https://www.commonwealthfund.org/publications/issue-briefs/2020/jan/us-health-care-global-perspective-2019 Accessed 30 Jan 2020.
Wolff T, Minkler M, Wolfe SM, Berkowitz B, Bowen L, Butterfoss FD, et al. Collaborating for Equity and Justice: Moving Beyond Collective Impact. Nonprofit Q. 2016;23(4):42-53.
DeSalvo KB, Wang YC, Harris A, Auerbach J, Koo D, O'Carroll P. (2017). Public Health 3.0: A Call to Action for Public Health to Meet the Challenges of the 21st Century. Prev Chronic Dis. 2017;14:E78.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions. Medical Research Council Guidance. 2006. https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ Accessed 30 Jan 2020.
Lipsitz LA. Understanding health care as a complex system: the foundation for unintended consequences. JAMA. 2012;308(3):243-4.
Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.
Snowden DJ, Boone ME. A leader’s framework for decision making. A leader’s framework for decision making. Harv Bus Rev. 2007;85(11):68-76, 149.
Wolpert M, Rutter H. (2018). Using flawed, uncertain, proximate and sparse (FUPS) data in the context of complexity: learning from the case of child mental health. BMC Med. 2018;16(1):82.
Network for Improvement & Innovation in College Health. Community of Solutions Framework. n.d. https://collegehealthqi.nyu.edu/20x30/frameworks/community-of-solutions-framework/ Accessed 6 Jan 2020.
Carr T, Reed K, Rabbani R, Chandler C, Cook B, Henry B, et al. How Communities Build Capacity for Transformation: Using Meta-Ethnography to Explore Pathways and Dynamics. 2020. Manuscript in preparation.
Noblit GW, Hare RD. Meta-ethnography: Synthesizing qualitative studies. SAGE; 1988.
Major CH, Savin-Baden M. An introduction to qualitative research synthesis: Managing the information explosion in social science research. New York: Routledge; 2010.
Wimpenny K, Savin-Baden M. Exploring and Implementing Participatory Action Synthesis. Qual Inq. 2012;18(8):689-98.
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: A review of possible methods. J Health Serv Res Policy. 2005;10(1):45-53.
Schultz A. Collected papers 1. The problem of social reality. Boston: Martinus Nijhoff; 1972.
Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol. 2008;8:21.
Dhirasasna N, Sahin O. A Multi-Methodology Approach to Creating a Causal Loop Diagram. 2019;7(3):42.
Leviton LC. Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity. Annu Rev Public Health. 2017;38:371-91.
Scott VC, Alia K, Scaccia J, Ramaswamy R, Saha S, Leviton L, et al. Formative Evaluation and Complex Health Improvement Initiatives: A Learning System to Improve Theory, Implementation, Support, and Evaluation. Am J Eval. 2020;41(1):89-106.
Wandersman A, Snell-Johns J, Lentz BE, Fetterman DM, Keener DC, Livet M, et al. The principles of empowerment evaluation. In Fetterman DM, Wandersman A, editors. Empowerment evaluation principles in practice. New York: Guilford Press; 2005. p. 27-41.

Table 1

Title of table: SCALE Data Sources

SCALE Activity	Source Name	Source Description
Project Planning	Community Transformation Map	Self-assessment tool used quarterly by the coalitions to assess motivation and capability for change.
	Chart Your Path	Tool used by communities to develop their improvement plans.
	Driver Diagram	Visual documentation of theory of change.
	Monthly & Quarterly Progress Reports	Reports on goals, progress, future planning, and scale-up effort.
	Quarterly Vital Signs	Tool used quarterly by communities assess the health of their coalition (i.e. their core health transformation team).
	Racism & Inequity Plan	Documentation on community actions to address racism and inequity
	Meaningful Measurement Meeting	Notes from a one-day dedicated meeting conducted with each community to develop a customized measurement plan.
	Measure Focus Group	Focus group on each community’s measurement challenges, readiness, and sustainability conducted during the meaningful measurement meeting.
Coaching	Coach Calls	Interview with coaches about the coaching process and its effectiveness.
Progress Monitoring	Milestone Calls	Calls conducted by the implementation team quarterly to review the progress reports and plans for the coming quarter.
National and Regional Capacity Building	National Community Health Improvement Leadership Academies (CHILA) Surveys	Evaluations conducted at the end of each session and at the conclusion of the four national CHILAs, which were 3-5 day capacity building workshops.
	Regional CHILA Surveys	Evaluations conducted at the end of regional community CHILAs. These were 1-2 day capacity building workshops conducted by the communities in their own regions.
	Regional CHILA Focus Groups	Deep dive focus groups conducted with a few selected communities after they conducted their regional CHILAs about their ability to use and disseminate what they had learned through SCALE.
Scale-up Planning	Scale-up Interviews	Interviews conducted with a few selected communities about the activities, process, successes, and challenges related to scaling up the SCALE activities to other communities in the region.

Table 2

Title of table: Summary of Engagement and Synthesis Cycles

Cycle	Participants	Focus	Outputs
1	· Implementation team stakeholders · Community stakeholders · Evaluation team stakeholders	Community-level data validation by subgroup	Data summaries for each community
2	· Implementation team stakeholders · Community stakeholders · Evaluation team stakeholders	Cross-community synthesis by subgroup	First-order interpretations expressed as themes
3	· Implementation team stakeholders · Community stakeholders · Evaluation team stakeholders	Community-level synthesis by theme	Second-order interpretations expressed as community concepts
4	· Evaluation team stakeholders	Cross-community consolidation of concepts	Third-order interpretations expressed as overarching concepts

Table 3

Title of table: Example of Synthesis Process: Sustained Engagement of Community Members and People with Lived Experience

CYCLE 1
Raw data source	COMMUNITY A			COMMUNITY B
	Project Management System	Coaching Call Transcript Data	Milestone Completion Interview	Project Management System	Coaching Call Transcript	Milestone Completion Interview
Raw data type	Goals and Plans for "Meaningful Engagement of Residents Most Affected by Inequities"	Report to coaches about engagement activities	Answer to Interview Question: How do you describe the results your SCALE community has achieved by engaging people with lived experience, including community champions, in your work?	Goals and Plans for "Meaningful Engagement of Residents Most Affected by Inequities"	Report to coaches about engagement activities	Answer to Interview Question: How do you describe the results your SCALE community has achieved by engaging people with lived experience, including community champions, in your work?
Descriptive summaries	The community has people with lived experience in their coalition; includes people with lived experience throughout the planning process; and has utilized input from people with lived experience to guide strategic direction of park network and partnership	Their community champion (CC) meets, discusses, and plans with the community. They integrated people with lived experience in their Quarterly community meetings where they gather input for planning process.	The community is developing “Friends of Groups” for all parks. They utilized input from people with lived experience to guide the strategic direction as a park network and partnership.	They have integrated community members in their work. They trained CHIP implementation members on how to increase engagement in community and host listening sessions	Specificity has helped them identify suitable Community Champion for their work	"We have included people with lived experience in the Health Equity Action Labs and we were also able to provide an incentive... They were paid to provide input and be a liaison to the schools to inform the process and PDSAs each week to better connect folks to food."
CYCLE 2
Engaging People with Lived Experience (PLE) Themes	· Recruitment strategies for intentional targeting of community champions · People with lived experience, including community champions, co-design intervention with the community · Feedback was solicited in an effort to engage the community and better understand their wants/needs · Growing leadership of people with lived experience, including community champions, by training them as new leaders, and providing professional development and/or mentorship opportunities · Community engaged and involved youth/adolescents as people with lived experience in their work
CYCLE 3
Engaging PLE Concepts	COMMUNITY A			COMMUNITY B
	The process of bringing people with lived experience happened early on. People with lived experience helped to guide the strategic direction of the community's improvement efforts			Youth were integrated into the community's transformation work as leaders and people with lived experience.
CYCLE 4
Engaging PLE Overall Concept	The community engages people with lived experience in a number of roles, including as community champions, project leaders, trainers, organizers, key informants and participants throughout the course of the change process

Table 4

Title of table: Overarching Concepts and Operational Definitions

Concept Label	Concept Definition
Applying a theory of change to guide community efforts	The community first develops and then applies an explicit theory of change (TOC), whereby it conceptualizes specific ideas needed for change to direct its efforts towards community health and well-being improvement, create a transformational plan, and spread effective strategies to other communities.
Embedding people with lived experience into transformation work	The community engages people with lived experience in a number of roles, including as community champions, project leaders, trainers, organizers, key informants and participants throughout the course of the change process.
Building capabilities for community change by identifying and growing leaders	The community builds capability of community members to address complex community structural issues that are barriers to community well-being.
Building the capability of the core team engaged in transformation to engage in peer learning	A community works with partners as a coalition to more effectively direct its improvement efforts. Partners include people that have intimate knowledge of and/or experience in the community as residents, advocates, or through community-based organizational affiliations.
Creating access to those with specialized knowledge (e.g. in QI) for coaching and technical assistance	The community is proactive and intentionally uses support from specialists with topic specific and community-relevant knowledge.
Creating the atmosphere for authentic dialogue within and between communities	The community leaders develop relationships and engage community members to create space for, and improve ability to have, difficult or sensitive conversations.
Facilitating the formation of personal relationships and social connections across coalitions	The community forms personal relationships with peer communities and provide and receive support to one another to discuss and problem-solve common community challenges.
Explicitly and intentionally addressing racism and inequity within the community	The community makes efforts to identify and address the systems, policies and practices working within the community that reinforce structural racism and contribute to disparities and inequities.

Download PDF

Journal Publication

published 17 Nov, 2021

Read the published version in Journal of Participatory Research Methods →

Version 1

posted

You are reading this latest preprint version

Blending Participatory Action Synthesis and Meta-ethnography: An Innovative Approach to Evaluating Complex Community Health Transformation

Status:

Journal Publication

Version 1

Abstract

Figures

Contributions To The Literature

Background

Addressing Complexity in Public Health

The Need for New Evaluation Methods

Background – The SCALE Initiative

SCALE Theory of Change

SCALE Communities

Methods

Evaluation Questions

Evaluation Approach

SCALE stakeholders

SCALE data sources

Engagement Cycles

Synthesis Cycles

Results

Outputs from the synthesis process – an illustrative example

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Status:

Journal Publication

Version 1