The present results describe the seven phases of the framework used to identify, model and possibly refine the case of local innovation, in addition to planning for potential scale up for further feasibility and outcome testing (Figure 1). These phases are not intended to be sequential but iterative and not necessarily sequential.
- Identification of Innovative Local Practice
The first phase is to identify innovative local practice that warrants progression to modelling, possible refinement and outcome evaluation. The researchers identified the case study intervention through the DECIPHer research centre hosted School Health Research Network infrastructure (SHRN) (14). The network comprised 165 of all secondary schools in Wales (N=212) at the time of study, with representation from all 22 local authority areas. The Network seeks to optimise research collaboration between researchers, policy-makers and practitioners. One of the central mechanisms to encourage collaborative working is through a programme of knowledge exchange activities, including webinars and stakeholder meetings. At regional meetings, researchers present study data whilst practitioners share examples of innovative practice to improve staff and student health and wellbeing. The case study innovation had been presented at a stakeholder event, with the school gatekeeper following up the potential for research collaboration with the SHRN Manager. The Manager identified a relevant academic contact with the requisite expertise to assess the fit of the intervention with the centre’s research priorities, formulate preliminary research questions, consider an appropriate research design, and draw together a research team.
Given the characteristics of the local innovation and its history of implementation in the school, a pragmatic formative progress evaluation was decided upon. Criteria for informing this decision was: 1) Feasibility of programme theory modelling: The researchers questioned if an “intervention” (regardless of type) was in use and that a programme theory, contextual characteristics, implementation and outcomes could be characterised. The school had been recognised as delivering sector-leading, best practice in restorative practice and had been awarded a Restorative Service Quality Mark (RSQM) in 2010. As a consequence of this external validation the researchers felt that there was clear delivery of a restorative practice intervention . 2) Feasibility of implementation and scale-up: The researchers established that the restorative practice had been routinely used and resourced for a substantial period of time (i.e. eight years). The researchers further considered the future traction of the intervention and if it could be scaled-up for evaluation beyond the single case study schools, or was so contextually contingent no replication was feasible. There was no indication that the school was atypical so the intervention could not be transported to other secondary schools, and the school had been increasingly invited to share their practices with other schools at a national level due to being recognised as sector leading; 3) Research Co-production: The researchers consulted with the school to ensure they were prepared to participate in a research study and would potentially be committed to future research.
- Scoping review to identify programme theory; outcomes; contextual characteristics that influence programme theory and implementation
The second phase is to engage in a scoping or systematic review of the existing scientific research to develop a preliminary understanding of the intervention. This can inform the development of an initial logic model, which can serve as the basis to model the real-world case example. A review further supports consideration of the effects of such interventions, and potential iatrogenic pathways that might be attended to in the primary research (15).
Programme Theory and Outcomes: Across the studies there was a lack of specificity around the underpinning programme theory. Rather there were broad principles of how restorative approaches may work, largely through the building, maintaining and restoring of relationships, where individuals take responsibility for their actions and positively engage in relationship repair and conflict resolution (10, 11). This may be further supported by changes in classroom management practices and school ethos. The INCLUSIVE intervention provides one of the most theoretically informed approaches (12, 18), hypothesising that through restorative practices students are more likely to engage with schools’ pedagogic practices and embrace rules and ethos. As a result, school connectedness increases and relationships improve. A range of activities at the targeted, universal and whole-school level can be considered as restorative. The approach may be most effective when it is fully adopted at the system level (19, 20).
Evaluations of school based restorative approaches have identified a range of measurable intervention outcomes (12, 19, 21, 22). At the student level these include improving mental health and wellbeing (12), social and emotional competencies, including empathetic attitudes and self-esteem (23), improved academic attainment (22), reduced bullying (23) and fewer school exclusions (21, 22). There has been limited consideration of staff level outcomes and iatrogenic pathways remain largely underdeveloped.
Contextual characteristics that influence implementation and programme theory: The researchers mapped key system-level characteristics that might inhibit the programme across different contexts and prevent planned implementation. The Context and Implementation of Complex Intervention Framework (CICI) (24) was used as a framework for mapping context and implementation. As these are often not fully considered when modelling interventions, Table 2 provides a worked example of how they were understood based on the evidence-base. Although existing research findings did not map onto all of the CICI domains, a number of influences emerged across papers. Epidemiological: Implementation is strengthened by an increase in the prevalence of bullying within the specified context, leading to more support for such approaches (22, 23). Political: There is increased support for restorative approaches where there is alignment with political/policy priorities, which has often led to direct government funding (12, 19, 21, 25). Ethical: Restorative approaches are congruent with a belief in a fair and just society where citizens are respected. In such cirumstances they are viewed as a more ethical approach to punitative or criminalised responses (21). {Wong, 2011 #15}
- Establishment of a TransDisciplinary Action Research Group (TDAR)
The third phase is to establish a TDAR Group, which is intended to support the effective collaboration between diverse stakeholders (26). Within the existing literature, this approach emphasises co-production where non-academics are active agents in research and strives for equal, mutually beneficial and reciprocal relationships that value public, practitioner and policy-maker knowledge and experience to the same degree as academic knowledge (27). It is underpinned by the principles of action research, and its tenets has been increasingly deployed in guidance around intervention development to ensure that approaches are maximally responsive to the contexts and populations where they are to be implemented (1). While dominant terminology uses the term transdisciplinary, it may be more useful to think more in terms of creating a trans-professional group that draws together different professional identities, such as policy-makers, local education authority representatives, researchers, teachers and of course students. Within a pragmatic formative process evaluation, TDAR can help to bring a comprehensive and nuanced understanding of the intervention that is being modelled, in addition to a rich awareness of the context in which it has been originally delivered.
A Transdisciplinary Action Research Group (TDAR) comprising diverse researchers (i.e. sociology, public health, psychology and epidemiology) and members of the school community who were on the Senior Management Team. The group comprised eight members. It should be noted that students were not represented, meaning that their perspective was only accommodated during the research. Future studies should better represent the target population in the TDAR group. The group met routinely throughout the duration of the study. Its function was to oversee study conduct, ensure that the study design and processes were being shaped by practice perspectives, support the development of an initial candidate logic model and to build relationships to support knowledge translation. It further aided the decision-making about future evaluation (Phase 7), where stakeholders could share views on the value of information from an outcome evaluation and the different types of evidence that would support practice moving forward.
- Co-production of Intervention Logic Model with Stakeholders
The fourth phase is the co-production of a logic model with key stakeholders to identify the underpinning programme theory, contextual characteristics, implementation practices and outcomes. Participants developed the logic model from the initial construction undertaken by the TDAR group following the scoping review. The Wisconsin template was used (28). The output of the logic model from both Phase 4 and Phase 5 is presented Table 3. A more detailed consideration of context and implementation, as mapped across the CICI framework, is presented in Table 4.
Programme theory: Both staff and students stated confidence and self-efficacy as being important to the programme theory. Students spoke about feeling between equipped to take ownership of their learning, ask for help, and take risks with complex topics, which was largely a consequence of involvement in classroom and school-level decision making. Meanwhile staff suggested that improvements in confidence in the classroom, combined with having the opportunity and skills to express their thoughts and feelings following student conflict, had reduced stress:
STAFF FG1;3: So … it certainly has made a difference in terms of my wellbeing, giving me more confidence within the classroom … it’s not just looking after student wellbeing, but also staff wellbeing.
The central mechanism for both of these groups of stakeholders was a change in relationships. Students mentioned peer relationships frequently, while staff emphasised relationships between staff and students: In the later instance, one member of staff suggested that circle time redresses power imbalances, creating more supportive interactions:
STAFF FG2; 14- …the starting with them … with them was to sort of have a circle time in and listen to them. Find out what they need from me and let them know what I need from them. Erm, and just … just not being afraid really to sort of break down any barriers between sort of thoughts and feelings …
Through a shift towards trustworthy and responsive relationships, the school was considered to offer a more positive and supportive culture. These changes led to students experiencing increased school connectedness. This process was further enhanced through a distributed leadership model, involving students in key decision making, such as the design of a new building or appointment of a staff member, with one commenting ‘we’ve had a huge impact with everything in school.’
Additionally, students felt that the instilling of restorative practices had improved the school’s reputation in the community, and relative to other local schools. This had enhanced school connectedness and thus motivation to engage in positive behaviours and improve academic attainment:
STUDENT FG1;5: Because when I first came to the school, … we were known as “down the hill” and now it’s “the comp”. Like things have changed. …
Beyond intended causal pathways, participants considered iatrogenic pathways, which have largely been overlooked in the previous modelling of restorative approaches. This identification illustrates the particular strength of co-production and learning from interventions already in routine practice. For example, participants indicated that the school’s improved reputation following adoption of the intervention had led to over-subscription, which had limited access in the community and placed a resource burden on the school.
Outcomes: Logic modelling indicated three key sets of outcomes, which are largely congruent with existing restorative approaches. For both student and staff the reported primary outcome was improved student mental health and wellbeing:
STUDENT FG1; 2- I think wellbeing in the school is kind’ve increased massively ..,I’ve got a brother who is 5 years older than me but he came to this school as well and he’s told me stories about how there used to be fights every week and people would set off fire extinguishers... then you look at our school now and honestly I’d be surprised if I heard about a fight because it just doesn’t happen anymore...(laughs) yeah it’s not common any more. I think generally school life has transformed and everything is more positive now. I rarely hear people talk badly about teachers um, everything here seems to be more positive and I think that contributes to all the points these guys have brought up about feeling secure and happy in the environment.
Additional outcomes are presented in Table 3.
Contextual characteristics that influence implementation and programme theory: Drawing on the factors identified in the scoping review, the co-production process explored key contextual features that could support the implementation of the intervention and ensure the programme theory worked as intended. These factors were often explained in relation to the reason why restorative practices were initiated.
Epidemiological: Data indicated that the school had reached a tipping point, and preparedness for change was due to perceptions of increasingly poor levels of mental health and wellbeing and high levels of fixed-term and permanent exclusions. Existing practices based on merit and punishment were considered punitive and ineffectual in addressing the problem:
STAFF FG1; 2: …we were just finding we were going round and round and round in circles and not really making progress.
Political: The policy context in Wales was increasingly orientated to support the prioritisation of mental health and wellbeing of children and young people, particularly within the educational context. The Well-being of Future Generations Act (2015) in Wales has mandated organisational and culture change to enhance metal health related outcomes. Meanwhile the Donaldson educational review on curriculum reform has outlined six key priorities, such as wellbeing, alongside an acknowledgement of the synergy between wellbeing and educational outcomes (Donaldson, 2016). Although in the case study school, restorative practice had been implemented for 8 years prior to data collection in 2016, and advance of these political and educational changes, these policy priorities supported its continued implementation. Socio-economic: Participants acknowledged that the case study school had a lower than national average level of free school meal entitlement and a high level of academic achievement. Thus, whilst the school cannot necessarily be characterised as atypical, there was acknowledgment that the intervention may be more difficult to implement in a more challenging context with higher levels of disadvantage:
STAFF FG1; 6 – I think there’s more focus on students’ perspectives here um, which students value more. Generally the behaviour here is better than at schools that I’ve taught at previously, though I’d say those schools are working within a different concepts, there are inherently gonna be more issues because of the intake that they have.
Socio-cultural: Participants identified entrenched pedagogic practices that were the antitheses of restorative approaches, namely punitively orientated interactions with students. There was indication that staff could orientate to the default approach, which could lead to extensive variation in practice:
STAFF FG1;2 – varied yeah, it is varied across the school: you can see a restorative conversation happening in quite a negative tone in one space, but in another it can be very effective so...and that’s hard for young people as well because young people say “I’ve just had a restorative” (said in an angry voice) and actually it’s like hang on a second, that’s not a restorative
Participants also suggested potential incongruence between the social and emotional competencies required for the effective delivery of a restorative approach, and a socio-cultural context that did not always privilege vulnerability and emotional openness. To mitigate against such issues, participants identified the importance of senior leadership vision and commitment as part of the implementation plan to ensure realignment of the school ethos with the restorative practice approach and staff commitment to training and delivery. Moreover, the school adopted a rather organic diffusion process, initially securing training to a small team of pastoral staff to ensure their buy in and capacity for modelling the approach before expanding to more diverse professional roles. Eventually working groups were established to ensure continued change to the socio-cultural context, with a Behaviour Research Group reviewing how the restorative practices could be sensitively translated into the setting.
- Confirmation of Logic Model with Stakeholders
The fifth phase of the process entails confirmation of the logic model with stakeholders. Commonly studies present logic model development as a static phase, but to ensure meaningful co-production multiple opportunities for input are required. The second round of data collection with participants provided clarity on a number of uncertainties that remained following Phase 4 and elicited aspects of the intervention and context that had not yet been identified. In particular, participants focused on the socio-ecological domains beyond the inter-personal, notably family and community level processes. For example, family-based activities emerged, particularly the delivery of parenting skills, to ensure some congruence between the school ethos and family dynamic:
STAFF FG2;13: We’re working with parents on the approach we would take in school particularly where children have reflected and said ‘well if I did that at home this is what would happe’n …or this is what I see at home. And that ongoing communication and collaboration with parents is really important and it’s quite a long journey for some.
Taken together, Phase 4 and Phase 5 provided a nuanced and contextually sensitive understanding of the local innovation.
- Planning for Intervention Refinement
The sixth phases progresses to planning for intervention refinement where required. A knowledge exchange event was hosted at the school in order to feedback the study findings and discuss the logic model generated during Phase 4 and 5 (Figure 2). The purpose and impacts of the meeting were fourfold. First, it aimed to provide an additional opportunity to discuss the intervention and verify that the research team who interpreted the data, and the TDAR group who supported this process, had an adequate understanding of the intervention. Second, by highlighting remaining areas of uncertainty or challenges with delivering the intervention, stakeholders were able to identify where further intervention refinement was required and where barriers needed to be addressed to ensure that proposed the programme theory was being activated as intended. Third, the event served to strengthen partnership between stakeholders. Fourth, it aimed to reassert the emotional investment of the school (29). To progress to further evaluation, where the school may be required to support the sharing and delivery of practices within other institutions, it was deemed important for the school feel committed to both the intervention and research. Reflecting with stakeholders provided a positive experience that renewed enthusiasm, with many commenting on how much the school had achieved since the initial introduction of the intervention.
A central issue to reflect upon at this stage, is the potential for different stakeholder groups to have different perspectives on what the refinements are required. In the present case example, there were no significant disagreements and discussion generally focused on removing implementation barriers. However, this issue may arise and the processes for resolving potential conflict needs further consideration.
While the co-production literature is important in recognising the need to take account of all perspectives, the notion of functional fidelity may have practical use in helping accommodate a range of different needs and understandings. This recommends that changes to interventions can be permitted and activities can be flexible as long as the central programme theory is being activated (6, 30). In refining an intervention then, the focus may be on expanding activities or providing different versions of components for different stakeholders in accordance with their needs. The extent to which these refinements support the programme theory can guide what is permitted.
This approach does not come without its own challenges, as they may not be a single programme theory in operation, and different stakeholders can have different interpretations of how an intervention works. Hence achieving modifications to ensure the programme theory is activated can be problematic, and in this case there may need to be several iterations of modelling and refinment to reach an understanding of a theory that most closely approximates the mechanisms that are being activated in the context.
- Planning for Outcome Evaluation
The seventh phase comprises planning for future outcome evaluation if appropriate. Where outcome evaluation is warranted, the type of evaluation would be most suitably assessed against the phases of evaluation prescribed by the MRC: pilot and feasibility trial, a randomised controlled trial; natural experiment or other quasi-experimental design; and then longer-term implementation evaluation (3, 31). Further work is required to refine decision-making about the most suitable evaluation approach, and an a priori progression criteria similar to that used in feasibility trials may be helpful. Potential criteria to be considered are: 1) The evaluability of the intervention(32); 2) The Value of Information (VOI), which weighs the cost of obtaining evidence against the need for certainty amongst stakeholders (33); and 3) the applicability of the existing evidence. For example, Aarons et al. have developed a framework for ‘borrowing evidence’, which assesses the similarities of different interventions and contexts to see if the outcomes of evaluations conducted elsewhere have relevance in the new context in question(34).
In the present case study, planning is primarily being conducted through the TDAR group, drawing on both practice and academic expertise. The SHRN infrastructure offers a particular opportunity to continue with pragmatically orientated innovation evaluation, through the conduct of a pragmatic feasibility and outcome trial. As of 2020 the network includes 100% of the 212 state-funded schools in Wales, providing a complete sample frame for randomisation. A sample of students at each participating school complete bi-annual surveys of their health and wellbeing, and provided data is collected at appropriate times, these surveys could be exploited as the data source for outcome measurement. As popular innovations, such as that selected for the case study, are gaining traction within systems, it is imperative that we have responsive study designs. Use of routine data, such as that collected through the SHRN survey data offers such responsivity, although the evidence generated is arguably less scientifically robust than that provided by RCTs.