The present results describe the six phases of the framework used to identify and model the case of local innovation, in addition to planning to optimise delivery and conduct future feasibility and outcome testing (Figure 1). These phases are not necessarily sequential and may require repeating a number of times.
- Identification of Innovative Local Practice
The first phase is to identify innovative local practice that warrants progression to modelling and possible outcome evaluation. The researchers identified the case study intervention through the DECIPHer research centre hosted School Health Research Network infrastructure (SHRN) (14). The network comprised 165 of all secondary schools in Wales (N=210) at the time of study, with representation from all 22 local authority areas. The Network seeks to optimise research collaboration between researchers, policy-makers and practitioners. One of the central mechanisms to encourage collaborative working is through a programme of knowledge exchange activities, including webinars and stakeholder meetings. At regional meetings, researchers present study data whilst practitioners share examples of innovative practice to improve staff and student health and wellbeing. The innovative practice was presented at a stakeholder event, with the school gatekeeper following up the potential for research collaboration with the SHRN Manager. The Manager identified a relevant academic contact with the requisite expertise to assess the fit of the intervention with the centre’s research priorities, formulate preliminary research questions, consider an appropriate research design, and draw together a research team.
Given the characteristics of the local innovation and its history of implementation in the school, a pragmatic formative progress evaluation was decided upon. Criteria for informing this decision was: 1) Feasibility of programme theory modelling: The researchers questioned if an “intervention” (regardless of type) was in use and that a programme theory, contextual characteristics, implementation and outcomes could be characterised. The school had been recognised as delivering sector-leading, best practice in restorative practice and had been awarded a Restorative Service Quality Mark (RSQM) in 2010. As a consequence of this external validation the researchers felt that there was clear delivery of a restorative practice intervention. 2) Feasibility of implementation and scale-up: The researchers established that the restorative practice had been routinely used and resourced for a substantial period of time (i.e. eight years). The researchers further considered the future traction of the intervention and if it could be scaled-up for evaluation beyond the single case study school, or was so contextually contingent no replication was feasible. There was no indication that the school was atypical so the intervention could not be transported to other secondary schools, and the school had been increasingly invited to share their practices with other schools at a national level due to being recognised as sector leading; 3) Research Co-production: The researchers consulted with the school to ensure they were prepared to participate in a research study and would potentially be committed to future research.
- Scoping review to identify programme theory; outcomes; contextual characteristics that influence programme theory and implementation
The second phase is to engage in a scoping or systematic review of the existing scientific research to develop a preliminary understanding of the intervention. This can inform the development of a programme theory, which can serve as the basis to model the real-world case example. A review further supports consideration of the effects of such interventions, and potential unintended causal pathways that might be explored in the primary research (12).
Causal Mechanisms and Outcomes: Across the studies there was a lack of specificity around the underpinning programme theory. Rather there were broad principles of how restorative approaches may work, largely through the building, maintaining and restoring of relationships, where individuals take responsibility for their actions and positively engage in relationship repair and conflict resolution (10, 11). This may be further supported by changes in classroom management practices and school ethos. The INCLUSIVE intervention provides one of the most theoretically informed approaches (12, 17), hypothesising that through restorative practices, students are more likely to engage with schools’ pedagogic practices and embrace rules and ethos. As a result, school connectedness increases and relationships improve. A range of activities at the targeted, universal and whole-school level can be considered as restorative. The approach may be most effective when it is fully adopted at the system level (18, 19).
Evaluations of school based restorative approaches have identified a range of measurable intervention outcomes (12, 18, 20, 21). At the student level these include improving mental health and wellbeing (12), social and emotional competencies including empathetic attitudes and self-esteem (22), improved academic attainment (21), reduced bullying (22) and fewer school exclusions (20, 21). There has been limited consideration of staff level outcomes and unintended causal pathways remain largely underdeveloped.
Contextual characteristics that influence implementation and programme theory: The researchers mapped key contextual characteristics that might influence the activation of the programme theory and impact planned implementation. The Context and Implementation of Complex Intervention Framework (CICI) (23) was used as a framework for mapping context and implementation. Table 2 shows how the domains of the Framework were populated from the evidence-base. Although existing research findings did not map onto all of the CICI domains, a number of influences emerged across papers. Epidemiological: Implementation is strengthened by an increase in the prevalence of bullying within the specified context, leading to more support for such approaches (21, 22). Political: There is increased support for restorative approaches where there is alignment with political/policy priorities, which has often led to direct government funding (12, 18, 20, 24). Ethical: Restorative approaches are congruent with a belief in a fair and just society where citizens are respected. In such cirumstances they are viewed as a more ethical approach to punitative or criminalised responses (20). {Wong, 2011 #15}
- Establishment of a TransDisciplinary Action Research Group (TDAR)
The third phase is to establish a TDAR Group, which is intended to support the effective collaboration between diverse stakeholders (25). The model of TDAR strives for equal, mutually beneficial and reciprocal relationships that value public, practitioner and policy-makers knowledge and experience to the same degree as academic knowledge (26). Therefore, while dominant terminology uses the term transdisciplinary, it may be more useful to think in terms of creating trans-professional models of practice. t is underpinned by the principles of action research, and its tenets has been increasingly deployed in guidance around intervention development to ensure that approaches are maximally responsive to the contexts where they are to be implemented (1). Within a pragmatic formative process evaluation, TDAR can help to bring a comprehensive and nuanced understanding of the intervention that is being modelled, in addition to a rich awareness of the context in which it has been originally delivered.
A TDAR group was set up including researchers from different disciplines (i.e. sociology, public health, psychology and epidemiology) and members of the school community who were on the Senior Management Team. The group comprised eight members. It should be noted that students were not represented, meaning that their perspective was only accommodated during the research. Future studies should better represent the target population in the TDAR group. The group met routinely throughout the duration of the study. Its function was to oversee study conduct, ensure that the study design and processes were being shaped by practice perspectives, support the development of a candidate programme theory and to build relationships to support knowledge translation. It further aided the decision-making about future evaluation (Phase 6), where stakeholders could share views on the value of information from an outcome evaluation and the different types of evidence that would support practice moving forward.
- Co-production and Confirmation of Programme Theory with Stakeholders
The fourth phase is the co-production and confirmation of the programme theory, and associated logic model, with key stakeholders to identify the underpinning causal mechanisms, contextual characteristics, implementation practices and outcomes. Participants developed the programme theory through the first round of focus groups. An initial logic model constructed by the TDAR group from the scoping review findings served as a tool to guide this work. The Wisconsin template was used (27). The output of the logic model is presented in Table 3. A more detailed consideration of context and implementation, as mapped across the CICI framework, is presented in Table 4.
Causal Mechanisms: Both staff and students stated confidence and self-efficacy as being important to the programme theory. Students spoke about feeling equipped to take ownership of their learning, ask for help, and take risks with complex topics, which was largely a consequence of involvement in classroom and school-level decision making. Meanwhile staff suggested that improvements in confidence in the classroom, combined with having the opportunity and skills to express their thoughts and feelings following student conflict, had reduced stress:
STAFF FG1;3: So … it certainly has made a difference in terms of my wellbeing, giving me more confidence within the classroom … it’s not just looking after student wellbeing, but also staff wellbeing.
The central mechanism for both of these groups of stakeholders was a change in relationships. Students mentioned peer relationships frequently, while staff emphasised relationships between staff and students: In the later instance, one member of staff suggested that circle time redresses power imbalances, creating more supportive interactions:
STAFF FG2; 14- …the starting with them … with them was to sort of have a circle time in and listen to them. Find out what they need from me and let them know what I need from them. Erm, and just … just not being afraid really to sort of break down any barriers between sort of thoughts and feelings …
Through a shift towards trustworthy and responsive relationships, the school was considered to offer a more positive and supportive culture. These changes led to students experiencing increased school connectedness. This process was further enhanced through a distributed leadership model, involving students in key decision making, such as the design of a new building or appointment of a staff member, with one commenting ‘we’ve had a huge impact with everything in school.’
Additionally, students felt that restorative practices had improved the school’s reputation in the community, and relative to other local schools. This had enhanced school connectedness and thus motivation to engage in positive behaviours and improve academic attainment:
STUDENT FG1;5: Because when I first came to the school, … we were known as “down the hill” and now it’s “the comp”. Like things have changed. …
Beyond intended causal pathways, participants considered unintended pathways, which have largely been overlooked in the previous modelling of restorative approaches. This identification illustrates the particular strength of co-production and learning from interventions already in routine practice. For example, participants indicated that the school’s improved reputation following adoption of the intervention had led to over-subscription, which had limited access in the community and placed a resource burden on the school.
Outcomes: Participants identified three key sets of outcomes, which are largely congruent with existing restorative approaches. For both students and staff the reported primary outcome was improved student mental health and wellbeing:
STUDENT FG1; 2- I think wellbeing in the school is kind’ve increased massively ..,I’ve got a brother who is 5 years older than me but he came to this school as well and he’s told me stories about how there used to be fights every week and people would set off fire extinguishers... then you look at our school now and honestly I’d be surprised if I heard about a fight because it just doesn’t happen anymore...(laughs) yeah it’s not common any more. I think generally school life has transformed and everything is more positive now. I rarely hear people talk badly about teachers um, everything here seems to be more positive and I think that contributes to all the points these guys have brought up about feeling secure and happy in the environment.
Additional outcomes are presented in Table 3.
Contextual characteristics: Drawing on the factors identified in the scoping review, the co-production process explored key contextual features that could support the implementation of the intervention and ensure the programme theory worked as intended. These factors were often explained in relation to the reason why restorative practices were initiated.
Epidemiological: Data indicated that the school had reached a tipping point, and was ready to change. This was largely was due to perceptions of increasingly poor levels of mental health and wellbeing among students, in addition to high levels of fixed-term and permanent exclusions. Existing practices based on merit and punishment were considered punitive and ineffectual in addressing the problem:
STAFF FG1; 2: …we were just finding we were going round and round and round in circles and not really making progress.
Political: The policy context in Wales has been increasingly focused on supporting mental health and wellbeing of children and young people, particularly within the educational context. The Well-being of Future Generations Act (2015) in Wales has mandated organisational and culture change to enhance metal health related outcomes. Meanwhile the Donaldson educational review on curriculum reform has outlined six key priorities, such as wellbeing, alongside an acknowledgement of the synergy between wellbeing and educational outcomes (Donaldson, 2016). Although in the case study school, restorative practice had been implemented for 8 years prior to data collection in 2016, and so was in advance of these political and educational changes, these policy priorities support its continued implementation. Socio-economic: Participants acknowledged that the case study school had a lower than national average level of free school meal eligibility and a higher level of academic achievement. Thus, whilst the school cannot necessarily be characterised as atypical, there was acknowledgment that the intervention may be more difficult to implement in a more challenging context with higher levels of disadvantage:
STAFF FG1; 6 – I think there’s more focus on students’ perspectives here um, which students value more. Generally the behaviour here is better than at schools that I’ve taught at previously, though I’d say those schools are working within a different concepts, there are inherently gonna be more issues because of the intake that they have.
Socio-cultural: Participants suggested that schools tend to have entrenched pedagogic practices that are the antitheses of restorative approaches, namely punitively orientated interactions with students. There is always the risk that staff can orientate to the default approach, which can lead to extensive variation in practice:
STAFF FG1;2 – varied yeah, it is varied across the school: you can see a restorative conversation happening in quite a negative tone in one space, but in another it can be very effective so...and that’s hard for young people as well because young people say “I’ve just had a restorative” (said in an angry voice) and actually it’s like hang on a second, that’s not a restorative
Participants also suggested a potential mismatch between the social and emotional competencies required for the effective delivery of a restorative approach, and a socio-cultural context that does not always privilege vulnerability and emotional openness. To mitigate against such issues, participants identified the importance of senior leadership vision and commitment as part of the implementation plan to ensure realignment of the school ethos with the restorative practice approach and staff commitment to training and delivery. Moreover, the school adopted a rather organic diffusion process, initially securing training to a small team of pastoral staff to ensure their buy in and capacity for modelling the approach before expanding to more diverse professional roles. Eventually working groups were established to ensure continued change to the socio-cultural context, with a Behaviour Research Group reviewing how the restorative practices could be sensitively translated into the setting.
Following the initial round of focus groups to co-produce the programme theory, further work was undertaken by the TDAR to refine their understanding and to create another iteration of the logic model. The second round of focus groups with staff and students was intended to confirm these outputs Importantly they provided clarity on a number of uncertainties that remained following the first round of focus groups and elicited aspects of the intervention and context that had still not been identified. In particular, participants focused on the socio-ecological domains beyond the inter-personal, notably family and community level processes. For example, family-based activities emerged, particularly the delivery of parenting skills, to ensure some congruence between the school ethos and family dynamic:
STAFF FG2;13: We’re working with parents on the approach we would take in school particularly where children have reflected and said ‘well if I did that at home this is what would happe’n …or this is what I see at home. And that ongoing communication and collaboration with parents is really important and it’s quite a long journey for some.
Taken together, this phase provided a nuanced and contextually sensitive understanding of the local innovation. At this point it is important to consider the potential for different stakeholder groups to have different perspectives on the programme theory. In the present case example, there were no significant disagreements. However, it may arise and the processes for resolving potential conflict needs further consideration.
- Planning to Optimise Intervention Delivery in Local Contexts
The fifth phase progresses to planning to optimise the intervention delivery in the local context. A knowledge exchange event was hosted at the school (Figure 2). The primary purpose of the event was to reflect on the research findings and to identify if there was a mismatch between the hypothetical programme theory that should underpin the approach, and the reality of implementing it within a real-world setting. This was important in exploring if intervention delivery could be optimised to overcome contextual issues that had led to barriers to implementation (e.g. prioritisation of academic achievement), as identified in Phase 4. This helpful when moving forward to feasibility and outcome evaluation, as it provides some assurance that a future evaluation would not be assessing a sub-optimally delivered approach.
The event also served to address an additional two aims. First, it strengthened partnership between stakeholders. Second, it reasserted the emotional investment of the school (28). To progress to further evaluation, where the school may be required to support the sharing and delivery of practices within other institutions, it was deemed important for the school feel committed to both the intervention and research. Reflecting with stakeholders provided a positive experience that renewed enthusiasm, with many commenting on how much the school had achieved since the initial introduction of the intervention.
- Planning for Outcome Evaluation
The sixth phase comprises planning for future outcome evaluation if appropriate. Where outcome evaluation is warranted, the type of evaluation would be most suitably assessed against the phases of evaluation prescribed by the MRC guidance for developing and evaluating interventions: pilot and feasibility trial, a randomised controlled trial; natural experiment or other quasi-experimental design; or longer-term implementation evaluation (3, 29). Further work is required to refine decision-making about the most suitable evaluation approach, and an a priori progression criteria similar to that used in feasibility trials may be helpful to guide next steps after the pragmatic formative process evaluation. Potential criteria to be considered are: 1) The evaluability of the intervention(30); 2) The Value of Information (VOI), which weighs the cost of obtaining evidence against the need for certainty amongst stakeholders (31); and 3) the applicability of the existing evidence base to the local context. For example, Aarons et al. have developed a framework for ‘borrowing evidence’, which assesses the similarities of different interventions and contexts to see if the outcomes of evaluations conducted elsewhere have relevance to the new context in question (32).
In the present case study, planning is primarily being conducted through the TDAR group. The SHRN infrastructure offers a particular opportunity to continue with pragmatically orientated innovation evaluation, through the conduct of a pragmatic feasibility and outcome trial. As of 2020 the network includes 100% of the 210 state-funded schools in Wales, providing a complete sample frame for randomisation. A sample of students at each participating school complete bi-annual surveys of their health and wellbeing, and provided data is collected at appropriate times, these surveys could be exploited as the data source for outcome measurement. As popular innovations, such as that selected for the case study, are gaining traction within systems, it is imperative that we have responsive study designs. Use of routine data, such as that collected through the SHRN survey data offers such responsivity, although the evidence generated is arguably less scientifically robust than that provided by RCTs.