The present results describe the seven phased framework utilised to identify, model and refine the whole-school based restorative approach, in addition to planning for potential scale up for further feasibility and outcome testing (Figure 1). These phases are not intended to be sequential but iterative, with stages 4 and 5 particularly necessitating revisiting on a number of occasions.
- Identification of Innovative Local Practice
The first phase is to identify innovative local practice that warrants progression to theoretical modelling, refinement and outcome evaluation. We identified the case study intervention through the DECIPHer research centre hosted School Health Research Network infrastructure (SHRN) (15). The network comprised 165 of all secondary schools in Wales (N=212) at the time of study, with representation from all 22 local authority areas. The Network seeks to optimise research collaboration between researchers, policy-makers and practitioners, particularly in regard to knowledge translation. One of the central mechanisms to encourage collaborative working is through a programme of knowledge exchange activities, including webinars and stakeholder meetings. At regional meetings researchers present study data whilst practitioners share examples of innovative practice to improve health and wellbeing. The case study intervention had been presented at a stakeholder event, with the school gatekeeper following up the potential for research collaboration with the SHRN Manager. The manager identified a relevant academic contact with the requisite expertise to a TDAR group. The group decided the most appropriate evaluation design moving forward.
On deciding to progress to a pragmatic formative progress evaluation, we considered the following key aspects of the intervention, system and extant research: 1) Feasibility of theory modelling: We questioned if we could establish that an “intervention” (regardless of type) was in use and that we could, in conjunction with stakeholders, characterise intervention theory, outcomes, implementation and key contextual characteristics. The school had been recognised as delivering sector-leading, best practice in restorative practice and had been awarded a Restorative Service Quality Mark (RSQM) in 2010. As a consequence of this external validation we felt that there was clear delivery of a restorative approach and there was adherence to core processes. 2) Feasibility of implementation and scale-up: We established that the restorative practice had been routinely used and resourced for a substantial period of time (e.g. 8 years). We further considered the future traction of the intervention and if it could be scaled-up for evaluation beyond the single case, or was so contextually contingent no replication was feasible. There was no indication that the school was atypical so the intervention could not be transported to other secondary schools, and indeed the school had been increasingly invited to share their practices with other schools at a national level due to being recognised as sector leading; 3) Research Co-production: We consulted with the school that they were prepared to participate in a research study and would potentially be committed to future research.
- Scoping review to identify intervention theory of change; outcomes; antecedent and emergent contextual characteristics implementation
The second phase is to engage in a scoping or systematic review of the existing scientific research to develop a preliminary understanding of the intervention. From here an initial logic model can be constructed, which can serve as a normative referent to model the real-world case example. A review further supports consideration of the effects of such interventions, and potential iatrogenic pathways that should be attended to in the primary research (12). Table 2 presents key retrieved studies that describe restorative approaches and conducted process and/or outcome evaluations.
Theory of Change and Outcomes: Across the studies there was a lack of specificity around the underpinning theory of change. Rather there was broad principles of how restorative approaches may work, largely through the building, maintaining and restoring of relationships, where individuals take responsibility for their actions and positively engage in relationships repair and conflict resolution. (16, 17). This may be further supported by changes in classroom management practices and school ethos. The INCLUSIVE intervention provides one of the most theoretically informed approaches (18), hypothesising that through restorative practices students are more likely to engage with schools’ pedagogic practices and embrace rules and ethos. As a result, school connectedness increases and relationships improve. A range of activities at the targeted, universal and whole-school level can be considered as restorative. The approach may be most effective when it is fully adopted at the system level, with modification across settings to align with localised problems (19, 20). Evaluations of school based restorative approaches have identified a range of measurable intervention outcomes (18, 19, 21, 22). At the student level these are: mental health and wellbeing (18); social and emotional competencies, including empathetic attitudes and self-esteem (23); academic attainment (22), incidences of bullying (23); and school exclusions (21, 22). There has been limited consideration of staff level outcomes. Iatrogenic pathways remain largely underdeveloped.
Antecedent and emergent contextual characteristics implementation: We mapped key system-level characteristics that might comprise the theory of change across different contexts and inhibit planned implementation. We employed the Context and Implementation of Complex Intervention Framework (CICI) (24) as a hermenutic tool for mapping contextual and implemntation factors (Table 2). Although the extent evidence base did not map onto all specified domains, a number of influences emergeed across papers. Epidemiological Implementation had been supported by an increase in the prevalence of bullying within the specified context, leading to more support for such approaches (22, 23). Political: There had been increased support for restorative approaches due to an alignment with political/policy priorities, which had often led to direct governemnt funding (18, 19, 21, 25). Ethical:Resrotative approaches reflected belief in a fair and just society where citizens are respected. Such interventions are viewed as a more ethical approach to punitative or criminalised responses (21){Wong, 2011 #15}
- Establishment of a Transdisciplinary Action Research Group (TDAR)
The third phase is establishment of a TDAR Group, which is intended to support the effective collaboration between research, policy, frontline staff and community stakeholders (26). It is underpinned by the principles of action research, and its tenets has been increasingly deployed in guidance around intervention development to ensure that approaches are maximally responsive to the contexts and populations where they are to be implemented (1). Within a pragmatic formative process evaluation, TDAR can help to bring a comprehensive and nuanced understanding of the intervention that is being modelled, in addition to a rich awareness of the context in which it has been originally delivered. We established a Transdisciplinary Action Research Group (TDAR) comprising researchers and members of the school community, who routinely met throughout the duration of the study to guide research conduct. It further supported planning around dissemination and consideration of future evaluation.
- Co-production of Intervention Logic Model with Stakeholders
The fourth phase is the co-production of a logic model with key stakeholders to identify the underpinning theory of change, outcomes, and contextual characteristics that impact upon causal pathways and implementation practices. Participants developed the logic model from the initial construction undertaken by researchers following the scoping review, and using the Wisconsin template (27). The output of the logic model from both phase 4 and phase 5 is presented Table 3. A more detailed consideration of pertinent contextual factors, as mapped across the CICI framework, is presented in Table 4.
Theory of change: Participants identified the range of activities that a school may undertake at the individual, group, organisational and community level, with interactions across these levels. The central mechanism activated by these activities is a change in the natures of relationships, particularly between staff and students: For example, circle time redresses power imbalances, creating more supportive interactions:
STAFF FG2; 14- …the starting with them … with them was to sort of have a circle time in and listen to them. Find out what they need from me and let them know what I need from them. Erm, and just … just not being afraid really to sort of break down any barriers between sort of thoughts and feelings …
Through a shift towards trustworthy and responsive relationships, the school was considered to offer a more positive and supportive culture. These changes led to students experiencing increased school connectedness, which was explained as the central process underpinning observed outcomes and has been identified in other approaches. This was further enhanced through a distributed leadership model, involving students in key decision making, such as the design of a new building or appointment of a staff member, with one commenting ‘we’ve had a huge impact with everything in school.’
Outcomes: Logic modelling indicated three key sets of medium-term outcomes, which are largely congruent with existing restorative approaches. First, both student and staff reported feeling that the intervention had improved student mental health and wellbeing:
STUDENT FG1; 2- I think wellbeing in the school is kind’ve increased massively ..,I’ve got a brother who is 5 years older than me but he came to this school as well and he’s told me stories about how there used to be fights every week and people would set off fire extinguishers... then you look at our school now and honestly I’d be surprised if I heard about a fight because it just doesn’t happen anymore...(laughs) yeah it’s not common any more. I think generally school life has transformed and everything is more positive now. I rarely hear people talk badly about teachers um, everything here seems to be more positive and I think that contributes to all the points these guys have brought up about feeling secure and happy in the environment.
Second, there was indication of improved staff wellbeing, with participants claiming increased confidence, whilst having the opportunity and skills to express their thoughts and feelings following student conflict had reduced stress:
STAFF FG1;3: So … it certainly has made a difference in terms of my wellbeing, giving me more confidence within the classroom … it’s not just looking after student wellbeing, but also staff wellbeing.
Third, was a range of educational outcomes, relating to attainment, attendance and fixed-term and permanent exclusions. Involvement in classroom and school level decision-making, combined with support teacher-student relationships, had developed students’ confidence to take ownership of their learning, request help when required, and take risks with more complex topics as they were less concerned about making mistakes: Improved relationships had also motivated some students to attend school more frequently. Additionally, students felt instilling of restorative practices had improved the school’s reputation in the community, and relative to other local schools, which had increased school connectedness, and thus motivation to engage in positive behaviours and improve academic attainment:
STUDENT FG1;5: Because when I first came to the school, … we were known as “down the hill” and now it’s “the comp”. Like things have changed. …
Beyond positive intervention outcomes, participants considered iatrogenic pathways, which have largely been overlooked in previous modelling of restorative approaches. This identification illustrates the particular strength of co-production and learning from interventions already in routine practice. For example, participants indicated that the school’s improved reputation following adoption of a restorative approach had led to over-subscription, which had limited access in the community and placed a resource burden on the school.
Antecedent and emergent contextual characteristics and implementation: Drawing on the factors identified in the scoping review as a starting point, the co-production process explored key contextual features that participants felt impacted on implementation and the theory of change. Epidemiological: Data indicated that the school had reached a tipping point, and preparedness for change was due perceptions of increasingly poor levels of mental health and wellbeing and high levels of fixed-term and permanent exclusions. Existing practices based on merit and punishment were considered punitive and ineffectual in addressing the problem:
STAFF FG1; 2: …we were just finding we were going round and round and round in circles and not really making progress.
Political: The policy context in Wales was increasingly orientated to support the prioritisation of mental health and wellbeing of children and young people, particularly within the educational context. The Well-being of Future Generations Act (2015) in Wales has mandated organisational and culture change to enhance metal health related outcomes. Meanwhile the Donaldson educational review on curriculum reform has outlined six key priorities such as wellbeing, alongside an acknowledgement of the synergy between wellbeing and educational outcomes (Donaldson, 2016). Socio-economic: Participants acknowledged that the case study school had a lower than national average level of free school entitlement and high level of academic achievement. Thus, whilst the school cannot necessarily be characterised as atypical, there was acknowledgment that the intervention may be more difficult to implement in a more challenging context with higher levels of disadvantage:
STAFF FG1; 6 – I think there’s more focus on students’ perspectives here um, which students value more. Generally the behaviour here is better than at schools that I’ve taught at previously, though I’d say those schools are working within a different concepts, they’re are inherently gonna be more issues because of the intake that they have.
Socio-cultural: Participants identified entrenched pedagogic practices that were the antitheses of restorative approaches, namely punitively orientated interactions with students. There was indication that staff could orientate to the default approach, which could lead to extensive variation in practice:
STAFF FG1;2 – varied yeah, it is varied across the school: you can see a restorative conversation happening in quite a negative tone in one space, but in another it can be very effective so...and that’s hard for young people as well because young people say “I’ve just had a restorative” (said in an angry voice) and actually it’s like hang on a second, that’s not a restorative
Participants also suggested potential incongruence between the social and emotional competencies required for the effective delivery of a restorative approach, and a socio-cultural context that did not always privilege vulnerability and emotional openness: To mitigate against such issues, participants identified the importance of senior leadership vision and commitment as part of the implementation plan to ensure realignment of the school ethos with the restorative approach and staff commitment to training and delivery. Moreover, the school adopted a rather organic diffusion process, initially securing training to a small team of pastoral staff to ensure their buy in and capacity for modelling the restorative approach before expanding to more diverse professional roles. Eventually working groups were established to ensure continued change to the socio-cultural context, with a Behaviour Research Group reviewing how the restorative practices could be sensitively translated into the setting.
- Confirmation of Logic Model with Stakeholders
The fifth phase of the process entails confirmation of the logic model with stakeholders. Commonly studies present logic model development as a static phase, but to ensure meaningful co-production multiple opportunities for input are required. The second round of data collection with participants provided clarity on a number of uncertainties that remained following phase 4 and elicited aspects of the intervention and context that had not yet been identified. In particular, participants focused on the higher level of the socio-ecological domains rather than inter-personal aspects, notably family and community level processes. For example, family based activities emerged, particularly the delivery of parenting skills, to ensure some congruence between the school ethos and family relational dynamic:
STAFF FG2;13: We’re working with parents on the approach we would take in school particularly where children have reflected and said ‘well if I did that at home this is what would happe’n …or this is what I see at home. And that ongoing communication and collaboration with parents is really important and it’s quite a long journey for some.
The additional processes and activities elicited during this phase are presented emboldened and italics in Table 3.
- Planning for Intervention Refinement
The sixth phases shifts towards prospective intervention developmental work, and planning for the refinement of intervention. We hosted a knowledge exchange event at the school site in order to feedback the study findings and discuss the logic model generated during phase 4 and 5 (Figure 2). The purpose and impacts of the meeting were fourfold. First, it aimed to provide an additional opportunity to reach consensus and verifying the underpinning causal mechanisms of the intervention, ensuring that the research team who interpreted the data and the TDAR group who support this process, had an adequate understanding of the intervention. Second, by highlighting where there remains areas of uncertainty or challenges with delivering the intervention, stakeholders were able to identify where further intervention refinement was required and where barriers needed to be addressed to ensure that proposed causal pathways were not being disrupted. Thirdly, the event served to strengthen partnership between researchers and the school. Fourth was to reassert the emotional investment of stakeholders, which is important to the continued delivery of the intervention (28). To progress to further evaluation, where the school may be required to support the sharing and delivery of practices within other institutions, we deemed it important that the school feel committed to both the intervention and research. Reflecting with stakeholders provided a positive experience that renewed enthusiasm, with many commenting on how much the school had achieved since the initial introduction of the intervention.
- Planning for Outcome Evaluation
The seventh phase comprises planning for future outcome evaluation if appropriate. In some instances the prior developmental phase may necessitate revisiting multiple times before phase 7 can be considered. Where outcome evaluation is warranted, the type of evaluation would be most suitably assessed against the phases of evaluation proscribed by the MRC: pilot and feasibility trial, a randomised controlled trial; natural experiment or other quasi-experimental design; and then longer-term implementation evaluation (3, 29). In this case study, planning is primarily being conducted through the TDAR group, drawing on both practice and academic expertise to assess if the intervention is sufficiently understood to progress to outcome evaluation and the appropriate study design that might be employed. The SHRN infrastructure offers a particular opportunity to continue with pragmatically orientated innovation evaluation, through the conduct a pragmatic feasibility and outcome trial. As of 2018 the network includes 100% of the 212 state-funded schools in Wales, providing a complete sample frame for randomisation. A sample of students at each participating school complete bi-annual surveys of their health and wellbeing, and provided data is collected at appropriate times, these surveys could be exploited as the data source for outcomes. As popular innovations, such as that selected for the case study, are often gaining traction within systems, it is imperative that we have responsive study designs. Exploitation of routine data, such as that collected through the SHRN survey data offers such responsivity, although the evidence generated is less scientifically robust than that provided by RCTs.