Tailoring Research Evidence and Theory (TREAT) into Allied Health Journal Clubs: A Hybrid Effectiveness-Implementation Study

Allied Health Professionals (AHPs) commonly use journal clubs (JCs) to support Evidence-Based Practice (EBP). There is however little research regarding implementing JCs in the long term, and their impact on EBP use and skills in AHPs. This study investigated the impact of implementing a structured JC format, called “TREAT” that was specically tailored for each club, on EBP skills, condence, use and resultant changes in clinical practice over 16 sessions for AHPs in a public health service. The study also investigated AHP’s attendance, adherence, satisfaction and barriers and enablers to implementing the format. A mixed methods hybrid-effectiveness implementation design was employed, guided by the Knowledge-to-Action EBP and attitudes were (Adapted tailored journal club culture questionnaire) at baseline, and after and 16-monthly Satisfaction and impact on clinical practice were explored using questionnaires at latter two time-points, with free-form responses identifying enablers and barriers to EBP culture and implementation. Data on attendance and adherence to the format were also collected.

While JCs are commonly used by health professionals, further knowledge is required to optimise their implementation and effectiveness. Illic et al., reported in their recent systematic review that research is needed to investigate the effectiveness of JCs in increasing health professionals' EBP knowledge and implementation within clinical settings [7]. A systematic review by Harris et al., further identi ed that JCs rarely incorporate evidence-informed components and theory important for them to run successfully and sustainably [8]. Additionally, engagement of health professionals in JC's can vary, potentially due to the JCs not being implemented using principles of adult learning which target the needs of clinician learners [8,9].
To address some of these shortcomings, in 2016, we developed a structured JC format-TREAT (Tailoring Research Evidence and Theory) [5]. TREAT uses 11 components evidenced in the literature and theory to enhance a JC's effectiveness, which are then tailored to the local context [5,8,10]. We undertook a cluster randomised controlled trial (RCT) investigating the initial effectiveness and feasibility of the TREAT format in a group of 126 allied health professionals (AHPs) across nine JCs in a single health service.
Each JC was randomly allocated to either the TREAT JC format or standard JC format 1 hr/month for 6 months [5]. While AHPs reported signi cantly greater satisfaction with the TREAT format and gave positive qualitative feedback regarding the format, there were no signi cant improvements in their EBP skills, attitudes, knowledge and/or practice, compared to the standard format. We proposed that the 6month implementation period may not have been long enough for important changes to be identi ed and further investigation is required to understand the impact of TREAT JC participation over a longer period [5].
As a follow up to this RCT, we conducted a qualitative study guided by behaviour change theory. Nineteen clinicians who participated in the RCT were interviewed 6 months after the trial period had ended to explore their experience with sustaining the TREAT JC format [11]. These clinicians reported that there were bene ts of maintaining the format within their local settings and identi ed several individual and service level factors in uencing sustainability. The results of the study were synthesised into a range of potential implementation strategies that may enhance opportunity, motivation and capability for sustainability of JCs within local contexts [11]. These included, amongst others, having JC portfolio holders, upskilling senior staff, email reminders and departmental leadership advocating attendance, however it is unknown which of these strategies may be most effective. Moreover, it remains unclear what the impact on EBP skills and use would be from implementing TREAT JCs using a more tailored approach, whereby implementation strategies addressing unique local contextual barriers are applied to enhance long term sustainability. As research in structured allied health JCs to date has generally been within a six-month period or less [5,12], research is also needed to explore the effectiveness of structured JCs implemented over a longer time period.

Objectives
To address these identi ed gaps in knowledge, our primary aim was to investigate which implementation strategies that addressed local barriers to implementing the TREAT JC format were most effective in promoting AHPs JC attendance, culture and satisfaction. The secondary aims were to; investigate the impact of implementing the structured TREAT JC format over 16-monthly sessions on AHPs' EBP skills, con dence, use and changes in clinical practice; evaluate satisfaction, attendance, and adherence to the TREAT format; and describe barriers and enablers to its implementation. The present manuscript presents the results of these secondary aims, while the primary aim will be reported as part of a realist evaluation (currently in preparation).

Study Design
We conducted a hybrid implementation-effectiveness study (type 3) design [13] using a mixed methods approach guided by the Knowledge to Action (KTA) cycle [14]. Data was collected at baseline, and at two follow up time points (after participation in up to 10 and 16-monthly TREAT JC sessions). Due to the primary implementation focus of this trial and secondary effectiveness focus, all participants received the TREAT JC format, with reporting guided by the EQUATOR Standards for reporting implementation studies [15]. Ethical approval for the study was sought (HREC/15/QGC/310) prior to commencement.

Participants
Participants were recruited from any AHP JC (both new and existing) within a single non-metropolitan Australian public health service which consisted of two main hospitals and several community centres.
Line manager support was gained prior to consenting participants within JCs. In addition, research mentors who supported the JCs were invited to participate in an audit of their support activities. Procedure: An overview of the study procedure including assessment time points and alignment with steps of the KTA cycle is found in Table 1.

Pre-implementation
Prior to implementing the JCs, at least two clinicians from each JC volunteered to be facilitators and attended a four-hour in-house Evidence-Based Practice workshop run by Bond University. This workshop focussed on developing a PICO (Population, Intervention, Comparator, Outcome) based question and search strategy, identifying different research designs and critically appraising articles. An AHP researcher was also allocated to be the research mentor for each JC. Each of these researchers received approximately 4 hours of training (combination of face to face and reading a written facilitator manual) in facilitation of the TREAT JC format, with a focus on their role in the project.
Participants in each JC then completed a baseline assessment and chose topics for their upcoming JCs during a 60min session facilitated by the research mentor. Topic selection involved the JC members brainstorming a list of clinically relevant topics they wanted to discuss in their JC. Clinicians then voted on which topics they perceived to be most relevant and important to their clinical practice. The topics with the most votes were prioritised to be discussed rst and presenters to discuss these topics were allocated.

Identi cation of Implementation Strategies
Approximately 1 week after this session, the research mentor met with the JC clinician facilitators and 1-2 other JC members to discuss a summary of the baseline assessment results including barriers identi ed in the Journal Club Readiness and Culture Questionnaire (described in Outcome Measures). Clinicians received a list of potential implementation strategies (see Supplementary File 1) derived from qualitative ndings in a previous study [11] which targeted barriers that related to Capability, Opportunity or Motivation of the COM-B model of behaviour change [16]. Clinicians were also encouraged to generate other implementation strategies to address barriers speci c to their context as identi ed from their baseline assessment. Following this discussion, the research mentor in collaboration with the clinician facilitators nalised an implementation plan for each participating JC (see Supplementary File 2 for an example implementation plan).

Intervention
The TREAT JCs were run for one hour, once a month over approximately 16 months, with some exibility and rescheduling required over holiday and busy service periods. Key components of the TREAT structure included: group appraisal of articles using the Critical Appraisals Skills Programme (CASP) structured appraisal tools [17], discussion of the evidence in the context of clinical practice, and having set facilitator, presenter and scribe roles in the JC. An example session plan is found in Supplementary File 3 with further details published previously [5]. To build the clinician facilitator's con dence and capability in facilitating, the rst two TREAT JC sessions were usually facilitated by the research mentor, followed by two sessions of co-facilitation by both the research mentor and the clinician facilitator, and then two sessions led by the clinician facilitator (with the research mentor still present). The research mentor remained contactable for support between sessions throughout the duration of the 16-month implementation.

Outcome measures
The effect of participating in the TREAT JC format on evidence based pro ciency was evaluated through the use of the Journal Club Readiness and Culture (JCC) questionnaire, Adapted Fresno questionnaire, EBP questionnaire (EBPQ), and in uence of journal club on clinical practice questionnaire (IJC). The JCC, Adapted Fresno and EBPQ were completed at baseline, 10-months, and 16-months. The IJC and two additional Likert scale satisfaction questions of the JCC were completed at 10 and 16-months.
Journal club readiness and culture questionnaire The JCC questionnaire was developed based on mechanisms found to promote JC sustainability [11]. Participants rated 11 items on a 5-point Likert scale, with potential responses ranging from strongly disagree to strongly agree. Participants then completed an open-ended questionnaire regarding barriers and facilitators to EBP and running of JCs within their teams.

EBP pro ciency
Changes to EBP pro ciency were monitored using the EBP Questionnaire (EBPQ) and the Adapted Fresno Test. The EBPQ is a 24-item questionnaire which asks participants to self-rate their (1) attitudes towards, (2) con dence in, and (3) use of EBP on three separate subscales [18]. The Adapted Fresno Test [19] is an objective measure of EBP skills which involves the participant answering 7 questions related to one of four clinical scenarios. The Adapted Fresno test was scored by a member of the research team (JW), who was blinded to the participant's JC and the time point at which the test was completed. Intra-rater reliability of the Adapted Fresno has been published previously and found to be high [20].
In uence of journal club on clinical practice The in uence of JCs on clinical practice was evaluated via a tailored questionnaire which included questions for each JC session regarding whether or not the participant attended and if the article appraised was relevant or not to them, and a 5-point Likert scale which asked participants to rate whether the evidence from the article appraised during each JC session led to changes in practice or con rmed their current practice. Where the participant indicated a change in practice, they were subsequently asked to identify what the change was from a list of multiple-choice options (e.g., updated clinical guideline, adopted new treatment strategy).

Fidelity measure
Clinician facilitators completed an audit questionnaire after every JC session to identify which key components of the TREAT format were adhered to in each session, with a space for additional comments and re ective questions also provided. After 10 monthly sessions, research mentors were asked to complete a custom multiple-choice questionnaire to indicate what level of support they provided within each JC session to their allocated JC. Multiple choice options ranged from a maximal level of support (i.e., facilitating the session) to no support provided (i.e., not attending the session and not providing any active support). The option for free form comments was also provided.

Data analysis
Descriptive statistics (frequencies and percentages) were calculated for demographic information and mean and standard errors were calculated for EBPQ, Adapted Fresno Total, and JCC questionnaire scores at baseline, 10-months and 16-months. Sample size was estimated based on a simple two-sided paired ttest that could detect a change of 6 points on the EBP-Q with 80% power at a signi cance level of 0.05. The EBP-Q was chosen as the primary outcome due to its wide use in measurement of EBP pro ciency in health professionals, with a change of 6 points on the EBP-Q con dence scale (i.e., self-reported knowledge) being considered the minimum clinically important change by the study authors. To detect this change, it was calculated (G*Power 3.1) that a sample of 28 participants was needed. Within group changes for quantitative measures (i.e., Adapted Fresno, EBPQ, Likert scale questionnaires) between time points were initially assessed with paired t-tests. Subsequently, mixed effects regression models were used with the JC and individual participant identity considered as random effect levels. Time-point was considered the variable of interest and the effects from baseline to 10-months and baseline to 16-months reported. Prior EBP training, gender, age group and total JC sessions attended were considered as xed effect covariates. Where a covariate effect was seen with p<0.05 it was included in the model and adjusted time-point effects were reported.
Qualitative content analyses were used for responses to open ended questionnaire questions for the 10 month and 16-month data. In this approach, one of the authors (JW or CB) coded meaning units into categories and subcategories [23] using NVivo software. These were then checked by a second author (RW). Once all data was analysed, qualitative and quantitative data were brought together, and qualitative comments were used to help understand quantitative results.

Results
A total of 132 unique clinician participants consented to take part in the study across six JCs. Demographic information of participants is found in Table 2. Most participants were female (n=111, 84%) with 2 to 10 years of clinical experience (n=79, 59%), and were currently working in the inpatient acute setting (n=67, 51.5%). Of these participants, 79 (61.2%) completed baseline assessment, 63 (48.8%) completed the 10-month assessment and 47 (49%) completed the 16-month assessment. Two of the JCs did not participate in the 16-month assessment due to changes in clinical services provided during the COVID-19 pandemic, resulting in their JCs requiring an extended break after 12 sessions. In addition, four research mentors completed the support audit (median years of post-doctorate experience=4.5 years, range 4 to 11 years).

EBPQ and Adapted Fresno Test
As shown in Table 3, on the EBPQ participants self-reported improvements in their EBP con dence at 10and 16-months compared to baseline. Minimal changes were apparent in EBP attitudes and EBP use across time points. On the Adapted Fresno Test EBP skills were found to be improved at 10 and 16months.
Journal Club Culture Questionnaire Clinician ratings of their manager's expectation for them attend JC and their access to expertise and resources outside the team were found to improve compared to baseline at 10-months, with the latter continuing to improve 16-months following JC participation (see Table 3). An increase at 16-months compared to baseline was also found for information sharing within the team. At the 10-month and 16month time points, the average ratings of agreement to wanting the JC to continue and recommending it to others were approaching 4 out of 5 (mean= 3.7 and 3.9 respectively), indicating general positive agreement with these statements.
When covariates were taken into consideration, prior EBP training was found to have an effect on the Adapted Fresno Total scores, with adjusted changes from baseline to 10 months being 6.4 (95% CI: 0.23-12.5, p=0.042) and baseline to 16 months 7.8 points (95% CI, 1.09, 14.9, p=0.023). Gender and total attendance were found to have a signi cant effect on the group participation item of the JCC questionnaire leaving a negligible effect of time-point at 10 months; 0.01 (95% CI: -0.17, 0.19, p= 0.90) and 16 months; 0.11 (95% CI -0.06, 0.29, p=0.37) compared to baseline.

Qualitative responses of Journal Club and Culture questionnaire
Five categories were identi ed in the questionnaire responses: 1) enablers and 2) barriers to EBP culture, 3) enablers and 4) barriers to JC; as well as 5) suggestions for improvement. Most commonly reported categories and subcategories are shown in Table 4 for both 10 month and 16-month time points. In general, responses did not conceptually differ across time points. Some barriers to JC, including lack of preparation due to time constraints and variable attendance, that were evident at the 10-month time point were not reported at the 16-month time point. The most commonly reported enablers to EBP culture were team educational opportunities (with "Journal club" being the most frequently reported of these opportunities); and having collaborators and people as resources including "EBP champions" and "Research Fellows". The most predominant barrier was time and competing caseload demands of clinicians, "Time constraints to research EBP. Competing clinical and non-clinical priorities" (P089), as well di culties implementing EBP as described as "possible lack of con dence in integrating EBP into clinical practice if there is a change required" (P027).
Enablers to JC related to structural components of the format included choosing clinically relevant topics, "having a team consensus on choosing articles relevant to the group" (P119) and active participation from the group, with one clinician commenting, "Group participation is important for the success of journal club" (P063). Supportive team factors were also commonly reported as enablers, including "Sharing between all team members "(P106) and "Open mindedness and respect of differing opinions" (P128). Conversely, barriers to JC included topics not being relevant, lack of active participation or preparedness and general time constraints, for example "time constraints in preparing for JC i.e. prereading article." (P005).  7 11 Reduced knowledge, skills & confidence 10 6 Lack of preparation generally due to time constraints 22 Increased education and support  Structure changes  Scheduling changes  Changes to preparedness  Topic selection suggestions   19  14  7  7  3   7  18 3 3 11 In uence on clinical practice

Suggestions for improvement
Across the 16 sessions, 64 clinicians reported that they updated guidelines, processes or pathways as a result of JC participation, 88 clinicians reported adopting new treatment strategies or resources, 30 clinicians reported starting new quality projects and 6 clinicians reported starting new research projects. One clinician also reported discontinuing a current practice due to lack of evidence as appraised in a JC session (See Supplementary le 4).

Treatment delity
As depicted in Figure 1, most components of the TREAT format were adhered to across sessions.
Components reported to have the highest adherence (100%) were having a relevant topic, a facilitator present, group appraisal and application to clinical context discussed. Items with lowest adherence were reviewing actions from the minutes (38.1%) and seeking library support (51.8%). The level of support provided across sessions by research mentors is shown in Table 5. The rst two sessions were all facilitated by the research mentor, while in subsequent sessions, the level of support varied across JCs.
However, a general pattern of reduced support as the sessions progressed was evident in all JCs.

Discussion
Clinicians who participated in the TREAT JCs improved their EBP skills, and reported improvements to EBP con dence and changes to clinical practice. Overall clinician satisfaction and adherence to the TREAT JC was reasonably high with several enablers and barriers to implementing JCs identi ed.
Although attendance varied across the extended data collection period, ndings support the feasibility of implementing a JC using the TREAT format for up to a 16-month period.
This is the rst study to explore longer term impacts of structured allied health JCs. Previous research implemented the format for 6 months with a 6-month follow up [5]. As such, this research adds to the current evidence base for the impact of JCs, particularly for AHPs. Indeed, a recent systematic review highlighted that the majority of evidence to date has been from the medical professions [7] and recommended the need for further research to support the effectiveness of JCs in improving knowledge, attitudes and EBP implementation for all health professionals. Improved EBP skills identi ed on the Adapted Fresno Test and improved self-reported con dence on the EBPQ suggest that JCs may not only maintain levels of EBP skills in AHPs but increase them. This is particularly important for AHPs, for whom EBP skills have been reported to degrade the longer AHPs work clinically [2], thus impacting on their ability to provide evidence-based care.
The improved EBP skills seen in the present study are in contrast to Wenke et al., [5] who found no quantitative changes to EBP skills 6 months after implementation of TREAT JCs. This may support the notion that more than 6 months of JC participation is required to consolidate EBP skills and self-e cacy.
The results of this study also imply that the Adapted Fresno Test may be more sensitive or relevant to AHPs as it uses AHP-based clinical scenarios in contrast to the Assessing Competence in EBP (ACE) tool [22] used by Wenke et al., [5] which uses a medical clinical scenario to assess EBP skills.
Other important changes that were found from the JCC questionnaire, were improved access to expertise and resources outside the team and sharing of knowledge within the team. These results may indicate that teams may have felt more equipped to run the JCs using the TREAT format. This has been identi ed as an important factor in sustainability of JCs [11]. The reduced support provided by the research mentors over time, similarly, may indicate that clinician facilitators felt more independent in running the JC. This increased independence was most likely as a result of the clinician's improved self-e cacy (as re ected by improved self-reported EBP con dence on the EBQ), which has also been seen in AHPs receiving other EBP training [23]. The increased con dence of clinicians in running the JCs may have been enabled by reports of strong EBP culture within teams, wherein EBP was valued and supported. This positive culture may have been reciprocatively sustained by the ongoing participation and outcomes of the JC, as re ected in ratings of EBP being valued by the team being consistently high on the JCC questionnaire over time.
While the self-reported frequency of use of EBP on the EBPQ did not have meaningful changes over time, a large proportion of clinicians self-reported that they adopted new treatment strategies and changed clinical processes following participation in JC sessions. This nding adds to the evidence base demonstrating the impact and value of JCs on real-world clinical practice, something which has had little exploration previously [3]. Even so, di culties with applying evidence into clinical practice, known to be in uenced by a myriad of factors [24], was also reported by participants as a barrier to general EBP culture. This barrier may have restricted the number of clinical changes made from the JC evidence. The ability to apply evidence in practice is also based on the strength of the study being appraised, and while the evidence base for AHP care is growing, evidence for many areas of practice are still emerging [25] and therefore may not always be at a level where results can be readily applied in practice, as identi ed by some clinicians.

Limitations and Future Directions
Certain limitations can be identi ed for the present study. Firstly, due to a high turnover of participants, fewer data points at 10-and 16-month assessments were identi ed compared to baseline. High turnover has been reported in previous JC research [5,26]. The nature of clinicians rotating in and out of teams every 3-6 months is re ective of usual allied health practice for larger departments, particularly for newer graduate clinicians so that they can gain skills across a variety of caseloads. In addition, while four of the six JCs in the present study sustained the format for 16 sessions and anecdotally have continued to use the JC several months after the study completion, two JCs were unable to continue the format due to unforeseen service changes related to the COVID-19 pandemic, resulting in fewer data points at 16months. COVID-19 related impacts including staff being redeployed to other areas, meant one JC was not viable to continue, whilst other services put a pause on certain non-clinical activities due to a focus on clinical service provision as part of their business continuity plans.
Clinical changes were self-reported by clinicians and similar changes may have been counted more than once by different participants, therefore future research may bene t from auditing these changes to further substantiate and describe them. Generalisability of the results may be limited by the fact that participants were all from one hospital and health service setting. It is therefore unclear what the impacts of TREAT JCs may be across other contexts within Australia and internationally including primary health care and private practice.
Future research should investigate the use of TREAT JCs in other contexts, as well as broadening to the nursing and medical professions. Viability of the TREAT model in terms of time demands upon clinicians could also be explored. While the TREAT format is designed to minimise the impact and burden on clinician time, clinicians continued to report reduced time as a barrier to implementing the JC in their local contexts. Further exploration regarding how to enhance the e ciency of the format and handover of tasks and roles to new staff should be undertaken, as well as determining the optimal frequency of the JC sessions required to have a bene cial effect. Further integration of ongoing training particularly for new staff joining the team in the JC format and basic EBP processes is also indicated.

Clinical implications
This study highlights that a structured JC format can be implemented in a public hospital and health service environment with allied health clinicians and sustained for up to 16 months. External support from a more experienced researcher appeared bene cial for building clinician facilitator con dence and increasing internal capacity. This bene t was particularly relevant in the rst few sessions, and the tapering off of support over time appeared to be an effective approach. However, it was found that for some teams who are less con dent in EBP, this support may need to be extended and more prominent. It is important that clinicians consider their local barriers and enablers to JC implementation prior to commencing their JC, to identify key implementation strategies tailored to their context to support sustainability. The structured format with clear roles and expectations of members shared within the team may enable sustained use of the TREAT JC format. Other supportive team factors such as a welcoming and non-threatening environment for sharing, supportive managers and active participation from members and existing research and clinical knowledge may also increase sustainability. Due to changing social distancing requirements arising from the COVID-19 pandemic, JCs may need to consider an online format. Indeed, within the present study, some teams with staff across multiple locations were already using videoconferencing and anecdotally some of the continuing JCs are using the online format at the time of writing with good success.

Conclusion
To ensure that the clinical care provided by AHPs is based on the best available evidence, there must be investment in interventions which help build EBP skills, con dence, and behaviours in everyday practice [2]. The present study demonstrated that participation in a structured TREAT JC may improve AHPs EBP skills, con dence and application to clinical practice. While opportunities for future re nement of the structured format have been identi ed, this study demonstrated the value and sustainability of the TREAT JC format within AHP practice settings, being able to be sustained for up to 16 monthly sessions. The potential for broader application to other professions and settings should be explored.