A total of 41 participants, 22 from the UK and 19 from Australia and 12 RTCs were interviewed - three ARCs in the UK, seven AHRTCs and two CIRHs in Australia.
Five major themes emerged from the interview thematic analysis (1) Dissonant metrics and drivers for healthcare improvement and research; (2) Different models of leadership; 3) Public –patient involvement and research co-production; (4) Workforce development; and (5) Barriers to collaboration. Quotes are presented as verbatim comments from participants.
Theme 1. Dissonant metrics and drivers for healthcare improvement and research
Participants from both countries identified significant challenges in integrating applied health care and improvement approaches with more rigorous discovery and implementation research (relevance vs rigour). Dissonant metrics between academic and healthcare sectors underpin this tension. Academic organisations remain focused on traditional metrics of grants, publications and citations, while healthcare focuses on service outcomes (length of stay, occasions of service and patient outcomes),
“Yes we need to publish papers, but the [health] trusts are much more interested in whether we are actually improving service delivery, contributing to training, or whether we have actually implemented a quality improvement program that is of local relevance” (Participant 19, UK).
This dissonance was echoed by Australian participants, “…I think historically …we do basic science and then you look for a use for it and really it should be the other way around – we have got a clinical problem and how do we address that…” (Participant 27, Aus). This highlights how RTCs effectively need to have one foot in healthcare improvement and one in research, investing in healthcare improvement initiatives as well as traditional rigorous studies. In the UK, local healthcare providers need service evaluations and rapid cycle improvement studies in their collaborations with the ARCs, but these research approaches are not always valued by the academic sector, which often requires more rigorous research: “…Innovation does not have an evidence base to support it. It is about trialling and taking risks with new things. The academics need to have a whole rationale for why something would work.” (Participant 13, UK).
Participants from Australia also identified challenges in ensuring health research is relevant to the needs of front-line staff, “…we weren’t interested in somebody developing an App or improving the specificity of a test. We wanted some change to a model of care that could be implemented straight away if it was shown to be useful” (Participant 37, Aus). Being subject to traditional research metrics means that academics continue to privilege discovery research and clinical trials, which often comes at the expense of implementation and healthcare improvement studies that include processes for stakeholder engagement and co-production of research.
This has had important consequences for the ARCs and means that, although they were established to build multi stakeholder and co-produced research, their impact continues to be measured by the more traditional academic metrics. In fact, the ARCs that did focus more on co-creation and collaborations in their first round of funding experienced particular challenges in demonstrating traditional success metrics after working with multiple stakeholders over a five year funding cycle, “They didn’t get the findings that were appropriate because it takes time … for that less mainstream model” (Participant 12, UK).
The use of traditional metrics was also a significant issue for the Australian RTCs, “The NHMRC may talk of translation ...translation isn’t really funded … there is still no incentive to do that” (Participant 32, Aus). However, participants did refer to the national AHRA alliance between RTCs and described how it was enabling, “…a clear framework that we can all be accepting nationally about knowledge translation and impact so we … don’t have a lot of duplicated effort or repetition when we are trying to report these things” (Participant 29, Aus). The MRFF was identified as a potential enabler of applied research and translation, “…you could almost see the MRFF as an NIHR [National Institute for Health Research] equivalent for Australia … with NHMRC at the discovery end and the MRFF in the middle, translation approach” (Participant 29, Aus).
Participants from both countries identified difficulties in trying to demonstrate clinical and community impact, when constrained and measured by traditional academic metrics. They also identified challenges demonstrating both national and local impacts, “for the national they need big studies, big papers, big impacts and yet their vision was that it [the ARC] would have local relevance” (Participant 21, UK).
There is clearly a need to establish impact and success criteria that capture participation, collaboration and co-production, and are relevant to and prioritised by stakeholders and funders, but, “…how do we show government – like how do we collect the data that’s needed to show … we are being moved from the research front to the practice front” (Participant 20, UK). Ultimately, healthcare and academic metrics need to integrate and align to achieve a more holistic understanding and model of impact in both local healthcare improvement (a priority for health services) and larger transformational programs and rigorous traditional evidence (favoured by researchers).
Theme 2: Different models of leadership
Participants from both countries concurred that the complex, cross-sector collaborations that RTCs seek to embed, and their mission to translate evidence into practice, deems leadership a crucial determinant of their success. In the academically led UK ARC models, many RTC directors had academic backgrounds with demonstrated excellence in research that largely informed the Centre’s strategic themes. Here, academic metrics may contribute to now largely discredited ‘top-down’ leadership approaches, where themes are developed and chosen according to a ‘push’ model, “…not because they naturally hang together …but because the kind of CLAHRC period which started four years ago, wanted each theme led by an internationally recognised sort of research leader” (Participant 18, UK).
Participants affirmed a need to re-align this ‘top-down’ leadership approach, “It is very much top down and it’s quite interesting they actually got…leadership development programs to try and create distributed leadership. But then there still seems to be quite a lot of control from the top” (Participant 13, UK). Distributive and collective models of leadership were identified as an important alternative, especially in Australia where the RTCs are health service led, “…there’s no question in my mind that the most powerful leadership in these kinds of organisations are leaders who are able to work collaboratively with others – who’d distribute leadership” (Participant 29, Aus). Some of the key personal qualities that support this leadership approach were identified, “…I don’t assume that I have the answers, so I’m there primarily as a facilitator and I respect the process and prioritised outcome and advocate for it, even if it’s not what I think is most important” (Participant 38, Aus).
Training needs to challenge the notion of leadership in hierarchies and instead focus on “…handling institutional complexity … because you’re bringing together so many different parties and interests and their job is about being skilled as kind of like a conductor” (Participant 29, Aus). The importance of leadership provided by middle managers was also emphasised, “Often the middle management are the people that set the strategy for the organisation and that actually facilitate a lot of the new collaborations….the Executives, are too busy … they’re away or they’re just engaged with other things – I think we could do with a middle [layer of leaders] … there isn’t really a next level” (Participant 24, Aus). This also applied to managers situated in health services, “…there hasn’t been enough focus on building the capacity of managers” (Participant 26, Aus). On the basis of equity, the dominance of women in the healthcare workforce, and the preference of women for distributive leadership, participants identified an urgent need for “…training in leadership and development of leadership in women” (Participant 38, Aus), who are often operating in the middle level management of RTCs due to well-recognised barriers to career advancement and structural issues in medicine and health care that have yet to be overcome (20).
3: Public-patient involvement and research co-production
Participants were unanimous on the importance of stakeholder engagement, yet they identified that Patient and Public Involvement (PPI in UK terminology) or Consumer and Community Involvement (CCI in Australian terminology), researchers and health professionals often lack training and knowledge about research and healthcare improvement. There was consensus on the need to improve understanding of what PPI or CCI actually entails, what is most effective, and what processes are needed to guide meaningful and genuine co-production of research, translation, and healthcare improvement with communities.
The UK has made significant advances at systems and grassroots levels, in progressing meaningful PPI, when compared with Australia, where system level changes are in their infancy and efforts remain fairly limited. In the UK, PPI advisors described system level changes, including that PPI is mandatory for all research grant applications funded by the NIHR. Other system level strategies for advancing PPI included frameworks and a collaboration with the NIHR to produce guides and conduct reviews that support PPI endeavours. The James Lind Alliance (JLA) was cited as an example with a framework and toolkit for bringing together patients, carers, and clinicians in priority-setting exercises to identify ‘treatment uncertainties’ and enable meaningful engagement. The JLA is, “…a very specific method for making those joint priorities, but that’s only for a number of different areas [treatment uncertainties]” (Participant 5, UK). Another NIHR initiative described by participants was the “Going the Extra Mile (Breaking Boundaries)” Review (20) that identifies characteristics of co-production and principles for PPI in healthcare research,
“…we have very strong policy support at the highest levels and so when we did our Breaking Boundaries Review …. which was about reviewing the landscape nationally and internationally… we worked with the whole review panel to create recommendations which were very specific and are now being rolled out across the NIHR” (Participant 6, UK).
All the ARC participants reported having PPI as separate or cross cutting themes, “… the level of PPI within CLAHRCs is high …. We meet three times a year and we meet with the NIHR and we provide a report where we share what we’re doing” (Participant 6, UK). In the UK, PPI advisor positions are publicly advertised with clear role descriptions (22) and come with financial support, “…which is 20 pounds per hour, 75 per half day, 150 pounds for a day and we are very transparent about what’s involved” (Participant 12, UK).
CCI was identified as crucial in Australia for the ongoing evolution of the RTCs “…consumers should be involved in every formal part of the structure”. However, there were diverse views about what PPI in research and healthcare might actually look like or how consumers might have a role in the national research agenda, “The consumers are very focused on the local experience …our projects are often migrating into different contexts” (Participant 25, Aus). While consumers have had limited involvement in healthcare planning and advocacy in Australia for the past decade, participants expressed a low awareness of how to co-produce research and no formal processes existed for recruiting or training CCI members. The question of how to ensure representative consumer voices was also raised, “…The challenge, of course, with community engagement is who are the right representatives” (Participant 41, Aus).
Australia has had very limited policy or funding incentives for CCI, and limited training and experience, but system, organisational and individual level strategies are now being prioritised nationally, with the RTCs designated a significant role. Indeed, the RTCs have come together nationally to create a CCI framework, establish priorities, undertake a national scoping exercise and co-design strategies including training and a knowledge hub that will accelerate and deliver culture change in this area. The RTCs have drawn and learnt significantly from the experiences in the UK RTCs in this work.
In the UK, participants reported progress in measuring the impact of PPI. For example, the Guidelines for Reporting the Impact of Patient and Public Involvement in Research (GRIPP2) have been developed in the UK to improve the transparency and reporting of PPI impacts,“…they’re [the NIHR] keen to create a consistency of reporting” (Participant 6, UK). Narratives and case studies were also utilised in the UK to improve understanding of how PPI affects the experiences of patients, researchers and the public. The question of how to capture less formal impacts of PPI remains elusive, “Sometimes someone will say to me that I didn’t say anything in that meeting and I said …but that’s fine because the fact that you were there changed everybody’s way of thinking and so how you measure that …I don’t have the answers” (Participant 6, UK). Another challenge for PPI advisors related to the time-frames needed to develop research proposals and protocols, “…it takes the academics two months to get their heads around what they can do, what’s feasible” (Participant 6, UK). Feedback to PPI leads from ARC members was also reported to be inconsistent and potentially tokenistic, raising questions about the authenticity of PPI in some instances, “…I was really shocked at the statistic of 50% of them [PPI advisors] being asked to be involved and never hearing anything again…”. (Participant 17, UK). Of more concern is the fact that, “…A lot of them had not even been told that the thing [proposal] they helped with had been funded.” (Participant 17, UK) More work is needed to close the loop and establish true partnerships with patients, public, consumers and communities.
Theme 4: Workforce development
There was wide agreement among participants that the ability to implement and translate research evidence into practice, and to strengthen collaborations between research, policy, and practice requires particular skills and capabilities. Key themes for workforce development included a range of ‘global skills’ and the potential importance of dedicated roles: knowledge brokers (or dedicated translation roles).
In the UK, knowledge brokers were deployed in several ARCs or in partner organisations as intermediaries between academics and healthcare providers. Their roles and titles varied across ‘diffusion fellows’, ‘improvement fellows’, and ‘boundary spanners’. One participant explained that, “Improvement science fellows are the navigators, while knowledge brokers facilitate diffusion” (Participant 11, UK). In this context, knowledge brokers were seen as having a broader remit than improvement fellows, but their exact roles were somewhat blurred. Despite variation, knowledge brokers largely operated outside the organisational hierarchy and were seen as making a valuable contribution – especially in working across professional boundaries, “…The people that emerged as being key in terms of having a brokerage role were the ones that had a hybrid background themselves” (Participant 11, UK).
However, there was variation across ARCs in how these roles operated “We had people from a range of quantitative and qualitative backgrounds, and a mix of clinicians and academic backgrounds, so they could mix their skills and projects and share their skills. These people are in demand.” (Participant 18, UK). However, diffusion fellows and knowledge brokers did not always have clinical ‘credibility’, “…they are relatively junior coming into the world and a setting that is highly professional” (Participant 21, UK).
The Fellows themselves acknowledged challenges with their bridging roles, especially the tension between the independence and isolation of the research-related dimensions of their role, and the inherently interdependent nature of knowledge brokering, “…I deal with the clinicians and they ask me about the research …but it’s my research not our research … ” (Participant 13, UK). They also identified tension between the independence and isolation their role entails, You do have the independence of your time [but] you are quite isolated and it’s like doing the PhD which is very, you know, your project” (Participant 13, UK). Another concern was a lack of clear career progression / paths for knowledge brokers, “…we need to be moving upwards and developing sort of middle grade faculty appointments because obviously we now have quite significant numbers of these people who have come through our CLAHRC” (Participant 21, UK). Career progression was a particular issue for knowledge brokers situated in clinical settings. The need for a critical mass or ‘army’ of knowledge brokers to ensure their work is sustained was identified, as was their role as change agents, “They need to want to change things. They need to be in the boat and rock the boat, but they can’t fall out of the boat. They have got to do it from within, embedded in the community but disposed towards change” (Participant 14, UK).
In Australia, universities have traditionally developed academic chair roles – positions funded by both academia and health to provide leadership in research and clinical care and to drive integration. Participants identified a greater need for a, “…hybrid clinician that leads and owns the research, but has got a strong set of academic credentials that we’ve [the AHRTC] helped develop” (Participant 28, Aus). In Australia, there has been little to no funding for health services, implementation or applied researchers (<5% of NHMRC funding), with a major gap in the workforce and skills. The funding models have also focused in the past on traditional research metrics, which make it difficult for active clinicians to compete with full time academics. Whilst this is changing, in this context, participants expressed concerns about the challenges experienced by clinical academics, “… I think we’ve lost a bit of ground…they are doing two impossible jobs, they are trying to be a good clinician and trying to be a good change agent and researcher … we are losing clinical academics” (Participant 32, Aus).
In terms of skills, UK participants identified program and service evaluation as crucial workforce capabilities for RTCs. This includes more than traditional program logic approaches or the efficacy and effectiveness paradigms of implementation research. Participants identified implementation frameworks such as RE-AIM and the Consolidated Framework for Implementation Research (CFIR), but also reported a low awareness and skills in these methods, among both clinical and academic staff.
Global skills such as digital health, communication, team-work, priority setting and leadership were also deemed important, “You need the communication skills, you need the leadership skills to influence people, you need to listen to the local authorities, and you need the technical skills to get the evidence.” (Participant 15, UK). Diplomacy skills, such as the ability to build and maintain positive relationships, link and network with others, and deal with conflict effectively, were important for maintaining partnerships with local authorities, governments, stakeholders, and end-users. Also, qualities such as, “…emotional intelligence, the empathy, the interest in those around them” (Participant 11, UK) were identified as important, as were team based models of care and project management.
Participants from one UK ARC described their program of education and workforce development on implementation that includes both accredited academic courses and more tailored workforce programs for clinicians. Implementation and improvement knowledge was embedded in all their themes, with a discrete cross-cutting theme in implementation science that enabled them to develop a suite of tailored education programs. These programs included a Master in Implementation and Improvement Science, a three-day Masterclass and whole day meetings with PPI representatives to clarify processes for identifying gaps and research priorities. Flagship programs, including mental health, were identified and initiated, “…we realised we needed to be offering a number of things’ (Participant 21, UK).
In Australia, with a more limited workforce in applied research, participants tended to cite traditional research skills as important, “…we run a whole program of courses on cost-effectiveness, on implementation, on study design, on statistics, meta-analysis…” (Participant 25, Aus). However, the need for workforce development programs to also address informal or global skills such as negotiating and communication was recognised, “…the intangibles, professionalism, patient safety, patient experience - professionalism is a big topic …how doctors interact with themselves and their patients” (Participant 41, Aus).
Overall, there was consensus that workforce development programs need to be tailored and that traditional unidirectional education alone is insufficient in the absence of opportunities and motivation to apply new skills in clinical settings, “What is the right approach to upskill people? It is not just sitting in a lecture theatre and giving them knowledge. You need to help them absorb this knowledge by taking it into their environment” (Participant 15, UK). This highlights the importance of integrating education and learning into healthcare, “It is more than workshops. It is situated learning. It is about them doing real life projects, leading them, and learning as they go along” (Participant 14, UK).
The UK experience is instructive in terms of recognising how partnerships between the ARCs and health services have facilitated opportunities for applied learning and the testing of new interventions, “…most of them work as incubators, so early incubators or later stage incubators”. There is significant potential, therefore, for RTCs to operate as ‘test beds’ for implementation and healthcare improvement, “It is about the ability to experiment, learn from mistakes, reflect, lead, and share collective responsibility” (Participant 15, UK).
Theme 5: Barriers to collaboration
All participants identified that collaboration was fundamental for RTCs in both the UK and Australia. However, in the UK, participants were more likely to identify collaboration as a key endeavour. In Australia, where RTCs are health service led and governance includes both health services and academic institutions, participants saw collaboration as integral to their structure and had a stronger focus on a ‘translation’ mission, consistent with policy and funders positioning of these entities. In the UK, “…it’s not about translation of existing research but about development of partnerships between a variety of stakeholders in order to find ways to improve health…” (Participant 20, UK).
However, a number of barriers to collaboration were identified. In the UK, ARCs compete and have a 5 year cycle of funding, presenting significant constraints to collaboration, “…the funding cycle can disrupt relationships as they start to get established” (Participant 7, UK). Participants also described how ARCs are annually ranked against each other often on traditional metrics, “We are not supposed to be competing against each other, but then when we submit the annual report and the NIHR funders give us feedback, they rank us all, and everyone knows what their number is” (Participant 4, UK). The focus on traditional competition, academic metrics and funding models for ARCs, means that genuine collaboration is not always rewarded, “Collaborating across CLAHRCs is not a priority because you are all competing for the same pot of funding” (Participant 11, UK). Even collaboration within ARCs is challenging, “…the sheer practicalities of it. Someone’s got to get from A to B” (Participant 7, UK).
The need for more national collaboration between ARCs was identified as crucial in any future iterations, “…you could have your individual CLAHRCs but you could have certain national themes like workforce … as priorities. And each CLAHRC would have to commit a certain amount of funding to those national priorities” (Participant 10, UK). One participant commented on the irony that, “…in some ways it might be easier to have an international network rather than UK based one, because one of the challenges in the UK is they compete for the same funding” (Participant 12, UK).
In Australia, there was evidence of direct learnings from the UK and a focus on avoidance of direct competition seen in the UK models. Here the accreditation by NHMRC (based on meeting criteria, not on competition across RTCs), the strong national collaboration afforded by the AHRA, and the agreement to award equal funding by the MRFF to all NHMRC accredited RTCs, has created a system whereby RTCs can genuinely collaborate and avoid competition. This has also allowed the Centres to jointly establish a range of national priorities, has facilitated collaboration across Centres, reduced competition and duplication, and accelerated sharing and the impact of the RTC work, “…collaboration from the national alliance of the nine centres has been remarkable…when we got our funding this year every centre director just said we will commit to continuing to implement these national frameworks…” (Participant 23, Aus). However, there was acknowledgement from some participants that a certain level of competition might continue and that Centres would need to identify where there was most value in collaboration. National system level initiatives include, “… setting up a national data record and sharing - agreeing on a set of privacy principles … but at a state level there are things …where we’ll compete” (Participant 38, Aus.).
Participants in Australia expressed a strong commitment to the AHRA and willingness to strengthen their national collaboration, “…there is a mechanism to firstly find out what’s going on across the other sites … we can be strategic” (Participant 28, Aus). In addition, the alliance was identified as a key factor in improving engagement with government, “…when we are having those individual centre discussions with governments and our own state governments we actually have come together and shared some of the principles, difficulties and communications so we can actually then go back with a shared voice” (Participant 37, Aus). Participants reported that the AHRA has convened national steering committees (to address system wide priorities) and different RTCs are working together to advance shared knowledge around these priorities. These include i) data driven health care improvement; ii) CCI; iii) clinical research facilitation; iv) Indigenous capacity building; and v) health services research and workforce development (17). Indeed, multiple participants maintained that AHRA has been the most significant enabling factor for collaboration and the acceleration and sharing of learning across Centres in Australia, “It’s open discussion, decision making and the delivery of the national frameworks …when we got our funding this year every Centre Director just said we will commit to continuing to implement these national frameworks…” (Participant 36, Aus.).
In the UK, high staff turnover was a factor identified as a barrier to collaborative relationships. The National Health Service (NHS) appears subject to political ideology; attempts to significantly reform the NHS are common and frequent. Even at a regional or organisation level this was identified as a significant challenge. Participants described how the NHS has experienced a number of leadership changes in recent years and this creates significant disruption, as researchers need to continually re-establish and re-strengthen their relationships, “the NHS is constantly changing, and people move on so quickly there. The four main NHS trusts with whom we work now have different chief executives and different people on their boards.” (Participant 9, UK). Staff attrition also impacts on priority setting processes, “Although we have annual stakeholder meetings, we hardly ever get the same audience. It’s always a different group of people” (Participant 16). The UK experience also highlighted structural barriers to collaboration, whereby ARCs themes can reinforce internal silos, “We are always trying to cross-refer and think about how all of the themes are working and whether aspects of what one theme does can have an impact on another. But often it doesn’t happen because they are quite specialised” (Participant 4, UK).
Relationships between leaders and their partners were crucial factors that enabled collaboration. In the UK, “What makes the difference is that the leaders are willing to put in the time and agree to communicate with our partners, spending many hours at coffees, dinners, lunch, and breakfast meetings to slowly build those relationships so you can get to the stage of trusting each other and understanding our visions” (Participant 20, UK). The emphasis on the importance of relationships highlights the challenge, but also the importance of capturing more informal types of communication as key enabling factors for collaboration, “We talk to our AHSN lead on a weekly basis, it is the pick-up-the-phone-and-ask-a-question type of relationship that makes a huge difference” (Participant 20, UK). There is a need to ‘unpack’ what these collaborations actually entail and how their impact and shape may be captured, “...what will these collaborative practices look like, how do we actually even, where do we begin …you know to unfold that” (Participant 13, UK).