Twenty four individuals or groups were invited to participate. Seventeen individuals took part in total, nine via individual interviews - and the rest via group-interviews (table 1). Interviews were conducted between October 2014 and February 2015. Three of the individual interviewees did not give permission for their interviews to be recorded. For two interviews the recording equipment failed so in total seven interviews (two group and five individual) were recorded and transcribed. We used contemporaneous field notes from the remaining five interviews.
Table 1 – Summary of interviews
Interview
| Number of Interviewee(s)
| Role
| Primary Care Clinician
|
Interview A
| 3
| Guideline Developer
| No
|
Interview B
| 1
| Guideline Developer
| No
|
Interview C
| 1
| Guideline Developer
| No
|
Interview D
| 2
| Primary care support and quality improvement
| Yes
|
Interview E
| 1
| Primary care educator
| Yes
|
Interview F
| 1
| Primary Care Commissioner
| Yes
|
Interview G
| 3
| Research Charity
| No
|
Interview H
| 1
| Chair of public funding panel
| Yes
|
Interview I
| 1
| Chair of public funding panel
| No
|
Interview J
| 1
| Health Technology Assessment
| No
|
Interview K
| 1
| Journal publishing systematic reviews
| No
|
Interview L
| 1
| Journal publishing systematic reviews
| No
|
Four main themes emerged from our interviews: How evidence is used; aspects of trials considered when assessing evidence; views on PRECIS-2 domains; and perceptions of pragmatic and explanatory trials.
How evidence is used
Interviewees who applied evidence from randomised trials in a clinical setting reported using evidence differently from those who produced evidence synthesis or guidelines. The latter followed a formal process starting with a systematic way of identifying evidence, critical appraisal and then production of a summary of the evidence as a review or guideline. Challenges in applying evidence came from combining the results of heterogeneous evidence and deciding whether evidence is applicable to the question of interest. In overcoming these challenges judgement has to be incorporated into the formal processes. The greatest area of judgement was around whether evidence is applicable to the question of interest.
"... the guideline development group members look at the evidence tables. And make a, what we call, a considered judgement on evidence from the trials, taking into account other aspects, whether it’s generalizable, whether it’s applicable." (Interview C, Guideline Developer)
For those applying evidence in a clinical setting, sources of evidence included guidelines or evidence summaries as well as less formal sources such colleagues or experts in a particular field. Due to time pressure and a need for accountability regarding decisions about patient care, the results from single studies were not commonly used to influence practice. We found a mistrust of early evidence due to the potential of trials to show greater benefit of new interventions than would be found in practice. To overcome this challenge some individuals employed a deliberate strategy of waiting before implementing new evidence. A further challenge in applying evidence was the limited resources to implement novel interventions.
"So most of the time, what you’ll find is on my computer screen on how, you know, NICE [The National Institute for Health and Care Excellence] or CKS [Clinical knowledge summaries] open up in a separate window, that I’ll just refer to if I need to with every patient."
(Interview E, Primary care educator)
“…let’s just take the new oral anticoagulant agents … There’s been some early meta-analyses, but everybody feels the early trials were always more optimistic…
(Interview D, Primary care support and quality improvement)
And then the question is, do you wish to be an early adopter, are the advantages so great that you want to take a risk or would you like to do it later.”
(Interview D, Primary care support and quality improvement)
Perceptions of pragmatic trials
Interviewees perceptions of pragmatic trials included enthusiasm, mistrust alongside limited knowledge of the term and misconceptions about its meaning. Interviewees from public research funders and journals were most enthusiastic about pragmatic trials, showing existing familiarity with the concept and positively expressing that those are the sort of trials they would be most interested in.
“We are more interested in funding pragmatic trials than explanatory and would like to see trials which were as pragmatic as possible...”
(Interview H, Research funder)
Interviewees from the research charity, involved in primary care education or from CCGs, and some interviewees involved in guideline development had little pre-existing knowledge of the term “pragmatic trials”, expressing misconceptions as to what the concept meant or it being a term they were not familiar with prior to the interview.
Amongst those who used evidence from guidelines or evidence synthesis, pragmatic trials were welcomed as they simplified the judgement around whether evidence is applicable to clinical settings. There was concern, however, that they could be significantly different from trials in the existing evidence base leading to heterogeneity which could make meta-analysis more difficult.
“Pragmatic trials may have to be considered separately in meta-analysis due to heterogeneity with other trials”
(Interview J, Health technology assessor)
“I can just imagine that people would be really thrilled to see a pragmatic trial, of diabetes or something, [set here] or something and that people could then use it... It would be so much better for recommendations."
(Interview C, Guideline Developer)
Pragmatic trials also carried negative connotations, sometimes being viewed as inferior to more traditional approaches to randomised trials. There was also concern that the degree of pragmatism of trials could be used to manipulate the systematic review process.
“During discussions of evidence it can be difficult to consider the relative pragmatism of evidence – sometimes it is used by people to try and exclude evidence that doesn’t agree with their point.”
(Interview I, Research funder)
“Pragmatic can be a dirty word when describing trials, people like to shoot at them…”
(Interview K, Journal publishing systematic reviews).
Pragmatic trials were conflated with complex interventions and with alternative trial designs such as stepped-wedge or cluster randomised. Some interviewees also drew a distinction between pragmatic trials and randomised controlled trials.
"...to include these kind of trials, you know… the guideline developers would have to be really careful to explain the difference between this and RCT [randomised controlled trial]."
(Interview C, Guideline Developer)
Views on the PRECIS-2 domains (fig 1)
The population in the trials was by far the most discussed aspect of a trial’s design, with interviewees preferring trial populations to have few exclusions, including patients with comorbidities and older patients. Population is handled by PRECIS-2 over three domains: eligibility, recruitment and setting.
"Well I think, first of all, they want to know that people, that patients like mine, are entered."
(Interview D, Primary care support and quality improvement)
"So again, if you do it in your tertiary centres, then it becomes almost inapplicable in primary care, because I don’t have those resources. I don’t see all of those patients at that stage in that illness. I see them either way before or way after they’ve seen the tertiary care people. So yes I think setting is very important."
(Interview E, Primary care educator)
Aside from its part in determining the population in trials, Recruitment was an aspect of trial design not explicitly considered by most interviewees when assessing evidence. Amongst those who did consider the impact of recruitment on the relevance of trial results, there was concern that very intensive recruitment could bring being people into trials who would not usually present for treatment in routine practice.
“…we wouldn’t assess recruitment routinely, but it’s about whether recruitment is applicable to the question that we‘re trying to address.”
(Interview D, Guideline Developer)
"Sometimes recruitment can be too intensive and bring people into the trial who would not usually present for a condition"
(Interview I, Chair of funding panel)
The organisation domain focuses on the level of expertise and resources made available to deliver an intervention compared with what would be available in practice. Here, we identified a tension between a pragmatic and an explanatory approach. Those applying research in practice were more concerned about resourcing issues and preferred trials to test interventions that could be implemented with the limited resources available in primary care. Those from the research charity or guideline developers believed there was also a place for evidence from trials of interventions requiring resources over and above those currently available. They felt that sometimes research showed that a resource intensive intervention was effective could lead to those resources becoming available in routine care.
"The only thing in primary care that would be a limiting factor is the resources aren’t this sort of overflowing bucket"
(Interview E, Primary care educator)
"And sometimes groups will make the gold standard recommendation and that will push forward what resources are brought in"
(Interview C, Guideline Developer)
For flexibility of delivery interviewees producing evidence synthesis or guidelines favoured reduced flexibility as this allows more understanding to be gained as to what is causing any effect, makes it easier to include a trial in a meta analysis, and also reduces bias from other treatments being initiated. Reduced flexibility was also preferred as it allowed greater understanding of what the intervention being delivered in the trial actually is. For the clinicians interviewed there was an appreciation and expectation that trials would have less flexibility in the way interventions were delivered.
" After ten years of doing this I would prefer to see strict control [in how interventions are delivered] but if there are variation they need to be described properly so in attempting to make sense of this you can see what has happened."
(Interview B, Guideline Developer)
"We appreciate that you’ve got to stick to strict guidelines when you’re doing the research, otherwise it doesn’t become very accurate at the end of it. So as long as it’s not hugely different, we appreciate there is a little bit of leeway in real life, but we wouldn’t expect that in a clinical trial."
(Interview E, Primary care educator)
Poor adherence to interventions in trials was a concern as this can reduce the potential effect of a successful intervention. For some, best practice with regard to flexibility of adherence was for adherence issues to be identified prior to the trial and in the trial itself no extra steps to be taken to improve adherence. Others, notably the interviewees from the research charity, suggested knowing an intervention can be effective can lead to measures to help adherence being developed.
“Researchers shouldn’t include intense follow up to ensure adherence. Steps should be taken to collect as much primary end point data as possible but this is separate to ensuring that people adhere to the intervention.”
(Interview I, Chair of funding panel)
“The more flexibility there, the more you’re going to actually mask any real effect because of the amount of variation... ...find out why people aren’t adhering and what can we do to try and help people adhere to that particular exercise programme, for example. So, you know, I feel that the flexibility [of adherence] should be pushed tighter."
(Interview G, Research charity)
For follow up we identified a balance to be struck between collecting data that is useful for research and follow up influencing participant behaviour or increasing the burden on those taking part.
" ...in terms of having quite a lot follow-up, you can get some really useful answers and actually you probably want to do that. But it’s thinking about the way in which you follow up and so that you’re not actually influencing their behaviour and their any sort of clinical outcome through taking those measurements."
(Interview G, Research charity)
Where primary outcome and primary analysis was discussed all Interviewees favoured a more pragmatic approach with patient centred primary outcomes and intention to treat analysis. There was concern expressed about primary outcomes being measured at too early a time point.
Comments on applying PRECIS-2
The guideline developers, research charity and research funders considered PRECIS-2 a useful tool, with the guideline developers saying that it covered many of the areas of judgement they were required to make and those involved in research funding commenting that being able to justify design decisions across the PRECIS-2 domains would strengthen funding applications.
"we never used this instrument, it looks very helpful, we were often left to make the judgement... to what degree was it moving towards the pragmatic, and to what degree was it explanatory. (Interview B, Guideline Developer)
"if they thought about all of them in advance and they’ve got a good reason why it’s explanatory on this and it’s pragmatic on this one, then I think that’s, that would make a strong application coming through."
(Interview G, Research Charity)
Alternative uses for PRECIS-2 were also suggested including use as a teaching aid, trial reporting, and aiding judgements around applicability. It was noted that PRECIS-2 is subjective so care would have to be taken for reported PRECIS-2 scores to be justified.
“…actually in the paper they should put [PRECIS-2 wheels] in… you just want a quick summary. That might be helpful”
(Interview F, Primary Care Commissioner)
Issues raised by participants but not covered by PRECIS-2 domains
Outside of PRECIS-2 domains, interviewees raised issues around internal validity, in particular blinding where possible, and size of the trial - with larger trials preferred. Internal validity, sometimes referred to by interviewees as “quality”, was typically assessed before generalisability when developing guidelines or assessing research for funding, typically using risk of bias tools. Issues around internal validity were raised both by those involved in evidence synthesis and those applying evidence to practice.
"So if the quality is poor, that will get marked before people even think whether it’s generalizable or not."
(Interview C, Guideline Developer)
Of the factors unrelated to the design of trials, reporting of the trial was the most important issue outside PRECIS-2 domains, to be raised. Poor reporting was seen as an obstacle to using evidence in practice whereas good reporting was seen as something which could enhance the generalisability of evidence. Areas of reporting that were seen as most important included details of what the intervention was, how the intervention was implemented in the trial and discussion of the generalizability of results. If usual care is used as a comparator it was seen as important to report in detail what usual care consisted of.
"I think we’ve found that people have not been able to use evidence before because they haven’t been clear on what the usual care has been."
(Interview C, Guideline Developer)
Other issues that were raised included patient acceptability of the intervention, whether research was carried out in collaboration with practice and whether the intervention addressed an important clinical problem either effecting a large number of people or a particular problem for a minority that has been hard to reach.