Process and stages of RES
We present a full example of an RES in supplementary material, a snapshot is shown in Figure 1.
The key elements of the completed RES are shown in Box 1.
BOX 1: Structure of RES
- A headline summary of the certainty and relevance of available evidence
- A bulleted summary for each question addressed
- A description of the innovation
- A set of key questions
- A summary of the search process
- Detailed answers to each question addressed in terms of available evidence and its certainty and relevance.
- Bibliography of sources used to answer the questions
Timeframe and personnel
The RES is designed to produce a “good-enough” answer to contribute to decision-making in a short time-frame, rather than a perfect answer. The methods described are implemented using a median of up to two days of time for an experienced researcher with a background in evidence synthesis. More complex innovations may entail more resource for RES production. More details are provided in the framework (see supplementary information).(11)
Describing the innovation
The first stage is to briefly describe the innovation in terms of its nature and purpose (Box 2). This establishes the type of innovation (e.g. intervention/test/service delivery mechanism); the population or system that is targeted and the outcomes which should be considered. A comparator will usually also be identified through this process. This stage involves assessing and clarification of the information supplied by the sponsor.
Box 2. Example: Innovation description
Phagenyx is a device which is designed to reduce neurogenic dysphagia (dysfunction of eating).[1] This is dysphagia arising from the disruption of any of the neurological systems or processes involved in the execution of a coordinated safe swallow and occurs in people following stroke and in other conditions such as multiple sclerosis which impact muscle control. Dysphagia also occurs in people who have undergone sustained intubation for any reason. Phagenyx is classed as a pharyngeal electrical stimulation intervention and comprises a base station and a treatment catheter. It is applied over a period of days.
NICE guidance on the management of people with dysphagia following stroke states that they should be offered swallowing therapy at least three times a week, if they are able to participate, for as long as they continue to make functional gains. Therapy could include compensatory strategies, exercises and postural advice.[2]
Developing the questions
Using the description of the intervention, we formulate a series of questions based on the innovation description (Box 3). These begin with the most narrowly focused and move to wider category-based questions. These consider innovations in the same category and are key to production of a useful RES where evidence for the innovation is limited. Questions use the PICO(S) approach; defining the Population, Intervention (Innovation), Comparator, key Outcomes and Setting (where relevant).(30) The eligible study designs will always include existing evidence syntheses or, in their absence, the most relevant primary research design.
If the innovation is an intervention, then the questions will be ones of effectiveness and safety; where the innovation is (for example) a test or screening tool we consider accuracy as well as the impact on participants and health systems of implementing the technology. For complex interventions each core feature is described, and these are also taken into consideration in the question formation. When evaluating evidence for particular components of a complex intervention we are mindful of the fact that effectiveness in such an intervention may derive not solely from the additive effect of components but from their interaction with each other, as well as with the (often complex) system context.(31) We would therefore consider evidence relating to an individual component to be indirectly relevant to the innovation as a whole. There may be multiple questions of this form (to reflect different populations or comparators, for example).
Whilst we first focus on effectiveness evidence for the specific innovation being assessed, where this limited, we will explore evidence for (2) the category of innovations (“innovations like this”), and then (3) wider categories of relevant innovation (e.g. “innovations with a similar aim”). Categories are sometimes not obvious, particularly where the innovation is complex.(18-20, 22, 32) In the case of wider categories we may ultimately be looking at any intervention with a purpose similar to the index innovation or all interventions for the condition or issue under consideration (See Box 1) . These subsequent questions are designed to ensure that useful evidence can be provided where the evidence for the innovation itself is absent, limited or not directly relevant. They are also required when innovations are tests, where the evidence for available treatments should be considered as a whole in the absence of test-and-treat evaluations.(25) These additional questions are designed to be addressed only where the evidence for the first question(s) is considered insufficient. Implementation of the sequential question set is then flexible and sensitive to the nature of the identified evidence.
Box 3. Example: Key questions
- Focus on specific innovation: What is the evidence for the impact of Phagenyx for key outcomes in people with neurogenic dysphagia compared to other interventions or to usual care?
- Focus on innovation category: If there is limited evidence for Phagenyx, what is the evidence for the impact of similar interventions (pharyngeal electrical stimulation) for key outcomes in people with neurogenic dysphagia compared to other interventions or to usual care?
- Focus on wider relevant innovations: If evidence for pharyngeal electrical stimulation is limited, what is the evidence for the impact of interventions for neurogenic dysphagia more generally?
Types of evidence
Our focus is always on those study designs best able to answer the questions we have developed. We focus on identification of existing evidence synthesis (systematic reviews) where possible; where this is not possible, we focus on the most informative primary evidence. In the case of most innovations this is from comparative studies, giving priority to randomised controlled trials. Where appropriate to the questions we also include diagnostic accuracy or prognostic studies. There are also questions, especially where the focus of an innovation is on patient experience, where mixed methods or qualitative studies will be the most appropriate form of primary evidence.
Identifying evidence
We adopt a pragmatic and iterative approach to identifying evidence. This uses an initially narrow focus to maximise relevance and progresses to a broader evidence base as necessary. We search key resources including NICE guidance;(33) PubMed; and the Cochrane Library, which includes both the Cochrane Database of Systematic Reviews and the Cochrane Central Register of Controlled Trials.
We increasingly encourage sponsors to provide research evidence for the innovation, as would be the case with a submission to NICE; we routinely search the sponsor’s website. Where appropriate we will use subject/domain-specific resources, such as the webpage of a particular Cochrane Group,(34) or the ORCHA database of health apps.(35) Where required we consult with an information specialist. We also use reference checking or forward citation searching of relevant evidence syntheses and primary studies.
Critical appraisal
We use appropriate methods to critically appraise the different types of evidence we identify.
Cochrane reviews are generally considered to represent reliable evidence and we use their summaries and assessments of evidence certainty rather than re-appraising the evidence, unless there are issues around relevance. Where possible with other high quality systematic reviews we will also use the existing assessments of evidence from the review. This approach maximises the use of existing high quality evidence while improving timeliness. We consider the quality of non-Cochrane systematic reviews, using the signalling questions from ROBIS as a guide.(36) We consider the possibility of duplication of evidence between multiple evidence syntheses.(37)
Where there is no existing evidence synthesis, or we have concerns about the robustness or relevance of a systematic review, we consider primary evidence for the question. We also move to assessing primary research where the existing synthesis has only partially addressed a question, for example because eligibility criteria were narrower. Conversely, where a review has a broader remit, we may look at the included primary studies relevant to our question. In the example of the Phagenyx RES we looked at the subgroup of RCTs assessing Phagenyx within the Cochrane review of interventions for dysphagia in stroke.(38)
Assessment of primary studies considers both the capacity of the study design(s) to answer the question, and an assessment of the risk of bias in the identified studies of the study to produce overall judgements of reliability. Because of our narrow timescale, we do not undertake full assessments but, as with ROBIS, are guided by the domains used. For example, for randomised controlled trials we are guided by the criteria and considerations of the Cochrane Risk of Bias tool,(39) for other study designs we consider questions posed by tools such as ROBINS-I; QUIPS etc.(40, 41)
Relationship with GRADE
In forming judgements about the certainty of the evidence we are guided by the principles of GRADE.(17, 42) GRADE assesses the certainty of evidence through evaluation of several domains in order to produce an assessment of high, moderate, low or very low certainty. The first domain is the risk of bias in the evidence, which we consider as outlined above. This is considered alongside questions of imprecision, inconsistency, and direct relevance of the evidence, and publication bias. There are adaptations of GRADE for non-effectiveness questions.(43, 44)
Apart from risk of bias the domains most relevant to our rapid evidence syntheses are usually imprecision (because of small sample sizes) and indirectness (often a function of context): there is often insufficient evidence to determine inconsistency between studies for the initial questions because there are usually only a small number of studies. The evidence for category-level questions is often of higher certainty than the evidence for the innovation itself; here the domains of inconsistency (and completeness of evidence (publication bias)) are more likely to be considerations.
We bear in mind that where inconsistency is present (as at the wider category level), this may be a consequence of either – or both – differences in the interventions or the systems in which they are evaluated as well as differences in participants or outcome measures. While some interventions are clearly complex, even simple interventions are frequently implemented into complex health systems and this is especially true of those which would represent changes in patient management.(26) This especially includes diagnostic and prognostics test, for which we always primarily ask about the effect of testing on the people involved and their management.(25)
Imprecision is usually the consequence of small studies with insufficient participants; this results in wide confidence intervals and effect estimates which would be highly likely to change with further evidence. Indirectness is also often an issue for some or all of the evidence. Because innovations assessed are novel there is often only a partial evidence base, where the evidence may be only indirectly relevant to many of the people in the question, although directly relevant to the group represented in the studies.
Our considerations of relevance (which GRADE considers as (in)directness) are key to our assessments. In addition to the consideration of indirectness which informs our assessment of the certainty of the evidence we also consider the relevance of the evidence to the context and health system in which the innovation would be implemented – in this case Greater Manchester in the UK.