The present study analyzes preferences of decision-makers in healthcare and people involved in the preparation of evidence syntheses. The preference estimates may help to derive recommendations with regards to methodology and formats of rapid reviews. While rapid reviews aim to fulfill the demands of decision-makers in a timely manner, the resulting methodological shortcuts bear the risk that results may be less reliable than those of systematic reviews [20]. Literature on the overall effects of altered methods is scarce and analyses of the impact of methodological shortcuts on review quality did not show clear results [21, 22]. A literature search limited e.g. in terms of the number of databases presumably leads to a smaller number of studies being included in comparison to classical systematic reviews [23]. Thereby, risks for selection, retrieval and publication bias can increase which can distort the results of a review [8, 12], thus potentially leading to wrong decisions or recommendations [24]. If screening and data extraction are performed by one person, errors might remain undetected. For example, a recently published trial shows that single-reviewer abstract screening misses 13% of relevant studies [25]. Finally, a lack of quality assessment of the included articles may limit the validity of a rapid review as a whole [12, 26]. Despite these limitations, decision-makers have high expectations of the validity of rapid reviews. Research has shown that decision-makers in healthcare and guideline developers expect rapid reviews to provide answers similar to systematic reviews in at least 9 out of 10 cases [24].
Because of this apparent incompatibility, it was deemed important to investigate preferences of decision-makers and people involved in the preparation of evidence syntheses with regards to methodological shortcuts of rapid reviews. The method of DCE seems to be an appropriate approach in the present analysis since the preparation of rapid reviews is usually limited by financial and temporal resources. By the fact that the pairwise comparisons constrain trade-offs, the respondents are prevented from classifying all attributes as very important.
In the 2 specific scenarios, performing the data extraction by 2 persons in conjunction with quality assurance is very relevant. Similarly, screening by 2 persons with consensus and a full-text analysis of the literature are of great importance for the respondents. In scenario 2, the inclusion of several domains (economic, ethical, social, legal and organizational issues) shows a stronger preference than in scenario 1. It is conceivable that with the longer working time in scenario 2, participants ascribe importance to the consideration of several domains. However, the higher relevance of the additional domains to be included could have also been influenced by the fact that scenario 1 and 2 deal with different topics. The attributes ‘searches in several databases’ and ‘publication period to be considered’ show comparatively low preferences in both scenarios. Based on the expressed preferences, financial and temporal savings should not be realized by reducing the number of people involved in screening and data extraction. According to the participants of this survey, preferable methodological shortcuts involve restrictions in the number of databases, a consideration of a smaller number of HTA domains or a shorter publication period. Presumably, the absence of trade-offs is reflected in the answers to the final question on 11 attributes of an evidence synthesis. Even though the data extraction by 2 persons was also rated as most important, the other individual attributes were rated comparably high, so no clear results can be derived.
The present study has several limitations. First, due to the aggregation of the originally defined 11 attributes to 6 attributes, only a selection of attributes could be examined and thus, preferences could not be derived for all steps in the preparation of rapid reviews. However, this reduction of attributes was deemed necessary to enhance the feasibility of the DCE. Second, there is some evidence for a higher proportion of employees in statutory health insurance in scenario 1 than in scenario 2 and it cannot be ruled out that the results were influenced by these differences. This unequal distribution may be due to the fact that (i) these employees have felt an affinity to scenario 1 and (ii) that the link to the survey might have been shared among colleagues.
Finally, the fact that the 3 attributes ‘number of reviewers during data extraction’, ‘number of reviewers during screening’ and ‘full-text analysis’ have a similarly strong preference in the two scenarios suggests a generalizability. However, it must be taken into account that the importance of the individual work steps in practice depends on the respective research question. For example, the benefit of including further databases might be small for certain research questions. In the case of very short-term inquiries, it might be useful to focus on the clinical domains of safety and efficacy, and the economic assessment may be conducted at a later stage. Furthermore, a possible link between the attributes should be considered. Although, as described, the attributes do not overlap as such, they are part of a sequence in which alterations might affect process steps occurring later in time. For example, there might be little point to restrict the number of databases, in order to ultimately examine a large number of domains. The analysis principally reflects the view of German users and developers of evidence syntheses.