Recently, there has been an increase in the development of core outcome sets (COS) to overcome the heterogeneity in outcome selection across clinical trials for a broad spectrum of health conditions [1]. A COS is a standardised set of outcomes which should be measured and reported worldwide, as a minimum, in all studies for a specific health area/condition [2]. The standardised set of outcomes allows for the results across trials to be combined or compared, reduces the potential for reporting bias and ensures that outcomes are meaningful, relevant and useable, globally. Use of COS in trials and systematic reviews, can assist and strengthen the evidence base, resulting in improved quality of care worldwide. The development of a COS is a stepwise process that involves working with relevant stakeholders of a particular health condition/area to prioritise the core set from a larger number of outcomes [2]. Guidance for developing and reporting high-quality COS is evolving, however a number of methodological uncertainties still remain [2–6].
Involving patients/health service users from an early stage is recommended in COS development; however, the most appropriate way to facilitate inclusion remains largely unknown [2, 7]. Participation in Delphi surveys is the most popular method used for patient inclusion by COS developers, but mixed methods techniques are becoming increasingly popular [7, 8]. COS developers using mixed method techniques often conduct patient interviews as an adjunct to a systematic review of the literature to identify an initial list of potential outcomes for inclusion in a Delphi consensus survey. This reflects current COS development guidance which recommends that the initial list of outcomes is identified from multiple sources including systematic reviews of published studies, reviews of qualitative work, examination of items collected in national audit data sets and interviews or focus groups with key stakeholders, such as patients [2]. However, conducting these patient interviews increases the workload and adds additional costs, resources and time for the COS development team in the absence of clear evidence of impact on final outcome selection; as such, research on this topic is recommended [2, 9].
Once the preliminary list of outcomes has been, identified, the Delphi technique is the most commonly used method for rating the importance of these outcomes for including in the COS [2]. Using Delphi, which is an iterative survey method whereby relevant stakeholders are sent a series of questionnaires, known as ‘rounds’, and are asked to rate the importance of each included outcome for inclusion in the COS on a scale of some description, usually a Likert scale. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative provides guidance for using the Delphi technique to prioritise outcomes in developing a COS, but recognises also that there are a number of methodological uncertainties surrounding this method which need to be further explored [2, 4, 10]. For example, a variety of Likert scale scoring systems exist for use in COS development; however, it remains unclear which rating scale is the most appropriate for use in the Delphi phase of a COS development study, with one other study only identified that compared the use of two different Likert scales, a 3-point and a 9-point scale, for rating preliminary outcomes [11]. The authors of this study reported that the use of the 9-point Likert scale resulted in almost twice as many outcomes being rated as important compared with the 3-point Likert scale in the first Delphi round, but they did not, however, analyse how each scale impacted on the final COS, and the impact of using different rating scales on outcome selection, and the actual COS remains unclear. For this reason, in addition to evaluating the impact of patient interviews on outcome selection, we embedded a randomised trial within our PGP-COS development project to compare the impact of a 5-point versus a 9-point rating scale on preliminary outcome selection, and thus potentially, on the final agreed COS [9, 12, 13].
The objectives of this study were:
- To determine if including patient interviews as an adjunct to systematic review for identifying the initial list of outcomes influences the final COS.
- To evaluate the use of a 9-point versus a 5-point Likert scale in the Delphi phase of a COS development study on the number of “important” ratings received for each outcome in each round of the Delphi and on the final COS.
- To inform and contribute to COS development methodology by advancing the evidence base on COS development techniques.