Best-Worst Scaling (BWS) is a technique that belongs to the conjoint analysis family of methods that aim to identify peoples’ preferences and trade-offs that contribute to their choices about products or services. BWS surveys are arguably less cognitively demanding than other conjoint analysis methods such as discrete choice experiments (DCEs). BWS offer a better way of identifying the relative trade-offs between levels of attributes (compared with making judgements on groups of attributes within scenarios as in DCEs or simple comparisons using Likert scales) [23, 24]. DCEs typically present a series of alternative scenarios that convey information on multiple attributes with a range of variable levels, whereby respondents choose their preferred scenario [25]. Conjoint analyses make explicit the trade-off between the benefits of increasing one attribute at a cost of decreasing another. After careful consideration and expert advice, BWS was selected in this study as the preferred technique to elicit patient, relative/carer and public preferences.
In a BWS task, respondents are asked to select both the “best” (i.e. most preferred) and “worst” (i.e. least preferred) items in different subsets. The participant chooses “the pair that exhibits the largest perceptual difference on an underlying continuum of interest” [24, 26]. BWS has been used in a variety of health and health care settings. For instance, recently published studies have used BWS to understand patient, carer and public preferences on multiple aspects of healthcare, particularly on medical treatments [27–30]. In this study, we used the BWS profile/attribute case method (also referred to in the literature as ‘Case 2’), in which the importance of each contributing attribute of a decision is evaluated relative to all other attributes.
Two separate BWS tasks were designed. One focused on critical attributes of service organisation for EVT, and the second explored preferences on modelled outcomes for increasing the number of CSCs across England.
Selection of attributes and levels for the service organisation BWS task
Attributes and their levels for service organisation were derived from a survey (N = 147 responses) of stroke survivor (n = 27/18%); relatives/carer of stroke survivors (n = 51/35%) and other members of the public (n = 69/47%) views on service organisation for EVT in England (Appendix 1a & b). This resulted in four attributes with 2 × 2 × 2 × 2 = 16 possible combinations of attribute levels for the service organisation BWS task (Table 1).
Table 1
Attributes and levels in the service organisation BWS task
Attribute | Levels |
Secondary hospital transfer | Needed May be needed |
Clinical team expertise in EVT | Less experience/less specialised but a local service More experienced/specialised rather than a local service |
Length of stay in CSC | < 48 hours > 48 hours |
Travel time by emergency ambulance | < 45 minutes 45 minutes or longer |
Selection of attributes and levels for the modelled outcomes BWS task
Previous modelling studies indicated that increasing the number of CSCs from 24 (current number in England) to 30 would deliver both greatly enhanced population access to EVT plus health gains with a very high probability of being cost-effective [19, 20]. Therefore, three outcomes with levels informed by 24 and 30 CSCs were included as attributes in the modelled BWS task: (1) clinical effectiveness of EVT; (2) cost of setting up, maintaining and running CSCs; and (3) equity of access to EVT for eligible patients. Each modelled outcome attribute had two levels, although to reduce the cognitive burden on respondents the attribute level descriptors did not refer to numbers of CSCs.
Levels for the modelled clinical effectiveness of EVT attribute were described as modified Rankin Scale (mRS) outcomes derived from those published by the academic Highly Effective Reperfusion evaluated in Multiple Endovascular Stroke trials (HERMES) collaboration [11]. The mRS outcomes were defined as the proportion of patients that would likely be independent (mRS 0 to 2 – mild or no disability), dependent (mRS 3 to 5 - moderate or severe disability) or dead (mRS 6) after treatment with EVT within 4 hours (30 CSCs) or 4 to 6 hours (24 CSCs) at 90 days follow-up. The levels for effectiveness (41% and 52% recover with no/mild disability) reflected proportions of patients that would reach a CSC within the treatment window, and receive treatment with EVT, due to increasing the number of CSCs from 24 to 30 across England. These outcomes were taken from Saver et al [11] Fig. 1B, where the reduced effectiveness of EVT over time is estimated.
As opposed to use of currency values, the cost of EVT provision was represented using a metaphor to ensure that costs were evaluated against another hi-tech and potentially highly valued health service provision. After considerable consideration, the metaphor chosen was the equivalent cost of setting up, maintaining and running MRI scanners for 2 years. Like angiography suites for EVT, additional MRI machines involves the provision of hi-tech expensive imaging infrastructure with multiple different staff groups required to run them, yet is an item which would be at least somewhat familiar to many, even most, respondents. Costs were estimated via in-house costing work (HL) based on published data [11, 19, 31–38] and expert opinion. Equity was operationalised as proportions of patients with and without access to thrombectomy within 7 hours since onset of stroke symptoms. This resulted in three attributes with 2 × 2 × 2 = 8 attribute level combinations for inclusion in the modelled outcomes BWS task (Table 2).
Table 2
Attributes and levels in the modelled outcomes BWS task
Attribute | Levels | # CSCs |
Clinical effectiveness | 41% recover with no/mild disability; 49% recover with moderate/severe disability: 10% will die 52% recover with no/mild disability; 41% recover with moderate/severe disability; 7% will die | 24 30 |
Cost | Cost of setting up, maintaining and running 70 MRI scanners for 2 years Cost of setting up, maintaining and running 88 MRI scanners for 2 years | 24 30 |
Equity of Access | 71% will have access to thrombectomy within 7 hours of symptom onset; 29% will not 72% will have access to thrombectomy within 7 hours of symptom onset; 28% will not | 24 30 |
BWS Survey Design
The research programme PPI representative (DB) facilitated recruitment of 7 stroke survivors and their relatives/carers from the NE CRN (Stroke) PPI Panel in an iterative co-design process to develop the form and content of the BWS survey informed by best evidence on presentation of probabilities (numerical and graphical [39]) and guidelines on the design of information for people with aphasia [40]. Online pilot testing was conducted with 10 stroke survivors/carers and two researchers with expertise in choice experiments (TR, DC).
By their very design and construction, BWS tasks are repetitive; therefore, attempts were made to present the fewest possible number of questions to maximise both participation in and completion rates of the survey. However, to have enough degrees of freedom to provide individual parameter estimates across key subgroups, a minimum of 8 responses per participant for each BWS task is required [41].
A randomised block design was used to generate two blocks of 8 questions for the service organisation BWS task (with each participant only completing 1 of 2 blocks) and one block of 8 questions (completed by all participants) for the modelled outcomes BWS task. Attribute levels within each block was controlled by ensuring even numbers of levels for each attribute were included between blocks. Maximum difference scaling was used, whereby respondents stated their most and least (best or worst) preferred options for four attributes (with 2 levels each) in the service organisation BWS task (Fig. 2) and three attributes (with 2 levels each) in modelled outcomes BWS tasks (Fig. 3). In total, individual respondents were required to answer 2 × 8 (16) BWS questions
Figure 2. Attributes and levels for the service organisation BWS task
The final survey (Appendix 2) was delivered via on-line QualtricsXM platform using the ‘maximum difference scaling’ question type. A URL to the anonymous survey was generously hosted on the Stroke Association website and a link to the survey, along with information about the study was distributed to all Healthwatch services in England and for national Clinical Research Networks to distribute to patient groups. The survey also appeared in June 2019 edition of the NHS In Touch Newsletter. The introduction of the survey provided an overview of thrombectomy and aims of the study, including a link to a short film about thrombectomy produced by the BBC [42]. In order to help completion of the BWS questions, a short information film was embedded at the start explaining the repetitive nature of BWS surveys and the rationale for using this methodology in appropriate language and style for the intended PPI respondents [43].
Anonymised demographic data were collected at the end of the survey on age, gender, region of England, and respondent type (stroke survivor, relative/carer of a stroke survivor or member of the public).
Statistical Analysis
Standardised scores were calculated for each attribute level that ranged from − 1 to + 1, which represented the strength of preferences (best [positive scores] or worst [negative scores]) for the attribute level across the sample. These were calculated by establishing the difference between the frequency of an attribute level being chosen as best versus worst divided by the ‘availability’ of each attribute (the number of times it appeared across the design). Each attribute level was shown four times to each of the 105 respondents who fully completed the survey, giving a total attribute availability of 420. For example, if one level of an attribute was chosen as best (n = 320) and worst (n = 2), the difference (B-W) is 318, and the standardised score = 318/420 = 0.76 (positive preference). Corresponding 95% confidence intervals for standardised scores were generated by iteratively resampling with replacement each of 105 respondent’s data 1,000 times (referred to as bootstrapping). All analyses were conducted in Stata version 15 [44] and Microsoft Excel [45]. In other BWS studies, standardised scores have been shown to provide similar information to regression coefficients from a conditional logistic regression model, but with a greater ease of both interpretation and calculation [46, 47].