Trial design
A three-arm feasibility RCT comparing (1) SMART + treatment-as-usual (TAU) with (2) TAU and (3) active control (‘sham’) training + TAU. We decided on three arms because this will be most informative for the envisaged definitive trial design: It is crucial to include both passive and active control conditions in definitive trials of cognitive rehabilitation, in order to detect any effects over-and-above training-unspecific effects (e.g., (26). We decided against using a waitlist control because of evidence that waiting-list allocation can have negative effects (e.g., reducing self-management efforts over the waiting period) (27); moreover, the ethical imperative for providing SMART to all participants is unclear given the as-yet unknown acceptability of this experimental intervention.
Setting
The study will be set in two hospital-based neurology outpatient clinics for people with MS: in Nottinghamshire, UK (site information available from the corresponding author).
Eligibility criteria
Inclusion criteria
- Diagnosis of MS, received ≥3-months pre-enrolment (allowing for acute adjustment, as per other trials of cognitive rehabilitation, e.g., CRAMMS(16))
- Age 18-89 (to meet the standardisation criteria of psychometric assessments)
- Cognitive difficulties as assessed by Perceived Deficits Questionnaire (PDQ) self-report (≥27) and Symbol Digit Modalities Test (SDMT) performance (1.5 SDs or more below the normative reference-value)
- Able to read and speak English to standard necessary for completing assessment and intervention procedures
- Able and willing to access a computer/tablet/smartphone with internet connection throughout the study
- Able and willing to give informed consent
Exclusion criteria
- Currently receiving cognitive rehabilitation
- Previously received SMART training
- Vision or hearing problems precluding completion of procedures
Interventions
TAU
Participants in this arm will receive treatment-as-usual (TAU). Content of TAU for cognitive concerns, based on our clinical experience and knowledge, is often informational support from an MS Nurse with signposting to the MS Society/MS Trust websites.
SMART + TAU
Participants in this arm will receive treatment-as-usual (TAU) plus the experimental SMART intervention (theory-based cognitive training).
The standard SMART programme will be adapted for intervention (20). The programme involves presenting a series of logical reasoning problems, with corrective feedback after every response, in the course of training users to derive comparative relationships among novel stimuli (“nonsense words”). The complexity of the problem-solving tasks increases in a stepwise manner over 70 stages of training, requiring increasing relational abilities to progress. Novel stimuli and task configurations are used on every trial, in order to enhance far transfer (i.e., there is no single ‘set’ of correct answers to a single set of specific problem-solving tasks). Figure 1 provides examples of SMART training trials of varying complexity. Please see Additional File 1 for the Template for Intervention Description and Replication (TIDieR) checklist(28) which has been used to describe the intervention to facilitate replication of the intervention in the future, showing the different aspects that are required of an intervention (i.e., who delivers the intervention, how often, when and where).
Participants are typically encouraged to complete the SMART intervention for 30min per session, for a total of 1.5hours per week. However, SMART is incremental and can be completed at the participant’s own pace. Each stage includes a training and a test phase. During training, the participant is required to respond correctly to 16 consecutive exemplar tasks for that stage, within a time limit (typically 30 seconds per exemplar, to ensure fluency). Tasks continue until this criterion is reached, with corrective audio-visual feedback (‘correct’ or ‘wrong’) provided after each response. Mastery of training is confirmed by performance in a test phase: wherein the learner must respond correctly to a single finite block of 16 consecutive exemplar tasks without feedback. If they pass, they move onto the next stage. If they do not pass, they are directed to repeat both the training and test phase for that stage. It is expected that it would take approximately 12 weeks to complete all 70 stages of the intervention. However, participants will not be required to complete a specific number of stages. Improved cognitive performance has been shown for participants completing just 15 stages (on average) with no clear linear relationship between training completion and outcomes(18).
The intervention will be accessible to participants via their personal computer, tablet, or smartphone with internet connection. A resource-guide prepared for this study will be provided: providing technical information on accessing the programme and a description of how to work through each stage, including a visual chart for progress-tracking. Additionally, participants will receive telephone support from an Assistant Psychologist to facilitate intervention access and use. Training activity will be automatically logged for monitoring.
Active control (‘sham’) training + TAU
Participants in this arm will receive treatment-as-usual (TAU) plus a control (sham) cognitive training intervention: Sudoku. We selected Sudoku to control for expectancy effects based on popular conceptions that it broadly improves cognitive functions(19, 29), coupled with little evidence supporting this notion(30), and its use as an active control in similar trials(31). Sham-training will be delivered online, over the same timeframe/regimen as SMART treatment, and with telephone support to facilitate access – controlling for modality, schedule of engagement, and relational support. Training activity will be automatically logged.
Outcomes
Primary Endpoints:
The Primary endpoints in this study relate to the feasibility of proceeding to a Phase III trial. The primary endpoints are based on:
- Acceptability and feasibility of trial procedures
- Appropriateness of eligibility criteria, baseline and outcome measures, audio recording of support sessions, and randomisation protocol
- Recruitment and retention rates
- Intervention acceptability, including progression and completion rates
- Estimating sample size needed for a Phase III RCT
- Completion rates of outcome measures
Secondary endpoints:
The secondary endpoints are related to developing a cost-effectiveness framework for a Phase III trial, based on:
- Establishing methods for estimating intervention resource-use and costs
- Feasibility of our bespoke service and resource use questionnaire
- Acceptability of the outcome measures for use in estimating the cost per-QALY of the intervention
Exploratory endpoints:
The exploratory endpoints are related to signal of efficacy and indicative estimation of intervention effects (effect-sizes and 95% CIs) for the following outcome measures:
Primary outcome measures for exploratory estimation of effects:
- Perceived Deficits Questionnaire (PDQ) | Subjective cognitive functioning
- Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) | Objective cognitive performance (attention, language, visuospatial/constructional abilities, and immediate and delayed memory)
- Symbol Digit Modalities Test (SDMT) | Objective cognitive performance (processing speed)
Secondary outcome measures for exploratory estimation of effects:
- Generalized Anxiety Disorder Scale-7 (GAD-7) | Anxiety
- Patient Health Questionnaire-9 (PHQ-9) | Depression
- Modified Fatigue Impact Scale–5-Item (MFIS-5) | Fatigue
- Personal Questionnaire (PQ) | Participant-identified cognitive problems
- EQ-5D-5L | Health-related quality of life
- MS Impact Scale–29 (MSIS-29) | MS-specific health-related quality of life
- ICECAP-A | Capability wellbeing
Participant timeline
Table 1 depicts the schedule of enrolment, interventions, and assessments for participants. Over the course of screening and baseline assessment procedures, consenting eligible patients will complete a cognitive assessment battery and questionnaires assessing impact of living with MS, health-related quality of life, subjective cognitive difficulties, and service/resource-use. After completing baseline assessments, participants will be randomly allocated to one of the three intervention arms (using block randomisation with varying block-sizes to balance participant numbers across arms). Participants will complete follow-up assessments at 3- and 6-months post-randomisation; after the first follow-up assessment (>3 months post-randomisation) a sub-sample of participants will engage in feasibility-feedback interviews.
Sample size
For the purposes of the current study, we will continue to approach and recruit people until we have randomised 60 participants (20 participants per arm). This should provide us with sufficient information in informing the design of a Phase Ill RCT. Twelve participants per arm would serve to inform trial feasibility outcomes and provide minimally sufficient precision for preliminary parameter estimates(32, 33). We will recruit 20 per arm because: (1) with 20 per arm, our 95% confidence intervals for key estimates (such as trial retention and intervention completion rates) will be narrow enough (<±20%) that, in terms of our criteria for progression to a definitive trial, there will be no substantive misclassification (‘red’ as ‘green’ or vice-versa; please see Table 2) and (2) at least 15 per arm is recommended for estimating variance to inform sample size requirements for a 90%-powered main trial aiming to detect effects of moderate (clinically meaningful) magnitude. Given service-activity data and prevalence of cognitive difficulties in people with MS, this target is achievable over a 12-month recruitment phase.
Recruitment
Participants will be recruited from MS clinics in two centres (Nottingham City and North Nottinghamshire). Recruitment will first be opened in the Nottingham site and, depending on the recruitment rates, we will secondarily open recruitment in the North Nottinghamshire site. The initial approach will be from a member of the patient’s usual care team (MS Nurses or Neurologists), and information about the trial will be on display in the relevant clinical areas.
Postal invitation: In the recruiting MS clinics, the clinical teams have regular contact with all people diagnosed with MS in the community. The clinical staff will identify potential participants from hospital records. An invitation letter and Participant Information Sheet (PIS) will be sent to identified patients via post or email by the MS Nurses or a member of the clinical team. This invitation letter will include study information and research team details. Patients who are interested will contact the research team – whereupon a researcher will address any questions, ensure their understanding of the study, and arrange screening procedures. The researcher will explain that the screening procedures are to check that the patient meets study inclusion criteria.
Face-to-face invitation: In addition to the invitation letter, potential participants who attend clinic visits can be introduced to the study by their neurologist or MS nurses and given/sent the PIS. People who do not contact the research team will have a single phone call by the clinical team to enquire whether they remember receiving the PIS, and whether they would like further information about the study, where possible. If they do not wish to have further information, no further contact will be made by the researchers. If, however, they wish to have more information, the clinical team will request verbal consent to pass on their contact details to a researcher, who can provide them with more information about the trial. The clinical team will record the date and time when verbal consent was obtained to pass on contact details. The research team will then contact potential participants to address any questions, ensure their understanding of the study, and enquire whether they are still interested in taking part. If so, screening procedures will be arranged – the researcher will explain that the screening procedures are to check that the patient meets inclusion criteria.
Screening appointments will be arranged as suitable to the potential participant: The first screening procedure (completion of PDQ and demographic information) can be undertaken via an online survey platform or via telephone/video call with a researcher; the second screening procedure (completion of RBANS) must be undertaken with a researcher either via video call or in-person. Potential participants will be sent a Consent Form (CF) and (if requested) another copy of the PIS in advance of the first screening appointment, so that they have sufficient time and information to understand the study before consenting to the study and engaging in screening procedures.
We will explain to the potential participant that entry into the trial is entirely voluntary and that their treatment and care will not be affected by their decision, and that they can withdraw at any time. In the event of their withdrawal, it will be explained that their data collected so far may not be erased in accordance with the University’s Research Privacy Notice and information given in the Participant Information Sheet and we will seek consent to use the data in the final analyses where appropriate. If participants withdraw from the study interventions, the study team will ask if they are willing to remain in the trial and complete trial assessments. As this is a feasibility study, recruitment will continue until at least 60 participants have been randomised (20 to each group).
Participants will not be paid to participate in the trial. Travel expenses will be offered for any visits in excess of usual care.
Randomisation and Blinding
Participants will be individually randomised at baseline (after consent) in equal proportions to one of three groups (1:1:1 ratio) using block randomisation in permuted blocks of three and six.
Treatment allocation will be computer generated via the electronic trial database in Castor EDC (Castor Electronic Data Capture, available at: https://castoredc.com). Allocation sequence is concealed by Castor EDC until randomisation of a participant. Castor EDC is used to assign participants to the different groups.
Given the nature of the intervention, participants and the intervention-facilitating researchers will not be blinded. No unblinding procedures relating to potential adverse effects are therefore required. The assessments will be conducted by a Research Fellow who will be blind to treatment allocation. We will record any instances of unblinding to assess the feasibility of blinding outcome assessors to allocation.
Data collection methods
Screening and baseline measures
PDQ(34): to assess self-reported cognition (for screening and focal outcome measurement). Perceived cognitive function, measured using the 20-item PDQ, assesses cognitive functions most affected in MS: attention, memory, planning, and organisation. The PDQ is associated with objective cognitive performance in MS(35). We will use a cut-off of ≥27(36) to identify study-eligibility.
Symbol Digit Modalities Test (SDMT) (37): to assess objective cognitive performance (processing speed; for screening and focal outcome measurement). The SDMT(37) is a symbol substitution test that examines processing speed and attention and is reported as the most sensitive test for MS cognitive problems(38). Age-, education-, and gender-adjusted norms are available(39), and these will be used to define cognitive impairment (for study-eligibility) as: scoring 1.5 SDs or more below the normative reference-value.
The RBANS(40): to further assess objective cognitive performance (attention, language, visuospatial/constructional abilities, and immediate and delayed memory; for screening and focal outcome measurement). The RBANS is a brief test of cognitive abilities across multiple domains. Normative data provide age- and education-corrected scores(41) and these can be used to further define cognitive impairment as: scoring ≥1 SD below the mean on ≥1 RBANS- composite(42). The RBANS is validated in this population and has various strengths (including alternate forms to enable repeated assessment)(43).
Generalized Anxiety Disorder Scale-7 (GAD-7)(44) and Patient Health Questionnaire-9 (PHQ-9) (45)to assess distress – an important correlate of cognitive concerns. In this population, scores ≥10 in either anxiety or depression indicate clinical distress(46)
Modified Fatigue Impact Scale–5-Item (MFIS-5)(47): to assess fatigue and its impact on cognitive, physical, and psychosocial functioning. To identify possible fatiguing effects of intervention.
Personal Questionnaire (PQ)(48): to assess patient-described cognitive problems and their everyday impact (patient-generated outcome measure)
EQ-5D-5L(49, 50): provides health state utility values (HSUVs) and is NICE recommended for estimating the cost-per-QALY of interventions(51) (informing cost-effectiveness framework).
MS Impact Scale–29 (MSIS-8D; v.2): MS-specific QoL to additionally provide MS-specific HSUVs(52, 53) (informing cost-effectiveness framework).
ICECAP-A(54): Provides capability wellbeing values (reflecting ability to ‘do’ and ‘be’ the things that are important in life) and is NICE recommended for use in economic evaluations, alongside health measures(55) (informing cost-effectiveness framework).
Service- and resource-use questionnaire developed from our previous pilot-work (informing cost-effectiveness framework). Questionnaire design has been informed by the Database of Instruments for Resource Use Measurement (DIRUM)(56) and the core resource-use items(57).
These measures were selected because they have adequate psychometric properties, have been used in other trials with this population, are brief, and capture the outcomes our PPI group felt were most important for people with MS.
Follow-up measures
All participants will be assessed 3- and 6-months post-randomisation, using the same measures as at baseline. The assessing Research Fellow will be blind to allocation (any instances of unblinding will be recorded).
Intervention resource requirements
The resources needed to deliver the intervention (e.g., providing the SMART programme online for people with MS, facilitators’ time) will be assessed via participant case-records and discussion with the intervention developers.
Feasibility interviews
After the first follow-up assessment, 30 participants (10 from each arm) will be invited to participate in a semi-structured feedback interview. We considered the timing of this interview (whether to wait until after the final follow-up assessment) and ultimately decided that (at least for this feasibility study) we should keep interviews proximal to the main study procedures and intervention – particularly given the cognitive difficulties experienced by our participants and likelihood that later interviews would strain retrospective recall. Table 1 outlines the overall schedule of study assessments/procedures (including how the feedback interviews fit within this). We will use ‘maximum variation’ sampling to select a demographically and clinically diverse sample. We anticipate theoretical sufficiency with 30 participants but will extend recruitment if needed. We will use the Theoretical Framework of Acceptability to guide our interviews and analyses(58). These interviews will allow SMART participants to feed-back what they found helpful/unhelpful about intervention content and delivery, enabling us to further refine these. For those in control arms, interviews will explore their feelings about not receiving the intervention. All participants will be asked about the acceptability of research processes (including randomisation).
Participant feedback is often subject to the ‘halo effect’ produced by a perceived lack of independence of assessors. We will therefore train and supervise two patient-partners (PPs) to help us in this process, with support from the Research Fellow who is independent to intervention delivery. The process of involving PPs in conducting interviews engenders agency and capacity-building and enables PPs to bring their unique perspectives of being fellow patients, which may permit research-participants to be more open. We successfully used such PP engagement in the CRAMMS trial(16).
Up to 10 feedback interviews will be conducted by our PPs, who will be trained and supported to conduct them (by our PPI Lead), and the remainder will be conducted by the Research Fellow. PPs will have had DBS checks and will only conduct telephone/video-call interviews, following our Trust’s Volunteer Policy.
An interview schedule will be developed/piloted with the PPI group and PPs. Interviews will be audio-recorded with permission and analysed using Framework Analysis.
Data management
All study staff and investigators will comply with the principles of the Data Protection Act (2018) in protecting the rights of study participants with regards to the collection, storage, processing, and disclosure of personal information and will uphold the Act’s/Regulations core principles. Each participant will be assigned a study identity number, for use on CRFs other trial documents and the electronic database. Personal data, research data and the linking code will be stored electronically in separate locations: this will include using encrypted digital files within password protected folders and storage media. Personal information shall be stored separately to research data and will be kept secure and maintained. Personal data will be stored for 6 months following the end of the study, so that the Chief Investigator may provide participants with a summary of the research (should they wish to receive a copy). Data generated through this study will be available for inspection on request by the participating physicians, the University of Lincoln representatives, the REC, local R&D Departments and the regulatory authorities. Routine reviews of submitted data will be conducted to identify and follow-up on missing data, inconsistent data, data outliers, and potential protocol deviations that may be indicative of systematic or significant errors in data collection and reporting at a site.
Questionnaire and eCRF data will be collected and stored using Castor EDC. Data gathered using the Castor EDC platform will only be accessible to the research team, with access rights managed by the database manager (ER). Participants will be given a unique study identifying code, so that no identifiable information need be entered.
Data-Analysis
To specifically address the feasibility objectives of the proposed programme, our analysis will draw on multiple data sources – including qualitative data from post-trial feasibility interviews (see Table 1). Quantitative analyses (conducted using R and SPSS) will be primarily descriptive, focussing on key indicators of trial and cost-effectiveness analysis viability – including recruitment/attrition rates. Variability estimates will be computed for study outcomes and used to inform sample-size calculations for the definitive RCT. To identify signal of efficacy, we will estimate (group-level) effect-sizes (with 95% CIs) and proportions achieving (individual-level) reliable and clinically significant changes. To handle incomplete outcome data when testing for signal of efficacy (whether confidence intervals around effect-sizes preclude clinically important differences) we will estimate effects using intent-to-treat linear mixed modelling – an available-case method that can accommodate missing datapoints. A Statistical Analysis Plan has been developed by the trial statistician, in consultation with the UoL Clinical Trials Unit, and will be applied with oversight from the TSC/DMEC. Qualitative data will be purposively analysed (applying Framework Analysis with support of Nvivo software) to understand participant study-experiences and identify areas for development/revision towards a definitive trial.
Monitoring
Study conduct will be governed by a Joint Trial Steering Committee and Data Monitoring (and Ethics) Committee (TSC/DMEC)
Joint TSC/DMEC
The TSC/DMEC will have an independent chair, two clinical/academic members, two Patient and Public Involvement (PPI) members, and an independent statistician. The TSC/DMEC will provide independent oversight of the study and will meet (in person or by teleconference) at least every 6 months with more frequent meetings as necessary. This joint committee will safeguard the interests of trial participants – with particular reference to safety and the efficacy of the intervention – monitor the overall progress and conduct of the trial, monitor the outcome data regularly during data collection, and assist and advise the investigators so as to protect the validity and credibility of the trial.
Harms
Adverse events of participation in this study may be: (1) exacerbation of MS-related fatigue through engagement with the intervention and study procedures and (2) elevated distress if participants find that they are not performing as well as they think they should during cognitive assessments. There are no serious adverse events anticipated with participating in this study. In practice, with respect to (1), participants will be able to withdraw at any point – one purpose of this (feasibility) study is to understand whether the intervention and study procedures are acceptable, and withdrawal due to perceived burdensomeness/exacerbation of fatigue will be informative for addressing our feasibility aims and informing future intervention and trial design. With respect to (2), this is considered to be a low-probability risk, causing minimal distress (based on our experiences of running similar trials (e.g., CRAMMS [https://doi.org/10.1186/ISRCTN09697576]) but any such distress will be managed by the assessing psychologists who are qualified to deal with distress in an appropriately compassionate manner – and will make necessary referrals (to the participant’s GP) as needed.
All adverse events will be recorded and closely monitored until resolution, stabilisation, or until it has been shown that the study treatment / intervention is not the cause.
Participant removal from the study due to adverse events. Any participant who experiences an adverse event may be withdrawn from the study at the discretion of the Investigator.
Auditing
Compliance with the protocol will be assessed throughout using central monitoring techniques. This will be achieved through routine reviews of submitted data to identify and follow-up on missing data, inconsistent data, data outliers, and potential protocol deviations that may be indicative of systematic or significant errors in data collection and reporting at a site. An interim analysis will be conducted by the TSC/DMEC to check that the project is operating appropriately – examining consent, retention, and completeness of data. The TSC/DMEC can make a recommendation for continuation/stopping or highlight any concerns or areas that may need attention (e.g., changes in recruitment practices, strategies to improve retention, or similar).
Accidental protocol deviations may occur at any time. Accidental protocol deviations will be adequately documented on the relevant forms and reported to the Chief Investigator and Sponsor immediately. Deviations from the protocol which are found to frequently recur are not acceptable, these will require immediate action and could potentially be classified as a serious breach.
Dissemination policy
Dissemination will be multipronged to inform a wide audience of patients, carers, and clinicians.
- Trial participants will be offered a lay summary of findings.
- The wider public will be informed through Trust- and study-specific websites, and via press offices for the collaborating institutions.
- We will submit findings for presentations at relevant meetings: informing the academic community and fostering interest from potential collaborators for the definitive RCT (extending the network of research sites).
- We will publish feasibility results in (open access) peer-reviewed national and international journals and professional newsletters.
- Along with our PPI Advisory Panel, we will co-write for newsletters/webpages of relevant charities that reach patients and carers directly.
Regarding outcomes, study results will directly inform protocol-development for a fully powered, definitive RCT. Should this RCT demonstrate that SMART is clinically effective, there is a clear trajectory to benefit for patients, carers, and the NHS: increasing the availability and accessibility of treatment/self-management options for cognitive rehabilitation in MS – and thereby enabling improved service provision and reducing demands on services to manage sequelae of untreated cognitive deficits. If clinically effective, the low-resource nature of the intervention makes SMART more likely to be implemented – and a full cost-effectiveness analysis in the future RCT will indicate the relative value for money to the NHS/Personal Social Services of SMART. The remote accessibility of SMART (as an online intervention) is particularly beneficial in the context of COVID-19 and would enable swift and scalable implementation – consistent with the NHS digital healthcare agenda.
Protocol version and amendments
This publication is based on protocol version 2.0_25.11.2021.
Sponsor information
Sponsorship for this study is provided by the University of Lincoln (Sponsor ID 21002).