Study design
This is a pragmatic, formative, mixed methods service evaluation study to help support the implementation and sustainability of remote monitoring in rheumatology services. The study will follow a simultaneous mixed methods approach to generate both quantitative and qualitative data, which will be synthesised in the light of implementation and programme theory.
Table 1 outlines the detailed evaluation framework and metrics for the evaluation. The evaluation framework is structured based on the established generic logic model proposed by Proctor et al (13), in which patient-level outcomes are impacted by service-level outcomes, which in turn are influenced by implementation outcomes.
Table 1
Evaluation outcome and question(s)
|
Measure(s) / metric(s)
|
Data source(s) / collection method(s)
|
Patient outcomes
|
Do patients on the remote monitoring service have a positive experience?
|
Patient experience and satisfaction
|
Custom-made patient survey developed and administered electronically; patient interviews
|
Does the RM pathway influence patient willingness and/or ability to manage their condition?
|
Health Confidence Score (HCS)
|
Validated scale administered through the remote monitoring system
|
What impact does RM pathway have on disease status and wellbeing?
|
Rheumatoid arthritis impact of disease (RAID) score
|
Validated scale currently collected 4-weekly via the remote monitoring system
|
Service outcomes
|
What is the profile of patients using (and not using) the remote monitoring system (i.e. to understand potential health inequalities/access issues)?
|
Demographics (e.g. age, gender, ethnicity, postcode for IMD); RAID score
|
Demographic data from trust electronic patient records matched to RAID data collected through RM system and at face-to-face appointments.
|
Do patients and clinicians report fewer unnecessary appointments?
|
Patient/clinician experience
% of appointments timings changed (brought forward or pushed back) and/or format changed (face to face or video or telephone) pre and post RM implementation
|
Patient survey; patient and staff interviews
Remote monitoring system
|
Do patients gain savings in time and money?
|
Estimated cost analysis
|
Interviews (patients)
|
Does the service gain savings in time and money?
|
Estimated cost analysis
|
Interviews (staff); budget information - service managers/budget holders
|
Implementation outcomes
|
Do eligible patients opt-in to the remote monitoring system?
|
Number patients eligible opting in
|
Interviews with clinical team; % of all patients seen who opt-in/out
|
How often do patients complete RAID?
|
% of patients who complete RAID at 1,2,3,4, etc. months
|
Remote monitoring system
|
Do patients instructed to use the telephone helpline make contact?
|
% of patients instructed to use the telephone helpline who make contact
|
Clinical services telephone helpline data
|
What is the level of uptake/engagement by staff across clinical services?
|
Number of patients per sites referred to remote monitoring service; self-reported engagement
|
RM system; Interviews
|
How was the remote monitoring pathway developed and implemented?
|
Patient / staff perspectives
|
Evidence review; Interviews; Project documentation; Theory
|
What factors have impeded/supported its implementation?
|
Implementation frameworks; patient / staff perspectives
|
Evidence review; Interviews; Project documentation; Theory
|
Large-scale study feasibility
|
What is the feasibility of undertaking a subsequent larger-scale (impact) study to evaluate clinical, service and health economic impacts and implementation effectiveness?
|
• Recruitment and sample characteristics
• Preliminary data for sample size and power calculations
• Procedures and measures (e.g. data collection methods/tools, data access, volume, quality, data management and processing)
• Intervention acceptability and feasibility
• Resources and ability to manage study
• Description of key contextual factors affecting implementation
• Identify potential implementation strategies
|
Study documentation
|
Table footnote: please see Box 1 for the definition of all abbreviated terms used in this table. |
Stakeholder engagement and patient and public involvement
The evaluation is designed in a partnership with the clinical product owner and digital pathway coordinator (MM, ES), clinical lead (TG), a team of implementation/improvement scientists and evaluators (NS, AW, HS, OB, KW). The draft evaluation protocol was shared with stakeholders including the project clinical steering group and local commissioners, who offered inputs. Service user perspectives were obtained working with the patient and public involvement (PPI) coordinator at King’s Improvement Science, King’s College London (LD).
Seventeen GSTT service users involved in the original pilot study have consented to ongoing engagement in the evaluation. Developing plans include connecting with patients at the two other trusts involved, and with a wider group of public, service user and carers, friends and family recruited through the National Rheumatoid Arthritis Society (NRAS) database (nras.org.uk). A tiered approach is planned to engage people who would like to be involved, with three levels reflecting individual preference: participating in project meetings and contributing to project outputs, joining workshops and/or one to one discussions, and receiving updates via an electronic bulletin.
Theoretical frameworks
The study will be informed by a number of established, evidence-based implementation frameworks. These will used to guide data collection, analysis and provide a theory-informed lens for interpreting the implementation process. The following complementary frameworks will be applied:
-
The Proctor et al implementation outcome taxonomy and logic model, which links the outcomes of implementation efforts to observable improvements in services, which ultimately lead to improved patient outcomes. This taxonomy has been used to conceptualise the levels of evaluation that is being conducted (Table 1) (13).
-
The Expert Recommendations for Implementing Change (ERIC) framework of implementation strategies will be used to identify early approaches taken to implement the remote monitor service across sites, and to develop a potential implementation support package in the future (14).
-
The Exploration, Preparation, Implementation, Sustainment (EPIS) framework provides a temporal lens to explore the different stages of the implementation process; thereby, allowing the remote monitoring model to be understood from its development and initial piloting at one trust, to the current implementation and scale-up across an Integrated Care System (ICS), and sustainment (15). The framework also incorporates an analysis of barriers/drivers to implementation, as well as local and wider system-level contextual factors that may impact on them.
-
The evaluation of patient experience will be informed by the COM-B theory enabling a systematic evaluation of behavioural aspects. The COM-B theory is based on three interacting domains for behaviour change: capability, opportunity and motivation (16). Capability is defined as the individual’s psychological and physical capacity to engage in the activity concerned; this includes having the necessary knowledge and skills, e.g. self-efficacy. Opportunity is defined as all factors outside the individual that enable the behaviour or prompt it, e.g. support from a physician. Motivation is defined as brain processes that direct behaviour, e.g. goals, conscious and analytical decision-making, habitual processes and emotional response. The COM-B theory is increasingly used in evaluation research which involves patients’ behaviour and intentions in their engagement with new technologies. For example, the model has been deployed in the evaluation of eHealth, self-management interventions in patients with cardiovascular disease and persistent physical symptoms (17, 18). The model was successfully applied to the examination of decision aids in the context of shared decision-making in care for patients with rheumatoid arthritis (19).
Setting
The project will take place in rheumatology outpatient services in three National Health Service (NHS) trusts comprising six sites that collectively provide all rheumatology outpatient services across South East London, UK. The sites are Guy's and St Thomas' Hospital (GSTT), King's College Hospital (KCH), Princess Royal University Hospital - Orpington (PRUH), Queen Elizabeth Hospital (QEH), Queen Mary's Hospital (QMH), and University Hospital Lewisham (UHL). Following the 18-month pilot at GSTT in January 2019, the remote monitoring service was launched at the three trusts in October 2020. Remote monitoring will be offered to all patients with rheumatoid arthritis who meet the eligibility criteria – estimated as between 1,500 to 2,000 in total within the catchment areas of the six study sites. During the period of the evaluation 350-400 patients are estimated to be registered on the remote monitoring system.
Participants and recruitment
Both staff and patients involved in the remote monitoring service will be invited to take part. This evaluation will involve interviews with staff employed in rheumatology outpatient services across the three NHS trusts delivering the remote monitoring service (e.g. doctors, nurses, allied health professionals, administrators and service managers). Patients with rheumatoid arthritis accessing care via the remote monitoring service will be invited to participate in interviews and surveys.
Purposive sampling will be used for qualitative data collection to ensure the evaluation considers a range of perspectives that are representative of the population and implementation process. We will ensure clinical and non-clinical staff are represented from a range of sites, disciplines, and seniority. All eligible patients of different backgrounds (age, gender, ethnicity, disease longevity) will be approached. The recruitment will continue until the sample of approximately, 20-30 patients and 30 staff (estimated 10 per trust) is achieved, which is estimated to allow saturation to be met.
Data collection methods and sources
The evaluation will use (i) rapid evidence review and synthesis, (ii) service-level data routinely collected by Trusts and from the remote monitoring system, (iii) documentation produced in developing and implementing the remote monitoring service; (iv) data collected from staff via interviews, (v) data collected from patients via interviews and survey.
The rapid evidence review will involve database searches of Medline and Web of Science, combining the following Search/MESH terms: Rheumatoid Arthritis AND remote monitoring OR mobile app OR mobile applications OR smartphone app OR smartphone application OR telehealth OR tele-health OR telecare OR tele-care OR electronic health OR mhealth OR ehealth. Inclusion criteria will be research papers and protocols published in English; exclusion criteria will be articles focussing on ‘telemedicine’. Abstract and full text screening will be undertaken by two researchers (OB, NA), with any discrepancies discussed with a third reviewer; the data will be synthesised as a narrative review.
Interviews with staff and patients will focus on the overall experience of using the remote monitoring service, what works well and what does not; staff interviews will also explore implementation and feasibility aspects of introducing the service as an organisational innovation within their respective trusts. Interview topic guides have been developed incorporating the early experience of the service development team, the view of patients and service users involved in the study, and theoretical perspectives from implementation theory (see Appendix 2 for the draft topic guides). We anticipate that the topic guides will be fine-tuned after an initial one or two interviews and finalised.
The patient survey will be based on survey developed in the original pilot at GSTT to provide a comparison. The evaluation will use the RAID score (6) and Health Confidence Score (HCS) (20) as standardised PROMs for the condition. The RAID score is a validated 7-item PROM assessing pain, functional, disability, fatigue, sleep, coping, physical and emotional well-being. It will be collected every 4-weeks via the remote monitoring system (i.e. using an SMS link) for all registered users of the system. The Health Confidence Score is a 4-item validated measure for self-care health confidence, which provides self-reported information on patients’ confidence about knowledge about their health, self-management, access to help, and involvement in decision making. The HCS will be collected as part of a one-off online patient survey, which will be sent via SMS link to all registered users from the remote monitoring system. The predicted number of patients registered on the system during the evaluation is 350-400 people across all sites.
Initially, staff and patient data will be collected remotely in line with COVID-19 pandemic guidelines. However, the evaluation may use face-to-face approaches (e.g. interviews) if permitted by national and local guidelines relating to COVID-19 infection control and social distancing. Table 1 outlines the data collection methods and data sources in more detail.
Table 1 Evaluation Framework
Table footnote: please see Box 1 for the definition of all abbreviated terms used in this table.
Data analysis
For the rapid evidence review, relevant studies will be identified and relevant data extracted and tabulated in pre-formulated data extraction tables. We anticipate extracted data will include the study design and methods, the range if implementation outcomes assessed and how the assessment was carried out, the core findings and, finally, the limitations of the study. Study identification and data extraction will be undertaken by a minimum of two researchers, who will reach consensus on the process of identifying and extracting data from included studies; and resolve any disagreements via discussion (including the wider project team as required).
For the quantitative part of the study, we will use descriptive statistics (mean, standard deviation, range, percentages) to summarise patient satisfaction with the service. Administrative data (budget information, telephone helpline data, other service-level data) will be extracted in a form of descriptive summary tables and charts.
For the qualitative part of the study, we will follow five stages of framework analysis outlined by Ritchie and Spencer (24): familiarisation, identifying a framework, indexing, charting, mapping and interpretation. Interview recordings will be transcribed professionally, identified information will be removed and transcripts will be coded in NVivo. 10% of the interview data will be double coded and consensus will be reached through a dialogue. Emerging frameworks will be reviewed in light of the EPIS and COM-B theories and discussed among the researchers.