Phase 1
One hundred and twenty-one patients were recruited over three months. The completeness of the dataset was similar across centres, with minor variations. The majority of data items were >80% complete. However, 14/62 data items received >40% missing data. (Table 1) Further descriptive analyses highlighted incorrect completion of paper forms.
Three themes emerged from the focus groups: problematic data items (defined as >10% missing at >1 centre), format of clinician data forms and the role of digital data collection. Suggested solutions included minor changes to data item definitions and formatting. There were no refinements to the data items. Development of a digital data collection system was identified by all as essential.
Due to a lack of clear consensus, the original forms included a number of ways to record joint count data. This proved confusing and a unanimous decision was taken to collect joint count data on all 83 joints in a tabular format.
Table 1: CAPTURE-JIA data items with >40% missing data in paper pilot
Data item
|
% missing (if item required)
|
Relevant co-morbidities?
|
60
|
Macrophage activation syndrome?
|
100
|
Has the ILAR subtype changed since previous visit?
|
50
|
Morning stiffness lasting >15 minutes
|
42
|
History of any form of uveitis?
|
52
|
Date started uveitis medication?
|
50
|
Uveitis medication details
|
83
|
Counselled prior to new disease-modifying drug (DMARD) / biologic?
|
56
|
Enrolled in national biologic registries if new DMARD / biologic?
|
48
|
Joint count (homunculus or table format)
|
48
|
Physician assessment of systemic disease activity (VAS)
|
75
|
Erythrocyte sedimentation rate (ESR)
|
74
|
C reactive protein (CRP)
|
92
|
Plasma viscosity
|
100
|
Phase 2
The Agileware solution was exhibited at a national UK paediatric rheumatology conference (BSPAR 2019) where delegates visiting the exhibit were asked to complete a short survey (Table 2). 93% of responses indicated that Agileware was either Very Easy or Easy to use, suggesting minimal training would be required. NHS providers are at varying degrees of technical and transformational change, resulting in a mixed economy of patient data collection methodologies. A recurring theme across all responses was the limited resource / capacity with internal IT departments with an associated reluctance to take on more work and integrate external systems. Proposed solutions included: communication / discussions with individual IT departments, a national directive (for example mandated national audit) and local/national funding opportunities.
Table 2: Agileware Solution Clinician Feedback BSPAR 2019
Question
|
Responses
|
|
On a scale of 1-5 how easy did you find using Agileware?
|
Very easy
Easy
Neutral
Difficult
Very difficult
|
57%
36%
7%
-
-
|
On a scale of 1-5 how enthusiastic are you to use the Agileware system to collect the CAPTURE JIA dataset for JIA patients?
|
I would really like to
I would like to
I have no strong feelings
I have little interest
I have no interest
|
79%
21%
-
-
-
|
On a scale of 1-5 how likely is the adoption of Agileware at your hospital?
|
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
|
36%
21%
36%
7%
-
|
What is your local Electronic Patient Record (EPR) maturity?
|
EPR currently used in practice
EPR in development
Plans for future development of EPR
No plans for EPR
Don’t know
|
54%
23%
15%
-
8%
|
If patient data are collected electronically: what method do you use?
|
EPR
Dedicated database
Excel spreadsheet
|
56%
31%
13%
|
Phase 3
A) Prospective electronic pilot:
COVID-19 restrictions on ongoing NHS research and reduced staff availability resulted in recruitment from just two of the three study centres; seven paediatric rheumatology consultants and one paediatric rheumatology grid trainee participated in the electronic pilot with 38 patients recruited across Alder Hey and Newcastle Hospitals.
Data entry was reported to be neutral to easy overall, with clinicians reported completion times ranging from 11 to 30 minutes (average 20.7 minutes per form). There was a clear trajectory towards improved confidence and faster data entry with experience using the system. Data entry was significantly faster for new JIA patients; clinicians reported time-consuming delays associated with the identification of historical clinical data. Clinicians were positive about the system in a post-pilot clinician focus group, reporting that the system looks good, is easy to navigate and flows in accordance with the clinical consultation. There was universal agreement that the system is more user-friendly than paper data collection and the ability to review summary data at local hospital level was viewed as an important advantage.
It was not possible to directly export data from either hospital EPR to the Agileware system. Although the Agileware system houses data in accordance with the Data Protection and Security Toolkit (DPST) provided by NHS Digital, neither Trust had systems in place to allow new direct data exports during the study recruitment period.
Although the electronic forms were all complete, the number is too small to provide robust insight into data completeness and would not accurately represent reasons for missingness in a standard clinic setting at an overall and individual site level.
B) Longitudinal dummy data entry and analysis
A dummy dataset of 20 patients with linked data entries over multiple time points was developed by participating clinicians. Data were analysed in combination with the prospective electronic pilot data to ascertain whether the HQIP National Audit questions were answerable. (15)
Seven of the eleven audit questions could be answered within the Agileware system itself. A number of graphical additions to the database, namely percentages, ensured that the 7 audit questions can be built in individual site environments, allowing each paediatric rheumatology centre to independently monitor performance.
The remaining questions were not answerable within the Agileware system; they had to be extracted and analysed using a statistical package (Stata version 14.0). Once extracted, it was relatively simple to transform the data and produce answers to the audit questions. Due to the small dummy dataset, some more specific questions were not fully answerable, though we were able to generate proof of concept that they could be answered with a larger dataset.
Audit question analysis methodologies presented in Table 3.
Table 3: HQIP National Audit Question Analysis
Subject area
|
|
Proposed Question
|
Answerable within Agileware
|
Answerable within a statistical package
|
1. Categorisation
|
1A
|
What is the number of patients in each ILAR sub-group in the audit population?
|
Yes
|
Yes
|
1B
|
What is the proportion of patients in each ILAR sub-group, relative to the audit population?
|
Yes
|
Yes
|
2. Access
|
2
|
What is the median time for children with suspected JIA, from receipt of the referral letter in the Rheumatology department to the date of the first appointment offered in a rheumatology clinic? (modified PRH03)
|
Yes
|
Yes
|
(PRH03: children with newly diagnosed JIA should have access to a specialist paediatric rheumatology service* within 6 weeks of the referral being received by the specialist service)
|
|
|
3. Steroids
|
3A
|
What is the mean number of days to joint injection on a dedicated Paediatric GA list from date of decision to treat, for children of different ILAR sub-types? (PRH04)
|
Yes
|
Yes
|
(PRH04: Children with JIA who need to have intra-articular steroid injection(s) should wait no longer than 4 weeks for the procedure. Those needing general anaesthesia (GA) will have these performed on a Paediatric GA list.)
|
|
|
3B
|
What percentage of children of different ILAR sub-types is on oral (systemic) steroids at different times after their first Rheumatology clinic visit?
|
No
|
Yes
|
4. DMARDS
|
4
|
What is the median time from their first clinic visit to the decision to treat with methotrexate, for children of different ILAR sub-types?
|
No
|
Yes
|
5. Biologic therapies
|
5
|
What is the median time from their first clinic visit to the decision to treat with their first biologic therapy:
|
No
|
Yes
|
- for children of different ILAR sub-types?
|
|
|
- for different biologic therapies?
|
|
|
6 Uveitis
|
6
|
What is the median time from the patient's first clinic visit to the date of their first uveitis screening with an appropriate paediatric ophthalmic specialist, for patients of different ILAR sub-types? (modified PRH05)
|
No
|
Yes
|
(PRH05: Children with Juvenile Idiopathic Arthritis should have access to Uveitis screening within 6 weeks of diagnosis)
|
|
|
7. Clinic organisation
|
7A
|
What proportion of children who started a DMARD or biologic agent were counselled by a Paediatric Rheumatology Clinical Nurse Specialist (PRH01)
|
Yes
|
Yes
|
(PRH01: Children with established rheumatic diseases (and their carers) should be counselled by a Paediatric Rheumatology Clinical Nurse Specialist* before starting treatment with a DMARD or Biologic.)
|
|
|
7B
|
What proportion of children with JIA is seen in a specialist paediatric rheumatology clinic and what proportions in other clinic types (modified PRH02)
|
Yes
|
Yes
|
(PRH02: Children with Juvenile Idiopathic Arthritis (JIA) should have access to a paediatric rheumatology clinic* for follow-up appointments)
|
|
|
8. Research
|
8
|
What proportion of eligible patients has been recruited to the BSPAR Cohort Studies (BSPAR Etanercept and BCRD)?
|
Yes
|
Yes
|
C) Qualitative participant evaluation
A total of 40 CYP/parent pairs completed patient-data collection preference questionnaires (120 questionnaires received in total, one for each form type, e.g. CHAQ, PROM and PREM).
CYP / parent pairs were happy to complete the forms although there was a definite lack of consensus relating to paper or electronic formatting. Overall, 61/120 (50.8%) pairs highlighted a preference for paper forms with 48/120 (40%) specifying a preference for electronic forms.
Electronic forms, completed at home, were highest ranking for both the CHAQ (n=15/40) and PROM (n=12/40) but electronic and paper versions of the PREM (n=11/40 respectively), completed either at home or in the hospital waiting area, were ranked equally highly. If all forms were to be completed electronically, accessing the forms via personal mobile phone was favoured (n=87/120) over personal iPad/tablet (n=59/120) or hospital iPad/tablet (n=56/120). Nearly half of respondents would prefer a text reminder to complete the forms prior to the appointment (n=54/120), followed by email (n=27/140) or letter (n=36/120).
Completion of forms between appointments was equally divisive. Around 30% respondents would choose to complete the CHAQ (13/40) or the PROM (12/40) between appointments with 44/80 (55%) keen to avoid between-appointment reporting.
D) Qualitative clinician evaluation
Nine clinicians participated in the pilot with a total of 17 clinician feedback forms submitted (Table 4). Clinicians reported that the Agileware system was easy to neutral to use, becoming easier with increased familiarity.
Table 4: CAPTURE-JIA Electronic Pilot Clinician Feedback
1. What went well?
|
- Patients seemed enthusiastic about the concept of CAPTURE-JIA data collection.
- The Agileware system feels and looks “professional”, flows in accordance with the clinical consultation and becomes easier with familiarity.
|
2. What didn’t go so well?
|
- Some clinicians reported that the system occasionally crashed mid-data entry.
- Forms took much longer to complete for patients who were diagnosed many years earlier (and many of the data items were missing).
- Some data items may need a “not known” or “not checked today” tab (e.g. baseline data items, uveitis, height/weight).
- Some results may not be available at the time of completing the form (e.g. bloods). How would you advise centres to complete these forms?
|
3. What (if anything) would have improved the process?
|
- Recruiting new patients only rather than including historical patients.
|
4. Additional comments
|
- Fantastic to have developed an IT solution to support collection of the dataset.
- The forms are far easier to complete fully and less time consuming for newly diagnosed patients.
- The forms are considerably less time consuming if completed in retrospect (when all data items are readily available).
- May need to consider working with a lead clinician at each centre.
|
5. Timings
|
- Ranged from 11 to 30 minutes (average 20.7 minutes per form).
- Competing priorities may prevent dataset completion in the busy clinical setting.
- Data entry became faster with experience.
|
Phase 4
Further to clinician feedback, no changes were made to the consensus-agreed dataset, although a number of minor amendments to the database were implemented. The issue with the system freezing was isolated to one Trust and likely related to firewall restrictions.
The national youth research advisory panel (Your Rheum - https://yourrheum.org) provided additional PPI opinion regarding patient data collection methodologies. Feedback was collated from a virtual Your Rheum meeting involving 6 young people (5F:1M from Northern Ireland (3), Liverpool (1), Manchester (1), Sheffield (1)) and an online survey (n=8). The youngest attendee was 16 years old. Young people felt it important to collect data and appeared surprised that this was not happening already. Anonymity was a high priority. Paper forms were preferable to electronic (n=5/8), although participants did suggest development of a dedicated mobile phone/tablet app (n=4/8), QR code (n=4/8) or direct patient portal (n=4/8) if electronic completion was considered necessary.
A higher proportion of this group indicated that they would like to complete the forms more often in-between clinic appointments (n=5/8). As one young person commented “it’s important to capture ‘a difficult period’ and remember there are good and bad days/weeks”.