Sample:
The CHAQ and QoML PROMs are normally distributed in our general clinics, juvenile dermatomyositis (JDM), systemic arthritis and autoinflammatory subspecialty clinics. Therefore, patients seen in our systemic lupus erythematosus (SLE), neonatal lupus erythematosus (NLE), Kawasaki disease, and vasculitis clinics were not included in this study. All patients who were attending these aforementioned clinics were invited to participate, thereby representing the proportion of patients who would usually receive this questionnaire. A convenience sample of 225 clinic patients/caregivers consented to participate. Of the enrolled sample, 29 datasets were excluded due to missing or incomplete data, resulting in a questionnaire completion rate of 87%. Technical issues with the internet connectivity limited three participants from being able to successfully submit their e-form, whereas we were unable to locate the paper form for 10 participants despite them having submitted an e-form. An additional 11 patients had both uncompleted e-forms and uncompleted paper forms. Finally, five patients were unable to be timed accurately with their paper form as they were interrupted after starting the form (were called to see their healthcare team) and completed the form at a later time. As such, participants who were able to successfully complete both paper and e-forms did so before they saw their attending physician.
A total of 196 participants were included in the project. 21 participants were new patients to the clinic, whereas 175 were follow-up patients. As with our usual clinical practices, we allowed patients and caregivers to decide among themselves who completed the paper and e-form. There was no prescribed eligibility age for patients as secondary factors such as caregiver’s fluency in English and patient’s intellectual/developmental disability influenced this decision. The satisfaction questionnaire was completed by the person who completed the paper and e-forms. Over half (57%) of the PROMS were completed by the patient alone. 11% were completed together by the patient and caregiver. The remainder of the PROs were completed by the caregiver.
Equivalence of Paper Form vs. E-Form PROMs Assessed by ICCs:
The ICC estimates are reported in Table 1. All ICC measures were greater than 0.9 with a p-value < 0.001. According to Koo and Li, ICC scores above 0.9 indicate excellent reliability (40). When we stratified by who completed the PROs (patient versus caregiver) we observed no difference in the agreement of responses (data not shown).
Table 1. Intraclass Correlation Coefficients (ICC) Between Paper Forms and E-Forms for PROMs.
Patient/Caregiver Reported Outcome Measures
|
ICC
|
Quality of my Life (QoML)
|
|
Overall, my life is:
|
0.910
|
Considering my health, my life is:
|
0.910
|
Childhood Health Assessment Questions (CHAQ)
|
|
CHAQ score
|
0.966
|
How would you rate your child’s illness in the past week?
|
0.904
|
How much pain do you think your child has had because of his or her illness in the past week?
|
0.952
|
Considering all the ways that illness affects your child, rate how your child is doing
|
0.934
|
Agreement and Bias Between Paper Form vs E-Form PROMs Assessed by Bland-Altman Plots:
These are shown in Figures 1 and 2. Figures 1A and B have data points clustered towards 100 as most of our participants were happy with their quality of life. Figures 2A, B, C, and D have data points clustered towards 0 as most of our participants did not experience pain that impacts their quality of life. All plots show a certain degree of bias which is listed in the Table 2, along with the LoA.
Table 2. Bland-Altman Data Summary for PROMs.
Patient/Caregiver Reported Outcomes
|
Bias1
|
95% CI of LoA2
|
QoML
|
|
|
Overall, my life is:
|
-0.211
|
26.79 to -26.26
|
Considering my health, my life is:
|
0.266
|
21.75 to -22.17
|
CHAQ
|
|
|
CHAQ Score
|
0.021
|
0.392 to -0.350
|
How would you rate your child’s illness in the past week?
|
0.744
|
29.78 to -28.29
|
How much pain do you think your child has had because of his or her illness in the past week?
|
2.30
|
24.16 to -19.56
|
Considering all the ways that illness affects your child, rate how your child is doing:
|
3.05
|
23.31 to -17.22
|
1Positive values indicate that on average, paper forms score that many units more than the e-form. Negative values indicate that on average, paper forms score that many units less than the e-form.
2All scores (except the CHAQ score which is measured out of 3) are measured out of 100.
CI=confidence interval
LoA=limits of agreement
Completion Time:
Overall, the paper form took longer to complete when 2.5 minutes were added to the paper form completion time (Table 3). New patients took longer to complete the forms when compared to the follow-up patients. Excluding the processing time, paper forms took less time to complete than the e-forms.
Table 3. Mean and Median Completion Time of Paper and Electronic Forms (E-forms) by New and Follow-up Patients.
|
|
New
(n=21)
|
Follow-up
(n=175)
|
Overall
(n=196)
|
Paper Form
|
Mean (seconds)
|
323.8 (473.8)1
|
255.9 (405.9)
|
263.7 (413.7)
|
Median (seconds)
|
302.1 (452.1)
|
207.6 (357.6)
|
221.1 (371.1)
|
E-Form
|
Mean (seconds)
|
519.82
|
394.2
|
407.3
|
Median (seconds)
|
408
|
293
|
294.5
|
1Numbers in brackets include two and a half minutes added for manual scoring, verification, and data entry.
2Bolded numbers indicate the longest time to completion when comparing paper form to e-form.
Observed Benefits and Barriers to Electronic PROMs
Cost Comparison Analysis:
We identified the costs of all the resources associated with the paper forms and e-forms to accomplish the cost-comparison analysis. The cost per patient for each paper CHAQ/QoML was $1.23 CAD. The overall cost for the e-forms was $500, which included the two electronic tablets used to administer the e-form. Cost savings would be realized after 407 uses of the e-form, which – in our clinic – would take approximately four weeks.
Barriers
Barriers to completing the e-form included poor Wi-Fi connectivity in certain areas of the clinic. Patients and caregivers were not able to qualify their answers on the e-form, whereas they could write on the paper form. Sensitivity of the device being used for this project may have decreased the ability of patients/caregivers to select extreme end values (e.g., 0 or 100). Other noted barriers associated with devices were the limited number of devices available, the potential theft of devices, as well as the need to disinfect devices after use.
Satisfaction Survey Results:
83% of respondents indicated that they either preferred the e-form to the paper form or had no preference. One respondent stated: “It was great - easy to use. Easier for my daughter to complete with her arthritis” and another stated: “My daughter usually says 'Oh no, not again' when she is handed the paper format. She loved using the tablet format. It is much more user friendly for kids/teens.” Others commented on the e-form’s environmental and potential cost-savings as well.
More than 97% of participants agreed that both the paper form and e-form were easy to understand and navigate. Approximately 10% of participants made suggestions to improve the user experience. The respondents commonly reported difficulty selecting responses on e-form. “It was hard to select answers if they were on the extreme end of the sliding scale.” Another common suggestion included making “the text larger, and the select buttons bigger so it is easier to press.”
Three patients/caregivers indicated that they preferred the paper form for varying reasons. “I enjoy writing it with a pen in hand personally” was one reason cited. “The electronic version could go down (not work), and my kids would want to play with it (the tablet) when they see it.” “The paper version was just as fast to complete as the electronic version” were also mentioned by individuals indicating a preference for the paper form.