Response Rate of Patient Reported Outcomes: the Delivery Method Matters

doi:10.21203/rs.3.rs-141497/v1

Download PDF

Research article

Response Rate of Patient Reported Outcomes: the Delivery Method Matters

https://doi.org/10.21203/rs.3.rs-141497/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 21 Oct, 2021

Read the published version in BMC Medical Research Methodology →

You are reading this latest preprint version

Background: Patient Reported Outcomes (PROs) are subjective outcomes of disease and/or treatment in clinical research. For effective evaluations of PROs, high response rates are crucial. This study assessed the impact of the delivery method on the patients’ response rate.

Methods: A cohort of patients with a unilateral vestibular schwannoma (a condition with substantial impact on quality of life, requiring prolonged follow-up) was assigned to three delivery methods: email, regular mail, and hybrid. Patients were matched for age and time since the last visit to the outpatient clinic. The primary outcome was the response rate, determinants other than delivery mode were age, education and time since the last consultation. In addition, the effect of a second reminder by phone was evaluated.

Results: In total 602 patients participated in this study. The response rates for delivery by email, hybrid, and mail were 45%, 58% and 60%, respectively. The response rates increased after a reminder by phone to 62%, 67% and 64%, respectively. A lower response rate was seen in association with a low level of education and longer time interval since last outpatient clinic visit.

Conclusion: The response rate for PRO surveys is influenced by the delivery method. PRO surveys by regular mail yield the highest response rate, followed by hybrid and email delivery methods. Hybrid delivery combines good response rates with the ease of digitally returned questionnaires.

Health Economics & Outcomes Research

response rate

delivery method

patient-reported outcome

vestibular schwannoma

Patient Reported Outcomes (PROs) are increasingly used both for scientific purposes and in clinical practice. PROs measure the patients’ perceived symptoms, functioning, and health-related quality of life. The use of PROs in research improves the understanding of the patient's perspective on the disease, the sequelae, and therapy [1]. In addition, using PRO in clinical practice may improve patient-clinician communication and enhance patient outcomes [2, 3]. The implementation of PROs in routine practice can be challenging however due to technological and workflow barriers [2].

One such barrier might be the method of delivery, which may impact on the response rate. Nowadays, electronic delivery of questionnaires by email has largely replaced delivery by regular mail. However, a 2009 meta-analysis, which also covered non-medical studies, showed that email delivery had a lower response rate than delivery by regular mail [4]. Conversely, in a more recent report physicians’ response rates were similar or slightly better after email delivery compared to regular mail delivery [5, 6]. Studies of the effect of the delivery method on patients’ response rates (i.e. in a medical setting) are scarce and mostly performed in small sample sizes [7–9]. This study assesses three different delivery methods for PROs in a large cohort of patients with a unilateral vestibular schwannoma, a disease that is usually not life threatening but may have a considerable impact on daily functioning and quality of life and requires prolonged follow-up.

This study was part of a larger study on long term outcomes of vestibular schwannoma management. Vestibular schwannoma is a benign intracranial tumor, causing symptoms such as hearing loss, tinnitus, and balance problems due to pressure on adjacent structures. A small majority of these tumors is non-progressive and in these cases active surveillance is usually the management option of choice. In progressive tumors, surgery or radiotherapy is performed to prevent future complications such as brain stem compression or elevated intracranial pressure. After active intervention, active surveillance ensues in these patients too as they are followed-up for prolonged periods of time in order to identify possible recurrences.

Patients who participated in a survey study in 2014 were re-approached for participation in a survey in 2020 [10]. All patients were diagnosed with unilateral VS between 2003–2014. Patients with bilateral VS, other skull base pathologies or insufficient proficiency in the Dutch language to complete the questionnaires were excluded.

Several PRO measurements which are also used in today’s routine care were collected in this study. Patients received a general health-related quality of life (HRQL) questionnaire, the short form 36 (SF-36), and a disease-specific HRQL questionnaire, the Penn Acoustic Neuroma Quality-of-Life Scale (PANQOL)[11, 12]. In addition, patients were asked to complete the dizziness handicap inventory (DHI), the medical outcome study cognitive functioning scale (MOS-CFS), the decision regret scale and the productivity costs questionnaires (iPCQ)[13–15]. Combined, patients were asked to answer 117 questions.

Three different delivery methods were used: email, regular mail, and a hybrid of the two. Patients in the email group received an email invitation with a link to a digital informed consent form. After providing consent, patients were directed to digital questionnaires. Patients in the hybrid group were invited by regular mail with a letter including a unique code and a link to the digital informed consent form and the questionnaires. The regular mail group received an informed consent form, the printed questionnaires, and a stamped return envelope. After two weeks, patients received a first personally addressed reminder by email (email group) or mail (hybrid and regular mail group). After another two weeks, all non-responders were called once by phone for a second reminder. This phone call was performed by a researcher, not their treating physician. In all groups, patients could request a different delivery method.

Before introducing the electronic patient records in our hospital (in 2011), the patient’s email address was not registered. Therefore, an email address was available in a minority of the patients, making randomization impossible. Patients for whom the email address was registered were assigned to the email group. The patients in the other groups (regular mail and hybrid) were randomly assigned to one of these two delivery modes and matched with the patients in the email group for age and time since the last visit.

Descriptive statistics were used to calculate the frequencies of categorical variables. Because patients could switch delivery method, both an intention to treat and an as treated analysis were performed. The former analyzes the predefined delivery methods and the latter the actual delivery method, i.e., taking the patients who switched between delivery method into account. The outcome was the response rate per group, which was analyzed using a chi-squared test. We also assessed the effect of the second phone call reminder by a chi-squared test. In addition, the effect on the response rate of age, educational level, time elapsed since the last visit (in years), and the delivery method were analyzed using logistic regression with response rate as the dependent variable. The independent variables were selected based on their reported effect on response rates in previous literature [4, 8, 16]. Model assumptions for multicollinearity were checked by calculating the variance inflation factor (VIF) and goodness of fit was verified with a Hosmer Lemeshow test and model chi-squared test. A minimum sample size of 387 was required based on a power calculation for the primary outcome, which used the difference in response rates in previous research (effect size w = 0.2, α = 0.05, 1-β = 0.95).

All statistical analyses were performed in SPSS version 26 (Armonk, NY: IBM Corp). A p-value < 0.05 was considered as statistically significant. There were no missing data.

In total, 602 patients were approached, of which 45(7%) refused participation, 169 (28%) did not respond, and 388 (64%) responded. Only two percent (N = 9) completed less than 80% of the questions. Baseline characteristics of the patients in the three groups are shown in Table 1. As expected, the matching variables ‘age’ and ‘time elapsed since the last visit’ were equally distributed in all groups. The proportion of patients with a low level of education was larger in the mail group. Furthermore, 98 (16%) patients used the possibility to request a different delivery method: 74 (76%) preferred to receive a questionnaire by regular mail and 24 (24%) preferred to complete the questionnaire electronically (by email).

Table 1

Baseline characteristics and response rates. Baseline characteristics for both the intention to treat and the as treated analysis are shown. Time since last visit shows the years since the last consultation in the hospital. yrs = years
	Intention to treat			As treated
	Email (N = 202)	Hybrid (N = 204)	Mail (N = 196)	Email (N = 201)	Hybrid (N = 151)	Mail (N = 250)
Sex (female)	98 (49%)	107 (53%)	91 (46%)	97 (48%)	76 (50%)	123 (49%)
Age
< 50 yrs	11 (5%)	11 (5%)	9 (5%)	10 (5%)	11 (7%)	10 (4%)
50–59 yrs	40 (20%)	40 (20%)	38 (20%)	39 (19%)	35 (23%)	44 (18%)
60–69 yrs	62 (31%)	63 (31%)	61 (31%)	67 (33%)	53 (35%)	66 (26%)
70–79 yrs	68 (34%)	70 (34%)	67 (34%)	60 (30%)	41 (27%)	104 (42%)
> 79 yrs	21 (10%)	20 (10%)	21 (11%)	25 (12%)	11 (7%)	26 (10%)
Educational level
Low	55 (27%)	78 (38%)	84 (43%)	64 (32%)	48 (32%)	105 (42%)
Middle	59 (29%)	52 (26%)	51 (26%)	57 (28%)	42 (28%)	63 (25%)
High	88 (44%)	72 (35%)	60 (31%)	80 (40%)	60 (40%)	80 (32%)
Time since last visit
< 5 yrs	126 (62%)	125 (61%)	120 (61%)	122 (60%)	91 (60%)	160 (64%)
≥ 5 yrs	76 (38%)	82 (39%)	76 (39%)	81 (40%)	60 (40%)	90 (36%)
Response rate
After 1st reminder	91 (45%)	119 (58%)	118 (60%)	85 (42%)	77 (51%)	166 (66%)
After phone call	126 (62%)	136 (67%)	125 (64%)	120 (60%)	94 (62%)	173 (69%)
Different delivery method
Email	-	4	20
Mail	25	49	-

Intention to treat

The initial response rates for patients allocated to delivery by email, hybrid, and regular mail were 45%, 58% and 60%, respectively. This difference was statistically significant (p < 0.01). The reminder by phone raised the response rates mostly in the email group, leading to all groups having a similar response; 62%, 67%, and 64%, respectively (p = 0.65).

As treated

The requests for a different delivery mode resulted in a decrease in email (-0.5%; N=-1) and hybrid delivery (-26%; N=-53), and an increase in mail delivery (+ 28%; N = + 54), as I sown in Table 1. The response rate for the actual delivery method was 42% by mail, 51% by hybrid, and 66% by mail (χ², p < 0.001). After the second reminder (by phone), the response rate increased to 60%, 62% and 69%, respectively (χ², p = 0.09).

Reminder by phone

After the first reminder by either email or regular mail, 249 patients (41%) still did not respond and received a reminder by phone call. Nearly half of these (N = 123) initial non-responders answered the phone, of whom 48% (N = 59) did participate after the this phone call, 36% (N = 45) did not respond while they said to do so in the phone call, and 15% (N = 19) declined participation.

Logistic regression

The results of the logistic regression are shown in Table 2. Both models, intention to treat and as treated, met the model assumptions and goodness of fit tests. The intention to treat analysis showed that the probability of responding was lower when the email route was used (odds ratio(OR) 0.48, 95% confidence interval (CI) 0.32–0.73). There was no difference between the hybrid and the regular mail delivery methods. In addition, patients with a low level of education had a lower probability of completing the PROs (OR 0.49, 95% CI 0.32–0.74). Furthermore, patients whose last visit to the hospital had taken place less than five years before the survey had a higher probability of responding (OR 1.52, 95% CI 1.07–2.15).

Table 2

Results of logistic regression. In both regression models, response rate was the dependent variable. The independent variables were delivery method, age, educational level, and time since last visit. The 2 and the Hosmer and Lemeshow show the goodness of fit tests. Time since last visit shows the years since the last consultation in the hospital. OR = Odds ratio, CI = confidence interval, yrs = years
	Intention to treat			As treated
		OR	95% CI		OR	95% CI
Delivery method
Email	N = 202	0.48	(0.32–0.73)	N = 201	0.35	(0.24–0.52)
Hybrid	N = 204	0.90	(0.60–1.36)	N = 151	0.51	(0.33–0.79)
Mail (reference)	N = 196	-		N = 250	-
Age
< 50 yrs		0.79	(0.32–1.96)		0.82	(0.33–2.04)
50–59 yrs		1.16	(0.60–2.21)		1.17	(0.61–2.26)
60–69 yrs		1.49	(0.81–2.71)		1.58	(0.86–2.74)
70–79 yrs		1.69	(0.93–3.04)		1.58	(0.87–2.87)
> 79 yrs (reference)		-			-
Educational level
Low		0.48	(0.32–0.72)		0.49	(0.32–0.75)
Middle		0.77	(0.50–1.17)		0.77	(0.50–1.18)
High (reference)		-			-
Time since last visit
< 5 yrs		1.52	(1.07–2.15)		1.49	(1.05–2.12)
≥ 5 yrs (reference)		-			-
Model		< 0.001			< 0.001
Hosmer and Lemeshow		0.31			0.62

In the as treated analysis, the difference between the delivery methods was smaller and no longer statistically significant. However, the effects on the probability of responding of the education level and time elapsed since the last visit were similar.

This study showed that delivery by email results in a lower response rate compared to delivery by regular mail or hybrid delivery. Even when patients could choose their preferred delivery method, the response rate per email was lower.

The low response rate of email delivery is consistent with prior studies on patient response [7, 8]. Compared to other studies, we found smaller differences between the delivery methods, despite the older average age of patients in this study. An older population might be less familiar with internet or email, but 87% of Dutch elderly (> 65 years) have internet access and 72% used email in 2019. In the subgroup of 65–75 years, these percentages are even higher: 95% internet access and 83% use of email [17].

Although regular mail delivery has the highest response rate, there are some logistic disadvantages. To use the PROs, surveys on paper need to be digitized, which is time-consuming and error prone. This is especially cumbersome when PROs are used in a clinical context, and feedback is expected during a clinical consultation. In this light, the results of hybrid delivery are noteworthy since the response rate is close to regular mail delivery, but the PROs are completed and returned electronically. In practice, the use of a hybrid system could reduce workload of digitizing PRO outcomes, with comparable response rates to surveys by mail.

In addition, a phone call reminder can further increase response rates. In the current study, 48% of initial non-responders did respond after being reminded by a phone call. However, the advantage of this higher response rate should be weighed against the time investment needed.

There are some inherent limitations to this study. First, it was impossible to perform a randomized trial because an email address was not available for all patients eligible for inclusion. Although the missing email addresses were caused by a different registration system in the hospital in the past, we cannot be entirely sure that the differences between the groups are purely at random. Second, the participants in this study were probably prone to participate in a research survey because they had already participated in a previous study in 2014. This committed population may therefore have increased response rates. Conversely, a decreased response rate may have been caused by a prolonged time interval between the survey and the last consultation, as was observed in a number of participants and was associated with a lower probability of responding in this study. Last, the PRO response rates found in this cross-sectional research setting may not be representative of PRO response rates in a clinical setting, in which PROs are typically collected close before or after clinical consultation and serve a more direct clinical purpose. However, patient preferences with regard to the survey delivery method are probably equally applicable to both settings.

When using PROs, the response rate is an essential factor to consider. Various factors have been identified that influence the response rate, such as personally addressed invitations, shorter questionnaires, and financial incentives [18, 19]. A reminder by letter and/or phone call seems to be particularly important in increasing the response rate, as confirmed by the current study [18]. In addition, this study adds two other major factors: the initial delivery method and the ability to choose the desired delivery method.

The effectiveness of the increasing use of PROs in healthcare stands or falls by patients completing and returning the questionnaires. This response rate can be influenced by several aspects, and the current study suggests that the route of survey delivery is an important factor. Regular mail delivery seems to perform better than email delivery but is more time consuming as results need to be digitalized afterwards. Therefore, a hybrid delivery method in which patients receive a letter by regular mail with a code to access the survey digitally might be the best of both worlds.

PRO = Patient Reported Outcome, HRQL = Health Related Quality of Life, SF-36=Short Form 36, PANQOL= Penn Acoustic Neuroma Quality-of-Life Scale, DHI = dizziness handicap inventory, MOS-CFS= Medical Outcomes Study – Cognitive Functioning Scale, iPCQ= iMTA Productivity Costs Questionnaires

Ethics approval and consent to participate:

Participants have provided informed consent. Formal approval of the ethics committee was not required under Dutch law according to the Medical Ethics Committee of the Leiden University Medical Center. The committee did approve the data handling and privacy review of this study (N19.112).

Data availability:

The dataset analyzed during the current study is not publicly available but is available from the corresponding author on reasonable request.

Competing interests:

The authors declare that they have no competing interests

Funding:

The appointment of the first author (ON) is funded by a strategic fund of the Leiden University Medical Center. There was no impact of the funding body in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Authors’ contributions:

Study concept and design: ON, EH; data collection and analysis: ON; interpretation and implications of results: ON, PB, AS and EH; drafting manuscript: ON; critical revision of the manuscript and final approval: ON, PB, AS, EH

Acknowledgements:

Not applicable

Basch E, Abernethy AP, Mullins CD, Reeve BB, Smith ML, Coons SJ, Sloan J, Wenzel K, Chauhan C, Eppard W et al: Recommendations for Incorporating Patient-Reported Outcomes Into Clinical Comparative Effectiveness Research in Adult Oncology. J Clin Oncol 2012, 30(34):4249-4255.
Basch E: Patient-Reported Outcomes — Harnessing Patients’ Voices to Improve Clinical Care. N Engl J Med 2017, 376(2):105-108.
Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, Macgillivray S: What Is the Value of the Routine Use of Patient-Reported Outcome Measures Toward Improvement of Patient Outcomes, Processes of Care, and Health Service Outcomes in Cancer Care? A Systematic Review of Controlled Trials. J Clin Oncol 2014, 32(14):1480-1501.
Shih T-H, Fan X: Comparing response rates in e-mail and paper surveys: A meta-analysis. Educational Research Review 2009, 4(1):26-40.
Weaver L, Beebe TJ, Rockwood T: The impact of survey mode on the response rate in a survey of the factors that influence Minnesota physicians’ disclosure practices. BMC Med Res Methodol 2019, 19(1).
Beebe TJ, Jacobson RM, Jenkins SM, Lackore KA, Rutten LJF: Testing the Impact of Mixed-Mode Designs (Mail and Web) and Multiple Contact Attempts within Mode (Mail or Web) on Clinician Survey Response. Health Serv Res 2018, 53:3070-3083.
Palmen LN, Schrier JCM, Scholten R, Jansen JHW, Koëter S: Is it too early to move to full electronic PROM data collection? Foot Ankle Surg 2016, 22(1):46-49.
Nota SPFT, Strooker JA, Ring D: Differences in Response Rates between Mail, E-mail, and Telephone Follow-Up in Hand Surgery Research. Hand 2014, 9(4):504-510.
Feigelson HS, McMullen CK, Madrid S, Sterrett AT, Powers JD, Blum-Barnett E, Pawloski PA, Ziegenfuss JY, Quinn VP, Arterburn DE et al: Optimizing patient-reported outcome and risk factor reporting from cancer survivors: a randomized trial of four different survey methods among colorectal cancer survivors. J Cancer Surviv 2017, 11(3):393-400.
Soulier G, Van Leeuwen BM, Putter H, Jansen JC, Malessy MJA, Van Benthem PPG, Van Der Mey AGL, Stiggelbout AM: Quality of Life in 807 Patients with Vestibular Schwannoma: Comparing Treatment Modalities. Otolaryngology–Head and Neck Surgery 2017, 157(1):92-98.
Ware JE, Jr., Sherbourne CD: The MOS 36-item short-form health survey (SF-36). I. Conceptual framework and item selection. Med Care 1992, 30(6):473-483.
Shaffer BT, Cohen MS, Bigelow DC, Ruckenstein MJ: Validation of a disease-specific quality-of-life instrument for acoustic neuroma. The Laryngoscope 2010, 120(8):1646-1654.
Jacobson GP, Newman CW: The Development of the Dizziness Handicap Inventory. Arch Otolaryngol Head Neck Surg 1990, 116(4):424-427.
Bouwmans C, Krol M, Severens H, Koopmanschap M, Brouwer W, Roijen LH-V: The iMTA Productivity Cost Questionnaire. Value Health 2015, 18(6):753-758.
Stewart A, Ware J, Sherbourne C, Wells K: Psychological distress/well-being and cognitive functioning measures In: Measuring Functioning and Well-Being: The Medical Outcomes Study Approach. edn. Edited by Stewart A, Ware J. Durham, NC: Duke University Press; 1992: 102-142.
Matthews FE, Chatfield M, Freeman C, McCracken C, Brayne C, Cfas M: Attrition and bias in the MRC cognitive function and ageing study: an epidemiological investigation. BMC Public Health 2004, 4(1).
Internet; access and use [https://opendata.cbs.nl/statline/#/CBS/nl/dataset/83429NED/table?fromstatweb]
Nakash RA, Hutton JL, Jørstad-Stein EC, Gates S, Lamb SE: Maximising response to postal questionnaires – A systematic review of randomised trials in health research. BMC Med Res Methodol 2006, 6(1).
Edwards P: Increasing response rates to postal questionnaires: systematic review. BMJ 2002, 324(7347):1183-1183.

Download PDF

Journal Publication

published 21 Oct, 2021

Read the published version in BMC Medical Research Methodology →

Review #1 received at journal
07 Feb, 2021
Editorial decision: Major revision
07 Feb, 2021
Review #2 received at journal
29 Jan, 2021
Reviewer #2 agreed at journal
11 Jan, 2021
Reviewer #1 agreed at journal
10 Jan, 2021
Reviewers invited by journal
07 Jan, 2021
First submitted to journal
21 Dec, 2020
Editor assigned by journal
21 Dec, 2020
Submission checks completed at journal
21 Dec, 2020
Editor invited by journal
21 Dec, 2020

You are reading this latest preprint version

Response Rate of Patient Reported Outcomes: the Delivery Method Matters

Status:

Journal Publication

Version 1

Abstract

Background

Methods

Results

Intention to treat

As treated

Reminder by phone

Logistic regression

Discussion

Conclusion

List of abbreviations

Declarations

References

Status:

Journal Publication

Version 1