The effect of a web-push survey on physician survey response rates: a randomized experiment.

doi:10.21203/rs.2.12242/v1

Download PDF

Research article

The effect of a web-push survey on physician survey response rates: a randomized experiment.

https://doi.org/10.21203/rs.2.12242/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Achieving a high response rate for physicians has been challenging and with response rates declining in recent years, innovative methods are needed to increase rates. An emerging concept in survey methodology has been web-push survey delivery. In this delivery method, contact is made by mail to request a response by web. This study explored the feasibility of a web-push survey on a national sample of physicians. Methods: 1000 physicians across six specialties were randomly assigned to a mail only or web-push survey delivery. Each mode consisted of four contacts including an initial mailing, reminder postcard, and two additional follow-ups. Response rates were calculated using AAPOR’s response rate 3 calculation. Data collection occurred between Febuary – April 2018 and analyzed March 2019. Results: Overall reponse rates for the mail only vs. web-push survey delivery were comparable (51.2% vs. 52.8%). Higher response rates across all demographics were seen in the web-push delivery with the exception of pulmonary/critical care and physicians over the age of 65. The web-push survey yielded a greater response after the first mailing requiring fewer follow-up contacts resulting in a more cost-effective delivery. Conclusions: A web-push mail survey is effective in achieveing a comparable response rate to traditional mail only delivery for physicians. The web-push survey was more efficient in terms of cost and in receiving responses in a more timely manner. Future research should explore the efficiency of a web-push survey delivery across various health care provider populations.

Health Economics & Outcomes Research

Survey methodology

response rates

physician survey

web-push delivery

Physician surveys are an important tool in obtaining information on healthcare related attitudes, beliefs, and practices (1). Historically, achieving a high response rate for this population has been challenging, and response rates have been declining in recent years (2-4). Given the potential for low response rates in this population (5, 6), understanding research strategies that facilitate adequate response rates in physician surveys is critical to strengthen generalizability and reducing bias. A recent review has found that incentives, repeated follow up, and data collection mode are important factors related to response rates in general as well as for this target population (4).

Dillman’s Tailored Design Method (TDM) is an extensively researched method on enhancing response rates (7). Strategies included are a respondent friendly questionnaire, repeated follow up, and incentive (8, 9). The TDM has been successful in achieving a high response rate (range 56%-86%) from health care providers on a variety of health topics (10-14). More recently, other strategies have been explored for health care providers surveys to combat declining response rates including web based and mixed mode surveys (15). Web based survey delivery to physicians offers immediate survey delivery, real time data, and lower costs (16). Despite these advantages, studies have shown that physician response rates of web only surveys are lower than mail surveys (17, 18). In mixed mode survey methods, one mode of response is initially offered followed by an alternate mode in a follow up contact (15). While mixed mode surveys have been shown to facilitate response rates among the general population as well as physicians (19, 20), challenges remain in adapting this method to the physician population. Indeed, mail surveys tend to have higher responses rates for the physician population compared to online or mixed mode surveys (4). Furthermore, alternate modes of data collection like online are problematic given there is no comprehensive list of physician emails to function as a sampling frame (21) and while e-mail addresses are available in as the American Medical Association Physician Masterfile, they are not included for all physicians and may be inaccurate (22). Until there are comprehensive sampling frames of physician email addresses, mail delivery will remain a dominant sampling approach.

Recent work in survey methodology has explored web-push mail surveys, which may yield higher response rates than the paper-only survey delivery (23). In web-push surveys the initial correspondence contains a postal request to respond by web. Follow-up mailings include a reminder letter to complete the survey online followed by a paper questionnaire. Offering a paper survey after the initial web request has been shown to increase response rates in the general population (Dillman 2015). The extent to which this method could work for physician surveys is unknown – as such this paper examines the feasibility of a web-push delivery for physician surveys via split sample experiment aims.

This survey experiment was embedded in a larger national, repeated cross-sectional mail survey of physicians focused on attitudes and beliefs regarding tobacco use, smoking cessation and electronic cigarettes (24, 25). In brief, the sampling frame for the national study was compiled from the American Medical Association's master list, via the MMS vendor; 3000 board certified practicing physicians, equally distributed across six specialties: family medicine, internal medicine, obstetrics and gynecology, cardiology, pulmonary/critical care, and hematology/oncology, were randomly selected. From the national sample, we then randomly selected 1000 physicians and then randomly assigned them to one of two survey mode conditions: traditional mail or web-push survey. Data collection occurred between Febuary – April 2018 and data were analyzed in March 2019.

The first mailing contained an introductory cover letter and a $25 Starbucks® gift card; for those assigned to the mail survey mode, they also received a paper copy of the survey while those assigned to the web-push mode were given instructions in the cover letter how to complete the web-survey (i.e., the survey URL was provided with an anonymous login code). One week after the first mailing, we made a second mailing contact via postcard reminders to non-respondents; the web-push mode postcard contained the survey URL and login code. The third mailing to non-responders mirrored the first contact, minus the gift card. The fourth mailing to all non-responders was mixed mode and included a paper survey as well as a cover letter with instructions on how to complete the web-survey. As such, the last contact afforded all non-responders, regardless of condition, a choice for their survey mode (i.e., a web-push non-respondent could submit a paper survey and a mail survey mode non-respondent could now do the survey online).

Overall response rates were calculated using AAPOR’s response rate 3 calculation (26) which estimates the proportion of cases of unknown eligibility that are eligible using data gathered during fielding. For example, if a survey had 100 completes and 50 ineligibles, the eligibility rate among non-responders is assumed to be 66% (i.e., 100 divided by 150). We examined responses rates by mode condition after each physician contact as well as final response rates by demographics.

Overall, 362 physicians responded to the survey and were equally distributed across the two survey modes. A total of 158 cases were determined to be ineligible (i.e., death, retirement, no active medical license in state, not board certified, not providing outpatient care). More ineligibles were identified in the web-push condition (83 vs. 75). Of note, the first question on the survey screened for eligibility (i.e., providing outpatient care) and for the web-push condition, the survey terminated with a “no” response, whereas for the traditional paper survey, the participant would need to indicate “no” and mail the survey back. The small differences for “ineligibles” produces slightly different overall response rates, despite the same number of completes for each group.

As shown in Table 1, response rates did not significantly differ for the two data collection modes overall or within demographic subgroups, with both modes achieving a response rate just over 50%. Females had slightly higher response rates in both modes compared to males, but the web-push condition performed slightly better. Differences in response rate are noted across speciality groups with the highest response rates among Family and Internal Medicine, and the lowest among cardiologist. Slightly higher responses rates for the web-push condition were noted for family medicine and oncology, whereas slightly higher response rates were noted for traditional mail survey condition for obstetrics and gynecology. Response rates overall were highest for those under the age of 45 and declined by age. The traditional mail survey mode performed slightly better for those over the age of 65.

Figure 1 depicts cumulative survey completion for the two data collection modes. Differences between the two modes are noted within the first three weeks of fielding, such that the web-push yielded greater completions than the mail survey, after which the mail survey conditions catches up and keeps pace with the web-push condition. While, ultimately the two yielded nearly identical response rates, the shorter response time for the web-push condition yielded efficiencies such that there were fewer follow-up contacts in the web-push condition. Indeed, a total of 1714 mailings were prepared for the web-push condition vs. 1781 for the traditional mail survey. Additionally, as noted previously, we were able to identify a greater number of inelgible respondents in the web-push condition.

Post hoc analyses also considered financial efficiencies for each mode (Table 2). Overall, we find that the cost for the traditional mail survey was 25% higher than web-push. The main sources of savings included printing costs of the paper survey, postage, and labor costs associated with assembling the mailings and data entry. With the number of completes being equal between the two modes, this resulted in a cost per complete of $31.70 for the mail survey compared to $23.80 for the web-push.

To our knowledge, this is the first study examining the feasiblity of a web-push approach for physician surveys. Overall, we found that the web-push data collection mode obtained about the same response rate as the traditional paper and pencil mail survey. Moreover, this was without notable differences by demographics with a few exceptions. Physicians over the age of 65 and obstetrics and gynecology had lower response rates for the web-push condition than compared to the mail only condition. This is consistent with prior research which has found higher response rates among younger physicians for web based surveys (13). In addition, while previous literature has found response rates to be higher among male physicians (5, 16, 21), we found higher response rates for females in both delivery modes.

When comparing the response rates of both modes, we see that web-push, when combined with the tailored design method, is capable of achieving a comparable response rate to mail only delivery, with notable efficiencies. The shorter response time for the web push delivery allowed for a more prompt response time and identification of ineligibles resulting in fewer followups. With additional costs for the mail only delivery including cost of printing the paper survey, postage for the return envelope, and time taken for data entry, this resulted in the web-push delivery to be more cost-effective. This is consistent with previous research which found web based survey delivery as more cost-effective among physicians (27).

Limitations to note include the small sample size for specific subgroups (e.g., females) which did not yield statistically significant differnces, despite the fact that there were meaningful differences between the two survey modes for females and those over the age of 65. Likewise, caution should be considered regarding the lower response rate for cardiologists. In our sample, males and those over 65 years of age, who overall had lower response rates, were over-represented in this speciality and these two demographic sub-groups had lower response rates. As such, the response rate for cardiologists could be confounded by demographics.

With survey reponse rates among physicians continuing to decline, it is important to explore alternative delivery methods among this population. This study found that implementing a web-push delivery design was feasible as well as more cost-effective and resulted in comparable response rates than mail only delivery. For these reasons, the remaining sample of 2,000 physicians for our national survey was fielded using web-push delivery which achieved an overall response rate above 50%, a notable accomplishment in the context of the challenges with physician surveys. Further research should continue exploring the effectiveness of web-push delivery on a variety of health topics and health care provider populations.

TDM: Tailored Design Method

AAPOR: American Association of Public Opinion Research

Ethics approval and consent to participate: This study is approved by the Rutgers Institutional Review Board. All survey participants consented to participate.

Consent for publication: All participants consented for publication. Survey data is anonymous and de-identified

Availability of data and materials: The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests: The authors declare that they have no competing interests.

Funding: The study described was supported by grant number R01CA190444 from the National Institute of Health . The content does not necessarily represent the official views of the NIH.

Authors' contributions: All authors participated in designing and developing the study instruments and procedures. All authors assisted with drafting of the paper and interpreting results. CD is the principal investigator of the grant.

McLeod CC, Klabunde CN, Willis GB, Stark D. Health care provider surveys in the United States, 2000–2010: a review. Evaluation & the health professions. 2013;36(1):106-26.
Thorpe C, Ryan B, McLean S, Burt A, Stewart M, Brown J, et al. How to obtain excellent response rates when surveying physicians. Family practice. 2008;26(1):65-8.
VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response rates in surveys of physicians: a systematic review. Evaluation & the Health Professions. 2007;30(4):303-21.
Cho YI, Johnson TP, VanGeest JB. Enhancing surveys of health care professionals: a meta-analysis of techniques to improve response. Evaluation & the health professions. 2013;36(3):382-407.
Asch DA, Jedrziewski MK, Christakis NA. Response rates to mail surveys published in medical journals. Journal of clinical epidemiology. 1997;50(10):1129-36.
Cummings SM, Savitz LA, Konrad TR. Reported response rates to mailed physician questionnaires. Health services research. 2001;35(6):1347.
Dillman DA, Smyth JD, Christian LM. Internet, phone, mail, and mixed-mode surveys: the tailored design method: John Wiley & Sons; 2014.
Field TS, Cadoret CA, Brown ML, Ford M, Greene SM, Hill D, et al. Surveying physicians: do components of the “Total Design Approach” to optimizing survey response rates apply to physicians? Medical care. 2002;40(7):596-605.
Dillman DA. Mail and Internet surveys: The tailored design method--2007 Update with new Internet, visual, and mixed-mode guide: John Wiley & Sons; 2011.
Delnevo CD, Abatemarco DJ, Steinberg MB. Physician response rates to a mail survey by specialty and timing of incentive. American journal of preventive medicine. 2004;26(3):234-6.
Abatemarco DJ, Steinberg MB, Delnevo CD. Midwives' knowledge, perceptions, beliefs, and practice supports regarding tobacco dependence treatment. Journal of Midwifery & Women’s Health. 2007;52(5):451-7.
Steinberg M, Zimmermann MH, Gundersen D, Hart C, Delnevo C. Physicians' perceptions regarding effectiveness of tobacco cessation medications: are they aligned with the evidence? Preventive medicine. 2011;53(6):433-4.
Ferrante JM, Piasecki AK, Ohman‐Strickland PA, Crabtree BF. Family physicians' practices and attitudes regarding care of extremely obese patients. Obesity. 2009;17(9):1710-6.
Ferrante JM, Winston DG, Chen P-H, de la Torre AN. Family physicians' knowledge and screening of chronic hepatitis and liver cancer. Family medicine. 2008;40(5):345.
Millar MM, Dillman DA. Improving response to web and mixed-mode surveys. Public opinion quarterly. 2011;75(2):249-69.
Cunningham CT, Quan H, Hemmelgarn B, Noseworthy T, Beck CA, Dixon E, et al. Exploring physician specialist response rates to web-based surveys. BMC medical research methodology. 2015;15(1):32.
Lusk C, Delclos GL, Burau K, Drawhorn DD, Aday LA. Mail versus internet surveys: determinants of method of response preferences among health professionals. Evaluation & the Health Professions. 2007;30(2):186-201.
Dillman D. on Climbing Stairs Many Steps at a Time: The New Normal in Survey Methodology. 2015.
De Leeuw D. To mix or not to mix data collection modes in surveys. Journal of official statistics. 2005;21(2):233.
Beebe TJ, Locke GR, Barnes SA, Davern ME, Anderson KJ. Mixing web and mail methods in a survey of physicians. Health services research. 2007;42(3p1):1219-34.
Braithwaite D, Emery J, De Lusignan S, Sutton S. Using the Internet to conduct surveys of health professionals: a valid alternative? Family practice. 2003;20(5):545-51.
Klabunde CN, Willis GB, McLeod CC, Dillman DA, Johnson TP, Greene SM, et al. Improving the quality of surveys of physicians and medical groups: a research agenda. Evaluation & the health professions. 2012;35(4):477-506.
McMaster HS, LeardMann CA, Speigle S, Dillman DA. An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health survey of military spouses. BMC medical research methodology. 2017;17(1):73.
Singh B, Hrywna M, Wackowski OA, Delnevo CD, Lewis MJ, Steinberg MB. Knowledge, recommendation, and beliefs of e-cigarettes among physicians involved in tobacco cessation: a qualitative study. Preventive medicine reports. 2017;8:25-9.
Steinberg MB, Giovenco DP, Delnevo CD. Patient–physician communication regarding electronic cigarettes. Preventive medicine reports. 2015;2:96-8.
Smith T. A revised review of methods to estimate the status of cases with unknown eligibility. American Association for Public Opinion Research. 2009.
Scott A, Jeon S-H, Joyce CM, Humphreys JS, Kalb G, Witt J, et al. A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC medical research methodology. 2011;11(1):126.

Table 1. Final response rates by demographics for paper vs. web-push mode
	Paper	Web-push
Demographic Characteristics
Gender
Female	53.8%	59.5%
Male	49.8%	49.4%
Specialty
Cardiology	37.8%	41.6%
Family Medicine	56.2%	65.2%
Internal Medicine	64.8%	65.4%
Obstetrics/Gynecology	56.2%	49.4%
Oncology	40.2%	45.8%
Pulmonary/Critical Care	51.2%	47.4%
Age Group
under 45	57.9%	58.6%
45-54	45.5%	52.3%
55-64	47.0%	50.0%
65+	56.3%	48.9%

Overall	51.2%	52.8%

Table 2. Total cost of mailings for paper vs. web-push mode
	Paper	Web-push
Item
Paper Survey printing	$989.45	$267.96
Envelopes	$436.90	$299.72
Follow-up Postcard	$89.28	$81.72
Labels	$32.06	$30.76
Postage	$1,844.10	$1,518.30
Labor	$2,347.20	$2,110.80

Total	$5,738.99	$4,309.26

Download PDF

Version 1

posted

You are reading this latest preprint version

The effect of a web-push survey on physician survey response rates: a randomized experiment.

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Conclusion

List of Abbreviations

Declarations

References

Tables

Status:

Version 1