In this study, we investigated response patterns in the Swedish HEARTS study on retirement transition, in which a web-push methodology was used with paper questionnaires offered as an alternative to the web questionnaire in the last reminder.
Our results can be generalized into three main findings. First, most respondents answered by web (69%; n=4067 at baseline) and this was a rather stable group who continued to respond by web in subsequent waves. Paper-respondents on the other hand, that is, those who did not respond until they got a paper questionnaire with the last reminder, were fewer (31%; n=1845 at baseline) and were less stable in subsequent waves, with higher probability of non-response and changing response mode.
Second, compared with those who answered by web only, paper-respondents, as well as those that switched modes between waves, were more likely to be women, have a low level of education, being non-married and fully retired and reported more depression and poor self-rated health. In addition, the associations between retirement status and depression and to some extent poor self-rated health were stronger among web-respondents than among paper-respondents, although this result must be interpreted with caution, since some of these estimates changed substantially when adjusting for confounders.
Third, the differences between web and paper respondents were more pronounced in the longitudinal sample, compared with the baseline sample; the differences between web and paper respondents increased when taking into account the longitudinal response patterns, that is, dividing the longitudinal sample by those that answered by paper across all three waves, those that switched mode between waves and those who responded by web across all three waves.
Notable is also that the response rate in the HEARTS study is similar (9) or even higher (8) than recent studies using web-push methodology among younger age groups. This indicates that a web-push methodology may be efficient when collecting survey data among older adults, at least in countries with widespread internet use.
Our results are in line with previous studies of surveys of older individuals, where women and non-married (22), low educated (21, 22) and non-working people (21) were found to be less likely to answer a web-questionnaire. Our finding that people answering by web had better subjective health are both supported (22) and non-supported (21) by previous studies. However, different measures of health were used in the compared studies.
Our finding that the response rate in the subsequent wave was higher among web-respondents than among paper-respondents are also in line with the previous literature. It has been shown that the differences in response rate between paper and web surveys is lower among panel members than among one-time respondents (7). This suggest that given survey response by web one time, the likelihood of response to the next wave of a web-survey are higher than in a new sample where people are contacted for the first time. On the other hand, it should be mentioned that in the HEARTS study, the paper option was only offered in the last reminder. Hence, the paper-respondents in HEARTS are not comparable with paper respondents from a survey with a paper option in the first invitation. It is likely that some people from the web-sample would have preferred the paper version if they had the choice, without being less likely to participate in the subsequent wave. These results imply that the group of respondents that did not answer until they got a paper questionnaire are also the people that are most likely to not participate in a survey. It should also be mentioned that those respondents that once answered by paper might be less likely to answer by web in subsequent waves, as they know about the coming paper option.
From a previous study, we know that the attrition in the HEARTS study is associated with personality; people with higher scores on extraversion and neuroticism, and lower scores on agreeableness, were more likely to drop out (27). Results from the present study adds to that knowledge by showing that those who did not answer until they got a paper questionnaire, that is, the paper respondents, were also more likely to attrite from the study. Finally, our analyses also showed that differences between the longitudinal sample groups (web vs paper respondents) were greater than between web and paper respondents at baseline. This finding demonstrates that without the option of a paper questionnaire, the response group in HEARTS would have been even more selected over time if not a paper questionnaire would have been offered.
Implications of the Chosen Survey Design
It is not possible to estimate what the response rate in the HEARTS study would have been if data had been gathered using another survey mode. Previous studies show that web surveys in general produce approximately 10-11% lower response rate than other survey modes, such as paper and telephone (7, 28). A recent meta-analyses, including over 100 experiments, confirm these results and show a 12 % response rate difference between web-surveys and other modes (19). The exception is among students, were the results are more mixed. In one study among students, paper and web yielded the same response rate (3), but in another study the highest response rate was reached when both paper and web was offered (29). Further, in an experimental study of a highly internet-literate population, the offer of both web and paper did not improve response rate compared to only paper. Nevertheless, offering paper at a later stage, as an alternative to web, improved the response rate and was equivalent to the use of paper as the only alternative (30). Previous research also show that the number of reminders seems to be less efficient in web surveys than in other modes, such as paper (7, 13). Taken together, this implies that it is likely that the non-response rate in HEARTS would have been higher if not a paper questionnaire was offered as a response option, even if more reminders would have been used.
The major problem with non-response, in addition to the decreased statistical power, is largely related to the risk that the non-response is occurring non-random. Web and paper respondents differed significantly from each other in the HEARTS study, not only in sociodemographic factors, but also in self-reported health and certain psychological outcomes, both in levels and regarding the association with retirement status. In addition, these differences was compounded in the longitudinal sample. That is, the differences between web and paper respondents was more substantial in the longitudinal sample (i.e. among those who either answered by web or by paper across all three waves) compared with all those answering at baseline. Hence, we also conclude that without offering a paper questionnaire as an alternative, a small but important group would have been missing in subsequent waves in the HEARTS study.
The next question is whether the quality of the data in HEARTS depended on choice of survey design. We know that survey mode matter for the results and that it can be problematic to change survey mode across waves (31, 32). However, we also know that some of the differences between survey modes can be explained by changes in wording, structure and visual effect used in the different survey modes and it is therefore recommended to use questions as similar as possible when using multi-mode surveys (32). In the HEARTS study, wording and structure were as identical as possible in the paper and web questionnaire. In addition, the paper and web questionnaire were self-administered, which implies smaller differences than if one of the modes were self-administrated questionnaires and one was conducted by interview (31). Further, in a more recent paper, web, paper and telephone mode yielded similar results regarding political opinion and issues (33).
An important limitation in this study is that there is no gold standard to compare our results with, that is, we do not know the real population values for most of the studied variables. However, based on the differences we found between web and paper respondents and that we assume that the offered paper questionnaire contributed with data from a group that otherwise would have been missing, we believe that offering the paper questionnaire generated results closer to the true population values.
This hypothesis is supported by previous research, in which researchers found that a) the use of mail and web modes alone resulted in very different types of respondents, and b) a mix of web and mail obtains respondents quite similar to a mail-only design. From these findings the authors drew the conclusion that the type of people who respond via the web may also respond via mail but not the other way around. Hence, “when offering the web to general public household samples, it is important to provide a mail option to those who cannot or will not respond by Web.” (20).
Compared to other studies that used a design where different groups were offered different options (e.g., web or paper), the design used in the present study, as a part of the general set-up of the HEARTS study, makes it harder to draw clear conclusions in terms of mechanisms underlying differences. For example, the timing of the reply (how long it took for the respondents to reply) is to some extent confounded with the mode of response. One reason the group that answered via web as a reply to the email contact, may be because they prefer email as a contact, or, because they tend to reply to surveys directly instead of waiting. Due to this limitation in the design, we could not in a reliable way distinguish those that responded early via web as a response to the email contact from those that responded somewhat later via web as a response to the mail contact.
Although the HEARTs study is a survey comprising questions of relevance for the target population and therefore might motivate participation, the questionnaire is extensive and rather time-consuming. It is most likely that there are differences between people who felt motivated enough to answer the questionnaire and those who did not. Respondents in HEARTs are for example more educated compared with the general population (23). It might be that the differences between web and paper respondents, as well as the response patterns, would have been different in a less extensive and time-consuming survey.