In this study, we investigated response patterns in the Swedish HEARTS study on retirement transition, in which a paper questionnaire was offered as an alternative to the web questionnaire in the last reminder.
Our results can be generalized into three main findings. First, most respondents answered by web (69%; n = 4067 at baseline) and this was a rather stable group who continued to respond by web in subsequent waves. Paper-respondents on the other hand, were fewer (31%; n = 1845 at baseline) and were less stable in subsequent waves, with higher probability of non-response and changing response mode.
Second, compared with those who answered by web, paper-respondents were more likely to be women, low educated, non-married and fully retired and reported more depression and poor self-rated health. In addition, the associations between retirement status and depression and to some extent poor self-rated health were stronger among web-respondents than among paper-respondents, although this result must be interpreted with caution, since some of these estimates changed substantially when adjusting for confounders.
Third, the differences between web and paper respondents were more pronounced in the longitudinal sample, compared with the baseline sample; the differences between web and paper respondents increased when taking into account the longitudinal response patterns, that is, restricting the analyses to those that answered by paper all three waves and those who responded by web across all three waves.
Our results are in line with previous studies of surveys of older persons, were women and non-married (17), low educated (16, 17) and non-working people (16) were found to be less likely to answer a web-questionnaire. Our finding that people answering by web had better subjective health are both supported (17) and non-supported (16) by previous studies. However, different measures of health were used in the compared studies.
Our finding that the response rate in subsequent wave was higher among web-respondents than among paper-respondents are also in line with previous literature. It has been shown that the differences in response rate between paper and web surveys is lower among panel members than among one-time respondents (5). This suggest that given survey response by web one time, the likelihood of response to the next wave of a web-survey are higher than in a new sample were people are contacted for the first time.
From a previous study, we know that the attrition in the HEARTS study is associated with personality; people with higher scores on extraversion and neuroticism, and lower scores on agreeableness, were more likely to drop out (22). Results from the present study adds to that knowledge by showing that those who preferred a paper questionnaire, also are more likely to attrite from the study. Finally, our analyses also showed that web and paper respondents differed more after two follow-ups compared to baseline. This finding demonstrates that without the option of a paper questionnaire, the web responders would have been an even more selected group over time.
Implications of the Chosen Survey Design
It is not possible to estimate what the response rate in the HEARTS study would have been if data had been gathered using another survey mode. Previous studies show that web surveys in general produce approximately 10–11% lower response rate than other survey modes, such as paper and telephone (5, 23), except among students, were the results are more mixed. In one study among students, paper and web yielded the same response rate (2), but in another study the highest response rate was reached when both paper and web was offered (24). Further, in an experimental study of a highly internet-literate population, the offer of both web and paper did not improve response rate compared to only paper. Nevertheless, offering paper at a later stage, as an alternative to web, improved the response rate and was equivalent to the use of paper as the only alternative (25). Previous research also show that the number of reminders seems to be less efficient in web surveys than in other modes, such as paper (5, 11). Taken together, this implies that it is likely that the non-response rate in HEARTS would have been higher if not a paper questionnaire was offered as a response option, even if more reminders would have been used.
The major problem with non-response, in addition to the decreased statistical power, is largely related to the risk that the non-response is occurring non-random. Web and paper respondents differed significantly from each other in the HEARTS study, not only in sociodemographic factors, but also in self-reported health and certain psychological outcomes, both in levels and regarding the association with retirement status. In addition, these differences was compounded in the longitudinal sample. That is, the differences between web and paper respondents was more substantial in the longitudinal sample (i.e. among those who either answered by web or by paper all three waves) compared with all those answering at baseline. Hence, we also conclude that without offering a paper questionnaire as an alternative, a small but important group would have been missing in subsequent waves in the HEARTS study.
The next question is whether the quality of the data in HEARTS depended on choice of survey design. We know that survey mode matter for the results and that it can be problematic to change survey mode across waves (26, 27). However, we also know that some of the differences between survey modes can be explained by changes in wording, structure and visual effect used in the different survey modes and it is therefore recommended to use as similar questions as possible when using multi-mode surveys (27). In the HEARTS study, wording and structure were as identical as possible in the paper and web questionnaire. In addition, the paper and web questionnaire were self-administered, which implies smaller differences than if one of the modes were self-administrated questionnaires and one was conducted by interview (26). Further, in a more recent paper, web, paper and telephone mode yielded similar results regarding political opinion and issues (28).
Our results, together with findings from other studies, brings us to the conclusion that any potential bias due to the mix of modes in the HEARTS study is likely to be marginal compared to the more substantial differences we would have expected if not offering a paper alternative to the web-format.
Limitations
An important limitation in this study is that there is no gold standard to compare our results with, that is, we do not know the real population values for most of the studied variables. However, based on the differences we found between web and paper respondents and that we assume that the offered paper questionnaire contributed with data from a group that otherwise would have been missing, we believe that offering the paper questionnaire generated results closer to the true population values.
Although the HEARTs study is a survey comprising questions of relevance for the target population and therefore might motivate participation, the questionnaire is extensive and rather time-consuming. It is most likely that there is differences between people who felt motivated enough to answer the questionnaire and those who did not. Respondents in HEARTs are for example more educated compared with the general population (18). It might be that the differences between web and paper respondents, as well as the response patterns, would have been different in a less extensive and time-consuming survey.