Long-term Participant Retention and Engagement Patterns in an App and Wearable-based Multinational Remote Digital Depression Study

doi:10.21203/rs.3.rs-1602069/v1

Download PDF

Article

Long-term Participant Retention and Engagement Patterns in an App and Wearable-based Multinational Remote Digital Depression Study

https://doi.org/10.21203/rs.3.rs-1602069/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 17 Feb, 2023

Read the published version in npj Digital Medicine →

You are reading this latest preprint version

Recent growth in remote studies has shown the effectiveness of digital health technologies in recruiting and monitoring the health and behavior of large and diverse populations of interest in real-world settings. However, retaining and engaging participants to monitor their long-term health trajectories has remained a significant challenge. Uneven participant engagement combined with attrition over the course of the study could lead to imbalanced study cohort and data collection, which may severely impact the generalizability of real-world evidence.

We report findings from long-term participant retention and engagement patterns in a multinational remote digital depression study with up to two years of real-world behavior monitoring. In total, real-world engagement data from 614 participants with 14,964 surveys and 135,014 days of phone passive and wearable (Fitbit) data were analyzed using survival and unsupervised clustering methods. A considerable proportion of participants (N=415; 67.6%) were retained during the first 43 weeks of the study. Clustering participants’ long-term usage data of study apps and wearables showed three distinct subgroups with different engagement levels (most, middle, and least). Notable findings comparing participants' characteristics across these subgroups were: 1.) Participants (33.2%; N= 204) with the highest baseline depression severity (4 points higher PHQ8 score, p < .01) were in the least engaged group (median bi-weekly surveys completed = 4) compared to the most engaged group (37.6%; N = 231) that on average completed 20 bi-weekly surveys. 2.) A considerable proportion (44.6%; N = 91) of participants in the least engaged group still contributed wearable data for up to 10 months. 3.) The participants in the least engaged group also took significantly longer in responding to surveys in naturalistic settings (3.8 hours more, p < .001) and were younger (age difference = 5 years, p < .01) in comparison to participants in the most engaged group.

Our findings show various factors such as socio-demographics, app usage behavior, and depression severity can be linked to the long-term retention and density of real-world data collected in remote digital research studies. Finally, passive data gathered from wearables without additional participant burden showed advantages over active survey data for long-term monitoring, providing greater contiguity and duration of data collected. Together these findings could inform the design of future remote digital health research studies to enable equitable and balanced health data collection from diverse target populations.

To gain valuable insights into the etiology of depression and identify effective treatments tailored to individuals, large diverse cohorts-based studies are required to assess the underlying temporal patterns in real-world risk and protective factors of depression in individuals [1, 2]. However, dynamic day-to-day changes in behavior in naturalistic settings are not captured effectively by conventional clinical assessments that rely on infrequent in-person assessments and subjective retrospective reporting of symptoms [3]. Additionally, reaching and recruiting a large and diverse cohort in a cost-effective and timely manner continues to be a challenge for conventional clinical studies [4].

Due to increasing ubiquity and cost-effectiveness when compared to medical devices, smartphones and wearable devices provide researchers with the opportunity to monitor personalized daily behaviors and physiology over time for a large and diverse population [5–7]. When combined with scalable data collection platforms, these technologies provide high fidelity multimodal behavior sensing capabilities [8]. Several recent large-scale remote digital depression studies have shown the feasibility of technology-based remote data collection with real-world behaviors [9–12]. For example, sleep [13], social interactions [14], and mobility [15, 16] features, derived from digital apps, smartphones, or wearable devices, have been demonstrated to be significantly associated with depressive symptoms. Remote digital studies also offer an effective medium to reach and recruit from larger and more diverse populations [17] thereby considerably lowering the costs and time for creating cohorts of interest than conventional clinical studies [9].

Although previous remote digital studies have shown the feasibility and utility of leveraging smartphones and wearable technology for assessing behavioral changes in naturalistic settings, long-term participant retention and engagement remain a significant challenge [18, 19]. Moreover, differential recruitment and retention of participants can lead to imbalanced cohorts and biased data collection that can severely impact the generalizability of real-world evidence [20–23]. For example, Pratap et al. found that four specific indicators (referral by clinicians, older age of participants, compensation of participants, and having clinical condition [as opposed to being healthy]) were significantly associated with participant retention, and participant demographics were also associated with long-term engagement patterns in a cross-study evaluation of eight observational digital health studies conducted between 2014–2019 [20].

However, past studies investigated participant behavior and retention in the study for short follow-up periods and primarily based on active tasks (surveys) completed by participants using a limited set of variables of interest [20–23]. To be able to leverage digital health technology for assessing and managing complex chronic conditions (including psychiatric and neurological disorders), there is a need to further understand long-term participant engagement (months to years) in remote digital studies including the utility of active and passive data streams. Participants’ behaviors of answering surveys via the study app, such as time spent responding to surveys and completing surveys in naturalistic settings, may also reflect the participants’ interest [24–26], which in turn could be linked to long-term engagement in the study. Furthermore, passive data gathered via smartphones (e.g., Bluetooth and GPS data) and wearables (e.g., heart rate and sleep data) without additional participant input may show differential engagement patterns that need to be assessed further in comparison to active task-based data that requires active participation.

Utilizing data from the Remote Assessment of Disease and Relapse-Major Depressive Disorder (RADAR-MDD) dataset [10, 27], we evaluated the real-world factors impacting long-term participant retention and engagement in a large, multinational, longitudinal remote digital depression study. Specifically, we assessed three specific key questions using participant-level usage data of study apps and wearables: A) Is participant retention associated with real-world factors (such as socio-demographics, medium of data collection [smartphones and wearables], and severity of depressive symptoms)? B) Are there potential patterns in participants’ long-term engagement including differences between active and passive data streams collected via the study apps and wearables, respectively? C) And if there are significant differences in participants’ characteristics in the study across different long-term engagement patterns?

Cohort Characteristics

In total, we analyzed data from 614 participants recruited from three recruitment sites (350, 146, and 118 participants from KCL, CIBER, and VUMC, respectively) between November 2017 to April 2021. The cohort's median (IQR) age was 49 (32.0, 58.8) years with the majority being females (75.7%, N = 465/614) with a median baseline PHQ8 score of 10 (7.0, 16.0). Differences in participant characteristics across study sites were assessed by Kruskal-Wallis tests [28]. Participants recruited at the CIBER site had the highest median age (54.0 [49.0, 61.0]) across the three sites (KCL: 45.0 [30.0, 56.0] and VUMC: 40.0 [26.0, 57.8]) (p < .001). In addition, the CIBER site cohort also had a significantly higher median baseline PHQ8 score (15.5 [10.0, 19.0]) than the KCL (9.0 [6.0, 13.0] scores) and VUMC (8.0 [6.0, 14.0] scores) sites (p < .001). Participants at the KCL site were contacted more often (N = 5.0 [3.0, 8.0]) to be reminded about completing clinical assessments and in relation to technical issues during the primary observation period (the first 43 weeks) by the research teams compared to the other two sites (CIBER: 3.0 [2.0, 5.0], VUMC: 2.0 [1.0, 3.8]) (p < .001).

Table 1 summarizes socio-demographics, baseline depression scores, and the number of contact logs for the overall cohort and stratified by sites. Briefly, the secondary cohort had 313 participants with a median age of 51.0 [37.0, 59.0] years with the majority being females (75.1%, N = 235/313). The full set of secondary cohort descriptive statistics is summarized in Supplement Table 1.

Table 1

A summary of socio-demographics, baseline depressive symptom severity, and the number of contact logs of 614 participants in the RADAR-MDD study, with comparisons across the three study sites.
Characteristics	Total	KCL	CIBER	VUMC	P value
Number of participants, n	614	350	146	118
Baseline PHQ8 score (median [IQR])	10.00 [7.00, 16.00]	9.00 [6.00, 13.00]	15.50 [10.00, 19.00]	8.00 [6.00, 14.00]	< .001
Number of contact logs (median [IQR])	4.00 [2.00, 7.00]	5.00 [3.00, 8.00]	3.00 [2.00, 5.00]	2.00 [1.00, 3.75]	< .001
Age (median [IQR])	49.00 [32.00, 58.75]	45.00 [30.00, 56.00]	54.00 [49.00, 61.00]	40.00 [26.00, 57.75]	< .001
Female, n (%)	465 (75.7)	267 (76.3)	106 (72.6)	92 (78.0)	.56
Marital status, n (%)					.005
Single/separated/divorced/widowed	328(53.4)	185(52.9)	66 (45.2)	77 (65.3)
Married/cohabiting/LTR	286 (46.6)	165 (47.1)	80 (54.8)	41 (34.8)
Has children, n (%)	304(49.5)	152 (43.4)	111(76.0)	41 (34.8)	< .001
Years in education (median [IQR])	16.00 [13.00, 19.00]	17.00 [14.00, 19.00]	11.00 [9.00, 15.75]	16.50 [14.00, 20.00]	< .001
Annual income, n (%)					< .001
<15,000 (£/€)	152 (24.8)	74 (21.1)	47 (32.2)	31 (26.3)
15,000–55,000 (£/€)	348 (56.7)	203 (58.0)	92 (63.0)	53 (44.9)
>55000 (£/€)	98 (16.0)	72 (20.6)	7 (4.8)	19 (16.1)
Accommodation, n (%)					< .001
Own outright/with mortgage	323 (52.6)	169 (48.3)	105 (71.9)	49 (41.5)
Renting	236 (38.4)	151 (43.1)	27 (18.5)	58 (49.2)
Living rent-free	46 (7.5)	29 (8.3)	10 (6.8)	7 (5.9)

Participant Retention

For the primary cohort analysis, the participant retention (survival rate) at the end of the common maximum observation period (described in Methods) as quantified using Phone-Active, Phone-Passive, and Fitbit-Passive data streams were 54.6% (N = 335), 47.7% (N = 293), and 67.6% (N = 415), respectively. Similarly, for the secondary cohort, the participant retention rates measured by the three data streams were 48.2% (N = 151), 39.3% (N = 123), and 54.0% (N = 169), respectively. Figure 1 displays the Kaplan-Meier survival curves that show participant retention across two observation periods, stratified by three data streams.

To further assess the impact of multiple variables of interest (the baseline PHQ8 score, age, gender, marital status, children, years in education, annual income, accommodation type, and study site), we used multivariate Cox Proportional-Hazards models [29]. For each covariate a hazard ratio (HR) greater than 1 indicates the variable is positively associated with the risk of an event (stopping contributing data to the study), thus negatively associated with the length of participant retention time. Across the three data streams, age and study site were the two variables that significantly affected the risk of stopping contributing data to the study. Compared with the youngest group (18–30), participants in older age groups tend to stay in the study for a longer time. Especially, participants in the oldest group (> 60 years old) had the lowest risks of stopping contributing data for all three data streams (Phone-Active: HR = 0.54, p < .05; Phone-Passive: HR = 0.59, p < .05; Fitbit-Passive: HR = 0.48, p < .05). Participants in the KCL site had a significantly longer duration in the study compared with the CIBER site for Phone-Passive (HR = 0.64, p < .05) and Fitbit-Passive (HR = 0.53, p < .01) data. Participants in the VUMC site had the lowest risk of stopping contributing the Fitbit-Passive data (HR = 0.37, p < .01).

All three Cox Proportional-Hazards models met the proportional hazards assumption tested using the scaled Schoenfeld residuals [30]. Figure 2 shows the hazard ratio plots of the three data streams for the primary cohort. For the secondary cohort, participants of older age or recruited in the KCL site also have a longer duration in the study compared with the youngest group (18–30) or the CIBER site. Furthermore, male participants had lower risks (HR = 0.62, p < .05) of stopping contributing to the Phone-Passive data than females in the longer observation period. The hazard ratio plots for the secondary observation period are shown in Supplement Fig. 1.

Participants’ Long-term Engagement Patterns in the study

The density of long-term data contributed by participants, as represented by the data-availability vectors (Fig. 3a), was clustered by the unsupervised K-means method [31] across the three data streams separately. The optimal number of clusters was determined by elbow methods (Supplementary Fig. 2). In the primary observation period, three subgroups showing distinct participant engagement patterns (C1: most engaged, C2: middle engaged, and C3: least engaged), emerged across each data stream (Fig. 3b). Across the Phone-Active, Phone-Passive, and Fitbit-Passive data streams, we also found significant differences in participants’ survey response and completion times, socio-demographics, and baseline depression symptom severity across different long-term engagement patterns (Fig. 4 and Supplement Table 2- Table 4).

a) Long-term engagement patterns across active, passive, and wearable data streams: Participants completing the most bi-weekly surveys in the C1 cluster (37.6% of the cohort; N = 231) had a median (IQR) of 20.0 (18.0, 21.0) completed surveys as opposed to 4.0 (1.0, 6.0) completed surveys for the least engaged cluster (C3; 33.2% of the cohort; N = 204). Similarly, the data sharing patterns for passive data streams showed significant differences. Participants (42.2% of the cohort; N = 259) in the most engaged C1 cluster of the Phone-Passive data stream, shared phone-based passive data for a median (IQR) of 283 (257.0, 298.0) days as opposed to 32 (4.0, 67.5) days for the participants in the least engaged C3 cluster (33.7%; N = 207). Similarly, for the Fitbit-based data gathered passively, the most engaged C1 cluster with 66.29% participants (N = 407) shared the data for median (IQR) 294 (274.0, 301.0) days compared to just 18 (0, 67.0) days for the 17.6% participants (N = 108) in the least engaged cluster (C3). A notable proportion of participants in the C2 and C3 clusters of the Phone-Active data stream, despite completing a lesser number of longitudinal surveys (13 and 4 bi-weekly surveys respectively) contributed passive data from Fitbit for up to 10 months. Figure 3c shows this marked transition where 65.4% of participants from the C2 cluster and 44.6% from the C3 cluster of the Phone-Active data stream, transitioned to the most engaged C1 cluster of the Fitbit-Passive data stream.

b) Survey response and completion times: We also observed distinct differences in the survey response time (the time participants took to respond to survey notifications) and completion time (the time participants spent completing the survey) across the clusters. Participants in the most engaged C1 clusters of the Phone-Active data stream had significantly shorter survey response time (73.7 [31.3, 215.8] minutes) for the PHQ8 survey compared to 302.4 (122.3, 527.1) minutes for the least engaged C3 cluster (Fig. 4a) (p < .001). This finding is also consistent for subgroups in the Fitbit-Passive data stream (Fig. 4a) and RSES survey (Supplement Table 2 and Supplement Table 4). Participants in the least engaged cluster (C3) of the Phone-Active data stream had a significantly longer survey completion time (61.6 [46.1, 83.0] seconds) than C1 (50.3 [37.9, 69.0] seconds) and C2 (49.4 [40.0, 67.0] seconds) clusters (Fig. 4b) (p < .001). Likewise, the finding of survey completion time is consistent for the Fitbit-Passive data stream (Fig. 4b) and RSES survey (Supplement Table 2 and Supplement Table 4).

c) Baseline Depression symptom severity: The baseline PHQ8 scores of participants were significantly different across three subgroups (C1, C2, C3) across all three data streams. Overall participants in the least engaged cluster (C3) had significantly higher severity of depressive symptoms at enrollment (Fig. 4c). For example, participants in C3 for the Phone-Active data stream had a 4 points difference in the median baseline PHQ8 score (13.0 [7.0, 17.0]) compared to participants in the most engaged cluster (C1) with a median baseline PHQ8 score of 9.0 (6.0, 15.0). Similarly, C3 of the participants derived from both Phone-Passive and Fitbit-Passive data streams showed a statistically significant difference in the baseline PHQ8 scores compared with the most engaged cluster (C1) (Phone-Passive - C1: 9 [6.0, 15.0]; C3: 12 [8.0,17.0] and Fitbit-Passive - C1: 9 [6.0, 15.0]; C3: 13 [9.0, 17.5]) (p < .001).

d) Socio-demographics: Similar to the results of CoxPH models, age was also found to significantly impact the participants’ long-term engagement patterns (Fig. 4d). For the Phone-Active data stream, participants in C1 cluster had a significantly higher median (IQR) age of 53.0 (34.0, 61.5) years than participants in C2 (45.0 [31.0, 55.5]) and C3 (48.0 [32.0, 57.3]) clusters (p < .01). Similarly, for the Phone-Passive data stream, participants in the most active C1 cluster had the significantly highest median (IQR) age of 52.0 (36.5, 61.0) years across 3 clusters (C2: 46.5 [30.8, 56.3] years and C3: 46.0 [30.5, 57.5] years) (p < .05).

The Kruskal-Wallis tests for all variables of interest across subgroups for 3 data streams of the primary cohort are summarized in Supplement Table 2 – Table 4, respectively. For the secondary cohort with a longer observation period, clustering showed 4 clusters (C1-C4) are shown in Supplement Fig. 3 and the results of enrichment of characteristics in subgroups are summarized in Supplement Table 5 -Table 7 for 3 data streams, respectively. Results of the enrichment of characteristics for the secondary cohort are similar to the results of the primary cohort.

We report findings from a novel investigation into long-term participant retention and engagement patterns, from a large European multinational remote digital study for depression [10, 27]. Our findings show a significantly higher long-term participant retention than past remote digital health studies [18, 20–23]. We show several factors, including participants' socio-demographics, depression symptom severity, survey response and completion times, significantly impact long-term participant engagement and the density of real-world data collected in fully remote digital studies. Here we contextualize our key findings in the broader digital medicine context that may help inform the design and development of remote digital studies in the future. We also compare the utility of using active and passive data collection for long-term remote monitoring of behavior and health outcomes. Finally, we share some of the participant engagement strategies deployed by the RADAR-MDD consortium [10] as well as data-driven insights to help improve long-term participant engagement in future remote digital studies.

Key learnings from long-term participant engagement patterns in remote studies

One of the notable findings was that participants with higher severity of depression at the time of enrollment were found to contribute fewer data both actively and passively. For example, participants in the least engaged cluster (C3) had the highest depression severity at the baseline and were up to 16 times less likely to share data irrespective of the active or passive form of data collection. This indicates a potentially negative linkage between higher symptom severity and engagement in remote research. Participants having a higher depression severity at enrollment may be at a higher risk of not engaging with the study app for the long term. As a result, data collection in long-term observational studies could directly impact the generalizability of findings from real-world behavior monitoring. Similar to our findings, previous literature has shown baseline depression to be significantly associated with engagement [32]. A previous analysis [33] of the RADAR-MDD dataset was based on a smaller study cohort that completed depression surveys during the study rather than baseline and did not show significant association between engagement and depression severity.

We also observed participants’ time in responding to and completing surveys are significantly associated with their long-term engagement patterns. Participants with shorter survey response time and completion time tend to complete more active surveys and wear Fitbit for more days, which indicated that the survey response and completion time could reflect their interest in the study [24–26]. To our best knowledge, this paper is the first to quantitatively link the survey response and completion times to participants’ longitudinal engagement patterns in a remote digital health study. Such real-world metrics on participants’ survey completion generated from metadata collected by the study app may be potentially useful for passively assessing the quality of the active data. Finally, we observed that older participants tend to contribute more phone data (both active and passive data) during the study, which indicates age may be one of the demographic indicators for long-term engagement patterns.

Feasibility of collecting Active data and Passive data streams for long-term remote behavior monitoring in major depression

While there is growing interest amongst researchers in gathering real-world behavioral data without having to rely on episodic in-clinic assessments that may be subject to recall bias [34]; there is limited empirical research quantifying the long-term participant engagement differences between active (surveys) and passive data streams (smartphones and wearables). We compared the completion densities of active and passive data collected from surveys, smartphones, and wearable devices.

Passively gathered data from wearable devices showed the highest long-term engagement (C1 in Fig. 3b and Supplement Fig. 3) and the highest participant retention rates (Fig. 1) over both observation periods. The finding clearly shows that wearable devices with minimal participant burden could help researchers collect high-density data over the longer term. Another potential reason may be that the Fitbit app provides participants timely feedback related to their sleep quality and physical activity, which may increase participants’ interest in wearing Fitbit devices. We found a significant proportion of the participants, who completed fewer longitudinal surveys (C2 and C3 of the Phone-Active data stream) but were willing to contribute passive Fitbit data for significantly longer (Fig. 3c). This illustrates the value of wearable devices for long-term monitoring of participants who are unable to keep contributing active surveys (possibly due to burden or the severe depression symptoms in the present cohort).

On the other hand, we found that the passive data gathered from participants' phones had the lowest retention rate in both observation periods (Fig. 1). This might be due to some technical issues during the course of the study. Finally, a small but significant group of participants were not contributing either active or passive behavioral data (Fig. 3c). Further research is needed to understand the concerns of this subgroup to avoid the collection of unbalanced data in relation to symptom severity.

Long-term participant retention

We found that older participants (> 60 years old) have a lower risk of disengaging from the study app than the younger participants (< 30 years old) (Fig. 2 and Supplement Fig. 1). This finding is consistent with a previous cross-study evaluation of eight remote digital health studies [20], which indicated that age is a reliable factor for participant retention in the remote digital health study. Although there are several barriers (such as health problems, lack of interest, and problems with transportation) to recruiting older participants in conventional clinical trials [35, 36], they have been found to be willing to participate and engage in the remote digital studies [37, 38]. A possible reason for high participant retention in older participants is that mental health (age negatively associated with depression [39]) and available time (high retirement rates in old people) are two essential factors affecting participants’ engagement [21, 40, 41]. As we mentioned above, age also affects long-term engagement, therefore, age is associated with both the depth and density of data collection in digital studies. Moreover, predictive models built from data skewed towards older age groups may have limited utility in younger age groups.

The study site is the other factor we found that significantly affects the participant retention rates, however, this may be due to significant differences in participants' characteristics across the sites. Participants in the CIBER site cohort had significantly higher baseline depression severity than the other two study sites (6–7 points difference in PHQ8 score [p < .001]; Table 1). Higher depression severity is known to be linked to participants’ engagement [21, 40, 41] may be a potential reason for lower participant retention rates in the CIBER cohort. Furthermore, research teams at the KCL site contacted the participants more frequently, which we speculate may be another reason for participants from the KCL site having a lower risk of stopping contributing passive data than the CIBER site. However, there are still many differences across study sites that may affect participant retention, such as differences in enrollment date, technology issues, and app versions, that are difficult to compare directly. We further discuss these in the study limitations.

Potential reasons for high participant retention in the RADAR-MDD study

Here we discuss four strategies developed by the RADAR-MDD consortium [10] which may have helped increase long-term participant retention and engagement.

a) “Human-in-loop” [42]. The RADAR-MDD research team contacted participants for various reasons, such as reminding 3-month assessments, switching to a new phone, having their Fitbit devices broken, and congratulating them on achieving the 1-year milestone. Feedback and encouragement from the research team may increase participants’ engagement [41].

b) Monetary incentives. Compensation for participant time and monetary incentives are known to enhance engagement [41, 43]. Participants were given monetary incentives for enrolling in the study, taking part in clinical assessments (every 3 months), and additional interviews (e.g., 1-year interview) (see Method section). This cyclical compensation (every 3 months and 1 year) might incentivize participants to remain in the study. Although participants were not offered compensation for completing surveys remotely and sharing behavior data passively, existing monetary incentives could increase participants’ willingness to remain engaged in the study. Furthermore, participants were allowed to keep the Fitbit device after the completion of the study observation period which could have impacted their motivation to join and remain engaged in the study for a longer-term.

c) Participant-centric design. Participants’ lack of familiarity with how to use digital technologies (study apps) and lack of intrinsic motivation (not familiar with the value of the study) are two key barriers to long-term engagement [41]. Therefore, participants and patients were invited to provide input at all stages of the study process [10]. A patient advisory board comprising service users guided the early study protocol and study app design stages to implementation and analysis phases. They provided contributions to improving the study design and engagement motivation strategy and shaped how the technology was used [44]. This approach, called “participant-centered initiative” [45, 46], treats participants as partners in the entire research cycle, which could provide a means to improve participant retention and engagement in long-term digital health studies. A recent study demonstrated that the “participant-centric design” played an essential role in maximizing engagement in 2 successful UK phone-based studies [18].

d) Recruiting participants with the target disease of interest. The inclusion criteria in the present study required all participants to have at least one depressive episode in the last two years. Therefore, the study contains an enriched population with a specific clinical condition. Prior research has shown that participants with clinical conditions of interest in the study tend to remain engaged for significantly longer [20, 41]. Experiences of having depression may make participants aware of the benefits of regularly completing the self-assessment and getting feedback from clinical teams to realize their status of mental health [41].

Potential solutions to improve participant retention and engagement in remote digital research

Although the incentives and recruitment strategies discussed above increased participant retention in the present cohort, a notable proportion of the cohort (17.59%-33.71%) across active and passive data streams did not remain engaged in study over the long-term (C3 clusters in Fig. 3b). Long-term participant retention and engagement in remote digital studies therefore remains an active area of research. Several potential solutions could be learned from our findings. Participant characteristics, such as younger age, more depressive symptoms at baseline, delayed responses to remote surveys, could act as early indicators of a subgroup of participants at the higher risk of disengagement from the study. Targeted engagement strategies including tailored communication material and increased “Human-in-loop” interactions could be deployed to this subgroup. An alternative approach is to recruit more heavily from participants matching the characteristics of the low engagers which may help reduce the overall data imbalance.

Also, a near-real-time analytical-model-driven framework could be deployed to continually monitor the incoming data for known socio-technical biases. The algorithm could triage participants who are falling below an acceptable level of compliance to the study team in terms of data quantity and quality. This could help just-in-time identification of potential causes of unbalanced data collection in real-world settings such as missingness due to technical issues or differential decline in the participant engagement and allow for timely and targeted interventions.

Limitations

Several data collection changes were made during the course of the study. These included changes in versions of some surveys, fixing technical bugs such as missing notifications, adding of some surveys, and stopping the collection of high-resolution phone-based passive data. The technical differences of the two versions of Fitbit devices deployed in the study (Charge 2 and 3) were not tracked. These changes and technical issues could have impacted the engagement of participants, however, were not accounted for during the retention analysis.

The calculation of our data availability vectors treats withdrawal and disengagement as equivalent factors. This was a deliberate design choice to help ensure our findings were not biased against participants who withdrew. Moreover, fully accounting and understanding withdrawal and any links with disengagement, would require considerable subjective analysis to fully understand participants’ motivations for dropping out. Such analysis is beyond the scope of the presented work which considered object measure of engagement (i.e., data availability) only.

We found the study site is also a significant indicator that both affected participant retention and longitudinal participant engagement in data collection. However, there were a number of significant differences across the three study sites, such as study start time, age distribution, recruitment pool, depressive symptom severity distribution, and the outreach level (frequency of contact with participants) of research teams at different study sites. Furthermore, the education system, language, income levels, and currency are also different across European countries and could lead to inconsistencies in the comparison of participant responses to socio-demographic questions across sites.

This study demonstrated that the RADAR-MDD study had significantly higher participant retention than past digital studies. The engagement strategies and settings deployed in the study, such as “human-in-loop”, monetary incentives, participant-centric design, and a targeted clinical cohort, could contribute to increased participant retention and engagement. We found several indicators of age, baseline depression severity, survey response and competition times, and recruitment sites significantly impacted the depth and density of our real-world data collection in fully remote research, which could be used to identify the potential participants with low engagement who need more engagement incentives in future research. Furthermore, passive data gathered from wearables without participant burden showed advantages over active survey data collection for long-term monitoring, providing greater contiguity and duration of data collected. Together these findings could inform the design of future remote digital health research studies to enable equitable and balanced health data collection from diverse target populations.

RADAR-MDD Study design

Data used in this study was collected from the EU research program RADAR-MDD, which aimed to investigate the utility of smartphones and wearable devices to monitor depression in real-world settings and understand factors that could help predict relapse in MDD [10]. The study recruited 623 participants from 3 sites across 3 Europe countries (United Kingdom - King’s College London [KCL]; Spain - Centro de Investigación Biomédican en Red [CIBER]; Netherlands - Vrije Universiteit Medisch Centrum [VUMC]) and followed participants for up to 2 years [10]. Nine participants recruited from a second site in Spain were not included in the present analysis due to the small sample size. All participants in this study had a history of recurrent MDD with at least one episode within the last 2 years that meets DSM-5 diagnostic criteria for diagnosis of MDD. The detailed study protocol and descriptions of the dataset have been reported by Matcham et al. [10, 27].

The study used RADAR-base [8], an open-source platform, for smartphone-based health data collection. All participants were sent regular notifications reminding them to complete study-related health surveys via the study app [10]. Additionally, participants’ real-world behavior was gathered passively using a smartphone-based app and a Fitbit wearable (details below). Participants’ socio-demographics, medical history, lifetime depression history, and baseline mental health status were also collected during the participant enrollment session [10]. Participants were contacted by the research team for various reasons, such as reminding clinical assessment and technical issues, during the study.

The first participant was enrolled in November 2017 and the last participant was enrolled in June 2020, and the data collection was finished in April 2021 [27]. As a result of this rolling enrollment, the time in study for RADAR-MDD participants varies from 11 months to 24 months. There were temporal differences in participant recruitment across the three sites. The KCL site started participant recruitment first (November 2017) followed by the CIBER site (September 2018), and the VUMC enrolled participants later again (February 2019) [27].

Although participants were not financially reimbursed for providing data via study apps and the wearable, participants received £15/€20 for enrollment, £5/€10 for clinical assessments every 3 months, and £10/€10 for every additional qualitative interview completed [10].

The RADAR-MDD protocol was co-developed together with a patient advisory board who shared their opinions on several user-facing aspects of the study including the choice and frequency of survey measures, the usability of the study app, participant facing documents, selection of optimal participation incentives, selection and deployment of wearable device as well as the data analysis plan [10, 47]. All participants signed informed consent and the study had been approved by all local Ethics committees [10].

Primary Data streams

For evaluating long-term participant retention and engagement in the study, we classified the data collected by the study apps into three distinct categories: i) Phone active data - representing active tasks completed by participants via the study app, ii) Phone passive data - continuous data streams gathered by the smartphones without active input from participants, and iii) Fitbit passive data - continual physiological monitoring data collected through a wrist-worn Fitbit device during the observation period.

Phone Active Data

A variety of episodic survey-based assessments (active tasks) were administered via the study app during the course of the full study. The complete list of surveys and deployment details are covered in the study protocol [10]. However, with the focus on present research evaluating long-term engagement, we considered the two longitudinal surveys, Patient Health Questionnaire (PHQ-8 [48]) and Rosenberg Self-Esteem Scale (RSES [49]), that were conducted remotely once every two weeks. If the participants finished at least one of these two surveys, we considered they were engaging in the active assessments part of the study for the corresponding 2 weeks.

Phone Passive Data

The passive monitoring app unobtrusively and continuously collected information on participants’ phone usage (e.g. battery level logs, app use logs, and phone interaction data) and surrounding information (e.g., ambient light, nearby Bluetooth device count, and GPS location data) [10]. Following the phone passive data availability definition presented in [27], we considered a participant to be using their study phone and sharing the phone passive data on a given day if at least one passive data point was collected from their smartphone during the day.

Fitbit Passive Data

Participants were also required to wear a Fitbit Charge 2 or 3 wrist-worn during the follow-up time to provide passive measures of their sleep stages, steps, calories consumption, and heart rate. Matching the phone passive data availability definition, if at least one data point from the Fitbit-based data stream was available, we considered the participant to be wearing the Fitbit at least once during that given day.

Primary Outcomes

Metrics of engagement: We defined two key metrics to assess the participant’s engagement. (1) Duration in the study: the number of days between the first and last day of data contributed by the participant in a selected engagement observation period. (2) Longitudinal data-availability vector: a binary-encoded vector representing the density of the participant’s contributed data in an engagement observation period, where the i-th element of the vector represents the i-th day in the study and is set to 1 if a data point is contributed by the participant on that day or is set to 0 otherwise. To align the frequency of passive data streams (daily), for the Phone-Active data, we set the 14 elements (2-week period) of the data-availability vector to 1 in which a survey was completed by a participant. We calculated these two metrics of engagement for each of the three data streams (Phone-Active, Phone-Passive, and Fitbit-Passive data), respectively.

Variables of interest

A variety of factors may affect the duration and density of participants’ engagement in remote digital studies [20, 50, 51]. In this engagement study, we considered a variety of factors including participants' socio-demographics, the enrollment site, baseline depression symptom severity as well as app usage behavior (survey response time and survey completion time) as variables of interest. These are briefly described below.

Socio-demographics

Age, gender, education, marital status, income, and accommodation type, were recorded in the enrollment session.

Study-site

Participant recruitment site (KCL, CIBER, and VUMC).

Baseline Depression symptom severity

Depressive symptom severity was estimated by the PHQ8 survey administered through the study app at the time of enrollment and every subsequent two weeks. The PHQ8 contains 8 questions and the total score of PHQ8 ranges from 0 to 24 with increasing severity of depressive symptoms [48]. We considered the PHQ8 surveys completed at enrollment to represent the participants’ baseline depression severity.

Survey response and completion time

Survey response time is calculated as the time elapsed between the notification arrival time in the study app and the time at which participants started responding to the survey. Survey completion time was the total time participants spent completing the survey. Several studies suggested that the response time and the speed of answering questions could reflect the participants’ attitude strength to the survey [24–26]. Therefore, we used these two metrics to reflect participants’ interests and enthusiasm about the study and test whether they are linked to long-term engagement patterns. Both metrics were calculated for the two surveys (PHQ8 and RSES).

Statistical Analysis

We used a survival modeling approach [52] to assess participants' overall duration in the study (retention). Survival models are commonly used in medical research for exploring associations between time passed before some events occur and one or more predictor variables [53]. The survival models were also used in a recent participant retention study [20]. In our participant retention analysis, the event is the participant disengaging from the study app (stopping contributing data to the study) and the elapsed time is the duration in the study (described above).

Data Harmonization: We conducted our survival analysis on two separate observation periods. The first, referred to as our primary cohort, has an observation period of 43 weeks. This period matches the number of weeks between the date the last patient enrolled in RADAR-MDD (June 2020) and the end of data collection in RADAR-MDD (April 2021). Therefore, it represents the common maximum theoretical survival observation period for all participants enrolled in the RADAR-MDD study. We used this cohort for the presented primary analysis. We also defined a secondary cohort with a survival observation period of 94 weeks. This longer period of observation represents the maximum survival observation period for 50% of participants enrolled in RADAR-MDD. Using this secondary cohort, we aimed to investigate even longer-term participant behavior patterns in remote studies.

Participant Retention Analysis: We first used Kaplan-Meier curves [54] to measure the overall participant retention rates over the two observation periods for three data streams, respectively. To further assess the joint effect of multiple variables of interest on participants’ retention in the study, we used the Cox Proportional-Hazard (CoxPH) model [29]. We considered the baseline PHQ8 score, socio-demographics (age, gender, marital status, children, years in education, annual income, and accommodation type), and the study site as predictor variables. If the duration in the study of a participant is equal to the cutoff observation period, we consider the participant to be engaged in the study (no event). To minimize undue influence associated with periodic disengagement (i.e., some participants stop engaging for a while, then re-engage), the right-censoring method [54] was used for participants whose duration in the study was less than the observation period. We relaxed the determination of the event by considering 4 more weeks after the cut-off day. For example, if a participant’s last active survey was completed on Week 30 within the first 43 weeks (using the primary cutoff observation period), but if they completed more active surveys between Week 44-Week 47 (4-week extension), we still considered this participant was engaged in contributing active data in the study (no event). Otherwise, if there was no completed survey during 4 weeks after the cut-off day, we considered this participant stopped contributing active data in the study (the event happened i.e participant stopped contributing Phone-Active data to the study). Note, the same methodology was used to counter periodic disengagement in the Phone-Passive, and Fitbit-Passive data. To assess the joint effect of multiple variables of interest on retention, we used separate CoxPH models for Phone-Active, Phone-Passive, and Fitbit-Passive data across the two observation periods (43 weeks and 94 weeks). The assumptions of proportional hazards [55], which means that relative hazard remains constant over time with different predictors, of all CoxPH models were tested using the scaled Schoenfeld residuals [30].

Clustering Analysis

Long-term participant pattern modeling: We used an unsupervised K-means clustering method [50] to explore potential latent patterns of participant long-term engagement in the study using the longitudinal data-availability vector (defined above). The elbow method was used to determine the optimal number of clusters [56]. The Kruskal-Wallis test was used to assess any potential enrichment of variables of interest (described above) across the clusters [28]. The same approach was applied to the three data streams and across the two observation periods.

Acknowledgments

The RADAR-CNS project has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 115902. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA, www.imi.europa.eu. This communication reflects the views of the RADAR-CNS consortium and neither IMI nor the European Union and EFPIA are liable for any use that may be made of the information contained herein. The funding body has not been involved in the design of the study, the collection or analysis of data, or the interpretation of data. Participants in the CIBER site came from the following four clinical communities in Spain: Parc Sanitari Sant Joan de Déu Network services, Institut Català de la Salut, Institut Pere Mata, and Hospital Clínico San Carlos.

Participant recruitment in Amsterdam was partially accomplished through Hersenonderzoek.nl (www.hersenonderzoek.nl), a Dutch online registry that facilitates participant recruitment for neuroscience studies. Hersenonderzoek.nl is funded by ZonMw-Memorabel (project no 73305095003), a project in the context of the Dutch Deltaplan Dementie, Gieskes-Strijbis Foundation, the Alzheimer's Society in the Netherlands and Brain Foundation Netherlands.

This paper represents independent research part-funded by the National Institute for Health Research (NIHR) Maudsley Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

We thank all the members of the RADAR-CNS patient advisory board for their contribution to the device selection procedures, and their invaluable advice throughout the study protocol design.

This research was reviewed by a team with experience of mental health problems and their careers who have been specially trained to advise on research proposals and documentation through the Feasibility and Acceptability Support Team for Researchers (FAST-R): a free, confidential service in England provided by the National Institute for Health Research Maudsley Biomedical Research Centre via King’s College London and South London and Maudsley NHS Foundation Trust.

RADAR-MDD was conducted per the Declaration of Helsinki and Good Clinical Practice, adhering to principles outlined in the NHS Research Governance Framework for Health and Social Care (2nd edition). Ethical approval has been obtained in London from the Camberwell St Giles Research Ethics Committee (REC reference: 17/LO/1154), in London from the CEIC Fundacio Sant Joan de Deu (CI: PIC-128-17) and in The Netherlands from the Medische Ethische Toetsingscommissie VUmc (METc VUmc registratienummer: 2018.012 – NL63557.029.17).

Data availability

The datasets used for the present study can be made available through reasonable requests to the RADAR-CNS consortium. Please email the corresponding author for details.

Code availability

The complete code used for the analysis is available through a GitHub code repository (https://github.com/kcl-bhi/RADAR_MDD_Pariticipant_Engagement_and_Retention_Analysis).

Author’s contributions

A.P., Y.Z., A.F., and R.D. contributed to the design of the study. Y.Z. and A.P. extracted and summarized the active survey and passive data from the Remote Assessment of Disease and Relapse–Major Depressive Disorder study, planned and performed the analysis, and drafted the manuscript. M.H. is the principal investigator for the RADAR-MDD study. R.D., A.F., Y.R., Z.R., P.C., and C.S. have contributed to the development of the RADAR-base platform for data collection. F.M., K.W., C.O., A.I., F.L., S. Siddi, C.H.R., S. Simblett, J.M.H., B.W.J.H.P., and M.H. contributed to participant recruitment and data collection. A.F., I.M.G., V.A.N., T.W., P.A., M.H., and R.D. contributed the administrative, technical, and clinical support of the study. All authors were involved in reviewing the manuscript, had access to the study data, and provided direction and comments on the manuscript.

Competing interests

S.V. and V.A.N. are employees of Janssen Research and Development LLC. P.A. is employed by the pharmaceutical company H. Lundbeck A/S. D.C.M. has accepted honoraria and consulting fees from Apple Inc, Otsuka Pharmaceuticals, Pear Therapeutics, and the One Mind Foundation; has received royalties from Oxford Press; and has an ownership interest in Adaptive Health Inc. M.H. is the principal investigator of the Remote Assessment of Disease and Relapse–Central Nervous System project, a private public precompetitive consortium that receives funding from Janssen, UCB, Lundbeck, MSD, and Biogen.

Cai N, Choi KW, Fried EI. Reviewing the genetics of heterogeneity in depression: operationalizations, manifestations and etiologies. Human molecular genetics. 2020;29(R1):R10-R18.
Klasen F, Otto C, Kriston L, Patalay P, Schlack R, Ravens-Sieberer U. Risk and protective factors for the development of depressive symptoms in children and adolescents: results of the longitudinal BELLA study. European child & adolescent psychiatry. 2015;24(6):695–703.
Snyder M, Zhou W. Big data and health. The Lancet Digital Health. 2019;1(6):e252-e254.
Gilchrist G, Gunn J. Observational studies of depression in primary care: what do we know? BMC Family practice. 2007;8(1):1–18.
Liew CS, Wah TY, Shuja J, Daghighi B. Mining personal data using smartphones and wearable devices: A survey. Sensors. 2015;15(2):4430–4469.
Bardram JE, Matic A. A decade of ubiquitous computing research in mental health. IEEE Pervasive Computing. 2020;19(1):62–72.
Bailon C, Damas M, Pomares H, et al. Smartphone-based platform for affect monitoring through flexibly managed experience sampling methods. Sensors. 2019;19(15):3430.
Ranjan Y, Rashid Z, Stewart C, et al. RADAR-base: Open source mobile health platform for collecting, monitoring, and analyzing data using sensors, wearables, and mobile devices. JMIR mHealth and uHealth. 2019;7(8):e11734.
Pratap A, Renn BN, Volponi J, et al. Using mobile apps to assess and treat depression in Hispanic and Latino populations: fully remote randomized clinical trial. Journal of medical Internet research. 2018;20(8):e10130.
Matcham F, di San Pietro CB, Bulgari V, et al. Remote assessment of disease and relapse in major depressive disorder (RADAR-MDD): a multi-centre prospective cohort study protocol. BMC psychiatry. 2019;19(1):1–11.
Luik AI, Zuurbier LA, Direk N, Hofman A, Van Someren EJ, Tiemeier H. 24-hour activity rhythm and sleep disturbances in depression and anxiety: A population‐based study of middle‐aged and older persons. Depression and anxiety. 2015;32(9):684–692.
Cho YM, Lim HJ, Jang H, et al. A cross-sectional study of the association between mobile phone use and symptoms of ill health. Environmental health and toxicology. 2016;31
Zhang Y, Folarin AA, Sun S, et al. Relationship Between Major Depression Symptom Severity and Sleep Collected Using a Wristband Wearable Device: Multicenter Longitudinal Observational Study. JMIR mHealth and uHealth. 2021;9(4):e24604.
Zhang Y, Folarin AA, Sun S, et al. Predicting Depressive Symptom Severity Through Individuals’ Nearby Bluetooth Device Count Data Collected by Mobile Phones: Preliminary Longitudinal Study. Original Paper. JMIR Mhealth Uhealth. 2021;9(7):e29840. doi:10.2196/29840
Laiou P, Kaliukhovich DA, Folarin AA, et al. The Association Between Home Stay and Symptom Severity in Major Depressive Disorder: Preliminary Findings From a Multicenter Observational Study Using Geolocation Data From Smartphones. Original Paper. JMIR Mhealth Uhealth. 2022;10(1):e28095. doi:10.2196/28095
Zhang Y, Folarin AA, Sun S, et al. Longitudinal Relationships Between Depressive Symptom Severity and Phone-Measured Mobility: Dynamic Structural Equation Modeling Study. Original Paper. JMIR Ment Health. 2022;9(3):e34898. doi:10.2196/34898
Moore S, Tassé A-M, Thorogood A, Winship I, Doerr M. Consent processes for mobile app mediated research: systematic review. JMIR mHealth and uHealth. 2017;5(8):e126.
Druce KL, Dixon WG, McBeth J. Maximizing engagement in mobile health studies: lessons learned and future directions. Rheumatic diseases clinics of North America. 2019;45(2):159.
De Angel V, Lewis S, White K, et al. Digital health tools for the passive monitoring of depression: a systematic review of methods. NPJ Digital Medicine. 2022;5(1):1–14.
Pratap A, Neto EC, Snyder P, et al. Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants. NPJ digital medicine. 2020;3(1):1–10.
Simblett S, Greer B, Matcham F, et al. Barriers to and facilitators of engagement with remote measurement technology for managing health: systematic review and content analysis of findings. Journal of medical Internet research. 2018;20(7):e10480.
O’connor S, Hanlon P, O’donnell CA, Garcia S, Glanville J, Mair FS. Understanding factors affecting patient and public engagement and recruitment to digital health interventions: a systematic review of qualitative studies. BMC medical informatics and decision making. 2016;16(1):1–15.
Quisel T, Foschini L, Zbikowski SM, Juusola JL. The association between medication adherence for chronic conditions and digital health activity tracking: retrospective analysis. Journal of medical Internet research. 2019;21(3):e11486.
Bassili JN. Meta-judgmental versus operative indexes of psychological attributes: The case of measures of attitude strength. Journal of personality and social psychology. 1996;71(4):637.
Heerwegh D. Explaining response latencies and changing answers using client-side paradata from a web survey. Social Science Computer Review. 2003;21(3):360–373.
Fazio RH, Powell MC, Herr PM. Toward a process model of the attitude–behavior relation: Accessing one's attitude upon mere observation of the attitude object. Journal of Personality and Social Psychology. 1983;44(4):723.
Matcham F, Leightley D, Siddi S, et al. Remote Assessment of Disease and Relapse in Major Depressive Disorder (RADAR-MDD): recruitment, retention, and data availability in a longitudinal remote measurement study. BMC psychiatry. 2022;22(1):1–19.
Ostertagova E, Ostertag O, Kováč J. Methodology and application of the Kruskal-Wallis test. Trans Tech Publ; 2014:115–120.
Kumar D, Klefsjö B. Proportional hazards model: a review. Reliability Engineering & System Safety. 1994;44(2):177–188.
Grambsch PM, Therneau TM. Proportional hazards tests and diagnostics based on weighted residuals. Biometrika. 1994;81(3):515–526.
Wu J. Cluster analysis and K-means clustering: an introduction. Advances in K-means Clustering. Springer; 2012:1–16.
Chien I, Enrique A, Palacios J, et al. A machine learning approach to understanding patterns of engagement with internet-delivered mental health interventions. JAMA network open. 2020;3(7):e2010791-e2010791.
Matcham F, Carr E, White KM, et al. Predictors of engagement with remote sensing technologies for symptom measurement in Major Depressive Disorder. 2022;
Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. Journal of multidisciplinary healthcare. 2016;9:211.
Mody L, Miller DK, McGloin JM, et al. Recruitment and Retention of Older Adults in Aging Research: (See editorial comments by Dr. Stephanie Studenski, pp 2351–2352). Journal of the American Geriatrics Society. 2008;56(12):2340–2348.
Forsat ND, Palmowski A, Palmowski Y, Boers M, Buttgereit F. Recruitment and retention of older people in clinical research: a systematic literature review. Journal of the American Geriatrics Society. 2020;68(12):2955–2963.
Potts C, Bond R, Ryan A, et al. Ecological momentary assessment within a digital health intervention for reminiscence in persons with dementia and caregivers: user engagement study. JMIR mHealth and uHealth. 2020;8(7):e17120.
Ramsey AT, Wetherell JL, Depp C, Dixon D, Lenze E. Feasibility and acceptability of smartphone assessment in older adults with cognitive and emotional difficulties. Journal of Technology in Human Services. 2016;34(2):209–223.
Akhtar-Danesh N, Landeen J. Relation between depression and sociodemographic factors. International journal of mental health systems. 2007;1(1):1–9.
Kanstrup AM, Bertelsen P, Jensen MB. Contradictions in digital health engagement: An activity tracker’s ambiguous influence on vulnerable young adults’ engagement in own health. Digital health. 2018;4:2055207618775192.
Simblett S, Greer B, Matcham F, et al. Barriers to and Facilitators of Engagement With Remote Measurement Technology for Managing Health: Systematic Review and Content Analysis of Findings. Review. J Med Internet Res. 2018;20(7):e10480. doi:10.2196/10480
Goodday SM, Karlin E, Alfarano A, et al. An Alternative to the Light Touch Digital Health Remote Study: The Stress and Recovery in Frontline COVID-19 Health Care Workers Study. JMIR Formative Research. 2021;5(12):e32165.
Bentley JP, Thacker PG. The influence of risk and monetary payment on the research participation decision making process. Journal of medical ethics. 2004;30(3):293–298.
Birnbaum F, Lewis DM, Rosen R, Ranney ML. Patient engagement and the design of digital health. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2015;22(6):754.
Kaye J, Curren L, Anderson N, et al. From patients to partners: participant-centric initiatives in biomedical research. Nature Reviews Genetics. 2012;13(5):371–376.
Anderson N, Bragg C, Hartzler A, Edwards K. Participant-centric initiatives: tools to facilitate engagement in research. Applied & translational genomics. 2012;1:25–29.
Simblett S, Matcham F, Siddi S, et al. Barriers to and Facilitators of Engagement With mHealth Technology for Remote Measurement and Management of Depression: Qualitative Analysis. Original Paper. JMIR Mhealth Uhealth. 2019;7(1):e11325. doi:10.2196/11325
Kroenke K, Strine TW, Spitzer RL, Williams JB, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. Journal of affective disorders. 2009;114(1–3):163–173.
Greenberger E, Chen C, Dmitrieva J, Farruggia SP. Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: Do they matter? Personality and individual differences. 2003;35(6):1241–1254.
Baumel A, Muench F, Edan S, Kane JM. Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. Original Paper. J Med Internet Res. 2019;21(9):e14567. doi:10.2196/14567
Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. Journal of affective disorders. 2020;263:413–419.
Bewick V, Cheek L, Ball J. Statistics review 12: survival analysis. Critical care. 2004;8(5):1–6.
Singer JD, Willett JB. Modeling the days of our lives: using survival analysis when designing and analyzing longitudinal studies of duration and the timing of events. psychological Bulletin. 1991;110(2):268.
Rich JT, Neely JG, Paniello RC, Voelker CC, Nussenbaum B, Wang EW. A practical guide to understanding Kaplan-Meier curves. Otolaryngology—Head and Neck Surgery. 2010;143(3):331–336.
Kleinbaum DG, Klein M. Evaluating the proportional hazards assumption. Survival analysis. Springer; 2012:161–200.
Syakur M, Khotimah B, Rochman E, Satoto BD. Integration k-means clustering method and elbow method for identification of the best customer profile cluster. IOP Publishing; 2018:012017.

There is a conflict of interest S.V. and V.A.N. are employees of Janssen Research and Development LLC. P.A. is employed by the pharmaceutical company H. Lundbeck A/S. D.C.M. has accepted honoraria and consulting fees from Apple Inc, Otsuka Pharmaceuticals, Pear Therapeutics, and the One Mind Foundation; has received royalties from Oxford Press; and has an ownership interest in Adaptive Health Inc. M.H. is the principal investigator of the Remote Assessment of Disease and Relapse–Central Nervous System project, a private public precompetitive consortium that receives funding from Janssen, UCB, Lundbeck, MSD, and Biogen.

Supplement.docx
Supplementary Tables and Figures

Download PDF

Journal Publication

published 17 Feb, 2023

Read the published version in npj Digital Medicine →

Editorial decision: revise
25 Jul, 2022
Review #4 received at journal
24 Jul, 2022
Reviewer #4 agreed at journal
24 Jul, 2022
Review #2 received at journal
22 May, 2022
Reviewer #3 agreed at journal
17 May, 2022
Review #1 received at journal
12 May, 2022
Reviewer #2 agreed at journal
11 May, 2022
Reviewer #1 agreed at journal
09 May, 2022
Reviewers invited by journal
09 May, 2022
Submission checks completed at journal
05 May, 2022
Editor assigned by journal
27 Apr, 2022
First submitted to journal
27 Apr, 2022

You are reading this latest preprint version

Long-term Participant Retention and Engagement Patterns in an App and Wearable-based Multinational Remote Digital Depression Study

Status:

Journal Publication

Version 1

Abstract

Figures

Introduction

Results

Cohort Characteristics

Participant Retention

Participants’ Long-term Engagement Patterns in the study

Discussion

Key learnings from long-term participant engagement patterns in remote studies

Long-term participant retention

Potential reasons for high participant retention in the RADAR-MDD study

Potential solutions to improve participant retention and engagement in remote digital research

Limitations

Conclusion

Methods

RADAR-MDD Study design

Primary Data streams

Phone Active Data

Phone Passive Data

Fitbit Passive Data

Primary Outcomes

Variables of interest

Statistical Analysis

Clustering Analysis

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1