Effects of a mobile health intervention on activities of stress self-management for workers: A randomized controlled trial

DOI: https://doi.org/10.21203/rs.3.rs-988972/v1

Abstract

Background: Mobile health interventions are being widely tried because of their attractive advantages. However, there is not enough evidence for its effectiveness. This study aimed to evaluate the effects of mobile app-based stress management intervention (mSMI) on highly stressed workers.

Methods: A sample of 82 white-collar workers with elevated symptoms of perceived stress (Perceived Stress Scale-10≥22) were randomly assigned to the intervention or control group. The mSMI consisted of three modules: self-management including a work diary, counseling based on cognitive behavioral therapy and interventions focused on music, meditation, relaxation and image healings. Self-report data were collected the baseline and post-intervention. Study outcomes were perceived stress, anxiety, depression, and work engagement. Data were assessed using analysis of covariance with covariates.

Results: There was significantly reduced perceived stress from baseline to 6 weeks in mSMI (t=5.788, p<.001) and control group (t=3.184, p=.003). After adjusting for covariates, the between-group difference in the perceived stress was significantly different (F=4.051, p=.048); however, the effect size was small. There was no significant intervention effect on anxiety, depression, and work engagement. The process evaluation indicated that most participants (85.3%) were satisfied with the intervention and their mental health benefited.

Conclusion: This study found that mobile health intervention facilitated stress management for highly stressed workers. Further studies should address job-related outcomes and mental health symptoms in workers by applying the latest information technology and addressing the limitations of mobile interventions.

Trial Registration: Not applicable.

Introduction

Workers suffer from various mental health problems, including stress, depression, anxiety, posttraumatic stress disorder (PTSD), self-stigma [1], and insomnia [2]. These mental health problems may manifest as physical diseases, including cancer and heart disease [3], and extended work hours [4], workplace bullying [5], workplace violence [6], and disaster [7]. Therefore, workers should engage in mental health-promoting activities that facilitate inspection and management of their mental health.

To keep up with the advances in information technology, mental health interventions are being recently offered as internet-based programs [8, 9], with utilization of mobile communication and social media [10]. Mobile health (mHealth) interventions that utilize smartphone apps help overcome geographical distance and allow limited face-to-face access [11]. Further, mHealth self-care interventions can be simultaneously administered to many people, which makes them an inexpensive strategy [3].

A systematic review by Wang et al [11] on studies regarding mHealth interventions targeting stress, anxiety, depression, alcohol disorder, sleep disorder, suicidal behaviors, and PTSD found that 14 out of 16 apps significantly reduced mental health symptoms. Further, an analysis of 27 studies regarding mobile app-based interventions and short message service (SMS) demonstrated that mobile apps are highly effective (effect size 0.05–3.37) in reducing stress, anxiety, and depression immediately after the intervention. Moreover, SMS was found to be useful in delivering psychoeducation [12].

Recently, studies on mindfulness meditation mobile-app intervention for university students to reduce stress decreased perceived stress [13, 14]. However, there have been inconsistent reports by studies on workers. One study [15] found that an in-person intervention for workers with work-related stress was more useful than a smartphone-delivered intervention. Specifically, the in-person intervention reduced stress and emotional burnout as well as promoted well-being; contrastingly, the smartphone-delivered intervention only promoted well-being. A mindfulness-based mHealth intervention for healthcare workers during the coronavirus disease 2019 (COVID-19) pandemic significantly improved depression, anxiety, stress, and insomnia in the group receiving psychotropic medications or psychotherapy compared with the control group [16]. This shows that mHealth interventions are effective when coupled with other treatment modalities. Therefore, there remains limited evidence supporting the effectiveness of mHealth interventions in workers.

We aimed to develop and evaluate the effects of a mobile app-based stress management intervention (mSMI) for highly stressed workers familiar with mobile devices who require mental health management.

Methods

Study design

We used an equivalent comparison group pretest-posttest design to evaluate the effectiveness of mSMI in workers with high perceived stress.

Participants

We enrolled workers aged ≥20 years with high perceived stress (Perceived Stress Scale-10 [PSS-10] score ≥22 at the screening test [9]) who owned an Android-based mobile device and could use mobile apps. The exclusion criteria were a PSS-10 score<22, a self-reported diagnosis of mental illnesses, and current use of psychotherapy or psychotropic medications. This study was approved by the Institutional Review Board before data collection (CUIRB-2020-0041).

The sample size was calculated using the G* Power 3.1.7 software [17]. As a reference, we used the effect sizes reported by a study on an internet-based stress intervention for workers (Cohen d = 0.75) [8]. For an effect size of 0.70, significance level of 0.05, and power of 0.80 for two-tailed tests, the minimum sample size was calculated to be 68. Considering a 20% drop-out rate, 82 participants were required.

We performed convenience sampling of office workers at public administration, information technology (IT), education, and health industries through health managers and a recruitment announcement. Among 634 individuals who participated in the screening test, we enrolled 82 eligible participants (selection rate 12.9%). The enrolled participants were randomly allocated to the experimental or control group. At the 6-week posttest survey, there were 64 participants in all groups, indicating a dropout rate of 22.0% (Figure 1).

Intervention

To determine the contents of the mSMI, we searched for studies in databases using keywords, including “mental health,” “stress,” “workers,” “application,” “intervention,” and “self-management.” The following contents and features for the mobile app were identified through collaboration with a mental health care nurse. First, there was a “self-management” feature. Participants were instructed to log in the times they start and finish work to monitor their daily work hours as well as check their daily stress levels and work intensity. Additionally, the participants could record their self-stress management efforts and methods, including taking walks and exercising. Second, a “counseling” feature was added to help workers requiring counseling and assistance. Participants were asked to describe their feelings and thoughts during a specific problem situation, their current thoughts and feelings, and how they would deal with similar situations in the future. Here, the contents were devised based on cognitive behavioral therapy; additionally, two assigned mental health care professionals checked these notes and replied within 24 hours. Third, to facilitate stress relief, features and techniques such as music [18], meditation [14], relaxation [19], and image healing [20] were utilized. We included a poem recital feature with music and a breathing exercise [20] (Figure 2).

The contents of the developed app were evaluated by an expert panel comprising college professors, nurses, and clinical psychologists. Furthermore, all items were confirmed to have an item content validity index (I-CVI) ≥0.80. Additionally, the quality of the developed app was rated using the Mobile Application Rating Scale (out of 5) [21]. The app was rated 4.16 and 3.72 by five experts and five non-experts, respectively. After correcting errors and modifying the app based on the feedback, the mSMI (called MindCafe) for workers was finalized.

The mSMI was provided as a self-guided intervention. The certification process, how-to guides for each feature, and information regarding privacy policies were explained on the Frequently Asked Questions (FAQ) page. Participants were asked to use the app for 10 minutes twice a week for 6 weeks. During the intervention period, we sent out an SMS thrice a week to encourage participation [12]. The SMS contained messages for motivating the participants to utilize the counseling, music, meditation, and relaxation features. Further, we opened up a chat channel (Kakao channel) for participants to ask questions regarding the app during the intervention period, including app downloading problems, instructions for each feature, and data entry, which we responded to around the clock. Using self-report questionnaires, data were collected at the baseline (before intervention) and 6 weeks after completing the intervention. The control group did not undergo any intervention.

Measures

Perceived stress

We measured perceived stress using the PSS-10, which was developed by Cohen [22] and translated into Korean and validated by Lee and colleagues [23]. This scale comprises 10 items, with a higher score indicating higher perceived stress.

Anxiety and Depression

We measured anxiety and depression using the Hospital Anxiety and Depression Scale. This scale was developed for patients with physical illness; however, it has been validated for use in the general population and has been used for a large worker population in Korea [4]. This scale comprises 14 items, with seven items each for depression and anxiety. The total score for each subscale is 21, with a higher score indicating more severe anxiety or depression symptoms.

Work engagement

Work engagement was measured using the Utrecht Work Engagement Scale–Korean Version [24], which is the Korean version of the 9-item short form of the UWES developed by Schaufeli and Bakker [25]. This scale comprises three subscales, vigor (three items), dedication (three items), and absorption (three items), with a higher score indicating greater work engagement.

Statistical analyses

Statistical analyses were performed following the intention-to-treat principle (ITT) using the SPSS/WIN 24.0 software (IBM, Armonk, New York, USA). We used the multiple imputation procedure to impute missing sum scores for participants who didn’t complete the post intervention evaluation. Between-group differences in general characteristics were analyzed using the chi-square test, independent t-test, and analysis of variance. Within- and between-group differences in post-intervention changes were analyzed using a paired t-test and analysis of covariance (ANCOVA) with the baseline scores and age as covariates.

Results

Table 1 shows the results of the between-group homogeneity test. The mean age was 38.0 ± 8.96 and 32.7 ± 7.23 years in the experimental and control groups, respectively (t = 3.000, p = .004). There were no significant between-group differences in gender, marital status, education level, income, and workplace size.


 
Table 1

General Characteristics of Participants and Differences Between-group

Variables

Categories

Total

(N=82)

mSMI

(N=39)

Control

(n=43)

χ2/t/F

p

N (%) or Mean±SD

N (%) or Mean±SD

N (%) or Mean±SD

Gender

Male

Female

18(22.0)

64(78.0)

9(23.1)

30(76.9)

9(20.9)

34(79.1)

0.055

.815

Age(years)

<30

30~39

≥40

35.2±8.49

28(34.2)

32(39.0)

22(26.8)

38.0±8.96

8(20.5)

16(41.0)

15(38.5)

32.7±7.23

20(46.5)

16(37.2)

7(16.3)

3.000

7.876

.004

.019

Marital status

Unmarried

Married

44(53.7)

38(46.3)

17(43.6)

22(56.4)

27(62.8)

16(37.2)

3.032

.082

Education

≤University

≥Master

65(79.3)

17(20.7)

30(76.9)

9(23.1)

35(81.4)

8(18.6)

0.249

.618

Income

(won millions)

<400

400~499

≥500

49(59.8)

16(19.5)

17(20.7)

19(48.7)

8(20.5)

12(30.8)

30(69.8)

8(18.6)

5(11.6)

5.169

.075

Workplace size (number of employees)

<50

50~299

300~1,999

≥2,000

20(24.4)

17(20.7)

18(22.0)

27(32.9)

8(20.5)

9(23.1)

9(23.1)

13(33.3)

12(27.9)

8(18.6)

9(20.9)

14(32.6)

0.702

.873


Table 2 shows post-intervention changes within each group. There was a significant post-intervention reduction in perceived stress in the experimental group (t = 5.788, p < .001) and control group (t = 3.184, p = .003). However, there were no significant post-intervention changes in anxiety, depression, and work engagement.


 
Table 2

Changes of Post-intervention Within-group (mSMI=39, Control=43)

Variables

Categories

Baseline

(Mean±SD)

6 weeks

(Mean±SD)

Difference* (Mean±SD)

t-test

(p)

PSS-10

mSMI

24.79±2.34

21.20±4.15

3.60±3.88

5.788(<.001)

Control

24.44±2.56

22.23±4.87

2.21±4.56

3.184(.003)

Anxiety

mSMI

9.44±3.60

8.59±2.73

0.85±3.40

1.554(.129)

Control

8.67±3.29

8.89±3.06

-0.21±2.77

-0.506(.616)

Depression

mSMI

10.79±3.15

10.70±3.37

0.09±3.46

0.165(.870)

Control

9.81±2.66

9.93±2.70

-0.12±2.35

-0.330(.743)

UWES

mSMI

25.15±8.63

26.54±10.14

-1.38±5.83

-1.479(.147)

Control

24.26±8.23

25.33±8.24

-1.07±5.85

-1.203(.236)

*Scores at baseline–Scores at 6 weeks; PSS=Perceived Stress Scales; UWES=Utrecht Work Engagement Scale


Table 3 shows the between-group differences in the post-intervention scores. There were between-group differences in the age distribution. Moreover, since the baseline score affects the post-intervention score, we adjusted for age and baseline score as covariates. There was a significant between-group difference in the post-intervention perceived stress score (F = 4.051, p = .048); however, the effect size was small (0.23). There were no significant between-group differences in anxiety, depression, and work engagement scores.


 
Table 3

Differences in Post-intervention Between-group (mSMI=39, Control=43)

Variables

Difference in Means*

(95% CI)

ANCOVA

F(p)

Adjusted R2

PSS-10

-1.03 (-3.03 to 0.97)

4.051(.048)

.176

Anxiety

-0.30 (-1.58 to 0.98)

1.960(.165)

.272

Depression

0.77 (-0.57 to 2.11)

0.064(.801)

.251

UWES

1.21 (-2.84 to 5.25)

0.082(.776)

.601

*Post-intervention scores in the experimental group–Post-intervention scores in the control group; Controlling for baseline scores and age; 95% CI=95% Confidence Interval; ANCOVA=Analysis of covariance; PSS=Perceived Stress Scales; UWES=Utrecht Work Engagement Scale


Table 4 shows the users’ comments regarding their experiences and satisfaction with the mobile app. Most users (85.3%) were satisfied with the app. Two-thirds of the users commented that the mobile app increased their stress management awareness. Furthermore, more than half of the users stated that the mobile app motivated them to change their attitude and behavior towards stress management. Other comments included requests for alert settings, stress measurement using a sensor, music repeat, and periodic content updates.


 
Table 4

The Users’ Comments with Mobile Application (N=34)

Items

Categories

N (%)

Other comments

Overall satisfaction with mobile application

Satisfied

29 (85.3)

  Alarm settings

  Stress measurement using sensor

  Music repeat

  Periodic contents updates

Dissatisfied

5 (14.7)

Does this mobile application increase your stress management awareness?

Agree

23 (67.6)

 

No change

9 (26.5)

 

Disagree

2 (5.9)

 

Does this mobile application improve your attitude toward stress management?

Agree

18 (52.9)

 

No change

11 (32.4)

 

Disagree

5 (14.7)

 

Does this mobile application bring behavioral change in stress management?

Agree

20 (58.8)

 

No change

9 (26.5)

 

Disagree

5 (14.7)

 


Discussion

This study developed an mSMI for highly stressed workers and evaluated its effects. We found that mSMI effectively reduced perceived stress. This is consistent with previous findings that a mobile app-based stress management program (called Mind Healer) for nurses [18] and a mindfulness-based mHealth intervention for health care workers during the COVID-19 pandemic reduced perceived stress [16].

However, we found a small effect size of mSMI on perceived stress. A recent meta-review of seven meta-analyses on mobile app mental health interventions reported a small to medium effect size of mobile apps for stress management [26]. Contrastingly, a mobile application administered for 4 weeks to nurses had a high effect size on perceived stress [18]. Our observed small effect size could be attributed to the inadequate treatment dose. We instructed participants to use the mSMI for at least 10 minutes per session twice a week for 6 weeks. However, our logs showed that the participants accessed the app for an average of 10.4 minutes per week. Since we used a self-guided intervention, we delivered information regarding how-to instructions for each feature and personal information processing through the FAQ page and SMS. Additionally, we answered participants’ questions through a chat channel. Nevertheless, the participants did not adequately use the interventions. In the post-intervention satisfaction survey, participants mentioned that the periodic reminder text messages were stressful for them. There didn’t reinforce program engagement. Facilitating program engagement requires an approach that includes enhancing participants’ initial motivation, frequent and effective reminders, education on the importance of active participation, and financial and verbal incentives [27].

Also, as in the method of this study, the limitation of non-face-to-face interventions could have contributed to the failure to induce marked stress relief. A study [15] that administered a 6-week resilience training to health care workers with work-related stress reported that a smartphone-delivered intervention only improved well-being. Contrastingly, an in-person intervention reduced stress and emotional burnout as well as promoted well-being. Therefore, one or two face-to-face coaching sessions should be included to boost the effectiveness of mHealth interventions. If in-person interventions are not possible due to factors, including the recent COVID-19 pandemic, video conferences (e.g., Zoom, Meet, Teams, and Webex) should be conducted at the participants’ convenience to emphasize the importance of self-management and encourage steady participation.

In our study, mSMI was not effective for anxiety, depression, and work engagement. This is consistent with a previous report that a 4-week stress-management intervention for nurses using a mobile app did not significantly mitigate depression, anxiety, and emotional labor [18]. However, meta analyses presented [26, 28] that mobile-app interventions significantly improved on depressive or anxiety symptoms with medium effect size. In particular, a 3-week intervention to reduce stress for university students using a cognitive behavioral therapy-based mobile app (called Feel Stress Free) changed anxiety and depression symptoms [29]. The eHealth stress management for employee has already improved anxiety [9], insomnia [8, 9], burnout [8, 9], work engagement [9], and work experience [8]. Digital interventions improved work and social functions as well as reduced depression and anxiety [30]. Future studies on mHealth intervention should examine changes in mental health symptoms or job-related outcomes among workers. Anxiety and depression increase by 2.15 and 1.87 times, respectively, among manufacturing workers working >60 hours a week [4]. Therefore, there is a need for intervention studies to examine mental health symptoms with respect to extended work hours.

In our study, participants were generally satisfied with the app; however, they suggested several improvements, including incorporating the latest IT such as automatic stress measurement using a sensor or voice control features. User acceptability and adherence rates are especially critical for mobile interventions [31]. Specifically, emotionally relaxed, warm, and appealing designs should be applied to boost users’ adherence rates to ensure effective prevention and treatment of mental health conditions. Therefore, there is a need for customized mobile apps tailored to specific target populations by incorporating the latest technology to apps promoting mental health management.

This study has some limitations. First, the treatment dose was lower than planned, which could have compromised the internal validity. Second, the study was conducted during the COVID-19 pandemic; therefore, external factors, including COVID-19-related depression, may have influenced the intervention outcomes. However, this study is significant since we administered an individualized mHealth intervention on highly stressed workers for preventing mental health problems.

Conclusion

This study developed an mHealth intervention and evaluated its stress management effects on workers highly vulnerable to mental health problems. We found that a self-guided, non-face-to-face mSMI for 6 weeks reduced perceived stress; however, it did not improve anxiety, depression, and work engagement in office workers. Future studies should develop mHealth interventions addressing the shortcomings of mobile-based interventions.

References

  1. Lee YJ, Kim EH: A Systematic Review of Workplace Interventions for the Mental Health of Workers. J Korean Acad Psychiatr Ment Health Nurs. 2020, 29(2):155–167.
  2. Omeogu C, Shofer F, Gehrman P, Green-McKenzie J: Efficacy of a Mobile Behavioral Intervention for Workers with Insomnia. J Occup Environ Med. 2020, 62(3):246–250.
  3. Matcham F, Rayner L, Hutton J, Monk A, Steel C, Hotopf M: Self-help interventions for symptoms of depression, anxiety and psychological distress in patients with physical illnesses: a systematic review and meta-analysis. Clin Psychol Rev 2014, 34(2):141–157.
  4. Kang W, Park WJ, Jang KH, Lim HM, Ann JS, Cho SH, Moon JD: Comparison of anxiety and depression status between office and manufacturing job employees in a large manufacturing company: a cross sectional study. Ann Occup Environ Med. 2016, 28:47.
  5. Verkuil B, Atasayi S, Molendijk ML: Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data. PLoS One. 2015, 10(8):e0135225.
  6. Tarquinio C, Rotonda C, Houllé WA, Montel S, Rydberg JA, Minary L, Dellucci H, Tarquinio P, Fayard A, Alla F: Early psychological preventive intervention for workplace violence: A randomized controlled explorative and comparative study between EMDR-recent event and critical incident stress debriefing. Issues Ment Health Nurs. 2016, 37(11):787–799.
  7. Difede J, Malta LS, Best S, Henn-Haase C, Metzler T, Bryant R, Marmar C: A randomized controlled clinical treatment trial for World Trade Center attack-related PTSD in disaster workers. J Nerv Ment Dis. 2007, 195(10):861–865.
  8. Persson Asplund R, Dagöö J, Fjellström I, Niemi L, Hansson K, Zeraati F, Ziuzina M, Geraedts A, Ljótsson B, Carlbring P et al: Internet-based stress management for distressed managers: Results from a randomised controlled trial. Occup Enviro Med. 2018, 75(2):105–113.
  9. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, Lehr D: Self-guided internet-based and mobile-based stress management for employees: Results of a randomised controlled trial. Occup Environ Med. 2016, 73(5):315–323.
  10. Boulos MN, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP: Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform. 2014, 5(3):229.
  11. Wang K, Varma DS, Prosperi M: A systematic review of the effectiveness of mobile apps for monitoring and management of mental health symptoms or disorders. J Psychiatr Res. 2018, 107:73–78.
  12. Rathbone AL, Prescott J: The Use of Mobile Apps and SMS Messaging as Physical and Mental Health Interventions: Systematic Review. J Med Internet Res. 2017, 19(8):e295.
  13. Yang E, Schamber E, Meyer RML, Gold JI: Happier Healers: Randomized Controlled Trial of Mobile Mindfulness for Stress Management. J Altern Complement Med. 2018, 24(5):505–513.
  14. Huberty J, Green J, Glissmann C, Larkey L, Puzia M, Lee C: Efficacy of the Mindfulness Meditation Mobile App "Calm" to Reduce Stress Among College Students: Randomized Controlled Trial. JMIR Mhealth Uhealth. 2019, 7(6):e14273.
  15. Mistretta EG, Davis MC, Temkit M, Lorenz C, Darby B, Stonnington CM: Resilience Training for Work-Related Stress Among Health Care Workers: Results of a Randomized Clinical Trial Comparing In-Person and Smartphone-Delivered Interventions. J Occup Environ Med. 2018, 60(6):559–568.
  16. Fiol-DeRoque MA, Serrano-Ripoll MJ, Jiménez R, Zamanillo-Campos R, Yáñez-Juan AM, Bennasar-Veny M, Leiva A, Gervilla E, García-Buades ME, García-Toro M et al: A Mobile Phone-Based Intervention to Reduce Mental Health Problems in Health Care Workers During the COVID-19 Pandemic (PsyCovidApp): Randomized Controlled Trial. JMIR Mhealth Uhealth. 2021, 9(5):e27039.
  17. Faul F, Erdfelder E, Buchner A, Lang A-G: Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behav Res Methods. 2009, 41:1149–1160.
  18. Hwang WJ, Jo HH: Evaluation of the Effectiveness of Mobile App-Based Stress-Management Program: A Randomized Controlled Trial. Int J Environ Res Public Health. 2019, 16(21).
  19. Scott AJ, Webb TL, Rowse G: Self-help interventions for psychosis: a meta-analysis. Clin Psychol Rev. 2015, 39:96–112.
  20. Mayo-Wilson E, Montgomery P: Media-delivered cognitive behavioural therapy and behavioural therapy (self-help) for anxiety disorders in adults. Cochrane Database Syst Rev. 2013(9):Cd005330.
  21. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M: Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015, 3(1):e27.
  22. Cohen S: Perceived stress in a probability sample of the United States. In: The social psychology of health. Thousand Oaks, CA, US: Sage Publications, Inc; 1988: 31–67.
  23. Lee J, Shin C, Ko Y-H, Lim J, Joe S-H, Kim S, Jung I-K, Han C: The Reliability and Validity Studies of the Korean Version of the Perceived Stress Scale. Korean J Psychosom Med. 2012, 20(2):127–134.
  24. Kim WH, Park JG, Kwon B: Work Engagement in South Korea: Validation of the Korean Version 9-Item Utrecht Work Engagement Scale. Psychol Rep. 2017, 120(3):561–578.
  25. Schaufeli WB, Bakker AB, Salanova M: The Measurement of Work Engagement With a Short Questionnaire:A Cross-National Study. Educ Psychol Meas. 2006, 66(4):701–716.
  26. Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, Francoeur A, Pennou A, Khazaal Y: Mobile Apps for Mental Health Issues: Meta-Review of Meta-Analyses. JMIR Mhealth Uhealth. 2020, 8(5):e17458.
  27. Zeng Y, Guo Y, Li L, Hong YA, Li Y, Zhu M, Zeng C, Zhang H, Cai W, Liu C et al: Relationship Between Patient Engagement and Depressive Symptoms Among People Living with HIV in a Mobile Health Intervention: Secondary Analysis of a Randomized Controlled Trial. JMIR Mhealth Uhealth. 2020, 8(10):e20847.
  28. Linardon J, Cuijpers P, Carlbring P, Messer M, Fuller-Tyszkiewicz M: The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry. 2019, 18(3):325–336.
  29. McCloud T, Jones R, Lewis G, Bell V, Tsakanikos E: Effectiveness of a Mobile App Intervention for Anxiety and Depression Symptoms in University Students: Randomized Controlled Trial. JMIR Mhealth Uhealth. 2020, 8(7):e15418.
  30. Sin J, Galeazzi G, McGregor E, Collom J, Taylor A, Barrett B, Lawrence V, Henderson C: Digital Interventions for Screening and Treating Common Mental Disorders or Symptoms of Common Mental Illness in Adults: Systematic Review and Meta-analysis. J Med Internet Res. 2020, 22(9):e20581.
  31. Rismawan W, Marchira CR, Rahmat I: Usability, Acceptability, and Adherence Rates of Mobile Application Interventions for Prevention or Treatment of Depression: A Systematic Review. J Psychosoc Nurs Ment Health Serv. 2021, 59(2):41–47.