Quantitative findings
Training evaluation
Independent t-tests were conducted to assess whether case and non-case studies exhibited similar perception, knowledge and skills around eMH at pre- and post-training. Results showed that in general participants’ levels of confidence, skills and knowledge of eMH did not differ significantly across case and non-case studies. However, at pre-training, participants in the case studies exhibited higher confidence in communicating about wellbeing (t(60.218) = 4.068, p = .000) and perceived eMH to be more effective than those in the non-case studies (t(34.328) = 2.654, p = .012). While post-training, participants from the non-case studies exhibited higher levels of eMH knowledge than those in the case studies (t(64) = -3.033, p = .003). The means and standard deviations are reported in Table 3. Note that 10 is the maximum score.
Paired t-tests were conducted to compare levels of confidence, knowledge and skills of eMH from pre- to post-training for all participants. Results showed significant improvements across all measures. Mean (M) and Standard Deviation (SD) are shown in Table 4. For the pre- training evaluation questions on ‘accessibility’, ‘appropriateness’ and ‘effectiveness’ of e-mental health for Indigenous people, 9, 11, and 19 participants respectively selected ‘don’t know’, indicating insufficient knowledge to answer the questions. However, fewer participants, i.e., 5, 5, and 7 participants selected ‘don’t know’ for the same questions post-training, indicating increased knowledge on the topics.
Table 3. Pre- and post-training ratings for case and non-case study participants
|
Pre-training M (SD)
|
|
Post-training M (SD)
|
|
Item
|
Case
|
Non-case
|
Case
|
Non-case
|
1. Confidence with wellbeing concerns
|
7.61 (1.16)
|
6.10 (1.80)
|
7.66 (1.09)
|
7.58 (1.47)
|
2. eMH knowledge
|
4.19 (2.16)
|
3.56 (1.85)
|
5.73 (2.13)
|
7.07 (1.45)
|
3. Ipad/tablet competency
|
7.26 (2.26)
|
6.96 (2.37)
|
7.63 (1.97)
|
8.00 (1.50)
|
4. Computer competency
|
7.51 (1.73)
|
7.75 (1.54)
|
7.91 (1.64)
|
8.33 (1.08)
|
5. Confidence in using SSA
|
5.88 (2.37)
|
4.85 (2.31)
|
7.31 (1.79)
|
7.67 (1.43)
|
6. Confidence in using eMH
|
5.84 (2.18)
|
5.37 (2.25)
|
6.94 (1.92)
|
7.35 (1.33)
|
7. eMH referrals competency
|
5.55 (2.41)
|
5.30 (2.31)
|
6.55 (2.10)
|
6.79 (1.53)
|
8. Accessibility to eMH
|
4.47 (1.73)
|
3.44 (1.93)
|
6.56 (2.00)
|
6.11 (1.93)
|
9. Appropriateness of eMH
|
6.29 (1.60)
|
5.73 (2.50)
|
7.42 (1.68)
|
7.10 (1.85)
|
10. Effectiveness of eMH
|
6.81 (1.11)
|
5.25 (2.49)
|
7.11 (1.81)
|
7.02 (1.73)
|
|
|
|
|
|
|
Table 4. Pre- and post-training ratings for all participants
Item
|
Pre-training M (SD)
|
Post-training M (SD)
|
1. Confidence with wellbeing concerns
|
6.76 (1.72)
|
7.63 (1.29) **
|
|
2. eMH knowledge
|
3.84 (2.00)
|
6.48 (1.89) **
|
|
3. Ipad/tablet competency
|
7.06 (2.32)
|
7.84 (1.72) **
|
|
4. Computer competency
|
7.67 (1.62)
|
8.15 (1.35) **
|
|
5. Confidence in using SS app
|
5.31 (2.38)
|
7.51 (1.61) **
|
|
6. Confidence in using eMH
|
5.58 (2.21)
|
7.14 (1.61) **
|
|
7. eMH referrals competency
|
5.41 (2.34)
|
6.68 (1.80) **
|
|
8. Accessibility to eMH
|
3.86 (1.96)
|
6.03 (1.94) **
|
|
9. Appropriateness of eMH
|
5.93 (2.23)
|
7.13 (1.76) **
|
|
10. Effectiveness of eMH
|
5.82 (2.21)
|
7.14 (1.78) **
|
|
Note. **p < .001.
e-Index scores
Organisations enrolled in the implementation program participated in three e-index completion sessions (averaging 1.5 hour in length). Each session involved a combination of CEOs, senior managers, general staff, and IT consultants from the organisation completing the e-index, facilitated by a member of the research team. Participants from all four case studies were confident in their knowledge and experience to accurately complete the e-Index on behalf of their organisation. This is demonstrated by their ratings on the first e-Index item ‘confidence in index completion’ as 8 and above (Figures 2, 3, 4 and 5).
In general, there was an upward trend on most of the e-index items for all organisations, with some exceptions: Organisation 1 scores for ‘system fit’ and ‘integration through continuous quality improvement’, and Organisation 3 scores for implementation planning’, ‘service provision fit’, and ‘integration through continuous quality improvement’ (see Figure 2 and 4). Possible explanations for the lack of upward trend in some items are that the initial ratings were over-confident, that actions to accommodate e-MH were not achieved as predicted and/or that unforeseen implementation challenges arose. Nonetheless, the graphs depict improved organisational readiness across most aspects relating to eMH implementation for all organisations over the course of the implementation program.
eMH usage data
Follow-up assessment included a question regarding the frequency of use or referral to three culturally responsive eMH resources – the SS app, the Mindspot Indigenous Well-being course, and the Stayin’ on Track app. Due to a number of reasons including staff movement to a different organisation and disengagement from the research project, only 12 participants from the case studies and 3 participants from the non-case studies completed the follow-up assessment. In sum, participants from the case studies reported to have used or referred to eMH resources 67 times, while those in the non-case studies reported 7 times. Taking into account the difference in participant numbers, there is still a marked difference in use between the groups. i.e., on average participants in the case studies used eMH 5.58 times while those in the non-case used eMH 2.3 times.
Additionally, 6 out of 10 follow-up support records from the case studies demonstrated evidence of eMH use, in comparison to that of the non-case studies where only 2 out of 15 records exhibited evidence of eMH use.
Qualitative analyses
These analyses seek to understand how the implementation program enhanced organisational readiness for eMH implementation.
The analysis is presented according to the i-PARIHS framework’s elements and sub-elements, i.e., innovation, recipients, context, and facilitation. Each of the elements and sub-elements include the following discussions:
- Barriers to eMH implementation for case and non-case studies
- A narration of how the facilitation strategies (i.e. e-Index discussion - for case studies, and follow-up support - for both case and non-case studies) helped to address the challenges.
It is important to highlight that due to a limited number of culturally responsive eMH approaches to Indigenous Australian populations, many of the participants reported to use the Stay Strong app which incorporates both Indigenous and Western perspectives of wellbeing [10]. Therefore, many of the responses reported here were related to the use of that app.
Innovation
Definition: knowledge of the innovation (information from research and clinical and patient experience) and its alignment with local priorities and practice.
Underlying knowledge sources
Participants from all four case studies reported to have limited awareness and knowledge of eMH approaches during the first e-Index completion session. Their knowledge, however, increased throughout the implementation program. They reported to have learned more about the approach through resources provided by Menzies and by seeking feedback from service providers’ and their clients’ experience using eMH.
Degree of fit with existing practice and values
During follow-up support, many participants from both case and non-case studies reported difficulties integrating eMH approaches into usual practice. There were two main challenges. One related to the use of the SS app with a group of clients: “working with big groups of clients – difficult to give individual attention”, and with couples: “many staff are working with couples – this is presenting as an issue... A few staff were looking for solutions as the responses will be on one profile and the other person then feels it’s not theirs.” Another difficulty identified was in the one to one setting where clinicians struggled to find time to have wellbeing conversations if their usual practice was focused on practical support.
“Social support framework doesn’t allow the time to sit with people… too busy running around taking people to appointments… When people are focused on getting emergency services – housing, food, bills paid… there is no interest in doing the app.”
Therefore, as part of follow-up support, Menzies staff helped participants to brainstorm ways in which eMH could be integrated into their usual practice. For instance, discussions confirmed that each client did not require a separate iPad - “one iPad per support person rather than participant”, or in a family setting that families could work together - “short term solution to set up a 3rd profile which is a family one”. Additional ideas to create opportunities to use the SS app such as “offer food and sit with someone” or “sitting with people when on dialysis” were also discussed.
Usability
The technical aspect of eMH approaches raised challenges for participants from both case and non-case studies. Participants reported difficulties with setting up the SS app (e.g., downloading the app, setting up passwords, registering emails etc…), glitches within the app, and difficulty in printing care plan summaries. Additionally, having to remember to take iPads to appointments was a challenge for both groups. These issues were discussed, and solutions offered during follow-up support.
Recipients
Definition: Characteristics of the service providers (e.g., motivation, values, beliefs, skills and knowledge), and factors relating to the team’s culture, in supporting or resisting an innovation.
Motivation & goal-setting
The majority of the participants from both case and non-case studies demonstrated enthusiasm for using the SS app and eMH approaches and expressed the desire to use eMH more often in the future. However, records of follow-up support showed that only case study participants set explicit goals regarding when they would use the SS app and the number of clients with whom they would use it. Some case study participants also set plans to practice using the SS app and had discussions with researchers about using other eMH approaches.
Skills & Knowledge
Levels of IT competency and knowledge of the SS app functions differed across participants. Some were more confident and competent using the app than others. Nonetheless, follow-up support provided participants from both case and non-case studies with solutions to technical issues, ‘how to’ explanations and revisions of the app’s functions (e.g., “can go straight to Goal section, do not need to work through all app – menu bar on left hand side”). For some participants from the case studies, follow-up support also provided discussions around counselling skills and how to best use the SS app in wellbeing conversations with clients.
“Important discussion about not letting App drive the conversation, let the practice inform what goes into the app.”
Time, resources and support
Limited time was a major impediment to eMH implementation for participants from both case and non-case studies. In follow-up support, participants reported difficulties finding time in their usual practice to incorporate the SS app.
“All said they have so many resources and other things that they have to include, that it can be hard to add this in as well. None have had an opportunity to use it yet.”
Participants from case studies also had difficulties finding time to discuss internal eMH training plans with their manager.
“Difficulty in finding time to discuss training plans with manager. Unsure about expectations/training needs for team.”
Lack of resources was an issue mainly for the participants in the non-case studies. The case study organisations made efforts throughout the implementation program to increase access and availability of iPads, tablets and WIFI. This was documented within the e-index discussions at which time the organisations made plans to install WIFI in certain areas and purchase additional iPad/tablets.
The implementation program also encouraged the organisations to develop a support system for staff around eMH use. The e-Index prompted the organisations to identify internal facilitators to undergo the TtT workshop, and to incorporate eMH discussions in individual supervision sessions and team meetings. Nonetheless, this was an on-going challenge for some organisations. For example, one organisation initially reported in the first e-Index completion: “support in supervision sessions needs to happen. There has been minimal to no training in eMH”, and while they made progress and reported having allocated time in supervision sessions to support eMH use, they still reported in the last e-Index discussion that staff needed more support. This was confirmed in the follow-up support discussions as participants from this organisation were not confident in using the SS app. They also reported having not received directions from management in terms of how to integrate the app into practice.
Context
Definition: different layers of context from the micro (local setting) through the meso (organisational setting) and macro levels (external health system) that act to enable or constrain implementation.
Culture
All four case studies reported a culture that is supportive of staff learning and development and described having allocated time and resources for on-going training of staff. During e-index completion, one organisation also discussed the idea of providing incentives for eMH use by giving acknowledgement through newsletters and for staff to include one case study in their quarterly reporting.
Leadership and management support
Support from management was identified by some participants in the non-case studies as essential to eMH implementation. This aspect was addressed in the implementation program which adopted a whole of organisation approach. Across all four case studies, the CEO, senior managers, IT consultants and other staff attended e-index completion sessions, demonstrating the organisation’s commitment to eMH implementation. Nonetheless, as discussed above, participants from the case studies reported during follow-up sessions that they felt the need for additional support to carry out internal eMH trainings and to integrate eMH into practice.
“When asked if they had been given direction from management where/how to use the SS app, they hadn’t had direction but did know it was able to be inputted on database.”
Evaluation and feedback process
Most of the case studies reported having an existing evaluation and feedback process prior to eMH implementation. For instance, one organisation described a continuous improvement system while another had client feedback integrated within their service closure process. Facilitated by the e-Index and follow-up support discussions, the organisations continuously adapted their system to incorporate feedback on eMH utilisation. Strategies such as “put tablet/SS app usage as an agenda item for discussion at Team Meetings… and individual supervision sessions” and “have staff present to group on app use/experiences/thoughts on a rotating basis at team meetings”, were brainstormed.
Structure and systems
High-staff turnover was common across case and non-case studies. A few participants who underwent eMH training left their organisation, and one internal trainer from a case study moved to another organisation.
Policy and protocols around IT use and data management were identified as a challenge to eMH adoption. In response to the open-ended question regarding changes needed in the organisation to implement eMH, the majority of the participants from non-case studies highlighted issues concerning policies and protocols:
“Must be able to coincide with our policies”
“Systems around introducing iPads + downloading apps”
The implementation program aimed to address this aspect by using the e-Index to instigate discussions around policies and protocols. This successfully encouraged the case studies to discuss and initiate plans to adapt, or create, policies and protocols for IT use and data management. By the end of the program, all four organisations reported progress in adapting or creating procedures around use of devices, data security and storage of information, and protocols for saving and transferring clients’ data. However, finalising and translating policies into practice was a slow process, as during follow-up support participants still reported: “No consistent procedures around tablet storage, charging”.
“Staff aware that it is part of organisation policies and procedures and there is a movement towards it being used as an initial planning/assessment tool but still not being consistently used by all staff”
Integration of eMH into usual practice was identified as a major challenge. Although the e-Index completion sessions prompted the organisations to develop a concrete plan for integration of eMH within their care-planning pathway, some participants still displayed uncertainty about how to incorporate the SS app in practice during follow-up support discussions.
“Aware that managers are also discussing with Menzies changes to systems, processes etc. Unsure whether integrating app with assessment form will work – many assessments take place over the phone with people in remote communities”
Follow-up support, therefore, provided another avenue to help participants brainstorm ways in which they could use eMH in their practice. For example, in a follow-up support session it was recorded:
“Where does the SS app fit within practice framework was discussed. There was a lot of conversation around this as the staff had different ideas where they would use the tool within their practice… one is making plans for people, another is having the ipads in the drop in centre secured to ‘something’ and participants access the SS app on their own.”
Internal facilitation
As part of the implementation program, the case studies identified an internal facilitation team. This team consisted of senior managers and IT consultants and included service providers who were responsible for conducting internal eMH trainings and providing eMH support and supervision to other staff (internal trainers). The internal trainers underwent the Train the Trainer workshop which was designed to provide them with skills and knowledge to carry out eMH training independently. Follow-up support discussions demonstrated evidence of internal eMH trainings being carried out in the case study organisations and the trainers’ plans for future trainings. During follow-up support, Menzies staff assisted the internal trainers with resources, presentation materials and space to discuss their training plans. Menzies staff also offered to come into the organisation and observe the trainers deliver their planned eMH training.
“[participant] and [participant] stayed and developed a training for staff... Talked about catering and will provide afternoon tea. Developed a 2-3 hour SS App session delivered tomorrow.”