Measures at the participant level: Reach, barriers and facilitators
The target group for the trial, as defined in the governmental commission of the study, were people in treatment for moderate to severe mental illness in secondary care. The diagnostic screening of most participants at inclusion showed that 51% of the participants suffered from severe mental illness (psychosis or bipolar disorder), and 49% fulfilled criteria for moderate mental illness (primarily affective disorders). This indicates that the study population corresponds to the pre-defined target group.
An overview of study population characteristics is provided in table 2.
Table 2. Characteristics of the intervention group at baseline.
-------------------------INSERT TABLE 2-------------------------
The study population was relatively young (x = 35, SD 10.7) and education level was low. Nearly half of the participants had experienced violence, and one third had been involuntarily committed to a psychiatric hospital. Mean of previous years worked in main occupation was 7 (SD 7). The mean rating of health-related quality of life, measured by the EQ-5D visual analogue scale, was 58 (SD 18.3).
Barriers and facilitators to participation
Table 3 shows results from the statements inquiring about barriers and facilitators to participation in the intervention. Open-ended response categories were provided, but few respondents used these, and they did not generate any additional barriers or facilitators.
Table 3. Percentage of respondents agreeing to the statements about facilitators and barriers.
--------------------------------------INSERT TABLE 3 HERE------------------------------------
Two of the most frequently cited facilitators among participants regarded the employment specialist’s role: 94% of respondents agreed with the statement ‘Knowing that the employment specialist was available for me was helpful’, while 81% agreed with the statement ‘The regular follow-up from the employment specialist was helpful.’
Responding to a separate item regarding the employment specialist, 78% reported to be satisfied, while 13% reported to be dissatisfied. 9% reported to be neither dissatisfied nor satisfied (n=78; x = 4 SD 1.11).
All over, participants were very happy with the role of the employment specialist. Participant interviews gave further insight into the employment specialist’s role, emphasizing their availability, support, and consistent job focus. When talking about availability, informants emphasized that the employment specialist was quick to respond and to express their availability:
“She is really good, really efficient. Supportive, calls and asks me to call back, to call on her spare time. If it’s a good time it’s a good time, if it’s not a good time she calls me back up again. It’s been really nice. “
“I think he’s been really available, because even if he doesn’t answer my call immediately, he calls me back up, he is always there for me if anything comes up. So yes…I feel I have received very, very good follow-up from him. So I am very happy.”
There were a few exceptions to the positive descriptions of the employment specialist’s role. Two of the informants who were less satisfied with the intervention, described the interaction with the employment specialist as challenging, and one of these got a new employment specialist as the first one quit.
Although the interview didn’t contain questions about empowerment, it emerged as a topic when discussing the role of the employment specialist. Some participants said that the employment specialist ‘pushed’ them to keep going with the job search, and some had been confronted with their lack of motivation. This resulted in taking a more active role in the job search:
“It wasn’t a threat, but they said we can’t help you if you’re not interested. They deserved an honest answer to that. It was the best question I could have received, instead of them saying ‘We’re not wasting time on this...’ I woke up.”
“It has been positive for me to start working, yes. But I do feel there is a small pressure and that I have to push myself to say yes to working. I am supposed to start working and not sit at home. And I did get a job, so maybe it’s good that they push a little.”
One participant described how being listened to in the process was important:
“If he has come with a job suggestion and I have said that this is not for me, because I will not function well there, he has just put it away immediately, he is very accommodating like that.”
Another facilitator for participation indicated by the list of barriers and facilitators was the freedom to disclose or not: 93% of respondents agreed that ‘Being able to choose whether or not to be open about my illness’ was helpful. The interviews indicated that for some participants, choosing to disclose expanded the possibilities for practical help in the job search:
“Yes, it’s very nice because the employment specialist can call around for me, I have anxiety about talking on the phone sometimes. And she is with me in the conversations with employers so that I understand what is being said, and she can also inquire about salary.”
One participant reflected on the positive aspects of disclosing to potential employers:
“It might be that some employers think that they want to support it because it is kind of a good cause to help people get a job that maybe have a history of illness or have had problems, and because of that they can get a job.”
Most of the proposed barriers were not supported by participants. However, the most agreed-upon statement was ‘My illness was a barrier’ (46%, n=95). Considering the target group of the intervention, this number is not particularly high. One’s illness was generally not a specific theme in the interviews, at least not as a barrier for participation in IPS. One in six participants agreed to the statement ‘IPS was not what I expected’ (17%, n=77). Interview data provided further insight into the role of expectations. While some participants described being positively surprised by the intervention due to low expectations, others described having high expectations, and then being disappointed as the intervention progressed.
“I was promised employment within 6 weeks, and now I have waited for 13-14 months (…). I had expectations about follow-up from employment specialist and close cooperation between my doctor, the District Psychiatric Center, and a permanent position with full salary. And none of it has happened.”
One informant stated that the follow-up was simply different from what he had expected regarding his own involvement:
“Uuuhm, but the only expectation I did have that turned out not to be correct was that I kind of thought they had some sort of obligation to help me find a job so I didn’t have to do such an effort myself. But that was totally wrong. (…) It’s not like I can nag them and say ‘Hey, find me a job’, it’s more like they come alongside and back me up on the things I manage to do.”
Two items measuring satisfaction and perceived usefulness were included as another measure of barriers and facilitators from the participant perspective, as low scores on these measures could indicate poor quality in intervention delivery, and/or low engagement with the intervention among participants (table 4). However, participants were overall satisfied with the intervention (n=95; x=3.95 SD 0.97), and also found it useful (n=96; x=3.96 SD 1.06).
--------------------------------------INSERT TABLE 4 HERE------------------------------------
The participant interviews provided further insight into the positive responses on satisfaction and usefulness. For example, informants reported that the IPS follow-up had made them aware of their own competence and their own preferences. Most informants reported that IPS had increased the frequency of sending applications, and that they had learned more about the job interview setting. The informants emphasized that the focus on employment had been central in the follow-up:
“I haven’t got a job offer, but now I apply for jobs in a different way. I have been on many interviews, so that has improved as a result of this follow-up (…). I learned how to write an application, about motivation, qualities…”
The first thing we did was to go over what kind of jobs I wanted, then we got my CV sorted out, how to write an application, and have everything ready for sending the application. (…) And then we went out into the job market. We went step by step, one thing after another, in the right order.
Doing job-related activities in itself seemed to increase the motivation to obtain work:
“I do become happier and more positive in my everyday life when I have sent applications, and called about vacancies and stuff.”
“I enjoy working. It’s really nice that they have helped me to get started. Just having an appointment with them one day, having something to show up to, not sit there and do nothing, that helped me get started.”
Service provider perspective: Barriers and facilitators
Implementation issues at the service provider level were examined by fidelity reports and interviews with IPS teams. Results from the fidelity evaluations are presented in table 5. These evaluations were carried out approximately one year into the project period. All six centers had reached fair or good fidelity at this point, with a range of 89-109 points (median 99.5), where 125 was the highest possible score, and 75 the cut-off score for being an IPS intervention.
Table 5. Center scores on each item, mean scores of centers, mean of lowest and highest performing centres, total fidelity score and median for each centre.
--------------------------------------INSERT TABLE 5 HERE------------------------------------
Some trends are worth pointing out. First, a look at particularly low fidelity scores across all centers more than one year into the project period may indicate implementation barriers. Items with low scores (range 1-5) were ‘Community-based services’ (`x = 1 ) and ‘Job development – frequency’ (`x = 2). To receive a top score on ‘Community-based services’, the employment specialists must spend 65% or more of their time outside the office. This should be seen in relation to the item ‘Job development – frequency’, indicating frequency of contact with employers. The employment specialists reported in the interviews that it had taken quite some time to develop and understand their role, and that prioritizing tasks was demanding, as expressed in the following interview remarks:
“What we always have to challenge ourselves on is the use of time, considering time spent on internal meetings, participant meetings and employer contact. To obtain the optimal allocation of resources is pretty challenging.”
“Being an employment specialist is a pretty complex and difficult role, where you are a seller on the one hand, selling the best manpower there is, next you’re a facilitator, an employment specialist, and you can sometimes have a therapeutic approach at times when the therapist is not there. So, it is a very difficult role, and it takes time to be secure in it.”
Looking at the two items measuring “integration with health services”, which is an important IPS principle, their mean score across centers is 3. Although this is above the middle of the scale ranging from 1 to 5, it is worth pointing out that according to the quality thresholds defined by the program developers, these two items barely pass the “Fair fidelity” threshold .
On the positive side, all six centers received top scores on the items “Caseload size” and “Employer diversity”, and nearly top score on “Disclosure of disability to employers”. “Caseload size” means that the caseload for each employment specialist does not exceed 20 participants, in order to ensure close follow-up in all phases of the job search. “Employer diversity” refers to the diversity of workplaces where participants get jobs. It is used as an indicator of whether employment specialists are following participants’ own preferences, and not only working within the limits of their existing employer network. The item on disclosure measures the degree to which employment specialists provide information to participants about pros and cons of disclosing about their illness to an employer.
When looking at the differences on employment outcomes between the centers, three of the six centers performed particularly well, according to the evaluation report of the trial , though there were no apparent reasons for this difference. Overall fidelity scores varied somewhat between the centers, but the top three centers differed from the rest on two particular fidelity items. The high performing centers had an average of 1.2 points above the average of the less performing centers on the item ‘Integration of IPS with treatment team’, which is considered a crucial intervention component . The challenges related to this topic was addressed frequently in the interviews, and are illustrated by the following remark:
“I think that it is the greatest success and the greatest challenge, that integration process (…), how we feel that they [treatment team] talk and feel concerning work.”
However, the most striking difference was found between scores on the item ‘Job development – quality’ (indicating quality of employer contact), where the top performing centers had an average score of 2.7 points higher than the average of the less performing centers. The issue of employer contact was addressed in the interviews, as exemplified in the following remarks:
“And then you meet those…that even if you talk about employment and paid employment and ordinary employment, and they nod their head, you find out at the end of the conversation that they still expect free labor. We do a lot of work to change attitudes.”
“…the job development part, that’s something that for most of us, and definitely
for me, has been new and different, going out and being assertive both on the phone and in person (…) After a while I realized it’s been written in the manual the whole time, that we really need to have our main focus on job development. We’ve made some changes now this fall where we have set targets and try to reserve days and times to do that.”
“We’re selling in the close follow-up (…). But it is hard to compete [with other
work rehabilitation programs], that’s a barrier. Some employers have a clear
policy that they only use work practice, there is no other way in.”