Successes
<BLINDED> scored 9 out of 12 for fidelity to the design thinking process with a notable need for improvement around the ideation stage (Table 5).
Table 5: <BLINDED> HCD Evaluation Score
Empathise and Define
|
Comments
|
Points
|
Moderately Attempted: (2 points)
End users were engaged, felt respected, and were compensated for their participation in co-design sessions. (Compensation may be financially via cash or gift cards, transportation costs to get to sessions, provision of childcare during sessions, or other means.)
Empathy exercises were undertaken to understand lived experience of end users.
|
- · The organisational documents provided evidence of compensation, end user interviews and ethnographic work, CBPR and empathy exercises but there was no documentation regarding whether co-designers felt respected or recognised for their contribution. Instead, this information came from the surveys.
- · All co-designers selected ‘agree’ or ‘strongly agree’ to feeling respected and if they would encourage others to participate in a co-design session with the <BLINDED> team.
- The representativeness of the end user co-designers was also not apparent in any of the organisational documents but emerged as a theme during the interviews.
- · Co-designers’ views on how representative the end users were varied by the type of co-designer group the participant came from.
- · SMEs and UX/programmers generally agreed that there was enough end user representation at the co-design sessions. As one computer programmer said: “We focused bringing on more and more people from the refugee and migrant community which is really good … I don’t think we had a shortage of that diversity … in terms of cultural background it was quite well represented.”
- The end users themselves felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time.
- A more representative end user population would have accrued a higher score.
|
2
|
Ideate and Design
|
|
Points
|
Minimally Attempted: (1 point)
Learnings from empathy exercises were compiled.
Pre-determined solution was minimally modified in response to what was learned from end users.
|
- An extensive document review verified that the <BLINDED> team did not have a pre-conceived idea of what the prototype would be, and that the app’s features and structures arose in response to insights gained from the end user co-designers.
- The ideation stage took place within the design student teams and was tested at intervals with end users.
- Surveys and interviews with the co-designers revealed that one quarter of all survey participants marked ‘neither agree nor disagree’ to whether <BLINDED> confirmed group consensus for either the problem statement or the solutions to be prototyped. The remaining three quarters indicated they ‘agree’ or ‘strongly agree’ that <BLINDED> did in fact, achieve this.
- Whilst the issue of consensus did not emerge as a theme across all interviews, one computer programmer articulated the problem this way: “I feel like sometimes there was a bit of disconnect between what [<BLINDED> staff] wanted and maybe necessarily what the refugees wanted … they wanted a resource where they could find health information, locations specifically of hospitals, GPs, pharmacies … the actual health information, they would rather go directly to the source, rather than … reading it online.”
- Given the conflicting information received from the surveys and the interviews and the lack of documentation, the evaluator reported finding difficulty scoring <BLINDED> in this area.
- A lower score of 1 out of 3 was given to draw attention to this issue in the future.
- More thorough record keeping during this design stage will shed light on this process and the methods used in future HCD projects.
|
1
|
Prototype
|
|
Points
|
Satisfied: (3 points)
Multiple iterations of prototypes and/or MVPs created and tested with end user population.
End user feedback incorporated into subsequent iterations of solution.
Solution validated with subject matter experts and/or existing literature.
|
- End users from different backgrounds tested and contributed to iterations of the app on multiple occasions and continue to do so to this day.
- SME co-designers and their feedback lead to tangible changes in the app appearance, language accessibility and functionality.
- Organisational documents alone verified these requirements
- <BLINDED> received 3 out of 3 in this stage.
|
3
|
Launch and Share
|
|
Points
|
Satisfied: (3 points)
Product launched.
User testing completed to understand users’ experience and satisfaction.
User feedback incorporated into plans for future iterations.
Process or product results shared with program staff, co-designers, and wider community.
|
- <BLINDED> held a product release in August 2017 which included co-designers, partners, and funders and presented preliminary findings from the CBPR projects.
- <BLINDED> received 3 out of 3 in this stage.
|
3
|
Total Score (max score of 12):
|
|
9
|
Feedback obtained through both the interviews and surveys verified that the <BLINDED> team did complete all the steps of the design thinking approach. All survey respondents selected ‘agree’ or ‘strongly agree’ to statements assessing the collaborative nature of the group work and that they felt safe sharing their opinions (Table 3). Over 90% reported that they would participate in another <BLINDED> co-design session and that they would recommend participation to a friend or family member. In the interviews, a feeling of enjoyment from participating in the co-design sessions clearly emerged. All 13 evaluation participants reported feeling valued, appreciated, and/or respected during the co-design sessions.
When we went to … the students, they were very enthusiastic. They take our notes and they try to discuss with us … And as users … they take our notes and they try to improve. And we share, really. We share as a big group, as a teamwork. And we share our ideas together.
The surveys and interviews revealed important learnings for the <BLINDED> team when using HCD within this particular context.
Communication
First was the issue of communication. Three of the four end users mentioned that language was a barrier, despite assistance from other community members who acted as interpreters during the co-design sessions and their own self-assessment as being proficient in English. This likely exacerbated the second area for improvement: role clarification. This was a need reported by co-designers from all groups. An end user expressed how she came to understand her role in the co-design session:
After some time, I could realise what’s going on, and understand what I had to do. It wasn’t clear in the beginning. Like when I went there, I didn’t know why I’m going there. I just know that I want to be part of this, this is what I really wanted to do, and yeah, after some time I could understand what’s going on, but nobody explained me how.
Fragmentation of involvement
There was fragmentation of co-designers’ experiences during the sessions. Many spoke about not understanding the project’s entire process, wanting to be more involved but not receiving further invitations, or the need to build on previous sessions with end users. While end users were involved at every step of the co-design, different individuals participated at different points and in different ways. Very few end user co-designers were a part of the process from beginning to end. This led to feelings of disconnection and confusion, as one end user put it:
First, I was really interested, but after some time when I found like nobody’s calling you back, so I said no I don’t want to waste my time on this. But it’s something really helpful and I really like the idea of helping new arrivals from refugee and migrant backgrounds.
SMEs who facilitated meetings between <BLINDED> staff and end users also felt the desire to be more involved throughout the entire HCD project. One SME commented:
I think it would be good to have a follow up consultation on working on actual usage … because we haven’t sort of touched base again with those women to say, have you used it? … it was almost still in the design stage, and things hadn’t quite been finished. So, I think we could have a follow up that says, this is the latest version of the product, let’s have a play around with it, what do you think now? I think that would be really timely.
A third area for consideration in future co-design session is the importance of diversity and representation within end-user groups. The SMEs and UX/programmer generally agreed that there was sufficient end user representation at the co-design sessions:
We focused bringing on more and more people from the refugee and migrant community which is really good…I don’t think we had a shortage of that diversity … in terms of cultural background it was quite well represented.
The end users themselves, however, felt that there were groups within the Arabic-speaking population that were unrepresented. Some of the suggested groups include individuals who did not attend university, Arabic speakers with no or low English proficiency, middle and late middle age individuals, people with different levels of proficiency with mobile technology, and refugees who had just arrived to Australia compared to refugees who have been living in Australia for some time. The risks of only collaborating with end users who are university-educated, recently resettled and English-proficient were summarised by one end user:
Maybe we will use this program [<BLINDED>] and maybe never we will use this program because we can search … what we need by Google … but that program [<BLINDED>], it’s good for different level of the people.
Increased representation from within the end-user group should be considered for future HCD endeavours.
Finally, a survey question assessed whether co-designers felt that there was enough time for relationship building during the co-design sessions. End users responded most negatively to the statement, with half stating that they ‘strongly disagree’ or ‘disagree’ that there was enough time for this.