In the following, the methodology applied to test “VIDEA bewegt” for usability is described. This test forms the basis of a study to assess the effectiveness of the app "VIDEA bewegt" to increase physical activity, self-efficacy, and quality of life. This evaluation was registered with the German Clinical Trials Register (DRKS): Evaluation of an app-based activity intervention for statutory health insured people. DRKS-ID: DRKS00017392. (14 June 2019)
In order to evaluate usability, the think-aloud method was applied, as it is a common procedure within user-centred design processes . In think-aloud tests, participants are asked to verbalise their thoughts and impressions during a certain activity. In an app-based usability test, participants are asked to describe their experience when performing certain tasks within the app, with a special emphasis on what they notice, what bothers them, and causes problems, or what they enjoy while using the app. This procedure provides a deep insight into weaknesses and strengths of a system .
At the end of December 2018, a qualitative preliminary analysis with four participants was carried out in order to gain experience with the think-aloud method and judge the applicability of the concept. From January to March 2019, the plan for the specific think-aloud tests was further revised in cooperation with a communication scientist and the "VIDEA bewegt" team.
The app “VIDEA bewegt” (see screenshots 1 and 2), the intervention this paper focusses on, aims to sustainably increase physical activity of its users. Theoretical and practical videos provide the basis of the eight-step programme. Additional features such as documentation of physical activities and synchronising of step numbers, forum, and chat are included.
It is one of the first exercise apps to be officially covered by health insurances in Germany as a preventative intervention.
Screenshot 1 – Start screen
After launching the app, participants are presented a first overview of the app’s goals on five slides. By pressing the button "let's start" (“Los geht’s”) the registration process begins.
Screenshot 2 – Home
After registration, the welcome page of the app opens. At the top you can see a progress bar. The different stages are presented in the centre and the menu bar at the bottom contains the items: Stages, Focus, Activity, Video+, Exchange.
Selection of participants and study setting
As 75% of all usability problems can be identified with only four test interviews , we chose a sample size of ten test participants to discover as many usability problems as possible.
When selecting participants for the usability test, the aim was to cover a wide range of ages, while focusing on people aged 40 and older (see Table 1: Characteristics of participants). This is based on the fact that older people have greater difficulty using digital media, which makes usability particularly important . Plus, the risk for lifestyle-related health complications such as diabetes type II or cardiovascular diseases increases with age [32, 33]. Only participants aged older than 18 were selected. The interviews were conducted at home or at the workplace in a private, quiet, and neutral environment.
Test-participants were asked to work their way through the structure of the app based on several pre-defined goals in 20-30 minute-interviews, describing their thoughts, impressions and problems orally. They were using the app for the first time and were asked to imagine that they had discovered it in real life. A test phone was given to the participants for the test.
The participants received an information and declaration of consent from the interviewer which they agreed to orally for data protection reasons. They were also informed that they could terminate the think-aloud test at any point in time. The interviews were conducted by a total of three different team members. Of these, two were medical students and one was a psychologist.
In order to ensure a standardised and comparable procedure, a guideline was developed for the interviewers [see Additional file 1]. In this guideline, the introduction to the study, the description of the app "VIDEA bewegt", the test procedure, questions about previous experiences with health apps as well as the expectations to the product to be tested were formulated for all think-aloud sessions.
The test focused on six goals, on the basis of which the test participants got to know key components of the app. Instructions for the interviewers were attached to each goal in the guideline. Suggestions for questions prompting feedback, if none was given spontaneously, were included as well (e.g.: "What are you seeing?", "What do you notice?", "What do you want to do next?", “What problems have occurred?”). Such, test participants were motivated to continuously formulate their thoughts and utter them aloud.
The six tasks tested were:
- Registration in the app
- Watching three minutes of theory video
- Watching three minutes of exercise video
- Manual entry of 5000 steps
- Sending a test message in the chat
- Starting an additional video
Following the standardised test of the app and the related think-aloud, general questions were asked: whether participants could imagine using the programme in real-life, whether their expectations had been met and whether they missed certain app-features or felt existing features to be unnecessary. In addition, age, educational level and occupation of the participants were assessed.
In April 2019, all 10 test interviews were carried out within two weeks. The transcription and analysis of the transcripts took place in April and May 2019.
Consent to participate was given orally and no names were used during the interview. An anonymous transcription was carried out, which did not allow identification of the participants. Only members of the research team had access to the data stored on the servers of the partaking institutions. No personal data was collected at any time during the test.
The interview was recorded directly on the test smartphone. Both an audio track and the screen of the test smartphone were captured in order to be able to visually monitor any problems arising during analysis. The complete interviews were transcribed using a web software (otranscribe) [see Additional file 2]. Subsequently, the transcripts were evaluated using qualitative content analysis according to Mayring . Based on the research question, three broad categories were used to structure the deductive content analysis.
- ideas for improvement
- strengths of technology, content and design
- weaknesses of technology, content and design
Information on the test subjects also was a necessary category for structuring the content analysis (MAXQDA was also used for this analysis). The inductive formation of sub-categories, further describing each of the major categories above, was intended.
Each transcript was analysed independently by two team members. This process was supported by two experienced researchers of the Technical University of Dresden. Subsequently, the independent analyses were merged in one code system and discussed further. A codebook was created, for which the most important findings on strengths and weaknesses of the app were summarised and illustrated with examples.