Design
Using Q-sorting allows for the collection of both qualitative and quantitative data, and is helpful in providing a rich picture of participants’ understanding and opinions on a topic (13). To answer research question 1, Q-sorting provides insight into clusters of participants who sorted items similarly, so each of their viewpoints can be represented in the selection of scale items. To answer research question 2, the think-aloud data and the short interviews after the sorting task give insight into the associations and interpretations participants had when reviewing the items. To answer research question 3 and compose the final scale, we integrated the selected items with participants’ qualitative input from the think-aloud data and interviews.
Participants and setting
This study was conducted with 16 mental health professionals, recruited from three different suborganizations of one overarching Dutch mental healthcare organization (see Table 1 for demographic information of the participants). The three foundations involved in this study offer different types of mental healthcare: the first one provides primary mental healthcare, the second one provides secondary mental healthcare, and the third provides primary and secondary mental healthcare fully online. In primary mental healthcare, mild to moderate mental health problems are treated, while in secondary mental healthcare more complex and serious psychiatric disorders are treated. Involving professionals from these different types of mental healthcare gave us a more representative sample of the Dutch mental healthcare system. We recruited professionals that were employed at the organization at the time of the study, and who were fluent in Dutch. Among the 16 participants, one did not complete the quantitative part, citing the forced-choice rank-ordering's near-normal distribution as too restrictive for expressing their opinions. Therefore, we only analyzed the qualitative data for this participant and not their (incomplete) quantitative data. For another participant, the recording of the Q-sort meeting failed for technical reasons. For this participant, we only analyzed their quantitative data. Therefore, for both the quantitative and the qualitative data we ended up with 15 participants. Table 1 shows the demographic information of the participants of whom we analyzed the quantitative and qualitative data.
Table 1. Demographic information of the participants (n=15).
Characteristic
|
Quantitative data
n (%)
|
Qualitative data
n (%)
|
Age (years)
|
20-30
|
2 (13)
|
31-40
|
7 (47)
|
41-50
|
3 (20)
|
51-60
|
3 (20)
|
Gender
|
Female
|
14 (93)
|
13 (87)
|
Male
|
1 (7)
|
2 (13)
|
Profession
|
Psychologist
|
6 (40)
|
6 (40)
|
Mental health psychologist
|
2 (13)
|
2 (13)
|
Psychiatrist
|
2 (13)
|
2 (13)
|
Psychiatric nurse
|
1 (7)
|
1 (7)
|
Social psychiatric nurse
|
1 (7)
|
1 (7)
|
Mental health nurse specialist
|
2 (13)
|
1 (7)
|
Clinical psychologist and psychotherapist
|
1 (7)
|
2 (13)
|
Type of mental healthcare*
|
Primary
|
5 (31)
|
5 (31)
|
Secondary
|
4 (25)
|
3 (19)
|
Both primary and secondary, fully online
|
7 (44)
|
8 (50)
|
Type of technology use in treatment
|
Mainly face-to-face
|
2 (13)
|
1 (7)
|
Mainly digital
|
8 (53)
|
9 (60)
|
Combination of face-to-face and digital
|
5 (33)
|
5 (33)
|
* Please note: one participant was employed in two different mental healthcare foundations offering different types of care.
Materials
Item Construction
To develop the potential items for the scale, preparatory work was done drawing from the conceptualization and item development phases of scale development described by Alber et al. (37). We started with desk research on compassion, existing compassion scales, and technology evaluations. Next, we selected a suitable definition of compassion (1) to apply to technology. As preliminary work, a focus group with 6 mental health professionals was held to explore their opinions about DMHIs, blended treatment, and compassion. The focus group showed that compassion was indeed an important value to mental health professionals, and that they preferred offering blended treatment over providing either physical or digital treatment alone. Subsequently, a systematic scoping review was conducted to provide an overview of links between compassion and DMHIs in current scientific literature (12). Based on the previous activities, a first set of 35 items to evaluate DMHI use on compassion was generated and a feedback session was held with 6 experts from the research fields of compassion, eHealth development and scale development. Based on the feedback session, the items were refined.
The five elements of compassion as proposed by Strauss et al. (2016) formed the basis for our set of statements, or the ‘Q-sample’. These five elements were all represented equally, with 6 items for each compassion element. Additionally, 5 overarching items were added for a short form of the scale. These items did not refer to elements of compassion, but to compassion itself, for a more general assessment. Based on future psychometric studies, one most representative item for each compassion element could be selected for an alternative short form of the scale. All items referred to a ‘platform’ used between a client and professional in mental healthcare. By using ‘platform’ to describe forms of digital mental health interventions (DMHIs), we aimed to use a simple and succinct word that could refer to different types of DMHIs. The items were presented in Dutch, as our participants were Dutch-speaking.
The ideal number of items used in a Q-sort study depends on the complexity of the subject (38). All possible aspects of the research topic must be represented. Typically Q-samples consist of 20-50 statements (35). In our study, we used 35 statements, to maintain a balance between doing justice to the comprehensive construct of compassion and not making the sorting task too time-consuming for our participants.
Q-sort materials
The Q-sort task consisted of two parts (see Figure 1). For the first part, a basic distribution with three different columns was used. In the second part, a near-normal distribution was presented, with two items in each of the extremes, gradually increasing to seven items in the middle pile.
The Q-sort study was conducted online using Microsoft Teams and the free website ‘Qsortware’ (39), which allows participants to be invited via email to complete the Q-sort in their browser. The shiny application ‘QMethod’ (40) was used as a resource to analyze the Q-sort data, as well as Atlas.ti (version 24.0.0.29576) for Windows for qualitative analysis.
Procedure
The Q-sorts took place in online one-to-one sessions that lasted an average of 49 minutes (std. dev. 9 minutes, min. 31 minutes, max. 1h and 5 minutes). The sessions were recorded (screen and audio). All collected data were anonymized so that they could not be traced to the participants.
At the start of each session, the researcher provided background information on blended treatment (combining eMental health and face-to-face treatment (22)), as well as compassion and its elements, the aim of the current study and of the to be developed scale. Next, participants were invited to the Qsortware website and asked to share their screen via Microsoft Teams. Then, the first part of the sorting task was explained to participants, emphasizing that there were no right or wrong answers, and that our interest lay in the personal opinion of the participant. Therefore, participants were asked to think out loud during the whole sorting task. If a participant had been quiet for a while during sorting, they were asked to share what they were considering and how they made their sorting choices.
In the first part, participants sorted the Q-sample (35 items) into three different columns (see Figure 1, left) based on the following question: “how important are these items for evaluating whether technology supports compassion in treatment, according to you?”. The three columns had the following headings: ‘least important to measure’, ‘neutral’, and ‘most important to measure’. Above these three columns, the items were presented one after another, so that participants could drag them to one of the three columns and move them around as desired. The items were shuffled across compassion elements, but the order of the items was not randomized across participants, as the online program that we used did not allow for this. After all the items were sorted in the three columns and participants were satisfied with their distribution, they could continue to the second part of the sorting task.
In the second part, participants were shown a near-normal distribution (see Figure 1, right). Above the empty Q-sort distribution, the three columns from the first part were shown, containing the previously sorted items. Now, participants were again asked to distribute their previously sorted items into the near-normal distribution of the empty Q-sort. Again, participants were free to keep moving the items around until all positions were filled and they were fully satisfied with the resulting distribution.
When participants were finished with sorting the items into the distribution, the Qsortware website was closed and a short interview followed. In this interview, we asked participants what they thought of the task, what they thought of the items, how they made decisions during sorting, and whether any items were superfluous or missing according to them.
Data analysis
Quantitative Data
The Q-method is an inverted technique of factor analysis (35): the resulting data matrix has respondents as variables and items as cases. Thus, respondents are correlated instead of items, and factors represent clusters of participants with similar Q-sorts. The aim of the statistical analysis of the Q-sorts determine which distinct clusters represent shared viewpoints on the topic.
The quantitative data from the Q-sorts were analyzed with by-person factor analysis. First, a correlation matrix of all Q-sorts was produced, indicating the degree of correspondence between respondents. Then, we used Principal Component Analysis (PCA) with varimax rotation and automatic flagging of significant sorts (loadings) on each extracted factor. The analysis also identified confounding respondents (loading significantly on >1 factor) and non-significant respondents (not loading significantly on any factor). To find a factor model with a number of clusters that fit our data well, we examined whether all eigenvalues were greater than 1 (in line with (41)). Furthermore, we checked whether at least two Q-sorts loaded significantly on each factor and not on any other factor (in line with (42)), meaning that at least two Q-sorts had correlations of at least 0.5 with ‘their’ factor, and cross-loadings with other factors below 0.40 (in line with (43)). Next, we checked whether the clusters were interpretable in a substantive manner by checking whether we could find meaningful differences in viewpoints between the clusters. Based on the identified clusters, we selected the items that each cluster prioritized on average as most important.
Qualitative Data
The similarities and differences in prioritized items between the clusters were examined and interpreted to make a meaningful choice about the number of factors that fit our data, as well as to describe these clusters.
The full Q-sort sessions were transcribed verbatim, consisting of thinking out loud data and the short interviews after the sorting task. Thematic analysis was used to analyze this data following the guidelines of Braun and Clarke (44). Thus, relevant features of the data were systematically coded by one coder (CvL), generating initial codes. These codes were merged into potential themes. Separately from this bottom-up approach, the five elements of compassion were coded in a top-down manner, forming the theme “compassion”. All themes were then reviewed and a thematic map of the analysis was created. Next, the bottom-up themes were further defined and named. Using this coding scheme, a second coder (JA) also coded 13% of the data, with a percentage agreement of 75% with the first coder (CvL), which was deemed satisfactory. Differences in coding were discussed until consensus was reached, and two new codes were added (“Less active client with technology use” and “Integration of technology in workflow”). The first coder (CvL) then checked all the data again with the two added codes. Finally, the report was produced with a selection of illustrative quotes to help answer our research questions.