In this section, the results of the study are shown. First, the characteristics of the study participants are presented. Afterwards, the results of the qualitative content analysis with its categories are shown. In the third part, the results of the questionnaire are described.
3.1 Participants
All eight experts contacted responded to our invitation and participated in the study. We therefore achieved a participation rate of 100 %. One study participant took part in the study, but due to time constraints, the questionnaire could not be filled in, with the consequence that the following data are only available for seven participants. The characteristics of the study participants are shown in table 2.
Table 2: Characteristics of study participants
Characteristics
|
Options
|
Participants (n=7)
|
Gender
|
Female
|
4
|
Male
|
3
|
Age group
|
>59
|
1
|
50-59
|
1
|
40-49
|
2
|
30-39
|
3
|
Medical specialization
|
Paediatric Surgery
|
1
|
Neurology
|
1
|
Psychiatry and Neurology
|
1
|
Nephrology
|
1
|
Internal Medicine
|
1
|
Neuropediatric
|
1
|
Paediatric
|
1
|
Years of experience in the field of rare diseases
|
30
|
1
|
24
|
1
|
15
|
1
|
4
|
2
|
3
|
1
|
Prior experience with clinical decision support systems
|
Yes
|
4
|
No
|
3
|
The participants were predominantly female (n=4). The distribution of the age group was between 30-39 and older than 59 years. The study participants have different medical specialisations. It is noticeable that three study participants are working in paediatrics. The experience of the study participants in the field of RDs ranges from 3 to 30 years. In total, four of the study participants stated that they had already worked with any kind of CDSS.
3.1 Main themes by deductive category
The results of the qualitative content analysis are presented in the following sections. Results are reported by deductive categories of the category system. We provide selected quotations for each statement (see additional file 5). The following exemplary quotes are abbreviated by “Q” and numbered in ascending order.
3.1.1 Patient Overview (Category 1)
Information
One study participant stated that all the expected information about the patient were available in the patient’s overview (Q1). Two other study participants noted that more precise findings, such as doctor's letters, laboratory or imaging findings are necessary to determine who diagnosed the RD (Q2-Q4).
In the diagnosis history and symptom history (category 1.1 and 1.2), the study participants stated that the provided information were insufficient (Q5-Q8). One participant said: “So, basically, it is not enough for me. But it depends on whether this is a first, what are the symptoms. Then of course it is sufficient. But in principle I would like to have details. In the case of dyspnoea in particular, this is high dyspnoea, dyspnoea caused by stress, and in case of fever, how high are the temperatures really, is this a chronic increased body temperature or fluctuating symptoms. The same applies to paraesthesia, i.e. which extremities are affected. Paraesthesia can have very different qualities. Depressiveness would suffice and fatigue, which are more descriptive. But this is not enough for me. But to have a first overview, then yes.” (Q6)
On the lines of the symptom and the diagnosis history, the information on the family history (category 1.3) were not considered as sufficient by the study participants. Further information should be provided to illustrate the family relationships (Q9). For example, one respondent said that there was no information on whether a family member had similar diagnoses or symptoms. You would also expect a kind of family tree that shows the links between the family members and their illnesses (Q10-12). One study participant indicated that consanguinity can be a marker for a genetic disorder, but it is not interesting for all diseases (Q13). The participant stated: “In that in the constellation as it exists right now, this is not my first question. Nevertheless, this is interesting, because whenever you have to make a diagnosis, it is not uninteresting to investigate consanguinity. Because paediatrics 80% are genetic. That is super interesting. But in the symptom complexes that were given here, that would not have been my first question.” (Q13)
Usability
The participants agreed that the views of the patient overview, symptom history, diagnosis history and family history (category 1 - 1.3) are presented simply and clearly (Q14-17). One study participant explained: “Otherwise, the mask is clearly arranged.” (Q18)
Functionality
One participant stated that in addition to the information about the symptoms, the age of the patient and the age at which the symptoms occur is important. The participant recommended calculating the age and the age at the onset of symptoms (Q19). One study participant suggested to integrate a function to mark diagnoses as valid or not. This could, e.g. indicate whether a diagnosis was made in accordance with medical guidelines (Q20).
3.1.2 Execution of the similarity analysis (category 2)
Information
No data available for this category.
Usability
Before the similarity analysis could be carried out by the participant, one task of the task sheet required the user to select all MIRACUM locations where data matching should take place. In the course of the study, the study participants noticed that the function for selecting all locations is not immediately visible, as it is located in the lower part of the view (Q21-22). Two study participants stated that the selection of all locations must be placed in the table headline (Q21-22).
Functionality
Two study participants made statements about the speed of the calculation time of the similarity analysis. One participant rated this process as too slow (Q23), while another participant stated: “Well, that didn't take too long, 45 seconds. Maybe even 30.” (Q24).
3.1.3 Results of the similarity analysis (category 3)
Information
When presenting the overview of similar patients (category 3.1), the participants consider the date of birth as not relevant, whereas the age at the time of the diagnosis is much more relevant (Q25-26). The views of the patient timeline (category 3.3), medical history (category 3.4) and criteria for similarity analysis (category 3.5) also contain corresponding statements from participants (Q28-31). In addition, the patient comparison (category 3.2) should differentiate between the age of the patients and the age at the time of the diagnosis.
The information provided on the medical history (category 3.4) were rated as interesting and relevant (Q31-35). One study participant used an example to explain why the medical history is relevant for diagnosis: “History is always very helpful. Especially in rare diseases. Because then only the history of the disease provides the information. If you think it's from my field, neurology, psychiatry, for example. With atypical Parkinson's syndromes. If you start with a cross-sectional approach, then you can hardly tell the difference at the bottom of the scale. But then the progression of certain symptoms, like the increase or decrease, especially the increase. This will basically give you the information about the diagnosis. As I mentioned earlier, this varies greatly from illness to illness.” (Q36)
Apart from the advantages of such a view, one participant emphasised that the information in the medical history must be chosen in such a way that it really helps to answer the question (Q37). One study participant stated that only those parameters should be displayed in the medical history that are specifically rare: “I think something that is a common symptom of an extremely common disease, you don't need it here. Instead, these should be things that are specifically rare and therefore more specific.” (Q38).
When searching for an expert for a diagnosis (category 3.6), two participants discussed which persons should be considered as an expert. They explained that an expert should have published at a high quality level (Q39-40).
Usability
The participants rated the usability of the overview of similar patients (category 3.1) positively (Q41-42). One participant stated: “This is all very clear to me.” (Q43). Additionally, the patient timeline (category 3.3) was also rated as useful (Q43-46). One participant suggested to include more than one patient in the patient timeline view (Q47).
One study participant stated that the comparison of two patients (category 3.2) is helpful (Q48). He proposes to place the demographic data at the beginning of the table (Q49).
Two participants could not find the function to set the criteria for the similarity analysis (category 3.5) (Q50-51). However, they noted that a regular use of the system may make it easier to use (Q52-53). Two study participants proposed to configure the criteria of the similarity analysis before the analysis is performed (Q54-55). One participant stated that the function to search for experts for a diagnosis (category 3.6) is not intuitive: “No, not very intuitive. That what I did earlier. Oh, that was not related to the patient at all. Then it's perfectly fine. Then it is. You only have to know it once and then. Perfect. Okay.” (Q56)
One participant had difficulties to find the scatterplot (category 3.7) (Q57). Another participant also suggested that more information should be displayed when you click on a point in the scatterplot (e.g. information on diagnosis) (Q57-58). One participant said that the benefits of the scatterplot occur when there are a large number of patients. However, he rated the tabular presentation as better, as it allows to view several pieces of information at a glance (Q59).
Functionality
The study participants criticised the lack of transparency of the similarity algorithm (category 3.2) (Q61-62). They explained that it must be comprehensible how the algorithm leads to the results (Q63-Q67). One participant stated: “The displayed parameters. The anonymous patient was 63 years old when she was diagnosed. Similarity between the patients. Exactly here I miss again, which diagnosis have exactly matched?” (Q67)
One participant rated the search for an expert in the CDSS as a good feature (category 3.6). However, he suggested that it should be possible to search for diagnoses that do not only refer to the confirmed diagnosis of a patient. The diagnoses of the diagnosis history should also be included in the search (Q68). One participant noted that it would be helpful to include differential diagnoses in the system and to include them in the search (Q69).
Regarding the scatterplot (category 3.7), one participant suggested to use a spider chart to plot the individual components, such as symptoms, diagnoses and family history (Q70).
3.2 Results of the questionnaire
The first part of the questionnaire (questions 1-10), which is related to the SUS, resulted in a score of “73.21”. Thus, the CDSS achieved the usability rating “good”, according to Bangor et al. [40]. More details are shown in Table 3.
Table 3: Results of the System Usability Scale (SUS)
SUS-Item
|
Question
|
N (valid)
|
Mean
|
SD
|
1
|
I think that I would like to use this system frequently.
|
7
|
3,85
|
0,9897
|
2
|
I find the system unnecessarily complex.
|
7
|
1,71
|
0,4518
|
3
|
I thought the system was easy to use.
|
7
|
3,71
|
0,4518
|
4
|
I think that I need the support of a technical person to be able to use this system.
|
7
|
2,57
|
0,4949
|
5
|
I found the various functions in this system were well integrated.
|
7
|
3,85
|
0,8330
|
6
|
I thought there was too much inconsistency in this system.
|
7
|
2
|
0,5345
|
7
|
I would imagine that most people would learn to use this system very quickly.
|
7
|
4,57
|
0,4949
|
8
|
I found the system very cumbersome to use.
|
7
|
2
|
0
|
9
|
I felt very confident using the system
|
7
|
3,71
|
1,1606
|
10
|
I needed to learn a lot of things before I could get going with this system.
|
7
|
2,14
|
0,8330
|
Overall SUS score
|
73.21
|
n = 7 participants, mean rating (5-point scale strongly disagree = 1; disagree = 2; neutral = 3; agree = 4; strongly agree = 5), standard deviations (SD) and overall SUS score
The second part of the questionnaire (questions 11-21) with specific items to individual functionalities resulted in a rating from 3.42 to 4.28. Thus this correspond to the characteristic value from “neutral” to “agree”, according to the 5-level Likert scale. More detailed results of the questionnaire are shown additional file 6.