Pre-Test
Overall n = 19 participants (8 Paramedics, 6 Emergency Physicians, 3 EMTs and 2 call handlers) took part in the Pre-Test. Reliability was confirmed with Cronbachs Alpha and an α-value of 0,837.
The average SUS Score was 74,2% and was also confirmed with an α-value of 0,898 for internal consistency, which shows a satisfying internal consistency.
Changes:
As some questions used a past tense adaptations had to be made f.ex in to the present tense. The adaptation can be seen in Annex 1, which also includes an explanation for each of the changes.
Test
Participants and Qualifications
At the time of the study there were n = 308 registered professionals (which also included part-time employees and temporary staff), consisting of n = 238 non-physicians and n = 70 Emergency Physicians in the county. Of these n = 91 (29,5 5%) finished the complete survey and their results were included in the analysis (e.g. Figure 2).
The participants (n = 91) were on average 34,38 ± 10,89 years old 95% KI [33,27; 35,50]. The oldest being 59 and the youngest 19 years old.
Regarding sex n = 66 participants (72,55%) identified as male, while n = 25 (27,5%) as females.
n = 73 (80,2%) participants were qualified as emergency medical staff, which included paramedics (39,56%), EMTs (30,77%) and call handlers (9,89%) for emergency dispatch centres. n = 18 (19,8%) were qualified as emergency physicians (19,8%).
Referral of patients to the appropriate treatment centre
A Mann-Whitney U test was performed to evaluate the item ”The telemedicine system supports the referral of patients to the appropriate treatment centre” between physician and non-physicians. It showed that physicians significantly agreed with this item compared to non-physician medical staff [z = -2,074, p = .038].) This had a low effect size 𝑟 = .217.
A comparison within the group of non-physicians was also performed between EMTs (n = 24) 42,1% and Paramedics (n = 33) 57,9%.
48,5% of Paramedics and 33,3% of EMTs agreed with this item. The result was not significant [z = -1,845, p = .065].
Level of knowledge about telemedicine
n = 43 (47,3%) Participants were assigned to the group “little or no knowledge", n = 26 (28,6%) to the group “moderate level of knowledge” and n = 22 (24,2%) to group “high level of knowledge” group.
The item “The telemedical system leads to an improvement of treatment options” showed a significant difference [χ2 = 6,871, p = .032]. The following Post-hoc test showed a significant difference between the groups “little or no knowledge” and “high level of knowledge”
[z = -2,401, p = .049] as 90,3% of the “high level of knowledge” group agreed with this item compared to only 65,1% of the “little or no knowledge” group. The size of effect size was weak 𝑟 = .295.
Intended use of the telemedical system
Comparing for the intended use, 30,2% of the group “little or no knowledge” replied with a daily to weekly use while only 23,1% from the group with a “moderate level of knowledge” replied like this. From the group with a high level of knowledge 36,4% of participants replied with a daily to weekly use. There was no significant difference [χ2 = 9,521, p = .199] between the groups according to the Kruskal-Wallis-Test.
Comparing age groups
Participants were assigned to 3 age groups: n = 42 (46,2%) participants were in the age group 19–31 years, n = 31 (34,1% ) to the age group 32–45 and n = 18 (19,8%) to the age group 46–59 years.
The item “The tele-emergency physician performs higher-level supervisory and control functions” [χ2 = 12,958, p = .002] provided a significant group difference.
In the Post-hoc tests a significant group difference between the age groups “32–45” and “46–59” was seen [z = -3,356, p = .002] with a medium size of effect 𝑟 = .479.
Also a significant difference was described for the age groups „19–31“ and „46–59“ [z = -3,205, p = .004] with a medium size effect 𝑟 = .413.
The participants from the groups 19–31 and 32–45 years agreed more with this item than the group of the 46–59 year olds.
Frequency of intended use,
The group „19–31“ years replied with 33,3% to a daily to weekly use, while 32,3% of the group „32–45“ years and 16,7% of the group „46–59“ replied with 16,7%. There was no significant group difference [χ2 = 8,428, p = .215].
Request for support
Regarding the items which asked the participants in which area a request for support is likely, the answer “decision on diagnosis and therapy” n = 71 (58,7%) was chosen the most frequently, followed by “organisational support” n = 24 (19,8%), “manual skills” n = 16 (13,2%) and “no support needed” n = 4 (3,3%) and n = 6 (5,0%) for others (i.e. Table 1).
When comparing this in regard of the 3 age groups: n = 34 (81%) participants in the age group 19–31 years and n = 26 (83,9%) in the age group 32–45 requested support on “diagnosis and therapy”. Followed by “manual skills” n = 7 (22,6%) from the age group 32–45 years and n = 4 (22,2%) age group 46–59 years.
Table 1
Request for support per age group
| Age groups | |
Request for support | 19–31 (n = 42) | 32–45 (n = 31) | 46–59 (n = 18) | Overall (n = 91) |
Decision on diagnosis and therapy | n | 34 | 26 | 11 | 71 |
% of group | 81,0 | 83,9 | 61,1 | 77,17 |
Organizational support | n | 9 | 4 | 11 | 24 |
% of group | 21,4 | 12,9 | 61,1 | 26,37 |
Manual skills | n | 5 | 7 | 4 | 16 |
% of group | 11,9 | 22,6 | 22,2 | 17,58 |
No support needed | n | 2 | 1 | 1 | 4 |
% of group | 4,8 | 3,2 | 5,6 | 4,4 |
Regarding the qualification of participants, 75,8% of Paramedics (n = 25) and 95,8% of EMTs (n = 23) requested support on “diagnosis and therapy”.
System Usabilty Scale
With Cronbachs Alpha being .829, overall usability received a score of 68,68 (SD 12,76) 95% KI [67,37; 69,99] (i. e. Table 2).which allows a system to be described as usable [53].
Female and Male Participants
To compare the SUS Score in the female and male group a two sided t-test was performed.
There was no significant difference in SUS Score between the male group (M = 69.24, SD = 12, 95% KI [66.29; 72.18]) and female group (M = 67.20, SD = 14.74, 95% KI [61.12; 73.28]); [t(89) = .67952, p = .499]. (e.g Fig. 3 Label A)
Physicians and not physicians
There was no significant difference between the physician group (M = 68.89, SD = 10.44, 95% KI [63.70; 74.08]) and non-physician group (M = 68.63, SD = 13.33, 95% KI [65.72; 71.94]) in SUS Scores [t(89) = .076628, p = .939]. (e.g Fig. 3 Label B)
Availability of TNA System
There was no significant difference in SUS Scores between the group that had a TNA System available (M = 69.21, SD = 13.82, 95% KI [65.54; 72.88]) and not-available (M = 67.79, SD = 10.9, 95% KI [65.72; 71.59]); [t(89) = .51014, p = .611]. (e.g Fig. 3 Label C)
SUS and Age
A Pearson correlation coefficient was computed to assess a linear relationship between the participants age and SUS score. There was a no correlation between the two variables, [r(89) = − .002, p = .989]. (e.g Fig. 3 Label D)
Results Age Groups
A one-way ANOVA was performed to compare the effect of the age groups on the SUS Score. Homogeneity of variances was confirmed with Levene's Test. It revealed that there was no statistically significant difference in mean SUS Scores between at least two groups [F(1, 89) = .038, p = .845]. (e.g Fig. 4 Label A)
Qualifications,
When comparing the effects of the participants qualifications on the SUS Score, homogeneity of variances was confirmed with Levene's Test. No statistically significant difference in mean SUS Scores between at least two groups [F(5, 85) = [2.028], p = .083] could be seen. (e.g Fig. 4 Label B)
Experience
The effect of the participants experience on the SUS Score was also compared. Homogeneity of variances was confirmed with Levene's Test and it could be seen there was no statistically significant difference in mean SUS Scores between at least two groups [F(5, 85) = 2.029, p = .083]. (e.g Fig. 4 Label C)
Table 2
Usability evaluation with the System Usability Scale within the different groups
| SUS Score |
Categories | n | mean | SD | Min - Max |
Age group | 19–31 years | 42 | 69,46 | 14,7 | 37,5–100 |
32–45 years | 31 | 66,53 | 11,16 | 50–90 |
46–59 years | 18 | 70,56 | 10,38 | 50–92,5 |
Qualification | Call handler | 9 | 68,89 | 11,87 | 57,5–92,5 |
Emergency Physician | 18 | 68,89 | 10,44 | 50–85 |
EMT | 28 | 66,07 | 14,73 | 37,5–92,5 |
Paramedic | 36 | 70,56 | 12,54 | 50–100 |
Experience | < 2 years | 17 | 70,44 | 15,11 | 37,5–87,5 |
2–5 years | 17 | 65,59 | 11,27 | 42,5–87,5 |
6–10 years | 18 | 73,61 | 13,89 | 50–100 |
11–20 years | 24 | 66,25 | 11,37 | 47,5–92,5 |
21–30 years | 11 | 63,86 | 9,96 | 50–77,5 |
> 31 years | 4 | 80 | 7,36 | 70–87,5 |
Acceptability - Usability - Effectiveness
To explore the factorial structure in this sample 23 items (excluding sub-questions) were subjected to an exploratory factor analysis with orthogonal rotation. The Kaiser-Meyer-Olkin measure verified the sampling adequacy for the analysis, KMO = .830. Bartlett’s test of sphericity v2 (210) = 730.183, p < .001, indicating that correlation structure is adequate for factor analyses.
Two items „The TEP is also alerted in situations in which an emergency physician would not normally be called.“ (MSA - Value: .326) and „The TEP assumes higher − level supervisory and control functions.“ (MSA - Value: .350) were excluded beforehand as the MSA Values were < .5.
With the Kaiser’s criterion of eigenvalues greater than 1 and indicated by the scree plot a three-factor solution was yielded as the best fit for the data, accounting for 51.28% of the variance (e.g. Figure 5). The results of this factor analysis are presented in Table 3.
A rotated component matrix analysis was performed and showed that loading of all variables was > .30. Therefore all variables could be assigned to a factor.
Two variables existed for the items ”The TEP system supports the referral of patients to the appropriate treatment centre“ with a loading of .344 for factor 1 und a loading of .336 for factor 3. Also the loading of the item “The TEP system reduces my workload.“ with .372 for factor 1 and .398 for factor 3 existed. As crossloading was seen for these variables and the difference was < .2 these variable were excluded from further analysis.
Therefore the Kaiser-Meyer-Olkin measure was repeated and verified the sampling adequacy for the analysis, KMO = .834. As well as Bartlett’s test of sphericity v2 (171) = 672.248, p < .001, indicating that correlation structure is adequate for factor analyses.
As indicated by the scree plot, other possible factor solutions could be a 4- or 5-factor model (e.g Fig. 5). But such corresponding models don’t seem to exist in the available literature and marginally have a larger Eigenvalue than 1.
Table 3
| Faktor |
Variable | 1 Usability | 2 Effectiveness | 3 Acceptability |
The TEP system provides support in finding a diagnosis | 0,708 | | |
The TEP system is useful for my work | 0,707 | | |
The TEP system leads to an improvement in treatment options | 0,706 | | |
I think that the TEP system improves the quality of patient care | 0,664 | | |
I think that the TEP system increases diagnostic certainty | 0,660 | | |
The TEP acts as a supportive counsellor | 0,521 | | |
The TEP system leads to a delay at the scene of the emergency | | -0,700 | |
The TEP system leads to faster transport capability | 0,361 | 0,693 | |
The TEP system leads to significant time savings | 0,330 | 0,686 | |
I think that the TEP will lead to cost savings in the healthcare system | | 0,674 | |
The TEP system conserves the resources of the physical emergency physician | | 0,597 | |
I think that the TEP system leads to increasing costs for the healthcare system | | -0,567 | |
The TEP system leads to a faster start of treatment | | 0,461 | |
The TEP system enables outpatient care for patients | | 0,471 | |
The TEP system increases my workload | | | -0,832 |
The TEP system increases my documentation effort | | | -0,783 |
The TEP system disrupts the established structure of the emergency medical service | | | -0,705 |
I can imagine continuing to work with a TEP system | 0,326 | 0,308 | 0,510 |
I think the TEP system is sensible | | | 0,478 |
Korrelation and Regression
The following reliability analysis provided that Cronbachs Alpha for the factor „usability“ was .801, for „effectiveness“ .779 and for the factor „acceptably“ .805. All α-values were between .7- .9, which describes consistent subscales [36]
Table 4
Correlations for Accetability, Usability and Effectiveness
| Acceptability | Usability |
Usability | .570* | |
Effectiveness | .439* | .435* |
* p < .001 ** p < .05 | | |
A Pearson correlation coefficient was computed to assess the linear relationship between Acceptability and Effectiveness. Positive correlations between the two variables were seen, r(89) = .439, p < .001. and the size of effect was medium (i.e Table 4).
Between “usability and effectiveness positive correlation existed r(89) = .435, p < .001 with a medium size effect (i. e. Table 4).
Also between acceptably and usability a positive correlation could be analysed r(89) = .570, p < .001. with a strong effect size(i.e. Table 4).
Linear regression Model
Based on the earlier results and on Sauers-Ford et al. concept, the earlier analysed factors “acceptably” was analysed as a predictable variable and “usability" and “effectiveness” as predictor variables.
The regression model was: Acceptably = 0,472 * Usability + 0,163 * Effectiveness – 3,943.
The overall regression was statistically significant (R2 = .355, F(2, 88) = 25.801, p = .001).
It was found that “Usability” significantly predicted “Acceptably” (β = .467, p < .001).
It was found that “Effectiveness” significantly predicted “Acceptably” (β = .236, p = .014).
According to Cohen, the effect was strong: (𝑓2 = = 0,55)