Initial usability evaluation training for clinical engineering students

doi:10.21203/rs.3.rs-3117823/v1

Download PDF

Research Article

Initial usability evaluation training for clinical engineering students

https://doi.org/10.21203/rs.3.rs-3117823/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

We conducted usability evaluation training among clinical engineering students to determine user dissatisfaction with equipment that was difficult to use, and we verified the effectiveness of the training program.

Methods

We conducted a workshop in which 66 students evaluated the usability of appliances they normally use at home, and they commented on both the good and bad attributes. To verify the effectiveness of usability evaluation training, the comments of 61 students were categorized based on their descriptions, and their levels of comprehensiveness were verified. Additionally, we conducted text mining on 14 students’ comments regarding their impressions of the training program.

Results

Ninety percent of the students commented on the consistency and suitability of the presentation of information on the subject equipment. This represented a statistically significant higher number of comments than that for the other topics. However, the students were least likely to comment on the ease of listening to and viewing information on equipment displays, with 62% of the students commenting on this aspect. The students’ comments regarding their impressions of the training program included the lessons they had learned and specific plans on how they would apply their training in the future.

Conclusions

Students could use the target equipment and evaluate their usability while being aware of their cognitive characteristics and mental models throughout the training period. This training program also contributed to providing students with an interest in usability evaluation, its importance, confidence in usability evaluation, and renewed motivation to learn in the future.

usability training

healthcare student education

patient safety education design

human factors and ergonomics training

Improving the usability of medical devices is a crucial issue in ensuring patient safety. According to the ISO 9241-11:2018, usability is defined as “the extent to which a system, product, or service can be used by specific users to achieve specific goals effectively, efficiently, and satisfactorily within a specific context of use” [1]. In other words, it is important to design user interfaces of medical devices that enable healthcare providers to provide the intended medical treatment efficiently and without error. Indeed, there have been reports of medical incidents thought to have resulted from poor usability. For example, operational errors, such as entering an incorrect speed or time into intravenous smart pumps, are considered major causes of medication errors [2, 3, 4, 5]. Problems in usability can cause errors in medical practice, even among experienced doctors [6]. The Association for the Advancement of Medical Instrumentation (AAMI)/Food and Drug Administration (FDA) infusion device summit conducted in 2010 called for a reduction in errors by improving usability and user-interface development [7], and an increasing number of studies have been focusing on the safety of infusion devices [8]. To improve the work systems of healthcare providers effectively, it is desirable to apply human factors/ergonomics (HFE) methods [9]. The US national academies [10] and the World Health Organization (WHO) [11] have recommended that patient safety can be ensured by promoting the application of HFE methods in healthcare.

Human-centered design (HCD) is often applied as an ergonomic approach to usability. HCD is an approach to interactive system development that specifically focuses on making systems usable [1]. HCD mainly comprises the following four activities: understanding and specifying the context of use, specifying user requirements, producing design solutions, and evaluating the design [12]. In HCD, specifying the user requirements is essential, and it is necessary to determine user dissatisfaction with devices that are difficult to use. Communication between medical device designers and users to identify user problems affects the safety of medical devices [13]. Therefore, usability evaluation conducted by incumbent healthcare providers is crucial for determining issues specific to the health care domain [14].

Among healthcare providers, clinical engineers (CEs) are medical device specialists. In Japan, CEs are not only responsible for cardiopulmonary, respiratory, and dialysis care but also for a wide range of diseases, treatments, and medical equipment as well as the management of the entire medical facility’s safety [15]. Identifying user dissatisfaction is an important process in assessing usability, and the expectations for CEs involved with medical devices on a daily basis are significantly high. However, although CEs often evaluate the performance of medical devices with regard to patient care and safety related to electricity, gas, and electromagnetic waves, they rarely evaluate the usability of device interfaces. One possible reason behind this attribute is the lack of education on usability. Because CEs are essential in ensuring patient safety, it is necessary to ensure that they understand the concept of system safety based on HCD and HFE in their training schools [16, 17]. However, in training schools, there are few examples of usability education that focuses on all healthcare providers, and not just on CEs.

In this study, we conducted usability evaluation training for students enrolled at a CE training school. In addition, we verified the effectiveness of our training program by analyzing the results of the students’ usability evaluation. We also conducted text mining on the students’ comments regarding their impressions about the training program.

Overview of usability evaluation training

The objective of this training program was to enable the students comment on the usability of the equipment. This training program was conducted in classes on HFE between July 2021 and July 2022 among 66 third-year students enrolled at a CE training college. The HFE class series comprised 15 mandatory sessions, one of which provided usability evaluation training to all participants. First, as an introduction, we explained the purpose of the training program, the definition of usability, and the importance of usability evaluation in CE. We then explained the methods for usability evaluation.

Specifically, referring to Norman’s seven stages of the action model and the human–machine systems model [18, 19], we explained that the equipment should be evaluated from four perspectives: perception, comprehension, action, and feedback. The action model breaks down human actions into the following seven processes: forming a target, forming intention, specifying an action, executing an action, perceiving the state of the world, interpreting the state of the world, and evaluating the outcome [18]. The Human–machine systems model explains the relationship between humans and machines as follows: a user senses and processes a machine’s status through its display and operates a machine using its controller. These models represent usability evaluation and have been referenced in recent studies [20, 21]. During training, we aggregated these models and established that a device that can execute perception, comprehension, action, and feedback contributes to good usability. Regarding perception, information presented in the display of equipment was evaluated in terms of the ease of viewing and listening. Regarding comprehension, the information provided by the equipment was evaluated in terms of whether it was easy to understand and whether it could be used to determine the status of the equipment. Action was evaluated based on the ease with which a user could operate the equipment using its controller and obtain the intended outcome. Feedback was evaluated based on whether a user received complete and continuous feedback from the device regarding the results of the action.

In this training program, we conducted a workshop in which students evaluated the home appliances they frequently use from four perspectives: perception, comprehension, action, and feedback. The reason for using home appliances as the targets for evaluation was that the students had no experience in usability evaluation. Therefore, we believed that devices they were sufficiently familiar with would be more suitable for training. Each student had 20 min to evaluate the usability of their selected appliances, recording both good and bad attributes using photos and text. Next, in groups of 3–4, they were given 15 min to share their results. Finally, the representatives from each group presented the results of their usability evaluations, which were critiqued by the faculty.

Verification of training effectiveness

Two approaches were used to verify the effectiveness of the training program. We expected that the students who attended the training program would be able to evaluate usability comprehensively in the following four categories: perception, comprehension, action, and feedback. Therefore, we analyzed the results of the usability evaluations presented by 61 students who agreed to participate in this study. First, one researcher who was an expert in HFE (certified professional ergonomist) read all the evaluation comments written by the students and counted the number of topics described by each student regarding the four evaluation categories. The differences in the number of topics among the four categories were then tested using Friedman's test. If the test result was significant (p < 0.05), Bonferroni multiple comparison was conducted. The significance level of Friedman’s test was set at 5%. SPSS Statistics ver. 28 (IBM, Armonk, NY, USA) was used for statistical analysis.

Next, student feedback on usability training was analyzed to verify the effectiveness of the training program. Students submitted their impressions of any three classes as a final report after completing the 15-session HFE training series. This study analyzed the comments of all 14 of the students who submitted comments on usability training. Text mining was performed using Nvivo ver. 1.7.1 (QSR International, Burlington, MA, USA) and coded according to the topics described. Because multiple topics were often described in a single student’s comment, the analysis was conducted on a per-sentence basis. Based on the frequent word analysis of the comments from the coding results, the students’ subjective evaluations of the training program were identified.

Number of topics described by each student for each of the four usability evaluation categories

Table 1 shows the appliances that 61 students evaluated for usability. Approximately 40% of all the students involved in the study evaluated cooking appliances, such as microwave ovens and toasters. Thirty-eight students, or approximately 60% of the total, evaluated multi-functional products (e.g., smartphones and microwave ovens). The remaining 40% evaluated single-function products (e.g., remote controls and scales).

Table 1

Appliances evaluated by students
Product Categories	n (%)
Cooking appliances (microwave, toaster, etc.)	26 (42.6)
Computer peripherals (printers, etc.)	7 (11.5)
Seasonal appliances (AC, fans, etc.)	7 (11.5)
Household appliances (vacuum cleaner, washing machine)	6 (9.8)
Information appliances (PCs and smartphones)	6 (9.8)
Audio appliances (remote controls, speakers, etc.)	5 (8.2)
Beauty appliances (scales and hair irons)	2 (3.3)
Clock	1 (1.6)
Automobile	1 (1.6)

Figure 1 shows the average number of topics described by each student in each of the four evaluation categories. Comprehension had the highest mean number of topics per student at 2.2 ± 1.57, whereas perception had the lowest at 0.9 ± 0.88 (mean ± SD). Friedman’s test results showed χ² = 25.811, p < 0.001, with significant differences among the evaluation categories. The results of the Bonferroni multiple comparison are listed in Table 2. There were significant differences between comprehension and all other evaluation categories. In other words, the number of topics on comprehension was the highest.

Table 2

Bonferroni multiple comparison test results
Category A	Category B	Mean Difference Category A - B	Test statistic	P
Perception	Action	-0.3	-0.369	0.688
	Feedback	-0.3	-0.418	0.442
	Comprehension	-1.3	-1.049	0.000^a
Action	Feedback	< 0.1	-0.049	1.000
	Comprehension	-0.9	0.680	0.022^b
Feedback	Comprehension	-0.9	0.631	0.042^b
Note: Bonferroni correction, ^aStatistically significant (p < 0.01), ^bStatistically significant (p < 0.05).

Students’ impression comments on the training program

The comments presented by the 14 students comprised 42 sentences. After coding the comments according to their descriptive topics, 18 sentences (43%) described what the students had learned during training (learned content). The remaining 24 sentences (57%) contained plans on the ways in which the students would like to practice what they had learned when they become CEs in the future (plans for practice). For each of the two topics, the frequently occurring words in the relevant sentences are shown as word cloud diagrams (Fig. 2: learned content; Fig. 3: plans for practice). The orange letters in the center of the figure represent high-frequency word groups. The next high-frequency word groups are shown using bold black letters. Other word groups are shown in grey font.

A total of 99 words were described in the comments regarding learned content. Of these, ‘use,’ ‘medical,’ ‘devices,’ ‘easy,’ ‘equipment,’ and ‘error’ were described with high frequency, in that order. ‘Easy’ was described as meaning ‘easy to use’ or ‘easy to handle.’ ‘Error’ was described as ‘help prevent human error,’ ‘we can contribute to the prevention of human error,’ or ‘be useful in preventing incidents caused by operational errors.’

A total of 181 words were described in the comments regarding plans for practice. Of these, ‘medical,’ ‘equipment,’ ‘use,’ ‘devices,’ ‘usability,’ and ‘manufacturers’ were described with high frequency, in that order. ‘Manufacturers’ was described as ‘communicate with medical device manufacturers,’ ‘we should tell the medical equipment manufacturer what is difficult to use,’ or ‘giving our opinions to medical equipment manufacturers.’ Other specific words related to plans for practice, ‘purchasing’ and ‘questionnaires’ were commented on with high frequency. ‘Purchasing’ was described as ‘when purchasing and selecting medical equipment.’ ‘Questionnaires’ was described as ‘asking users to answer the questionnaire.’

Usability evaluation outcomes and training effectiveness

When students evaluated the usability of the appliances, comments on comprehension were the most frequently presented, whereas comments on perception were the least. Ninety percent of the students involved commented on comprehension, whereas 84% commented on feedback, 80% commented on action, and 62% commented on perception.

Regarding comprehension, the information provided by the equipment was evaluated in terms of whether it was easy to understand and whether it could be used to determine the status of the equipment. This would involve forming a target, forming intention, specifying an action, and interpreting the state of the world according to Norman’s seven stages of the action model [18]. These usability problems are caused by a mismatch between the mental model of the equipment’s designer and the user’s mental model [18]. The mental model represents the understanding in a person’s mind regarding the way in which things work. Users are forced to rely only on the information provided by the equipment and their mental model to understand the state and behavior of the equipment designed to reflect the designer’s mental model. Usability problems related to understanding the state and behavior of the equipment, i.e., comprehension, are partly owing to the designer’s inadequate understanding of the user’s mental model. In other words, when healthcare providers identify problems related to comprehension through usability evaluation and communicate them to medical device designers for improvement, they can significantly contribute to the safety of medical device usage. However, users with expertise in Business-to-Business (B to B) products, such as medical devices, attempt to master them even if they are difficult to use [14, 22, 23]. In other words, it is not easy for healthcare providers without specialized training in HCD, usability, or HFE to identify various problems with medical devices. Based on the above, it is a crucial outcome that students could evaluate many topics related to comprehension during training. Specifically, we established that students could evaluate the consistency and appropriateness of the information presented by the equipment while being aware of their mental models as users. This is similar to the cognitive walkthrough method, which is a typical usability evaluation method. This method evaluates usability by having the evaluator assume the role of a hypothetical user and actually use the target device while being aware of their cognitive characteristics and mental model [24, 25]. The cognitive walkthrough method is best suited for evaluating B to B products [22]. In this training program, students practiced usability evaluation with themselves as assumed users. However, if they could conduct evaluation by assuming the roles of other users and the characteristics of medical facilities in the future, they might be able to contribute to the improvement of medical device usability when they become CEs in the future.

However, approximately 40% of the students did not evaluate their perceptions. Regarding perception, information displayed by the equipment was evaluated in terms of the ease in viewing and listening. This may be owing to the fact that the appliances evaluated in this training program were multifunctional. Approximately 60% of the students evaluated multifunctional products (Table 1). For such products, the complexity of information structures, such as hierarchical menus, has become more problematic than the ease of viewing the information displayed, which has been emphasized in single-function home appliances in the past [26, 27]. Therefore, it is possible that students focused their comments on comprehension rather than perception. In future training programs, targeting not only home appliances but also medical devices with usability problems could enable students to practice usability evaluation regarding perception.

Students’ impressions regarding this training program

Using the ARCS model [28], we discuss the learning motivations that this training program provided to the students. This model is based on a four-part framework that addresses issues in instructional design related to one’s motivation to learn: attention (stimulating learner interest and intellectual curiosity), relevance (creating familiarity and meaning in the content), confidence (building confidence in the learner), and satisfaction (motivating learners to learn anew).

Students’ comments regarding learned content included ‘easy to use,’ ‘easy to handle,’ ‘helps prevent human error,’ ‘we can contribute to the prevention of human error,’ and ‘is useful in preventing incidents caused by operational errors.’ From this, students may have learned that it is important for medical devices to be easy to use and operate, and usability evaluation contributes to the promotion of patient safety and the prevention of human error. In other words, this training program may have provided the students with ‘attention and relevance’ involving learning about usability evaluation.

Next, comments regarding plans for practice included ‘communicate with medical device manufacturers,’ ‘we should inform the medical equipment manufacturer on what is difficult to use,’ ‘giving our opinions to medical equipment manufacturers when purchasing and selecting medical equipment,’ and ‘asking users to answer the questionnaire.’ In summary, the students could develop concrete plans for the future, such as actively communicating with medical device manufacturers about usability, practicing usability evaluation when purchasing or selecting medical devices, and conducting questionnaire surveys as a specific method for usability evaluation. Specific planning is believed to result from the confidence that students can conduct usability evaluations. It can also be concluded that this training program contributed to eliciting new learning motivations for the future (satisfaction).

Therefore, the training program may have provided students with each element of the ARCS and contributed to their motivation to learn more about usability evaluation.

Limitations

This study has several limitations. First, training was conducted at a single CE training school. Therefore, we could not verify the effectiveness of this training program in other schools or in other healthcare profession training schools. Second, this training program involved usability evaluations only on home appliances that students use frequently. Because this training program was the students’ first experience with usability evaluation, we targeted products that they were fully proficient in using. In future studies, additional refresher usability training should be conducted using medical devices used by CEs. It is also necessary to consider the intervals, how often, and using what content refresher training should be conducted after the initial training provided in this study. Finally, this study verified the short-term effects of training only. In future studies, it is necessary to verify the long-term effects of training programs, including behavioral changes among students and their contributions to usability evaluation after they become CEs.

In this study, we conducted usability evaluation training among clinical engineering students. In addition, to evaluate the effectiveness of the training program, the outcomes of the students’ usability evaluations were analyzed. We also applied text mining on the students’ comments regarding their impressions of the training program. As a result, students recorded many usability evaluation comments regarding the consistency and adequacy of the information presented on the displays of the subject equipment. In the training program, students could evaluate the usability of the target equipment while being aware of their cognitive characteristics and mental models as users. Analyses of the students’ comments on their impressions indicated that the training program provided students with a sense of interest and significance in learning about usability evaluation. Furthermore, students commented specifically on their plans to apply what they had learned through the training program, thereby suggesting that the training program contributed to providing students with confidence in usability evaluation and motivation to learn in the future. If we continue to conduct refresher usability training in the future, and if students can conduct usability evaluations by assuming the characteristics of other users and medical facilities, they may be able to contribute to improving the usability of medical devices when they become CEs. In other words, it is expected that this training program will positively contribute to the quality of healthcare and the safety of medical equipment.

ARCS: Attention, Relevance, Confidence, and Satisfaction

CE: Clinical Engineer

HCD: Human Centered Design

HFE: Human Factors/Ergonomics

Ethical approval and consent to participate

The ethical requirements associated with this study were in accordance with the Declaration of Helsinki. We emphasized and explained to the students involved that participating in this study was voluntary and declining to participate would have no influence on their grades. We also explained that consent to participate in the study could be withdrawn at any time, the results may be published after processing, and the students’ personal information would not be revealed in the publication. All the students provided written informed consent. The study design was approved by the Ethics Review Committee of Jichi Medical University (Approval No. 20-006).

Consent for publication

Not applicable.

Availability of data and materials

The datasets generated during and analyzed during the current study are not publicly available due to privacy and ethical concerns but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

Not applicable.

Authors' contributions

YM contributed to the conceptualization, methodology, validation, formal analysis, investigation, resources, data curation, writing (original draft, review, and editing), and visualization.

SS contributed to the conceptualization, methodology, data curation, writing (review and editing), and visualization.

Acknowledgments

We would like to thank Editage (www.editage.com) for English language editing.

ISO 9241-11. :2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en. Accessed 23 Jun 2023.
Giuliano KK. Intravenous Smart Pumps: Usability Issues, Intravenous Medication Administration Error, and Patient Safety. Crit Care Nurs Clin. 2018;30(2):215–24.
Keohane CA, Hayes J, Saniuk C. Intravenous medication safety and smart infusion systems: lessons learned and future opportunities. J Infus Nurs. 2005;28:321–8.
National Quality Forum. Critical paths for creating data platforms: intravenous infusion pump devices. 2012. https://www.qualityforum.org/Publications/2012/10/Critical_Paths_for_Creating_Data_Platforms__Patient_Safety__Intravenous_Infusion_Pump_Devices.aspx Accessed 23 Jun 2023.
Giuliano KK, Niemi C. The urgent need for innovation in I.V. smart pumps. Nurs Manage. 2015;46(3):17–9. 10.1097/01.NUMA.0000461066.79777.06.
Carayon P, Wetterneck TB, Rivera-Rodriguez AJ, et al. Human factors systems approach to healthcare quality and patient safety. Appl Ergon. 2014;45:14–25. 10.1016/j.apergo.2013.04.023.
Association for the Advancement of Medical Instrumentation (AAMI). Infusing patients safely. Priority issues from the AAMI/FDA infusion device summit. 2010:1–48. https://www.aami.org/docs/default-source/reports/aami_fda_summit_report.pdf. Accessed 23 Jun 2023.
Lyons I, Furniss D, Blandford A, et al. Errors and discrepancies in the administration of intravenous infusions: a mixed methods multihospital observational study. BMJ Qual Saf. 2018;27(11):892–901. 10.1136/bmjqs-2017-007476.
Werner NE, Ponnala S, Doutcheva N, Holden RJ. Human factors/ergonomics work system analysis of patient work: state of the science and future directions. Int J Qual Health Care. 2021;33(Suppl1):60–71. 10.1093/intqhc/mzaa099.
National Academy of Engineering (US) and Institute of Medicine (US) Committee on Engineering and the Health Care System. In: Reid PP, Compton WD, Grossman JH, Fanjiang G, editors. Building a Better Delivery System: A New Engineering/Health Care Partnership. Washington (DC): National Academies Press (US); 2005.
World Health Organization. Human Factors. Geneva: World Health Organization; 2016.
Komatsubara A. Human Centred Design on Medical Devices. J Jpn Soc Precis Eng. 2008;74(2):118–20. 10.2493/jjspe.74.118.
Vincent CJ, Li Y, Blandford A. Integration of human factors and ergonomics during medical device design and development: it's all about communication. Appl Ergon. 2014;45(3):413–9. 10.1016/j.apergo.2013.05.009.
Oga K, Takano J, Funai T, Komatsu T, Tokuda K. Evaluating the Effectiveness of Think-Aloud Protocol and Impact Analysis on the Usability Testing for Healthcare Professionals. Jpn J Ergon. 2022;58(1):3–10. 10.5100/jje.58.3.
Shonohara K. Status and future of clinical engineers, from viewpoints of a surgeon and educator. Japanese J Med Instrum. 2018;88(1):67–70. 10.4286/jjmi.88.67.
Hirose M, Tsukao H. Change on the education and future’s problem at clinical engineer’s educational facilities. Japanese J Med Instrum. 2018;88(1):48–53. 10.4286/jjmi.88.48.
Shinohara K, Idebuchi Y, editors. Rinshoukougaku kouza: Iyoukiki anzen kanrigaku 2nd Edition [Clinical Engineering Seminar: Safety Management of Medical Devices 2nd Edition]. Tokyo: Ishiyaku Publishers, Inc; 2015.
Norman D. The Design of Everyday Things. New York: Basic Books; 2013.
Chapanis A. Man-machine engineering. Belmont: Wadsworth; 1965.
Villani V, Lotti G, Battilani N, Fantuzzi C. Survey on usability assessment for industrial user interfaces. IFAC-Papers Online. 2019;52(19):25–30. 10.1016/j.ifacol.2019.12.078.
Lindblom J, Alenljung B. The ANEMONE: theoretical foundations for UX evaluation of action and intention recognition in human-robot interaction. Sensors. 2020;20(15):4284. 10.3390/s20154284.
Mizumoto T, Yamaoka T. Consideration of a Usability Evaluation Method of B to B Products, Based on Testing of a Medical Device. Bull Japanese Soc Sci Des. 2014;60(6):103–8. 10.11247/jssdj.60.6_103.
Shinohara K. Problems of Collaborative Research and Development for ME Devices in Japan. Trans Jpn Soc Med Biol Eng. 2008;46(3):304–8. 10.11239/jsmbe.46.304.
Polson PG, Lewis C, Rieman J, Wharton C. Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int J Man-Mach Stud. 1992;36(5):741–73. 10.1016/0020-7373(92)90039-N.
Wharton C, Rieman J, Lewis C, Polson P. The cognitive walkthrough method: A practitioner's guide. In Usability inspection methods. Hoboken: John Wiley & Sons, Inc.; 1994.
Ishii K, Harada E. Are hierarchical structures understood in everyday life? Factors that hinder understanding the menu structures of home appliances. Tsukuba Psychol Res. 2022;60:21–8.
Ziefle M, Bay S. How to Overcome Disorientation in Mobile Phone Menus: A Comparison of Two Different Types of Navigation Aids. Int J Hum-Comput Int. 2006;21:393–433. 10.1207/s15327051hci2104_2.
Keller JM. Development and use of the ARCS model of instructional design. J Instr Dev. 1987;10:2–10. https://doi.org/10.1007/BF02905780.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Initial usability evaluation training for clinical engineering students

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Introduction

Methods

Overview of usability evaluation training

Verification of training effectiveness

Results

Number of topics described by each student for each of the four usability evaluation categories

Students’ impression comments on the training program

Discussion

Usability evaluation outcomes and training effectiveness

Students’ impressions regarding this training program

Limitations

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1