Development of User Suitability Evaluation Method of Cognitive Training System Through Application x

Background: To check in advance whether there are any problems in use and to address these in the process of designing a medical device, it is necessary to evaluate user suitability. Yet currently, there are a few user suitability evaluation constructs established for unique technologies. Therefore, the development of user suitability evaluation constructs for the user suitability evaluation of self-developed computerized cognitive rehabilitation applications should be prioritized. Method: To evaluate the user suitability of the computerized cognitive rehabilitation application developed in this paper, we derived the user suitability evaluation constructs of user satisfaction, time satisfaction, diculty, comprehension, effectiveness, eciency, and error, and after clinical trials for user suitability evaluation, we analyzed the results and tried to identify and address the problems of the developed evaluation constructs. Twenty-four senior citizens over 55 years of age participated in the clinical trial, and performed cognitive rehabilitation training once a week for 3 weeks using our self-developed cognitive rehabilitation application. Result: As the weeks progressed, satisfaction increased in orientation training and concentration training. User suitability evaluation constructs for effectiveness, eciency, and error were appropriate, but there were insucient aspects in the evaluation scales for user satisfaction, time satisfaction, diculty, and understanding; Conclusion: we were able to derive an improvement plan by extracting the established user suitability evaluation scale. Trial registration: Ethics approval and consent to participate. The evaluation was conducted after obtaining Institutional Review Board approval from Dongguk University Ilsan Hospital (DUIH2020-07-001-001).


The Background of User Suitability
The concept of usability has been discussed since 1990. Usability, as de ned by Jakob Nielson, is a term that indicates how easy it is to use a product to achieve a user's desired purpose [1]. In the process of designing a product or service, it also indicates how to improve the product to make it easier for users to use. The sub-attributes of usability de ned by Jakob Nielson are 1. Learnability, 2. E ciency, 3. Satisfaction, 4. Memorability, and 5. Errors. 1.
Learnability refers to the ease with which users can perform basic tasks with a design they rst encounter. 2.
E ciency refers to the speed of task execution when users are familiar with the design. 3. Satisfaction refers to how pleasantly satis ed the user is when using the design. 4. Memorability refers to the ease with which the user can perform a task when user encounters the design again after a period of not using the design. 5. Errors refer to how often and how many errors users make when working with the design, and how easily they can overcome them.
User suitability has recently emerged as a metric in the process of developing medical devices, and research on developing evaluation methods suitable to the characteristics of products for user suitability evaluation in various elds is ongoing. In the usability test, a suitable evaluation method for the product should be developed so that all factors that hinder the training effect can be found and prevented according to the convenience, usage errors, cognitive errors, and perceptual errors that may occur in the user's direct use. Therefore, testing usability by establishing usability evaluation constructs that can clearly test the usability of the product in the development process before the actual product is made is one of the effective ways to check whether there is any problem with the product 1.2 The Need for User Suitability Page 3/24 A user suitability test provides many advantages while developing a product or service. Through a user suitability evaluation, problems can be found and addressed before a product's release, saving time and cost. Therefore, it is possible to reduce unnecessary processes in the product development process by establishing usability evaluation constructs that can clearly test the usability of the product and conducting the user suitability test. In addition, for medical devices, it is necessary to continuously check whether there are any problems after market release through follow-up management. At this time, through the usability test, it is possible to discover supplements and improvements to the product, and to con rm the needs and preferences of users.

The Background of User Suitability
Since no user evaluation method has been established for cognitive rehabilitation application, we analyzed previous studies to develop an evaluation method that is suitable for cognitive rehabilitation application. Looking at previous studies that evaluated user suitability of web services or applications in elds other than cognitive rehabilitation, Park Min-su et al. established effectiveness, e ciency, and error evaluation as user suitability evaluation scales for web usability evaluation, measured training performance, required time, and error rate, and con rmed whether users can process information as desired. For the success/failure of the task, the training performance was quanti ed through satisfaction by grade and the number of errors. Total execution time of the task was measured for the e ciency and the number of errors were measured for the error, and the satisfaction score was quanti ed using a 5-point Likert scale [2]. In addition, in the study of human-computer interaction Gee, Park Ji-hyeong et al. studied the effects of the repetitive context of the game and the potential effect of interactivity on job training, and is pursuing ongoing research on web usability evaluation, such as how the design, interactivity and response speed of an application give satisfaction to users and increase their loyalty to applications [3] [4]. Based on these previous studies, we tried to develop a user suitability evaluation method for cognitive rehabilitation training. In addition, we conducted clinical trials using the user suitability evaluation method developed to evaluate cognitive rehabilitation applications, and attempted to derive and address the problems of user suitability evaluation constructs through clinical trials. The clinical trial was conducted on subjects over the age of 55 with normal cognitive abilities, and the safety of the subjects was considered through IRB approval for safety in use.
Questionnaire for User Interface Satisfaction (QUIS) [5] and Satisfaction and Ease of use Questionnaire (USE) [6] have been heavily used among the cases of other questionnaires used in the existing user suitability evaluation, but their evaluation constructs are inappropriate to evaluate the user suitability of a computerized cognitive rehabilitation training system. Accordingly, Ahmed Mohammed Elaklouk developed evaluation constructs based on the constructs of the veri ed questionnaire for user suitability evaluation of the rehabilitation game system for cognitive de cits [7]. Therefore, in this paper, we developed evaluation constructs suitable for the computerized cognitive training system based on the frequently used questionnaire. In addition, we set up evaluation factors to measure the evaluation scale by referring to existing studies [8].

Cognitive Rehabilitation and Cognitive Training
Cognitive training is a program that trains learners in skills and strategies to improve speci c cognitive functions, and consists of training tasks that re ect that cognitive function in a prescribed way. For Cognitive rehabilitation, disability and health are judged at the body organ level, individual level, and social level with International Classi cation of Functioning, Disability and Health(ICF) introduced in 2001 [9]. The concept of cognitive rehabilitation is one of a treatment process that aims to restore, complement, and replace the deteriorated cognitive function from the point of view of the functional level of the body. From an individual and social level perspective, the focus will be on the limitations of daily life and social life due to a decline in cognitive function.

Subjects for Cognitive Training Evaluation
According to the Act on the Prohibition of Age Discrimination in Employment and the Promotion of Employment of the Elderly in Korea, a person aged 55 or older is referred to as an elderly person [10]. Zhengxiang Pan et al. studied the effect of familiar design on elderly users in wellness games for elderly users over 65 years of age. As a result, it was found that familiarity with design improved satisfaction and adoption potential among elderly users [11]. Based on this, in this study, we consider that the age group that perceives the most di culty in skillfully operating the application is the senior layer, and the improvement of the senior layer's familiarity in design will be an important factor to improve the satisfaction. Therefore, we recruited 25 adults over the age of 55 to use the self-developed application, and then conducted a user suitability evaluation. The Korean version of Mini-Mental State Examination (MMSE-K) was performed to screen the cognitive abilities of the seniors, who are the subjects of clinical trials.
Cognitive rehabilitation was designed with reference to the typical cognitive training tasks of each cognitive domain to proceed with each subdomain of orientation, concentration, and memory using the application. We also developed a questionnaire necessary for user suitability evaluation with user suitability questions for each construct on a 5-point scale, as shown in Table 4. A high score in each item of the user suitability questionnaire evaluation means that the user suitability is high. It is designed to give 1 point if it is very di cult or very inconvenient, 2 points if it is di cult or inconvenient, 3 points if it is moderate, 4 points if it is easy or convenient, and 5 points if it is very easy or very convenient.

Cognitive Rehabilitation Training Process
The usability evaluation period of the cognitive rehabilitation training application considering the user's safety through the IRB was performed once a week for 3 weeks. The trainings constituting the cognitive rehabilitation application are orientation, memory, and concentration, which are functions that recognize time, place, and humans. Table 1 shows the detailed composition of memory and concentration. In the rst week, before proceeding with full-scale training, the subjects lled out a consent form and a personal information questionnaire asking for personal information such as age, gender, academic background, and medical history. Cognitive rehabilitation training was conducted through a program the subject accessed through a tablet computer that proceeded through orientation, concentration, and memory, and after training the subjects lled out a questionnaire to evaluate satisfaction in terms of time satisfaction, use satisfaction, comprehension, and di culty.
Required time, number of errors, and training performance were recorded automatically at the end of each detailed training. Each detailed training required the subject to enter the correct answer before proceeding to the next stage.
The user suitability evaluation process conducted for 3 weeks is shown in Fig. 1, and MMSE-K was performed before the rst week of cognitive rehabilitation training and after the third week of cognitive rehabilitation training. The training was required to be performed by the subjects themselves, but the investigator waited in the same room so that the subjects could freely ask for help during the training. Table 2 shows factors that can be the subjects of user suitability evaluation in each detailed training of the cognitive rehabilitation training using applications.

Cognitive Rehabilitation Training Applications
This application allows the level of training to be classi ed according to the di culty of the task. In the area of orientation, there are a total of 5 questions, and as shown in Fig. 2, the current year, month, and day are asked, and then selecting a different material, selecting a speci c fruit, selecting a different purpose, and selecting a seagull and a sweet potato are included among the examples given.
In the area of search concentration, as shown in Fig. 3A, letters composed of numbers and English letters are displayed in a row, and it is a training to identify how many characters suggested in the lower left box appear in the presented strings. It consists of a total of 5 stages. In stage 1, a single-character string is presented, and the number of strings presented increases as the di culty increases. In the case of continuous concentration, as shown in Fig. 3B, the numbers are selected sequentially from small to large, but starting from the blue box containing the numbers between the blue box and yellow one to the other coloured boxes alternately. In stage 1, the numbers presented are sequential, and in stage 2, the numbers presented are out of sequence. In stage 3, as shown in Fig. 3C, the objective is to click from the small number to the large number alternately in the boxes of different colours. Colour matching concentration is a training nd a match between the colour of a word and the colour that the word means. There are 4 levels in total, and the number of colours presented increases as the di culty increases. In the case of memory concentration, as shown in Fig. 3D, it is a training to memorize pictures or words and select matching words by memorizing actual pictures, and consists of a total of 5 stages.
For word memory, the component is a training to remember consecutive words, as shown in Fig. 4A. It consists of a total of 5 stages, and the number of words presented increases as the di culty increases. General number memory is a training to remember consecutive numbers and their sequence as shown in Fig. 4B. It consists of a total of 5 stages, and the number of numbers that must be remembered increases as the di culty increases. For mental arithmetic number memory, it is a training in mental arithmetic, and there are a total of 5 questions. For colour memory, it is a training to remember colours as shown in Fig. 4C. It consists of a total of 5 stages, and the number of colours to remember increases as the di culty increases. Matching memory is a training to memorize words to nd images, and memorize images to nd words, as shown in Fig. 4D, and there are a total of 8 stages. As the di culty increases, the number of words and images presented increases.

User suitability evaluation method developed for cognitive rehabilitation applications
Evaluation constructs for user suitability evaluation of computerized cognitive rehabilitation training applications are user satisfaction, time satisfaction, di culty, understanding, effectiveness, e ciency, and error. In addition, the evaluation factors for measuring each construct were whether the execution of the task is possible/impossible for 'effectiveness,' task execution time for 'e ciency,' and the number of errors for 'error.' User satisfaction, time satisfaction, di culty, and understanding were measured on a 5-point scale using a questionnaire, and the evaluation points of the questionnaire are shown in Table 3. If incorrect answers are given more than 5 times, the investigator measured the effectiveness by making the task impossible to perform, and quanti ed the effectiveness, e ciency, and errors of each training for the record. In the meantime, the subjects lled out the user suitability questionnaire, completing the user suitability evaluation. Table 3 User suitability questionnaire evaluation points The evaluation scales for use satisfaction, time satisfaction, di culty, and understanding constructs are shown in Table 4, and each evaluation scale for the effectiveness, e ciency, and error constructs is shown in Table 5.
The constructs set for the user suitability evaluation suitable for computerized cognitive rehabilitation application were use satisfaction, time satisfaction, di culty, and understanding, with which we tried to check whether the use method of training through each application was easy, whether the time limit set for each training was appropriate, whether it was di cult to understand the training method because the subject was 55 years of age or older, and whether the di culty level was appropriate; also, we tried to check how well users can learn and use the system without errors with the constructs of effectiveness, e ciency, and errors. [12].  Table 4 shows the results of executing MMSE-K, which can measure a patient's cognitive function, on select seniors with normal cognitive ability among the recruited personnel. In the MMSE-K, a score of 24 or more out of 30 is judged to be normal. According to the MMSE-K results, none of the subjects had any cognitive problems.

Effectiveness results
For training performance, inability to perform did not occur in orientation, and 14 subjects were treated as unable to perform training because they failed to correct a problem more than 5 times in the concentration and memory training.
As a result of the training performance of the concentration training, as shown in Table 7, the number of incorrect answers decreased as the weeks progressed in the search concentration and continuous concentration, where inability to perform occurred in week 1, and the number of subjects who were unable to perform also decreased.
Colour concentration, where the largest number of subjects showed inability to perform in the 1st week, showed a decreased number of subjects who were unable to perform in the 3rd week. As a result of the performance of memory training, in Table 8, the number of subjects who were unable to perform in general number memory and mental arithmetic number concentration, which occurred in the 1st week, was smaller in the 3rd week than in the 1st week. Colour memory 0 0 0 Matching memory 0 0 0

Effectiveness results
In terms of the total satisfaction score, the evaluation of use satisfaction, time satisfaction, understanding, and di culty for orientation training was high, and evaluation of use satisfaction, time satisfaction, understanding, and di culty for concentration training was the lowest.
In the result of the orientation training questionnaire, upon adding up scores for each week for each item in Table 9, the score for question 3 for time satisfaction in Table 4 was the lowest at 8.08, and the score increased as the weeks progressed. Adding up each week of the 2nd question of use satisfaction and the 2nd question of time satisfaction in Table 4 showed the highest score result of 14.17. In the result of the questionnaire survey of concentration training, it was found that when each week was added up for each item in Table 10, as with the survey results of orientation, the score of item 3 of time satisfaction in Table 4 was the lowest at 8, and the score increased as the weeks progressed. In Table 5, the sum of use satisfaction, time satisfaction, understanding, and di culty of concentration training were the lower than that of other training, but the satisfaction with use satisfaction, time satisfaction, understanding, and di culty increased as the weeks progressed.
However, as the score for the user satisfaction question 2 of Table 4 decreased in the 3rd week compared to the 1st week, this shows that the ease of touch and click under the user satisfaction is lower, as shown in use satisfaction question 2 of Table 5. From the memory training questionnaire survey, through adding up each week for each item in Table 11, it was found that the item that received the lowest score of 8 points was item 3 of time satisfaction in Table 4, but the score increased as the weeks passed. With a total score of 13.5 for each week, the item with the highest score was item 2 of use satisfaction of Table 4. In Table 11, the sum of use satisfaction, time satisfaction, understanding and di culty of memory training decreased as the weeks passed, while the satisfaction with the preparation time before starting increased.

Error and E ciency Results
The evaluation scales for the error and e ciency evaluation constructs are error rate and time required, respectively. In orientation training, the results of the training for each stage are shown in Table 12, with the number of errors and time required decreasing as weeks passed.
Looking at Table 13 in the 4th stage of the search concentration training, the number of errors and the time required increased in the 3rd week compared to the 1st week. Table 14 shows the results of the reduction in the number of errors and the time required as the weeks passed in continuous concentration training. Referring to Table 15, as the di culty of color concentration increased, the number of errors decreased and the time required also decreased. As for the memory concentration in Table 16, the number of errors and the time required decreased as the weeks passed Looking at Table 17, in terms of word memory, the number of errors in the 4th stage, the highest di culty word memory training, increased as the weeks passed. In Table 18, the number of errors and time required increased in general number memory in stage 3 as the weeks passed. Table 19 shows the results of the increase in the number of errors and the time required in stages 4 and 5 as the weeks passed in mental arithmetic number memory training. In Table 20, the number of errors and the time required increased as the weeks progressed in the 5th stage of colour memory training. Table 21 shows that both the number of errors and the time required increased in the 3rd stage of matching memory. Table 12 The number of errors and the time required in each stage of orientation training

Discussion
In this paper, we developed an appropriate method for a user suitability evaluation that can save time and money by pre-evaluating the usability of a developed cognitive rehabilitation training program. The developed user suitability constructs were judged to be suitable for evaluating cognitive rehabilitation applications, and as the training is repeated in the course of conducting a clinical trial using the developed user suitability evaluation method, it was considered that the number of errors and the time required would be reduced as the time satisfaction, use satisfaction, understanding and di culty, which affect user satisfaction and loyalty, are improved.
As a result of the clinical trial, rstly, in terms of the effectiveness, e ciency, and error constructs, due to problems in the app's con guration, as weeks passed or higher levels of training progressed at some stages of training, the time required or number of errors unexpectedly increased; however, overall effectiveness and e ciency were improved and errors decreased. In addition, by measuring the time required and the number of errors, it was possible to determine how well the user was able to learn and use the system with the appropriate di culty setting. For example, as shown in Table 15, the time required and the number of errors decreased as the di culty increased, indicating that the di culty level in the rst stage was high for the user. In addition, as shown in Stage 4 of Table 13, Stage 4 of Table 17, and Stage 3 of Table 21, the time required and the number of errors increase as the weeks passed, con rming that there was a problem in learning and using this training. In other words, it was con rmed that the user suitability scales of training performance, time required, and number of errors were suitable to identify constructs of effectiveness, e ciency, and errors.
Second, through the experiment on the evaluation constructs for use satisfaction, time satisfaction, understanding and di culty, it was found that satisfaction increased as the weeks passed in the training of orientation and concentration; however, the overall satisfaction was decreased in the use satisfaction, time satisfaction, understanding, and di culty of memory training. In particular, the number of errors and the time required for memory training decreased as the weeks passed, but contrary to expectations, the time satisfaction construct of memory training also decreased as the weeks passed. As the subject repeated the same training as the weeks passed, the time satisfaction was expected to increase as a result of improving the e ciency of the system and reducing the error rate; however, it was considered that the time satisfaction was not improved because the subject had to wait for the answer for a while after suggesting a question due to the program con guration of the application. We judged that the problem of program con guration had an effect on time satisfaction because in the case of concentration training, time satisfaction improved as the weeks passed, despite having the same questionnaire as that of memory training.
Through the above experimental results, it is judged that the user suitability evaluation scale used for user satisfaction evaluation has limitations when it comes to analyzing user satisfaction, and the scale used for user satisfaction evaluation needs to be partially changed according to the detailed training characteristics of the computerized cognitive rehabilitation application. In a training in which a problem is presented and the subject has to wait until subject clicks an answer, it was considered meaningless to evaluate time satisfaction construct. Therefore, it is appropriate to exclude the construct of time satisfaction from the user suitability evaluation of memory training. In addition, in Table 5, the more weeks passed, the lower the ease of touch and click of use satisfaction was obtained.
However, since the evaluation constructs of the developed questionnaire were used equally in all training, it was di cult to determine which speci c training of the concentration training had di culties in the area of ease of touch and click. Also, as the weeks passed, the number of errors and the time required increased in the 3 rd stage of the general number memory training and in the 5 th stage of the color memory training. This is because unlike other training, the mental arithmetic number memory, general number memory training, and color memory training contained random elements, and the subject had to remember mental arithmetic, memory, and random color of random numbers. Accordingly, it was found that the number of errors and the time required increase when a number that is di cult to calculate or remember, or a similar color is presented, even for training that the subjects already know. Therefore, it is necessary to add constructs to check if training suddenly becomes di cult or easier.
Through the user suitability evaluation performed in this way, it was con rmed that it is necessary to develop evaluation constructs suitable for a speci c product or service, and appropriate evaluation scales for the developed evaluation constructs are required according to the characteristics of each program.

Conclusion
In this paper, we developed an evaluation method for a user suitability evaluation of computerized cognitive rehabilitation applications, and clinical trials were conducted for seniors aged 55 years or older to evaluate whether the developed user suitability evaluation method is appropriate. To evaluate the computerized cognitive rehabilitation application, we derived use satisfaction, time satisfaction, di culty, understanding, effectiveness, e ciency, and error as evaluation constructs, and through clinical trials, the above constructs were judged to be suitable as evaluation constructs for user suitability evaluation.
However, due to the program characteristics of the computerized cognitive rehabilitation application, we found that the evaluation elements and scales used for evaluation were not appropriate in the constructs of use satisfaction, time satisfaction, di culty, and understanding.
As such, it is considered that certain evaluation elements and scales used for evaluation need to be added or subtracted according to the product characteristics. Based on this research, we suggest that it is necessary to compose evaluation constructs according to the products or services for user suitability evaluation, and to develop evaluation scales and evaluation elements that re ect the characteristics of products or services. Figure 1 User suitability evaluation process conducted in clinical trials.