Unleashing the Potential of Turkish Chatbots: A Study on the Validity and Reliability of the Bot Usability Scale

doi:10.21203/rs.3.rs-4682608/v1

Download PDF

Research Article

Unleashing the Potential of Turkish Chatbots: A Study on the Validity and Reliability of the Bot Usability Scale

https://doi.org/10.21203/rs.3.rs-4682608/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Purpose

The objective of this study is to adapt and evaluate the Turkish version of the Chatbot Usability Scale (BUS-11) through a confirmatory factorial analysis method. The BUS-11 scale has been established in various languages except for Turkish; thus, its validation and dissemination could serve as a means to improve chatbot interaction satisfaction among the Turkish-speaking population and hence foster growth in Turkey's conversational agent market.

Methods

To achieve this aim, seven customer-oriented chatbots were rated on pre-designed tasks by participants. Data gathering involved utilizing TBUS-11 with UMUX-Lite scale translated into Turkish to gauge individuals' experience after interacting with Turkish-speaking bots.

Results

Results show that TBUS-11 has been demonstrated to be highly reliable with a strong convergent validity with the UMUX-Lite already validated in Turkish. Moreover, the collected data fit in a very satisfactory way the expected five-factor structure of the original version of the scale, thus confirming the psychometric properties of the TBUS.

Conclusion

The study successfully adapted the Chatbot Usability Scale (BUS-11) into Turkish, providing a reliable and valid tool for assessing chatbot usability in the Turkish-speaking market. This can potentially enhance user satisfaction and promote the growth of conversational agents in Turkey.

chatbot

scale adaptation

interaction satisfaction

usability

user experience

human-computer interaction

Chatbot usage has gained widespread popularity in recent years, particularly for business and customer service purposes (Følstad & Brandtzæg, 2017; Taylor et al., 2020). In many fields, including healthcare, finance, and e-commerce, chatbots have become a favored tool for interacting with consumers. Chatbots employ natural language for engaging in effective interactions with users and providing responses to their inquiries. Currently, chatbots are utilized in various domains to offer round-the-clock services. Chatbots possess the ability to address a minimum of 80% of typical customer queries and are accessible round-the-clock, indicating their versatility beyond just being limited to handling inquiries (Adam et al., 2021). Primarily employed for sales-related activities, followed by support and marketing functions, chatbots have demonstrated a notable impact on increasing sales figures with an average improvement of 67% (Press, 2019). Additionally, one-fourth of all transactions involve interactions mediated through these automated conversational agents (Press, 2019). However, in order to fully leverage the potential of chatbots, it is crucial that their usability be evaluated systematically. In other words, the effectiveness and usability of these chatbots remain a subject of inquiry.

On the other hand, the evaluation of user satisfaction with chatbots has become an important area of research to understand how effectively such agents might meet the needs of users enabling a positive experience (Jenneboer et al., 2022). Evaluation scales are available and widely used to assess the interaction quality perceived by the users of a digital system, for instance, reliable and concise scales such as the System Usability Scale (SUS) (Brooke, 1996), the Usability Metric for User Experience (UMUX) (Finstad, 2010), and UMUX-Lite (Lewis et al., 2013). The SUS is a “quick-and-dirty” questionnaire that consists of ten items using a five-point scale, which has been shown to have excellent psychometric properties. In contrast, UMUX employs fewer items in line with the International Organization for Standardization (ISO) definition of usability and utilizes a seven-point scale ranging from strongly disagree to strongly agree. UMUX Lite is a two-item instrument that has acceptable reliability, validity, and psychometric properties, making it a valuable tool for preliminary and rapid testing of user reactions to a prototype (Borsci et al., 2015). Nevertheless, while these tools have been widely used and validated for assessing user satisfaction with web interfaces when it comes to chatbots these scales may not fully capture the conversational aspects of user interaction with chatbots since they were not designed for this purpose (Borsci, Malizia, et al., 2022). In a similar fashion, as suggested by Lewis (2016) and Lewis and Sauro (2020), scales to assess the experience with vocally controlled interfaces were developed in the past, for instance, the Mean Opinion Scale (Salza et al., 1996), the Subjective Assessment of Speech System Interfaces is composed by (Hone & Graham, 2000) and the Speech User Interface Service Quality developed by (Polkosky, 2005). Nevertheless, voice-controlled interfaces are not comparable to chatbot systems.

The development of a novel measurement scale, known as the Chatbot Usability Scale (BUS-11), is one example of recent efforts to create an evaluation tool that can be used specifically for gauging users’ satisfaction following interactions with chatbots. This newly devised scale has been validated and established as fit-for-purpose according to research conducted by Borsci et al. (2022). The BUS-11 composed in its last version of 11 items (Borsci, Schmettow, et al., 2022) was extensively validated and it is currently available and validated in various languages: Dutch, Italian, German, and Spanish. The researchers utilized a comprehensive approach to develop the construct of such a scale (Borsci, Schmettow, et al., 2022). Additionally, the researchers utilized the UMUX-Lite scale (Lewis et al., 2013) to establish convergent validity as a dependent variable. The outcomes of previous studies suggested a positive correlation between BUS-11 and UMUX-Lite (Borsci, Schmettow, et al., 2022). BUS-11 can aid practitioners in comparing and benchmarking their chatbots during product evaluation, allowing them to enhance their performance. The researchers also suggested that the construct and the factors behind the BUS scale (Borsci, Malizia, et al., 2022) can be also utilized as an efficient checklist for chatbot designers during the development process.

Turkey has become a desirable e-commerce market to invest in by Turkish companies and foreign partnerships due to its population of over 80 million and the accelerating e-commerce volume (Akıl & Ungan, 2021). Along with the e-commerce market, Turkey is also a potential growth market for chatbots, with many businesses incorporating chatbots into their operations. CBOT, the leading chatbot developer in the country, claims that 90% of customer inquiries can be handled by customized chatbots with appropriate task customization (İçgözü, 2020). However, it is essential to ensure the evaluation metrics used for chatbots are not only specific but also valid and reliable. It is currently not possible to find a validated chatbot usability scale in the Turkish language. Therefore, further research in this area is highly necessary to ensure the efficacy and comprehensiveness of any evaluation tool used for Turkish chatbots. The validation in Turkey of the BUS-11 to facilitate the assessment and the comparability of the quality of interaction with chatbots might help practitioners in Turkey to benchmark their chatbots within the country and at international level, this might potentially unify and ameliorate the user experience offered by these system and thereby improve the performance of chatbots. Therefore, this study aims to adapt the BUS-11 scale for Turkish language to assess its validity and reliability in evaluating chatbots in Turkey.

2.1 Participants

The study comprised a sample of 199 participants (49% female), with a total of 557 evaluations provided for the chatbots. The study design entails participants engaging with and evaluating a selection of up to four chatbots out of a pool of seven using the Qualtrics Survey platform (see Appendix A for chatbots) The evaluation distribution was as follows: 100 participants assessed four chatbots, 55 participants assessed one chatbot, 30 participants assessed two chatbots, and 14 participants assessed three chatbots. Demographically, the majority (%75.38) of the group belonged to the age bracket between 18–24 years old; while there were around 39 individuals (%19.60) aged from 25 to 34 years old and 10 individuals (%5.02) of 35 to 44 years of age. The nationalities represented in this study included 13 participants hailing from Serbia, Romania, Afghanistan, and Azerbaijan. To ensure the results were accurate and reliable, a strict process was followed wherein only the valid responses from participants remained for further analysis.

2.2. Instruments and study procedure

To adapt the scale to Turkish, approval was sought from the authors who created BUS-11 scale and content validity was considered by consulting with two field experts and one assessment and evaluation expert. The adaptation study aimed at minimizing cultural differences. Language validity of the scale was ensured through a back-translation design aided by two proficient translators who are fluent in both cultures. One of the translators translated from English, the original language of the scale, into Turkish, and the other one translated the first translator's translation into English without seeing the original text. Subject matter specialists were consulted in ensuring linguistic equivalence between versions to ensure robust results that minimize sources error when adapting assessment instruments across languages. Additionally, three experts, including the developer of the BUS-11, agreed on the quality of the back translation.

The translated scale form was applied to a group of 27 participants (15 female, 12 male) in order to determine the items that were not understood and finalized as a result of the feedback received from the participants. All participants were undergraduate students in Turkey. In the pilot study, participants were assigned a task to convert their airline tickets into open ones using an airline chatbot available at https://www.flypgs.com. Following this interaction, they rated their level of satisfaction with the chatbot by answering questions. Regarding understandability, participants were asked to assess from 1 (not understandable to 5 very understandable) the wording of the items. The validity of these responses was measured through Cronbach’s Alpha which yielded an acceptable score of .76. Descriptive statistics were given in Table 1 for the pilot study. Overall, the mean score of each item was high, suggesting that participants had the impression to understand clearly the items. However, item 1 had the lowest mean score and highest standard deviation in terms of understandability. To reduce the risk of potential miscomprehension due to the wording of the translated items, a Turkish language teacher was consulted, who suggested a revision regarding items 1 and 8. So that, TBUS-11 was finalized (see Appendix B).

Table 1

Descriptive statistics of the pilot study of TBUS-11
TBUS Items	Usability of chatbot per item		Understandability assessment
	M	SD	M	SD
Item1	3.48	1.28	4.07	1.20
Item2	4.04	1.09	4.78	.57
Item3	4.19	.73	4.74	.59
Item4	4.26	.85	4.59	.84
Item5	4.26	.81	4.67	.67
Item6	3.78	1.21	4.70	.60
Item7	4.04	.98	4.59	.79
Item8	3.70	1.23	4.52	.84
Item9	4.22	.84	4.67	.67
Item10	4.00	.87	4.41	.88
Item11	4.63	.92	4.78	.57

[Insert Table 1 Here]

In the current study, websites with chatbots in various fields (banking, e-commerce, etc.) that speak Turkish and can be accessed directly (no membership required, etc.) were listed by the researchers. All chatbots were analyzed to avoid selecting chatbots developed by the same developers. In other words, chatbots with the same infrastructure, even if adapted for different sectors, were not selected to ensure diversity. Seven chatbots were determined. Scenarios and tasks were developed for each chatbot to enable participants to use the chatbot for a purpose and elicit particular information (see Appendix A). A two-part survey was designed. In the first part, demographic information was asked to participants. In the second part, participants were asked to choose one of the seven chatbots. They were given a scenario and task developed according to the chatbot they chose and asked to experience the chatbot with this scenario and task. Then, as a form of control, participants were asked to confirm whether they had completed the task and, even if they did not achieve the goal, they were asked whether they had an experience with the chatbot with which they could express their satisfaction. If participants answered positively, they were allowed to answer the items of TBUS-11 scale and the UMUX-Lite scale (Lewis et al., 2013). In addition, as a control question, the participants were asked to say “goodbye” to the chatbot at the end of each task, and then the participants were asked about the chatbot's response to “goodbye”. Thus, in terms of validity, it was aimed to determine whether the participants used the chatbot or not. Participants are allowed to choose up to four chatbots in a session.

Pilot Study Task: You purchased a discounted economy-class flight ticket from Pegasus Airlines from Istanbul to Amsterdam. However, due to incomplete visa procedures, you wish to have the ticket converted to an open one. Try using the chatbot on the website https://www.flypgs.com to find out when is the latest date you can use your open ticket and how much deduction there will be when you convert it to an open ticket.

2.3. Ethical Consent

Before commencing the research, ethical approval was obtained from the Istanbul Bilgi University Ethics Committee to ensure adherence to ethical research standards and protect the rights and well-being of the study participants.

2.4. Data Analysis

Descriptive and regression analyses were used to assess users’ satisfaction after the interaction with all the chatbots. The control of participant interaction was tested by a regression analysis to check the level of satisfaction (TBUS-11) declared by participants who answered correctly or incorrectly the control questions. If participants decided to skip the question, we counted the answer as incorrect. Moreover, a regression was used to test if participants who did not manage to finish the task resulted in less satisfied compared to the overall population.

The data was analyzed using R. After checking for the normality of the distribution by a Shapiro-Wilk test, a confirmatory factorial analysis (CFA) was performed using the structural equation modeling function of the ‘Lavaan’ package. Moreover, an appropriate estimator for non-normally distributed ordinal data was selected between a diagonally weighted least squares (DWLS) or an unweighted least squares (ULS) estimator as the literature suggested that these two are more reliable estimators than classic Maximum Likelihood Robust (MLR) approach for our type of data (Li, 2016; Mindrila, 2010).

Regarding the model fit the factor loading was considered acceptable when at least .6 and optimal at .7 and above (Hair et al., 2010). Model fit was established by looking at multiple criteria including (Cole, 1987; Hu & Bentler, 1999): the ratio between Chi-Square and degrees of Freedom below 3, the Comparative Fit Index (CFI) aiming for a value of .90 or higher. The root mean squared error approximation (RMSEA) aiming for values less than .07; the Standardized Root Mean Square Residual (SRMR) looking for a value below .08. The R package ‘ggplot’ was used to visualize the overall model. Cronbach’s Alpha was calculated for the overall scale and per each factor of the BUS 11. The minimum acceptable value for Cronbach’s α was considered to be ≥ .70 (α ≥ .9 = excellent; .9 > α ≥ .8 = good; .8 > α ≥ .7 = acceptable); below this value, the internal consistency was considered low (.7 > α ≥ .6 = questionable; .6 > α ≥ .5 = poor; .5 > α = unacceptable) (Tavakol & Dennick, 2011).

To establish convergent validity a Kendall tau correlation analysis was performed between BUS and UMUX-LITE. Finally, as suggested by previous validation studies in other languages (Borsci, Schmettow, et al., 2022), we tested by regression the potential effect of age on satisfaction with chatbots.

3.1. Control of participants' interaction

Out of 557 observations, only in 75 cases users decided to not answer the control questions, in 242 cases participants answered correctly, and in 240 cases they answered incorrectly, we decided to aggregate missed and incorrect answers: thus only 43.4% of our participants demonstrated a certain level of attention to the conversational exchange with chatbots till the end of the conversation. Despite that, the regression analysis suggests no significant differences between people who answered correctly or not correctly to the control questions in terms of satisfaction. This might indicate that even if participants did not remember how the chatbots presented their greetings, the users were able to reliably assess their experience. Therefore, we decided to keep all the observations, however, when performing the CFA we decided to also control the fit with the expected model separately for people who answered correctly or incorrectly to the check questions.

For 55 observation participants failed to achieve the goal but felt that they could assess the quality of the chatbots anyway. In such cases, the aggregate average satisfaction of participants was 68.6% (SD: 15.5%) compared to the average of the remaining 499 which was 77.8% (SD: 15.4%). Despite the 9.2% difference, a generalized linear model suggested that the difference between the satisfaction of people who achieved or failed the task with the chatbot was not significantly different.

3.2. Satisfaction of participants per each chatbot

Descriptive statistics of satisfaction of participants per chatbot were given in Table 2. The participants’ satisfaction with both TBUS and UMUX-LITE chatbots was generally high. According to the findings, Chatbot 1 received the highest level of satisfaction (TBUS = 80.2%, UMUX-LITE = 80.14%), whereas Chatbot 4 had a lower satisfaction rating overall (TBUS = 70.4%, UMUX-LITE = 69.86%). The regression analysis clarified that there is a significant effect on the satisfaction rated by participants due to the chatbots (F(6,550) = 3.5; p = .002). More specifically, compared to Chatbot 1, both Chatbot 5 (p = .04) and Chatbot 4 (p < .001) were ranked significantly less satisfactory by users in their interactions with them.

Table 2

Descriptive statistics for each chatbot
Chatbot	Number of observations	TBUS		UMUX-LITE
		M	SD	M	SD
1	70	80.2%	13.9%	80.1%	20.5%
2	86	79.3%	14.4%	80.1%	18.1%
3	45	77.3%	16.6%	77.5%	21.8%
4	72	70.4%	17.9%	69.8%	18.8%
5	95	74.5%	14.4%	76.3%	16%
6	128	78.2%	16.3%	79.3%	17.9%
7	61	77.8%	14.8%	81.3%	17.1%

[Insert Table 2 Here]

3.3. Confirmatory factorial analysis of the Turkish adapted version of BUS

The expected five factors of the BUS-11 identified in previous validation studies (Borsci, Prati, et al., 2022; Borsci, Schmettow, et al., 2022) were confirmed in the present experiment (see Table 3, Model 1) in particular the model resulted very solid even when participants incorrectly answered the check questions (Table 3, Model 2 incorrect). The best fit was of course obtained by considering the 242 observations of the participants who correctly answered the control questions (Table 3, Model 3).

Table 3

Fit measures of the data from the confirmatory factorial analysis*
Model	ChiSquare ratio	CFI	RMSEA	SRMR
Model 1 (all data)	2.14	.996	.045	.040
Model 2 (incorrect)	1.38	.997	.035	.049
Model 3 (correct)	1.07	.999	.017	.039
*Three models were tested: Model 1 with all the data, Model 2 with only data of people who incorrectly answered the control questions and Model 3 with the data of only those participants who answered correctly to the control questions.

[Insert Table 3 Here]

Despite the fit of Model 3 is certainly superior, it is reassuring that also in the case of participants who gave minimal attention to the task (Model 2), the scale remains solid and the fit with the data is excellent. Figure 1 represents Model 1 showing the relationship between items and factors. In line with previous studies a correlation of .940 between Factor 2 (Perceived quality of chatbot functions) and Factor 3 (Perceived quality of conversation and information provided).

3.4. Reliability, convergent validity of the adapted scale, and effects of individual characteristics

The scale seems to be very reliable with a Cronbach’s Alpha of .915. Moreover, in line with validation studies in other languages, the TBUS is strongly correlated to UMUX-LITE (Rt = .69, p < .001, see Fig. 2).

Users’ characteristics do not seem to affect the level of satisfaction measured by the TBUS. Specifically, despite a not significant trend (see Fig. 3), the age of users does not seem to affect their rating of satisfaction in our population.

Based on the TBUS scores, participants perceived the chatbots tested to be satisfactory with an average score ranging from 70% to a maximum of 80.2%. The TBUS-11 scale aligns with our expected results and well-coincides with the five original BUS-11 structures. Consequently, it provides Turkish users a reliable and standardized way of assessing their satisfaction level when interacting with chatbots. The TBUS scale aims to assess the different aspects that contribute to the conversational system’s quality, including Accessibility (F1, average loading of items: 0.64), Perceived quality of chatbot functions (F2, average loading of items: 0.84), Perceived quality of conversation and information providing (F3, average loading of items: 0.82), Perceived privacy and security (F4), and Time Response (F5). In line with findings from previous validation studies (Borsci, Prati, et al., 2022; Borsci, Schmettow, et al., 2022), there exists a robust correlation between Factor 2 and 3 on the TBUS scale. This indicates that the participants had difficulty distinguishing between factors related to functionality quality (items 3–5) and those associated with conversation and information quality (items 6–9).

The analysis suggested that the Turkish version of the scale is reliable even when participants misbehave or miss to achieve the expected goal of the task for any reason. Specifically, the TBUS analysis showed that these participants did not only experience on average a less satisfactory experience than those who paid attention and achieved the goals, but (more importantly) the data appropriately fit the expected model with the five factors structure. This is important because developers can consider quite reliable the insights from the TBUS-11 even when users were not able (for different reasons) to achieve the expected outcome.

There is an increasing interest in identifying modalities to assess chatbots and conversational agents in terms of the quality of interaction with humans (Mariani et al., 2023; Ren et al., 2018). The BUS scale is currently the only standardized scale available in multiple languages that can be reliably used to assess the satisfaction of people with different chatbots and to compare chatbots' quality in tasks of information retrieval. We proposed and validated, by involving Turkish-speaking chatbots and users, the Turkish adaptation of the BUS-11 scale. Our goal was to contribute to the domain of conversational agents in Turkey and to facilitate the implementation of high-quality Turkish chatbots by providing a relatively easy-to-use benchmarking tool. The TBUS-11 scale is a valid and reliable tool for measuring user satisfaction with chatbots. The scale is effective in capturing the essential aspects of satisfaction with chatbots and it is perfectly in line with the other versions of the BUS scale validated in other languages. The possibility to access a multilanguage scale such as the TBUS could potentially allow operators in academia and industry to assess their products against national and international systems. This may also in the long run feed cross-cultural comparisons of usage and satisfaction with conversational systems.

Future studies should explore additional factors that influence user satisfaction with chatbots, particularly for users who aim to use the chatbot but are unable to retrieve information or accomplish their intended tasks. The present study also did not investigate other potential aspects affect people’s experience with chatbots such as design patterns, trustworthiness etc. (Diederich et al., 2022; Seeger et al., 2021; Seeger & Heinzl, 2017). Although our current study did not detect a significant impact of age on satisfaction, it is important to recognize that our participant pool consisted primarily of cooperative undergraduate students who voluntarily participated. Previous research identified an effect of age on participants’ answers to the BUS scale (Borsci, Schmettow, et al., 2022) therefore future studies should compensate for that and also test satisfaction with a cohort more representative of people from different ages. In fact, technology usage can vary across different age groups in Turkey, and literature suggests that elderly people in Turkey may have different needs and expectations when using technological artifacts (Guner & Acarturk, 2020).

It is recommended to involve participants from different age groups and backgrounds in order to obtain a more representative sample of the population. Moreover, given the fast-paced changes in technology and user expectations, continued research must be conducted to update the scale as needed to ensure its relevance.

Author Contribution

All Authors: Collaboratively conducted the comprehensive data analysis and prepared the result tables. All authors were involved in the study's conceptualization, provided critical revisions, and approved the final version of the manuscript.S.B.: Responsible for preparing Figures 1-3, conducted data analysis, contributed to manuscript writing, and participated in the review process.N.B.A.: Conducted the survey, contributed to data collection and initial data analysis, and participated in manuscript writing and review.B.S.: Conducted the survey, contributed to data collection and initial data analysis, and participated in manuscript writing and review.

Declaration of Interest Statement

The authors report there are no competing interests to declare.

Adam, M., Wessel, M., & Benlian, A. (2021). AI-based chatbots in customer service and their effects on user compliance. Electronic Markets, 31(2), 427–445. https://doi.org/https://doi.org/10.1007/s12525-020-00414-7
Akıl, S., & Ungan, M. C. (2021). E-Commerce Logistics Service Quality. Journal of Electronic Commerce in Organizations, 20(1), 1–19. https://doi.org/10.4018/jeco.292473
Berkman M.İ., S., & Şahin, Ş. (2022). Exploring Usability as a Formative Construct through UMUX: A Multi-Language Approach for Turkish Adaptation. International Journal of Human-Computer Interaction, 1–25. https://doi.org/10.1080/10447318.2022.2121049
Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015). Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience. International Journal of Human-Computer Interaction, 31(8), 484–495. https://doi.org/10.1080/10447318.2015.1064648
Borsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022). The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents. Personal and Ubiquitous Computing, 26(1), 95–119. https://doi.org/10.1007/s00779-021-01582-9
Borsci, S., Prati, E., Federici, S., Malizia, A., & Schmettow, M. (2022). “Ciao AI”: The Italian adaptation and validation of the Chatbot Usability Scale. https://doi.org/10.31234/osf.io/3hcgy
Borsci, S., Schmettow, M., Malizia, A., Chamberlain, A., & van der Velde, F. (2022). A Confirmatory Factorial Analysis of the Chatbot Usability Scale: A Nultilanguage Validation. Personal and Ubiquitous Computing, 0123456789. https://doi.org/10.1007/s00779-022-01690-0
Brooke, J. (1996). SUS: A “Quick and Dirty” Usability Scale. In Usability Evaluation In Industry (1st ed., pp. 207–212). CRC Press. https://doi.org/10.1201/9781498710411-35
Cole, D. A. (1987). Methodological contributions to clinical research utility of confirmatory factor analysis in test validation research. Journal of Consulting and Clinical Psychology, 55(4), 584–594. https://doi.org/https://doi.org/10.1037/0022-006X.55.4.584
Diederich, S., Brendel, A. B., Morana, S., & Kolbe, L. (2022). On the Design of and Interaction with Conversational Agents: An Organizing and Assessing Review of Human-Computer Interaction Research. Journal of the Association for Information Systems, 23(1), 96–138. https://doi.org/10.17705/1jais.00724
Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323–327. https://doi.org/10.1016/j.intcom.2010.04.004
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the New World of HCI. ACM INTERACTIONS, 38–42. https://interactions.acm.org/archive/view/july-august-2017/chatbots-and-the-new-world-of-hci
Guner, H., & Acarturk, C. (2020). The Use and Acceptance of ICT by Senior Citizens: A Comparison of Technology Acceptance Model (TAM) for Elderly and Young Adults. Universal Access in the Information Society, 19(2), 311–330. https://doi.org/10.1007/s10209-018-0642-4
Hair, J., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate Data Analysis (7th ed.). Pearson Prentice Hall, Upper Saddle River, New Jersey
Hone, K. S., & Graham, R. (2000). Towards A Tool for the Subjective Assessment of Speech System Interfaces (SASSI). In Natural Language Engineering (Vol. 6, Issues 3–4). https://doi.org/10.1017/s1351324900002497
Hu, L., & Bentler, P. M. (1999). Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/https://doi.org/10.1080/10705519909540118
İçgözü, T. (2020). Covid-19 etkisiyle chatbot kullanımı bankacılıkta 5 kat, e-ticarette 2 kat arttı. Webrazzi. Retrieved from https://webrazzi.com/2020/05/20/pandemi-etkisiyle-chatbot-kullanimi-bankacilikta-5-kat-e-ticarette-2-kat-artti/ Accessed May 18, 2023
Jenneboer, L., Herrando, C., & Constantinides, E. (2022). The Impact of Chatbots on Customer Loyalty: A Systematic Literature Review. Journal of Theoretical and Applied Electronic Commerce Research, 17(1), 212–229. https://doi.org/10.3390/jtaer17010011
Lewis, J. R. (2016). Standardized Questionnaires for Voice Interaction Design. Voice Interaction Design, 1(1), 1–16. http://www.avixd.org.
Lewis, J. R., & Sauro, J. (2020). Three Questionnaires for Measuring Voice Interaction Experiences. MeasuringU.
Lewis, J. R., Utesch, B. S., & Maher, D. E. (April, 2013). UMUX-LITE - When there’s no time for the SUS. CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2099–2102, Paris, France https://doi.org/10.1145/2470654.2481287
Li, C. H. (2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21(3), 369–387. https://doi.org/10.1037/met0000093
Mariani, M. M., Hashemi, N., & Wirtz, J. (2023). Artificial Intelligence Empowered Conversational Agents: A systematic Literature Review and Research Agenda. Journal of Business Research, 161(March), 113838. https://doi.org/10.1016/j.jbusres.2023.113838
Mindrila, D. (2010). Maximum Likelihood (ML) and Diagonally Weighted Least Squares (DWLS) Estimation Procedures: A Comparison of Estimation Bias with Ordinal and Multivariate Non-Normal Data. International Journal for Digital Society, 1(1), 60–66. https://doi.org/10.20533/ijds.2040.2570.2010.0010
Polkosky, M. D. (2005). Toward a Social-Cognitive Psychology of Speech Technology: Affective Responses to Speech-Based e-Service [University of South Florida]. In Digital Commons. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=psyc4&NEWS=N&AN=2005-99018-095
Press, G. (2019). AI Stats News: Chatbots Increase Sales By 67% But 87% Of Consumers Prefer Humans. Forbes. Retrieved from https://www.forbes.com/sites/gilpress/2019/11/25/ai-stats-news-chatbots-increase-sales-by-67-but-87-of-consumers-prefer-humans/?sh=6477b17d48a3 Accessed May 5, 2023
Ren, Q., Li, S., Song, B., & Chen, C. (2018, December). The application of bounded online gradient descent algorithms for kernel based online learning in tourist number forecasting. Proceedings of the International Conference on Electronic Business (ICEB), Guilin, China, 800–804.
Salza, P. L., Foti, E., Nebbia, L., & Oreglia, M. (1996). MOS and Pair Comparison Combined Methods for Quality Evaluation of Text-to-Speech Systems. Acta Acustica United with Acustica, 82(4), 650–656.
Seeger, A.-M., & Heinzl, A. (2017). Human versus machine: Contingency factors of anthropomorphism as a trust-inducing design strategy for conversational agents. In Lecture Notes in Information Systems and Organisation (Vol. 25, pp. 129–139). Springer Heidelberg. https://doi.org/10.1007/978-3-319-67431-5_15
Seeger, A.-M., Pfeiffer, J., & Heinzl, A. (2021). Texting with Human-like Conversational Agents: Designing for Anthropomorphism. Journal of the Association for Information Systems., 22(4). https://doi.org/10.17705/1jais.00685
Tavakol, M., & Dennick, R. (2011). Making Sense of Cronbach’s Alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Taylor, M. P., Girard, S., Jacobs, K., Buvat, J., Subrahmanyam, K., Puttur, R., Shah, H., & B., A. (2020). Smart Talk: How Organizations and Consumers are Embracing Voice and Chat Assistants. In Capgemini. https://www.capgemini.com/wp-content/uploads/2019/09/Report_Conversational-Interfaces-1.pdf

No competing interests reported.

Appendix.docx

Download PDF

Reviewers agreed at journal
18 Jul, 2024
Reviews received at journal
18 Jul, 2024
Reviewers agreed at journal
17 Jul, 2024
Reviewers invited by journal
16 Jul, 2024
Editor assigned by journal
16 Jul, 2024
Submission checks completed at journal
05 Jul, 2024
First submitted to journal
03 Jul, 2024

You are reading this latest preprint version

Unleashing the Potential of Turkish Chatbots: A Study on the Validity and Reliability of the Bot Usability Scale

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusion

Figures

1. Introduction

2. Method

2.1 Participants

2.2. Instruments and study procedure

2.3. Ethical Consent

2.4. Data Analysis

3. Findings

3.1. Control of participants' interaction

3.2. Satisfaction of participants per each chatbot

3.3. Confirmatory factorial analysis of the Turkish adapted version of BUS

3.4. Reliability, convergent validity of the adapted scale, and effects of individual characteristics

4. Discussion

5. Conclusion

Declarations

Author Contribution

References

Additional Declarations

Supplementary Files

Status:

Version 1