Understanding User comfort and Expectations in AI-based Systems

doi:10.21203/rs.3.rs-3135320/v1

Download PDF

Research Article

Understanding User comfort and Expectations in AI-based Systems

https://doi.org/10.21203/rs.3.rs-3135320/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

This study investigated the impact of prior experience and education levels on user expectations in Artificial Intelligence (AI) based systems. The research aimed to determine whether these factors, individually or interactively, significantly influenced user expectations. Moreover, the effects of system interface, system feedback and system responsiveness on user comfort in AI-based systems were determined as well. The findings highlighted the importance of prior experience in shaping user expectations. It also suggests that educational level may have limited influence on user expectations. The choice of system interface and the responsiveness of the AI-based system significantly impact user comfort. The findings suggest for the creation of more user-friendly and comfortable interfaces. Understanding the various factors that influence user comfort and expectation, can aid the design and development of AI systems tailored to user backgrounds that better meet user needs and enhance their overall experience.

Artificial intelligence (AI)

user comfort

user expectations

system interface

system feedback

system responsiveness

Artificial Intelligence (AI) has revolutionized various aspects of our lives, ranging from communication and entertainment to healthcare and transportation. By simulating human intellect and carrying out operations that ordinarily call for human cognitive operations, AI has emerged as a powerful technology that holds immense potential for improving efficiency, accuracy, and decision-making in numerous domains. As a result, AI has gained significant popularity and is being widely adopted across industries and sectors.

One of the key reasons for the growing use and popularity of AI is its ability to process and analyse large volumes of data at incredible speeds. With the advent of big data and the proliferation of digital platforms, organizations are faced with an overwhelming amount of information. AI algorithms and machine learning techniques enable businesses to extract valuable insights, identify patterns, and make data-driven decisions that were previously unattainable. These advancements have paved the way for applications like virtual assistants, chatbots, recommendation systems, autonomous vehicles, and medical diagnostics, which have become an integral part of our daily lives. From streamlining manufacturing processes to optimizing supply chain management, AI technologies are enabling organizations to achieve higher levels of productivity and competitiveness. The popularity of AI is also driven by its ability to improve user experiences. Personalized recommendations, intelligent virtual assistants, and voice-activated devices have become commonplace, offering users convenient and tailored solutions. AI-powered systems are continuously learning and adapting to user preferences, providing a seamless and intuitive user experience.

Moreover, AI has opened new frontiers in scientific research and exploration. It has enabled breakthroughs in areas like drug discovery, genomics, climate modelling, and space exploration, by processing and analysing vast amounts of complex data to uncover insights and accelerate scientific advancements. Despite the numerous benefits and opportunities offered by AI, it is important to address ethical considerations and ensure responsible and unbiased AI development and deployment. Issues such as algorithmic bias, privacy concerns, and job displacement require careful attention and regulation to mitigate any potential negative impacts. The use and popularity of AI continue to grow rapidly, driven by its ability to process large volumes of data, automate tasks, enhance user experiences, and drive scientific discoveries. AI-driven language processing systems have emerged as powerful tools for understanding and processing human language. These systems employ methods like natural language processing (NLP) and machine learning to extract meaning, identify patterns, and generate coherent and contextually appropriate responses. Through the analysis of vast amounts of textual data, these systems can recognize entities, sentiments, and relationships within language, enabling them to provide valuable insights and automate various language-related tasks.

One key application of AI in language processing is in the development of chatbots and virtual assistants. These intelligent conversational agents can engage in human-like interactions, understanding user queries, and delivering relevant and accurate responses. They can assist users in tasks such as answering questions, providing recommendations, and performing simple transactions, enhancing customer service experiences and increasing operational efficiency. AI-driven language processing systems also play a significant role in machine translation, enabling the automatic translation of text from one language to another. Using sophisticated algorithms and neural networks, these systems have improved translation accuracy and fluency, making multilingual communication more accessible and convenient.

Moreover, AI models have been employed for sentiment analysis, allowing businesses to gauge public opinion, customer satisfaction, and brand sentiment by analysing large volumes of textual data from social media, customer reviews, and surveys. This helps organizations gain valuable insights into customer preferences, market trends, and brand reputation, empowering them to make data-driven decisions. Despite their advancements, AI-driven language processing systems still face challenges, such as understanding context, handling ambiguous language, and addressing biases present in training data. Ongoing research and development efforts focus on improving these systems' performance, accuracy, and ethical considerations to ensure fair and inclusive language processing.

AI-driven language processing systems have transformed the way we interact with technology and communicate with each other. They continue to advance the boundaries of language understanding, enabling more efficient and effective human-computer interactions and opening new possibilities in various industries and applications. AI-driven language processing systems leverage advanced algorithms and machine learning techniques to analyse, understand, and generate human language. By continuously learning from large datasets, AI models can improve their accuracy and performance over time, enabling them to provide more precise and contextually relevant responses. AI-driven language processing systems have the potential to revolutionize communication, streamline information retrieval, and enhance user experiences across a wide range of domains, including customer service, education, healthcare, and content generation.

A chatbots request is more likely to be complied with by users if the chatbot is made to appear more human-like and if the request is made in a way that makes the user feel like they have already committed to complying (Adam et al. 2021). The findings of this study carry implications for the development of chatbots and other forms of artificial intelligence. By making AI systems more socially present and by using techniques such as the foot-in-the-door technique, designers can increase the probability of users adhering to their requests. The considerable potential of incorporating human-like cues in chatbot applications is directly linked to higher customer satisfaction. This relationship is influenced by factors such as enjoyment, attitude, and trust. The findings from the research suggest that in order to stay competitive, food retailers should swiftly embrace cutting-edge technologies as a component of their omnichannel programme and integrate design elements that mimic human characteristics into their chatbot systems (Klein and Martinez 2022). A study by Google AI showed that their large language model, BERT, was able to attain cutting-edge performance on various NLP tasks (Devlin et al. 2018)⁹. Even in the presence of a language model, incorporating semi-supervised learning methods can enhance both the quality and diversity of outputs (van der Lee et al. 2022). In 2019, a study by Open AI showed that their large language model, GPT-2, was able to generate realistic and coherent text, even when given prompts that were open-ended or challenging.

The utilization of NLP and AI systems has become the favoured approach for enterprise management. These technologies are crucial in comprehending customers' needs and ensuring high levels of customer satisfaction. As a result, the adoption of NLP and AI techniques has become imperative in order to accomplish business objectives effectively (Mah et al. 2022). The AI has set foot in the educational settings as well and its significance is emphasized (Nazaretsky et al. 2022). Most language learning tools that incorporated artificial intelligence (AI) predominantly relied on machine learning and natural language processing techniques. Their primary functions included error identification, feedback provision, and language proficiency assessment. Following the usage of these tools, learners exhibited improvements in their language skills and knowledge (Woo and Choi 2021)¹⁰. The integration of chatbots in the learning process has the potential to enhance student engagement. By encouraging students to ask questions, offering adequate feedback during the review stage, and fostering a relaxed and friendly interaction, chatbots contribute to students' sense of recognition and promote an improvement in their academic performance (YF. Lee et al. 2022). Nevertheless, despite the advancements in technology, customers still encounter dissatisfying experiences when interacting with AI systems. These encounters often involve inadequate responses that fail to meet the user's expectations, creating a discrepancy between what the user anticipates and his comfort. Nevertheless, Indian users demonstrate a willingness to embrace AI generated decisions. The ability of AI to shape human behaviours has been legitimized, even in the absence of sufficient evidence regarding the system's capabilities (Kapania et al. 2022).The present study aims to contribute to the broader understanding of user-centered design principles in AI-based systems. The insights gained from this research can guide future system development efforts, leading to improved user satisfaction, engagement, and overall user experiences in AI interactions.

Objectives

In order to better understand the user comfort and expectations surrounding AI-based systems, this study sets out the following objectives.

Assess user comfort about AI-based systems.
Assess user expectations about AI-based systems.

The sampling procedure for this study involved selecting participants from different education levels in private universities in the Anand district, Gujarat, India. The goal was to ensure representation of individuals from various educational backgrounds. The first step was to identify the relevant strata based on the education levels within the population that is graduate, postgraduate and doctorate. Within each stratum, participants were randomly selected to ensure that the sample represented the diversity of education levels in the population. This was done using a random number generator. Participants were approached and invited to participate in the research. Voluntary consent with adequate information was obtained, and they were provided with the necessary information about the research objectives and procedures. Data were collected through the administration of the questionnaire measuring user expectations and user comfort in AI-based systems. The questionnaire utilized a Likert scale, spanning from strongly disagree to strongly agree. The questionnaire comprised of items related to user expectations for accuracy, explanation, and overall satisfaction with the system as well as system interface, system feedback, system responsiveness, and user engagement for user comfort. The statements included in the questionnaire were developed based on an extensive literature review and consultation with experts in the field. The questionnaire was administered to the participants either in an online format or in a paper-based format, depending on the participants convenience and preference. Clear instructions were provided at the beginning of the questionnaire to guide participants on how to respond and complete the survey.

User Comfort was measured by the tool with its manifestations in the following.

Ease of Use: This dimension captures the level of ease and convenience experienced by the user while interacting with the AI-based language processing system.
Perceived Comfort: This dimension assesses the user's subjective perception of comfort during their interactions with the system.
Satisfaction: This dimension measures the extent to which the user feels satisfied with the overall user experience of the AI-based language processing system.

User expectation was measured by the tool with its manifestations in the following.

Performance Expectations: This dimension evaluates the user's expectations regarding the system's performance and accuracy in understanding and responding to their inputs.
Personalization: This dimension assesses the user's expectations for personalized and tailored interactions with the AI-based system.

These dimensions provide a comprehensive assessment of user comfort and user expectations in AI-based systems, covering aspects related to ease of use, subjective comfort, satisfaction, performance expectations, and personalization.

Factorial design

In the study examining the effects of selected factors on user comfort and user expectations in AI-based systems, a factorial design was employed. The factorial design allowed for the simultaneous investigation of multiple independent variables and their interactions, providing a comprehensive understanding of their effects on the dependent variable.

The factorial design for the first objective involved manipulating three independent variables: system interface, system feedback, and system responsiveness for user comfort. Each independent variable has multiple levels or conditions. The system interface had two levels (text-based and voice-based), system feedback had two levels (no feedback and minimal feedback), and system responsiveness had three levels (slow, moderate, and quick). In this case, a 2x2x3 factorial design was used, resulting in a total of 12 unique conditions or combinations. The participants in the study were allocated in a random manner to one of the 12 conditions, ensuring an equal distribution of participants across each combination of independent variable levels. This random assignment helped control for potential confounding variables and contributes to the internal validity of the study.

The second objective, examining the effects of prior experience and education levels on user expectations in AI-based systems, the factorial design involved manipulating two independent variables: prior experience and education levels. The prior experience had two levels (with prior experience and without prior experience), and education levels had three levels (Graduate, Postgraduate and Doctorate). In this case, a 2x3 factorial design was employed, resulting in a total of 6 unique conditions or combinations. The participants were randomly allocated to one of the 6 conditions, ensuring an equal distribution of participants across each combination of independent variable levels.

Findings

The findings of the study were categorised objective-wise for clearer perspective.

Objective 1: Assess User Comfort about AI-based Systems

To fulfil the first objective, the study examined the main and interactive effects of system interface, system feedback and system responsiveness on user comfort in AI-based systems. By analysing the data using a three-way ANOVA, the aim was to determine if, system interface, system feedback and system responsiveness or their interaction significantly influence user comfort.

Table1: Descriptive statistics (User comfort)

System interface	System feedback	System responsiveness	Mean	Std. Deviation	N
Text-based	No feedback	Slow response	2.27	1.069	33
		Moderate response	3.00	1.107	32
		Quick response	2.57	1.089	14
		Total	2.62	1.124	79
	Minimal feedback	Slow response	2.63	.824	24
		Moderate response	2.33	.985	12
		Quick response	3.00	.894	11
		Total	2.64	.895	47
	Total	Slow response	2.42	.981	57
		Moderate response	2.82	1.105	44
		Quick response	2.76	1.012	25
		Total	2.63	1.041	126
Voice-based	No feedback	Slow response	2.75	.775	16
		Moderate response	3.61	.979	18
		Quick response	3.69	.855	13
		Total	3.34	.962	47
	Minimal feedback	Slow response	2.55	1.214	11
		Moderate response	3.55	.858	22
		Quick response	4.15	.631	46
		Total	3.76	.964	79
	Total	Slow response	2.67	.961	27
		Moderate response	3.58	.903	40
		Quick response	4.05	.705	59
		Total	3.60	.980	126
Total	No feedback	Slow response	2.43	1.000	49
		Moderate response	3.22	1.093	50
		Quick response	3.11	1.121	27
		Total	2.89	1.119	126
	Minimal feedback	Slow response	2.60	.946	35
		Moderate response	3.12	1.066	34
		Quick response	3.93	.821	57
		Total	3.34	1.082	126
	Total	Slow response	2.50	.976	84
		Moderate response	3.18	1.077	84
		Quick response	3.67	.998	84
		Total	3.12	1.121	252

Table 2: Tests of Between-Subjects Effects (User comfort)

Source	Type III Sum of Squares	Df	Mean Square	F	Sig.	Partial Eta Squared
Corrected Model	109.233^a	11	9.930	11.545	.000	.346
Intercept	1856.395	1	1856.395	2158.292	.000	.900
System interface	28.771	1	28.771	33.449	.000	.122
System feedback	.132	1	.132	.153	.696	.001
System responsiveness	23.271	2	11.636	13.528	.000	.101
System interface* System Feedback	.008	1	.008	.009	.923	.000
System interface* System Responsiveness	8.151	2	4.075	4.738	.010	.038
System feedback* System responsiveness	5.575	2	2.787	3.241	.041	.026
System interface* System feedback* System responsiveness	3.040	2	1.520	1.767	.173	.015
Error	206.429	240	.860
Total	2761.000	252
Corrected Total	315.663	251
R Squared = .346 (Adjusted R Squared = .316)

As the table 1 indicates, the sample size for realising objective 1 involved 252 participants ensuring to eliminate the redundant cases that disturbed the study. The table 1 also indicates the mean scores that help draw conclusions regarding the significant interactions, if found during the later stage.

Inferences about the main and interactive effects were drawn from Table 2. The main effect of system interface on user comfort was found to be statistically significant at the α = 0.05 level, F(1, 11) = 33.449, p = 0.000. The effect size, measured by partial eta squared, was 0.122, indicating that approximately 12.2% of the variance in user comfort can be attributed to the system interface.

The main effect of system feedback on user comfort was not found to be statistically significant at the α = 0.05 level, F(1, 11) = 0.153, p = 0.696. The effect size, measured by partial eta squared, was 0.001, indicating that only approximately 0.1% of the variance in user comfort can be attributed to the system feedback.

The main effect of system responsiveness on user comfort was found to be statistically significant at the α = 0.05 level, F(2, 11) = 13.528, p = 0.000. The effect size, measured by partial eta squared, was 0.101, indicating that approximately 10.1% of the variance in user comfort can be attributed to the system responsiveness.

The interactive effect of system interface and system feedback on user comfort was not found to be statistically significant at the α = 0.05 level, F(1, 11) = 0.009, p = 0.923. The effect size, measured by partial eta squared, was 0.000, indicating that the interaction between system interface and system feedback explains very little variance in user comfort, less than 0.1%.

The interactive effect of system interface and system responsiveness on user comfort was found to be statistically significant at the α = 0.05 level, F(2, 11) = 4.738, p = 0.010. The effect size, measured by partial eta squared, was 0.038, indicating that the interaction between system interface and system responsiveness explains approximately 3.8% of the variance in user comfort.

The interactive effect of system feedback and system responsiveness on user comfort was found to be statistically significant at the α = 0.05 level, F (2, 11) = 3.241, p = 0.041. The effect size, measured by partial eta squared, was 0.026, indicating that the interaction between system feedback and system responsiveness explains approximately 2.6% of the variance in user comfort.

The interactive effect of system interface, system feedback, and system responsiveness on user comfort was not found to be statistically significant at the α = 0.05 level, F (2, 11) = 1.767, p = 0.173. The effect size, measured by partial eta squared, was 0.015, indicating that the interaction between system interface, system feedback, and system responsiveness explains approximately 1.5% of the variance in user comfort.

Table 3: Pairwise comparisons of system interface

(I)System interface	(J)System interface	Mean Difference (I-J)	Std. Error	Sig.^b	95% Confidence Interval for Difference^b
					Lower Bound	Upper Bound
Text-based	Voice-based	-.749^*	.130	.000	-1.004	-.494
Voice-based	Text-based	.749^*	.130	.000	.494	1.004
Based on estimated marginal means
*. The mean difference is significant at the .05 level.

The table 3 helped in the pairwise comparisons of system interface, which revealed a significant mean difference between the text-based and voice-based system interfaces (M = -0.749, p = 0.000, α = 0.05). This indicates that users rated the text-based system interface 0.749 units lower than the voice-based interface. Conversely, the mean difference between the voice-based and text-based system interfaces was 0.749, indicating that users rated the voice-based interface 0.749 units higher than the text-based interface (p = 0.000, α = 0.05). These findings suggest that there are significant differences in user comfort based on the type of system interface utilized.

Table 4: Pairwise comparisons of System responsiveness

(I)System responsiveness	(J)System responsiveness	Mean Difference (I-J)	Std. Error	Sig.^b	95% Confidence Interval for Difference^b
(I)System responsiveness	(J)System responsiveness	Mean Difference (I-J)	Std. Error	Sig.^b	Lower Bound	Upper Bound
Slow response	Moderate response	-.574^*	.154	.000	-.877	-.271
Slow response	Quick response	-.806^*	.162	.000	-1.124	-.487
Moderate response	Slow response	.574^*	.154	.000	.271	.877
Moderate response	Quick response	-.232	.160	.149	-.547	.084
Quick response	Slow response	.806^*	.162	.000	.487	1.124
Quick response	Moderate response	.232	.160	.149	-.084	.547
Based on estimated marginal means
*. The mean difference is significant at the .05 level.

The table 4 helped in the pairwise comparisons of system responsiveness which revealed a significant mean difference between the slow response and moderate response conditions (M = -0.574, p = 0.000, α = 0.05). This indicates that users rated the slow response condition 0.574 units lower than the moderate response condition. The pairwise comparisons revealed a significant mean difference between the slow response and quick response conditions (M = -0.806, p = 0.000, α = 0.05). This indicates that users rated the slow response condition 0.806 units lower than the quick response condition. However, there was no significant mean difference between the moderate response and quick response conditions (M = -0.232, p = 0.149, α = 0.05). This suggests that there is no difference in user ratings between the moderate response and quick response conditions.

Table 5: Post hoc test (User comfort)

Tukey HSD
(I)System responsiveness	(J)System responsiveness	Mean Difference (I-J)	Std. Error	Sig.	95% Confidence Interval
(I)System responsiveness	(J)System responsiveness	Mean Difference (I-J)	Std. Error	Sig.	Lower Bound	Upper Bound
Slow response	Moderate response	-.68^*	.143	.000	-1.02	-.34
Slow response	Quick response	-1.17^*	.143	.000	-1.50	-.83
Moderate response	Slow response	.68^*	.143	.000	.34	1.02
Moderate response	Quick response	-.49^*	.143	.002	-.83	-.15
Quick response	Slow response	1.17^*	.143	.000	.83	1.50
Quick response	Moderate response	.49^*	.143	.002	.15	.83
Based on observed means. The error term is Mean Square (Error) = .860.
*. The mean difference is significant at the .05 level.

In the current study, as indicated by Table 5, Tukey's HSD post hoc test was conducted to examine pairwise differences between the response conditions (slow, moderate, and quick). The pairwise comparisons using Tukey's Honestly Significant Difference (HSD) post hoc test revealed the following:

The mean difference between the slow response and moderate response conditions was -0.574 units, indicating that users rated the slow response condition significantly lower than the moderate response condition (p < 0.001, α = 0.05).
The mean difference between the slow response and quick response conditions was -1.17 units, indicating that users rated the slow response condition significantly lower than the quick response condition (p < 0.001, α = 0.05).
The mean difference between the moderate response and quick response conditions was -0.49 units, suggesting that users rated the moderate response condition significantly lower than the quick response condition (p = 0.002, α = 0.05).

All reported p-values were below the chosen significance level of 0.05, indicating statistical significance in the observed differences between the respective response conditions.

The reason for conducting the post hoc test is to identify which specific pairs of response conditions significantly differ from each other in terms of user comfort. The post hoc test allows for a more detailed analysis of pairwise differences, providing insights into the specific comparisons that contribute to the overall significant result in the ANOVA. The differences in the results of the pairwise comparisons obtained through Tukey's HSD post hoc test compared to the initial pairwise comparisons are due to the adjustment made by the post hoc test to control for Type I error inflation. The post hoc test adjusts the significance levels for multiple comparisons, resulting in more stringent criteria for determining statistical significance.

Objective 2: Assess User expectations about AI-based Systems

To fulfil the second objective, the study examined the main and interactive effects of prior experience and education levels on user expectations in AI-based systems. By analysing the data using a two-way ANOVA, the aim was to determine if prior experience, education levels, or their interaction significantly influence user expectations.

Table 6: Descriptive Statistics (User Expectations)

Prior Experience	Education level	Mean	Std. Deviation	N
With prior experience	Graduate	3.34	.815	47
	Postgraduate	3.63	.755	49
	Doctorate	3.57	.788	23
	Total	3.50	.791	119
Without prior experience	Graduate	2.83	1.264	52
	Postgraduate	2.70	1.282	50
	Doctorate	2.24	1.091	17
	Total	2.69	1.254	119
Total	Graduate	3.07	1.100	99
	Postgraduate	3.16	1.149	99
	Doctorate	3.00	1.132	40
	Total	3.10	1.123	238

Table 7: Tests of Between-Subjects Effects (User Expectations)

Source	Type III Sum of Squares	df	Mean Square	F	Sig.	Partial Eta Squared
Corrected Model	46.183^a	5	9.237	8.484	.000	.155
Intercept	1827.932	1	1827.932	1678.899	.000	.879
Prior Experience	42.063	1	42.063	38.633	.000	.143
Education level	1.987	2	.994	.913	.403	.008
Prior Experience * Education level	5.174	2	2.587	2.376	.095	.020
Error	252.594	232	1.089
Total	2581.000	238
Corrected Total	298.777	237
a. R Squared = .155 (Adjusted R Squared = .136)

As the table 6 indicates, the sample size for realising objective 2 involved 238 participants ensuring to eliminate the redundant cases that disturbed the study as well as eliminating cases using SPSS randomly that contributed to the large variance in sample size in each stratum, which contributes to the limitation of this study. The table 6 also indicates the mean scores that help draw conclusions regarding the significant interactions, if found during the later stage.

Inferences about the main and interactive effects were drawn from Table 7. The main effect of prior experience on user expectations was found to be statistically significant, F (1, 5) = 1678.899, p < 0.001. The effect size, indicated by partial eta squared, was large, accounting for 87.9% of the variance in user expectations. This suggests that prior experience has a substantial influence on user expectations in the context of the study.

The main effect of education level on user expectations was not found to be statistically significant, F (2, 5) = 0.913, p = 0.403. The effect size, as indicated by partial eta squared, was small, accounting for 0.8% of the variance in user expectations. This suggests that education level may not have a significant impact on user expectations in the context of the study.

The interaction effect of prior experience and education level on user expectations was not found to be statistically significant, F (2, 5) = 2.376, p = 0.095. The effect size, as indicated by partial eta squared, was small, accounting for 2% of the variance in user expectations. This suggests that the combination of prior experience and education level may not have a significant joint impact on user expectations in the context of the study.

Table 8: Pairwise comparisons (User Expectations)


(I)Prior Experience	(J)Prior Experience	Mean Difference (I-J)	Std. Error	Sig.^b	95% Confidence Interval for Difference^b
					Lower Bound	Upper Bound
With prior experience	Without prior experience	.925^*	.149	.000	.632	1.219
Without prior experience	With prior experience	-.925^*	.149	.000	-1.219	-.632
Based on estimated marginal means
*. The mean difference is significant at the .05 level.

The table 8 helped in the pairwise comparisons of system interface, which revealed a significant mean difference in user expectations between individuals with prior experience and those without prior experience (mean difference = 0.925, p = 0.000, α = 0.05). This indicates that individuals with prior experience had higher expectations compared to those without prior experience.

Post hoc tests are typically conducted when there are three or more levels of a categorical independent variable. Since, there were only two levels of prior experience, which means there were no additional pairwise comparisons to be made beyond comparing the two groups directly.

Solangi et al. (2018) examined the utilization of Natural Language Processing (NLP) techniques for the purpose of analyzing opinions and sentiments. The present study facilitates the same by analysing the factors affecting user comfort and user expectations. AI's implementation in academia and education encompasses various aspects such as providing assistance and constructive feedback for learning (Alqahtani et al. 2023). The present research supports the above by providing evidence of how the system responsiveness enables the user comfort. The development of CoAuthor as a writing "collaborator" (M. Lee et al. 2022) proved to be beneficial in drawing a conclusion about the significance of the current research which highlights how interactions, feedback, and responsiveness play a vital role in fostering user comfort. Dang et al. 2023 provided insights into the interaction between humans and generative models, contributing to our understanding of how humans and AI can effectively collaborate, moreover, research discovered that an escalation in collaborative activity positively impacted multiple aspects of user experience efforts (Feng et al. 2023), which are in consonance with the findings indicating the effect of user responsiveness and feedback on the comfort of the user.

The present study’s findings that indicated the users’ comfort in using the voice-based interface is in consonance with the results identifying the use of speech based interactive systems to be coveted and technology facilitating natural, authentic and seamless conversations (Savcheva & Foster 2018, Strayer et al. 2017, Oesterreich et al. 2023, Wang et al. 2023). The conversational quality of AI chatbots has a considerable impact on user satisfaction (Hsu & Lin 2023) is augmented by the present study’s findings stipulating the importance of feedback and responsiveness on user comfort.

The findings indicating that "browsers" and "surfers" have a significant influence on e-satisfaction (Bressolles et al. 2014) supports the present study’s findings that prior experience has an impact on user satisfaction. The results of the current study are corroborated by Chattaraman et al. 2019 which suggests that digital assistants that are socially oriented result in better social consequences experienced by older users who have a high level of proficiency in utilising the internet and require less assistance with task-related activities. This further emphasizes the influence of prior experience on user expectations. Grimes et al. 2021 indicated that the user's pre-existing expectations significantly influence their evaluation of conversational agents, surpassing the impact of the agents' actual performance, which supports the findings of the present study.

Implications

The preference for voice-based interfaces and the importance of system responsiveness highlight the need for designing AI systems that prioritize user comfort. Developers should focus on creating intuitive and user-friendly interfaces, with special attention to voice-based interactions. Additionally, optimizing system response times can significantly enhance user satisfaction and overall comfort.

Considering the impact of prior experience on user expectations, it is crucial to provide comprehensive training programs for users to familiarize themselves with AI-based systems. Training sessions can focus on improving user proficiency and knowledge, ensuring that users with prior experience are equipped to meet their higher expectations. Offering personalized user support services can assist individuals in navigating AI systems and addressing their specific concerns. Helpdesk support, FAQs, and online tutorials can provide valuable assistance to users, ensuring a smooth and comfortable experience. The study suggests the need for further research to explore additional factors that contribute to user comfort in AI systems. Future studies can investigate variables such as system aesthetics, natural language processing capabilities, and context-awareness to gain a deeper understanding of their influence on user comfort. As AI-based systems become increasingly prevalent, it is crucial to take into account the ethical ramifications of user comfort. Developers should prioritize user privacy, data security, and transparency in system operations to build trust and ensure a comfortable user experience. Additionally, investigating the lasting impacts of user expectations on user satisfaction as well as system adoption would provide valuable insights for system development and improvement.

In this study, investigation of how different aspects of AI-based systems affect user comfort were done. It was found that the type of system interface and the speed of system response have a significant impact on user comfort. Users generally preferred voice-based interfaces over text-based interfaces, finding them more comfortable to use. Additionally, they felt more at ease when the system responded quickly to their input, highlights the importance of designing AI systems with user comfort in mind. By understanding and addressing users' preferences for system interface and responsiveness, we can create more satisfying and enjoyable experiences for users.

It was found that prior experience had a significant impact on user expectations, with individuals who had prior experience having higher expectations compared to those without prior experience. However, education level did not show a significant effect on user expectations. These findings suggest that individuals who have prior experience with similar systems tend to have higher expectations when using AI-based systems. This could be because they are familiar with the capabilities and potential benefits of such systems.

It is important for system developers and designers to consider the influence of prior experience on user expectations. By understanding and accommodating these expectations, they can enhance user satisfaction and overall user experience with AI-based systems. Additionally, this research opens avenues for further exploration in understanding the nuanced factors influencing user comfort in AI systems.

The authors declare no competing interests.

Data availability

All the data gathered during this study are included in the supplementary information files.

Author contributions

Conceptualisation: Anjali Dave; Methodology: Anjali Dave; Formal analysis and investigation: Anjali Dave and Ankur Saxena; Writing-original draft preparation: Anjali Dave; Writing- review and editing: Anjali Dave; Supervision: Avdhesh Jha

Compliance with Ethical standards

No funding was received for conducting this study. The authors declare no competing interests.

Adam, M., Wessel, M., Benlian, A.: AI-based Chatbots in Customer Service and their Effects on User Compliance. Electron. Markets. 31(2), 427–445 (2021)
Alqahtani, T., Badreldin, H.A., Alrashed, M., Alshaya, A.I., Alghamdi, S.S., bin, Saleh, K., Albekairy, A.M.: The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research. Research in Social and Administrative Pharmacy, (2023). https://doi.org/10.1016/j.sapharm.2023.05.016
Bressolles, G., Durrieu, F., Senecal, S.: A consumer typology based on e-service quality and e-satisfaction. J. Retailing Consumer Serv. 21(6), 889–896 (2014)
Chattaraman, V., Kwon, W.S., Gilbert, J.E., Ross, K.: Should AI-Based, conversational digital assistants employ social-or task-oriented interaction style? A task-competency and reciprocity perspective for older adults. Comput. Hum. Behav. 90, 315–330 (2019)
Dang, H., Goller, S., Lehmann, F., Buschek, D.: Choice over control: How users write with large language models using diegetic and non-diegetic prompting. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–17. Association for Computing Machinery, New York, NY, USA (2023)
Feng, K.K., Li, T.W., Zhang, A.X.: Understanding Collaborative Practices and Tools of Professional UX Practitioners in Software Organizations. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–20. Association for Computing Machinery, New York, NY, USA (2023)
Grimes, G.M., Schuetzler, R.M., Giboney, J.S.: Mental models and expectation violations in conversational AI interactions. Decis. Support Syst. 144, 113515 (2021)
Hsu, C.L., Lin, J.C.C.: Understanding the user satisfaction and loyalty of customer service chatbots. J. Retailing Consumer Serv. 71, 103211 (2023)
https://: arxiv.org/abs/1810.04805, last accessed 2023/06/14
https://: arxiv.org/abs/2111.04455, last accessed 2023/06/14
Kapania, S., Siy, O., Clapper, G., SP, A.M., Sambasivan, N.: “Because AI is 100% right and safe”: User Attitudes and Sources of AI Authority in India. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–18. ACM New York, USA (2022). https://doi.org/10.1145/3491102.3517533
Klein, K., Martinez, L.F.: The impact of anthropomorphism on customer satisfaction in chatbot commerce: an experimental study in the food sector. Electronic Commerce Research 1–37 (2022). https://doi.org/10.1007/s10660-022-09562-8
Lee, M., Liang, P., Yang, Q.: Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–19. Association for Computing Machinery, New York, NY, USA (2022)
Lee, Y.F., Hwang, G.J., Chen, P.Y.: Impacts of an AI-based chatbot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Education Tech. Research Dev. 70(5), 1843–1865 (2022)
Mah, P.M., Skalna, I., Muzam, J.: Natural Language Processing and Artificial Intelligence for Enterprise Management in the Era of Industry 4.0. Appl. Sci. 12(18), 9207 (2022). https://doi.org/10.3390/app12189207
Nazaretsky, T., Ariely, M., Cukurova, M., Alexandron, G.: Teachers' trust in AI-powered educational technology and a professional development program to improve it. Br. J. Edu. Technol. 53(4), 914–931 (2022)
Oesterreich, T.D., Anton, E., Schuir, J., Brehm, A., Teuteberg, F.: How can I help you? Design principles for task-oriented speech dialog systems in customer service. Inform. Syst. e-Business Manage. 21(1), 37–79 (2023)
Savcheva, D., Foster, M.E.: Comparing User Responses to Limited and Flexible Interaction in a Conversational Interface. In: Proceedings of the 6th International Conference on Human-Agent Interaction, pp. 368–370. Association for Computing Machinery, New York, NY, USA (2018)
Solangi, Y.A., Solangi, Z.A., Aarain, S., Abro, A., Mallah, G.A., Shah, A.: Review on natural language processing (NLP) and its toolkits for opinion mining and sentiment analysis. In: 2018 IEEE 5th international conference on engineering technologies and applied sciences, (pp. 1–4). IEEE, Bangkok, Thailand doi: (2018). 10.1109/ICETAS.2018.8629198
Strayer, D.L., Cooper, J.M., Turrill, J., Coleman, J.R., Hopman, R.J.: The smartphone and the driver’s cognitive workload: A comparison of Apple, Google, and Microsoft’s intelligent personal assistants. Can. J. Experimental Psychology/Revue canadienne de psychologie expérimentale. 71(2), 93 (2017)
van der Lee, C., Ferreira, T.C., Emmery, C., Wiltshire, T.J., Krahmer, E.: Neural data-to-text generation based on small datasets: Comparing the added value of two semi-supervised learning approaches on top of a large language model. Comput. Linguistics, 1–56 (2022)
Wang, B., Li, G., Li, Y.: Enabling conversational interaction with mobile ui using large language models. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–17. Association for Computing Machinery, New York, NY, USA (2023)

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Understanding User comfort and Expectations in AI-based Systems

Status:

Version 1

Abstract

Introduction

Review of related literature

Methodology

Discussion

Implications

Conclusion

Statements and Declarations

References

Additional Declarations

Status:

Version 1