Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

doi:10.21203/rs.3.rs-4469443/v1

Download PDF

Research Article

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

https://doi.org/10.21203/rs.3.rs-4469443/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

This study investigates the performance of generative artificial intelligence (AI) in evaluating the acceptance of generative AI technologies within higher education guidelines, reflecting on the implications for educational policy and practice. Drawing on a dataset of guidelines from top-ranked universities, we compared generative AI evaluations with human evaluations, focusing on acceptance, performance expectancy, facilitating conditions, and perceived risk. Our study revealed a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI, suggesting that generative AI can accurately reflect human judgment in this context. Further, we found positive associations between ChatGPT-rated acceptance and performance expectancy and facilitating conditions, while a negative correlation with perceived risk. These results validate generative AI evaluation, which also extends the application of the Technology Acceptance Model and the Unified Theory of Acceptance and Use of Technology framework from individual to institutional perspectives.

Educational Psychology

artificial intelligence

higher education

guidelines

UTAUT

In Philip K. Dick’s visionary novel “Do Androids Dream of Electric Sheep?”, the line between human and artificial intelligence (AI) blurs, prompting readers to ponder the capabilities of AI. Inspired by the great work, our study explores the performance of AI in academia. Given the widespread debates on the use of generative AI such as ChatGPT at universities worldwide, we particularly focus on the performance of generative AI, in assessing the acceptance of generative AI in higher education policies. Specifically, moving beyond the novel’s speculative fiction, our study compares generative AI evaluation with human evaluation and validates it using Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework.

Performance of AI

Recent research across various disciplines demonstrates that generative AI technologies perform well in passing professional and academic exams. A growing body of medical research has examined the performance of AI in passing exams. In the United States Medical Licensing Exam (USMLE), generative AI performed at or near the passing threshold of 60% accuracy (Kung et al., 2023). Similarly, another study found that generative AI performed at a level comparable to a third-year medical student on USMLE Step 1 and Step 2 exams (Gilson et al., 2023). Further, a previous study found that generative AI achieved a 79.9% correct response rate on the Japanese Medical Licensing Exam, notably outperforming the average examinee by 17% on hard questions (Takagi et al., 2023). Furthermore, a prior study found that generative AI outperformed the Japanese medical residents on the General Medicine In-Training Examination, particularly in areas requiring detailed medical knowledge and difficult questions (Watari et al., 2023).

Another body of research on engineering and computer sciences has examined the performance of AI in passing exams. Computer science research found that generative AI achieved a score that just met the passing of a computer science exam focusing on algorithms and data structures (Bordt & von Luxburg, 2023). Engineering research found that generative AI performed well across various tasks, including theoretical questions, programming, and practical circuit design, with a cumulative grade of 73% (Elder et al., 2023). A recent study found that generative AI was capable of solving simple math problems and addressing undergraduate-level mathematics questions (Frieder et al., 2024).

Management and financial research has also examined the performance of AI in passing exams. Previous research found that in the Operations MBA final exam, generative AI offered correct answers with excellent explanations for basic questions. (Terwiesch, 2023). An experimental study found that generative AI performed exceptionally well in economics exams, scoring higher than the average college student in both microeconomics and macroeconomics tests (Geerling et al., 2023). A previous study also found that generative AI performed well in solving basic finance problems in solving basic undergraduate finance problems with an 85% accuracy rate (Yang & Stivers, 2024).

Thus, recent studies in fields such as medicine, management, and engineering, provide evidence that AI technologies exhibit impressive performance in certain task completions, but there is little research concerning the performance of AI in education policy. The opportunities and challenges of adopting AI-generated content in educational policy are widely debated, and university guidelines reflect this divide (McDonald et al., 2024; Moorhouse et al., 2023). However, quantitatively assessing the acceptance of generative AI presents a challenge. To address this gap, we investigate the performance of AI in analyzing the text of guidelines, offering a method to measure the acceptance of generative AI in higher education. Specifically, this study examines the performance of generative AI in text evaluation by exploring the following two questions. Can generative AI evaluate the acceptance of generative AI as effectively as humans? Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk?

Evaluating Acceptance of Generative AI: Generative AI and Human Perspectives

For the first question, we argue that generative AI is capable of evaluating the acceptance of generative AI as effectively as humans given the previous literature on the performance of Ai in text evaluation. An experimental study examined the performance of generative AI feedback on writing and its preference among English as a new language students (Escalante et al., 2023). The experimental group received feedback from generative AI and the control group received feedback from their human tutor. The results showed that students who received feedback from generative AI did improve their writing skills comparable to students who received feedback from their human tutors. Those students were also split fairly evenly in their preferences between generative AI and human feedback, showing that each form of feedback has its own perceived benefits. Further, another study investigates the use of generative AI for automated essay scoring in assessing TOEFL essays (Mizumoto & Eguchi, 2023). Mizumoto and Eguchi (2023) compared AI-generated scores to human benchmarks and explored the effect of incorporating linguistic features on scoring accuracy. The results revealed that generative AI could effectively score essays with a level of accuracy and reliability, especially when combined with analysis of linguistic features. Furthermore, prior research explored the performance of generative AI in supporting English learning as a foreign language teacher (Guo & Wang, 2023). Guo and Wang (2023) asked teachers to evaluate both generative AI’s feedback on student writing and human teachers’ feedback. The results showed that generative AI generated more feedback than teachers, distributing attention evenly across content, organization, and language aspects. Overall, these findings suggest that generative AI performs certain tasks including text evaluation as effectively as humans, if not more so in some aspects.

Hypothesis 1

Human-rated acceptance of generative AI is positively associated with generative AI-rated acceptance of generative AI

Validating Generative AI Evaluation: Expectancy, Conditions, and Risk

As for the second question, we argue that we can validate its evaluation by confirming its correlations with evaluations of other aspects that research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework focuses (Abdaljaleel et al., 2024; Ben Arfi et al., 2021; Davis, 1989; Polyportis & Pahos, 2024; Venkatesh et al., 2003). Venkatesh et al. (2003) developed UTAUT as a thorough synthesis of previous technology acceptance studies, by reviewing the existing models, including TAM. UTAUT includes four key constructs: performance expectancy, effort expectancy, social influence, and facilitating condition. Especially, we focus on perceived risk as well as performance expectancy and facilitating conditions. Our focus was informed by recent discourse and research articles in the respective contexts, in which universities expect generative AI to improve teaching and learning activities and prepare themselves for support and resources for it, but are concerned about academic integrity (Abdaljaleel et al., 2024; Crompton & Burke, 2024; Harvard University, 2023; Imperial College London, 2023; Stanford University, 2023).

Performance expectancy is the extent to which using a technology will provide benefits to individuals in performing certain activities (Venkatesh et al., 2003, 2012). In the context of generative AI in higher educational policies, benefits involve the degree to which generative AI tools will support and improve teaching and learning activities, contributing to educational outcomes in academic research and activities. A large body of research has shown that performance expectancy can be a determinant of AI technology acceptance (Andrews et al., 2021; Chatterjee & Bhattacharjee, 2020; Guggemos et al., 2020; Raffaghelli et al., 2022). A recent survey showed that performance expectancy had a positive effect on students’ attitudes towards the generative AI (Foroughi et al., 2023). That is, when students believed that using the chatbot would bring them benefits such as convenience, efficiency, or effectiveness in their educational tasks, they had a positive attitude toward the chatbot. Provided that performance expectancy has a positive effect on accepting new technologies, as universities expect AI to greatly assist in personalized learning, efficient data analysis, and creative academic endeavors, they should accept generative AI. Therefore, we hypothesize,

Hypothesis 2

Performance expectancy is positively associated with acceptance of generative AI

Facilitating conditions are defined as individuals’ perceptions of the availability of resources and support necessary for performing activities (Brown & Venkatesh, 2005; Venkatesh et al., 2003). Within the context of generative AI in higher education policies, these resources and support specifically include the extent to which universities provide the essential technical, academic, and policy frameworks required for the effective integration of generative AI tools. Numerous studies have suggested that facilitating conditions are another determinant of individuals’ acceptance of AI technologies (Cabrera-Sánchez et al., 2021; Chatterjee & Bhattacharjee, 2020; Kwak et al., 2022). More recently, a survey study found that facilitating conditions had the strongest effect on students’ intentions and actual usage of generative AI (Habibi et al., 2023), indicating that when students had access to the necessary resources for using generative AI, such as a laptop and internet connection, they were not only more intent on using generative AI but also used it more frequently. Given these findings on facilitating conditions and technology acceptance, universities that provide the required resources and support for leveraging generative AI should endorse its acceptance. Accordingly, we hypothesize,

Hypothesis 3

Facilitating conditions are positively associated with acceptance of generative AI

Perceived risk refers to an individual’s subjective evaluation of the potential negative outcomes associated with a specific action (Abdaljaleel et al., 2024; Ben Arfi et al., 2021; Zhang et al., 2019). In the context of generative AI in higher education policies, such risk covers the degree to which a university perceives potential risks associated with the use of generative AI tools in academics such as cheating, misinformation use, and copyright violation, which undermine student learning. Some studies have suggested that perceived risk is a potential determinant of individuals’ acceptance of new technologies (Ben Arfi et al., 2021; Zhang et al., 2019). Recent research found the negative effect of perceived risk on acceptance of generative AI, which showed that when students and faculty member believed that using generative AI for answering academic queries is risky, they did not think that generative AI in higher education is good for society (Jain & Raghuram, 2024). A multinational study replicated this result (Abdaljaleel et al., 2024). If perceived risk has a negative effect on accepting new technologies, as universities perceives potential risks associated with the use of generative AI tools in higher education, they should be cautious about endorsing its acceptance. Thus, we hypothesize,

Hypothesis 4

Perceived risk is negatively associated with acceptance of generative AI

Present Study

The present study rigorously examines the performance of generative AI, a generative AI model, in assessing the acceptance of generative AI in guidelines from top-ranked universities worldwide. We aim to understand how well generative AI evaluation corresponds with human judgments in terms of acceptance, performance expectancy, facilitating conditions, and perceived risk associated with the use of generative AI. The purpose is twofold: first, to investigate the performance of generative AI in accurately reflecting human perspectives on the adoption of AI within educational policies; and second, to examine the validity of generative AI evaluation from existing technology acceptance frameworks, including TAM and UTAUT.

To ensure a systematic and efficient selection of guidelines, we used the Quacquarelli Symonds (QS) World University Rankings 2023 to identify universities for inclusion (https://www.topuniversities.com/qs-world-university-rankings). We conducted an a priori power with GPower (Faul et al., 2007). A minimum sample size of 26 was required to detect a large effect with 80% statistical power and an α of 5% in correlation analysis. Based on these criteria, we decided to collect guidelines from the top 50 ranked universities. The research ethics committee at the Center for International Education and Exchange of Osaka University approved the study procedure prior to data collection. Appendixes in this study can be found on the Open Science Framework site: https://osf.io/7qshb/?view_only=2de75d6f87e24a7ab3633c8272c8a058. The datasets used in this study are available from the corresponding author on a reasonable request.

Keyword-based Search

This study performed a keyword-based web search of the Google search engine in private browsing mode, after log-out from personal accounts and erasure of all web cookies and history (Jobin et al., 2019; Piasecki et al., 2018). We used the following keywords during the search: [generative AI guidelines], [generative AI policies], [generative AI statementes], and [UNIVERSITY NAME]. [UNIVERSITY NAME] was entered as the name of a university in the top 50 of the QS university rankings. The Google results up to the 200th listings for each Google search were followed and screened for generative AI guidelines only. Within these link listings, we identified 35 non-duplicate documents. We continued to monitor the literature in parallel with the data analysis and until June 30, 2023, to retrieve eligible documents that were released after our search was completed.

Inclusion Criteria

Based on our inclusion criteria, target documents (including statements, guidelines, and notices) included in the final synthesis were (i) written in English; (ii) issued by institutional entities from universities; (iii) mentioned explicitly in their title/description to generative AI or related notions, (iv) expressed a normative stance defined as a preference for a particular course of action related to generative AI.

Assessment of Acceptance of Generative AI

We used a generative AI to measure the acceptance of generative AI in higher education following two steps (i.e., Chat GPT). To measure the acceptance of generative AI in higher education, we operationally defined it as extent to which an institution supports or restricts the use of generative AI technologies by students in academic settings, as reflected in the university’s official guidelines and policies. This definition was derived from previous studies that linked a university’s stance on generative AI to academic integrity (Bin-Nashwan et al., 2023; Cotton et al., 2024; Eke, 2023). Based on this definition, we created four levels of acceptance (See the details in Appendix 1): Level 1, strongly against, prohibits AI use without permission, treating violations as misconduct. Level 2, against, views unauthorized AI as plagiarism, requiring ethical adherence and permission. Level 3, neutral, neither endorses nor bans AI, stressing academic integrity and ethical usage with necessary permissions. Level 4, supportive, encourages responsible AI use, acknowledging its educational value, with guidelines for proper application in assessments.

Next, we evaluated the acceptance of generative AI in higher education using two methods. First, for the human rating, we had three native English undergraduate students independently review 35 university guidelines (two students were male), categorizing them into predefined levels of acceptance. We assessed the reliability of their combined ratings and found it to be acceptable (α = .73). Thus, we calculated the average of these ratings to represent the human-rated acceptance of generative AI in higher education. Second, for the generative AI rating, we asked ChatGPT for classifying each university guideline into the established levels of acceptance (See an example; https://chat.openai.com/share/ae65480c-7721-42fb-9901-493c5dba542a). To ensure accuracy, ChatGPT evaluated each guideline three times. The reliability of these ratings was also found to be high (α = .82). We used the average of these ratings as the ChatGPT-rated acceptance of generative AI in higher education.

Assessment of Performance Expectancy, Facilitating Conditions, Perceived Risk

To measure performance expectancy, we operationally defined it as the degree to which a university perceives potential benefits associated with the use of generative AI tools in academic settings (Chatterjee & Bhattacharjee, 2020; Foroughi et al., 2023; Raffaghelli et al., 2022). Based on this definition, we created four levels of performance expectancy (See the details in Appendix 2). Level 1, Low Expectancy, sees minimal benefits, remaining skeptical of significant improvements. Level 2, Cautious Expectancy, holds a cautious outlook, expecting moderate enhancements in specific areas. Level 3, Moderate Expectancy, anticipates noticeable gains in performance and efficiency, viewing AI as a supportive tool for academic tasks. At Level 4, High Expectancy, the university foresees substantial improvements in education and research, expecting AI to advance personalized learning, data analysis, and creative academic processes.

To measure facilitating conditions, our study operationally defined it as the extent to which universities provide the necessary technical, academic, and policy support for the effective integration of generative AI tools in academic settings (Cabrera-Sánchez et al., 2021; Chatterjee & Bhattacharjee, 2020; Habibi et al., 2023). Based on this definition, we created four levels of facilitating conditions (See the details in Appendix 3). At Level 1, Minimal Facilitation, there’s hardly any support or AI resources. Level 2, Basic Support, offers basic AI tools and a general guide for use, but lacks in-depth resources and training. At Level 3, Supportive Infrastructure, the support is better, with good access to AI technology, some specialized training, and guidelines for ethical use. Level 4, Highly Facilitative, features a strong infrastructure with advanced AI labs, extensive training programs, and comprehensive support policies, ensuring widespread access to AI for all academic activities.

To measure perceived risk, this study operationally defined it as the degree to which a university perceives potential risks associated with the use of generative AI tools in academic settings (Abdaljaleel et al., 2024; Ben Arfi et al., 2021; Jain & Raghuram, 2024). Based on this definition, we created four levels of perceived risk (See the details in Appendix 4). Level 1, Embracing with Guidance, shows the least concern for risks, focusing on ethical use and responsible AI integration with supportive guidelines. Level 2, Conditionally Accepting, balances potential risks and benefits, allowing AI use under ethical and citation standards. Level 3, Cautiously Permissive, acknowledges higher risks, implementing stricter guidelines to prevent misuse. At Level 4, Highly Cautious, the university enforces severe restrictions due to concerns like academic dishonesty, diminishing learning quality, and intellectual property issues, with strict consequences for violations.

Similar to ChatGPT rating in acceptance of generative AI, we asked ChatGPT to categorize each set of guidelines into predefined levels concerning performance expectancy, facilitating conditions, and perceived risk. ChatGPT conducted this classification task three times for each guideline to ensure precision. The reliability of ChatGPT’s ratings across these evaluations was satisfactory, with .92 for performance expectancy, .71 for facilitating conditions, and .89 for perceived risk. We used the average ratings from these assessments.

Correlation Analysis

To test our hypotheses, we conducted Pearson correlation analysis between human-rated acceptance, ChatGPT-rated acceptance, performance expectancy, facilitative conditions, and perceived risk. Figure 1 presents overall correlation matrix for these variables. As Fig. 2 showed, there was a strongly positive correlation between ChatGPT-rated acceptance and human-rated acceptance (r = .85, p < .001). This means that generative AI evaluates texts as accurately as humans. As shown in Fig. 3, ChatGPT-rated acceptance was also positively correlated with performance expectancy (r = .53, p = .001). This indicates that as universities expect AI to enhance teaching and learning activities, they become more accepting of generative AI. Figure 4 presented a similar correlation between ChatGPT-rated acceptance and facilitating conditions (r = .48, p = .003). That is, universities that provide the resources for using generative AI are more accepting of it. Finally, unlike those aspects of generative AI (Fig. 5), ChatGPT-rated acceptance was negatively correlated with perceived risk (r = − .60, p < .001). This suggests that universities aware of the potential risks associated with using generative AI tools in higher education should be cautious in endorsing their acceptance.

We investigated the performance of generative AI in assessing the acceptance of its own use within higher education guidelines. The results confirmed all four hypotheses: Hypothesis 1 was supported by a strong positive correlation between human-rated and ChatGPT-rated acceptance. This study supported Hypothesis 2 by finding a positive relationship between performance expectancy and acceptance. Hypothesis 3 was confirmed by the positive correlation between facilitating conditions and acceptance. Finally, Hypothesis 4, which posited a negative correlation between perceived risk and acceptance, was supported. These findings collectively show the performance of generative AI in text evaluation.

Evaluating Acceptance of Generative AI

Building on a growing body of research on the performance of AI, our study addresses the first research question: Can generative AI evaluate the acceptance of generative AI as effectively as humans? Previous studies have shown that AI, including generative AI, performs impressively in various academic and professional exams, equalling or exceeding human performance across fields such as medicine, engineering, and finance (Frieder et al., 2024; Gilson et al., 2023; Kung et al., 2023; Takagi et al., 2023; Terwiesch, 2023; Watari et al., 2023; Yang & Stivers, 2024). Moreover, prior studies find that generative AI can effectively perform tasks similar to humans, such as providing writing feedback, evaluating essays, and supporting language learning (Escalante et al., 2023; Guo & Wang, 2023; Mizumoto & Eguchi, 2023). Our study first found a strong positive correlation between ChatGPT-rated and human-rated acceptance of generative AI in educational policies. Thus, in response to our research question, the strong positive correlation between ChatGPT-rated and human-rated acceptance provides evidence for the notion that generative AI can evaluate the acceptance of generative AI within educational policies as effectively as human evaluators.

Validating Generative AI Evaluation

Our study also contributes to the extensive research on the Technology Acceptance Model (TAM) or the Unified Theory of Acceptance and Use of Technology (UTAUT) framework by investigating the second research question: Can we validate its evaluation with criterion evaluations of other aspects, including performance expectancy, university conditions, and perceived risk? Previous studies have suggested that performance expectancy, facilitating conditions, and perceived risk are associated with the acceptance of AI technologies, underlining their role in technology adoption decisions (Ben Arfi et al., 2021; Cabrera-Sánchez et al., 2021; Chatterjee & Bhattacharjee, 2020; Guggemos et al., 2020; Kwak et al., 2022; Zhang et al., 2019). Recent studies indicate that performance expectancy and facilitating conditions are positively associated with students’ attitudes and usage of generative AI, while perceived risk is negatively associated with the acceptance of generative AI in education (Foroughi et al., 2023; Habibi et al., 2023; Jain & Raghuram, 2024). Our study found a positive correlation between ChatGPT-rated acceptance of generative AI and performance expectancy, indicating that universities’ expectations of AI enhancing teaching and learning activities are associated with greater acceptance of generative AI. Similarly, there was a positive correlation between ChatGPT-rated acceptance of generative AI and facilitating conditions, which suggests that universities equipped with the necessary resources for using generative AI are likely to accept it. Conversely, the negative correlation between ChatGPT-rated acceptance of generative AI and perceived risk shows that universities aware of the potential risks for generative AI in education are reluctant to accept it. These results expand the application of TAM or UTAUT framework from individual to institutional perspectives, indicating that generative AI can accurately evaluate and distinguish between various aspects of generative AI in educational policies. This answers our second research question, validating generative AI’s assessments against key criteria such as performance expectancy, facilitating conditions, and perceived risk.

Practical Implications

This study has implications for those who currently use or intend to use generative AI because it provides insights into the performance of AI technologies. As generative AI becomes more prevalent in society, expectations have risen substantially. Several articles highlight that emerging technologies such as generative AI often experience stages of initial enthusiasm, followed by disappointment, and finally, practical application, cautioning against overestimating their immediate impact (Eulerich et al., 2023). For some users, their expectations may not be fulfilled probably due to too much expectation. Here is a question; What can generative AI perform and to what extent? Addressing the question is crucial to avoid betraying users’ expectations. Our study answers this question in text evaluation by demonstrating that generative AI can evaluate guidelines on four levels, akin to humans, thereby contributing to the practical application of generative AI in society.

Furthermore, our findings inform research practices on educational policy for generative AI. Educational policy research collected and quantitatively analyzed policy documents and guidelines about generative AI from higher education institutions (McDonald et al., 2024). Another research categorizes the guidelines into different nine fields, including authorship, acknowledgment, plagiarism, advice on assessment design and tasks, detection of GAI use, responsible agent, proper use, improper use, and communication with students (Moorhouse et al., 2023). These studies help us understand how higher education institutions are navigating the integration of generative AI technologies into teaching activities and the development of guidelines to support instructors. However, we still do not know the degree to which each higher education institution accepts generative AI. Then researchers are required to rate it from guidelines, but the process of human rating of texts can be fraught with issues that fatigue, subjectivity, and inconsistency may produce unreliable results (Hussein et al., 2019; Mizumoto & Eguchi, 2023). Generative AI provides a faster, more consistent alternative, potentially transforming rating method, as the reliability of the ChatGPT rating was higher than the reliability of the human rating in this study.

Limitations

We acknowledge some limitations of this study. First, this study focuses on the top 50 universities according to the QS World University Rankings, which may not fully represent the diverse range of higher education institutions globally. As such, the findings may not be applicable to smaller, less research-intensive institutions. Second, this study captures a snapshot of university policies about generative AI at a specific timeframe. Given the fast-paced evolution of both AI technology and societal norms towards it, these policies and guidelines are subject to change. Therefore, the findings might quickly become outdated, limiting their long-term relevance. Third, while our study effectively quantifies acceptance levels and correlates them with various factors, we did not focus on qualitative aspects of university guidelines and policies toward generative AI. The rich context underlying these documents could provide deeper insights into institutional stances, which our quantitative approach might not fully capture.

This study demonstrates that generative AI can effectively evaluate the acceptance of generative AI in higher education guidelines, corresponding to human judgments. The positive correlations between generative AI-rated acceptance and performance expectancy and facilitating conditions, along with its negative correlation with perceived risk, validate the generative AI evaluations within the frameworks of technology acceptance theories. These findings highlight the performance of AI in assisting policy analysis and decision-making in higher educational contexts.

Acknowledgements

We would like to thank our research collaborators.

Funding

This research received no specific grant from any funding agency.

Availability of Data and Materials

The datasets used in this study are available from the corresponding author on reasonable request.

Abdaljaleel, M., Barakat, M., Alsanafi, M., Salim, N. A., Abazid, H., Malaeb, D., Mohammed, A. H., Hassan, B. A. R., Wayyes, A. M., Farhan, S. S., Khatib, S. El, Rahal, M., Sahban, A., Abdelaziz, D. H., Mansour, N. O., AlZayer, R., Khalil, R., Fekih-Romdhane, F., Hallit, R., … Sallam, M. (2024). A multinational study on the factors influencing university students’ attitudes and usage of ChatGPT. Scientific Reports, 14, 1983. https://doi.org/10.1038/s41598-024-52549-8
Andrews, J. E., Ward, H., & Yoon, J. (2021). UTAUT as a model for understanding intention to adopt AI and related technologies among librarians. Journal of Academic Librarianship, 47, 102437. https://doi.org/https://doi.org/10.1016/j.acalib.2021.102437
Ben Arfi, W., Ben Nasr, I., Khvatova, T., & Ben Zaied, Y. (2021). Understanding acceptance of eHealthcare by IoT natives and IoT immigrants: An integrated model of UTAUT, perceived risk, and financial cost. Technological Forecasting and Social Change, 163, 120437. https://doi.org/https://doi.org/10.1016/j.techfore.2020.120437
Bin-Nashwan, S. A., Sadallah, M., & Bouteraa, M. (2023). Use of ChatGPT in academia: Academic integrity hangs in the balance. Technology in Society, 75, 102370. https://doi.org/https://doi.org/10.1016/j.techsoc.2023.102370
Bordt, S., & von Luxburg, U. (2023). Chatgpt participates in a computer science exam. ArXiv Preprint ArXiv. 2303.09461.
Brown, S. A., & Venkatesh, V. (2005). Model of adoption of technology in households: A baseline model test and extension incorporating household life cycle. MIS Quarterly, 29, 399–426. https://doi.org/10.2307/25148690
Cabrera-Sánchez, J.-P., Villarejo-Ramos, Á. F., Liébana-Cabanillas, F., & Shaikh, A. A. (2021). Identifying relevant segments of AI applications adopters – Expanding the UTAUT2’s variables. Telematics and Informatics, 58, 101529. https://doi.org/https://doi.org/10.1016/j.tele.2020.101529
Chatterjee, S., & Bhattacharjee, K. K. (2020). Adoption of artificial intelligence in higher education: a quantitative analysis using structural equation modelling. Education and Information Technologies, 25(5), 3443–3463. https://doi.org/10.1007/s10639-020-10159-7
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61, 228–239. https://doi.org/10.1080/14703297.2023.2190148
Crompton, H., & Burke, D. (2024). The Educational Affordances and Challenges of ChatGPT: State of the Field. TechTrends, 68, 380–392. https://doi.org/10.1007/s11528-024-00939-0
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–340. https://doi.org/10.2307/249008
Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology, 13, 100060. https://doi.org/https://doi.org/10.1016/j.jrt.2023.100060
Elder, C., Pozek, G., Horine, S., Tripaldelli, A., & Butka, B. (2023). Can artificial intelligence pass a sophomore level digital design laboratory? SoutheastCon 2023, 861–868. https://doi.org/10.1109/SoutheastCon51012.2023.10115116
Escalante, J., Pack, A., & Barrett, A. (2023). AI-generated feedback on writing: insights into efficacy and ENL student preference. International Journal of Educational Technology in Higher Education, 20, 57. https://doi.org/10.1186/s41239-023-00425-2
Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2023). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. SSRN Electronic Journal. http://dx.doi.org/10.2139/ssrn.4452175
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. https://doi.org/10.3758/BF03193146
Foroughi, B., Senali, M. G., Iranmanesh, M., Khanfar, A., Ghobakhloo, M., Annamalai, N., & Naghmeh-Abbaspour, B. (2023). Determinants of iIntention to use ChatGPT for educational purposes: Findings from PLS-SEM and fsQCA. International Journal of Human–Computer Interaction, 1–20. https://doi.org/10.1080/10447318.2023.2226495
Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., & Berner, J. (2024). Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 699--744.
Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? American Economist, 68, 233–245. https://doi.org/10.1177/05694345231169654
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9, e45312. https://doi.org/10.2196/45312
Guggemos, J., Seufert, S., & Sonderegger, S. (2020). Humanoid robots in higher education: Evaluating the acceptance of Pepper in the context of an academic writing course using the UTAUT. British Journal of Educational Technology, 51, 1864–1883. https://doi.org/https://doi.org/10.1111/bjet.13006
Guo, K., & Wang, D. (2023). To resist it or to embrace it? Examining ChatGPT’s potential to support teacher feedback in EFL writing. Education and Information Technologies. https://doi.org/10.1007/s10639-023-12146-0
Habibi, A., Muhaimin, M., Danibao, B. K., Wibowo, Y. G., Wahyuni, S., & Octavia, A. (2023). ChatGPT in higher education learning: Acceptance and use. Computers and Education: Artificial Intelligence, 5, 100190. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100190
Harvard University. (2023). Initial guidelines for using ChatGPT and other generative AI tools at Harvard. Retrieved from https://huit.harvard.edu/news/ai-guidelines (July, 13, 2023).
Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208.
Imperial College London. (2023). Generative AI tools guidance: College guidance on the use of generative AI tools. Retrieved from https://www.imperial.ac.uk/about/leadership-and-strategy/provost/vice-provost-education/generative-ai-tools-guidance/ (March, 1, 2023).
Jain, K. K., & Raghuram, J. N. V. (2024). Gen-AI integration in higher education: Predicting intentions using SEM-ANN approach. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12506-4
Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 9, 389–399. https://doi.org/10.1038/s42256-019-0088-2
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2, e0000198. https://doi.org/10.1371/journal.pdig.0000198
Kwak, Y., Seo, Y. H., & Ahn, J.-W. (2022). Nursing students’ intent to use AI-based healthcare technology: Path analysis using the unified theory of acceptance and use of technology. Nurse Education Today, 119, 105541. https://doi.org/https://doi.org/10.1016/j.nedt.2022.105541
McDonald, N., Johri, A., Ali, A., & Hingle, A. (2024). Generative artificial intelligence in higher education: Evidence from an analysis of institutional policies and guidelines. ArXiv Preprint ArXiv:2402.01659.
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2, 100050. https://doi.org/https://doi.org/10.1016/j.rmal.2023.100050
Moorhouse, B. L., Yeo, M. A., & Wan, Y. (2023). Generative AI tools and assessment: Guidelines of the world’s top-ranking universities. Computers and Education Open, 5, 100151. https://doi.org/https://doi.org/10.1016/j.caeo.2023.100151
Piasecki, J., Waligora, M., & Dranseika, V. (2018). Google search as an additional source in systematic reviews. Science and Engineering Ethics, 24, 809–810. https://doi.org/10.1007/s11948-017-0010-4
Polyportis, A., & Pahos, N. (2024). Understanding students’ adoption of the ChatGPT chatbot in higher education: the role of anthropomorphism, trust, design novelty and institutional policy. Behaviour & Information Technology, 1–22. https://doi.org/10.1080/0144929X.2024.2317364
Raffaghelli, J. E., Rodríguez, M. E., Guerrero-Roldán, A.-E., & Bañeres, D. (2022). Applying the UTAUT model to explain the students’ acceptance of an early warning system in Higher Education. Computers & Education, 182, 104468. https://doi.org/https://doi.org/10.1016/j.compedu.2022.104468
Stanford University. (2023). Generative AI policy guidance. Retrieved from https://communitystandards.stanford.edu/generative-ai-policy-guidance (February, 16, 2023).
Takagi, S., Watari, T., Erabi, A., & Sakaguchi, K. (2023). Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: Comparison study. JMIR Medical Education, 9, e48002. https://doi.org/10.2196/48002
Terwiesch, C. (2023). Would chat GPT3 get a Wharton MBA: A prediction based on its performance in the operations management course. Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27, 425–478. https://doi.org/10.2307/30036540
Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly, 36, 157–178. https://doi.org/10.2307/41410412
Watari, T., Takagi, S., Sakaguchi, K., Nishizaki, Y., Shimizu, T., Yamamoto, Y., & Tokuda, Y. (2023). Performance comparison of ChatGPT-4 and Japanese medical residents in the general medicine in-training examination: Comparison study. JMIR Medical Education, 9, e52202. https://doi.org/10.2196/52202
Yang, C., & Stivers, A. (2024). Investigating AI languages’ ability to solve undergraduate finance problems. Journal of Education for Business, 99, 44–51. https://doi.org/10.1080/08832323.2023.2253963
Zhang, T., Tao, D., Qu, X., Zhang, X., Lin, R., & Zhang, W. (2019). The roles of initial trust and perceived risk in public’s acceptance of automated vehicles. Transportation Research Part C: Emerging Technologies, 98, 207–220. https://doi.org/https://doi.org/10.1016/j.trc.2018.11.018

The authors declare no competing interests.

Download PDF

Version 1

posted

You are reading this latest preprint version

Performance of Artificial Intelligence: Does artificial intelligence dream of electric sheep

Status:

Version 1

Abstract

Figures

Introduction

Performance of AI

Evaluating Acceptance of Generative AI: Generative AI and Human Perspectives

Validating Generative AI Evaluation: Expectancy, Conditions, and Risk

Present Study

Method

Inclusion Criteria

Assessment of Acceptance of Generative AI

Assessment of Performance Expectancy, Facilitating Conditions, Perceived Risk

Results

Correlation Analysis

Discussion

Evaluating Acceptance of Generative AI

Validating Generative AI Evaluation

Practical Implications

Limitations

Conclusion

Declarations

Acknowledgements

Funding

Availability of Data and Materials

References

Additional Declarations

Status:

Version 1