ChatGPT in Oral and Maxillofacial Radiology Education

doi:10.21203/rs.3.rs-3566948/v1

Download PDF

Research Article

ChatGPT in Oral and Maxillofacial Radiology Education

https://doi.org/10.21203/rs.3.rs-3566948/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Introduction Artificial intelligence refers to the ability of computer systems or machines to perform cognitive functions and tasks that are similar to those of humans. The aim of this study is to assess the knowledge and interpretative abilities of ChatGPT-versions by administering a dentomaxillofacial radiology exam, comparing its performance with that of dentistry-students in Türkiye, and questioning the effectiveness of different language options. Methods This study is a descriptive research comparing the data obtained by presenting the midterm exam questions, which were routinely administered to 4thyear dental students, to ChatGPT versions 3.5 and 4 in both Turkish and English. Results Firstly the 20 test questions were evaluated. The p-value was <0.05, there is a significant difference between the ChatGPT answer sheets. ChatGPT-4 in English demonstrated the highest performance. Answer sheets of chatGPT-4 in Turkish and in English demonstrated the best performance with 5 correct answers in open-ended questions. Based on the answers obtained from 89 students to 20 test questions, a class profile was created. ChatGPT answer sheets and class-profile were analyzed, the p-value was p<0.05. Class-profile ranked first as ChatGPT-4 in English. It was found a significant difference between the answer-sheets of ChatGPT and the class-profile for open-ended questions(p<0.10). The most successful results were obtained from ChatGPT-4 in Turkish and English, as well as the class profile. Conclusion As a cautionary note to dental students, it is important to mention that ChatGPT's knowledge and perception in the field of oral and maxillofacial radiology are not yet sufficient, particularly for use in examinations.

Artificial intelligence(AI) refers to the ability of computer systems or machines to perform cognitive functions and tasks that are similar to those of humans. These functions include image or language recognition, learning, decision-making, problem-solving and other processes that are typically performed by humans. AI systems use algorithms to perform tasks that were once the sole domain of human intelligence. As AI technology continues to evolve, its applications are becoming increasingly common in various fields(Huh,2023;Keser,2021;Yüzbaşıoğlu,2021;Sur,2020).

A large language model (LLM) is a type of machine learning model that uses deep learning algorithms. They are also known as auto regressive language models and ChatGPT(Chat Generative Pre-trained Transformer) is the latest among that class. ChatGPT is an AI language model developed by Open AI (San Francisco, California) using the GPT architecture. It is designed to generate human-like responses to prompts or questions, and it uses advanced algorithms to understand and process natural language. ChatGPT's ability to engage in human-like conversations is a direct result of its AI capabilities. ChatGPT is a chatbot that uses artificial intelligence to create advanced and natural interactions between humans and machines. ChatGPT is used in many applications, including customer service, chatbots, text-based dialogue systems, automatic text completion tools, and more(Thurzo,2023;Gilson,2023;Cascella,2023;Ollivier,2023).

A short time after the announcement of the first version of the model, GPT-1, in 2018, GPT-2 was introduced in 2019 with a much larger data set and analytical power compared to the first one. With the release of GPT-3 in 2020, the series of models that can understand and generate natural language was formed and it became the largest language model of all time. The currently default version, GPT-3.5, has been used with the addition of code understanding and generation capabilities to its superior language understanding ability. As of March 14, 2023, the limited beta version of GPT-4 has been released. Currently, GPT-3.5 is an advanced version for both text and code, and it has been reported that it will have image analysis capabilities in the future(Gilson,2023;Ollivier,2023;).

In the past decade, the clinical use of AI programs in the medical profession has gained significant popularity, including in the field of dentistry. Artificial intelligence has various applications in dentistry, not only in clinical are as such as orthodontics, endodontics, dental radiology, periodontics, dentomaxillofacial surgery, diagnosis, treatment or prosthetics but also in dental education and patient management. Dental education may not be limited to clinical simulator studies only, but it can also involve opportunities for students to reinforce or question the theoretical knowledge they acquire during their education. An artificial intelligence tool like ChatGPT can provide many benefits for dental students in their education such as answering students' questions related to dentistry and assisting them in their learning process. For this purpose, demonstrating the abilities of ChatGPT can provide very important information for dental students in terms of their education and can demonstrate the reliability of ChatGPT for their learning. Dental students should be able to evaluate the accuracy of medical and dental information produced by AI and have the ability to create reliable and verified information for patients. Therefore, the ability of ChatGPT, a recently developed AI chatbot, to accurately answer dental exam questions should be determined(Huh,2023;Ding,2023;Agrawal,2022;Nguyen,2021;Park,2019).

The aim of this study is to assess the knowledge and interpretative abilities of ChatGPT versions by administering a dentomaxillofacial radiology exam, comparing its performance with that of dentistry students in Turkiye, and questioning the effectiveness of different language options.

The null hypothesis of this study is that ChatGPT’s exam result will be more successful compared to the exam results of students who have taken this course.

This study is a descriptive research comparing the data obtained by presenting the midterm exam questions, which were routinely administered to 4th year dental students, to ChatGPT versions 3.5 and 4 in both Turkish and English. Since there were no human or animal experiments involved, there was no requirement for informed consent or ethical approval from the faculty review boards. On April 14, 2023, a dentomaxillofacial radiology midterm exam was conducted for 4th year students at University of Health Sciences, Gülhane Faculty of Dentistry. These exam questions were implemented in two distinct versions of ChatGPT, each supporting both Turkish and English languages, and were administered twice in each version and language. As a result, a total of four ChatGPT answer-sheets were generated.

All of the exam papers were assessed by the same evaluators (Associate Professor Buğra ŞENEL, Assistant Professor Hilal PEKER ÖZTÜRK). The Turkish answers obtained were manually written on a copy of the official exam paper and shuffled with all the exam papers to ensure objectivity in the assessment conducted by the evaluators. The Turkish and English versions of the same questions were administered twice to ChatGPT's 3.5 and 4th versions. And the average of the obtained results was incorporated into the final assessment. The study involved the comparison of answer sheets generated using ChatGPT. These answer sheets were compared with students’ performance. This comparative analysis aimed to evaluate the performance of ChatGPT in relation to the dentistry students and also to assess the variations and consistencies within the ChatGPT-generated answers. The examination consisted of 28 questions, including 20 multiple-choice questions and 8 essay and short-answer questions. The 8 essay or short-answer questions were evaluated on a scale of 5 points. According to the assessment by the evaluator, if a score of 2.5 or higher was obtained for each question, it was considered correct, whereas if the score was below 2.5, it was considered incorrect both in the evaluation of students’ answer-sheets and in the evaluation of ChatGPT’s answer-sheets.

A total of 8 answer-sheets of chatGPT were evaluated in terms of multiple-choice questions and open-ended question type. The purpose of this was to compare the success performance of ChatGPT in both the multiple-choice and essay-style questions, examining how well it performed in each type of question relative to the others. The topics covered in this exam were taught to students over a total of 12 weeks and 24 lecture hours between November 2022 and April 2023. The total number of participating students was 89, and there were no exclusion criteria.

The overall performance of the 89 participating students and ChatGPT were evaluated. All statistical analyses were conducted using IBM SPSS Statistics 21.0(IBM Corp., Armonk, NY, USA) and MS Excel 2007. Descriptive statistics, such as the number, mean, and standard deviation, were calculated for the variables.

The 20 test questions were evaluated by chi-square test. Since the p-value is 0.019(p < 0.05), there is a significant difference between the ChatGPT answer sheets. The exam performances were evaluated to determine the nature of the difference (Table-1).

ChatGPT-4 in English demonstrated the highest performance by correctly answering 15 out of 20 multiple-choice questions, whereas answer sheet of chatGPT-4 in Turkish and ChatGPT-3.5 in English had 15 questions correctly. On the otherhand, chatGPT-3.5 in Turkish were showed poor performance with 10 correct answers.

There is a weak significant difference between the ChatGPT answer sheets in terms of open-ended questions. The p-value is calculated as 0.073(p < 0.10). Answer sheets of chatGPT-4 in Turkish and in English demonstrated the best performance with 5 correct answers. The poor performance in open-ended questions belonged to ChatGPT-3.5 in Turkish with only 2 correct answers (Table-2). The question-based success chart for ChatGPT versions’ answer sheets has been generated for both multiple-choice and open-ended questions (Fig. 1).

Based on the answers obtained from 89 students to 20 test questions, a class profile was created(Table-3). ChatGPT answer sheets and the answers of students(class-profile) for the 20 test questions were analyzed using the Kruskal-Wallis test, the p-value being 0.018 (p < 0.05) indicates a significant difference. Class-profile (students) ranked first with 15 correct answers as ChatGPT-4 in English.

Another class profile was created for open-ended questions of 89 students. The class-profile’s question-based success chart has been developed for both multiple-choice and open-ended questions (Fig. 2). Kruskal-Wallis test was found a significant difference between the answer sheets of ChatGPT models and the class-profile’s answers for open-ended questions(p = 0.076, p < 0.10). There was a weak significant difference. Students achieved similar performance with some answer sheets of ChatGPT. The most successful results in terms of open-ended questions were obtained from ChatGPT-4 in Turkish and English, as well as the class profile, by correctly answering 5 out of the 8 open-ended questions.

When the answers were evaluated totally in the present study, class-profile and English version of ChatGPT-4 got 70 out of 100 and showed the highest performance. ChatGPT-4 in Turkish version got 67 out of 100 showed 2nd highest performance. The worst one was ChatGPT-3.5 in Turkish got 40 out of 100. The correct and incorrect answers given to both multiple-choice and open-ended questions, along with the total scores obtained, are provided in Table-3 for all answer-sheets.

The performance of ChatGPT-3.5 and ChatGPT-4 versions in English language remained unchanged, demonstrating consistent performance in both the first and second evaluations. In open-ended questions, while in ChatGPT-4, both the first and second evaluations in English produced the same results, in ChatGPT-3.5, the second evaluation yielded better results compared to the first evaluation. In multiple-choice questions, the first evaluation of ChatGPT-3.5 in Turkish has shown better performance compared to the second evaluation. In open-ended questions, both evaluations have yielded the same results. In version 4, the second evaluation conducted in Turkish has produced better results in both multiple-choice and open-ended questions compared to the first evaluation. Based on these results, in the Oral and Maxillofacial Radiology exam ChatGPT has provided more consistent results in English.

The null hypothesis of this study that ChatGPT’s exam result will be more successful compared to the exam results of students was rejected; In multiple-choice questions, English version of ChatGPT-4 has demonstrated the highest performance, but same with the class profile(students). And in open-ended questions, ChatGPT-4 in English and in Turkish and the class profile have yielded equivalent results.

As of January 2023, ChatGPT has reached 100 million users, establishing itself as the artificial intelligence program with the highest user ratio (Eysenbach,2023). For researchers and practitioners to use ChatGPT correctly and to prevent potential problems, it is essential to have a good understanding of ChatGPT's capabilities and limitations (Cascella,2023).

However there is a limited number of studies available that have examined ChatGPT in the medical and dental fields so far (Fatani,2023).

In a similar study performed by Huh (Huh,2023), the answers of medical school students and ChatGPT were compared in an examination conducted in the field of medical parasitology. The examination success performance of medical school students has been found to exceed that of ChatGPT (Huh,2023). In the present study, success has been evaluated based on two question types: open-ended questions and test questions. Additionally, two different versions of ChatGPT, in English and Turkish, were subjected to the examination twice each. Unlike Huh's study, in this research, ChatGPT-4 in English joined the highest academic achievement with the class profile both in terms of the total result of the exam and the correct answers to the test questions.

In open-ended questions, both students and ChatGPT-4 in Turkish and English achieved the same level of success. However, ChatGPT 3.5 provides different results for Turkish and English. It is thought that some language models give different answers in different languages, it is because of the complexity of these models, training data and language characteristics. Therefore, the model may give different answers depending on the language and it is not always possible to achieve perfect consistency.

The study has revealed another finding that the first and second evaluations of the results of the Oral and Maxillofacial Radiology exam conducted in English by ChatGPT are more consistent with each other compared to those conducted in Turkish language. In fact this consistency has reached 100% in version-4.

According to the evaluated versions of chatGPT, the highest success rate in multiple-choice questions is 75%, whereas in open-ended questions, it decreases to 62.5%.

In this study, within the field of oral and maxillofacial radiology, ChatGPT has demonstrated relatively superior performance in multiple-choice questions compared to open-ended questions. However, the results from both sets of questions indicate that there are some areas where the knowledge of ChatGPT appears to be lacking.

This study results may be applicable not to all dentistry fields, but specifically to the field of oral and maxillofacial radiology. However, it would be said that the knowledge of ChatGPT is not highly sufficient for advanced levels in this field.

In their study, Khurana and Vaddi have identified the potential applications of ChatGPT within the realm of Oral and Maxillofacial Radiology. They have highlighted several areas where ChatGPT can be utilized, including generating oral radiology reports, responding to multiple-choice questions, aiding in scientific writing, contributing to dental education by assisting in presentation creation, providing feedback on student assignments, and aiding in the preparation of academic content outlines. However, they have noted that ChatGPT exhibits limitations in addressing queries involving image-based questions (Khurana,2023).

The performance of ChatGPT has been evaluated in a study for the United States Medical Licensing Examination(USMLE) Step 1, 2, and 3. The examination encompassed a mixture of open-ended questions with variable inputs and multiple-choice questions, single answer questions. As observed in the present study, ChatGPT demonstrated its lowest performance in step-1 assessment, which involved open-ended questions, followed by its performance in step-2 and 3. The outcomes of the study are consistent with those of human subjects (Alkaissi,2023).

According to the results obtained from this study, it has been confirmed that ChatGPT-4 has been enhanced and improved compared to version 3.5.

In another study Ali and his colleagues utilized ChatGPT to generate clinical letters of patients. According to the obtained results, the correctness rates of the letters created using ChatGPT were high scores compared to the letters generated by the physicians themselves (Ali,2023). One of the crucial aspects of studies lies in the act of writing. This is because a study can only be effectively introduced to the literature and the realm of academia through well-crafted composition. However, it is imperative to recognize that ChatGPT should not serve as the sole source. It is advisable to seek support from ChatGPT for language enhancement and rectification of certain errors.

The most significant advantage of ChatGPT can be evaluated as its ability to quickly understand the given information and reach evidence-based conclusions faster than humans (Salvagno,2023). In essence, anyone can ask questions on any topic and receive satisfying answers quickly through ChatGPT (Fatani,2023).

Overall in multiple-choice questions, ChatGPT-4 in English has demonstrated the highest performance, same as the class profile. However, in open-ended questions, ChatGPT 4 in English and in Turkish and the class profile have yielded equivalent results as the highest performance. The different results obtained in ChatGPT-3.5 evaluations in Turkish and English necessitate new researchs about evaluation in different languages. Therefore, new studies need to be conducted in different languages with different contents.

As a cautionary note to dental students, it is important to mention that ChatGPT's knowledge and perception in the field of oral and maxillofacial radiology are not yet sufficient, particularly for use in examinations. However, it is rapidly continuing to evolve and improve.

Acknowledgement

We would like to thank Dr. Mustafa Çağrı PEKER, who made the statistics of the study.

Declaration of Conflicting Interests

The authors declare that there is no conflict of interest.

Funding

The authors received no financial support for the research, authorship and publication of this article.

Huh, S. (2023). Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination? A descriptive study. J Educ Eval Health Prof, 20(1).
Keser, G., & Pekiner, F. N. (2021). Attitudes, Perceptions and Knowledge Regarding the Future of Artificial Intelligence in Oral Radiology Among a Group of Dental Students in Turkey: A Survey. Clin Exp Health Sci, 11, 637–641.
Yüzbaşıoğlu, E. (2021). Attitudes and perceptions of dental students towards artificial intelligence. Journal Of Dental Education, 85, 60–68.
Sur, J., Bose, S., Khan, F., Dewangan, D., Sawriya, E., & Roul, A. (2020). Knowledge, attitudes, and perceptions regarding the future of artificial intelligence in oral radiology in India: A survey. Imaging Sci Dent, 50, 193–198.
Thurzo, A., Strunga, M., Urban, R., Surovková, J., & Afrashtehfar, K. I. (2023). Impact of artificial ıntelligence on dental education: a reviewand guide for curriculum update. Educ Sci, 13, 150.
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., et al. (2023). How does chatgpt perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR MedEduc, 9, e45312. Feb 8;.
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023). Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal Of Medical Systems, Mar 4(1), 33.
Ollivier, M., Pareek, A., Dahmen, J., et al. (2023). A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic research. Knee Surgery, Sports Traumatology, Arthroscopy, 31, 1190–1192.
Ding, H., Wu, J., Zhao, W., Matinlinna, J. P., Burrow, M. F., & Tsoi, J. K. H. (2023). Artificial intelligence in dentistry-a review. Front Dent Med, 4, 10.3389/fdmed.2023.1085251.
Agrawal, P., & Nikhade, P. (2022). Artificial intelligence in dentistry: Past, present, and future. Cureus, 14(7), e27405.
Nguyen, T. T., Larrivee, N., Lee, A., Bilaniuk, O., & Durand, R. (2021). Use of artificial intelligence in Dentistry: Current clinical trends and research advances. J Can DentAssoc, 87, l7.
Park, S. H., Do, K. H., Kim, S., Park, J. H., & Lim, Y. S. (2019). What should medical students know about artificial intelligence in medicine? J Educ Eval Health Prof, 16, 18.
Eysenbach, G. (2023). The role of ChatGPT, generative language models, and artificial intelligence in medical education: A conversation with ChatGPT and a call for papers. JMIR Med Educ, 9, e46885. 10.2196/46885.
Fatani, B. (2023). ChatGPT for future medical and dental research. Cureus, 15(4), e37285. 10.7759/cureus.37285.
Khurana, S., & Vaddi, A. (2023). ChatGPT from the perspective of an academic oral and maxillofacial radiologist. Cureus, 15(6), e40053. 10.7759/cureus.40053.
Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15, e35179. 10.7759/cureus.35179.
Ali, S. R., Dobbs, T. D., Hutchings, H. A., & Whitaker, I. S. (2023). Using ChatGPT to write patient clinic letters. Lancet Digit Health, 3, 3–7. 10.1016/S2589-7500(23)00048 – 1.
Salvagno, M., Taccone, F. S., & Gerli, A. G. (2023). Can artificial intelligence help for scientific writing? Critical Care, 27, 75. 10.1186/s13054-023-04380-2.

Table-1 ChatGPT answer sheets’ correct and incorrect answers to the 20 multiple-choice questions

group	incorrect answers	correct answers	total questions	success ranking
chatGPT 3.5 Turkish	10	10	20	3
chatGPT 3.5 English	6	14	20	2
chatGPT 4 Turkish	6	14	20	2
chatGPT 4 English	5	15	20	1

Table-2 ChatGPT answer sheets’ correct and incorrect answers to the 8 open-ended questions

group	incorrect answers	correct answers	total questions	success ranking
chatGPT 3.5 Turkish	6	2	8	3
chatGPT 3.5 English	4	4	8	2
chatGPT 4 Turkish	3	5	8	1
chatGPT 4 English	3	5	8	1

Table-3 The number of correct answers(N) and scores(S) of all answer sheets and total scores obtained from them. The number of multiple-choice questions were 20, and the number of open-ended questions were 8.

Answer-Sheet	Correct answers to the test-questions *N/S*	Correct answers to open-ended questions *N/S*	Totally Score S
Class-Profile(Students)	15/45	5/25	70
Chatgpt 3.5/ Turkish	10/30	2/10	40
Chatgpt 3.5/ English	14/42	4/20	62
Chatgpt 4/ Turkish	14/42	5/25	67
Chatgpt 4/ English	15/45	5/25	70

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

ChatGPT in Oral and Maxillofacial Radiology Education

Status:

Version 1

Abstract

Figures

INTRODUCTION

METHODS

RESULTS

DISCUSSION

CONCLUSION

Declarations

References

Tables

Additional Declarations

Status:

Version 1