Performance of GPT-4 in Oral and Maxillofacial Surgery Board Exams: Challenges in Specialized Questions

doi:10.21203/rs.3.rs-4711032/v1

Download PDF

Research Article

Performance of GPT-4 in Oral and Maxillofacial Surgery Board Exams: Challenges in Specialized Questions

https://doi.org/10.21203/rs.3.rs-4711032/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Purpose

The aim of this study was to evaluate the performance of GPT-4 in answering oral and maxillofacial surgery (OMFS) board exam questions, given its success in other medical specializations.

Methods

A total of 250 multiple-choice questions were randomly selected from an established OMFS question bank, covering a broad range of topics such as craniofacial trauma, oncological procedures, orthognathic surgery, and general surgical principles. GPT-4's responses were assessed for accuracy, and statistical analysis was performed to compare its performance across different topics.

Results

GPT-4 achieved an overall accuracy of 62% in answering the OMFS board exam questions. The highest accuracies were observed in Pharmacology (92.8%), Anatomy (73.3%), and Mucosal Lesions (70.8%). Conversely, the lowest accuracies were noted in Dental Implants (37.5%), Orthognathic Surgery (38.5%), and Reconstructive Surgery (42.9%). Statistical analysis indicated significant variability in performance across different topics, with GPT-4 performing better in general topics compared to specialized ones.

Conclusion

GPT-4 demonstrates a promising ability to answer OMFS board exam questions, particularly in general medical topics. However, its performance in highly specialized areas reveals significant limitations. These findings suggest that while GPT-4 can be a useful tool in medical education, further enhancements are needed for its application in specialized medical fields.

GPT-4

oral and maxillofacial surgery

board exams

artificial intelligence

medical education

Artificial intelligence (AI) systems are revolutionizing various domains of medicine by providing advanced tools for differential diagnosis generation, clinical decision support, and the analysis of imaging, physiologic, and genomic data [1]. Among these systems, large language models (LLMs) such as ChatGPT, developed by OpenAI, have shown significant potential in medical education and professional examinations.

ChatGPT, particularly its latest iteration GPT-4, represents a substantial advancement over its predecessor GPT-3.5. It was trained on a diverse corpus of text through both supervised and unsupervised learning techniques, followed by reinforcement learning with human feedback [2]. While earlier versions of ChatGPT demonstrated near-passing performance on general medical examinations, GPT-4 has exhibited notable improvements across various specialized medical exams, including the United States Medical Licensing Examination (USMLE) and several board certifications in fields such as neurosurgery, dermatology, orthopedic surgery, and radiology [3–6].

The performance of GPT-4 on these exams underscores its enhanced capability to understand and process complex medical information. For instance, GPT-4 scored significantly better than GPT-3.5 on the USMLE, achieving a 20% improvement in accuracy [3]. This suggests that GPT-4 can not only recall factual information but also perform higher-order reasoning, a critical aspect of medical problem-solving.

GPT-4o introduces several features and novelties that distinguish it from previous models. Notably, it integrates a multimodal input capability, allowing it to process and generate text based on both textual and visual inputs, though this functionality was not publicly available at the time of this study [2]. Additionally, GPT-4o boasts enhanced contextual understanding and problem-solving abilities, achieved through a more extensive and nuanced training dataset. This model also incorporates improved fine-tuning processes, leveraging human feedback to refine its responses more effectively. These advancements are designed to enhance GPT-4o’s accuracy, coherence, and relevance, making it a powerful tool for complex and specialized applications, such as those required in medical board examinations

Despite these promising results in other medical specialties, the performance of GPT-4 on oral and maxillofacial surgery (OMFS) board exam questions has not yet been examined. OMFS is an extremely specialized subspecialty, encompassing a wide range of complex clinical scenarios, from craniofacial trauma to oncological procedures, requiring a deep understanding of both surgical principles and clinical management. Given the unique challenges and specificity of OMFS, it is uncertain how well GPT-4 will perform in this domain, despite its success in other fields.

This study aims to evaluate GPT-4's performance on OMFS board exam questions, addressing this gap in the current research. By employing a 250-question mock board examination, we will assess GPT-4's proficiency in this specialized area. The objectives are to determine the accuracy of GPT-4's responses, compare its performance to that of human examinees, and identify any specific strengths or limitations in its ability to handle specialized medical content. By doing so, we aim to elucidate the potential role of advanced LLMs in medical education and their possible future applications in clinical practice.

This study was designed to evaluate the performance of GPT-4o on oral and maxillofacial surgery (OMFS) board exam questions. A total of 250 questions were randomly selected from a comprehensive OMFS question bank. Each question was categorized into specific topic areas to facilitate a detailed analysis of GPT-4o's performance across different domains within OMFS.

The 250 questions were randomly sampled from an established OMFS question bank, ensuring a broad representation of the subject matter typically encountered in board examinations. The questions included multiple-choice formats, covering various topics such as craniofacial trauma, oncological procedures, orthognathic surgery, reconstructive surgery, and general surgical principles.

Each question was assigned to one of several predefined categories based on its primary topic. These categories were designed to reflect the key areas of knowledge required for OMFS board certification. The categorization was performed by experienced oral and maxillofacial surgeons to ensure accuracy and relevance.

The primary outcome measure was the accuracy of GPT-4o's responses, which were classified as either correct or incorrect. For each question, GPT-4o's answer was compared against the correct answer as provided in the question bank.

Statistical Analysis

Statistical analysis was conducted using R, a language and environment for statistical computing. The performance of GPT-4o was analyzed overall and within each topic category. Key metrics included the overall accuracy rate, accuracy rates by category,

Descriptive statistics, including means, standard deviations, and accuracy rates, were calculated to summarize GPT-4o's performance.

Chi-square tests were employed to assess the differences in accuracy across different topic categories.

Data Handling

All data, including the questions, correct answers, and GPT-4o's responses, were stored and processed securely to maintain the integrity of the analysis. The data was anonymized to ensure that no personal or identifiable information was included in the study.

Tools and Software

The analysis was performed using R (version 4.1.2), which provided the necessary statistical functions and capabilities for comprehensive data analysis. The ggplot2 package was used for generating visualizations to illustrate the performance results across different categories.

GPT-4o's performance on the 250 OMFS board exam questions was assessed by categorizing the questions into specific topic areas and recording whether the answers were correct or incorrect. Overall, GPT4o achieved a score of 62% correct answers

The questions were distributed across several key topics within oral and maxillofacial surgery, as shown in Fig. 1. The largest proportion of questions were related to Infection (15%), followed by Oncology (14%), Trauma (13%), and Mucosal Lesions (10%). Other topics included Anatomy, Pharmacology, Reconstructive Surgery, Orthognathic Surgery, TMJ/TMD, Implantology, General Dentistry, Surgical Techniques and Material Science, and Congenital Deformities and Syndromes.

Figure 2 displays the accuracy of GPT-4o's answers across the different topics. The highest accuracy was observed in Pharmacology (92.8%), followed by Anatomy (73.3%) and Mucosal Lesions (70.8%). In contrast, the lowest accuracies were noted in Dental Implants (37.5%), Orthognathic Surgery (38.5%), and Reconstructive Surgery (42.9%). Interestingly the AI answered more generalized questions with higher accuracy, while the level of correct answers decreases the more specific the topic of the questions is (Fig. 3).

The present study investigated the ability of the AI-based chat program "ChatGPT 4o" to answer 250 specific questions in the field of oral and maxillofacial surgery. The AI was able to answer 62% of the questions correctly, meaning that, depending on the passing mark on the test, would be able to pass the very challenging and specific board exam of oral and maxillofacial surgery

The use of freely accessible chat programs like "ChatGPT" is rapidly expanding, with an increasing pool of retrievable information and improving accuracy [7, 8]. The medical landscape, too, cannot escape the rapid development and targeted information provision by AI programs [9]. There is a steep rise in the provision and use of AI-based applications both in the area of general medical information and in the field of clinical inquiries, patient education, teaching, and many other areas [10–12]. A sensitive subtopic currently is the performance of freely accessible chatbots in answering exam questions in the medical context [13]. To exemplify this for the field of oral and maxillofacial surgery (OMF), we had "ChatGPT 4o“ answer 250 prep questions for the OMF-board exam in English.

Despite the predominantly very specific and thematically detailed questions, the AI, which is fundamentally designed as a tool for general inquiries across all known subject areas, was able to perform at a level that would likely be sufficient to pass a real exam in this field. For a more precise analysis of the data and the AI's success rate, the 250 questions were previously divided into a total of 13 meaningful subtopics (Fig. 2).

There were significant differences in the correct answering of questions between the different groups. We attribute this primarily to two factors. Evaluating all the data, it became evident that there is a particular vulnerability in answering questions that feature complex semantic relationships or are designed to mislead the examinee. For instance, in an anatomical question, asking about a landmark in anesthetizing the maxillary nerve, the given answer was correct. However, questions requiring nuanced understanding and the ability to navigate potentially misleading information were incorrectly answered (Fig. 3). For example, the question was about the correct orientation of an incision. While the relevant hint for the answer was not given (to answer correctly the orientation of the relaxed skin tension lines (RSTLs) would have been significant) the underlying muscle was mentioned, and Chat-GPT answered wrong, tanking the muscle as a landmark for the incision. The second main problem identified was a clear gradient between the groups when it came to the degree of specialization of the individual question complexes. It was found that question clusters dealing with more general topics performed significantly better than more specialized question groups. For example, questions involving specific knowledge about the relative risk of complications in a cleft palate surgery were answered wrong. The less accessible information was for Chat GPT, the higher the chance for an incorrect answer. In this study, four of the five topics that performed better than the average of 62% were of a very general nature (pharmacology, anatomy, infections, and oncology). The field of traumatology matched the average. On the other hand, the specific topics performed below average. Particularly highly specialized topics with minimal overlap with other medical areas (dental implants and orthognathic surgery) performed significantly poorly. This problem is already known from recent literature and has been analyzed in several scientific works [14–16]. The chatbot "ChatGPT 4.0" is not specialized in any particular topic but is intended to provide cross-disciplinary assistance in virtually every field. It is therefore understandable that topics of a more general nature can be answered better than specialized topic complexes. Besides the database related to the retrievable knowledge of specific topics, the lack of specialization in expert knowledge, and the increasing complexity of questions in specific genres, context dependency is of particular importance in successfully answering questions. This means that the precision of the answer depends on the AI understanding a specific context and correctly placing it within a particular question [17].

The main weakness of this study is the number of questions examined. With 250 questions, one must assume that the results are not entirely representative. In addition, the question groups were divided into a total of 13 categories, which led to relatively small group sizes, thereby losing representativeness. Even though there is a clear trend that more specialized topics are answered less frequently correctly than more general ones, this should be further investigated through more extensive studies with a larger number of questions.

Funding and Conflict of interest:

No funding was provided for this study. The authors declare no conflicts of interest related to this study.

Compliance with Ethical Standards

The authors declare that this study was conducted in compliance with ethical standards of the declaration of Helsinki. Approval from a relevant ethics was unnecessary for this kind of investigation, since no Patient data was used.

Author Contribution

FBW and NH designed the study and wrote the manuscript, FBW performed the tests, NH prepared the figures, all authors reviewed the manuscript

Oermann EK, Kondziolka D (2023) On Chatbots and Generative Artificial Intelligence. Neurosurg April 92(4):665
https://cdn.openai.com/papers/gpt-4.pdf [accessed 8. Juli 2024].
Nori H, King N, McKinney SM, Carignan D, Horvitz E Capabilities of GPT-4 on Medical Challenge Problems. 20. März 2023 [zitiert 8. Juli 2024]; Verfügbar unter: https://www.microsoft.com/en-us/research/publication/capabilities-of-gpt-4-on-medical-challenge-problems/
Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepaño C (2023) u. a. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 9 Februar 2(2):e0000198
Ali R, Tang OY, Connolly ID, Zadnik Sullivan PL, Shin JH, Fridley JS (2023) u. a. Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations. Neurosurg Dezember 93(6):1353
Elias ML, Burshtein J, Sharon VR (2024) OpenAI’s GPT-4 performs to a high degree on board-style dermatology questions. Int J Dermatol 63(1):73–78
Kaplan J, McCandlish S, Henighan T, Brown TB, Chess B, Child R (2020) Scaling Laws for Neural Language Models [Internet]. arXiv; [accessed 8. Juli 2024]. http://arxiv.org/abs/2001.08361
Ziegler DM, Stiennon N, Wu J, Brown TB, Radford A, Amodei D (2020) Fine-Tuning Language Models from Human Preferences [Internet]. arXiv; [zitiert 8. Juli 2024]. Verfügbar unter: http://arxiv.org/abs/1909.08593
Tian S, Jin Q, Yeganova L, Lai PT, Zhu Q, Chen X (2024) u. a. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings Bioinf 1 Januar 25(1):bbad493
Alsadhan A, Al-Anezi F, Almohanna A, Alnaim N, Alzahrani H, Shinawi R (2023) u. a. The opportunities and challenges of adopting ChatGPT in medical research. Front Med 22 Dezember 10:1259640
Sahu PK, Benjamin LA, Singh Aswal G, Williams-Persad A (2024) ChatGPT in research and health professions education: challenges, opportunities, and future directions. Postgrad Med J 1 Januar 100(1179):50–55
Kim TW (2023) Dezember. Application of artificial intelligence chatbots, including ChatGPT, in education, scholarly work, programming, and content generation and its prospects: a narrative review. J Educ Eval Health Prof. 27. ;20:38
Gutiérrez-Cirlos C, Carrillo-Pérez DL, Bermúdez-González JL, Hidrogo-Montemayor I, Carrillo-Esper R, Sánchez-Mendiola M (2023) ChatGPT: opportunities and risks in the fields of medical care, teaching, and research. Gaceta médica de México Oktober 159(5):382–389
Huang K, Altosaar J, Ranganath R, ClinicalBERT (2020) Modeling Clinical Notes and Predicting Hospital Readmission [Internet]. arXiv; [accesed 8. Juli 2024]. Verfügbar unter: http://arxiv.org/abs/1904.05342
Lee J, Yoon W, Kim S, Kim D, Kim S, So CH (2020) u. a. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinf 15 Februar 36(4):1234–1240
Gururangan S, Marasović A, Swayamdipta S, Lo K, Beltagy I, Downey D et al (2020) Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks [Internet]. arXiv; [accesed 8. Juli 2024]. http://arxiv.org/abs/2004.10964
Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P Language Models are Few-Shot Learners. Adv Neural Inf Process Syst, 33, 1877–1901

No competing interests reported.

Download PDF

Reviews received at journal
28 Jul, 2024
Reviewers agreed at journal
19 Jul, 2024
Reviewers invited by journal
19 Jul, 2024
Editor assigned by journal
10 Jul, 2024
Submission checks completed at journal
10 Jul, 2024
First submitted to journal
09 Jul, 2024

You are reading this latest preprint version

Performance of GPT-4 in Oral and Maxillofacial Surgery Board Exams: Challenges in Specialized Questions

Status:

Version 1

Abstract

Purpose

Methods

Results

Conclusion

Figures

Introduction

Materials and Methods

Statistical Analysis

Results

Discussion

Declarations

Funding and Conflict of interest:

Compliance with Ethical Standards

Author Contribution

References

Additional Declarations

Status:

Version 1