Performance of ChatGPT in Dentistry: Multi-specialty and multi-centric study

doi:10.21203/rs.3.rs-3247663/v1

Download PDF

Research

Performance of ChatGPT in Dentistry: Multi-specialty and multi-centric study

https://doi.org/10.21203/rs.3.rs-3247663/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Artificial Intelligence (AI) powered tools have transformed the field of healthcare. A recently launched large language model, ChatGPT has gained significant traction due to its communicative interface and relevance of the responses generated. This tool could be utilized in dentistry for dental education, and possibly as a clinical decision support system. Hence, it is imperative to evaluate the accuracy of the model in relation to the responses generated for dental-related queries.

Methods: This multi-centric study involved 27 subject experts from nine dental specialties of various institutions and 2 heads of institutions. A total of 243 questions were formulated and the answers generated by ChatGPT (version: 3.5) were rated in terms of accuracy (6-point Likert), completeness (4-point Likert), and relevance (5-point Likert).

Results: The mean accuracy of the ChatGPT-generated answers was 4.61 (SD 1.575), with a median of 5.33. For completeness, mean score was 2.01 (SD 0.793), and the median was 2.33. Regarding relevance, a mean of 3.13 (SD 1.590) and median of 3.67 were obtained. The highest ratings were observed for answers related to Oral Medicine and Radiology, as well as for open-ended questions, and questions labelled as easy in terms of difficulty.

Conclusion: The promising results observed in the study promote the application of ChatGPT for retrieving dental information. However, it is crucial to exercise caution and seek advice from a qualified healthcare for dental health-related queries. Further large-scale testing of the model is necessary before incorporating it into dental clinical practice.

Artificial Intelligence

ChatGPT

Dentistry

Large Language Model

Natural Language Processing

Utilization of Artificial Intelligence (AI) powered tools in the delivery of healthcare has seen a recent surge in popularity. From diagnosis of dental conditions, disease prediction, prognostic evaluation, and clinical decision-making, AI has a vast array of applications in dentistry.¹ One of the AI-powered tools that has had a tremendous impact on the general population and the healthcare industry is Chat Generative pre-training transformer or ChatGPT. Recently launched in November 2022, ChatGPT is a Large Language Model (LLM) developed by OpenAI (OpenAI, L.L.C., San Francisco, CA, USA) which has the potential to revolutionize the field of dentistry.²

ChatGPT, a major breakthrough in the field of Natural Language Processing (NLP) mimics human language processing abilities using deep learning and neural networks to generate human-like text. Owing to its training on massive dataset involving textbooks, research papers, webpages, it could process an array of information and provide concise and appropriate responses to human prompts. ChatGPT version 3.5 is easily accessible and free for the public to use, further contributing to its success. Recently ChatGPT has been utilized in healthcare for generating literature reviews and summaries, analysis of clinical datasets, scientific and academic writing, documentation and streamlining clinical workflow.³

The dental applications of ChatGPT remain to be studied comprehensively given its recent launch. However, few studies have briefly discussed the role of the language model in dental telemedicine, treatment planning, text mining from electronic health records, dental education, and patient education.^4,5 In dental education, ChatGPT proposes to facilitate teaching of complex dental procedures by generating step-by-step instructions, create interactive educational content through quizzes and flashcards, generate case scenarios to improve basic knowledge of students, and assist dental students in proficient communication with patients through generation of simulated patient interactions.³

Although ChatGPT has shown tremendous potential in other areas, the accuracy and relevance of the responses to dentistry-related queries remain in question. Therefore, this multi-centric study was undertaken to evaluate the performance of ChatGPT-generated responses to questions from multiple dental specialties. If proven effective, it could be utilized by dental students and staff to answer common dental queries in routine practice. Furthermore, the model could also be used as a decision support system to aid in diagnosis and treatment plans based on the details prompted by the user that could steer the course of dental education and healthcare.

A multi-centric observational study was carried out to determine the accuracy, completeness, and relevance of answers generated by ChatGPT to questions from major dental specialties. The objective of this testing was to determine the accuracy and relevance of the AI language tool in dental education rather than to evaluate accuracy of responses to patient-centred queries relating to common dental symptomatology. The study was exempted from review by Institutional Ethics Committee since no data was collected from patients or external sources. A total of 27 subject experts from different universities across the country were approached by the senior investigators for collecting questions from 9 dental specialties and for rating the answers generated by ChatGPT (version: 3.5). 2 heads of institutions were also invited to re-evaluate few selected AI-generated responses.

2.1 Questions Dataset:

Three subject experts working as faculty in various dental institutes, each from the nine dental specialties of Conservative dentistry and Endodontics, Oral and Maxillofacial Surgery (OMFS), Oral Medicine and Radiology (OMDR), Oral Pathology, Orthodontics, Paediatric and Preventive dentistry, Periodontics, Prosthodontics, and Public Health Dentistry (PHD), were invited to participate in the study. Each subject expert formulated a set of 9 random specialty-specific clinical as well as non-clinical questions relevant to the dental curriculum in English language. A specific set of guidelines were given to the experts that included formulating questions with unambiguous answers, and involving core concepts of various dental specialties. The set entailed 3 binary questions with yes/no; right/wrong; true/false as answers, 3 multiple choice questions or one word answer questions, and 3 open-ended questions with descriptive answers. The rationale behind choosing these types of questions was incorporation of the variety of questions usually used within the dental curriculum to test the students. Hence, 27 questions were invited from each specialty, totalling to 243 questions from the nine dental specialties. Each question was validated through discussion between the three experts and any duplicate question was substituted. The subject experts labelled the difficulty level of each question as Easy, Medium, and Hard.

2.2 Generation of answers:

The set of questions were entered on the web version of ChatGPT 3.5 (Mar14 version) at http://chat.openai.com by a single investigator to ensure consistency. Separate chats were created for each specialty and following prompt was entered to ensure specificity and appropriateness of the answers generated: “Please answer the following questions of (name of Dental Specialty). Be specific and incorporate any standard guidelines applicable”. All the answers generated by the chatbot were tabulated specialty-wise and sent for evaluation.

2.3 Performance evaluation:

Three subject experts from each specialty rated all the answers generated by ChatGPT for that specialty and a mean score was obtained. Additionally, 3 open-ended questions with descriptive answers were randomly selected from each specialty and sent to 2 heads of institutions for rating. The answers generated by ChatGPT were rated in terms of Accuracy (6-point Likert Scale with following description 1: Completely incorrect, 2: More incorrect than correct, 3: Approximately equal correct and incorrect, 4: More correct than incorrect, 5: Nearly all correct, and 6: correct; Completeness (3-point Likert Scale with description as 1: Incomplete (Addresses some aspects of the question, but significant parts are missing), 2: Adequate (Addresses all aspects of the question and provides the minimum amount of information required to be considered complete), 3: Comprehensive (Addresses all aspects of the question and provides additional information or context beyond what was expected); and Relevance in relation to the length of answer generated was tested on a 5-point Likert Scale ranging from 1: Very irrelevant ( Lengthy answer with no relevance to the question asked), 2: Irrelevant, 3: Quite relevant, 4: Relevant, to 5: Very relevant (Concise and to-the point answer). Completely incorrect (accuracy = 1) answers were not graded for completeness, and incomplete answers were not graded for relevance. (Graded as 0 in the scale). The scales used in the present study was adapted from a previously conducted study of similar nature.⁶

2.4 Statistical analysis:

The average score for each subject was descriptively (mean, standard deviation) listed and then compared based on difficulty levels (easy, medium and hard), on the category (accuracy, completeness, relevance) and type of question (multiple-choice questions, binary and descriptive) using t-test. Mann-Whitney U or Kruskal Wallis testing was done when comparing between the groups. Responses for selected questions graded by faculty and the deans were compared using paired sample t-test. The comparisons were considered significant if p < 0.05 at 95% CI. All statistical analysis was performed using SPSS software version 21.0 (SPSS Inc).

A total of 243 questions were evaluated, 27 each from the 9 specialties. The list of questions formulated for each specialty along with their AI-generated answers has been provided as a supplementary file. [Appendix 1]

3.1 Overall performance:

The mean and median for accuracy of AI-generated answers (n = 243) was found to be 4.61 (SD 1.575) and 5.33 respectively. Mean value for completeness was 2.01 (SD 0.793) and median of 2.33 was noted. Mean score of 3.13 (SD 1.590) and median of 3.67 was obtained for relevance of the AI-generated answers. 11 answers were rated as completely inaccurate by all three experts (mean accuracy score = 1). These questions along with the ChatGPT generated answers are listed in the supplementary file. [Appendix 2] 49 answers were rated completely inaccurate by at least one subject expert. 68 AI-generated answers were rated as completely accurate (mean accuracy score = 6) by all three subject experts.

3.2 Specialty-wise performance:

AI-generated answers were found to be the most accurate, complete, and relevant for OMDR [mean accuracy 5.63 (SD 0.854), completeness 2.36 (SD 0.497), relevance of 4.04 (SD 1.149)] whereas the least accurate results were observed for orthodontics-related questions. [mean accuracy 3.36 (SD 1.821), completeness 1.28 (SD 0.918), relevance 1.68 (SD 1.743)]. [Figure 1] The performance rating for all dental specialties is listed as Table 1.

Table 1

Dental specialty-wise performance of ChatGPT
Specialty	Rating	n	Minimum	Maximum	Mean	Std. Deviation
Oral Pathology	Accuracy	27	1	6	4.41	1.701
	Completeness		0	3	1.94	.801
	Relevance		0	5	3.10	1.622
Oral and Maxillofacial Surgery	Accuracy	27	2	6	5.19	1.099
	Completeness		1	3	2.20	.483
	Relevance		1	5	3.32	1.256
Periodontics	Accuracy	27	1	6	4.67	1.720
	Completeness		0	3	2.10	.895
	Relevance		0	5	3.15	1.567
Pedodontics	Accuracy	27	2	6	4.58	.932
	Completeness		1	3	2.11	.462
	Relevance		0	5	3.43	1.065
Oral Medicine and Radiology	Accuracy	27	2	6	5.63	.854
	Completeness		1	3	2.36	.497
	Relevance		0	5	4.04	1.149
Conservative Dentistry and Endodontics	Accuracy	27	1	6	4.21	1.558
	Completeness		0	3	1.80	.813
	Relevance		0	5	2.46	1.491
Prosthodontics	Accuracy	27	1	6	4.47	1.693
	Completeness		0	3	2.02	.947
	Relevance		0	5	3.16	1.794
Orthodontics	Accuracy	27	1	6	3.36	1.821
	Completeness		0	3	1.28	.918
	Relevance		0	5	1.68	1.743
Public Health Dentistry	Accuracy	27	2	6	5.02	1.536
	Completeness		1	3	2.31	.666
	Relevance		1	5	3.80	1.344

3.3 Difficulty-wise performance:

35.4% (n = 86) of the questions were easy, 40.7% (n = 99) were of medium level, and 23.9% (n = 58) were hard. The distribution of difficulty level in respect to different dental specialties has been depicted in the figure. [Figure 2] An example of easy question from Orthodontics was “ What are clear aligners?”, medium was “ What is wagon wheel effect in straight wire orthodontic appliance?” and of hard was “What is Eastman correction in orthodontic cephalometry?”. The mean score for easy questions was found to be the highest in terms of accuracy [mean 5.03 (SD 1.475)], completeness [mean 2.15 (SD 0.760)], and relevance [mean 3.48 (SD 1.437)]. Medium and hard questions were almost of comparable accuracy, completeness, and relevance.

3.4 Performance as per the type of questions:

ChatGPT-generated answers for open-ended questions were found most accurate with mean accuracy 4.75 (SD 1.200), followed by binary questions [mean 4.71 (SD 1.680)] and multiple-choice questions [mean 4.39 (SD 1.778)]. Completeness and Relevance of the answers also followed a similar trend with the highest rating received for descriptive answers. [mean completeness 2.05 (SD 0.628); mean relevance 3.18 (SD 1.355)]. Most accurate answers to binary questions was seen for OMFS whereas OMDR received most accurate answers to multiple-choice and open-ended questions. The evaluation of AI-generated responses across all dental specialties in relation to the type of questions entered has been depicted in the table. [Table 2]

Table 2

Performance as per the type of questions across various specialties.
	Type of question	Accuracy	Completeness	Relevance
Oral Pathology	Binary	4.19 (2.174)	1.96 (1.02)	3.26 (1.935)
	MCQ	4.7 (1.947)	2.15 (0.852)	3.48 (1.651)
	Descriptive	4.33 (0.85)	1.7 (0.455)	2.56 (1.247)
Oral & Maxillofacial Surgery	Binary	5.85 (0.338)	2.19 (0.377)	3.59 (0.909)
	MCQ	5.26 (1.234)	2.33 (0.527)	3.48 (1.345)
	Descriptive	4.44 (1.080)	2.07 (0.547)	2.89 (1.472)
Periodontics	Binary	3.78 (2.236)	1.70 (1.160)	2.11 (1.590)
	MCQ	4.63 (1.628)	2.19 (0.973)	3.26 (1.801)
	Descriptive	5.59 (0.324)	2.41 (0.147)	4.07 (0.147)
Pedodontics	Binary	4.70 (1.369)	2.15 (0.580)	3.48 (1.365)
	MCQ	4.41 (0.830)	2.04 (0.455)	3.52 (1.015)
	Descriptive	4.63 (0.455)	2.15 (0.377)	3.30 (0.873)
Oral Medicine & Radiology	Binary	5.52 (0.852)	2.22 (0.236)	4.19 (0.747)
	MCQ	5.59 (1.222)	2.30 (0.772)	3.67 (1.818)
	Descriptive	5.78 (0.333)	2.56 (0.289)	4.26 (0.434)
Conservative Dentistry & Endodontics	Binary	4.37 (1.611)	1.96 (0.754)	2.22 (1.364)
	MCQ	3.30 (1.837)	1.41 (1.103)	2.00 (1.922)
	Descriptive	4.96 (0.564)	2.04 (0.309)	3.15 (0.915)
Prosthodontics	Binary	4.89 (1.599)	2.26 (0.954)	3.81 (1.564)
	MCQ	4.85 (1.425)	2.26 (0.778)	3.70 (1.359)
	Descriptive	3.67 (1.908)	1.56 (1.014)	1.96 (1.933)
Orthodontics	Binary	3.26 (1.786)	1.15 (0.603)	1.00 (1.323)
	MCQ	3.00 (2.115)	1.11 (1.179)	1.70 (2.065)
	Descriptive	3.81 (1.651)	1.59 (0.909)	2.33 (1.691)
Public Health Dentistry	Binary	5.81 (0.242)	2.63 (0.423)	4.44 (0.745)
	MCQ	3.74 (2.165)	1.93 (0.969)	2.89 (1.929)
	Descriptive	5.52 (0.294)	2.37 (0.200)	4.07 (0.324)

3.5 Evaluation of responses by Heads of Institutions:

Two heads of institutions evaluated a set of randomly chosen 27 open-ended questions, 3 from each dental specialty. The mean accuracy was found to be 5.13 (SD 0.804) and a median of 5.50 was noted. Similarly, mean 2.33 (SD 0.519) and median 2.50 was obtained for completeness, and a mean of 3.89 (SD 1.204) and median of 4.00 was calculated for relevance of the AI-generated answers. The dental specialty-wise rating revealed most accurate answers for Periodontics [mean 5.83 (SD 0.289)] and least accurate answers were observed for Prosthodontics [mean 4.17 (SD 1.155)]. [Table 3]

Table 3

Dental specialty-wise performance rating by heads of institutions.
Specialty	Rating	n	Minimum	Maximum	Mean	Std. Deviation
Oral Pathology	Accuracy	3	5	6	5.17	.577
	Completeness		3	3	2.67	.289
	Relevance		4	4	3.83	.289
Oral and Maxillofacial Surgery	Accuracy	3	5	6	5.00	.500
	Completeness		2	3	2.17	.289
	Relevance		4	5	4.17	.289
Periodontics	Accuracy	3	6	6	5.83	.289
	Completeness		3	3	2.50	.000
	Relevance		5	5	4.67	.289
Pedodontics	Accuracy	3	5	6	5.33	.289
	Completeness		3	3	2.83	.289
	Relevance		4	5	4.50	.866
Oral Medicine and Radiology	Accuracy	3	5	6	5.50	.500
	Completeness		2	3	2.67	.577
	Relevance		4	5	4.50	.866
Conservative Dentistry and Endodontics	Accuracy	3	5	6	5.50	.500
	Completeness		2	3	2.33	.577
	Relevance		4	5	4.00	.866
Prosthodontics	Accuracy	3	4	6	4.17	1.155
	Completeness		2	2	1.67	.289
	Relevance		1	5	2.50	1.803
Orthodontics	Accuracy	3	3	6	4.33	1.258
	Completeness		1	3	2.00	.866
	Relevance		0	4	2.67	2.309
Public Health Dentistry	Accuracy	3	5	6	5.33	.764
	Completeness		2	3	2.17	.289
	Relevance		4	5	4.17	.577

3.6 Agreement between Subject experts and Head of Institutions:

The rating of the 27 randomly selected questions by the heads of the institution were compared with the rating given by the faculty of the respective specialty. The average accuracy rating given by the heads of institution was 5.13 (SD 0.84) and by the faculty was 4.7 (SD 1.463) which was not statistically significant (p = 0.141) The difference in the rating meant that the faculty and the heads of the institution did not equally agree with the accuracy of the answers provided by ChatGPT. However, both the heads of the institutions and the subject experts rated similarly for the completeness (p = 0.036) and relevance of the answers (p = 0.022).

ChatGPT has garnered significant attention owing to its user-friendly platform and interactive interface. Within a relatively short timeframe, it has gained substantial traction and is being widely employed in the healthcare sector for diverse applications. However, the ease of access to the model exposes it to potential misuse and any misinformation provided especially in the context of healthcare can have detrimental effects. Moreover, as the platform is open to the general public, patients seeking dental advice might utilize the chatbot to address their dental-related concerns. Consequently, it becomes imperative to subject the model to comprehensive testing, thus identifying the potential limitations and establishing necessary precautions for its application in addressing dental queries. Hence, this study aims to evaluate the performance of ChatGPT in terms of accuracy, completeness, and relevance of the answers generated by the language model in response to clinical and non-clinical dental-related questions.

In terms of accuracy, a relatively high mean score of 4.61 (SD 1.575), and median of 5.33, was noted indicating adequate correctness of the information provided in relation to the dental queries. Completeness, as assessed on a 3-point Likert yielded a mean score of 2.01 (SD 0.793) and median 2.33. Hence, while ChatGPT responses were informative, there is a scope for improvement in incorporating comprehensive details. Relevance received a mean score of 3.13 (SD 1.590) and median of 3.67. Despite the utilization of prompts in each chat session to guide ChatGPT into producing specific results with standard guidelines, a general trend that was observed across the various specialties was wordiness of the answers generated with no relevance to the questions asked.

Specialty-wise, the model scored highest in OMDR [mean accuracy 5.63 (SD 0.854), completeness 2.36 (SD 0.497), relevance of 4.04 (SD 1.149)]. OMDR constitutes clinical, and radiographic diagnosis as well as the medicinal management of all common oral and maxillofacial conditions. Hence, this finding could be extrapolated to the application of ChatGPT as a clinical decision-support system. However, further testing with multiple clinical case scenarios needs to be performed to support its translatability into clinics. The language model scored least in Orthodontics [mean accuracy 3.36 ( SD 1.821), completeness 1.28 (SD 0.918), relevance 1.68 (SD 1.743)]. The poor performance could be attributable to the complexity and specificity of orthodontics-related inquiries and technical nature of the orthodontic concepts requiring precise judgment. Moreover, there is a probability that the dataset used for training ChatGPT might have limited representation of orthodontics-related content as opposed to other dental specialties. Hence, robust training of the model is required in this sector.

In relation to the difficulty level, ChatGPT performed better on easier questions [mean accuracy 5.03 (SD 1.475)] which is suggestive of limited effectiveness of the model in handling medium or difficult dental related queries. Although the difference was not very substantial and the model dealt with hard or medium-level questions efficiently. Regarding the types of questions entered, ChatGPT was most accurate in answering open-ended questions [mean 4.75 (SD 1.200)]. Furthermore, superior results were noted for descriptive answers rated by heads of institutions with a mean accuracy of above 5 for each specialty except for Prosthodontics and Orthodontics. Hence, it can be inferred that ChatGPT demonstrates higher proficiency in generating subjective responses as opposed to objective ones. However, it is crucial to acknowledge the potential overestimation of the accuracy scores in open ended questions that is attributable to several possible explanations when compared with multiple-choice or binary questions offering limited alternatives.

During prompting, few observations were made, including occasional loading errors that necessitated regenerating the response or refreshing the page. Notably, a warning message was displayed at the bottom of the webpage indicating the possibility of ChatGPT producing inaccurate information about people, places, or facts. This reflects the model’s limitations in dealing with eponyms and proper nouns. For instance, in the field of OMFS, when asked about the name of procedure where zygomatic arch is fractured to prevent Temporomandibular Joint dislocation, ChatGPT generated the response “zygomatic arch osteotomy”, which is indeed the actual technique used. However, the correct answer should have been Dautrey’s procedure. Furthermore, the model was sensitive to the prompts entered and minor rephrasing produced entirely different responses.

Additionally, there were instances where ChatGPT exhibited “hallucination effect” generating completely inaccurate and irrelevant responses, noted previously in other studies.⁷ This was observed in questions related to Conservative Dentistry and Endodontics such as “condition exhibiting radiographic picture of Bull’s eye” and “J-shaped radiolucency”, where Central Giant Cell Granuloma and Periapical Cemento-osseous Dysplasia were produced as answers. Hence, a notable limitation was the lack of precautionary warning accompanying the model’s responses, and the authoritativeness with which the model presents the answer which could be potentially misleading. Therefore, it is imperative that the model exercises caution when uncertain and avoid attempting to answer rather than the providing misinformation.

ChatGPT 3.5 is a relatively recent language model and only few studies have been conducted for assessing its feasibility. In a similar study as the present one, accuracy of the answers generated by ChatGPT in response to 284 medical questions was tested. Mean accuracy of 4.4 and median of 5 was determined. The study also noted an improvement in the scores when reassessed after 17 days, suggesting the continuously updating nature of the language model.⁶ Further, a study evaluated the performance of ChatGPT on the United States Medical Licensing Exams (USMLE). It was observed that the model performed at or near the passing threshold with high level of concordance in its explanations.⁸ These findings, along with the findings observed in the present study favours the use of ChatGPT in medical and dental education as well as clinical decision making.

The launch of ChatGPT has elicited varied response within the scientific community regarding its potential applications in healthcare, accompanied by concerns regarding the misuse of technology. Certain ethical and legal concerns may arise in this context, including plagiarism, authorship concerns, infringement of copyright laws, and accountability of the generated content.⁹ There are also certain limitations to ChatGPT that need to be acknowledged. As a text-based AI language model, ChatGPT 3.5 is unable to process visual information, rendering it unsuitable for image-related tasks. This is however addressed in the updated version ChatGPT 4.0 which incorporates image-based prompts. Secondly, the model’s generated output highly depends on the input phrase and minor tweaks, or rephrasing could change the response substantially. As stated earlier, hallucination effect is a concern whereby plausible-sounding text might be generated with no relevance to the original question. Moreover, the reference source used for generating information is conspicuous, which raises concerns regarding the validity of responses. ChatGPT has a knowledge cutoff dated September 2021. Therefore, the model might not produce accurate information regarding latest technological updates. Additionally, as with any AI-based technology, significant concerns surrounding user privacy and cybersecurity mandates the need for adequate safeguarding mechanisms in place to protect user’s data and prevent potential misuse.²

Limitations of the study

Although the study was carried on a large multi-centric scale, a moderate dataset of questions specific to the specialties was utilized for testing. We propose conducting comprehensive specialty-wise testing with a substantial dataset specifically focused on dental inquiries of that specialty. Secondly, the questions provided were self-validated by subject experts and not subjected through an audit externally. Thirdly, the rating of the responses was subjective and could have inter-observer variations. To mitigate potential biases, all AI-generated responses of a specialty were rated by three experts, and a mean score was obtained. Further, two heads of institutions re-assessed the descriptive questions from each specialty. The medium used for questions and answers was English, which limits the obtained accuracy data for English speakers only.

ChatGPT showcased promising performance in terms of accuracy, completeness, and relevance of the responses generated to dental-related questions. While the study promotes the use of the language model as a valuable resource to retrieve information regarding dental queries, caution should be exercised given its potential limitations. Complete reliance on technology for AI-generated information is not advisable and any dental health-related queries must be brought to qualified dental health care professionals for proper resolution. Future improvements should inculcate training of the model in specialized areas of dentistry and inclusion of precautionary warnings to mitigate potential errors and dissemination of misinformation.

Sources of Funding: None

Conflicts of Interest: None

Author Contributions:

Priyanshu Kumar Shrivastava: Contributed to conception, design, data acquisition, analysis, and interpretation, drafted and critically revised the manuscript.

Dr. Arpita Rai: Contributed to conception, design, data acquisition, analysis, and interpretation, drafted and critically revised the manuscript.

Dr. Ranjit J Injety: Contributed to design, data analysis, and interpretation, drafted and critically revised the manuscript.

Dr. Sanjay Singh: Contributed to design, data acquisition, and analysis, critically revised the manuscript.

Dr. Ashish Jain: Contributed to design, data acquisition, and analysis, critically revised the manuscript.

Dr. Amit Vasant Mahuli: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Anita Parushetti: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Anka Sharma: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr Arvind Sivakumar: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Bindiya Narang: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Farheen Sultan: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Gaurav Shah: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Gokul Sridharan: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Jeyaseelan Augustine: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Madhu Ranjan: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Neelam Singh: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Nishant Mehta: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Nishat Sultan: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Panchali Batra: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Sangita Singh: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr Sapna Gokul: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Sayani Roy: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Shabina Sachdeva: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Sharmila Tapashetti: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Simpy Amit Mahuli: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr Sridhar Kannan: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Sugandha Verma: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Tushar: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Vijay Yadav: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Vivek Gupta: Contributed to data acquisition, and analysis, critically revised the manuscript.

Dr. Deborah Sybil: Contributed to conception, design, data acquisition, analysis, and interpretation, drafted and critically revised the manuscript.

All authors gave their final approval and agree to be accountable for all aspects of the work.

Shan T, Tay FR, Gu L. Application of artificial intelligence in dentistry. Journal of dental research. 2021;100(3):232-44.
Introducing ChatGPT. c2015 – 2023. OpenAI [accessed on 15/05/2023]. Available from: https://openai.com/blog/chatgpt.
Sallam M, Salim N, Barakat M, Al-Tammemi A. ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J.;2023;3(1):e103.
Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. Journal of Esthetic and Restorative Dentistry. 2023;1-5.
Huang H, Zheng O, Wang D, Yin J, Wang Z, Ding S, Yin H, Xu C, Yang R, Zheng Q et al 2023. ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model. [accessed on 18/05/2023]: arXiv preprint arXiv:2304.03086. https://doi.org/10.48550/arXiv.2304.03086
Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, Chang S, Berkowitz S, Finn A, Jahangir E et al. 2023. Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model. Res Sq [Preprint].2023; rs.3.rs-2566942.
Sallam, M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare. 2023; 11;887.
Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health.2023;2(2): e0000198.
Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications,advantages, limitations, future prospects, andethical considerations.Front. Artif. Intell. 2023;6:1169595.

There is no duality of interest

SupplementaryFile.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Performance of ChatGPT in Dentistry: Multi-specialty and multi-centric study

Status:

Version 1

Abstract

Figures

1. INTRODUCTION

2. MATERIAL AND METHODS

2.1 Questions Dataset:

2.2 Generation of answers:

2.3 Performance evaluation:

2.4 Statistical analysis:

3. RESULTS

3.1 Overall performance:

3.2 Specialty-wise performance:

3.3 Difficulty-wise performance:

3.4 Performance as per the type of questions:

3.5 Evaluation of responses by Heads of Institutions:

3.6 Agreement between Subject experts and Head of Institutions:

4. DISCUSSION

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1