Effectiveness of Various General large language models in Clinical Consensus and Case Analysis in Dental Implantology: A Comparative Study

doi:10.21203/rs.3.rs-4613092/v1

Download PDF

Research Article

Effectiveness of Various General large language models in Clinical Consensus and Case Analysis in Dental Implantology: A Comparative Study

https://doi.org/10.21203/rs.3.rs-4613092/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

This study evaluates and compares ChatGPT-4.0, Gemini 1.5, Claude 3, and Qwen 2.1 in answering dental implant questions. The aim is to help doctors in underserved areas choose the best LLMs(Large Language Model) for their procedures, improving dental care accessibility and clinical decision-making.

Methods

Two dental implant specialists with over twenty years of clinical experience evaluated the models. Questions were categorized into simple true/false, complex short-answer, and real-life case analyses. Performance was measured using precision, recall, and Bayesian inference-based evaluation metrics.

Results

ChatGPT-4 exhibited the most stable and consistent performance on both simple and complex questions. Gemini performed well on simple questions but was less stable on complex tasks. Qwen provided high-quality answers for specific cases but showed variability. Claude-3 had the lowest performance across various metrics. Statistical analysis indicated significant differences between models in diagnostic performance but not in treatment planning.

Conclusions

ChatGPT-4 is the most reliable model for handling medical questions, followed by Gemini. Qwen shows potential but lacks consistency, and Claude-3 performs poorly overall. Combining multiple models is recommended for comprehensive medical decision-making.

large language models

artificial intelligence

dental implantology

clinical decision-making

case analysis

Artificial intelligence is the capability of machines or computer systems to perform tasks that typically require human intelligence.(1) It involves simulating human-like intelligence in machines through tasks such as logical reasoning, learning, problem-solving, and decision-making. In 2022, generative softwore like ChatGPT, based on large (LLMs), was officially launched. These models aim to simulate human conversations, understand the meanings of words and phrases like humans, and use this understanding to generate new content based on training data.(2) LLMs have demonstrated their effectiveness in semantic parsing by converting natural language into logical forms to complete complex reasoning tasks, enhancing their effectiveness in dental medical Q&A systems and robotic planning tasks(3). This highlights the potential and transformative capabilities of LLMs in natural language processing(NLP) and dentistry.

During clinical consensus and case analysis, models like ChatGPT have already shown some clinical decision-making capabilities. Stephanie Cabral et.al used 20 clinical cases to evaluate the ability of LLMs to handle medical data and demonstrate clinical reasoning, comparing them with attending physicians and residents. The results showed that large models were superior to doctors in handling medical data and using a recognizable framework measured by the R-IDEA score for clinical diagnosing. Other clinical decision-making results showed no statistical significant difference between doctors and chatbots(4). In providing medical information, Yazid K Ghanem et.al. evaluated the content and quality of medical information on acute appendicitis generated by ChatGPT, Bard (Gemini), and Claude2. The results showed that various general large models scored high in quality assessment of appendicitis medical information, with overall readability far exceeding public recommendation levels(5). These findings demonstrate that the potential of various LLMS like ChatGPT in medicine has been validated.

In the field of dentistry, Hossein Mohammad-Rahimi et. al tested the validity and reliability of ChatGPT, Bard, and Bing on endodontics-related questions, and all three chatbots achieved acceptable reliability levels (Cronbach's alpha > 0.7)(6). Arman Danesh et.al asked ChatGPT3.5 and ChatGPT4 questions from three different sources: INBDE Bootcamp, ITDOnline, and a board-style question list provided by the National Dental Examination Board(7).This demonstrates the potential of various general models as tools for daily dental diagnosis and research.

The above studies invited relevant experts to evaluate the results of various large models on specific questions using different forms of Likert scales. However, the information output by various large models, as part of medical information, needs to meet the requirements of carrying medical text information (completeness, practicality, reliability) and the requirements of medical text attributes (clarity, simplicity, logic, authority, and adherence to relevant legal and medical ethical principles). Additionally, as an extension of the NLP field, the traditional evaluation criteria for various artificial intelligence are precision, recall, and the F1 score calculated from both. These objective indicators were not reflected in the above studies, leading to a lack of comprehensive quantitative evaluation of model assessment, which is an area that needs further exploration.

In the field of dental implants, the auxiliary capabilities of various large models have not been explored.Dental implants, as a branch of dentistry, combine complex knowledge from oral surgery, periodontics, and prosthetics, requiring professional doctors to have extensive medical knowledge and excellent clinical decision-making abilities(8–10). However, global oral healthcare resources, especially for doctors capable of implant restoration, are extremely unevenly distributed.The emergence of general large models provides an opportunity to break this imbalance.Various general large models such as Gemini, Claude3, and Qwen, trained on different web corpora, have not yet been explored for their capabilities in the field of dental implants.The purpose of this study is to quantitatively evaluate and compare the performance of various general large models in the field of dental implants and the performance of the same large model on questions of different difficulty levels, thereby comprehensively exploring the ability of existing general large models to transition from clinical consensus to scenario applications in the field of dental implants.

2.1 Research Design

This study quantitatively evaluates and compares the qualitative differences in capabilities among ChatGPT-4.0, Gemini 1.5, Claude 3, and Qwen 2.1, in answering a variety of questions. We invited two dental implant specialists with over fifteen years of clinical experience each, possessing extensive knowledge and skills in oral implants. The study was conducted from February 2024 to May 2024. All questions were categorized into simple questions (i.e., true/false or numerical fill-in-the-blank questions), complex questions (requiring brief answers covering multiple aspects), and case analyses (real-life cases) to be posed to different models. The materials for the questions were sourced from the International Team for Implantology (ITI)Clinical Guidebook Series for Oral Implant Treatment, extracting ten questions, five of which were true/false, and five were complex short-answer questions. Additionally, six completed real-life cases from daily clinical practice were selected, with questions based on the chief complaint, medical history, physical examination, and auxiliary examinations. Questions focused on diagnosis, treatment plans for the chief complaint, and the most suitable treatment plan that meets patient requirements. (Due to rule restrictions, Gemini could not be used for specific case analyses.) Each model was asked simple questions 20 times in new windows, while complex questions and case analyses were posed 20 times.The specific process is shown in Fig. 1.

2.2 Question Design and Evaluation Criteria

2.2.1 Simple Questions

Simple questions are divided into true/false questions and numerical fill-in-the-blank questions. These questions aim to test the model's understanding and accuracy of clear, single answers. For true/false questions, given their clear judgment points, the study used binary (yes/no) questions with specific prompt formats (only yes or no answers) for the models to answer. For numerical answers or numerical range questions, due to the rigor of medical issues, the model's response must match the answer in the guidebook or fall within the range of values provided in the guidebook. For single-element fill-in-the-blank questions, the model must provide an answer conceptually identical to the intended meaning to be considered correct.

2.2.2 Complex Questions and Case Analyses

For complex questions and case analysis problems, their answers are multi-faceted. The standard answers in the guidelines and the various elements in the professional treatment answers of case analyses can be annotated.Based on previous research on medical texts, to ensure that the requirements of carrying medical information and the attributes of medical text structure are met, the scoring criteria for complex questions and Case Analyses need to cover completeness(11, 12)、reliability(12, 13)、clarity(12, 14), simplicity(12, 15)、logic(12, 16)、authority(12, 17), and adherence to relevant legal and medical ethical principles.For text credibility, it is necessary to consider the evaluation system based on Bayesian inference commonly used in evaluating artificial Intelligence(AI) generated texts, namely precision and recall(18).Given the complexity of medical scenarios, we referenced the EQIP(Expanded Quality Information Patient) expanded scale for assessing the quality of patient information documents, which includes content, structure, and identification data(19).A set of scoring criteria for medical texts generated by LLMs was developed based on these considerations, as shown in Table 1.

Table 1

Scoring rules for medical texts output by large language models
Evaluation Criteria	Full Score
Text Content
Completeness (Should provide comprehensive information)	3 points
Reliability (Whether the answers given are reliable)	3 points
Text Structure
Practicality (Whether the answer provides detailed explanations for practical situations)	1 point
Authority (Whether authoritative citations are provided when quoting answers)	1 point
Compliance with Laws and Ethical Principles (Whether relevant experts should be consulted/Whether medical ethics should be consulted when dealing with complex issues)	1 point
Clarity (Whether there are any ambiguities or contradictions)	1 point
Logic	1 point
Simplicity	1 point
Total Score

This evaluation form is divided into two parts: text content and text structure. Evaluators need to first annotate and break down the key points of the standard answers from the guidelines and the responses from various LLMs. Then, they calculate TP (True Positives) = factors in the model's answers that match the standard answers; FP (False Positives) = new factors added by the model that are not mentioned in the standard answers; FN (False Negatives) = factors mentioned in the standard answers but omitted in the model's answers. This expresses the comprehensiveness of the model's responses. Completeness corresponds to the model's recall rate, which is the number of FN (cases where ChatGPT did not provide correct medical advice but was actually needed). Precision (with a maximum score of 3) is calculated as TP (fully matched information) / (TP + FN) * 3, representing the accuracy of the model's provided medical advice or answers that match professional guidelines or are considered accurate. Reliability is expressed by precision (with a maximum score of 3), calculated as TP / (TP + FP) * 3, indicating how much of the model's advice aligns with the guidelines.

The study compares the key points of complex questions in the guidelines and case(In the actual medical records, the personal information of patients was anonymized, retaining only the chief complaint, present illness history, past medical history, family history, physical examination, and auxiliary examination parts that reflect the patient's condition. These were input into various large-scale models, requesting them to output diagnoses, treatment plans, and optimal treatment strategies.) analyses with the diagnoses, treatment plans, and suitable treatment plans provided for the patients, matching the key points of the model's answers with those in the guidelines.

Text completeness indicates the elements mentioned in the standard guidelines but not omitted by the model. Reliability represents the proportion of factors mentioned by the model that are consistent with the guidelines. Practicality evaluates whether the model can provide valuable answers based on specific analysis of the current issue, such as whether the model can analyze specific problems when providing follow-up analysis. Authoritativeness checks whether the model can provide relevant authoritative citations when adding answers. Legal and ethical principles assess whether the model acknowledges its limitations as an AI and reminds users to consult relevant medical professionals. Clarity evaluates whether the output text is unambiguous, with appropriate sentence and word meaning. Logic assesses whether the model's response is well-organized. Accessibility checks whether the expressions used are free from obscure terms and long, complex sentences.The scoring proportions for each criterion in this study refer to the EQIP expansion scale, which balances the scores for medical content and text structure at 50% each.

The present study invited two dental implant experts to use the above rules to annotate the key points in the guidelines and compare them with the key points extracted from the anonymized responses of the models to determine precision and recall. They further reviewed the model's responses to complex questions and scored them based on their clinical experience and the patient treatment records, considering the diagnosis, chief complaint, and the most suitable treatment plan. The results were saved. Responses from models that completely misunderstood or hallucinated were scored as 0.

Through this design, this study ensures the scientific rigor and accuracy in evaluating the performance of various LLMs in the field of dental implantology.

2.4 Statistical Methods

2.4.1 Sample Size Calculation

To evaluate the adequacy of the sample sizes for each group, a preliminary study was conducted. This study involved a small-scale experiment using various large models on a simple question (12 data sets, with each group answering 3 questions). The expected average score was determined to be 9 out of a possible 12 points, with a standard deviation of 0.1. Using the general formula\(\text{N}=\text{Z}²{\sigma }²/\text{d}²\) (20)for estimating sample size, where Z for a 95% confidence level is 1.96, the standard deviation is 0.05, and the margin of error is 0.1, the minimum sample size was calculated to be 4.

For complex questions and case analyses (with a margin of error of 0.1 and a standard deviation of 0.2), similar small-scale experiments were conducted. The minimum sample size for each group of questions was found to be 16.Increasing the sample size improves the reliability and accuracy of statistical results, and fully considers the complexity and diversity of problem types. Therefore, the sample size for each group was uniformly set at 20.

Tables 2 and 3 present the simple and complex judgment questions provided to the large model.

Table 2

Presentation of simple and complex questions posed to different general large language models
Question ID	Question Content
1	What are the patient-identifying factors required for aesthetic implant restoration of anterior teeth?
2	How to obtain long-term soft tissue stability in aesthetic implant of anterior teeth?
3	What are the biological types of gum in the implant treatment area?
4	What are the requirements for transitional dentures during the healing period?
5	What are the types of barrier membranes used for bone defects around implants? What are the advantages of each?
6	How many millimeters from the adjacent tooth is required for the ideal placement of the implant in the proximal and distal aesthetic areas of the tooth?
7	How long after tooth loss is delayed dental implant placement typically performed?
8	What is the average depth of the alveolar crest square soft tissue (including gingival groove depth) around the implant?
9	What is the torque range required for immediate implantation to sustain the initial stability of a single tooth?"
10	Does insufficient keratinized gingival mucosa (KAM) play a decisive role in maintaining peri-implant soft tissue health while ensuring good oral hygiene?

Table 3

Presentation of Clinical Cases Posed to Different General Large Language Models
Case ID	Case Focus
1	Immediate implant placement: Diagnosis, treatment planning, therapeutic strategies
2	Maxillary sinus lift implant: Diagnosis, treatment planning, therapeutic strategies
3	GBR surgery: Diagnosis, treatment planning, therapeutic strategies
4	Immediate loading of implants: Diagnosis, treatment planning, therapeutic strategies
5	Early loading implant surgery: Diagnosis, treatment planning, therapeutic strategies
6	All-on−6 procedure for advanced periodontal disease: Diagnosis, treatment planning, therapeutic strategies

2.4.2 Consistency Test

To evaluate the scientific validity of the scoring method, this study first had two experts score the aforementioned complex questions and case analyses based on the evaluation criteria. The Intraclass Correlation Coefficient (ICC) test was used to compare the consistency of their scores across different types of questions.

2.4.3 Simple Questions

The accuracy rates of each model for different simple questions were statistically analyzed. To assess the accuracy of the generated responses by the large models, the exact match ratio for the total question set and each individual question was calculated along with their 95% confidence intervals (Wilson score interval). The t-test was then used to calculate the significance of differences between groups.

2.4.4 Complex Questions/Case Analyses

All answers were stored in an Excel spreadsheet (Microsoft). This study used the following methods:

1. Mann-Whitney U test to pairwise compare the responses of different models to complex questions or case analyses to determine if there were significant differences.

2. GraphPad Prism 8.0.1 (GraphPad Software, Inc.) was used for data visualization and graph plotting.

3.1 Consistency

The consistency test (as shown in Table 4) resulted in Intraclass Correlation Coefficients (ICC) of 0.965 and 0.797 for the two dentists' scores on complex questions and case analyses, respectively. This indicates that the average consistency of scores among multiple raters is very high. When evaluating complex questions and case analyses, the average scores given by several dentists with similar expertise levels show high consistency.

Table 4

Inter-Rater Reliability Analysis of Dentists' Ratings (ICC Analysis)
Analysis Type / Type	ICC	95% CI
Complex Problems
- Single Measure ICC (A,1)	0.396	0.162–0.904
- Average Measure ICC (A,K)	0.965	0.890–0.997
Case Analysis
- Single Measure ICC (A,1)	0.136	0.065–0.289
- Average Measure ICC (A,K)	0.797	0.633–0.911

3.2 Simple Questions

Tables 5 display the performance of each model on simple questions and the significance of the comparisons between models. Gemini Advanced achieved the highest accuracy rate at 0.80, while Qwen had the lowest accuracy rate at 0.60. The accuracy rates for ChatGPT-4 and Claude-3 were 0.74 and 0.72, respectively. The standard deviations for all four models were similar, ranging from 0.05 to 0.07.

Table 5

Accuracy and Significance Testing of Different Models on Simple Questions
Group Comparison		p-value
ChatGPT-4 vs Qwen		0.035
ChatGPT-4 vs Claude-3		0.752
ChatGPT-4 vs Gemini Advanced		0.316
Qwen vs Claude-3		0.074
Qwen vs Gemini Advanced		0.002
Claude-3 vs Gemini Advanced		0.187
Model	Accuracy	CL95% (Wilson Score Interval)
ChatGPT-4	0.74	0.604 to 0.841
Qwen	0.6	0.462 to 0.724
Claude-3	0.72	0.583 to 0.825
Gemini	0.8	0.670 to 0.888

When comparing the models, the differences between ChatGPT-4 and Gemini Advanced, as well as between ChatGPT-4 and Claude-3, were not significant. However, the differences between ChatGPT-4 and Qwen, as well as between Qwen and Gemini Advanced, were significant. Additionally, the difference between Qwen and Claude-3 was close to reaching significance.

3.3 Complex Questions

To evaluate the performance of various LLMs in solving complex tasks, this study conducted a comparative analysis of four major models: ChatGPT-4, Qwen, Claude-3, and Gemini. The evaluation criteria included average score, standard deviation, minimum value, 25th percentile, median, 75th percentile, maximum value, and standard error of the mean (SEM). These analysis results are summarized in Table 6 and the significance differences between groups are shown in Fig. 2. Compared to ChatGPT-4, Claude-3 had a p-value of 0.001, indicating a very significant difference, with Claude-3 performing significantly worse than ChatGPT-4. The p-value between Claude-3 and Gemini was 0.033, indicating a significant difference, with Gemini performing significantly better than Claude-3. The p-values for the remaining group comparisons were all greater than 0.05.Table 6 illustrates the performance of each large-scale model in tackling complex problems.

Table 6

The performance and significance of different models in diagnostic scores
Metric		Mean	Standard Deviation	Minimum	Median	Maximum
ChatGPT4	SASDPLLM	9.83	2.1	3	10	12
	SASTRLLM	9.28	2.41	3	10.3	12
	SASTSLLM	10.02	1.23	4.5	10	12
	SASCPMLM	7.99	1.95	5	10	12
Qwen	SASDPLLM	10.9	0.68	9.3	11	12
	SASTRLLM	9.07	2.24	5	9	12
	SASTSLLM	7.89	4.46	0	10	12
	SASCPMLM	7.05	2.88	0	10.5	12
Claude3	SASDPLLM	9.88	1.56	3	10	12
	SASTRLLM	9.24	2.41	3	10.3	12
	SASTSLLM	9.83	2.31	3	10	12
	SASCPMLM	5.64	2.28	2	5.4	10
Gemini	SASCPMLM	6.81	2.82	2	7.5	11.4

SASDPLLM = Statistical Analysis of Scores in Diagnostic Performance of Various Large Language Models

SASTRLLM = Statistical Analysis of Scores in therapeutic regimen of Various Large Language Models

SASTSLLM = Statistical Analysis of Scores in therapeutic schedule of Various Large Language Models

SASCPMLM = Statistical analysis of scores for complex problems of language models

The specific case analysis results are shown in Fig. 3. In the diagnostic module, across 120 trials, Claude-3 had an average score of 9.88 with a standard deviation of 1.56, while ChatGPT-4 had an average score of 9.83 with a standard deviation of 2.10, showing greater variability than Claude-3. The Qwen algorithm showed the highest average score of 10.90, combined with the smallest standard deviation of 0.68, indicating higher consistency in its scores. Statistical analysis of significance showed that Qwen had significant differences compared to Claude-3 (p = 0.001) and ChatGPT-4 (p = 0.002), while the p-values for comparisons with other groups were all greater than 0.05.

Regarding treatment plans and planning modules: Claude-3 had average scores of 9.41 and 9.83, with standard deviations of 2.43 and 2.31, respectively. ChatGPT-4 had average scores of 9.28 and 10.02, with standard deviations of 2.41 and 1.23, respectively. Qwen had average scores of 9.07 and 7.89, with standard deviations of 2.24 and 4.46, respectively. In terms of p-values, the scores given by different models for treatment plans and planning modules did not show significant differences.

In this study, a new evaluation method was proposed, which objectively considers the content and structure of AI-generated text when evaluating various models on medical questions. This evaluation method aligns with the commonly used AI evaluation metrics (precision and recall) to minimize the subjectivity of expert evaluations of different large models. Additionally, this study uses this evaluation method to assess the performance of various general large models in the field of dental implants across different question dimensions, thereby filling the research gap in the implantology field regarding the performance of these models.

When handling simple medical questions, the Gemini Advanced model performed the best, with the highest accuracy rate (0.80) and the smallest standard error (0.057).The p-value for comparison with ChatGPT-4 was 0.3158, indicating no statistically significant difference.Research by Masalkhi et.al. showed that Gemini AI and ChatGPT-4 performed comparably on simple ophthalmology questions(21).Additionally, studies by Andrew Mihalache et.al. demonstrated that Gemini and ChatGPT had accuracy rates of 74% and 79%, respectively, on simple medical questions(22, 23).Based on the analysis in this study, the performances of Claude-3, ChatGPT-4, and Gemini were roughly comparable, with no statistically significant differences between them (p-values greater than 0.05). Their accuracy rates were 0.74, 0.72, and 0.80, respectively. These studies indicate that these models have similar capabilities in handling simple medical questions. In contrast, Qwen performed the worst among these models, with an accuracy rate of only 0.60, showing statistically significant differences compared to all other models.

For handling complex medical questions, ChatGPT-4 demonstrated the highest average evaluation score (7.99 ± 1.95), indicating strong performance when faced with complex issues. Its median score was 8.3, suggesting that a few extreme answers lowered the average score, but overall, ChatGPT-4 showed good stability and consistency in its performance on complex questions. This result is consistent with the findings of Harriet Louise Walker et.al., who observed similar reliability in ChatGPT's responses to hepatopancreatobiliary (HPB) diseases(24).Compared to other models, ChatGPT-4's performance on complex questions was significantly better than Claude-3 (p = 0.001), with no significant difference compared to Gemini and Qwen (p > 0.05). Qwen's average accuracy (7.05 ± 2.88) was slightly lower than ChatGPT-4's, but its median score was the highest (8.5), indicating that Qwen provided more appropriate answers in certain cases, though with lower consistency. Claude-3 performed the worst on complex questions, with the lowest accuracy (5.64 ± 2.28) and significant differences compared to both ChatGPT-4 and Gemini (p = 0.001 and p = 0.033, respectively).

When handling specific medical case questions, the average scores of ChatGPT-4, Qwen, and Claude-3 were similar, with no significant differences (p > 0.05). Standard deviation analysis showed that ChatGPT-4 and Claude-3 had relatively consistent performance, while Qwen's performance was more variable.

In terms of diagnosis, Claude-3 demonstrated consistency (9.88 ± 1.56), whereas ChatGPT-4 and Qwen showed greater variability (9.83 ± 2.10 and 10.9 ± 0.68, respectively). Regarding treatment plans, Qwen showed stability (9.07 ± 0.29), while ChatGPT-4 and Claude-3 had a broader range of choices. For treatment planning, Qwen's data distribution was wider (7.89 ± 4.46), indicating higher personalization and adaptability, but also greater uncertainty.

Overall, in medical case questions, ChatGPT-4 performed the best, followed by Gemini. Qwen provided high-quality answers in certain cases but had less stability. Claude-3 performed relatively poorly across various aspects. Qwen excelled in diagnosis, possibly due to its efficient feature extraction capability. However, its performance in treatment decision-making and planning did not show the same advantage, with a large standard deviation reflecting instability in specific plan designs. Notably, p-values indicate significant differences in several comparisons. The p-values for Qwen versus Claude-3 and ChatGPT-4 in diagnosis were 0.001 and 0.002, respectively, indicating statistically significant differences in diagnostic performance. In contrast, the p-values for treatment plans and planning were all greater than 0.05, indicating no significant differences in these areas. This demonstrates that all the large models have some degree of clinical decision-making ability in the field of implants. Additionally, observations of the models' scores revealed that Qwen had a higher number of completely incorrect responses (scores of 0, including hallucinations and misunderstandings of cases) in treatment planning compared to the other two models.

This study has several limitations. The field of dental implants is complex and diverse. Since this study extracted guidelines from ITI(International Team for Implantology) and randomly selected real cases, it did not encompass all aspects of the implant field. Due to space constraints, the study did not delve into the specific score differences in the evaluation criteria for different questions where significant differences between models were observed. Furthermore, insights from multidisciplinary perspectives on pre-implant periodontal treatment and post-implant restoration could provide more comprehensive treatment recommendations. Such interdisciplinary integration is crucial for dental implant practice, especially in complex situations like insufficient bone volume or occlusal disorders.Lastly, the various large models did not adequately explain the technical key points and difficulties in treatment planning. This might be due to the training data for these models coming from online sources, which do not include deep, specialized information from the field of dental implants.

This study introduces a novel evaluation method, providing an objective assessment of LLMs in dental implant research. ChatGPT-4 demonstrates the most consistent performance, while Gemini and Qwen show variable results, and Claude-3 performs poorly. The findings highlight significant differences between models, recommending their use as auxiliary tools in medical decision-making, with a focus on integrated results from multiple models.

1. AI: Artificial Intelligence

2. LLM: Large Language Model

3. NLP: Natural Language Processing

4. ITI: International Team for Implantology

5. TP: True Positives

6. FP: False Positives

7. FN: False Negatives

8. EQIP: Expanded Quality Information Patient

This list should cover the abbreviations used in your document and provide their full forms for reference.

Ethics approval and consent to participate

The research protocol received evaluation and authorization from the Ethics Committee of the Zhejiang Provincial People's Hospital (No. QT2023249). All patient medical records used in this study were included with the informed consent of the patients.

Consent for publication

All authors have read and approved the final manuscript and consent to its publication.

Availability of data and materials

Not applicable

Competing interests

The authors declare that they have no competing interests.

Funding

Not applicable

Authors' contributions

Yuepeng Wu, as the first author, participated in the scoring and supervised all the procedures. Yukang Zhang was involved in the scoring and designed the study. Mei Xu and Chen Jin Zhi collected the clinical data. Yuchen Zheng, as a corresponding author, prepared the manuscript and interpreted the data. All authors critically discussed the results, revised the manuscript, and approved the final version.

Acknowledgements

I would like to extend my sincere gratitude to the dental experts at Zhejiang Provincial People's Hospital for their invaluable assistance in providing data evaluation and scoring. Their expertise and meticulous efforts have significantly contributed to the quality and reliability of this research.

Morandín-Ahuerma F. What is Artificial Intelligence? Int J Res Publ Rev [Internet]. 2022 [cited 2024 May 23];03(12):1947–51. https://ijrpr.com/uploads/V3ISSUE12/IJRPR8827.pdf.
Abd-alrazaq A, AlSaad R, Alhuwail D, Ahmed A, Healy PM, Latifi S et al. Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions. JMIR Med Educ [Internet]. 2023 Jun 1 [cited 2024 May 23];9(1):e48291. https://mededu.jmir.org/2023/1/e48291.
Huang H, Zheng O, Wang D, Yin J, Wang Z, Ding S et al. ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model. Int J Oral Sci [Internet]. 2023 Jul 28 [cited 2024 May 23];15(1):29. https://www.nature.com/articles/s41368-023-00239-y.
Cabral S, Restrepo D, Kanjee Z, Wilson P, Crowe B, Abdulnour RE et al. Clinical Reasoning of a Generative Artificial Intelligence Model Compared With Physicians. JAMA Intern Med [Internet]. 2024 May 1 [cited 2024 May 23];184(5):581. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2817046.
Ghanem YK, Rouhi AD, Al-Houssan A, Saleh Z, Moccia MC, Joshi H et al. Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitis. Surg Endosc [Internet]. 2024 May [cited 2024 May 23];38(5):2887–93. https://link.springer.com/10.1007/s00464-024-10739-5.
Mohammad-Rahimi H, Ourang SA, Pourhoseingholi MA, Dianat O, Dummer PMH, Nosrat A. Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics. Int Endod J [Internet]. 2024 Mar [cited 2024 May 23];57(3):305–14. https://onlinelibrary.wiley.com/doi/10.1111/iej.14014.
Danesh A, Pazouki H, Danesh K, Danesh F, Danesh A. The performance of artificial intelligence language models in board-style dental knowledge assessment. J Am Dent Assoc [Internet]. 2023 Nov [cited 2024 May 23];154(11):970–4. https://linkinghub.elsevier.com/retrieve/pii/S0002817723004580.
Greenstein G, Cavallaro J, Tarnow D. Dental Implantology: Numbers Clinicians Need to Know. Compend Contin Educ Dent Jamesburg NJ 1995. 2019;40:e1–26.
Shah KK, Sivaswamy V. Assessment of Knowledge on Implant Abutment and Platform Switching among Dental Specialists Practicing Implantology. J Long Term Eff Med Implants [Internet]. 2023 [cited 2024 May 23];33(1):31–7. https://www.dl.begellhouse.com/journals/1bef42082d7a0fdf,1e500c424a6ff4b7,3078424a4e6a02b2.html.
Fatani B, Almutairi ES, Almalky HA, Mubarki MI, Al-Safadi A. A Comparison of Knowledge and Skills Related to Up-to-Date Implant Techniques Among Prosthodontists, Periodontists, and Oral Surgeons: A Cross-Sectional Study. Cureus [Internet]. 2022 Oct 17 [cited 2024 May 23]; https://www.cureus.com/articles/113059-a-comparison-of-knowledge-and-skills-related-to-up-to-date-implant-techniques-among-prosthodontists-periodontists-and-oral-surgeons-a-cross-sectional-study.
Macrina FL. 212797. 1995 [cited 2024 May 23]. SCIENTIFIC INTEGRITY: AN INTRODUCTORY TEXT WITH CASES. https://repository.library.georgetown.edu/handle/10822/545773.
Sharma S. How to Become a Competent Medical Writer? Perspect Clin Res [Internet]. 2010 Mar [cited 2024 May 23];1(1):33. https://journals.lww.com/picp/fulltext/2010/01010/how_to_become_a_competent_medical_writer_.8.aspx.
Adams S. Under Construction: Reviewing and Producing Information Reliability on the Web [Internet]. 2006 [cited 2024 May 23]. https://repub.eur.nl/pub/7841/.
Collier R. A call for clarity and quality in medical writing. Can Med Assoc J [Internet]. 2017 Nov 20 [cited 2024 May 23];189(46):E1407–E1407. http://www.cmaj.ca/lookup/doi/10.1503/cmaj.171265.
Plavén-Sigray P, Matheson GJ, Schiffler BC, Thompson WH. The readability of scientific texts is decreasing over time [Internet]. 2017 [cited 2024 May 23]. http://biorxiv.org/lookup/doi/10.1101/119370.
Stuyt PMJ. Why don’t medical textbooks teach? lack Log Differ diagnosis. 2003;61(11).
Roger A, Aïmeur E, Rish I. Towards ethical multimodal systems [Internet]. arXiv; 2023 [cited 2024 May 23]. https://arxiv.org/abs/2304.13765.
Rijsbergen V. Information retrieval; 2nd ed.; Butterworth, 1978. J Libr [Internet]. 1979 [cited 2024 May 23];11:237. https://cir.nii.ac.jp/crid/1370572092821121024.
Charvet-Berard AI, Chopard P, Perneger TV. Measuring quality of patient information documents with an expanded EQIP scale. Patient Educ Couns [Internet]. 2008 Mar 1 [cited 2024 May 23];70(3):407–11. https://www.sciencedirect.com/science/article/pii/S0738399107004739.
Sullivan LM. Essentials of Biostatistics for Public Health. 第第 4 版. Jones & Bartlett Learning; 2022. p. 400.
Masalkhi M, Ong J, Waisberg E, Lee AG. Google DeepMind’s gemini AI versus ChatGPT: a comparative analysis in ophthalmology. Eye [Internet]. 2024 Feb 14 [cited 2024 May 23]; https://www.nature.com/articles/s41433-024-02958-w.
Mihalache A, Popovic MM, Muni RH. Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmol [Internet]. 2023 Jun 1 [cited 2024 May 23];141(6):589. https://jamanetwork.com/journals/jamaophthalmology/fullarticle/2804364.
Mihalache A, Grad J, Patil NS, Huang RS, Popovic MM, Mallipatna A et al. Google Gemini and Bard artificial intelligence chatbot performance in ophthalmology knowledge assessment. Eye [Internet]. 2024 Apr 13 [cited 2024 May 23]; https://www.nature.com/articles/s41433-024-03067-4.
Walker HL, Ghani S, Kuemmerli C, Nebiker CA, Müller BP, Raptis DA et al. Reliability of Medical Information Provided by ChatGPT: Assessment Against Clinical Guidelines and Patient Information Quality Instrument. J Med Internet Res [Internet]. 2023 Jun 30 [cited 2024 May 23];25:e47479. https://www.jmir.org/2023/1/e47479.

No competing interests reported.

Download PDF

Reviewers invited by journal
13 Aug, 2024
Editor invited by journal
04 Jul, 2024
Editor assigned by journal
04 Jul, 2024
Submission checks completed at journal
04 Jul, 2024
First submitted to journal
20 Jun, 2024

You are reading this latest preprint version

Effectiveness of Various General large language models in Clinical Consensus and Case Analysis in Dental Implantology: A Comparative Study

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

1. Introduction

2. Material and Methods

2.1 Research Design

2.2 Question Design and Evaluation Criteria

2.2.1 Simple Questions

2.2.2 Complex Questions and Case Analyses

2.4 Statistical Methods

2.4.1 Sample Size Calculation

2.4.2 Consistency Test

2.4.3 Simple Questions

2.4.4 Complex Questions/Case Analyses

3. Results

3.1 Consistency

3.2 Simple Questions

3.3 Complex Questions

4. Discussion

5. Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1