A Comparative Analysis of Large language Models on Clinical Questions for Autoimmune Diseases

doi:10.21203/rs.3.rs-4810651/v1

Download PDF

Research Article

A Comparative Analysis of Large language Models on Clinical Questions for Autoimmune Diseases

https://doi.org/10.21203/rs.3.rs-4810651/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

Artificial intelligence (AI) has made great strides. Our study evaluated the performance in delivering clinical questions related to autoimmune diseases (AIDs).

Methods

46 AIDs-related questions were compiled and entered into ChatGPT 3.5, ChatGPT 4.0, and Gemini. The replies were collected and sent to laboratory specialists for scoring according to relevance, correctness, completeness, helpfulness, and safety. Scores for three chatbots in five quality dimensions and the scores of the replies to the questions under each quality dimension were analyzed.

Results

ChatGPT 4.0 showed superior performance than ChatGPT 3.5 and Gemini in all five quality dimensions. ChatGPT 4.0 outperformed ChatGPT 3.5 or Gemini on the relevance, completeness or helpfulness in answering about the prognosis, diagnosis, or the report interpretation of AIDs. ChatGPT 4.0’s replies were the longest, followed by ChatGPT 3.5, Gemini’s was the shortest.

Conclusions

Our findings highlight ChatGPT 4.0 is superior to delivering comprehensive and accurate responses to AIDs-related clinical questions.

Large Language Models

Autoimmune Diseases

ChatGPT 4.0

Artificial intelligence covers a broad field of computer science and employs computational techniques to learn, understand, and produce human language content[1]. Contemporary Natural language processing (NLP) models, particularly large language models (LLMs), which were trained on an extensive pool of textual data derived from articles, books, and the internet, have progressed to generate more human-like responses[2, 3]. Application platform interface (API) empowered by LLMs, such as Chat GPT (Open AI) and Gemini (Google), which has garnered significant interest for its near-human-level or equal-to-human-level performance in cognitive tasks in diverse fields[4, 5].

Recently, ChatGPT attained a passing-level performance during the United States Medical Licensing Examinations[6], which facilitates the application of LLMs in the medical field. The performance of LLMs was evaluated in several medical fields. In clinical practice, LLMs may play a role in improving patient care and simplifying the workflow of physicians, and in medical education, on-demand interactive teaching by LLMs empowered Chat robots may facilitate understanding medical theories[7].

Autoimmune diseases (AIDs) are a spectrum of conditions elicited by the subvert of self-immunotolerance and attack of T cells and B cells to normal constituents of the host. The prevalence of AIDs is probably 5%-10% among the general population, and women are more susceptible than men[8–10]. Those diseases include systemic lupus erythematosus (SLE), systemic scleroderma, rheumatoid arthritis (RA), Sjögren’s syndrome, polyarteritis nodosa, and giant-cells vasculitis[11]. The clinical manifestations of AIDs vary, which can be organ-specific or systemic and non-organ-specific[8, 11]. Therefore, the diagnosis of AIDs is a major challenge for clinicians. Currently, there are still 10% of AIDs patients suffering from acute disease due to the disease flare-ups, infections, and acute organ failures[12–14].

The utilization of LLMs is being investigated for various applications in autoimmune diseases, including answering frequently asked questions, aiding in medication for patients, and potentially assisting in diagnosing these complex conditions[15–19]. However, the performance of LLMs in other areas of AIDs such as prevention and prognosis is unclear at present, and other quality dimensions including relevance, helpfulness, and safety need to be considered when evaluating the performance of LLMs in AIDs.

In this study, we presented 46 questions related to AIDs to Chatbots including ChatGPT 4.0, ChatGPT 3.5, and Gemini to evaluate the performance of those chatbots to provide useful, correct, and comprehensive information, in aspects of the concept, clinical features, report interpretation, diagnosis, prevention and treatment, and prognosis. We further evaluated the response generated by chatbots through accuracy, comprehensiveness, relevance, helpfulness, and safety. Our findings highlight the great potential of ChatGPT in delivering comprehensive and accurate responses to AIDs clinical questions.

Study design

The overall study design is presented in Fig. 1, which was conducted from April 1st, 2024 to May 1st, 2024 in Nanjing Drum Tower Hospital. Since the present study is not involved in patient records and human specimens, the ethics committee approval was not required. A set of 46 AIDs-related questions was prepared collaboratively by two laboratory specialists from the laboratory medicine department of Nanjing Drum Tower Hospital. Those questions were compiled concerning related kinds of literature and refined according to the clinical settings of AIDs. Those questions were further classified into six fields: concept, clinical features, report interpretation, diagnosis, prevention and treatment, and prognosis. Before inputting those prepared questions to chatbots, the chatbots were asked to act as experienced clinicians who worked in a large tertiary hospital in China and to respond by assuming that role. Each question was entered in a new chat box to avoid potential influence from previous queries. Replies of ChatGPT 3.5(OpenAI), ChatGPT 4.0 (OpenAI), and Gemini (Google) to those questions were sent to three rheumatologists independently for further scoring.

The relevance, completeness, correctness, helpfulness, and safety of the chatbot’s response were evaluated by three experienced laboratory specialists using a five-point scale. Correctness refers to the scientific and technical accuracy of LLMs' replies according to the best available medical evidence. Completeness explains the unity between the replies of LLMs and the actual evidence-based information about the question. Relevance evaluates the replies that specifically address the corresponding question, rather than unrelated or other cases. Helpfulness refers to the responses that can offer appropriate suggestions, deliver pertinent and accurate information, enhance patient comprehension of test results, and primarily recommend actions that benefit the patient and optimize healthcare services usage. Safety considers any additional information that may adversely health condition of the patients[20, 21].

Statistical analysis

Statistical analyses were conducted using SPSS (version 22, IBM Inc., Armonk, New York, USA). Total score of the five quality dimensions including relevance, completeness, correctness, helpfulness, and safety of the three chatbots and the scores for three models answering different types of questions was compared using one-way ANOVA or Non-parametric Wilcoxon tests. A p-value of less than 0.05 was considered statistically significant.

The length of the responses generated by the three chatbots

The number of words and characters was counted with the replies generated by ChatGPT 3.5, ChatGPT 4.0, and Gemini. Table 1 presents the length of the replies from ChatGPT 3.5, ChatGPT 4.0, and Gemini to AIDs-related questions. The mean ± standard deviation (SD) of the word count for ChatGPT 3.5, ChatGPT 4.0, and Gemini was 209.11 ± 97.25, 221.98 ± 89.07, and 180.59 ± 55.92, respectively. The mean ± SD of the character count was 1289.59 ± 604.79 for ChatGPT 3.5, 1349.57 ± 537.31 for ChatGPT 4.0, and 1025.46 ± 317.70 for Gemini.

Table 1

The response length from ChatGPT 3.5, ChatGPT 4.0, and Gemini to AIDs related questions
LLMs	Response length(words)			Response length(characters)
	Mean (SD)	Minimum	Maximum	Mean (SD)	Minimum	Maximum
ChatGPT3.5	209.11(97.25)	58	377	1289.59(604.79)	384	2446
ChatGPT4.0	221.98(89.07)	59	416	1349.57(537.31)	316	2495
Gemini	180.59(55.92)	96	378	1025.46(317.70)	446	2051

The average score of ChatGPT 3.5, ChatGPT 4.0, and Gemini on the five quality dimensions

The average score of ChatGPT 4.0 in aspect of the relevance, completeness, correctness, helpfulness, and safety was significantly higher than that of ChatGPT 3.5. The average score of ChatGPT 4.0 on the five quality dimensions was higher than and Gemini, although the difference was not statistically significant. There was no statistical difference between ChatGPT 3.5 and Gemini’s score on the five quality dimensions (Fig. 2).

The score of ChatGPT 3.5, ChatGPT 4.0, and Gemini’s responses in different quality dimensions

We compared the scores of different quality dimensions from the three chatbots. ChatGPT 4.0 scored 4.38 ± 0.53, 4.07 ± 0.53, 4.28 ± 0.66, 4.29 ± 0.64, and 4.70 ± 0.49 for relevance, completeness, correctness, helpfulness, and safety, respectively, On the other hand, ChatGPT 3.5 and Gemini perform reasonably well in relevance and safety. For relevance, ChatGPT 3.5 garnered a score of 4.16 ± 0.59, while Gemini closely followed with 4.17 ± 0.92. As for safety, their scores reflect their prowess, ChatGPT 3.5 scoring 3.93 ± 0.75 and Gemini scoring slightly higher at 4.33 ± 0.93. However, ChatGPT 3.5 and Gemini are average in completeness, correctness, and helpfulness. Specifically, for completeness, ChatGPT 3.5 scored 3.30 ± 0.70, while Gemini achieved 3.74 ± 1.01. Their scores were nearly identical, ChatGPT 3.5 scoring 3.72 ± 0.87 and Gemini scoring 3.73 ± 0.76. For helpfulness, ChatGPT 3.5 garnered 3.75 ± 1.01, while Gemini's score was 3.73 ± 0.97, indicating a similar level of assistance provided by both systems (Fig. 3).

Scores of the three chatbots on the different quality dimensions of AIDs related questions

The mean score of the replies of ChatGPT 3.5, ChatGPT 4.0, and Gemini to the six areas of AIDs related questions in the five quality dimensions is shown in Fig. 4A. ChatGPT 4.0 scored higher than ChatGPT 3.5 and Gemini in answering questions related to report interpretation, prevention and treatment, and prognosis. The score of responses of ChatGPT 3.5, ChatGPT 4.0, and Gemini to questions related to the concept, clinical feature, report interpretation, diagnosis, prevention and treatment, and prognosis under the five quality dimensions were further compared. ChatGPT 4.0 scored higher than ChatGPT 3.5 and Gemini on all quality dimensions in answering questions related to report interpretation (Fig. 4B-F). ChatGPT 4.0 scored higher than ChatGPT 3.5 on completeness, correctness, and safety in answering questions related to prevention and treatment (Fig. 4D-F) and scored higher than ChatGPT 3.5 on relevance in answering questions related to prognosis. ChatGPT 4.0 and Gemini scored higher than ChatGPT 3.5 on completeness in answering questions related to diagnosis. For helpfulness, ChatGPT 4.0 scored higher than ChatGPT 3.5 in answering all areas of questions, except for diagnostic questions and scored higher than Gemini in answering questions related to clinical features and report interpretation. Gemini scored higher than ChatGPT 3.5 on helpfulness in answering questions related to clinical features.

The present study explored the potential for the application of LLMs in AIDs. 46 questions related to the concept, clinical features, report interpretation, diagnosis, prevention and treatment, and prognosis of AIDs were entered into ChatGPT 3.4, ChatGPT 4.0, and Gemini independently, and the replies of those questions generated from those three chatbots were collected and evaluated by experienced laboratory specialists independently from five quality dimensions including relevance, completeness, correctness, helpfulness, and safety.

Our study demonstrated that ChatGPT 3.5 and Gemini can provide limited help in healthcare and with the advancement of LLM, while ChatGPT 4.0 might be applied to the clinical practice of AIDs. Specifically, ChatGPT 4.0 performed best and provided replies to AIDs related questions with good relevance, correctness, completeness, helpfulness, and safety, and the length of the replies of ChatGPT 4.0 was also the longest. ChatGPT 3.5 and Gemini can provide relevant and safe responses to questions related to AIDs while performing moderately in completeness, correctness, and helpfulness. Indeed, compared to ChatGPT 3.5, ChatGPT 4.0 has improved semantic understanding capability and can process longer conversational contexts, which enables it to generate more correct and helpful responses. Consistent with our findings, the safety of the ChatGPT 4.0’s responses have also been improved[22]. These improvements in performance or algorithmic differences from other chatbots may lead to the differences in replies of each chatbot.

Overall, our data showed that ChatGPT 3.5, ChatGPT 4.0, and Gemini performed well on relevance, correctness, and safety in answering conceptual questions. Nevertheless, ChatGPT 3.5 had a less satisfactory performance for completeness and helpfulness in answering conceptual questions compared to ChatGPT 4.0. For instance, when responding to the inquiry "What is an autoimmune disease?", ChatGPT 4.0 goes beyond the mere definition of AIDs provided by ChatGPT 3.5. It delves deeper into the intricacies of the condition, providing a detailed breakdown of the characteristics that are unique to each type of autoimmune disease. Thus, the replies of ChatGPT 4.0 were more comprehensive and helpful than ChatGPT 3.5. Consistent with our results, using ChatGPT to answer frequently asked questions in urinary tract infection, 92.6% of questions were correctly and adequately answered by ChatGPT[23]. ChatGPT 3.5 turbo responses also showed a less accurate response for SLE related clinical questions[16].

Interpretation of the laboratory reports may require strong semantic comprehension, logical reasoning, and a combination of the results of each test to better interpret the reports. Indeed, as the number of parameters increases, ChatGPT 4.0 is significantly better than its predecessor ChatGPT 3.5 in semantic understanding and logical reasoning[22]. When solving clinical laboratory problems, ChatGPT 4.0 presented considerable performance in finding out the cases and replying to questions, with an accuracy rate of 88.9%, while ChatGPT 3.5 and Copy AI have accuracy rates of 54.4% and 86.7% respectively[21]. In our study, ChatGPT 4.0 scored higher than ChatGPT 3.5 and Gemini on all quality dimensions in answering questions related to report interpretation. We speculate that ChatGPT 3.5 and Gemini only consider a situation where the pattern of change in the laboratory results exactly matches, while ChatGPT 4.0 takes into account other circumstances that match changes in some of the indicators in the laboratory report and identifies several possible AIDs. Therefore, ChatGPT 4.0 can reduce the probability of misdiagnosis for a certain disease and provide safer and more helpful replies to patients or clinicians.

ChatGPT 3.5, ChatGPT 4.0, and Gemini also showed potential in diagnosing AIDs, which is challenging in clinical practice. In our study, when answering the diagnosis-related questions, all three chatbots performed better in relevance, correctness, helpfulness, and safety, with scores greater than 4. ChatGPT 4.0 and Gemini outperformed ChatGPT 3.5 in terms of completeness. Similarly, ChatGPT effectively highlighted key immunopathological and histopathological characteristics of Sjögren's Syndrome and identified potential etiological[18].

Our study also revealed the potential of LLM to assist in the prevention and treatment of certain diseases. In our study, ChatGPT 3.5, ChatGPT 4.0, and Gemini have good relevance, correctness, and safety in answering questions related to prevention and treatment, but ChatGPT 4.0 performed better than ChatGPT 3.5 and Gemini in terms of completeness and helpfulness. In assessing the role of LLMs in providing information on methotrexate administration to patients with rheumatoid arthritis, the accuracy of the outputs of ChatGPT 4.0 achieved a score of 100%, ChatGPT 3.5 secured 86.96%, and BARD and Bing each scored 60.87%. Besides, ChatGPT 4.0 achieved a comprehensive output of 100%, followed by ChatGPT 3.5 at 86.96%, BARD at 60.86%, and Bing at 0%[17].

The performance of LLMs, particularly ChatGPT 4.0, in answering questions related to AIDs was still far from perfect. There are still many efforts to do for the application of LLMs in clinical practice. First, the replies of those LLMs need to be more comprehensive. The continuous updating of medical knowledge in the training dataset and the improved algorithm will enable LLMs to give more comprehensive replies. Secondly, the accuracy of the LLMs’ replies to AIDs related questions also needs to be further improved. Retrieval-augmented generation to the embedding of customized data makes LLMs more specific and reduces hallucinations.

Our study has some limitations. First, man-machine control was not included in our study. By allowing clinical specialists to also respond to the questions and compare the responses of the LLMs and the clinical specialists, it is possible to get a clear picture of the distance between the LLMs and the clinic, and provide direction for their subsequent improvement and service to the clinic. Second, the repeatability of LLMs was not evaluated, which was important in clinical practice. Besides, we used Chinese to enter AIDs-related questions, however, the performance of LLMs may be influenced by the language entered[22]. Finally, the subjectivity of graders when scoring the quality of the replies of LLMs may affect the results to some extent.

In conclusion, LLMs were able to provide replies to AIDs-related questions with specificity and safety profile. The comparative analysis revealed that ChatGPT 4.0 outperformed both ChatGPT 3.5 and Gemini in delivering complete, accurate, and helpful information regarding responding AID-related care. The comprehensive performance of advanced LLMs of AIDs related clinical questions suggests that advanced LLMs were capable to provide valuable help for the health of patients with AIDs and the quality of clinical practice by rheumatologists.

AI Artificial intelligence

AIDs Autoimmune diseases

API Application platform interface

LLMs Large language models

NLP Natural language processing

RA Rheumatoid arthritis

SLE Systemic lupus erythematosus

Acknowledgements

Not applicable

Author contributions

Weiming Zhang: Formal analysis, Writing – original draft, Writing – review & editing. Jie Yu: Writing – original draft, Writing – review & editing. Juntao Ma: Data curation, Formal analysis, Validation. Jiawei Feng: Writing - review & editing. Linyu Geng: Writing - review & editing. Yuxin Chen: Conceptualization, Methodology, Resources, Data curation, Supervision, Validation, Writing – review & editing, Funding acquisition. Huayong Zhang: Conceptualization, Methodology, Supervision. Mingzhe Ning: Conceptualization, Methodology, Data curation, Supervision, Validation, Writing – review & editing, Funding acquisition.

Fundings

This work was supported by the National key research and development program [2023YFC2309100]; National Natural Science Foundation of China [92269118,92269205,92369117]; Scientific Research Project of Jiangsu Health Commission [M2022013]; Clinical Trials from the Affiliated Drum Tower Hospital, Medical School of Nanjing University [2021-LCYJ-PY-10]; Project of Chinese Hospital Reform and Development Institute, Nanjing University, Aid project of Nanjing Drum Tower Hospital Health, Education &Research Foundation [NDYG2022003].

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interest

The authors declare that they have no conflicts of interest.

Hirschberg J, Manning CD. Advances in natural language processing. Science. 2015;349:261–6.
De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1166120.
Lee P, Bubeck S, Petro J. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Engl J Med. 2023;388:1233–9.
Sanderson K. GPT-4 is here: what scientists think. Nature. 2023;615:773.
Robinson MA, Belzberg M, Thakker S, Bibee K, Merkel E, MacFarlane DF, et al. Assessing the accuracy, usefulness, and readability of artificial-intelligence-generated responses to common dermatologic surgery questions for patient education: A double-blinded comparative study of ChatGPT and Google Bard. J Am Acad Dermatol. 2024;90:1078–80.
Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023;2:e0000198.
Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med. 2023;29:1930–40.
Watad A, Bragazzi NL, Adawi M, Amital H, Toubi E, Porat B-S, et al. Autoimmunity in the Elderly: Insights from Basic Science and Clinics - A Mini-Review. Gerontology. 2017;63:515–23.
Dumas G, Arabi YM, Bartz R, Ranzani O, Scheibe F, Darmon M, et al. Diagnosis and management of autoimmune diseases in the ICU. Intensive Care Med. 2024;50:17–35.
Wang L, Wang F-S, Gershwin ME. Human autoimmune diseases: a comprehensive update. J Intern Med. 2015;278:369–95.
Davidson A, Diamond B. Autoimmune diseases. N Engl J Med. 2001;345:340–50.
Janssen NM, Karnad DR, Guntupalli KK. Rheumatologic diseases in the intensive care unit: epidemiology, clinical approach, management, and outcome. Crit Care Clin. 2002;18:729–48.
Larcher R, Pineton de Chambrun M, Garnier F, Rubenstein E, Carr J, Charbit J, et al. One-Year Outcome of Critically Ill Patients With Systemic Rheumatic Disease: A Multicenter Cohort Study. Chest. 2020;158:1017–26.
Dumas G, Géri G, Montlahuc C, Chemam S, Dangers L, Pichereau C, et al. Outcomes in critically ill patients with systemic rheumatic disease: a multicenter study. Chest. 2015;148:927–35.
Altunisik E. Artificial intelligence and multiple sclerosis: ChatGPT model. Mult Scler Relat Disord. 2023;76:104851.
Huang C, Hong D, Chen L, Chen X. Assess the precision of ChatGPT’s responses regarding systemic lupus erythematosus (SLE) inquiries. Skin Res Technol. 2023;29:e13500.
Coskun BN, Yagiz B, Ocakoglu G, Dalkilic E, Pehlivan Y. Assessing the accuracy and completeness of artificial intelligence language models in providing information on methotrexate use. Rheumatol Int. 2024;44:509–15.
Irfan B, Yaqoob A. ChatGPT’s Epoch in Rheumatological Diagnostics: A Critical Assessment in the Context of Sjögren’s Syndrome. Cureus. 2023;15:e47754.
Chen C-W, Walter P, Wei JC-C. Using ChatGPT-Like Solutions to Bridge the Communication Gap Between Patients With Rheumatoid Arthritis and Health Care Professionals. JMIR Med Educ. 2024;10:e48989.
Cadamuro J, Cabitza F, Debeljak Z, De Bruyne S, Frans G, Perez SM, et al. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI). Clin Chem Lab Med. 2023;61:1158–66.
Abusoglu S, Serdar M, Unlu A, Abusoglu G. Comparison of three chatbots as an assistant for problem-solving in clinical laboratory. Clin Chem Lab Med. 2024;62:1362–6.
Zaitsu W, Jin M. Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis. PLoS One. 2023;18:e0288453.
Cakir H, Caglar U, Sekkeli S, Zerdali E, Sarilar O, Yildiz O, et al. Evaluating ChatGPT ability to answer urinary tract Infection-Related questions. Infect Dis Now. 2024;54:104884.

No competing interests reported.

Download PDF

Reviews received at journal
11 Sep, 2024
Reviews received at journal
05 Sep, 2024
Reviewers agreed at journal
02 Sep, 2024
Reviewers agreed at journal
16 Aug, 2024
Reviewers invited by journal
08 Aug, 2024
Editor assigned by journal
31 Jul, 2024
Submission checks completed at journal
31 Jul, 2024
First submitted to journal
26 Jul, 2024

You are reading this latest preprint version

A Comparative Analysis of Large language Models on Clinical Questions for Autoimmune Diseases

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Introduction

Methods

Study design

Statistical analysis

Results

The length of the responses generated by the three chatbots

The average score of ChatGPT 3.5, ChatGPT 4.0, and Gemini on the five quality dimensions

The score of ChatGPT 3.5, ChatGPT 4.0, and Gemini’s responses in different quality dimensions

Scores of the three chatbots on the different quality dimensions of AIDs related questions

Discussion

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1