Advancing Rheumatology Practice with AI Assistance: Evaluating ChatGPT's Performance in Real-world Cases

doi:10.21203/rs.3.rs-3340373/v1

Download PDF

Article

Advancing Rheumatology Practice with AI Assistance: Evaluating ChatGPT's Performance in Real-world Cases

https://doi.org/10.21203/rs.3.rs-3340373/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The launch of ChatGPT, an advanced language model created by OpenAI, has sparked attention throughout the world. ChatGPT offers human-like responses and the potential for a wide range of applications, including medical decision-making. However, questions remain regarding its validity, the spread of false information, and its ethical implications for healthcare. While other studies have investigated ChatGPT's diagnostic capabilities, this study fills a research gap by assessing how well it performs in real-world rheumatology case scenarios, offering light on its possible use in managing rheumatologic patients.

Methods

The study encompassed 32 challenging rheumatology cases. Data for each case was divided into four categories: 1) initial presentation, history, and review of systems; 2) physical examinations; 3) workup results; and 4) final diagnosis. Data was transformed into prompts for ChatGPT, simulating real-time interaction. Four stages of questioning were used to progressively build the case. Recommendations were evaluated based on correctness, completeness, and potential harm or cost implications.

Results

The percentage of comprehensive answers (totally correct, totally complete, no extra-costs, no harm) for physical examinations, workups, differential diagnosis, and treatment were 65.6%, 50%, 40,6% and 40,6% respectively. ChatGPT was successful in 65.6% of the cases to suggest the correct diagnosis first in the list of differential diagnoses. None of ChatGPT responses included suggestions that would result in unnecessary costs or harm to the patient. ChatGPT recommendations for physical examinations, workups, differential diagnosis and treatment were totally correct in 75%, 65.63%, 53.13% and 50% of instances; and they were totally complete in 81.25%, 62.50%, 62.50%, 59.38% of instances respectively.

Conclusions

Our study highlights the effectiveness of ChatGPT in supporting rheumatology practice by offering precise and comprehensive suggestions across various stages of patient cases. While the AI model displays potential, its performance is inconsistent when faced with higher levels of scenario complexity.

AI assistance

Rheumatology practice

ChatGPT performance

Real-world cases

medical decision-making

The release of ChatGPT, a language model of significant size, trained on a vast range of datasets and texts found online, by OpenAI (OpenAI, L.L.C., San Francisco, CA, USA) in November 2022, has rapidly captured the attention of users around the world(1). The generative pre-trained transformer (GPT) architecture of this artificial intelligence (AI) model enables ChatGPT to accurately predict output based on input text and provide a human-like response to a variety of prompts, while also delivering a comprehensible and grammatically correct response(2). ChatGPT has the ability to emulate other intelligent functions performed by humans, such as judgment, decision-making, and reasoning, making it a promising candidate to improve areas related to these tasks, including medical practice and education(3). Recent studies have indicated that ChatGPT has multiple applications in medicine, such as providing insights into data analysis, enhancing scientific writing, managing and interpreting large electronic health records (EHR) databases, and improving medical education for both patients and healthcare professionals(4). This language model could also play a role in more practical aspects of healthcare, such as establishing differential diagnoses based on provided data and clinical decision-making(5).

Rheumatological disorders, such as rheumatoid arthritis (RA) and systemic lupus erythematosus (SLE), are characterized by complex multifactorial etiologies, and their management often requires long-term patient assessment(6). Decision-making for patients with chronic disease courses and various treatment options typically relies on physicians' experience and clinical judgment, developed over time based on disease characteristics and individual responses to different treatments(7). To improve the management of these patients, studies have already assessed the utility of technologies such as activity trackers or AI-based symptom checkers on rheumatologic patients(8, 9). With logical reasoning and evidence-based judgment, large language models like ChatGPT could also aid both rheumatologists and patients in the diagnosis, treatment, and follow-up process. However, it is still unclear to what extent ChatGPT's statements are clinically accurate and whether it could benefit or harm patients' management.

Despite the promising potentials of ChatGPT, concerns have been raised regarding the AI's reliability and potential for misuse in the field of healthcare(10). Owing to its training on vast amounts of uncategorized internet data and texts, ChatGPT may generate seemingly acceptable yet false statements in response to user input, a phenomenon referred to as hallucination(11). Such occurrences have the potential to spread misinformation and misguide healthcare professionals and patients. Furthermore, ethical and legal considerations, as well as data security, have concerned researchers with respect to ChatGPT(12, 13).

Several studies have been conducted to evaluate the effectiveness of ChatGPT in building diagnostic and clinical decision-making capabilities(14, 15). While one study has suggested that ChatGPT may be a valuable tool for gathering information on rheumatological diseases(16), no research has yet been undertaken to assess its practical decision-making abilities for patients with rheumatologic conditions. In the present study, we sought to address this gap by subjecting ChatGPT to real-world rheumatology cases and analyzing its performance across four different stages, from initial presentation to final diagnosis. Our findings provide valuable insights into the reliability of AI-assisted tools such as ChatGPT in the management of rheumatologic patients.

Study design and cases

The data collection for this study was conducted using ChatGPT web version in May 2023. This study consists of 32 medical cases from the book “Challenging Cases in Rheumatology and Diseases of the Immune System”(17) consisting of a list of rheumatologic patients with a variety of diseases. The data corresponding to each case was categorized into four parts including: 1. initial presentation, history, and review of systems (ROS) 2. physical examinations 3. workup results (imaging, laboratory results, biopsies, and consultations) and 4. final diagnosis. The data was extracted and changed to text, paraphrased, and then given to ChatGPT as a different prompt for each case. Similar to a real-world scenario, we did not introduce the whole case at once, but we rather fed the data into ChatGPT in four different stages and asked for recommendations at each stage.

Questions and prompts

As shown by Fig. 1, at stage one to four, we asked for recommendations using the following format:

[Information given about the case] + [ question about the case]

The information given about the case at each stage was appropriately added up while going forward. The questions for stage one to four included:

1. What physical examinations should be done by the physician?
2. What further workups should be ordered by the physician?
3. What are the differential diagnoses for this patient?
4. What is the correct treatment for this patient?

Evaluations and scoring

The recommendations were assessed based on correctness, completeness, comprehensiveness, costs and harms. The evaluations were performed by two separate rheumatologists with > 10 years of clinical experience and the average score was reported. A third rheumatologist reassessed the evaluations in cases of disagreement. For each question, correctness was evaluated by a Likert score from 1 to 5, corresponding to totally incorrect, mostly incorrect, uncertain, mostly correct and totally correct. Completement score of recommendations was evaluated from 1 to 5 corresponding to very poor, poor, only basics, almost complete, totally complete. Each recommendation was assessed for causing possible harm or causing extra costs to the patient with a yes/no score. For each question a comprehensive answer was considered an answer that was “totally correct”, “totally complete” and avoided unnecessary costs and harms to the patient.

Analysis

All data was analyzed and represented using IBM SPSS software version 26 for windows.

Table 1 shows the overview of ChatGPT performance on 4 consecutive stages of this study.

The percentage of comprehensive answers (totally correct, totally complete, no extra-costs, no harm) for Physical examinations, Workups, DDx and treatment were 65.6%, 50%, 40,6% and 40,6%. Although additional information was given to ChatGPT by moving through each new stage, this trend showed a slight decrease in percentage of comprehensive answers as the study proceeded.

In the third stage (Giving DDx based on history, physical examinations, and workup results), ChatGPT was successful in 65.6% of the cases to suggest the correct diagnosis first in the list of differential diagnoses.

In a total of 4 questions per case (32 * 4 = 128), none of ChatGPT answers included suggestions that created unnecessary costs or harm to the patient.

Table 1

The overview of recommendations given by ChatGPT to rheumatology cases
	Count (total = 32)	Percentage
Comprehensive* Physical Examinations	21	65.6%
Comprehensive Workups	16	50.0%
Comprehensive DDx	13	40.6%
Comprehensive treatment	13	40.6%
First DDx is correct**	21	65.6%
Suggestions with Unnecessary costs	0	0%
Suggestions with harm to the patient	0	0%
Comprehensive: A comprehensive answer is an answer that is “totally correct”, “totally complete” and avoids unnecessary costs and harms to the patient *It demonstrates the prevalence of answers that include the correct diagnosis as the first diagnosis in the list of differentials. DDx: Differential Diagnosis

Stage 1: What physical examinations should be done by the physician?

At this stage, the correctness and completeness of recommendations of ChatGPT for the necessary physical examination were assessed (Fig. 2). At this stage the given information about the case included initial presentation, history, and ROS.

Regarding the Correctness, ChatGPT recommendations were 75% totally correct, 18.75% mostly correct, and 6.25% uncertain (meaning that the evaluator could not decide about it).

ChatGPT answers were not Mostly or totally incorrect in any of the cases about physical examination.

Regarding the Completeness, ChatGPT recommendations were 81.25% Totally complete, 15.63% Almost complete and in 3.13% included only basics information.

ChatGPT answers were not poor or very poor in terms of completeness in any of the cases.

Stage 2: What further workups should be ordered by physician?

At this stage, the correctness and completeness of recommendations of ChatGPT for the further necessary workups were assessed (Fig. 3). At this stage the given information about the case included initial presentation, history, and ROS and physical examinations.

As shown in Fig. 3,

Regarding correctness in workup recommendations, ChatGPT was 65.63% totally correct, 28.13% mostly correct and 6.25% uncertain. In none of the cases, the answers were totally or mostly incorrect.

Regarding completeness of workup recommendations, ChatGPT was 62.50% totally complete, 21.88% mostly complete and 15.63% included only basics. In none of the cases, the answers were poor or very poor in representing necessary workup information.

Stage 3: What are the differential diagnoses for this patient?

At this stage, the correctness and completeness of recommendations of ChatGPT for the differential diagnoses were assessed (Fig. 4). At this stage the given information about the case included initial presentation, history, and ROS and physical examinations and workup results (imaging, laboratory results, biopsies and consultations).

As shown in Fig. 4,

Regarding correctness in differential diagnoses, ChatGPT was 53.13% totally correct, 40.63% mostly correct and 6.25% uncertain. In none of the cases, the answers were totally or mostly incorrect.

Regarding completeness of differential diagnoses, ChatGPT’s answer was 62.50% total complete, 28.13% mostly complete and 9.38% included only basics. In none of the cases, the answers were poor or very poor.

Stage 4: What is the correct treatment for this patient?

At this stage, the correctness and completeness of recommendations of ChatGPT for the treatment were assessed (Fig. 5). At this stage the given information about the case included initial presentation, history, and ROS and physical examinations and workup results (imaging, laboratory results, biopsies and consultations) and final diagnosis.

As shown in Fig. 5,

Regarding correctness in treatment recommendations, ChatGPT answers were 50% totally correct, 31.25% mostly correct and 18.75% uncertain. In none of the cases, the answers were totally or mostly incorrect.

Regarding completeness of differential diagnoses, ChatGPT’s answer was 59.38% totally complete, 31.25% mostly complete and 9.38% included only basics. In none of the cases, the answers were poor or very poor.

Trends in appropriateness of ChatGPT answers in different stages of the study

In terms of correctness (Figs. 2 to 5), the percentage of totally correct answers given by ChatGPT showed a slight decrease: 75% (for treatment), 65.63% (for workups), 53.13% (for DDx) and 50% (for treatment). This is contrary to the increase in the given information to ChatGPT at each stage.

As the study proceeded, the percentage of totally correct answers decreased, giving an increase in the percentage of mostly correct and uncertain answers.

The average score (quantitative scores from 1 to 5) of correctness is shown in Fig. 6. The mean scores of correctness show a slight decrease in each stage as the study proceeds.

In terms of completeness (Figs. 2 to 5), the percentage of totally complete answers given by ChatGPT also showed a slight decrease: 81.25% (for treatment), 62.50 (for workups), 62.50% (for DDx) and 59.38% (for treatment). This is contrary to the increase in the given information to ChatGPT at each stage.

As the study proceeded, the percentage of totally complete answers decreased, rising to the percentage of almost complete and only basics answers.

Figure 7 corresponds to the average completeness scores (1 to 5) at each stage of the study. The means of completeness scores were 4.81, 4,47, 4,53, 4,50 respectively.

As the most popular AI-based chatbot, ChatGPT is sought out by users seeking medical advice for their ailments. This may entail constructing a diagnosis based on the user's symptoms, formulating work-up recommendations, and proposing treatment options for their health conditions. As the frequency of these types of interactions with ChatGPT increases, concerns arise regarding the appropriateness of the chatbot's responses and the potential for harm to the user. In our investigation, we tried to assess ChatGPT's performance across four distinct stages (physical examinations, workup, differential diagnosis, and treatment) when presented with real-world rheumatologic cases. The majority of the AI's answers were accurate, comprehensive, and without any adverse consequences or additional costs for the patient when evaluated by experienced rheumatologists. However, there was a modest decline in the accuracy and comprehensiveness of ChatGPT's responses as we progressed to the subsequent stage.

Several studies have highlighted the potential of ChatGPT in clinical applications. According to Haver et al.(18), the AI has demonstrated accurate recommendations for breast cancer prevention and screening. Barat et al. arrived at a similar conclusion after investigating ChatGPT's recommendation appropriateness with respect to general questions in the field of interventional radiology(19). However, both studies noted a decrease in response accuracy as the inquiries became more specific and detailed. Consistent with prior research, the present study also observed ChatGPT's favorable accuracy and appropriateness in addressing challenging rheumatological disorders. Furthermore, the AI demonstrated superior performance in early stages when presented with symptoms or physical examinations, as opposed to later stages when more information about the patients was requested to determine appropriate treatment.

The cost-effectiveness of ChatGPT's medical recommendations should also be evaluated to prevent additional costs and eliminate harmful suggestions. According to Nastasi et al.'s investigation(20), ChatGPT's recommendations in response to various medical advice-seeking vignettes were found to be in compliance with clinical guidelines and did not lead to any extra harm in almost all cases (97%). Furthermore, ChatGPT's approach of considering social factors and insurance status of patients to determine the best treatment option was observed in over half of the cases (55%). However, Nastasi et al. have highlighted that most of the AI-generated responses were provided without any reliable personalized advice and merely relied on background information. Our study findings showed that none of ChatGPT's recommendations resulted in any harm or additional costs. Nevertheless, it is important to acknowledge the general nature and lack of specificity in the chatbot's suggestions, particularly in the later stages.

There exist several limitations for researchers in the field of LLMs such as ChatGPT. The information AI is educated on includes massive, uncategorized texts that are publicly available on the internet(5). This may provide ChatGPT with inappropriate data that ultimately results in incorrect conclusions and potentially false responses to users' prompts, particularly in the medical field where the credibility of recommendations is of utmost importance. Additionally, ChatGPT's training datasets are restricted to information that was available as of September 2021(21). In clinical settings with continuously evolving guidelines on the diagnosis and treatment of diseases, this limitation can seriously impede the AI's performance in this field. Furthermore, the novelty of LLMs and AI-based chatbots and the dearth of research on their implementation in medical environments could constrain the capacity to engage in a comprehensive discourse on this issue.

Our findings suggest that ChatGPT has the capability to serve as a valuable tool in supporting healthcare professionals in diagnosing and managing rheumatologic patients. However, the study also highlights an incontinency in accuracy and comprehensiveness as the complexity of the scenario increases, indicating that the model might face challenges when dealing with more intricate medical situations.

While ChatGPT shows promise, it's important to acknowledge its limitations, such as potential for misinformation and its reliance on training data that may not include the latest medical knowledge. The ethical and legal implications of using AI in healthcare decision-making also need to be carefully considered. Continued research and development, as well as close collaboration between AI developers and medical professionals, are crucial to harnessing the full potential of AI in improving patient care and outcomes in the field of rheumatology and beyond.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

Not applicable.

CONSENT TO PUBLISH

Not applicable.

AVAILABILITY OF DATA AND MATERIALS

The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.

COMPETING INTERESTS

The authors of this article declare no competing interests.

FUNDING

There was no grant for this research from any funding source in the public, private, or non-profit sectors. The writers are entirely responsible for the content.

AUTHORS CONTRIBUTIONS

All authors contributed equally.

ACKNOWLEDGEMENTS

We would like to thank Dr. Ghasem Rahmani for their clear guidance with the study design and preliminary analysis and case evaluations.

Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, et al. Language models are few-shot learners. Advances in neural information processing systems. 2020;33:1877-901.
Floridi L, Chiriatti M. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines. 2020;30:681-94.
Liebrenz M, Schleifer R, Buadze A, Bhugra D, Smith A. Generating scholarly content with ChatGPT: ethical challenges for medical publishing. The Lancet Digital Health. 2023;5(3):e105-e6.
Sallam M, editor ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. Healthcare; 2023: MDPI.
Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. Journal of Medical Systems. 2023;47(1):33.
Sangha O. Epidemiology of rheumatic diseases. Rheumatology. 2000;39(suppl_2):3-12.
Goldblatt F, O'Neill SG. Clinical aspects of autoimmune rheumatic diseases. The Lancet. 2013;382(9894):797-808.
Davergne T, Pallot A, Dechartres A, Fautrel B, Gossec L. Use of wearable activity trackers to improve physical activity behavior in patients with rheumatic and musculoskeletal diseases: a systematic review and meta‐analysis. Arthritis care & research. 2019;71(6):758-67.
Gräf M, Knitza J, Leipe J, Krusche M, Welcker M, Kuhn S, et al. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatology International. 2022;42(12):2167-76.
Deng J, Lin Y. The benefits and challenges of ChatGPT: An overview. Frontiers in Computing and Intelligent Systems. 2022;2(2):81-3.
Azamfirei R, Kudchadkar SR, Fackler J. Large language models and the perils of their hallucinations. Critical Care. 2023;27(1):1-2.
Mijwil M, Aljanabi M, Ali AH. Chatgpt: Exploring the role of cybersecurity in the protection of medical information. Mesopotamian journal of cybersecurity. 2023;2023:18-21.
Ray PP. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems. 2023.
Rao A, Kim J, Kamineni M, Pang M, Lie W, Succi MD. Evaluating ChatGPT as an adjunct for radiologic decision-making. medRxiv. 2023:2023.02. 02.23285399.
Hirosawa T, Harada Y, Yokose M, Sakamoto T, Kawamura R, Shimizu T. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: A pilot study. International journal of environmental research and public health. 2023;20(4):3378.
Uz C, Umay E. “Dr ChatGPT”: Is it a reliable and useful source for common rheumatic diseases? International Journal of Rheumatic Diseases. 2023.
Mahmoudi M. Challenging cases in rheumatology and diseases of the immune system: Springer Science & Business Media; 2012.
Haver HL, Ambinder EB, Bahl M, Oluyemi ET, Jeudy J, Yi PH. Appropriateness of Breast Cancer Prevention and Screening Recommendations Provided by ChatGPT. Radiology. 2023;307(4):e230424.
Barat M, Soyer P, Dohan A. Appropriateness of Recommendations Provided by ChatGPT to Interventional Radiologists. Canadian Association of Radiologists Journal. 2023:08465371231170133.
Nastasi AJ, Courtright KR, Halpern SD, Weissman GE. Does ChatGPT provide appropriate and equitable medical advice?: A vignette-based, clinical evaluation across care contexts. medRxiv. 2023:2023.02. 25.23286451.
Liu J, Wang C, Liu S. Utility of chatgpt in clinical practice. Journal of Medical Internet Research. 2023;25:e48568.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Advancing Rheumatology Practice with AI Assistance: Evaluating ChatGPT's Performance in Real-world Cases

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

INTRODUCTION

METHODS

Study design and cases

Questions and prompts

Evaluations and scoring

Analysis

RESULTS

Stage 1: What physical examinations should be done by the physician?

Stage 2: What further workups should be ordered by physician?

Stage 3: What are the differential diagnoses for this patient?

Stage 4: What is the correct treatment for this patient?

Trends in appropriateness of ChatGPT answers in different stages of the study

DISCUSSION

CONCLUSIONS

Declarations

References

Additional Declarations

Status:

Version 1