Progression of Large Language Models for Clinical Decision Support: An Evaluation for Rare and Frequent Diseases using GPT-3.5, GPT 4 and Naïve Google Search

doi:10.21203/rs.3.rs-3433351/v1

Download PDF

Article

Progression of Large Language Models for Clinical Decision Support: An Evaluation for Rare and Frequent Diseases using GPT-3.5, GPT 4 and Naïve Google Search

https://doi.org/10.21203/rs.3.rs-3433351/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 06 Mar, 2024

Read the published version in Nature Communications →

Version 1

posted

You are reading this latest preprint version

Large Language Models (LLMs) like ChatGPT have become increasingly prevalent. Even without medical approval, people will use it to seek health advice, much like searching for diagnoses on Google.

We performed a systematic analysis of GPT-3·5 and GPT-4 for suggesting diagnosis, examination steps and treatment of newly processed 110 medical case reports from different clinical disciplines. Balanced groups of rare, less frequent and frequent diseases were used as input. For the diagnosis task a naïve Google search was performed as benchmark comparison. Performance was assessed by two independent physicians using a 5-point Likert scale.

The results showed superior performance of GPT-4 over GPT-3·5 considering diagnosis and examination and superior performance over Google for diagnosis. With the exception of treatment, better performance on frequent vs rare diseases was evident for all approaches.

In conclusion, the LLMs showed growing potential for medical question answering in two successive major releases. However, several weaknesses and challenges necessitate the utilization of quality-controlled and regulated types of AI-models to qualify as medical applications.

Health sciences/Health care

Health sciences/Health care/Prognosis

Evidence before this study: Large Language Models (LLMs), including the ChatGPT series, started to become available for the broad public from November 2022. Even though these models are not designed for clinical consultations and carry appropriate disclaimers, patients will still use them as consultation tools. Only few preliminary analyses have been conducted to assess the applicability as clinical decision support tools. We have conducted two search phases on PubMed, Web of Science and grey literature using the keywords “clinical decision support”, “chatgpt”, “gpt”, “large language models” from 2022 till February 2023 and updated this through the end of September 2023. As a result, we could not find a single study that comprehensively assessed the core decision support tasks of suggesting diagnosis, examination and treatment options given a set of clinical cases across different specialties in terms of clinical accuracy.

Added value of this study: The study shows an increase in clinical accuracy from GPT-3·5 to GPT-4 in all three different tasks. Moreover, the performances of diagnosis recommendation showed superior accuracy compared to naïve Google-search. The models performed worse for cases with lower disease prevalence. Despite their overall high overall performance (reaching at least 4/5 points for all three tasks), there is still a high number of instances, where neither of the two models achieved the top score.

Implications of all the available evidence:

The findings show an increase in performance after the major update from GPT-3·5 to GPT4 and it is foreseeable that future LLMs will continue to do so. Our findings show that using these tools without consulting trained medical experts could result in confusion or even health complications for patients. The risk of amplification bias, non-transparent training procedures still provide major roadblocks in order to generate consistent high accuracies for different patient populations or to identify mistakes in the output due to artificial hallucination. These LLMs are still in their infancy and are not designed to function as fully matured, regulated medical devices. As patients increasingly turn to these tools for preliminary consultations, it's crucial for healthcare professionals to articulate both the capabilities and limitations of these emerging technologies.

The company OpenAI has achieved significant advancements in the perception of Artificial Intelligence (AI) by using Large Language Models (LLMs) in broad segments of the population with the introduction of ChatGPT with its initial version GPT3·5 and its recent release GPT-4 [1]. In many areas, these LLMs demonstrate great potential, especially when processing pure text, such as summarizing, rewriting, but also generating new acquisitions of texts or programming code [2, 3]. Moreover, they tempt the user to consult the system as a personal assistant especially with pending questions. However, the nature of existing text corpora, which is the basis for training such AI systems, can contain inconsistencies, incompleteness or information bias [4, 5]. Therefore, ChatGPT is frequently criticized for spreading false information with persuasive rhetoric and artificial hallucination [6, 7]. This is particularly crucial or threatening in fields such as medicine, when confused patients attempt to associate symptoms with diagnoses and therapies. Although these models are not primarily designed for this request, the practice will find its way into people's everyday life in a similar way to googling symptoms by both patients [8, 9] and physicians [10, 11]. Moreover, preliminary studies have already demonstrated significant potential in utilizing ChatGPT within the medical field. Research has shown that ChatGPT can approach or exceed the passing threshold in medical exams such as the United States Medical Licensing Exam (USMLE) or Advanced Cardiovascular Life Support (ACLS) exam [12, 13]. In addition to exam simulations, previous works have also illustrated the potential benefit of ChatGPT in everyday medicine, for instance, in medical writing by extracting information from electronic health records, assisting with literature searches or providing guidance on writing style and formatting [14]. Furthermore, entire specialized fields have discussed the use of LLMs, such as enhancing services in dental telemedicine or by improving patient-centered care in radiology [15, 16]. The diagnostic capability of GPT-3·5 has also been demonstrated in its basic aspects, as it can generate a well-differentiated diagnosis list for common chief complaints [17]. We aim to extend upon research of this nature through our comprehensive analysis. To the best of our knowledge, there has been no prior work on thorough evaluation encompassing various clinical decision support tasks, diagnostic and therapeutic capabilities.

We aim at investigating clinical accuracy of two well-established successive LLMs – GPT-3·5 and GPT-4 - regarding three key levels of clinical decision-making: 1) diagnosis, 2) examination and 3) treatment.

Within each of these three levels, the influence of a disease’s frequency on performance is additionally evaluated (categories rare, less frequent and frequent). It is known that machine learning models perform poor on problems with less or no training data available. Based on the known incidence and prevalence rates of the diseases, we consider our assessment is apt to provide different levels of difficulties to the AI. In particular, patients with rare diseases form a large global population when taken together and still run a higher risk of being undiagnosed [18]. These patients are in need of diagnostic support and are thus assumed to use these AI tools more frequently.

As input for the assessment, we consider clinical reports representing a broad disease entity spectrum. The case reports were extracted, processed to first-person perspective in layman language and translated from German clinical casebooks. Two major publishers requiring licensed access were taken into account. The restricted access, the unavailability in English, and the subsequent conversion into layman terms collectively serve as safeguards ensuring that ChatGPT was not trained on this input. Thereby, we can provide an un-biased testing framework.

On all three levels – diagnosis, examination and treatment – we compare the performance of GPT-3·5 vs GPT-4. Particularly for diagnosis, we compare these performances to Google search engine as important benchmark comparison.

In our in-depth analyses, we present a comprehensive evaluation of the current performance trajectory of two successive LLM versions with respect to their capacity to facilitate diagnostic decision-making, recommend appropriate diagnostic examination procedures, and propose treatment options across a wide spectrum of diseases, including those that are rare, less frequent, and frequent.

An overview of the workflow we applied to select clinical case reports, to query GPT-3·5 and GPT-4 and to evaluate the results is illustrated by the flowchart in Figure 1.

Selecting clinical case reports

In order to base our study on a representative, unbiased selection of realistic cases, we screened all case reports published in German casebooks by Thieme and Elsevier. Main clinical specialties internal medicine, surgery, neurology, gynecology and pediatrics were selected. A total of five casebooks from Thieme and seven from Elsevier were utilized. Of these, five books from each publisher were explicitly associated with the aforementioned medical specialties. Additionally, two supplementary casebooks from Elsevier focusing on "Rare Diseases" and "General Medicine" were analyzed to further include disease cases of very low incidence and from outpatient settings. This selection generated a pool of 1,020 cases.

To study the performance of ChatGPT in relation to a disease’s incidence, cases were categorized in three frequency groups as follows: A disease is considered “frequent” or “less frequent” if its incidence is higher than 1:1,000 or 1:10,000 per year respectively. A disease was considered as “rare” if the incidence is lower than 1:10,000. Only if no information of incidence was available, the disease was considered “rare” if the prevalence is lower than 1:2,000. Power calculations were conducted to infer a sample size of 33-38 per group in order to achieve a total power of 0·9 (details in Supplementary Information, section 1·1 Power calculation).

To limit the scope of the subsequent analysis, a random sampling of 40% of the total number of cases was performed, taking into account the distribution of sources, resulting in the examination of 408 cases.

The majority of cases followed a consistent structure of subsequent sections including medical history, current symptoms, examination options and findings, actual diagnosis, differential diagnoses and treatment recommendations. For each of the medical subspecialties, a physician reviewed all cases and included a case for further analysis when the following criteria were met: (1) the patient is able to provide information on medical history him-/herself (e.g. excluding severe traumatic patients), (2) images are not required for diagnostic purposes, (3) diagnosis does not overly rely on laboratory values, (4) the case is not a duplicate. A total of 153 cases were identified as conforming to the established inclusion criterion.

Incidence rates for the selected case studies were researched and sorted into three previously mentioned frequency groups. To ensure a balanced representation, we aimed to include an equal number of cases from each medical specialty while considering both the incidence rates and the publishing source. Sampling from each book and within each frequency group continued until we reached predefined limits: a maximum of seven cases per book or up to twenty cases for each medical specialty.

Included cases were manually translated to English by using the translation tool Deepl.com, followed by a human review to ensure both linguistic accuracy and quality. In order to mimic a true patient situation, cases were processed to patient layman language. Medical history was provided in first person perspective, containing only general information and avoiding clinical expert terminology. Layman readability was independently checked by two non medical researchers and alternative laymen terms provided if expert terminology was used. In cases of disagreement for laymen translation, consensus was achieved by a third non medical researcher.

An overview of the included cases, distributed to publisher, clinical specialty and disease frequency is provided in Table 1. Detailed information including the exact patient medical history is provided in Supplementary Data S1.

Table 1: Number of cases per publisher and clinical specialties, considering all, rare, less frequent and frequent diseases.

Querying GPT-3·5 and GPT-4

For generating the patient queries, the analysis plan reads as follows:

Open a new ChatGPT conversation.
Suspected diagnosis: Write patient medical history and current symptoms, add “What are most likely diagnoses? Name up to five.”
Examination options: “What are the most important examinations that should be considered in my case? Name up to five.”
Open New ChatGPT conservation
Treatment options: Write patient medical history and current symptoms and add: “My doctor has diagnosed me with (specific diagnosis X). What are the most appropriate therapies in my case? Name up to five.”

All prompts were systematically executed between 3 April 2023 and 19 May 2023 through the website https://www.chat.openai.com. Output generated with GPT-3·5 and GPT-4 for each of the three queries is provided in Supplementary Data S1.

Querying Google

Symptoms were searched and the most likely diagnosis was determined based on the first 10 hits reported by Google. Search, extraction and interpretation was performed by a non medical expert, mimicking the situation of a patient.

Based on medical history of every case, search strings were defined: “baby(opt.) child(opt.) diagnosis <symptoms> previous <previous illness>(opt.) known <known disease> <additional information>(opt.)”

Symptoms were extracted based on the information provided in medical history. Optionally, if a previously resolved illness existed and was considered relevant by the non medical expert, the word “previous” was added, followed by information on the disease. If a known condition, e.g. hay fever, existed and was considered relevant, the word “known” was added, followed by information on the disease.

The search was performed in incognito mode using https://www.google.com. For every case, the first 10 websites are evaluated. Websites were scanned for possible diagnoses. The non medical expert compared the symptoms characterizing each diagnosis on the search results to those provided by the medical history. Information on e.g. age or sex – available in the medical history but not by the search string – was additionally taken into account. For example, ectopic pregnancy is not considered possible for a male case with abdominal pain. For cases considering children below the age of 1, “baby” was added to the search string. For cases considering children below the age of 16, “child” was added to the search string.

Solely information available on the websites was evaluated. No further detailed search on a specific diagnosis was performed if, e.g., only limited information on a disease’s characteristics was provided by a website.

A maximum of five most likely diagnoses were determined. If more than five diagnoses appear equally likely, the most frequently reported diagnoses were selected. The Google search strings as well as identified five diagnoses are available in Supplementary Data S1.

Performance Evaluation

Assessment of the answers generated with GPT-3·5, GPT-4 and Google was conducted independently by two physicians. The AI’s output was reviewed in relation to solutions provided by the casebooks. In cases of uncertainty, further literature was consulted. Each physician scored clinical accuracy based on a 5-point Likert scale according to Table 2. The final score is calculated as the mean of the two individual scores.

Table 2. Assessment scheme for diagnosis, examination and treatment recommendations.

Most of all relevant options were not mentioned.

All or most of the system’s generated options were redundant or unjustified.

Some or many relevant options were not mentioned.

Some of the system’s generated options were redundant or unjustified.

Most of all relevant options were mentioned.

Some of the system’s generated options were redundant or unjustified.

Most of all relevant options were mentioned.

Few of the system’s generated options were redundant or unjustified.

All of the relevant options were mentioned.

There was no redundant or unjustified option mentioned.

All statistical analyses were conducted using R 4.3.1 [19]. To assess inter-rater reliability, weighted Cohen’s kappa and 95% confidence intervals (CIs) were calculated for each of the three tasks, using the R package “DescTools” [20]. To explore the performance of the AI learning models, an independent evaluation of diagnosis, examination and treatment was conducted. The performance of GPT-3·5 vs GPT-4 was evaluated applying a paired one-sided Mann-Whitney test (R package “base”, function wilcoxon.test()). Considering diagnosis, paired one-sided Mann-Whitney tests comparing GPT-3·5 vs Google and GPT-4 vs Google were additionally performed. Possible influence of the diseases’ incidence (rare vs less frequent vs frequent) was investigated using non-paired one-sided Mann-Whitney test. Using Bonferroni correction [21], a conservative adjustment for multiple testing was applied (for examination and treatment: 1 test comparing GPT-3·5 to GPT-4, 3 tests within group GPT-3·5, and 3 tests within group GPT-4 resulting in n=7; for diagnosis: 3 tests comparing GPT-3·5 to GPT-4 to Google, 3 tests within group GPT-3·5, 3 tests within group GPT-4, and 3 tests within group Google resulting in n=12).

Inter-rater reliability

Regarding diagnosis, highest levels of agreement can be observed with κ = 0·8 for GPT-3·5, κ = 0·76 for GPT-4 and κ = 0·84 for Google. Examination is characterized by κ = 0·53 for GPT-3·5 and 0·64 for GPT-4. With respect to treatment, we observe κ = 0·67 for GPT-3·5 and κ = 0·73 for GPT-4. According to Landis et al. 1977, this corresponds to substantial (0·61 − 0·8) to almost perfect (0·81 − 1) agreement [22]. The findings do not provide any evidence of consistent rating discrepancies favoring one rater over the other. Details of results are provided in Supplementary Figure S2 and Supplementary Data S1.

Performance evaluation of GPT-3·5, GPT-4 and Google

Figure 2 sums up the results of pairwise comparisons (bubble plots) as well as performance of each tool in relation to disease frequency (cumulative frequency plots). Performance distributions of all models for different tasks are summarized with bar plots in Supplementary Figure S3. Information on the median performance and 95% confidence intervals is provided in Supplementary Table S1, p values and adjusted p values for all tests performed in Supplementary Table S2.

With respect to diagnosis, all three tools were evaluated. Pairwise comparison showed a significantly better performance of GPT-4 (median: 4·5, IQR=[3·81;4·75]) against both GPT-3·5 (median: 4·25, IQR=[3·0;4·75], p = 0·0033) as well as Google (median: 4·0, IQR=[2·75;4·75], p = 0·0007). However, no significant difference between GPT-3·5 and Google was observed (p = 1). Considering disease frequency, plots in Fig. 2A indicate a continuously better performance for frequent compared to rare diseases. This observation was made for all tools (dark blue line – frequent – rising steeper compared to light blue line – rare). GPT-3·5 performed significantly better on frequent compared to rare diseases (p < 0·0001), while GPT-4 showed significant results for frequent vs rare (p = 0·0003) as well as for less frequent vs rare (p = 0·0067). For Google, no differences between rare and less frequent diseases were observed (Fig. 2A). Despite some differences being visible compared to frequent diseases, results are not significant.

Considering examinations, we compared GPT-4 (median: 4·5, IQR=[4·0;4·75]) to GPT-3·5 (median: 4·25, IQR=[3·75;4·5]). Pairwise comparison showed superior performance of GPT-4 (p < 0·0001). Evaluating performance of the two models in relation to disease frequency, results indicated superior performance of GPT-3·5 with respect to frequent diseases. However, these results were not significant. GPT-4 showed comparable performance for both frequent and less frequent diseases, but significantly better performance compared to rare (p = 0·0203).

Regarding treatment options, comparing performance of GPT-4 (median, 4·5, IQR=[4·0;4·75]) to GPT-3·5 (median: 4·25 (IQR=[4·0;4·69]) fewer differences were observed. Results in Fig. 2C indicateed superior, however, not significant performance of GPT-4 (p = 0·0503). Any influence of the diseases’ frequency on performance could not be observed.

Exploratively, we considered the influence of clinical specialty on the different approaches’ performance. Information on the median performance and 95% confidence intervals is provided in Supplementary Table S3, cumulative frequency plots can be found in Supplementary Figure S4. For all clinical specialties, GPT-3·5 and GPT-4 showed a comparable performance, considering all three levels – diagnosis, examination and treatment (un-adjusted p values not significant). Yet, data indicates a slightly superior performance of GPT-4 regarding examination in internal medicine (un-adjusted p value 0·0369) and treatment in pediatrics (un-adjusted p value 0·0315). In the specialty of surgery, no differences between GPT-3·5, GPT-4 and Google can be observed. However, data indicates a better performance of GPT-4 over Google for all the other clinical specialties – gynecology, internal medicine, neurology and pediatrics.

Our study shows results of a systematic and comprehensive testing for clinical decision support tasks with the models GPT-3·5, GPT-4 and naïve Google search in a broad spectrum of diseases entities and disease incidences. The results indicated performance progression of GPT-4 compared to its predecessor GPT-3·5 and Google search.

All of the three approaches performed worst in the most difficult task – the diagnosis recommendation. Despite the high median performances of the best performing model GPT-4, the full score of 5/5 points was reached only in 18, 24 and 26 out of 110 cases for the tasks diagnosis, examination and treatment respectively. All tools performed worse in rare diseases compared to frequent ones, which was anticipated as rare diseases are under-represented in large text corpora on the internet.

From a technical perspective, the results are promising as GPT-3·5 and GPT-4 were never trained particularly for such clinical decision support tasks and perform in a considerable amount of cases significantly better than naïve Google search while saving search time and utilizing a more intuitive chatbot functionality to users. From a clinical point of view, we have to conclude that there are few but still a noticeable number of cases, in which both of the two models could not reach the highest score. This could lead to misunderstandings or medical issues if patients rely on these outputs without contacting trained health professionals. To illustrate an example, there was one case with a patient having pheochromocytoma, a rare neuroendocrine tumor that causes hypertension. The models have correctly picked alpha-adrenergic and beta-adrenergic blockade as potential medication treatment. However, they failed to emphasize that the beta-adrenergic blocker should never be applied first, because the blockade of vasodilatory peripheral beta-adrenergic receptors with unopposed alpha-adrenergic receptor stimulation can further increase blood pressure [23]. A post-hoc test was conducted by re-entering the treatment question after finalizing our results, which generated an improved answer with correct ordering of both agent though we used the same model versions. This occurs likely due to continuous model optimizations at the back-end. Our results can therefore only present performance snapshots.

Even if the chatbots and search engines clearly output warning marks, that their services cannot replace medical consultation, it is well-known that patients will still use these tools to build a first opinion [8, 9] or ignore even such warnings. Health professionals, and patient organizations or patient science approaches should be aware of this and put more emphasis in communicating the limitations of such systems.

Strengths and limitations

The large number of medical case reports from different major medical disciplines and different disease frequencies is a strength of this this study. All cases were extracted from non-English, non-open access sources and further processed by non medical experts. This procedure comes with strengths and limitations. While it reduces the risk of prior training bias, ensures layman comprehension and thus mimics patient-generated input, it is a type of constructed but not original patient input.

As mentioned, the analyzed LLMs are continuously being updated through fine-tuning by their providers. This could lead to slightly different results if our test cases are re-entered, which limits the reproducibility of the exact performance results. Thus, our results need to be interpreted as snapshots of their time. To take this into account, we studied not only one LLM but two different successive versions with major updates (GPT-3·5 vs GPT-4). In this way we could show improved performance by the significantly updated model, which is explainable by technical progressions and updated text corpora – GPT-3·5’s training input only goes to 2021. It is foreseeable that future models could be retrained for the most current text input more efficiently. In particular, task-specific medical LLMs could be connected to quality-approved literature databases such as PubMed and UpToDate, will no longer be outdated and will continuously learn timely after newest published research articles are online. As with all clinical decision support tools that qualify the status of a medical device, LLMs will have to face high regulatory requirements and their approval will be significantly more cumbersome in the light of the low transparency of data input and low explainability in the training process.

Our study shows promising performance results by current LLMs for clinical decision support tasks. Apart from medicolegal issues that arise in the context of medical devices and data protection regulations, at the current state, they should not be used for medical consultation alone as the performance lacks consistent high accuracy. However, the study evaluated noticeable performance progress going from GPT-3·5 to GPT-4 and outperforming naïve Google search in terms of performance and time-consumption.

Both patients and health providers will have to know how to handle such generated texts and be aware of performance limitations caused by amplification bias, lacking transparency in the training process and artificial hallucination. Future models will likely be capable of efficiently incorporating quality approved up-to-date knowledge but will still need to face strict regulations regarding data protection and medical devices in order to be used with medical purpose.

Anticipating forthcoming advancements of domain and task-specific LLMs and leveraging current strengths, such as enhanced conversational capabilities, holds the potential to substantially enhance the interaction between patients and AI systems, as well as between doctors and AI systems in the realm of healthcare. We hypothesize that some of the future AI systems in healthcare will increasingly involve LLMs. Simultaneously, these systems will need to intensify their commitment to adhering to transparency, interpretability and stringent regulatory frameworks pertaining to data security and medical device regulations.

Conflict of Interest

The authors declare no conflict of interests.

Varghese J, Chapiro J. ChatGPT: The transformative influence of generative AI on science and healthcare. Journal of Hepatology [Internet]. 2023 Aug 5 [cited 2023 Sep 7]; Available from: https://www.sciencedirect.com/science/article/pii/S0168827823050390
Deng J, Lin Y. The Benefits and Challenges of ChatGPT: An Overview. Frontiers in Computing and Intelligent Systems. 2022;2(2):81–3.
Surameery NMS, Shakor MY. Use Chat GPT to Solve Programming Bugs. International Journal of Information Technology & Computer Engineering (IJITC) ISSN : 2455-5290. 2023 Jan 28;3(01):17–22.
Zheng H, Zhan H. ChatGPT in Scientific Writing: A Cautionary Tale. The American Journal of Medicine. 2023 Aug 1;136(8):725-726.e6.
Yang H. How I use ChatGPT responsibly in my teaching. Nature [Internet]. 2023 Apr 12 [cited 2023 Apr 16]; Available from: https://www.nature.com/articles/d41586-023-01026-9
Beutel G, Geerits E, Kielstein JT. Artificial hallucination: GPT on LSD? Crit Care. 2023 Apr 18;27(1):148.
Alkaissi H, McFarlane SI. Artificial Hallucinations in ChatGPT: Implications in Scientific Writing. Cureus [Internet]. 2023 Feb [cited 2023 Apr 16];15(2). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9939079/
North F, Ward WJ, Varkey P, Tulledge-Scheitel SM. Should You Search the Internet for Information About Your Acute Symptom? Telemedicine and e-Health. 2012 Apr;18(3):213–8.
Cocco AM, Zordan R, Taylor DM, Weiland TJ, Dilley SJ, Kant J, et al. Dr Google in the ED: searching for online health information by adult emergency department patients. Medical Journal of Australia. 2018;209(8):342–7.
Tang H, Ng JHK. Googling for a diagnosis—use of Google as a diagnostic aid: internet based study. BMJ. 2006 Nov 30;333(7579):1143–5.
Vallo Hult H, Master &#214, Stlund C, P&#229, Lsson P. Why and When Physicians Google: Resident Physicians’ Information-Seeking Strategies During Patient Consultations. In: Healthcare Transformation with Informatics and Artificial Intelligence [Internet]. IOS Press; 2023 [cited 2023 Sep 8]. p. 580–3. Available from: https://ebooks.iospress.nl/doi/10.3233/SHTI230563
Kung TH, Cheatham M, Medenilla A, Sillos C, Leon LD, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health. 2023 Feb 9;2(2):e0000198.
Fijačko N, Gosak L, Štiglic G, Picard CT, Douma MJ. Can ChatGPT pass the life support exams without entering the American heart association course? Resuscitation [Internet]. 2023 Apr 1 [cited 2023 Sep 26];185. Available from: https://www.resuscitationjournal.com/article/S0300-9572(23)00045-X/fulltext
Biswas S. ChatGPT and the Future of Medical Writing. Radiology. 2023 Apr;307(2):e223312.
Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. Journal of Esthetic and Restorative Dentistry [Internet]. [cited 2023 Sep 26];n/a(n/a). Available from: https://onlinelibrary.wiley.com/doi/abs/10.1111/jerd.13046
Jeblick K, Schachtner B, Dexl J, Mittermeier A, Stüber AT, Topalis J, et al. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports [Internet]. arXiv; 2022 [cited 2023 Sep 26]. Available from: http://arxiv.org/abs/2212.14882
Hirosawa T, Harada Y, Yokose M, Sakamoto T, Kawamura R, Shimizu T. Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study. International Journal of Environmental Research and Public Health. 2023 Jan;20(4):3378.
Griggs RC, Batshaw M, Dunkle M, Gopal-Srivastava R, Kaye E, Krischer J, et al. Clinical research for rare disease: Opportunities, challenges, and solutions. Molecular Genetics and Metabolism. 2009 Jan 1;96(1):20–6.
R: The R Project for Statistical Computing [Internet]. [cited 2023 Sep 8]. Available from: https://www.r-project.org/
Signorell A, Aho K, Alfons A, Anderegg N, Aragon T, Arachchige C, et al. DescTools: Tools for Descriptive Statistics [Internet]. 2023 [cited 2023 Sep 8]. Available from: https://cran.r-project.org/web/packages/DescTools/index.html
Bonferroni C. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze. 1936;8:3–62.
Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159–74.
Lenders JWM, Duh QY, Eisenhofer G, Gimenez-Roqueplo AP, Grebe SKG, Murad MH, et al. Pheochromocytoma and Paraganglioma: An Endocrine Society Clinical Practice Guideline. The Journal of Clinical Endocrinology & Metabolism. 2014 Jun 1;99(6):1915–42.

There is NO Competing Interest.

Download PDF

Journal Publication

published 06 Mar, 2024

Read the published version in Nature Communications →

Version 1

posted

You are reading this latest preprint version

Progression of Large Language Models for Clinical Decision Support: An Evaluation for Rare and Frequent Diseases using GPT-3.5, GPT 4 and Naïve Google Search

Status:

Journal Publication

Version 1

Abstract

Figures

Research in context

Introduction

Materials and Methods

Results

Inter-rater reliability

Performance evaluation of GPT-3·5, GPT-4 and Google

Discussion

Strengths and limitations

Conclusion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 1