Examining the Quality of General Purpose AI based Recommendation System for Over-the-Counter Medication

doi:10.21203/rs.3.rs-4869677/v1

Download PDF

Research Article

Examining the Quality of General Purpose AI based Recommendation System for Over-the-Counter Medication

https://doi.org/10.21203/rs.3.rs-4869677/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

This study investigates the quality of the over-the-counter medicines recommended by commercial general purpose artificial intelligence (AI) platforms by analyzing the relationship between two categorical data. 139 tests were performed using publicly available over the counter medication database from US FDA and two widely available commercially available general purpose AI platforms. A chi-square analysis was conducted to examine the relationships. Results indicated that there is a statistically significant difference between the quality of over-the-counter medication recommendations from the FDA Database-based system and those recommended by commercially available generative AI systems with chi-square statistic 0.1208, the p-value of .728145, not significant at p < .01 for one AI system and the chi-square statistic = 0.9988, the p-value of .317613, not significant at p < .01 for second AI system. These results indicate that recommendations made by widely available generative AI platforms are not suitable for medical advice. Additionally, the study found that there is no statistically significant difference between the quality of over-the-counter medication recommendations from two commercially available generative AI systems with the chi-square statistic = 61.4366, the p-value of < 0.00001, significant at p < .01. This result suggests that the recommendations made by these generative AI systems are consistently inaccurate. These findings highlight the need for the development of specialized generative AI systems with the involvement of domain experts, to ensure accuracy and reliability for providing the OTC medication recommendations. Table of Contents

The advancement and commercialization of generative artificial intelligence (Gen-AI) has revolutionized many fields including healthcare. Gen-AI has shown the ability to process vast amounts of data and reduce the human computer interaction gap by generating human-like responses. As a result, Gen-AI based systems are increasingly being utilized for various applications, including the recommendation of over the counter (OTC) medications. However, the reliability and quality of these recommendations remain under scrutiny, raising important questions about their efficacy and safety. This study seeks to address two critical research questions; (a) is the quality of over-the-counter medication recommendations by commercially available Gen-AI systems adequately reliable? and (b) is there a significant difference in the quality of over-the-counter medication recommendations between two commercially available Gen-AI systems? This research aims to ascertain Gen-AI based systems’ reliability and identify any discrepancies between different systems. This investigation is crucial for ensuring that AI-driven healthcare solutions meet the necessary standards for patient safety and efficacy. This study will benefit consumers and healthcare providers in making informed decisions of using Gen-AI in healthcare.

The increasing integration of artificial intelligence (AI) into healthcare has sparked both excitement and concern, particularly regarding the use of generative AI (Gen-AI) systems for providing medical advice. Among the various applications, one of the most notable is the recommendation of over the counter (OTC) medications. These systems are designed to assist consumers by offering personalized advice based on input symptoms and conditions, potentially improving accessibility to healthcare information. However, the reliability and consistency of these recommendations are critical, given their potential impact on public health.

Historically, OTC medication advice has been provided by healthcare professionals, including pharmacists and physicians, who rely on their expertise and experience to recommend appropriate treatments. The shift towards AI-driven recommendations is driven by the promise of greater accessibility and efficiency. Gen-AI systems, trained on extensive medical databases and employing advanced natural language processing techniques, can theoretically provide rapid and accurate advice. However, the quality of this advice must be rigorously evaluated to ensure it meets the necessary standards of accuracy, safety, and reliability.

Several studies have examined the general capabilities of AI in healthcare, highlighting both the potential benefits and the risks associated with its use. Issues such as data quality, algorithmic bias, and the contextual understanding of AI systems have been identified as significant challenges. These concerns are particularly pertinent in the context of OTC medications, where incorrect or inappropriate recommendations can lead to adverse health outcomes. To address these concerns, this study aims to answer two pivotal research questions: Is the quality of over-the-counter medication recommendations by commercially available Gen-AI systems adequately reliable? Is there a significant difference in the quality of over-the-counter medication recommendations between two commercially available Gen-AI systems?

By systematically evaluating Gen-AI based systems' performance, this research seeks to provide a full assessment of their reliability and consistency. This includes analyzing the accuracy of the recommendations by comparing the recommendations with the purpose for the OTC medication identified by FDA as a bases for comparing their adherence to established medical guidelines. Additionally, by comparing different Gen-AI systems, the study aims to identify any significant differences in their remediation quality performance, thereby offering insights into the variability and robustness of AI-driven healthcare solutions.

The findings of this research will contribute to provide clarity with the readiness and the role of AI in healthcare, providing valuable information for consumers, healthcare providers, and policymakers.

Research Questions

Despite their growing use, the reliability and quality of these recommendations remain under scrutiny, raising significant concerns about their efficacy and safety. This study seeks to address the following critical research questions:

Q1. Is the quality of over-the-counter medication recommendations by commercially available Gen-AI system adequately reliable?

Q2. Is there a significant difference in the quality of over-the-counter medication recommendation between two commercially available Gen-AI systems?

By investigating these questions, this research aims to determine the reliability of Gen-AI-based systems and identify improvement opportunities.

Hypotheses

H1_o. There is no statistically significant difference between the quality of over-the-counter medication recommendation between the FDA Database based system and medication recommended by commercially available Gen-AI.

H1_a. There is a statistically significant difference between the quality of over-the-counter medication recommendation between the FDA Database based system and medication recommended by commercially available Gen-AI.

H2_o. There is no statistically significant difference between the quality of over-the-counter medication recommendation between two medications recommended by two commercially available Gen-AI.

H2_a. There is statistically significant difference between the quality of over-the-counter medication recommendation between two medications recommended by two commercially available Gen-AI.

Nature of the Study

This is a quantitative study with primary aim to evaluate the reliability and quality of over the counter (OTC) medication recommendations provided by commercially available generative artificial intelligence (Gen-AI) systems. This research was an essential first step for determining and hence ensuring that AI-driven healthcare solutions meet necessary standards for patient safety and efficacy. This study employs a comparative, quantitative research design. By comparing two commercially available Gen-AI systems with FDA Database of OTC Medications, the research was aimed to quantify and analyze the reliability and quality of their OTC medication recommendations.

OTC medication recommendations from two different commercially available Gen-AI systems were collected for a set of common health conditions for which OTC medications are typically recommended were used for the prompts. A diverse range of health conditions such as cold, headache, fever, allergies were included to ensure comprehensive evaluation. Multiple instances of recommendations for each health condition were collected to account for variability. The collected data was analyzed using statistical methods to compare the quality of recommendations between the two Gen-AI systems and inferential statistics (chi-square test) was employed to identify significant differences between the systems. The study focuses on the quality of OTC medication recommendations and did not cover prescription medications or other types of healthcare recommendations.

Significance of the Study

The findings of this study were to offer insights for consumers and healthcare providers regarding the reliability of AI recommendations for over-the-counter drugs without any manipulation of the model. These insights will help determine whether the AI model is suitable for providing medical recommendations or if it requires the expertise of domain experts to ensure its accuracy and reliability.

In recent years, technological advancements have transformed various sectors worldwide, streamlining tasks through automation, including in the healthcare industry. However, alongside these innovations, concerns have emerged, particularly regarding the implementation of artificial intelligence (AI) in medical settings. Understanding AI's functionality is crucial for patients to mitigate its potential negative implications in healthcare. Research by Adus et al. (2023) underscores the importance of patient participation in this process. Patients suggested that physicians should serve as intermediaries between AI technology and patients, as doctors remain the primary end-users. This role can bridge the gap between advanced AI applications and patient needs, fostering better acceptance and utilization of AI in healthcare. However, questions arise about how physicians' self-perception and patients' perception of them will change when AI assumes major responsibilities, potentially relegating doctors to tasks like patient communication and accuracy verification (Svensson & Jotterand, 2022).

A survey of 300 participants revealed that 30–40% of doctors were aware of AI, and 85% believed it could be used in healthcare. While 75% thought doctors utilizing AI would replace those not using it, 90% believed they should be involved in developing these products. Concerns were also raised, with 55% believing that inadequate information or controversial opinions might cause issues (Orlova et al., 2023). To ensure the trustworthiness of medical AI models, it is essential to evaluate their reliability, competency, and intentions. This framework can help determine if AI is ready for large language models and represents their trustworthiness using these three parameters (Fenster, 2023).

In developing healthcare AI technology, algorithmic discrimination (AD) can occur at three points: measurement error since available data may be incorrect or incomplete, selection bias since specific data used to train AI may be unrepresentative, and feedback loop bias as data may represent historical patterns of discrimination (Starke et al., 2021). Another concern is the "black box" nature of AI algorithms, where users do not fully understand the data analysis process, potentially leading to bias. AI in healthcare involves training models using large datasets to recognize patterns and make predictions. AI can assist physicians in interpreting complex medical data, streamlining diagnoses, and personalizing treatment plans (Starke et al., 2021).

Currently, AI, particularly deep learning, can match or exceed human performance in specialized tasks. Simpler technologies like the "PREDICT" regression model also benefit healthcare by estimating cardiovascular risks and providing treatment recommendations. The explainability of deep neural networks remains an active research area (Warren, 2019). AI can identify new indications for existing drugs. IBM Watson Oncology analyzes patient medical records to suggest evidence-based and personalized treatment options for specific malignancies. Watson has also demonstrated the ability to classify proteins involved in heart disease. Swarm technology is another AI concept that could allow groups of doctors to combine their thinking into a virtual collective. However, challenges remain. There is limited data on how these systems perform when dealing with multiple diseases simultaneously, and unsupervised learning from medical records has not been fully demonstrated (Orlova et al., 2023).

Generative AI, such as ChatGPT, is being utilized in patient data analysis and real-time medical decision-making. It aids healthcare professionals in making informed decisions based on patients' symptoms, vital signs, and laboratory results. This technology can rapidly detect clinical deterioration and produce tailored medical care recommendations. It is also being applied to disease prevention, prognosis, and treatment planning. ChatGPT complements healthcare professionals' clinical knowledge and experience in quality assessment, effectiveness, usability, and accuracy when analyzing patients' potential diseases. The process involves ChatGPT analyzing both its trained database and patients' clinical data to produce accurate diagnoses and predict treatment outcomes. It also uses clinical outcomes to identify incongruencies in patient lab tests, potentially enhancing patient care (Henley, 2023).

One of the studies that has provided evidence for the efficacy of generative AI is the one developed by (Qarajeh et al., 2023). It evaluated the efficacy of four AI models—ChatGPT 3.5, ChatGPT 4, Bard AI, and Bing Chat—in categorizing foods based on their potassium and phosphorus content for chronic kidney disease (CKD) patients. The models were tasked with classifying items as high or low in potassium (149 items) and high in phosphorus (91 items) using 240 food items from the Mayo Clinic Renal Diet Handbook. Results demonstrated varying accuracies across models. For potassium content, ChatGPT 4 and Bing Chat performed best with 81% accuracy, followed by Bard AI (79%) and ChatGPT 3.5 (66%). Regarding phosphorus content, Bard AI achieved perfect accuracy (100%), with Bing Chat (89%), ChatGPT 3.5 (85%), and ChatGPT 4 (77%) following. These findings highlight the potential of AI models in renal dietary planning while also indicating areas for improvement to enhance their utility in clinical settings.

A quantitative research method was employed using a systematic approach. The study involved exploring various well-recognized and industry-leading large language models (LLMs) capable of recommending over the counter (OTC) medicines through their application programming interfaces (APIs). After a thorough evaluation, OpenAI's ChatGPT and Anthropic's Claude AI were selected for the study. Although Google’s Gemini LLM was considered, it was not selected due to its safety features, which prevented it from consistently recommending medications.

The reference OTC medicines, and their related attributes were sourced from the U.S. Food and Drug Administration (FDA) website. Data files were downloaded in JSON format from the official website, and the necessary data columns, including brand name, generic name, active ingredient, and purpose, were extracted, and stored in an Amazon RDS database. A list of symptoms treatable by OTC medications was compiled to generate test cases for evaluating the accuracy and quality of the LLMs' recommendations. The required number of test cases was determined using GPower analysis, which aids in calculating the sample size necessary for conducting a chi-square analysis when comparing categorical variables. The GPower tool’s priori test suggested 146 test cases, based on an effect size of 0.3 and a power of 0.95.

All test cases were cross-referenced with the FDA-recommended medications in the RDS database, ensuring that the symptom was present in the "purpose" column, thus allowing the creation of a list of potential medicines along with their active ingredients for each symptom. These test cases were then phrased in natural language commonly used by humans when interacting with LLMs and were submitted to ChatGPT and Claude AI using a standardized prompt. The outputs from the LLMs were then assessed for accuracy and quality and converted into a binary yes/no format for statistical analysis using the chi-square test.

During the quality checks, 7 test cases were removed due to duplication or errors identified upon further analysis. The final sample size was recalculated to correspond with the number of symptoms used to generate the LLMs' recommendations.

Assumptions

The medication list obtained from the FDA database was assumed to be accurate and complete. The large language model (LLM) was expected to return medication recommendations regardless of the patient's age, sex, or medical history.

Limitations

It is important to acknowledge the limitations inherent in this study. First, the accuracy of the various large language models (LLMs) was evaluated using a specific database of over-the-counter drugs sourced from the FDA, which contained some omissions and errors. Additionally, while models such as ChatGPT and Claude demonstrated varying levels of accuracy, the underlying reasons for these differences were not explored. Furthermore, generative AI models are subject to continuous improvement through updates and patches, meaning that the conclusions drawn from this study may not remain valid over time. Finally, the G*Power tool determined that 145 test cases were required for optimal results in the chi-square test. However, the final number of test cases used was 139, as the initial list was refined to eliminate incorrect entries and duplicates.

Delimitations

The omissions and errors in the FDA database were manually corrected to enhance the quality of the study's results. Future research could explore the underlying factors contributing to the successes and limitations of the models, providing a more comprehensive understanding of the functioning of large language models (LLMs). Additionally, repeating this study in the future is recommended to address the challenges posed by the continuous improvement of generative AI.

Ethical Assurances

This research does not involve any studies with human participants or animals conducted by the authors. The research was performed by extracting and inspecting publicly accessible files and tools. All data analyzed during this study are publicly available. This research was not funded by any organization and hence there were no commercial or financial relationships that could be construed as a potential conflict of interest in the research.

Results of the finding supported the first hypothesis that there is a statistically significant difference between the quality of over-the-counter medication recommendations from the FDA Database-based system and those recommended by commercially available generative AI systems. The null hypothesis was rejected as the chi-square statistic was 0.1208 with the p-value of .728145, not significant at p < .01 for FDA / ChatGPT and the null hypothesis was rejected as the chi-square statistic was 0.9988 with the p-value of .317613, not significant at p < .01 for FDA / ClaudeAI. These results indicate that recommendations made by widely available generative AI platforms are not suitable for medical advice.

Results of the finding did not support the second hypothesis. There is no statistically significant difference between the quality of over-the-counter medication recommendations from two commercially available generative AI systems. The null hypothesis was accepted as the chi-square statistic was 61.4366 with the p-value of < 0.00001, significant at p < .01 for FDA / ClaudeAI. This result suggests that the recommendations made by these generative AI systems are consistently inaccurate.

Other Observations:

In some instances, Commercial general purpose generative AI systems recommended prescription medications even when explicitly asked to provide over-the-counter options.

Implications

The findings from this study carry significant implications for both the development and deployment of generative AI systems in the healthcare domain, particularly in the context of providing over the counter (OTC) medication recommendations.

Firstly, the rejection of the null hypothesis in the first hypothesis (H1a) emphasizes the clear disparity in the quality of recommendations between an FDA Database-based system and those generated by commercially available generative AI platforms. This suggests that generative AI systems currently available to the public are not yet reliable or accurate enough for making medical recommendations. Consequently, this raises concerns about the potential risks of misinformation and incorrect medication guidance, which could lead to adverse health outcomes for users who rely on these systems without further consultation with a healthcare professional.

The acceptance of the null hypothesis in the second hypothesis (H2o) highlights that there is no significant difference in the quality of recommendations between different generative AI platforms (Claude AI and ChatGPT), with both showing consistently inaccurate results. This suggests a broader, systemic issue with the way these AI models are trained and utilized for medical advice, rather than an isolated problem with one Specific platform. It underscores the need for caution in the deployment of such technologies in sensitive areas like healthcare, where inaccuracies can have dire consequences.

Given these findings, the implications are clear: there is an urgent need for the development of specialized generative AI systems that are rigorously trained and validated with the assistance of domain experts, particularly in healthcare. Such systems must be subject to stringent testing and evaluation before being deemed safe and reliable for public use. Moreover, these findings should encourage policymakers and regulatory bodies to establish strict guidelines and oversight mechanisms for the use of AI in medical and health-related contexts to ensure that users are not misled by inaccurate information.

Finally, healthcare providers and the public should be made aware of the current limitations of generative AI in providing medical advice, and the importance of consulting qualified healthcare professionals for accurate and safe medication recommendations.

Commercial general purpose generative AI systems are not suitable for providing over-the-counter medication recommendations. It is recommended that a specialized generative AI system be developed, with the involvement of domain experts, to ensure accuracy and reliability in such critical applications.

Adus, S., Macklin, J., & Pinto, A. (2023). Exploring patient perspectives on how they can and should be engaged in the development of Artificial Intelligence (AI) applications in health care. BMC Health Services Research, 23(1). https://doi.org/10.1186/s12913-023-10098-2
Fenster, J. (2023). An antidote for patients: Combatting the discriminatory effects of artificial intelligence in healthcare. The Fordham Urban Law Journal, 50. https://doi.org/https://ir.lawnet.fordham.edu/ulj/vol50/iss2/6
Henley, S. (2023). Generative Artificial Intelligence-based clinical decision support in screening, prevention, and treatment choices in medical care. Contemporary Readings in Law and Social Justice, 15(1), 27. https://doi.org/10.22381/crlsj15120232
Orlova, I. A., Akopyan, Zh. A., Plisyuk, A. G., Tarasova, E. V., Borisov, E. N., Dolgushin, G. O., Khvatova, E. I., Grigoryan, M. A., Gabbasova, L. A., & Kamalov, A. A. (2023). Opinion research among Russian physicians on the application of technologies using artificial intelligence in the field of Medicine and Health Care. BMC Health Services Research, 23(1). https://doi.org/10.1186/s12913-023-09493-6
Qarajeh, A., Tangpanithandee, S., Thongprayoon, C., Suppadungsuk, S., Krisanapan, P., Aiumtrakul, N., Garcia Valencia, O. A., Miao, J., Qureshi, F., & Cheungpasitporn, W. (2023). AI-powered renal diet support: Performance of CHATGPT, Bard Ai, and Bing Chat. Clinics and Practice, 13(5), 1160–1172. https://doi.org/10.3390/clinpract13050104
Starke, G., van den Brule, R., Elger, B. S., & Haselager, P. (2021). Intentional machines: A Defence of trust in Medical Artificial Intelligence. Bioethics, 36(2), 154–161. https://doi.org/10.1111/bioe.12891
Svensson, A. M., & Jotterand, F. (2022). Doctor ex machina: A critical assessment of the use of Artificial Intelligence in health care. The Journal of Medicine and Philosophy: A Forum for Bioethics and Philosophy of Medicine, 47(1), 155–178. https://doi.org/10.1093/jmp/jhab036
Warren, J. (2019). How will ai change health care delivery? New Zealand Medical Student Journal. http://www.nzmsj.com/how-will-ai-change-health-care-delivery.html

No competing interests reported.

AppendixA.docx

Download PDF

Reviews received at journal
07 Oct, 2024
Reviews received at journal
07 Oct, 2024
Reviews received at journal
05 Oct, 2024
Reviewers agreed at journal
27 Sep, 2024
Reviewers agreed at journal
26 Sep, 2024
Reviewers agreed at journal
26 Sep, 2024
Reviews received at journal
19 Sep, 2024
Reviewers agreed at journal
09 Sep, 2024
Reviewers invited by journal
06 Sep, 2024
Editor assigned by journal
16 Aug, 2024
Submission checks completed at journal
12 Aug, 2024
First submitted to journal
06 Aug, 2024

You are reading this latest preprint version

Examining the Quality of General Purpose AI based Recommendation System for Over-the-Counter Medication

Status:

Version 1

Abstract

Introduction

Background

Nature of the Study

Significance of the Study

Literature Review

Research Method

Assumptions

Limitations

Delimitations

Ethical Assurances

Findings

Other Observations:

Implications

Conclusions

References

Additional Declarations

Supplementary Files

Status:

Version 1