Assessing the Statistical Abilities of GPT-4 as a Zero-shot Reasoner: Strengths and Weaknesses

doi:10.21203/rs.3.rs-2958780/v1

Download PDF

Research Article

Assessing the Statistical Abilities of GPT-4 as a Zero-shot Reasoner: Strengths and Weaknesses

https://doi.org/10.21203/rs.3.rs-2958780/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

The field of artificial intelligence (AI) has witnessed remarkable advancements in natural language processing and understanding (NLP and NLU), particularly through the development of large language models like GPT-4. These models have demonstrated promise across various domains, including medicine. However, the extent to which they can reason statistically as zero-shot reasoners remains largely unexplored.

Objective

This study aims to evaluate GPT-4's statistical abilities as a zero-shot reasoner and identify its strengths and weaknesses in addressing statistical inquiries.

Methods

A diverse set of statistical questions was extracted from published articles and inputted into the GPT-4 chatbot. The responses generated by GPT-4 were assessed for accuracy and compared against the correct answers. Logistic regression analyses were performed to determine the impact of statistical topics and AI tasks on the model's performance.

Results

GPT-4 achieved an overall accuracy of 74% in responding to the statistical questions. The analysis revealed that the model excelled at comprehending well-structured and unambiguous questions, showcasing proficiency in entity recognition and information integration. However, it encountered difficulties with complex statistical concepts, data interpretation, and questions involving ambiguity or convoluted structures.

Conclusion

GPT-4 demonstrated strengths in understanding statistical concepts when presented clearly. However, it faced challenges with more intricate tasks and synthesizing information from multiple sources. Suggestions for improvement include fine-tuning the model using advanced statistical datasets, incorporating external knowledge sources, optimizing prompt engineering techniques, and enabling visual information processing.

Implications: This study offers insights into the strengths and weaknesses of GPT-4 as a zero-shot reasoner in statistical tasks. Although the model exhibits potential in basic statistical reasoning, caution should be exercised in relying solely on its responses without human supervision for comprehensive statistical analysis and interpretation. Additionally, given that most researchers in the medical field may lack statistical expertise, leveraging language models can be valuable for addressing their statistical inquiries.

The field of artificial intelligence has gained significant attention in recent years, including in the medical field^1–3. One of its subfields, natural language processing and understanding (NLP and NLU, respectively), has garnered even more attention in recent months due to new architectural structures known as transformers, which have led to significant advancements in this field⁴. Natural language processing and understanding is a branch of artificial intelligence that focuses on the interaction between computers and human language, with the goal of enabling computers to process and understand human language and perform various tasks such as speech recognition, syntactic analysis, information extraction, among others.

The evolution of the architecture of these technologies has allowed for the development of large language models. These models are complex algorithms and architectures based on NLP and NLU that have been extensively trained on large amounts of text data to generate contextually relevant and coherent responses. After training on various sources such as books, web articles, documents, conversations, and more, the model can generate new texts based on the inputs it receives, known as prompts⁵. The model recognizes the input prompt and then performs the requested task, such as answering questions, translating texts, suggesting completions for given texts, among others. To achieve this, the model utilizes various tasks for which it has been trained, such as contextual understanding, text classification, text normalization, among others, in order to accurately respond to the input prompt^6,7.

One of the abilities of these models is the ability to perform reasoning for input prompts even without being explicitly trained for a specific task, known as zero-shot reasoning⁵. An example of zero-shot reasoning in the medical field could be as follows: a user inputs the following statement into a chatbot (which is essentially a conversational interface between a large language model and a user): "I have a persistent cough, sore throat, and difficulty breathing." Even if the model has never seen this specific statement during training, it can perform untrained reasoning and make inferences about possible medical conditions associated with these symptoms. Based on the prior knowledge acquired during training on medical tasks, the model can identify that these symptoms may indicate various conditions such as the common cold, flu, asthma, bronchitis, or even a more serious respiratory infection⁸.

Although these models demonstrate high efficiency in responding as zero-shot reasoners, they can make errors because they are extrapolating their prior knowledge to make inferences in tasks or domains not encountered during training⁶. One solution to improve their efficiency and extract the best content from these models is prompt engineering, which involves carefully and strategically creating prompts to guide the behavior of large language models in response to requested tasks⁹. However, the most commonly used tools utilizing this technology are online chatbots, and users may not always have the necessary knowledge to create refined prompts to extract the best responses from the models. This has led to the need to understand the capabilities of these models as zero-shot reasoners in various tasks, such as statistics.

The field of statistics poses challenges for all professionals, and several studies have indicated that it is an area of limited proficiency among healthcare professionals, such as medical residents, medical students, among others^10–12. Knowledge of statistics, as well as other research concepts, is essential in the medical field, as statistical errors in this area are unethical, time-consuming, costly, and detrimental to humanity and science¹³. The lack of proficiency in a subject, coupled with a powerful tool like large language models, can create the impression that even in the face of incorrect answers, the user may perceive them as true¹⁴.

Given these considerations, it is essential to test large language models to understand their statistical abilities as zero-shot reasoners. In this article, the GPT-4 (generative pre-trained) model, the most advanced model currently available, will be tested with various questions that assess statistical literacy and statistics in the healthcare context to understand its strengths and weaknesses when prompted as a zero-shot reasoner.

We conducted a search for articles published in journals or books that included questions related to statistics to test medical knowledge or general statistical literacy. We sought articles with heterogeneous question formats, including both direct and concise questions, as well as more complex and lengthy ones^10,11,15. The questions were then classified into statistical subtopics. The questions were inputted into the GPT-4 chatbot exactly as extracted from the published articles to simulate the GPT-4's behavior as a zero-shot reasoner. The tables were transformed into Markdown format for input into the model, as it is the currently accepted format. Questions with complex tables or images of tables and graphs were excluded due to the models' current inability to interpret them. After each question was inputted, the chatbot was reseted to ensure that the conversation history did not influence subsequent responses and to preserve the zero-shot reasoning.

Following the GPT-4's output, we cataloged the responses and compared them with the correct answers. The model was then questioned about the necessary tasks in artificial intelligence, such as NLP and NLU, required to answer each question. All questions, along with their respective classifications into statistical topics and the required tasks, can be found in the indexed file (Indexed 1).

Statistical analysis was performed using the Python software^16–18. We calculated the accuracy of correct answers and the 95% confidence interval of the model. To analyze the influence of statistical topics and AI tasks on the model's accuracy, we conducted three logistic regression analyses. The first regression analyzed the isolated variables of the statistical topics, the second regression analyzed the isolated variables of the AI tasks, and the third regression considered all variables to better understand how the synergy between statistical topics and AI tasks could weaken or strengthen the model's accuracy.

We presented 54 questions to the GPT-4 chatbot, of which it correctly answered 40, resulting in an accuracy of 74% (95% CI, [59.4%, 84.5%]). The number of questions answered by statistical topics and AI tasks can be found in Tables 1 and 2, respectively. Figuress 1 and 2 represent the bar graphs showing the percentages from Tables 1 and 2, respectively. Some questions encompassed multiple statistical topics and AI tasks, as noted in the tables and graphs and can be visualized in the comprehensive table of questions and their classifications (Indexed 1). The three logistic regression models exhibited different accuracies and log odds coefficients, as shown in Table 3.

Figure 1. Bar plot of corrected and incorrect questions answered by the model, by statistical topic.

Table 1. Statistical Topic count according to correct and incorrect answers, and total sum of appearances

Figure 2. Bar plot of corrected and incorrect questions answered by the model, by AI task.

Table 2. AI Task count according to correct and incorrect answers, and total sum of appearances

Table 3. Accuracy and log odds coefficients of the three logistic regression models, and absolute differences when considering models with statistical topic, AI task, or both.

A detailed analysis of the results from the three logistic regressions allows us to infer the strengths and weaknesses of GPT-4 as a zero-shot reasoner for questions related to statistical knowledge.

Logistic regression with statistical topics as predictor variables (Figure 3)

Figure 3. Contribution of Statistical Topic to correct answering, as indicated by the log odds coefficient of the logistic regression model incorporating only Statistical Topic related variables.

The most influential statistical topics for the GPT-4's ability to provide correct responses were Inferential Statistics, Sampling Theory, and Multivariate Analysis. Conversely, topics such as Descriptive Statistics, Test Performance, Power Analysis, and Comparison showed negative coefficients, indicating that the model may have difficulties in handling these concepts. This classification model achieved an accuracy of 0.72 with these predictor variables, ranking as the second-highest accuracy among the three models, partly due to a greater occurrence of statistical topics for the logistic regression.

Logistic regression with AI tasks as predictor variables (Figure 4)

Figure 4. Contribution of AI Task to correct answering, as indicated by the log odds coefficient of the logistic regression model incorporating only AI Task related variables.

In the second logistic regression model, which focused on AI tasks, the most relevant variables were Text Normalization, Text Classification, and Knowledge Representation. These tasks showed positive coefficients, suggesting that GPT-4 was more effectively trained in these specific areas. On the other hand, the Contextual Understanding task exhibited a negative coefficient, indicating that the model may have more difficulties in comprehending the context in which the questions are posed. This second regression model achieved an accuracy of 0.63 with the predictor variables solely from AI tasks, presenting the lowest accuracy among the regression models. This could be partly explained by the smaller number of variables and the high correlation among them, as observed in Table 2 (number of AI tasks).

Logistic regression with both statistical topics and AI tasks as predictor variables (Figure 5)

Figure 5. Contribution of Statistical Topic and AI Task to correct answering, as indicated by the log odds coefficient of the logistic regression model incorporating all variables.

The third logistic regression model, which included all variables, provided a more comprehensive view of GPT-4's performance. In addition to the aforementioned statistical topics and AI tasks, other relevant variables included Error Analysis, Probability Theory, and Regression Analysis. Conversely, variables such as Descriptive Statistics, Power Analysis, and Experimental Design showed negative coefficients, indicating that the model may face more difficulties in these specific areas. The logistic regression model that incorporated all dummy variables from statistical topics and AI tasks as predictors achieved the highest accuracy among the three classification models, at 0.81.

The three logistic regression models served to understand how GPT-4 performs as a zero-shot reasoner in different statistical tasks and topics, as well as varied question formats. The differences in log odds coefficients among the three logistic regression models indicate that the topic itself or the specific task alone is not the most important factor for the model to provide correct answers as a zero-shot reasoner, but rather other factors related to the question's format and content, and how both the statistical topic and AI task correlate. From this analysis, we can infer that other factor associated with better accuracy of the model are:

Clarity and specificity: Questions with clearer and more specific language reduce ambiguity and facilitate the model's understanding.
Familiarity with the topic: The model appears to have a better grasp of statistical concepts and methodologies in questions that fall within its training.
Straightforward structure: Questions structured in a manner that enables the model to identify key entities and relationships more easily.

To visualize the key terms related to the questions, Word Clouds were created (Figures 6 and 7) to display the most frequently occurring terms in correctly and incorrectly answered questions, respectively.

Figure 6. Word Cloud of the most frequently occurring term in correctly answered questions.

Figure 7. Word Cloud of the most frequently occurring term in incorrectly answered questions.

In summary, given these analyses, we highlighted some of the strengths and weaknesses of the model, as follows:

Strengths of the model:

Accurate understanding of statistical concepts and methodologies when presented clearly.
Ability to perform logical reasoning and synthesis when questions are well-structured and specific.
Strong performance in entity recognition and integration of information from different sources.

Weaknesses of the model:

Struggles with complex statistical questions, higher-order concepts, or data interpretation.
Difficulty in handling ambiguity and integrating information when the question structure is more convoluted.
Limited ability to accurately identify and relate key entities and concepts in more challenging questions.

Overall, the model demonstrates a good understanding of statistical concepts and methodologies when the questions are clear and well-structured. However, it encounters difficulties when the questions involve complex concepts, ambiguity, or convoluted structures. This highlights the model's limitations in handling complex tasks and integrating information from various sources as a zero-shot reasoner.

Limitations

This study has several limitations. Firstly, GPT-4 is currently unable to process visual information, such as plots and graphs, which limits its applicability in answering questions that require the interpretation of visual data. Secondly, we only assessed the model's performance in basic statistical tasks, and its ability to handle more advanced statistical concepts remains unknown. Thirdly, the performance of GPT-4 may be influenced by the quality of the prompts used, and alternative prompt engineering approaches may yield different results. Lastly, the number of statistical topics covered, as well as the number of AI tasks analyzed, were unbalanced, which should be addressed in future studies with a larger number of questions.

Suggestions:

To improve the performance of GPT-4 in similar question scenarios, we suggest the following (Figure 8):

Figure 8. Suggestions to Improve GPT-4 performance in answering statistical questions.

Fine-tuning: Fine-tune GPT-4 using a more specific dataset that focuses on advanced statistical concepts, study designs, and real-world applications. This will help the model acquire deeper domain knowledge and improve its performance on complex statistical questions.
Incorporate external knowledge: Enhance the model's ability to access external knowledge sources, such as databases or textbooks, to supplement its existing knowledge and improve its reasoning capabilities.
Prompt engineering: Improve the model's performance in reasoning tasks, especially those in which it performed poorly, by optimizing the input prompts and structuring the questions accordingly.
Incorporate visual processing: Add the capability to process visual information, such as plots and graphs, to enhance the model's understanding and reasoning for statistical questions that involve visual data interpretation.

Conclusion

Large language models, such as GPT-4, demonstrate promise in answering basic statistical questions. However, they also exhibit weaknesses, particularly in handling complex questions that require multiple reasoning steps or a deeper understanding of statistical concepts. Our findings suggest that while these models may be useful for certain tasks in medical research, they should not be solely relied upon for comprehensive statistical analysis or interpretation without human supervision.

To enhance the accuracy and efficiency of large language models, future research could focus on incorporating visual information processing, improving their understanding of advanced statistical concepts, and refining prompt engineering techniques. Additionally, ongoing efforts to train these models with larger and more diverse datasets can contribute to their overall performance as zero-shot reasoners in various domains.

Availability of data and materials

All data generated or analyzed during this study are included in this published article [and its supplementary information files].

Ethics approval and consent to participate: Not applicable.

Consent for publication: Not applicable.

Availability of data and materials: Data and materials are available in the indexed file.

Competing interests: The authors declare no competing interests.

Funding: This research has no funding sources.

Authors' contributions: Douglas Costa, João Victor, and Celso Nakano were involved in data collection and data analysis. Douglas Costa and Mario Luiz were involved in writing the manuscript. All authors were involved in reviewing the manuscript.

Acknowledgements: Not applicable

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Rong G, Mendez A, Bou Assi E, Zhao B, Sawan M. Artificial Intelligence in Healthcare: Review and Prediction Case Studies. Engineering. 2020;6(3):291-301. doi:https://doi.org/10.1016/j.eng.2019.08.015
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7
Rajkomar A, Dean J, Kohane I. Machine Learning in Medicine. N Engl J Med. 2019;380(14):1347-1358. doi:10.1056/NEJMra1814259
Vaswani A, Shazeer N, Parmar N, et al. Attention is All you Need. In: Guyon I, Luxburg U Von, Bengio S, et al., eds. Advances in Neural Information Processing Systems. Vol 30. Curran Associates, Inc.; 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Brown TB, Mann B, Ryder N, et al. Language Models are Few-Shot Learners. CoRR. 2020;abs/2005.1. https://arxiv.org/abs/2005.14165
Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners. OpenAI blog. 2019;1(8):9.
Radford A, Narasimhan K. Improving Language Understanding by Generative Pre-Training. In: ; 2018.
Lewis PSH, Denoyer L, Riedel S. Unsupervised Question Answering by Cloze Translation. CoRR. 2019;abs/1906.04980. http://arxiv.org/abs/1906.04980
Huang L, Ma S, Zhang D, Wei F, Wang H. Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt. Published online 2022.
Schield M. Assessing Statistical Literacy: Take Care (Extracts). In: Assessment Methods in Statistical Education: An International Perspective. ; 2010:133-152. doi:10.1002/9780470710470.ch11
Jenny MA, Keller N, Gigerenzer G. Assessing minimal medical statistical literacy using the Quick Risk Test: a prospective observational study in Germany. BMJ Open. 2018;8(8):e020847. doi:10.1136/bmjopen-2017-020847
Miles S, Price GM, Swift L, Shepstone L, Leinster SJ. Statistics teaching in medical school: Opinions of practising doctors. BMC Med Educ. 2010;10(1):75. doi:10.1186/1472-6920-10-75
Altman DG. The scandal of poor medical research. BMJ. 1994;308(6924):283-284. doi:10.1136/bmj.308.6924.283
Petroni F, Rocktäschel T, Lewis PSH, et al. Language Models as Knowledge Bases? CoRR. 2019;abs/1909.01066. http://arxiv.org/abs/1909.01066
Windish DM, Huot SJ, Green ML. Medicine residents’ understanding of the biostatistics and results in the medical literature. JAMA. 2007;298(9):1010-1022. doi:10.1001/jama.298.9.1010
van der Walt S, Colbert SC, Varoquaux G. The NumPy Array: A Structure for Efficient Numerical Computation. Comput Sci Eng. 2011;13(2):22-30. doi:10.1109/MCSE.2011.37
Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine Learning in {P}ython. J Mach Learn Res. 2011;12:2825-2830. http://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf
Python.org. https://www.python.org/

Variables	Statistical topics RL coefficients	AI Tasks RL coefficients	All variables RL coefficients	Absolute difference
RL model Accuracy	0.72	0.63	0.81

Question answering	-	0.00	0.00	0.00
Contextual understanding	-	-1.51	-1.44	0.09
Text normalization	-	0.57	0.58	0.01
Knowledge representation	-	0.75	0.76	0.01
Text classification	-	0.75	0.67	0.08

Descriptive statistics	-0.60	-	-0.60	0.00
Test performance	-0.56	-	-0.41	0.15
Power analysis	-0.56	-	-0.46	0.10
Comparison	-0.45	-	-0.36	0.09
Bayesian statistics	-0.41	-	-0.15	0.26
Regression analysis	-0.27	-	-0.31	0.04
Experimental design	-0.06	-	-0.07	0.01
Probability theory	-0.04	-	-0.35	0.31
Nonparametric statistics	0.13	-	0.23	0.10
Time series analysis	0.22	-	0.05	0.17
Error analysis	0.25	-	0.41	0.16
Multivariates analysis	0.43	-	0.57	0.14
Sampling theory	0.80	-	0.51	0.29
Inferential statistcs	1.03	-	0.51	0.52

Table 3. Accuracy and log odds coefficients of the three logistic regression models, and absolute differences when considering models with statistical topic, AI task, or both.

No competing interests reported.

Indexed1QuestionswithTopicsandTasks.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Assessing the Statistical Abilities of GPT-4 as a Zero-shot Reasoner: Strengths and Weaknesses

Status:

Version 1

Abstract

Figures

Introduction

Methodology

Results

Discussion and Conclusion

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1