Enhancing Mixed Methods Research with your generative AI assistant – A Tutorial and Evaluation for Scholars

doi:10.21203/rs.3.rs-4176435/v1

Scholars from all disciplines can benefit from learning how to use generative Artificial Intelligence (GAI or AI) for data analysis. The current article used real data to demonstrate the analysis and synthesis of mixed methods research (MMR) data with generative AI. I also explore how reliable and valid data outputs are and how to improve the use of generative AI for research. The demonstration data used emanated from a study done in South Africa, with a quantitative sample size of 969 first-year engineering students and, for the qualitative part, 14 second-year students. In the current article, I compare my original analysis to ChatGPT results. Generative AI is a mind tool that is ideal when utilised with human insight to check the cohesion, consistency, and accuracy of the data analysis. The current content is geared towards enhancing methodological application regardless of field or discipline and includes access to a prompt library and examples of using outputs. For the qualitative analysis, I found that ChatGPT could detect similar themes but missed some, and its write-up was shallower than our human version. The quantitative analysis was accurate for the descriptive statistics, but the researcher had to use best judgment to select the correct inferential analysis. A quantitative and qualitative analysis should be conducted separately in generative AI before asking the bot for help with mixed methods research. I give guidelines and a tutorial on how to use chatbots in an ethically responsible and scientifically sound manner for research in social and human sciences.

Generative Artificial Intelligence (AI or GAI)

Chat Generative Pre-Trained Transformer (ChatGPT) also known as Chad

Mixed Methods Research (MMR)

Large-Language Models (LLMs)

Chatbots

data analysis techniques

Recent advances in generative Artificial Intelligence (AI) are changing how we conduct scholarly research. We are challenged to use new tools in all educational disciplines [1–4]. The rapidly shifting technological landscape requires us to experiment with new tools, share what we have learned to support academics and students and engage in intellectual curiosity [5, 6]. Therefore, I showcase in this article how to use a generative AI platform to analyse quantitative and qualitative data and integrate findings for mixed methods research (MMR) using generative AI as an assistant. Chat Generative Pre-Trained Transformer (ChatGPT) has emerged as a dominant role player in the generative AI field, and higher institutions are reporting that students and academics are using the platform increasingly [7–9]. Some researchers [10] focus on the positive role that generative AI can play in higher education, for example, aiding in managing large classes and online learning. Other researchers focus on how generative AI will impact assessment [11], how it can be used to promote student learning and motivation [12] and which frameworks to use when integrating Chatbots into teaching and learning [13]. Most articles conclude that we need more research and guidelines on using generative AI in higher education [13, 14].

The current article is not the first exploration of how to use generative AI in higher education [15]; many good publications exist [16, 17]. The current article adds a new approach by demonstrating the application of generative AI in mixed methods research (MMR) and providing prompts and guidelines for use. My work here is intended to be a resource for students and the academy, showing how generative AI can feed into the research process for academic advancement [17]. Consequently, I use the following terms, which are distinct but related: generative Artificial Intelligence (AI or GAI), Chatbots and Large-Language Models (LLMs). By generative Artificial Intelligence (AI), I mean AI systems trained to produce new content based on but divergent from their training data [18]. Chatbots interface AI and humans by applying LLMs to interact through text or voice [19]. LLMs serve as the basis for many types of generative AI and were designed to understand and function through human language [20]. Conceptualising AI in terms of human interactions is functional and was the creators' intention when they relied on language as the medium of interaction. LLMs were designed to mimic natural human interactions, making it seem like you are speaking with someone [21]. I take a similar approach in this article, treating ChatGPT as an assistant and telling the story of how to use its functionalities in a more approachable, humanistic way [22]. Generative AI continues to evolve; for example, ChatGPT-3.5 is a text-to-text model, and ChatGPT-4 is a data-to-text model [23]. More exploration is needed to understand artificial intelligence's role in higher education, especially in research [24]. The usefulness of the current article is in its practical application to broaden our understanding of generative AI in higher education, and I present a tutorial which can be used by professors and students [25]. I used mixed methods research (MMR) for the tutorial and application because MMR offers us a robust framework for addressing many multifaceted issues in higher education. MMR explores complex human phenomena [26, 27]. Moreover, by analysing mixed methods data in ChatGPT, I create sections on quantitative and qualitative analysis that researchers from these disciplines could use even if they are not interested in MMR. The prompts and exercises in the current paper were piloted on over 100 academics in a South African university. Members of the academy from all Faculties and most departments at my university attended the training and provided feedback. I subsequently refined the tutorial based on the pilot.

Technological revolutions

Before the 2000s, software was syntax-based, and researchers had to write instructions [28–30]. This often led to considerable effort to fix and manage mistakes in writing syntax, apart from the knowledge and skills needed to conduct and interpret the data analysis [31]. The Graphical User Interface (GUI, ‘gooey’) revolution brought about easier use of statistics, where pointing and clicking was all the programming knowledge required [32]. At each phase of software evolution, the researcher needs to understand what they are doing and the statistical choices [33] or qualitative methods they employ [34]. The roles and responsibilities remain; the researcher must be well grounded in their understanding of data analysis theory and application. However, the researcher can now use the Chatbot as an assistant to suggest further analyses, conduct some of the analysis, and obtain help with interpreting outputs. At each step of the evolutionary process, there is less pressure on the researcher to know the underlying nuts and bolts of software and more opportunity to focus on the data. Researchers are only beginning to explore all the options that generative Artificial Intelligence offers [35]. Understandably, significant shifts are frightening, and the generative AI revolution in the academic world is no different. Luca Mari summarises this experience eloquently (2023, p.29):

Copernicus showed us our cosmological non-centrality

Darwin showed us our biological non-originality

Chatbots are showing us our cognitive non-uniqueness

Some of our shared fears include the idea that AI will fundamentally replace us, for example, being able to do work for which we were once considered crucial [36]. Another fear is that AI will reduce creativity, innovation and original work, replacing our humanity [37]. Hinks [38] found that negative expectations of artificial intelligence's role in our society are associated with reduced life satisfaction. Students are worried about the impact generative AI will have on their development, and the academy is concerned about academic integrity and the impact on assessment [39]. The Council on Higher Education [40] suggests that treating generative AI as a "crime" or plagiarism outright will negatively impact teaching and learning.

How should we deal with GAI in higher education? To navigate these existential crises and our trepidation, we need to engage with our fears reasonably, as Chatbots are here to stay [41]. We must find ethically sound ways to use the new tools to advance human knowledge and development. Our friends from the philosophy of science can help us to think critically about the latest existential crises. Heidegger was worried that technology would strip us of our humanity (forgetting to "be"), Idhe was concerned with its role in our daily lives, and Stiegler focused on how technology could disrupt our cultural memories [42, 43]. Hui [43] merges these ideas and sees technology as the conduit through which we understand our world (he calls this “cosmotechnics”). Baker and Hui are linked to the idea from Ihde [44] that science is the conceptual side of a technological society and that scientists should reimagine the role of technology and its ethical use. Metaphysics, cultural shifts and geopolitical implications cannot be ignored when significant technological changes occur [45]. New technologies shape the world around us, reshape how we see ourselves and lead to new traditions, potentially replacing our cultural heritages and requiring profound pluralism from us [43, 46]. In the research world, we measure what we value, pushing us to question what is valuable. While I focus on the practicalities in the current paper, I am also proposing a person-centred, meaning-making approach to scholarship and integrating artificial intelligence into our research [47]. I explore this idea further by examining how researchers can ethically apply the new GAI tools.

Ethical and safe use of generative AI in research

There is nothing inherently wrong with using generative AI during data analysis as long as the researcher does so responsibly [48]. For the reader's purposes, I recommend using generally established ethical guidelines for generative AI use in research [49, 50]:

being transparent in your use of generative AI during your research processes
being responsible for the inputs and how you use the outputs
protecting the data privacy of your participants
using AI for specific purposes in the research process and reporting the use as one would with any software [51]
considering and limiting potential bias from the AI platform
emphasis on the humanisation and centrality of participants in research

As a researcher, you need to understand that Chatbots are tools. The user still needs to have the required foundational knowledge to use the instrument for the intended purpose in the same way you need to know why you select statistical tests when using GUI software.

Designers of Chatbots add guardrails to prevent offensive, inflammatory or dangerous outputs in attempts to protect the users [52, 53]. However, jailbreaking still occurs; some individuals can find ways around the guardrails [54]. Jailbreaking is a complex issue and can be part of the innovative strategies of users. Guardrails have downsides, such as over-correction and replacing one type of bias with another – users need to be aware of this when employing Chatbots [55]. While the designers of generative AI try to protect users with guardrails, the researcher needs to understand settings to ensure safer use. To enhance the ethical use of ChatGPT and your data, you can go to privacy settings, choose “Settings” – “Data control” – “Chat History and Training”, then switch off the history and training [23]. In Fig. 1, I show the guidelines for the ethical use of AI and Chatbots in MMR and data analysis.

I advise that you refrain from uploading any sensitive data. Ensure you remove any identifying information from data sets before using generative AI to analyse the data. Use resources such as ChatGPT responsibly by rephrasing, repurposing, and reintegrating responses, just as you would information from other sources [56, 57]. Critically evaluate all the information you receive from AI platforms and apply your interpretation. Be transparent about how and when you used artificial intelligence in your writing, analysis, and presentation. Cite the Large-Language Model (LLM) you used. Chatbots are sophisticated but only sometimes reliable, explainable and never accountable for what they produce [58]. Therefore, the researcher is responsible for fair and ethical use.

Research questions

How can generative Artificial Intelligence assist with data analysis of Mixed Methods Research (MMR)?
To what extent does generative Artificial Intelligence yield reliable and valid findings for MMR data analysis?

The data used as a demonstration in the current article comes from a project that focused on enhancing first-year engineering education in South Africa. The data were collected as part of a concurrent triangulation research design, and the findings were reported in various publications (Author & Author, 2020; Author et al., 2022). We wanted to help first year engineering students by understanding student motivation, expectations, and perspectives and how these align with lecturer insights and curriculum design.

Instruments

The quantitative data were collected using the Academic Pathways of People Learning Engineering Survey (APPLES), which contains sections on learning expectations, beneficial and enjoyable educational experiences, skills and abilities required to become an engineer, motivation for studying engineering and post-graduate expectations [59, 60]. The APPLES survey was adapted to have forced-choice options, a decision for which evidence is presented in Author (2024). The qualitative data were collected using in-depth interviews for which the researchers designed a guide. The survey contained two open-ended questions where additional qualitative data were collected. In the open-ended questions, students were asked to explain why they chose engineering and provided any other information they deemed appropriate.

Sample

The quantitative data collection yielded a sample of 969 first-year engineering students, who were primarily men (71%), Caucasian (58%) or Black African (26%) and came from middle-class or affluent homes (81%). Students were, on average, 19 years old. The students who participated in the in-depth interviews (14) were all Black African by the study's design, predominantly male (9 out of 14 participants) and came from varied socio-economic backgrounds.

Data Analysis

In the original analysis, I used the Rasch dichotomous model via Winsteps© 5.4.0.0 [61] and SPSS (Statistical Package for the Social Sciences) to generate descriptive and inferential statistics [62]. The qualitative interview data were transcribed and then analysed by two research psychologists using reflexive thematic analysis (TA) (Author et al., 2022). After that, the specialist inspection was conducted by two engineering educators, and the final themes and write-up were the result of a consensus being reached among the team of four researchers [63–65].

For the current article and demonstration, I reran the analysis through ChatGPT’s Advanced Data Analysis bot, and my custom settings are shown Table 1. Custom ChatGPTs are available, and instructions can be customised [23]. I recommend that the user try out custom GPTs for themselves; the current tutorial utilises the Advanced Data Analyst, one of the original customs ChatGPTs [23]. Setting up custom instructions by clicking on your profile is recommended, and many good resources are available on the internet for custom instructions [66].

To use the custom instruction shown in Table 1, click on your profile icon in ChatGPT (bottom left-hand corner), choose the Custom instructions option, and paste in the text. The text provided is a generic guideline, and can be adjusted to suit specific individual preferences and should reflect the type of research you are conducting [67].

Table 1

Custom instructions for mixed methods analysis in ChatGPT
Question	Heading	Specification
What would you like ChatGPT to know about you to provide better responses?	Key Responsibilities	Integration of qualitative and quantitative data analysis
		Coding, thematic analysis, and statistical analysis
		Synthesis of findings from both data types to generate comprehensive insights
	Knowledge or expertise	Expertise in qualitative coding thematic analysis, and quantitative statistical methods
		Familiarity with integrating data from different sources to form cohesive conclusions.
		Proficiency in software tools for both qualitative (e.g., NVivo, ATLAS.ti) and quantitative analysis (e.g., SPSS, R)
	Current Projects	[Add your projects which reflect mixed methods]
	Jargon or Terminology	Qualitative coding, Reflexive Thematic Analysis, mixed methods integration, statistical analysis, multimodal data analysis, Rasch Analysis
	Goals and Objectives	To create rich, multidimensional interpretations from diverse data sets, consider context, and apply interdisciplinary knowledge to enrich interpretations.
	Goals and Objectives	Create meaningful interpretations from textual data, consider context, apply interdisciplinary knowledge to interpretations, limit bias
	Interactions	Engage in both social science and statistical data analysis
		Narrative and analytical writing for diverse audiences, including academic, policy, and public sectors
		Educational and practical application of mixed methods research findings
How would you like ChatGPT to respond?	Tone	Professional, empathetic, understanding, and clear
	Level of detail	Provide comprehensive narratives that integrate findings from both qualitative and quantitative data, supported by quotes and statistical evidence
	Preferred References	Scientific papers, psychological and educational frameworks, academic journals, primary sources across both qualitative and quantitative fields
	Examples or analogies	Use examples from journal articles, research reports, or thesis chapters that effectively demonstrate the integration of mixed methods
	Avoidance of Ambiguity	Present clear, analytical findings that seamlessly blend insights from both qualitative and quantitative research, grounded in theory and empirical evidence.
	Resources links	Access to interdisciplinary data science libraries, mixed methods research papers, and online courses focusing on integrated data analysis techniques
	Follow up questions	Research impact, future studies
	Problem-solving method	Apply a systematic approach to mixed methods data analysis, detailing the process of qualitative coding, thematic generation, and quantitative statistical analysis, including the integration phase, to draw overarching conclusions
	Confidence of answers	Include a confidence level with each piece of advice or information provided, especially regarding integrating data and interpreting mixed methods analysis results.

Ethical considerations

The present study was approved by the ethics committee of the Faculty of Engineering, Built Environment & IT (EBIT) at the University of Pretoria (EBIT/46/2020). The study was performed in accordance with the ethical guidelines of the University of Pretoria. All participants in this work have provided freely given, informed consent to participants in the study. The author affirms that the participants also provided informed consent for their data to be used for publication purposes.

Here, I present the roles that generative AI can play in the various types of data analysis. I give prompt examples, and I evaluate the reliability and validity of using ChatGPT for MMR. When asking ChatGPT to conduct analysis, I recommend that you give it frameworks, evaluation criteria or any other standards that it could use to improve outputs [68]. Ask ChatGPT to be brutally honest, to look for problems and lack of cohesion, to evaluate your writing critically, and to find ways to overcome positivity bias. ChatGPT can act as a research assistant who is available, teachable, and inexpensive. Meet your artificial assistant, ChatGPT, also referred to here as Chad (the only good Chad you will ever meet). The Chad is a naïve but well-meaning intern working for $20 monthly. Chad needs much supervision and makes silly mistakes but can add value when managed well. Chad is a well-mannered Chatbot (thanks to the guardrails) with much potential. Unfortunately, he is a people-pleaser, which can sometimes be annoying.

Qualitative data analysis with ChatGPT

ChatGPT and other similar generative AI platforms have immense capacity to analyse multimodal data [16]. ChatGPT can play several roles in qualitative analysis, including coding, where you ask ChatGPT to generate codes for textual data based on frameworks or theories. In addition to textual analysis, ChatGPT can detect sentiment, such as the emotional tone of the text [69]. ChatGPT can also generate themes and evaluate qualitative analysis for consistency by acting as a specialist inspector. The possibility of generating visualisations and tables with ChatGPT-4 is also helpful in quantifying or summarising aspects of the qualitative data. You can paste a limited amount of qualitative text into ChatGPT-3.5. In ChatGPT-4, the Advanced Data Analysis module makes it possible to upload data, ask for Excel codes per line, do more checks and comparisons, and create figures. Consider ChatGPT as an additional coder or a research assistant in this process. However, crucially, do your analysis and not rely solely on the Chatbot as it may miss rich aspects of your data or misinterpret the text.

In my experience, inductive coding does not work well in generative Artificial Intelligence settings, and I advise you to avoid asking the Chatbot to cold code. Instead, ChatGPT should be given both an analytical framework and a theoretical or conceptual framework to guide the coding and theme generation, as shown in Fig. 2 (see also the Prompt Library in the appendix). Examples of analytical frameworks to consider using include thematic, narrative, content, and discourse analysis. Another suggestion is to code the first few lines or paragraphs to show ChatGPT as examples.

ChatGPT assigns generic codes such as "other" when unsure or struggling to interpret the text. Such codes and themes require additional investigation as complex and valuable information is often submerged into these commonly named themes. When facing token restrictions, analyse a few interviews at a time; after that, ask ChatGPT to adjust themes based on additional interviews. As can be seen in Fig. 2, Chad provides well-written themes and quotes the participants as requested. The theoretical framework has also been woven into his answers.

Figure 3 demonstrates the prompt used to ask ChatGPT to code open-ended survey items. I uploaded the Excel file with the responses and labelled the columns where I wanted the bot to add the codes per line.

In addition to the downloadable Excel file Chad produced, reasonable themes were also written, as seen in Fig. 3. Checking the Excel file to see how every line was coded was very useful, and many vague statements had been given the code "other", and I could compare this to the human coding, which was more nuanced and consequential.

Quantitative data analysis with ChatGPT

Generative Artificial Intelligence (GAI) fulfils additional roles that GUI software did to a limited extent or not at all. ChatGPT can assist with cleaning data, for example, finding out-of-range values in data files. The Advanced Data Analysis custom ChatGPT can also scan your data file and recommend analysis. Here, I suggest sharing a codebook with the Chatbot with variables, labels, categories, and any required explanations, such as the Level of measurement. Use the suggestions from the Chatbot with care, as ChatGPT might suggest analysis that cannot or should not be done with the type of data presented. Chad can quickly generate descriptive statistics, as I will demonstrate here, as well as a paragraph with initial interpretation for the researcher to consider using. Commonly used inferential statistics can also be conducted in ChatGPT, with the bonus of interpretation. Currently, generative AI is fine for run-of-the-mill statistical analysis. Using generative AI to run more sophisticated statistical models requires further exploration. For example, I tried to conduct some structural equation modelling (SEM) with ChatGPT, but Chatbot has not yet been able to handle such complex analysis.

In Fig. 4, I show the prompt asking for Chad's recommended data analysis.

As can be seen in Fig. 4, Chad recommends many different types of analysis for my specific data set. However, not all of them are feasible or applicable. Nonetheless, this is a good starting point and ChatGPT may suggest types of analysis you had yet to consider.

Figure 5 is where I asked ChatGPT to produce descriptive statistics in a specified table format. I also asked for an interpretation and write-up of the table. While I did not fully agree with its interpretation, this again provides a good departure point to help the researcher start the writing process.

In Fig. 6, I illustrate the prompt for asking for inferential statistics from the Chad. Instead of asking for specific analysis, I asked ChatGPT to explore various relationships and to report statistical significance in an interpretative paragraph.

As can be seen from the response, the GAI accurately interprets the p-values, though it fails to report the effect sizes. I continued prompting beyond this point to obtain all the information I needed. Remember that just like a real assistant, you should keep speaking with the bot until you receive everything you need. See the appendix for the other prompts used.

Mixed methods data analysis with ChatGPT

A wide range of MMR analyses can be aided by generative AI, including concurrent triangulation, sequential explanatory and exploratory analysis, transformative designs (here, you must supply the paradigm) and multi-stage evaluations. ChatGPT-4 can also produce more complex figures and tables based on its own or your analysis, for example, side-by-side joint displays, statistics displayed by theme or vice versa and interview questions joint displays [70]. I have also created some infographics with ChatGPT-4, but these still require much input from the researcher, and it may be easier just to create your own in a different software environment. In ChatGPT-3.5, you can paste your findings from both the quantitative and qualitative analysis and work step by step with the bot to integrate analysis and output. To successfully use ChatGPT for mixed methods analysis, I recommend that you separately analyse the quantitative and qualitative data and combine the findings into a single document before asking the bot to assist with MMR analysis. In Fig. 7, I prompt ChatGPT to use my combined quantitative and qualitative findings document and conduct a sequential explanatory analysis using this study's theoretical framework. I specified both the analytical and the theoretical (or conceptual) frameworks as this yields the best results. Researchers should use comprehensive frameworks when conducting and analysing MMR data, as shown by Corrigan and Onwuegbuzie [71].

The output received in Fig. 7 was sparse, but further prompting into aspects reported by Chad led to more valuable outputs.

Based on the findings, I asked ChatGPT to create a joint table (see Fig. 8). While I would not use the table in its current format in a publication, I would use this summary to guide my writing and inform the creation of other graphics.

I recommend using generative AI to create initial tables, figures and text based on the quantitative and qualitative data. The final integration should be based on the aims of the study. When asking ChatGPT to create side by side type of tables, check this against your own version to make sure everything relevant is included. Write your interpretations based on your research aims and objectives; do not rely solely on the bot.

Reliability and validity of generative AI application

In Table 2, I compare the original analysis done by the research team and the outputs received from ChatGPT-4's Advanced Data Analysis Chatbot.

Table 2

Comparison of original quantitative and qualitative analysis with ChatGPT
Type	Original (Researchers)	ChatGPT version
Qualitative interviews	Theme: The academic program	Theme: Academic and Personal Growth
Qualitative interviews	Theme: Student support	Theme: Transition Challenges
Qualitative interviews	Theme: The role of peers in student success	Theme: Importance of Social Connections
Qualitative interviews	Theme: Behavioral and attitudinal factors
Quantitative data analysis	Demographic profile (Table & Text)	Demographic profile (Table & Text)
Quantitative data analysis	Recommended non-parametric analysis	Recommended non-parametric but also other not applicable analysis
Quantitative data analysis	Non-parametric analysis and interpretation	Non-parametric analysis done but less detail was provided
Open-ended survey questions	Theme: Intrinsic Psychological Motivation	Theme: Personal Interest and Passion
Open-ended survey questions	Theme: Engineering as Social Good
Open-ended survey questions	Theme: Focused on career opportunities	Theme: Career aspirations
Open-ended survey questions	Theme: Intrinsic behavioural motivation	Theme: Influence of Technology and Innovation
Open-ended survey questions		Theme: Curiosity and Problem-Solving

Comparing the original qualitative analysis done by the team of researchers to the ChatGPT themes, most were similar. The content of the themes is also consistent, but the most significant difference was related to the extent of coverage. ChatGPT found most of the same themes but gave shorter, less rich write-ups. ChatGPT also missed a theme we identified (behavioural and attitudinal factors).

In terms of the quantitative analysis, the Chatbot suggested creating demographic profiles (for example, gender, age, and ethnicity), mean distributions (for continuous variables) and summaries for the categorical variables. Other suggestions included cross-tabulations, non-parametric analysis, factor analysis (for constructs in the survey) and regression analysis to estimate predictors of the continuous variables. All the former suggestions I considered myself and are reasonable. Further suggestions, which I had yet to consider, included cluster analysis to group the students (this was not feasible with the data set), creating profiles within the engineering student population, and outcomes based on educational preferences. These last suggestions were exciting but irrelevant for my data set, once again pointing to the fact that the scholar needs to understand their data and the statistical possibilities. The demographic tables, figures, and interpretations provided by ChatGPT were of high quality and a good match for our own demographic profiles. When I asked ChatGPT to suggest inferential analysis, it did suggest the same non-parametric tests that I initially used, but it also suggested less useful and sometimes irrelevant analysis.

While descriptive statistics can quickly and reliably be done using ChatGPT, researchers should use their knowledge and judgment of statistics when deciding which tests to run in generative AI. The outputs are deemed acceptable and accurate and could be used in a research report or journal article, with the caveat that the text interpretation should be rewritten. Look for inaccuracies or exaggerations in the Chatbot’s output; for example, in my outputs, I found the claim that the sample is diverse to be an exaggeration.

The open-ended survey questions revealed most of the same themes, but again, ChatGPT's descriptions and unpacking of the themes were shallow when compared to human analysis. ChatGPT also missed a theme when compared to the human coders. At the same time, Chad had a theme that the humans did not identify. This comparison between human and artificial analysis can create avenues for identifying additional themes. ChatGPT contributes to analysis, but human judgement and insight should also be present throughout the process.

Troubleshooting

As is the case with all technology, ChatGPT can experience technical difficulties. Troubleshooting for ChatGPT includes clearing the history, cookies, and cache for the "all-time" option. Next, you can restart the session. Lastly, use different browsers (for example, Firefox works well) or disable all the Chrome extensions.

Generative AI is here to stay, and we must find ways to use it ethically and responsibly [3]. This is why the current study demonstrates how to conduct MMR analysis and integration using an artificial intelligence platform, ChatGPT. I also acknowledge that all types of technology have drawbacks, and a lack of transparency could call into question our research's methodological integrity, one of the biggest threats we face with GAI [72]. In this paper, I provide guidelines on using generative AI for mixed methods data analysis while balancing potential pitfalls. The main findings and recommendations from the current article are shown in Fig. 9.

In the quantitative section, I demonstrate how ChatGPT can quickly provide suggestions for data analysis techniques. Most of the suggestions are valid, and the bot has also improved since I used it last year, as it can now more accurately identify parametric from non-parametric statistical models. Providing the Chatbot with a codebook facilitates the data analysis, like setting up variables in software such as SPSS. Generative AI can provide accurate descriptive and inferential statistical outputs, but it’s ability to conduct more sophisticated analysis such as structural equation modelling (SEM) has yet to be tested.

In the qualitative section, the Chatbot should be provided with an analytical and a theoretical model to improve the analysis. ChatGPT does not handle inductive coding well, and I suggest providing the bot with examples of how you want it to code. You could also specify if you want it to act as a splitter or lumper. For more informative text, such as interviews, ChatGPT produces coherent narrative themes and can quote participants. It was less useful for shallow qualitative data, such as answers to open-ended questions. I suspect the latter is due to the inherent vagueness of the responses, which even human coders find challenging.

Mixed methods research (MMR) analysis works best if the quantitative and qualitative analyses are done separately before asking the bot to help integrate the findings. When the QUAL and QUANT are neatly combined in a document with clear headings, a prompt can be provided to the GAI wherein one asks it to help synthesise the findings. Here, I recommend specifying which MMR analytical approach you want it to use. The side-by-side tables it produces are also helpful, but as always, the researcher must add their own insights.

The reliability and validity of straightforward tasks, such as producing descriptive statistics, are perfect in the generative AI environment. More interpretative outputs, such as narratives derived from qualitative data, are less reliable and valid when using ChatGPT. While it can detect most of the same themes, it sometimes misses them. The outputs provided by the GAI are also less rich and tend to be shallower summaries. Therefore, the researcher can consider the GAI an assistant in coding but should still create their own themes and write-ups to compare to bot outputs. Overall, the reliability and validity of using generative AI in analysing MMR data is acceptable. The researcher needs to have the required foundational knowledge and be discerning, rephrasing, repurposing, and reintegrating the outputs before final publication. One of the main advantages of using generative AI is the quick turnaround time for data analysis. Speedier analysis is most definitely desirable in a world where rapid publication is required in higher education [34].

Lastly, we may ask ourselves why we would create new knowledge in a world where artificial intelligence can produce equivalent artefacts. Yet there have always been others doing something similar, and who could do it better than we could. Excellent poets, writers, scholars, artists, and every type of creator have existed for millennia. We continue to create despite others doing the same work because everyone has a unique point of view. Different from another person or Chatbot. We create new knowledge as scholars because we need to contextualise what we do for our settings. We add value through our humanity, and others crave that human connection. Human production of creative work will become even more desirable in the future when machines make so much. Creativity is a natural part of who we are – we are our most authentic selves when we create. Authentic self-expression through knowledge creation can fulfil an identity need to feel connected to ourselves and our world through adding new insights and intellectual artefacts.

Significance of the findings

The current paper demonstrates innovative ways to use GAI for quantitative, qualitative, and mixed methods data analysis. I also showcase new ways to integrate findings with the assistance of generative AI to improve the interpretation of mixed methods data. The reader is provided with guidelines on using GAI for efficiency and automation to make handling large quantities of data easier. Generative AI misses themes sometimes but can also reveal themes and patterns humans overlook. Therefore, I state the case for combining human insight with generative artificial intelligence to obtain results beyond the abilities of either; the sum of the parts is indeed greater. My study can be used across many disciplines or for cross-disciplinary work where mixed methods are prominent. I have detailed my steps and provided a prompt library for easy replicability and application. The current paper serves as capacity building for researchers globally to advance their use and understanding of AI tools in research methodology. I explicitly addressed ethical considerations in this paper to clarify privacy, consent, and practical implications for researchers who want to use generative AI. My findings here are meant to strengthen digital literacy in a fast-developing technological landscape.

Limitations of the study

The tutorial and application were limited to an available data set, and the example was drawn from higher education. I only used one platform, ChatGPT, and I used the paid version. Using AI in data analysis has its limitations, as discussed in the paper. The quality of the results depends significantly on the knowledge and skills of the researcher. The platform user should have the requisite knowledge of mixed methods data analysis, synthesis, and application. Chatbots cannot replace human insight.

Furthermore, the fact that bias and unfairness are built into artificial generative intelligence is well known [73]. Therefore, the researcher must actively manage the inputs and how the outputs are used, aware that there may be systemic, computational and human biases embedded in chats [74]. The landscape of generative AI is also changing rapidly [75], and the current paper is limited to the time and space in which it was written. ChatGPT is not the only platform available to researchers, and future researchers could look at platforms such as Perplexity (see https://www.perplexity.ai/), Jenni (see https://jenni.ai/) and Consensus (https://consensus.app/).

Funding statement

No funding was received for the current work.

Conflicts of interest

There is no conflict of interest.

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Code availability

Not applicable.

Consent statement

All participants in this work have provided freely-given informed consent to participants in the study. The author affirms that the participants also provided informed consent for their data to be used for publication purposes.

Author Contribution

I am the only author of this paper; I confirm the single authorship.

Acknowledgement

The Living Lab for Innovative Teaching at the University of Pretoria (LLITUP) supported the development of a workshop to test the ideas presented in this paper. I am very grateful to them for their time, inputs and congeniality.

Howard, J., Artificial intelligence: Implications for the future of work. American Journal of Industrial Medicine, 2019. 62(11): p. 917-926.
Dwivedi, Y.K., et al., Artificial Intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. International Journal of Information Management, 2021. 57: p. 101994.
Ågerfalk, P.J., Artificial intelligence as digital agency. European Journal of Information Systems, 2020. 29(1): p. 1-8.
Ooi, K.-B., et al., The potential of generative artificial intelligence across disciplines: Perspectives and future directions. Journal of Computer Information Systems, 2023: p. 1-32.
Dewey, J., M.C. Pautz, and M.K. Diede, How do we Address Faculty Burnout? Start by Exploring Faculty Motivation. Innovative Higher Education, 2023.
Strzelecki, A., Students’ Acceptance of ChatGPT in Higher Education: An Extended Unified Theory of Acceptance and Use of Technology. Innovative Higher Education, 2023.
Bahroun, Z., et al., Transforming education: A comprehensive review of generative artificial intelligence in educational settings through bibliometric and content analysis. Sustainability, 2023. 15(17): p. 12983.
Wikipedia. ChatGPT. 2024 [cited 2024, March 4; Available from: https://en.wikipedia.org/wiki/ChatGPT.
Rawas, S., ChatGPT: Empowering lifelong learning in the digital age of higher education. Education and Information Technologies, 2023.
Ruiz-Rojas, L.I., et al., Empowering education with generative artificial intelligence tools: Approach with an instructional design matrix. Sustainability, 2023. 15(15): p. 11524.
Ratten, V. and P. Jones, Generative artificial intelligence (ChatGPT): Implications for management educators. The International Journal of Management Education, 2023. 21(3): p. 100857.
Hmoud, M., et al., Higher Education Students’ Task Motivation in the Generative Artificial Intelligence Context: The Case of ChatGPT. Information, 2024. 15(1): p. 33.
Koh, E. and S. Doroudi, Learning, teaching, and assessment with generative artificial intelligence: towards a plateau of productivity. Learning: Research and Practice, 2023. 9(2): p. 109-116.
Dempere, J., et al., The impact of ChatGPT on higher education. Dempere J, Modugu K, Hesham A and Ramasamy LK (2023) The impact of ChatGPT on higher education. Front. Educ, 2023. 8: p. 1206936.
Islam, I. and M.N. Islam, Exploring the opportunities and challenges of ChatGPT in academia. Discover Education, 2024. 3(31).
Davidson, T., Start generating: Harnessing generative artificial intelligence for sociological research. Retrieved from osf. io/preprints/socarxiv/u9nft, 2023.
Pack, A. and J. Maloney, Using Generative Artificial Intelligence for Language Education Research: Insights from Using <scp>OpenAI</scp>'s <scp>ChatGPT</scp>. TESOL Quarterly, 2023. 57(4): p. 1571-1582.
Goodfellow, I., et al., Generative adversarial networks. Communications of the ACM, 2020. 63(11): p. 139-144.
Adamopoulou, E. and L. Moussiades. An overview of chatbot technology. in IFIP international conference on artificial intelligence applications and innovations. 2020. Springer.
Hadi, M.U., et al., A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Korteling, J., et al., Human-versus artificial intelligence. Frontiers in artificial intelligence, 2021. 4: p. 622364.
Cooms, S. and V. Saunders, Poetic inquiry: a tool for decolonising qualitative research. Qualitative Research Journal, 2024. 24(1): p. 3-22.
OpenAI. ChatGPT (September 25 Version) [Large language model]. 2024; Available from: https://chat.openai.com.
Barros, A., A. Prasad, and M. Śliwa, Generative artificial intelligence and academia: Implication for research, teaching and service. Management Learning, 2023. 54(5): p. 597-604.
Costello, G.R., K.R. Davis, and O.S. Crocco, Learning by Doing: Student & Faculty Reflections on a Collaborative Model for Conducting and Publishing Mixed Methods Research in a Graduate Course. Innovative Higher Education, 2022. 47(6): p. 1067-1084.
Malina, M.A., H.S. Nørreklit, and F.H. Selto, Lessons learned: advantages and disadvantages of mixed method research. Qualitative Research in Accounting & Management, 2011. 8(1): p. 59-71.
Onwuegbuzie, A.J. and C. Poth, Special Issue: Mixed Methods. International Journal of Qualitative Methods, 2015. 14(2): p. 122-125.
Coolidge, F.L., Statistics: A gentle introduction. 2020: Sage Publications.
Chambers, J.M., Statistical Computing: History and Trends. The American Statistician, 1980. 283.
Abatan, S.M. and M. Olayemi, The Role of Statistical Software in Data Analysis International Journal of Applied Research and Studies (iJARS) 2014. 3(8).
Klinke, S. Statistical user interfaces. Center for Applied Statistics and Economics (CASE) REport, 2004. 35.
Martinez, W.L., Graphical user interfaces. Wiley Interdisciplinary Reviews: Computational Statistics, 2011. 3(2): p. 119-133.
Field, A.P., Discovering statistics using IBM SPSS statistics. 2024: Sage publications limited.
Vindrola-Padros, C. and G.A. Johnson, Rapid Techniques in Qualitative Research: A Critical Review of the Literature. Qualitative Health Research, 2020. 30(10): p. 1596-1604.
Kantor, J., Best practices for implementing ChatGPT, large language models, and artificial intelligence in qualitative and survey-based research. JAAD international, 2024. 14: p. 22-23.
Cave, S. and K. Dihal, Hopes and fears for intelligent machines in fiction and reality. Nature machine intelligence, 2019. 1(2): p. 74-78.
McClure, P.K., “You’re fired,” says the robot: The rise of automation in the workplace, technophobes, and fears of unemployment. Social Science Computer Review, 2018. 36(2): p. 139-156.
Hinks, T., Artificial Intelligence Perceptions and Life Satisfaction. Journal of Happiness Studies, 2024. 25(1): p. 1-14.
Smolansky, A., et al. Educator and student perspectives on the impact of generative AI on assessments in higher education. in Proceedings of the tenth ACM conference on Learning@ Scale. 2023.
Council on Higher Education, C. A Closer Look at Plagiarism within the Context of Student Academic Writing in an Era of Generative Artificial Intelligence. Briefly Speaking, 2024. 27.
Atkinson, R., Don't fear AI (Volume 2). 2019: European Investment Bank.
Baker, M., The Western Mathematic and the Ontological Turn: Ethnomathematics and Cosmotechnics for the Pluriverse, in Indigenous Knowledge and Ethnomathematics. 2023, Springer. p. 243-276.
Hui, Y., For a cosmotechnical event: In honor of Don Ihde and Bernard Stiegler. Reimagining Philosophy and Technology, Reinventing Ihde, 2020: p. 87-102.
Ihde, D., Technology and the lifeworld: From garden to earth. 1990.
Fisher, W.P., Meaning and Method in the Social Sciences1. Human Studies, 2004. 27(4): p. 429-454.
Roth, P.A., Meaning and method in the social sciences: A case for methodological pluralism. 2019: Cornell University Press.
Fisher Jr, W.P. and S.J. Cano, Ideas and Methods in Person-Centered Outcome Metrology. Person-Centered Outcome Metrology, 2023: p. 1.
Schlagwein, D. and L. Willcocks, ‘ChatGPT et al.’: The ethics of using (generative) artificial intelligence in research and science. Journal of Information Technology, 2023. 38(3): p. 232-238.
Rothenberger, L., B. Fabian, and E. Arunov. Relevance of Ethical Guidelines for Artificial Intelligence-a Survey and Evaluation. in ECIS. 2019.
Skorburg, J.A., K. O'Doherty, and P. Friesen, Persons or data points? Ethics, artificial intelligence, and the participatory turn in mental health research. American Psychologist, 2024. 79(1): p. 137.
O’Kane, P., A. Smith, and M.P. Lerman, Building transparency and trustworthiness in inductive research through computer-aided qualitative data analysis software. Organizational Research Methods, 2021. 24(1): p. 104-139.
Wang, Y. and L. Singh, Adding guardrails to advanced chatbots. arXiv preprint arXiv:2306.07500, 2023.
Abdelkader, H., et al., ML-On-Rails: Safeguarding Machine Learning Models in Software Systems A Case Study. arXiv preprint arXiv:2401.06513, 2024.
Boxleitner, A., Pushing Boundaries or Crossing Lines? The Complex Ethics of ChatGPT Jailbreaking. The Complex Ethics of ChatGPT Jailbreaking (October 17, 2023), 2023.
Kamalov, F., D. Santandreu Calonge, and I. Gurrib, New era of artificial intelligence in education: Towards a sustainable multifaceted revolution. Sustainability, 2023. 15(16): p. 12451.
Cheng, M.W.T. and I.H.Y. Yim, Examining the use of ChatGPT in public universities in Hong Kong: a case study of restricted access areas. Discover Education, 2024. 3(1).
Daniela, F.-T. Academic writing and ChatGPT: Students transitioning into college in the shadow of the COVID-19 pandemic. Discover Education, 2024. 3, 1-10 DOI: 10.1007/s44217-023-00076-5.
Mari, L., Chatbots: facing a cultural revolution and trying to understand it (a non-technical perspective), B. Seminar, Editor. 2023: UC Berkeley.
Donaldson, K.M., et al., Scaling up: Taking the Academic Pathways of People Learning Engineering Survey (APPLES) national, in 2008 IEEE Frontiers in Education Conference. 2008, IEEE Computer Society. p. F4H-6-F4H-11.
Sheppard, S.D., et al., Exploring the Engineering Student Experience: Findings from the Academic Pathways of People Learning Engineering Survey (APPLES). 2010, Center for the Advancement of Engineering Education Technical Report CAEE-TR-10-01.
Linacre, J.M., Winsteps® (Version 5.4.0.0) 2023: Portland, Oregon: Winsteps.com.
IBM, IBM SPSS Statistics for Windows (Version 28.0). 2023, IBM Corp.
Braun and Clarke, One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 2021. 18(3): p. 328-352.
Braun, V. and V. Clarke, Thematic analysis : a practical guide. 2022, London ;: SAGE.
Braun, V. and V. Clarke, Using thematic analysis in psychology. Qualitative Research in Psychology, 2006. 3(2): p. 77-101.
Surach, D., AI foundations: Learn to leverage market moving AI tools. 2024, YouTube: YouTube.
Surach, D., 5 ChatGPT Hacks | Take ChatGPT to the NEXT Level!, in AI Foundations. 2023: YouTube.
Stapleton, A., How To Write An A+ Essay Using AI in 3 Simple Steps. 2023: YouTube.
Owoahene Acheampong, K. and M. Nyaaba, Review of Qualitative Research in the Era of Generative Artificial Intelligence. Matthew, Review of Qualitative Research in the Era of Generative Artificial Intelligence (January 7, 2024), 2024.
Guetterman, T.C., M.D. Fetters, and J.W. Creswell, Integrating Quantitative and Qualitative Results in Health Science Mixed Methods Research Through Joint Displays. Ann Fam Med, 2015. 13(6): p. 554-61.
Corrigan, J.A. and A.J. Onwuegbuzie, Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. 2020.
Grimes, M., et al., From scarcity to abundance: Scholars and scholarship in an age of generative artificial intelligence. Academy of Management Journal, 2023. 66(6): p. 1617-1624.
Schwartz, R., et al., Towards a standard for identifying and managing bias in artificial intelligence. NIST special publication, 2022. 1270(10.6028).
Hwang, G.-J., et al., Vision, challenges, roles and research issues of Artificial Intelligence in Education. 2020, Elsevier. p. 100001.
Nishant, R., M. Kennedy, and J. Corbett, Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda. International Journal of Information Management, 2020. 53.

Appendix A –Prompt Library

Prompts for Qualitative Analysis

Square brackets indicate aspects you should modify to suit your need.

Prompt 1 – You are a qualitative researcher. Use [Reflexive thematic analysis], according to Braun and Clark, to analyse the interview and generate themes. Use the [Self-Determination Theory] to interpret the data. The themes should be written as narrative findings in paragraphs, give each theme a name that reflects the main finding of that theme. Use quotes from the interviewee to support your themes. Only analyse the interviewee responses.

Prompt 2 – Attached are open-ended responses. Code and classify the responses according to [theory] so that you create themes, write up the themes as narratives with quotes from respondents to support your argument. Cite the frequencies of how many times a theme was mentioned. Make the excel sheet available with the codes completed.

Prompts for Quantitative Analysis

Prompt 3 – I have cross-sectional survey data, please recommend analyses I could do with my data set. The variables include [demographic questions, such as gender, age, and race]. The constructs measured in the questionnaire include [motivation to attend university and long-term career goals]. [Codebook pasted with prompt].

Prompt 4 – Summarise the constructs [add list here] in the attached data set to identify trends. Create a table and write a paragraph to illustrate your findings.

Prompt 5 – Please create demographic profiles for my data. Create a table in [APA 7] style as the output with a paragraph which briefly describes the student profile.

Prompt 6 – Redo the analysis, consider that most of the demographic variables are categorial in nature. Add the categories to the table to make it easier for the reader to understand.

Prompt 7 - Based on the previous data, run inferential statistical analysis and identify any items that are significant. Include p-values and effect sizes. Show the results in tables with a summary of your interpretation.

Prompts for Mixed Method Analysis

Prompt 8 – I have findings from [a questionnaire], and I also conducted [in-depth interviews]. The results from the quantitative and qualitative analyses are attached. Please conduct a [sequential explanatory analysis] and help me to identify patterns, achieve triangulation and integrate the results. Use the [Incentive Theory] to interpret the results. Write two pages based on your sequential explanatory analysis.

Prompt 9 – Based on the findings from the previous prompt, create a statistics-by-themes joint table display of the findings. Make the table a downloadable Excel.

OR

Create a side-by-side joint display of the quantitative and qualitative findings. Make the display downloadable.

Prompt 10 – Produce a two-page document of your findings and ensure your report clearly distinguishes between quantitative and qualitative findings and how they integrate. Based on your findings, suggest practical or theoretical implications. Reflect on the effectiveness of the mixed-methods approach for the data collected.

Prompts for obtaining more from Chatbots

Prompt 11 – Please give me more information about your analysis and findings

Prompt 12 - You are the student's supervisor. Evaluate the problem statement according to the guidelines. Identify the short-comings and list them. Give the student valuable tips to improve the problem statement. Problem statement guidelines: a) What do we already know about the problem? (Use recent and relevant studies to substantiate, preferably studies no older than 10 years) b) What do we need to know about the problem? (Gap in literature/scholarly field) c) Why does it matter? (importance of the study) d) Conclude with how your study will address the problem.

Prompt 13 - I want you to evaluate the writing. Act as a critic; be ruthless. Analyse the text and tell me where it can be better.

Prompt 14 - Please write a review of the attached document and highlight both the strengths and the weaknesses of the submission. Please be as constructive and specific as possible when offering recommendations. Use the following guidelines when examining the document [insert journal guidelines or evaluation criteria here]

No competing interests reported.