GPT-4 versus human authors in clinically complex MCQ creation: a blinded analysis of item quality

doi:10.21203/rs.3.rs-4831476/v1

Download PDF

Research Article

GPT-4 versus human authors in clinically complex MCQ creation: a blinded analysis of item quality

https://doi.org/10.21203/rs.3.rs-4831476/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

MCQs are a popular assessment format in medical education. Creating clinically complex MCQs can be a time-consuming task for subject matter experts. Large language models such as GPT-4, a type of generative artificial intelligence (AI), are a potential tool for MCQ design.

Clinically complex human-generated MCQs, at both novice and expert level, were compared with AI MCQs. A generic prompt for GPT-4 was engineered, which included item-writing guidance, example MCQs, and key learning points. A standardised scoring system was developed for a consensus panel to objectively evaluate each item, blinded to the author, on categories including content validity, scope, item anatomy, cognitive skill level, item-writing flaws (IWFs), feedback comprehensiveness, veracity, adequacy of clinical reasoning, and global impression of fitness for use.

Analysis showed that all groups (novice, expert, and AI) were able generate items within scope. Expert items performed better than Novice items in all categories. Expert items performed better than AI in content validity, feedback veracity and clinical reasoning. They also tended to test higher order cognitive skills. There was no difference in the global impressions of Expert and AI items, which suggests they may be comparable overall.

With adequate prompt engineering, GPT-4 can produce MCQs testing clinically complex concepts for medical assessment. The quality of AI outputs is comparable to experts, however human validation is necessary to ensure content validity. The AI-generated explanatory feedback was adequate in veracity and clinical reasoning, which may serve as an educational tool for learners.

Multiple choice question

higher order cognitive skills

Bloom’s taxonomy

artificial intelligence

prompt engineering

ChatGPT

The multiple choice question (MCQ) is ubiquitous in medical education, able to efficiently test a broad range of content and, if well-constructed, can assess higher-order cognitive skills at least as effectively as open-ended formats (Schuwirth & Van Der Vleuten, 2003; Palmer et al., 2010; Hift, 2014). Constructing high-quality, context-rich MCQs typically requires a significant investment of time and expertise from subject matter experts, posing a perpetual challenge for institutions to develop and maintain a comprehensive bank of up-to-date exam questions. Content shortages often force faculties to reuse items verbatim in subsequent assessments, which can influence students’ learning strategy toward rote memorisation over conceptual understanding (Papinczak et al., 2012; Jobs et al., 2013). The use of lower-quality items with item-writing flaws compromise examination validity (Haladyna & Downing, 1989; Pham et al., 2018) and may disproportionately disadvantage higher-performing students (Tarrant & Ware, 2008).

Even with the input of subject matter experts, maintaining consistently high quality, fidelity, novelty, and volume of assessment items is problematic for faculties. Standard workflows for generating assessment material are inherently resource-intensive and widely variable in output quality, and evidence-based interventions to enhance item construction - such as targeted faculty training (Naeem et al., 2012; Gupta et al., 2020) or peer review (Abozaid et al., 2017; Smeby et al., 2019) - only compound the resource requirement. These limitations have prompted exploration of alternative avenues for item generation at scale, including leveraging student authors (Shah et al., 2019; Pham et al., 2023) and, more recently, artificial intelligence.

A Large Language Model (LLM) is a form of artificial intelligence (AI) comprising a neural network trained via machine learning against a vast corpus of text to perform Natural Language Processing tasks including content generation, summarisation, translation, classification, dynamic dialogue, and others. This versatile technology has been met with cautious optimism by educators eager to explore its potential to revolutionise learning, teaching, and assessment but cognisant of its limitations and pitfalls (Abd-Alrazaq et al., 2023; Benítez et al., 2024). For example, the tendency of LLMs to “hallucinate” - that is, to invent inaccurate or unfounded information and silently integrate it into its output - is unacceptable in the safety-critical field of healthcare (Giuffrè et al., 2024). The availability of publicly accessible interfaces, most notably OpenAI’s ChatGPT (Chat Generative Pre-trained Transformer), has precipitated an explosion of interest in the capabilities and operationalisation of these systems; among such uses, LLM-aided generation of MCQ items has emerged as an appealing avenue to address the challenges of assessment content development for medical faculties. However, the formal evidence base remains nascent and the ideal process for reliably and efficiently producing high-quality medical content with generative AI remains unknown.

Although early research is accumulating regarding the basic feasibility of using ChatGPT to generate medical MCQs, methodologies and study quality are highly heterogeneous and reports of resulting MCQ item quality are mixed (Artsi et al., 2024). Structural characteristics of AI-generated items, encompassing cognitive skill level, flaws, and item anatomy, have been only sparsely evaluated. Klang et al. (2023) used ChatGPT-4 to generate a 210-item MCQ examination across internal medicine, surgery, obstetrics/gynaecology, psychiatry, and paediatrics, which was reviewed by specialist clinicians in each field blinded to the AI-generated nature of the items. Of the generated items, 15% were judged to require revision for structural flaws, ranging from 30% of the surgical items to 0% of the psychiatry items, and one item (0.5%) was deemed completely invalid. This appears to be an improvement from the previous version of ChatGPT. Using ChatGPT3.5, 60% of 40 dermatology items generated were judged unsuitable for examination use by two non-blinded board-certified dermatologists against criteria of accuracy, complexity, and clarity (Ayub et al., 2023), and 68% of 60 immunology items were either incorrect, significantly misleading, or only appropriate for use with substantial modification (Ngo et al., 2024).

Evaluations of the psychometric performance of AI-generated items are even more scant: work at Gazi University in Turkey has included a study in which two ChatGPT-generated hypertension MCQs demonstrated acceptable discrimination (point-biserial correlations of 0.41 and 0.39) in a medical student examination (Kıyak et al., 2024), and another in which 15 general medical MCQs were generated, of which six (40%) demonstrated acceptable discrimination (point-biserial correlation >0.30) and ten (66%) were judged unsuitable for use (Coşkun et al., 2024). An examination of the appended items in Coskun et al. (2024) and Ngo et al. (2024) show items generated at low clinical complexity levels, testing knowledge and recall. There is a paucity of data on the ability of AI to generate high complexity clinical material emulating the advanced clinical reasoning expected of medical graduates.

Comparative data evaluating AI-generated materials versus those authored by humans is also minimal. Cheung et al. (2023) performed a blinded expert analysis with fifty each of ChatGPT-generated versus human-generated MCQs, finding comparable average ratings of question appropriateness, clarity, relevance, discriminative power, and suitability for graduate-level examination. However, when compared head-to-head, human-generated items outperformed AI items on each scale in the majority of instances (60% of the time for cumulative rating score), with AI items showing wider variability on each metric.

Among the factors that may explain this wide variability in output quality, besides the rapidly advancing technology that underpins the LLM functionality, is a lack of standardisation in input. The output quality of an LLM is critically dependent upon the ‘user prompt’, the specific instruction provided to direct its behaviour and generate desired outputs (Giray, 2023; Heston & Khun, 2023). ‘Prompt engineering’ refers to the iterative process by which users of LLMs critically evaluate and craft their prompts to optimise responses. This process may involve serially refining prompts based on outputs; adding contextualising information such as target audience, key learning points, desired level of complexity, etc; and providing example content for the model to emulate. Specific reference to action verbs from Bloom’s taxonomy has also been recommended (Jacobsen & Weber, 2023). It is perhaps partly explanatory to note that the higher-quality outputs reported in the literature were generated using prompts that included a full example examination (Klang et al., 2023) or relevant chapters from two authoritative textbooks (Cheung et al., 2023), although lower-quality outputs were also derived from credible source material such as Continuing Medical Education articles from a top-tier specialty journal (Ayub et al., 2023). A thorough understanding of prompt engineering may help educators and curriculum designers produce quality relevant material, and as such it is likely a necessary skill for effectively leveraging this technology (Heston, 2023; Meskó, 2023).

From this survey of the available evidence, it is clear there is much to learn about the inherent capabilities, influencing factors, and optimisation processes of generative AI in producing medical MCQ materials. Nonetheless, the potential for AI to reduce the resource burden of item generation for medical faculties holds significant appeal, if its operationalisation can be optimised and its output validated.

Aims

Evaluate and compare the structural qualities of clinically complex MCQs authored by humans (both novice and expert constructors) versus Generative Pre-trained Transformer 4 (GPT-4)-generated items; and
Evaluate the comprehensiveness and articulated clinical reasoning of explanatory feedback on items generated by humans (both novice and expert) and GPT-4.

Research questions:

What is the structural quality of clinically complex MCQs generated by GPT-4?
Can GPT-4 articulate accurate explanatory feedback for clinically complex MCQs?
What points in the workflow of AI-assisted MCQ generation still necessitate human input?

This mixed-methods study was performed to critically evaluate the structural quality of AI-generated MCQs in comparison to human-generated items. AI items were prospectively generated, while human-authored items were retrospectively sourced from an existing content bank, as further detailed below.

Item structure and test blueprint construction

A single-best-answer MCQ format was employed. A complete item included a detailed contextual stem, a question, five options (with correct answer indicated), and explanatory text articulating the logic of correct versus incorrect options. As explained below, the explanatory text was evaluated separately from the other elements due to it being not universally included in standard MCQs.

A test blueprint was constructed to emulate a standard medical school examination at the level of the graduating student. Content areas included Medicine, Surgery, Paediatrics, Obstetrics, Gynaecology, Psychiatry, Population Health, and General Practice. 125 items were included, comprised of 40 from three sources, Novices, Experts, and AI. A surplus of 5 Expert human-generated items were included in the scoring process, as they were intended for future use in a mock examination where only satisfactory items would be included, and an element of redundancy was required for that purpose. An excerpt of the test blueprint is given in Appendix 1.

Development of a standardised scoring system

A standardised scoring rubric was developed to facilitate consistent evaluation of human and AI-generated MCQs. This incorporated elements such as content validity (encompassing factual accuracy, fidelity, and realism), scope, correct item anatomy, specific item-writing flaws, and cognitive skill level. This rubric drew on established frameworks, including modified Bloom’s taxonomy and item-writing guidelines (Haladyna et al., 2002), and is presented as Tables 1-3. A global impression criterion was included as a proxy for whether the item was considered fit for use in a summative examination for graduating medical students. A separate secondary evaluation was undertaken to assess the quality of each item’s explanatory feedback text for comprehensiveness, veracity, and articulation of clinical reasoning.

Table 1. Standardised scoring rubric of all MCQs

CORE ITEM ELEMENT	Score key
Content validity: The item has content validity, being factually accurate and realistic to clinical practice	Entirely does not meet criteria Mostly does not meet criteria Mostly meets criteria Entirely meets criteria
Within scope: The item tests concepts that are within scope for the target audience of a graduating medical student	Entirely does not meet criteria Mostly does not meet criteria Mostly meets criteria Entirely meets criteria
Item anatomy: The anatomy of the item is correct and complete	Entirely does not meet criteria Mostly does not meet criteria Mostly meets criteria Entirely meets criteria
Item-writing flaws (IWF): How many item-writing flaws are present? What type of flaws are present?* ● Content ● Style ● Formatting ● Stem ● Options	● Numeric count ● Type of IWF also documented
Cognitive skill level: What is the cognitive skill level of the item?	Using a modified Bloom’s taxonomy: Level I: Remembering Level II: Understanding Level III: Applying, analyzing, evaluating, and creating
Global impression (structural): Global impression of the stem, question, and options: This item is fit for use in a summative examination for graduating med student	No (unsalvageable) No (major further editing) Yes (minor further editing) Yes (no further editing)
SCORING OF EXPLANATORY TEXT	Score key
Feedback comprehensiveness: The feedback was appropriately comprehensive, addressing the correct option and distractors	Entirely does not meet criteria Mostly does not meet criteria Mostly meets criteria Entirely meets criteria
Feedback veracity and clinical reasoning: The science and clinical reasoning in the feedback was satisfactory	1. Entirely does not meet criteria 2. Mostly does not meet criteria 3. Mostly meets criteria 4. Entirely meets criteria
Global impression (overall): This item, including its written feedback, is fit for use in a summative examination for graduating med student	No (unsalvageable) No (major further editing) Yes (minor further editing) Yes (no further editing)

* Referencing item-writing guidelines as laid out by (Haladyna et al., 2002).

Table 2. Modified Bloom’s taxonomy

Level	Cognitive domains
Level I	Remember (identifying and retrieving information)
Level II	Understand (interpreting and summarizing information)
Level III	Apply, analyze, evaluate, and create (implementing, organizing, and critiquing information)

Table 3. Examples of item-writing guidelines (adapted from Haladyna et al., 2002).

● Content concerns: Use novel material to test higher level learning. Paraphrase textbook language or language used during instruction when used in a test item to avoid testing for recall.

● Formatting concerns: Format the item vertically instead of horizontally

● Style concerns: Use correct grammar, punctuation, capitalisation, and spelling

● Writing the stem: Include the central idea in the stem instead of the choices; word the stem positively, avoid negatives such as NOT or EXCEPT.

● Writing the choices: Place choices in a logical or numerical order; keep the length of choices about equal; avoid All-of-the-above; avoid giving clues to the right answer, such as pairs or triplets of options that clue the test-taker to the correct choice; make all distractors plausible.

Human-generated MCQs – Novice and Expert

A total of 85 human-generated MCQs were sourced from an existing Australian commercial medical education provider (eMedici2 Pty Ltd, Adelaide, Australia; https://emedici.com). This content bank is derived from submissions by medical students and junior doctors, which pass through a pipeline of peer review, expert clinician review, and editorial approval prior to acceptance. Human authors are provided detailed written item-writing guidelines referencing style and item anatomy, at the time of item submission. Only items tagged as testing higher-order cognitive skills were included in this study. Each item was otherwise randomly selected from the content bank based on its recorded topic by two authors who were otherwise blinded to the content of the item. Of the 85 human items, 40 were written by a non-expert and had not passed through a peer review or other editorial process, and as such were deemed at ‘Novice’ level of authorship, while 45 had been edited and/or approved by subject matter experts and were thus deemed ‘Expert’ level. Subject areas were matched between groups.

AI-generated MCQs

GPT-4 was used in this study (model number: gpt-4-0125-preview) based on favourable reported performance against the Massive Multi-task Language Understanding benchmarks (OpenAI, 2023). A programmed script was used in GPT-4 with the prompt and key learning points - created as below - to generate outputs in an unsupervised fashion such that all authors were blinded to the GPT-4 outputs.

Prompt engineering

Construction of a tailored prompt for GPT-4 took place across three reference group meetings by the six-panel team (the authors) who have broad educational, item-writing, clinical, and technical expertise. The aim was to develop a generic prompt template that maximised the potential of GPT-4 to produce structurally sound items testing higher order cognitive skills and could be easily adapted for a wide range of learning points or item topics with minimum subsequent human effort. The prompt was engineered incrementally, with each output assessed subjectively until the quality was deemed to be at the ceiling point. The prompt template included:

Information on the setting and the target audience of the MCQ;
The inclusions and exclusions in the clinical stem to meet basic item anatomy requirements;
Advice on avoidance of specific item-writing flaws, with instructions sourced from a full taxonomy of item-writing guidelines by Haladyna et al. (2002), as outlined in Table 3;
Instruction on the number of question options and distractors;
Instruction to produce explanatory feedback including clinical reasoning for the answer and distractors of the MCQ;
Instruction to include references to recent peer-reviewed articles;
Five examples of peer-reviewed, high quality MCQs covering a range of medical topics; and
A key learning point of the intended MCQ in the form of a factual statement, which included the question topic (in accordance with the test blueprint – Appendix 1).

Variability of GPT-4 outputs

Among the input variables of the GPT-4 interface is ‘temperature’, which broadly determines the level of variety in subsequently generated text. This parameter ranges from 0 to 2, with a lower value resulting in more consistent outputs. Preliminary investigations have yet to identify the ideal temperature for medical MCQ generation (Agarwal et al., 2024), and it is likely to vary in different settings. To maximise reproducibility, we used a temperature of 0.0.

To confirm the predictability of outputs at temperature 0.0, six learning points were used to generate three consecutive outputs without interval prompt modification. These 18 items were evaluated by a consensus panel of five authors against the scoring rubric, then independently reviewed by another author. The results of this variability testing are given in Appendix 2.

References

The veracity of the references generated by GPT-4 to support the generated feedback, were also evaluated from the 18 ‘variability testing’ items. These were evaluated against the criteria: ‘The references included were real, relevant to the MCQ, formatted, and peer-reviewed.’ The items were scored on a scale of 1 (entirely does not meet criteria), 2 (mostly does not meet criteria), 3 (mostly meets criteria), to 4 (entirely meets criteria). Specific inaccuracies in the references were documented.

Item appraisal - consensus panel scoring

All items, AI and human-generated, were pooled and then evaluated in random order by a consensus panel of five authors blinded to the origin of the item using the prespecified scoring rubric. One duplicate item was identified and excluded.

Examples of a novice, expert, and AI-generated item used in this study are presented in Appendix 3.

The panel, by majority vote, also recorded their prediction for whether the item was authored by a novice, expert, or GPT-4.

Ethics approval

This project received approval from the University of Adelaide Human Research Ethics Committee (HREC-2023-285).

Data analysis

There was no identifiable data involved in this study. Mean scores for measures of item quality between author types were compared using ANOVA with post-hoc Bonferroni or Tamhane tests as appropriate (the latter was used where the largest variance was at double or higher compared to the lowest). The distribution of the global impression scores (including and excluding feedback) were tallied and presented as a percentage of items. The distribution of item-writing flaws and cognitive skill level were tallied by author type. Summary descriptions are provided for the frequency of correct answer identification and placement, and assessments of quality of referencing. A p value <0.05 was considered significant.

Item quality evaluation - individual structural characteristics

The scoring of items based on their source (Novice, Expert, AI) is summarised in Table 4. One duplicate Novice item was identified and removed prior to data analysis.

For human-authored MCQs, Expert items outperformed Novice items in all categories except for appropriateness of scope and the comprehensiveness of explanatory text, for which no difference was observed between any groups. Expert items modestly outperformed AI items in mean content validity scores (3.98 vs 3.73, p<0.001), cognitive skill level (2.58 vs 2.25, p<0.05), and clinical reasoning in feedback text (3.96 vs 3.65, p<0.001), while no significant difference was observed in item anatomy, scope, number of item-writing flaws, or feedback comprehensiveness. AI items modestly outperformed Novice items in item-writing flaws (0.80 vs 1.33, p<0.05); no significant difference was observed for all other parameters.

Table 4: Item quality scores by author group.

	Expert (n=45)	Novice (n=39)	AI (n=40)	Group differences
CORE ITEM ELEMENTS
Content validity (1-4)	3.98 (0.15)	3.33 (0.90)	3.73 (0.60)	EN, EA
Within scope (1-4)	3.89 (0.38)	3.92 (0.35)	4 (0)	n/s
Item anatomy (1-4)	4 (0)	3.79 (0.41)	3.90 (0.30)	EN*
Item writing flaws (count)	0.82 (0.91)	1.33 (1.24)	0.80 (1.04)	EN, NA
Cognitive skill level (1-3)	2.58 (0.54)	2.26 (0.64)	2.25 (0.54)	EN, EA
EXPLANATORY TEXT
Comprehensiveness (1-4)	3.96 (0.20)	3.79 (0.52)	3.85 (0.36)	n/s
Veracity and clinical reasoning (1-4)	3.96 (0.21)	3.41 (0.85)	3.65 (0.58)	EN, EA

The nature of any significant difference is given in the final column. ‘EN’ denotes a statistically significant difference between Expert and Novice items, ‘EA’ denotes a difference between Expert and AI items, and ‘NA’ denotes a difference between Novice and AI items. All data are presented as mean (SD). Asterisks denote P<0.05 (*) or P<0.001(*). ‘n/s’ denotes no significant difference.

Item quality evaluation - global impressions

Global impression scores of structural quality for each item are summarised in Table 5.

Overall, excluding the explanatory text section, Expert items were most frequently deemed fit for purpose with or without minor edits (95%), followed by AI items (86%), then Novice items (61%). Only Novice items included some deemed unsalvageable (13%). Including the explanatory text section resulted in Expert items deemed most fit for purpose with or without minor edits (95%), followed by AI items (85%), then Novice items (59%).

Table 5: Summary of global impression scores for item structure, by group. Score counts are given as n (%), mean scores are given as mean (SD). Asterisks (**) denote a difference versus the other groups with P<0.001.

	GLOBAL IMPRESSION SCORE
	1	2	3	4	Mean score
Fit for purpose	No	No	Yes	Yes	–
Edits required	Unsalvageable	Major	Minor	None	–

EXCLUDING EXPLANATORY TEXT SECTION
Expert (n=45)	0 (0%)	2 (4%)	19 (42%)	24 (53%)	3.49 (0.59)
Novice (n=39)	5 (13%)	10 (26%)	13 (33%)	11 (28%)	2.77 (1.01)**
AI (n=40)	0 (0%)	6 (15%)	13 (33%)	21 (53%)	3.38 (0.74)
INCLUDING EXPLANATORY TEXT SECTION
Expert (n=45)	0 (0%)	2 (4%)	19 (42%)	24 (53%)	3.49 (0.59)
Novice (n=39)	5 (13%)	11 (28%)	14 (36%)	9 (23%)	2.69 (0.98)**
AI (n=40)	0 (0%)	6 (15%)	14 (35%)	20 (50%)	3.35 (0.74)

When averaged within groups, mean scores for global impression whether including or excluding explanatory text were similar between Expert and AI items. However, AI items outperformed Novice items in global scores both including and excluding explanatory text (3.38 vs 2.77, p<0.001; and 3.35 vs 2.69, p<0.001; respectively).

Item-writing flaws (IWFs)

The number and type of IWFs per item across author groups is presented in Table 6. Though the overall rate of IWFs differed only slightly between Expert, Novice, and AI items (0.8 vs 1.3 vs 0.8, respectively, p<0.05), Novice items were least likely to have zero IWFs and most likely to have 3 or more. Most IWFs belonged to the category ‘Writing the choices’ across all author groups, using the item-writing guidelines outlined in Table 3.

Table 6: Summary of item-writing flaws between groups. All data are presented as counts unless otherwise specified.

		Expert (n=45)	Novice (n=39)	AI (n=40)
IWFs per item, count distribution
	0	21	10	21
	1	13	16	10
	2	9	7	6
	3	2	3	2
	4	0	2	1
	5	0	1	0
IWFs per item, mean (SD)		0.8 (0.91)	1.3 (1.24)	0.8 (1.04)
IWFs by type, count distribution
	Content concerns	8	14	10
	Style concerns	0	5	0
	Formatting concerns	0	0	0
	Writing the stem	11	14	2
	Writing the choices	18	19	30

Cognitive skill level

Table 7 summarises the distribution of cognitive skill level across author groups. The majority of Expert items were at cognitive skill level III, compared to level II when for Novice or AI items. Novice items were most likely to be assigned a cognitive skill level of I.

Table 7: Distribution of assigned cognitive skill levels vis a vis Bloom’s modified taxonomy. Data presented as n (%).

	COGNITIVE SKILL LEVEL
	I	II	III
Expert (n=45)	1 (2%)	17 (38%)	27 (60%)
Novice (n=39)	4 (10%)	21 (54%)	14 (36%)
AI (n=40)	2 (5%)	26 (65%)	12 (30%)

Correct option veracity and placement

The correct answer was appropriately indicated (i.e. the option indicated to be correct by the author was corroborated by panel consensus) in 100% of Expert, 90% of Novice, and 85% of AI-generated items. Significant differences were observed in the distribution of the correct option placement, as shown in Table 8. Both AI-generated (45%, p=0.028) and Expert-authored (33%, p=0.002) items disproportionately positioned the correct answer as option C versus the Novice-authored items.

Table 8: Distribution of correct option position. All data are presented as percentages.

	Option position of correct answer
	A	B	C	D	E
Expert (n=45)	9	20	33	29	7
Novice (n=39)	15	33	13	26	13
AI (n=40)	8	15	45	25	8

References

The references generated by GPT-4 varied in quality. Of the 18 variability testing items’ 52 references that were scored (Appendix 2), the average score against the stated criteria was 3.06 out of 4 (range 2-4), with six items scoring 2, five items scoring 3, and seven deemed perfect with a score of 4. The most common flaws were an incorrect DOI, the reference being an old edition of a guideline, or incorrect details, as summarised in Table 9.

Table 9. Summary of types of errors in the references generated by GPT-4

Flaw in reference	Count
Incorrect DOI	7
Incorrect URL	2
Old edition	5
Incorrect details	5
Not peer reviewed	3
Non-existent	2

This study has added to the growing evidence base demonstrating that pre-trained generative AI systems can produce medical MCQs of broadly comparable structural quality to expert item-writers, with several important caveats relevant to educators interested in leveraging this technology. MCQs remain ubiquitous in medical education, and the resource-intensiveness of generating high quality items has prompted the exploration of new avenues of content generation. We present the most comprehensive evaluation to date of clinical medical MCQs generated by GPT-4. Taken together, these results suggest significant efficiency gains are feasible in MCQ creation via the conscientious integration of generative AI but underscore the ongoing necessity of human expert review as part of such a workflow to ensure quality, veracity, and fitness-for-purpose of AI-generated materials.

Granular appraisal of the intrinsic properties of MCQ items in this study yielded a rough hierarchy of quality from Novice-authored items (written by medical students or junior doctors and otherwise unedited), through AI-generated items (produced by GPT-4), to Expert items (written, edited, and/or approved by subject matter experts and experienced item-writers), with each successive group matching or outperforming the former on particular metrics. Regarding human authors, 39% of Novice items fell into a global impression category signifying either outright unsuitability or a need for major editing to achieve fitness for purpose, while 95% of Expert items were deemed fit for purpose requiring minor editing at most. This aligns with previously demonstrated distinctions between student-authored and expert-authored MCQs (Pham et al., 2023) and highlights that engaging medical trainees to author MCQs, while known to be beneficial for learning (Touissi et al., 2022), nonetheless necessitates a careful review process to guide item development to a usable standard.

The interposition of AI capability between Novice and Expert human authors in our analysis is a novel finding, though perhaps intuitive to educators in light of the constellation of medical knowledge, clinical experience, and pedagogical training required to generate high-quality MCQs with practical relevance and verisimilitude. Existing comparative data is sparse but corroborative: the only other such study as yet found that expert authors tended to outperform AI on assessments of quality when MCQs on the same subject matter were compared head-to-head, having achieved comparable quality scores in aggregate (Cheung et al., 2023). As less direct context, GPT-4 has also been shown to summarise medical information into clinical synopses (Van Veen et al., 2024) or patient information sheets (Currie et al., 2023; Lockie & Choi, 2024; Verran, 2024) to standards that variably exceed, meet, or fall short of those by expert clinicians. Conversely, while AI has not been compared to student or trainee authors on the specific task of MCQ generation prior to our study, GPT-4’s ability to interpret clinical MCQ material has been repeatedly shown to rival or exceed that of medical students and trainees - cohorts corresponding to the Novice group in our dataset - in the profusion of reports in which it achieved passing grades in qualifying examinations around the world (Abbas et al., 2024; Knoedler et al., 2024; Maitland et al., 2024; Meyer et al., 2024; Rojas et al., 2024; Tanaka et al., 2024). Taken together, the available evidence loosely suggests that GPT-4’s proficiency in handling clinical MCQ materials falls somewhere toward the upper end of a range between students/trainees and experts.

Focussing on the AI-generated items in this study, multiple results are important to highlight in evaluating the applicability of this technology to medical MCQ creation. AI items demonstrated high quality in aggregate, with 85% rated as fit for use upfront with minor edits at most, 95% deemed to test higher-order cognitive skills (modified Bloom’s level II or III), and excellent average scores across content validity (3.7/4), scope (4/4), and item anatomy (3.9/4). Though prior non-comparative studies have reported widely variable AI-generated MCQ quality (Artsi et al., 2024), our results support the view that expert-standard items at a clinically complex content level are within the capability of GPT-4. This high-quality output is likely closely dependent upon the input of a detailed prompt invoking best-practice item-writing principles, the inclusion of high quality example MCQs, and a well-articulated learning point, which was a feature of this study and strongly endorsed in literature (Meskó, 2023). Development of such a prompt - especially one that encompasses local requirements for structure, content, and style - necessitates human input at the outset of an AI-based workflow. A generic prompt that relies on the encoded clinical knowledge in the LLM minimises the human input required at subsequent stages of MCQ design. In this study, the provision of a factually verified key learning point by the human superseded the need to provide GPT-4 with any lengthy reference texts.

However, it is also crucial to consider not just the average quality across an entire exam, but outputs on a per-item basis in a safety-critical field like healthcare where high minimum standards of quality and veracity are necessary for all individual assessment items (Giuffrè et al., 2024). In this regard, it must be emphasised that 1 in 7 AI-generated items were deemed unfit for use without major edits (versus 1 in 25 by human experts), 1 in 7 indicated an erroneous correct answer, and almost half had correct answers default positioned as option C. Any of these issues would severely compromise assessment validity if left unchecked, and their presence indicates the vital necessity of incorporating human expert review prior to deployment of AI-generated items. Some of these issues may arise from technical limitations of GPT-4 capability, but it is also very likely that the AI engine is recapitulating the shortcomings of its training data: for example, a pervasive ‘middle bias’ is known to exist in MCQ assessments (Attali & Bar‐Hillel, 2003), in which correct options are disproportionately clustered in middle positions (e.g. option C in an A-E structure), and it is likely that this tendency has been encoded into the engine via its training data – along with other more pernicious biases of which educators must be cognisant (Zack et al., 2024).

Though detailed technical inquiry into LLM clinical reasoning was beyond our scope (readers are directed to Liévin et al. (2024), this study demonstrated GPT-4’s capability to generate cogent and cohesive explanatory text that, while marginally outperformed by human experts on ratings of veracity and articulated reasoning, ultimately equalled experts and exceeded novices on global impression. The complete fabrication of references by GPT-4 in this study also echoes known issues with LLM hallucination. This contrasts to prior literature in which 32-76% of generated MCQ explanations were deemed valid (Agarwal et al., 2023; Choi, 2023; Ngo et al., 2024), and may reflect interval improvement from previous versions of the GPT engine. Explanatory feedback to MCQs may enhance the acquisition or consolidation of contextualised knowledge and clinical reasoning for learners, functioning as a ‘virtual teaching assistant’. GPT-4 represents a highly efficient potential avenue to generate such material.

LIMITATIONS

● This mixed-methods study used a combination of pre-existing (Novice and Expert) and prospectively generated (AI) items, which introduces a lack of standardisation between instructions given to human authors and the GPT-4 prompt that may have influenced the respective item quality.

● GPT-4 ‘temperature’: Preliminary investigations have yet to identify the ideal temperature for medical MCQ generation (Agarwal et al., 2024), and it is likely to vary in different settings. A temperature of 0.0 was selected in this study to maximise external validity and reproducibility but may potentially have constrained output quality.

● Cognitive level and IWFs were considered contributory to item quality, based on standard item-writing guidance, however impact on discriminatory power is empirically inconsistent (Tarrant & Ware, 2008; Caldwell & Pate, 2013; Ali & Ruit, 2015; Pais et al., 2016; Rush et al., 2016; Pham et al., 2018).

● This study explicitly focussed on items requiring higher-order cognitive processing. GPT-4 was recently shown to make the most errors with lower-order reasoning in a set of psychiatric MCQs (Herrmann-Werner et al., 2024), and the output quality in this study is therefore not generalisable to the creation of medical MCQs targeting these lower-order processes.

FUTURE DIRECTIONS

The results of an evaluation of the psychometric properties of AI-generated MCQs will be reported by the authors in a future article. Further areas of study include exploring the quality of outputs using other LLMs, exploring the role of clinical images in AI-assisted design of MCQs, and evaluating the educational value of interacting with AI-generated explanatory feedback on student learning. More sophisticated prompt engineering with automatic provision of reference texts could also be explored, as well as investigating fine-tuning processes of LLMs with large volumes of high quality MCQs.

In summary, this study suggests that while human experts most reliably produce superior-quality complex clinical MCQs, GPT-4 is capable of producing comparable items in most instances and outperforms human novices in this task. An AI-integrated workflow for creating such items would still necessitate direct human expert input at multiple points, including prompt engineering in accordance with item-writing guidelines and local pedagogy, formulation of curriculum-specific key learning points, and validation of subsequent outputs with editing as required.

Project contributions

HW and SCK conceived the project idea.

HW and EP designed the study.

DL constructed the ethics application, which HW contributed to.

HW, DL, and SCK conducted the literature review.

All authors contributed to the design of the standardised scoring rubric, prompt engineering, and to data collection.

HW, PD, and TZ prepared the items for scoring.

EP conducted the statistical analysis.

HW and SCK drafted the manuscript, which all authors then contributed to and provided critical review. The tables were prepared by both HW and SCK.

Acknowledgements

None.

Declaration of interest statement

Authors HW, TZ, SCK, and PD are co-directors of eMedici, a commercial medical education platform. DL and EP have no conflicts of interest to declare. The authors did not receive support from any organization for the submitted work.

Abbas, A., Rehman, M. S., & Rehman, S. S. (2024). Comparing the Performance of Popular Large Language Models on the National Board of Medical Examiners Sample Questions. Cureus, 16(3). https://doi.org/10.7759/cureus.55991
Abd-Alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alrazak, S. A., & Sheikh, J. (2023). Large language models in medical education: opportunities, challenges, and future directions. JMIR Medical Education, 9(1), e48291. https://doi.org/10.2196/48291
Abozaid, H., Park, Y. S., & Tekian, A. (2017). Peer review improves psychometric characteristics of multiple choice questions. Medical teacher, 39(sup1), S50-S54. https://doi.org/10.1080/0142159X.2016.1254743
Agarwal, A., Mittal, K., Doyle, A., Sridhar, P., Wan, Z., Doughty, J. A., Savelka, J., & Sakr, M. (2024). Understanding the Role of Temperature in Diverse Question Generation by GPT-4. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2,
Agarwal, M., Goswami, A., & Sharma, P. (2023). Evaluating ChatGPT-3.5 and Claude-2 in answering and explaining conceptual medical physiology multiple-choice questions. Cureus, 15(9). https://doi.org/10.7759/cureus.46222
Ali, S. H., & Ruit, K. G. (2015). The Impact of item flaws, testing at low cognitive level, and low distractor functioning on multiple-choice question quality. Perspectives on medical education, 4, 244-251. https://doi.org/10.1007/s40037-015-0212-x
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education, 24(1), 354. https://doi.org/10.1186/s12909-024-05239-y
Attali, Y., & Bar‐Hillel, M. (2003). Guess where: The position of correct answers in multiple‐choice test items as a psychometric variable. Journal of Educational Measurement, 40(2), 109-128. https://doi.org/10.1111/j.1745-3984.2003.tb01099.x
Ayub, I., Hamann, D., Hamann, C. R., & Davis, M. J. (2023). Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis. Cureus, 15(8).
Benítez, T. M., Xu, Y., Boudreau, J. D., Kow, A. W. C., Bello, F., Van Phuoc, L., Wang, X., Sun, X., Leung, G. K.-K., & Lan, Y. (2024). Harnessing the potential of large language models in medical education: promise and pitfalls. Journal of the American Medical Informatics Association, 31(3), 776-783. https://academic.oup.com/jamia/article-abstract/31/3/776/7588721?redirectedFrom=fulltext
Caldwell, D. J., & Pate, A. N. (2013). Effects of question formats on student and item performance. American journal of pharmaceutical education, 77(4), 71. https://doi.org/10.5688/ajpe77471
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., Wong, R., & Co, M. T.-H. (2023). ChatGPT versus human in generating medical graduate exam multiple choice questions—A multinational prospective study (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS ONE, 18(8), e0290691. https://doi.org/10.1371/journal.pone.0290691
Choi, W. (2023). Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs. BMC Medical Education, 23(1), 864. https://doi.org/10.1186/s12909-023-04832-x
Coşkun, Ö., Kıyak, Y. S., & Budakoğlu, I. İ. (2024). ChatGPT to generate clinical vignettes for teaching and multiple-choice questions for assessment: A randomized controlled experiment. Medical teacher, 1-7. https://doi.org/10.1080/0142159X.2024.2327477
Currie, G., Robbie, S., & Tually, P. (2023). ChatGPT and patient information in nuclear medicine: GPT-3.5 versus GPT-4. Journal of Nuclear Medicine Technology, 51(4), 307-313. https://doi.org/10.2967/jnmt.123.266151
Giray, L. (2023). Prompt engineering with ChatGPT: a guide for academic writers. Annals of biomedical engineering, 51(12), 2629-2633. https://doi.org/10.1007/s10439-023-03272-4
Giuffrè, M., You, K., & Shung, D. L. (2024). Evaluating ChatGPT in medical contexts: the imperative to guard against hallucinations and partial accuracies. Clinical Gastroenterology and Hepatology, 22(5), 1145-1146. https://www.cghjournal.org/article/S1542-3565(23)00835-2/pdf
Gupta, P., Meena, P., Khan, A. M., Malhotra, R. K., & Singh, T. (2020). Effect of faculty training on quality of multiple-choice questions. International Journal of Applied and Basic Medical Research, 10(3), 210-214. https://doi.org/10.4103/ijabmr.IJABMR_30_20
Haladyna, T. M., & Downing, S. M. (1989). Validity of a taxonomy of multiple-choice item-writing rules. Applied measurement in education, 2(1), 51-78. https://doi.org/10.1207/s15324818ame0201_4
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333. https://doi.org/10.1207/S15324818AME1503_5
Herrmann-Werner, A., Festl-Wietek, T., Holderried, F., Herschbach, L., Griewatz, J., Masters, K., Zipfel, S., & Mahling, M. (2024). Assessing ChatGPT’s mastery of Bloom’s taxonomy using psychosomatic medicine exam questions: mixed-methods study. Journal of medical Internet research, 26, e52113. https://doi.org/10.2196/52113
Heston, T. F. (2023). Prompt engineering for students of medicine and their teachers. arXiv preprint arXiv:2308.11628.
Heston, T. F., & Khun, C. (2023). Prompt engineering in medical education. International Medical Education, 2(3), 198-205. https://doi.org/10.3390/ime2030019
Hift, R. J. (2014). Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine? BMC Medical Education, 14, 1-18. https://doi.org/10.1186/s12909-014-0249-2
Jacobsen, L. J., & Weber, K. E. (2023). The promises and pitfalls of ChatGPT as a feedback provider in higher education: An exploratory study of prompt engineering and the quality of AI-driven feedback. https://doi.org/10.31219/osf.io/cr257
Jobs, A., Twesten, C., Göbel, A., Bonnemeier, H., Lehnert, H., & Weitz, G. (2013). Question-writing as a learning tool for students–outcomes from curricular exams. BMC Medical Education, 13, 1-7. https://doi.org/10.1186/1472-6920-13-89
Kıyak, Y. S., Coşkun, Ö., Budakoğlu, I. İ., & Uluoğlu, C. (2024). ChatGPT for generating multiple-choice questions: evidence on the use of artificial intelligence in automatic item generation for a rational pharmacotherapy exam. European journal of clinical pharmacology, 80(5), 729-735. https://doi.org/10.1007/s00228-024-03649-x
Klang, E., Portugez, S., Gross, R., Brenner, A., Gilboa, M., Ortal, T., Ron, S., Robinzon, V., Meiri, H., & Segal, G. (2023). Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4. BMC Medical Education, 23. https://doi.org/10.1186/s12909-023-04752-w
Knoedler, L., Alfertshofer, M., Knoedler, S., Hoch, C. C., Funk, P. F., Cotofana, S., Maheta, B., Frank, K., Brébant, V., & Prantl, L. (2024). Pure Wisdom or Potemkin Villages? A Comparison of ChatGPT 3.5 and ChatGPT 4 on USMLE Step 3 Style Questions: Quantitative Analysis. JMIR Medical Education, 10(1), e51148. https://doi.org/10.2196/51148
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024). Can large language models reason about medical questions? Patterns, 5(3).
Lockie, E., & Choi, J. (2024). Evaluation of a chat GPT generated patient information leaflet about laparoscopic cholecystectomy. ANZ Journal of Surgery, 94(3), 353-355. https://doi.org/10.1111/ans.18834
Maitland, A., Fowkes, R., & Maitland, S. (2024). Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework. BMJ open, 14(3), e080558. https://doi.org/10.1136/bmjopen-2023-080558
Meskó, B. (2023). Prompt engineering as an important emerging skill for medical professionals: tutorial. Journal of medical Internet research, 25, e50638.
Meyer, A., Riese, J., & Streichert, T. (2024). Comparison of the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination: observational study. JMIR Medical Education, 10, e50965. https://doi.org/10.2196/50965
Naeem, N., van der Vleuten, C., & Alfaris, E. A. (2012). Faculty development on item writing substantially improves item quality. Advances in health sciences education, 17, 369-376. https://doi.org/10.1007/s10459-011-9315-2
Ngo, A., Gupta, S., Perrine, O., Reddy, R., Ershadi, S., & Remick, D. (2024). ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Academic Pathology, 11(1), 100099. https://www.academicpathologyjournal.org/article/S2374-2895(23)00031-3/pdf
OpenAI. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Pais, J., Silva, A., Guimarães, B., Povo, A., Coelho, E., Silva-Pereira, F., Lourinho, I., Ferreira, M. A., & Severo, M. (2016). Do item-writing flaws reduce examinations psychometric quality? BMC Research Notes, 9, 1-7. https://doi.org/10.1186/s13104-016-2202-4
Palmer, E. J., Duggan, P., Devitt, P. G., & Russell, R. (2010). The modified essay question: its exit from the exit examination? Medical teacher, 32(7), e300-e307. https://doi.org/10.3109/0142159X.2010.488705
Papinczak, T., Peterson, R., Babri, A. S., Ward, K., Kippers, V., & Wilkinson, D. (2012). Using student-generated questions for student-centred assessment. Assessment & Evaluation in Higher Education, 37(4), 439-452. https://doi.org/10.1080/02602938.2010.538666
Pham, H., Besanko, J., & Devitt, P. (2018). Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question. MedEdPublish, 7. https://doi.org/10.15694/mep.2018.0000225.1
Pham, H., Court-Kowalski, S., Chan, H., & Devitt, P. (2023). Writing Multiple Choice Questions—Has the Student Become the Master? Teaching and Learning in Medicine, 35(3), 356-367. https://doi.org/10.1080/10401334.2022.2050240
Rojas, M., Rojas, M., Burgess, V., Toro-Pérez, J., & Salehi, S. (2024). Exploring the Performance of ChatGPT Versions 3.5, 4, and 4 With Vision in the Chilean Medical Licensing Examination: Observational Study. JMIR Medical Education, 10, e55048. https://doi.org/10.2196/55048
Rush, B. R., Rankin, D. C., & White, B. J. (2016). The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education, 16, 1-10. https://doi.org/10.1186/s12909-016-0773-3
Schuwirth, L. W., & Van Der Vleuten, C. P. (2003). Written assessment.(ABC of learning and teaching in medicine). British Medical Journal, 326(7390), 643-646. https://doi.org/10.1136/bmj.326.7390.643
Shah, M. P., Lin, B. R., Lee, M., Kahn, D., & Hernandez, E. (2019). Student-written multiple-choice questions—a practical and educational approach. Medical Science Educator, 29, 41-43. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8368101/pdf/40670_2018_Article_646.pdf
Smeby, S. S., Lillebo, B., Gynnild, V., Samstad, E., Standal, R., Knobel, H., Vik, A., & Slørdahl, T. S. (2019). Improving assessment quality in professional higher education: Could external peer review of items be the answer? Cogent Medicine, 6(1), 1659746. https://doi.org/10.1080/2331205X.2019.1659746
Tanaka, Y., Nakata, T., Aiga, K., Etani, T., Muramatsu, R., Katagiri, S., Kawai, H., Higashino, F., Enomoto, M., & Noda, M. (2024). Performance of generative pretrained transformer on the national medical licensing examination in Japan. PLOS Digital Health, 3(1), e0000433. https://doi.org/10.1371/journal.pdig.0000433
Tarrant, M., & Ware, J. (2008). Impact of item‐writing flaws in multiple‐choice questions on student achievement in high‐stakes nursing assessments. Medical Education, 42(2), 198-206. https://doi.org/10.1111/j.1365-2923.2007.02957.x
Touissi, Y., Hjiej, G., Hajjioui, A., Ibrahimi, A., & Fourtassi, M. (2022). Does developing multiple-choice questions improve medical students’ learning? A systematic review. Medical Education Online, 27(1), 2005505. https://doi.org/10.1080/10872981.2021.2005505
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., & Seehofnerová, A. (2024). Adapted large language models can outperform medical experts in clinical text summarization. Nature medicine, 30(4), 1134-1142. https://doi.org/10.1038/s41591-024-02855-5
Verran, C. (2024). Artificial intelligence-generated patient information leaflets: a comparison of contents according to British Association of Dermatologists standards. Clinical and Experimental Dermatology, llad461. https://doi.org/10.1093/ced/llad461
Zack, T., Lehman, E., Suzgun, M., Rodriguez, J. A., Celi, L. A., Gichoya, J., Jurafsky, D., Szolovits, P., Bates, D. W., & Abdulnour, R.-E. E. (2024). Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. The Lancet Digital Health, 6(1), e12-e22. https://doi.org/10.1016/S2589-7500(23)00225-X (Erratum in: Lancet Digit Health. 2024 Jul;6(7):e445. doi: 10.1016/S2589-7500(24)00120-1)

Competing interest reported. Authors HW, TZ, SCK, and PD are co-directors of eMedici, a commercial medical education platform. DL and EP have no conflicts of interest to declare. The authors did not receive support from any organization for the submitted work

Appendix.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

GPT-4 versus human authors in clinically complex MCQ creation: a blinded analysis of item quality

Status:

Version 1

Abstract

INTRODUCTION

METHODS

RESULTS

DISCUSSION

LIMITATIONS

CONCLUSION

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1