Interviews were conducted with 11 educators in the medical program. Six were Problem Based Learning (PBL) tutors, 5 were university employed clinicians in academic teaching and research positions. Six were female. Five PBL tutors were employed in ongoing academic appointments, and one was employed on a sessional basis. Of the eleven participants, only two had received formal training in assessment. Participants described different levels of formal responsibilities and engagement in item writing (see Figure 1). Formal responsibility was determined by their school assigned roles as coordinators of medical program components, or as lecturers and teachers who are formally invited to submit items. In contrast, PBL tutors may informally engage in item writing, but it is not a requirement of their teaching roles. Level of engagement was determined by the participant’s self-described involvement. Seven participants were classified as ‘responsible and engaged’, two were classified as ‘not responsible but engaged’ and two were ‘not responsible and not engaged’.
When asked the features of a good item, all described typical features such as being matched to important and clinically relevant knowledge, of appropriate difficulty and discrimination, covered what students were supposed to know, and clearly written.
Something that’s relevant. Something that they actually need to know. It’s worded well, so it’s not ambiguous. [INT6]
There were four major themes, comprising
- Who’s Responsible? or responsibility for item writing,
- Item writer Motivations, Barriers and Enablers,
- How Much Expertise? The content expertise required to write items and
- Differences in the writing process, particularly between clinicians and non-clinicians.
Who’s responsible?
Participants who were ‘responsible and engaged’ referred to increased responsibility coming with a new academic role, prompting their increased engagement with item writing.
I think it’s one of the responsibilities I think you have as an academic, that that’s part of the package. [INT8]
When asked who they believed should be writing questions for medical school examinations, participants cited basic scientists who deliver or facilitate content in the medical program, and clinicians with their practice based expertise. Some participants specified that only clinicians who held an academic position were responsible. Whether they were formally responsible or not, participants from the engaged groups highlighted the importance of ‘a team effort’ in generating sufficient items.
Motivation, Barriers and Enablers
In addition to formal responsibility, the intellectual challenge of item writing, the importance of effectively assessing future doctors, and of testing content that writers were personally vested in, further motivated those who were engaged:
It’s also quite a good intellectual challenge… it’s a constant upskilling of myself. [INT5]
Less engaged writers cited a lack of time, multiple responsibilities and item writing not being a priority as key reasons that they were not more involved in the process.
I think that to be in involved in that process would take a lot of time which I don’t have right now, and I’d be interested in participating at some point, but probably not until I’ve finished my PhD. [INT2]
Selected barriers and enablers are discussed below.
Social interactions and relationships
This subtheme relates to how interactions with peer writers and with students assisted with item writing.
Participants found peer review and feedback significantly facilitated item writing. While self-review was important, participants found that this skill needed time to develop and feedback from more experienced writers, particularly when they first began writing questions, was invaluable.
It’s important to have a peer review, so that if you’re writing a short answer question, that the question you’re asking, the intention of the question and the concepts, the way that it’s phrased, it’s clearly understood by others. [INT11]
Student interaction through teaching led to identification of content that students struggled with and thus ideas for items.
I think it’s just the interaction of seeing their thought processes, seeing whether they’ve actually grasped the knowledge, even though I’m not teaching them the knowledge… idea of you know, where their strengths are, where their weaknesses are… [INT4]
In addition to peer review, many thought the quality of the examination paper as a whole was improved by input from a variety of writers from different backgrounds. As well as the practicality of sharing the item writing workload for busy academics and clinicians with multiple responsibilities, a collaborative approach meant writers felt a sense of responsibility to contribute their fair share:
This is a team game and if I don’t do it… and then the load falls on one person or two people - (a) that’s not fair and (b) it doesn’t give a broad enough perspective on the subject [INT8]
Organisational processes
Some did not recall being informed about the medical curriculum and existing database of items when initially recruited as item writers, nor when they were appointed to the university. For one writer based at the clinical school, this was a significant challenge:
I also don’t know - I don’t actually know what the curriculum is in general over the five years. So I don’t know what level people are meant to be at. [INT6]
There was no recall of being oriented as a new writer into the writing process, though some participants did receive resources and guidance from experienced writers when they asked for it.
Unpredictable scheduling of requests for new items was cited as a challenge. Those who were not formally responsible, and thus not overseeing the process, noted that items were often only requested near examination periods. Writing then needed to be urgently completed while still handling other responsibilities. While one participant felt that they were not given clear deadlines, another recalled receiving a deadline without much notice.
To sum, communication and administrative issues were barriers, particularly for those who felt engaged but were not formally responsible:
I didn’t readily have access to that, and I was unsure as to whether those lectures had been updated or not… I was unable to make contact with the lecturer. So, I think it’s a systems issue. Having the most recent lecture slides available, made available to the people that are going to create the exam questions, that’s very important. [INT11]
How much expertise?
Competency in item writing requires both content expertise and writing expertise. While some participants suggested that item writing is a technical process and thus expertise could be facilitated by training measures, one participant stated that it is an intuitive process.
Participants across all three groups perceived content expertise as an important requirement and writing outside of one’s discipline was a major barrier for both engaged and unengaged writers. One ‘responsible and engaged’ writer suggested that clinicians could write outside their discipline due to having shared core medical knowledge and training at a level relevant for students. However, even writers with a broad range of content expertise still found the prospect of writing outside of their discipline challenging:
As a GP (general practitioner) it’s very easy for me to say oh look I’m a generalist I can write a variety of questions right across the board in lots of areas… but I think once you start writing exams it’s a completely different kettle of fish to, to doing that. [INT5]
One participant felt confident about writing outside of their discipline, provided there was access to information resources and the opportunity to work with content experts:
I think if I had the information, if I had information and if I was able to then check in with an expert in the field, then yeah, something I’d be capable of. [INT11]
More generally lack of confidence was a significant challenge for both current and potential item writers across all three groups. Despite much teaching experience, one potential writer cited a lack of specific item writing expertise as a key reason they had not been involved:
I think you need to put me in the category of even though I’ve had a lot of experience educating people, I haven’t had very much experience in generating the questions. [INT2]
Participants outside of the medical profession, such as those with an allied health background, also lacked confidence writing outside of their profession:
Because I don’t actually have a clinical background or a basic science background, so I probably wouldn’t be the best person to ask to write questions for the exams. [INT10]
Differences in the writing process
In particular differences were described between medical clinician participants, and those from other disciplinary backgrounds. Clinicians found clinical experience a stronger inspiration for items than student interaction through teaching:
Probably less so, it’s probably more my clinical experience as being the part and parcel of writing questions rather than the actual PBL side of it. [INT5]
Medical clinicians were able to draw on knowledge about clinical medicine and experiences with patients to choose topics relevant to medical practice, but found it challenging to transfer this implicit knowledge into writing, using appropriate words:
You actually find yourself coming up with a question fairly easily on a certain subject for argument’s sake let’s say chlamydia but then trying to actually come with a very exacting question requires all the time. [INT5]
Alluding to unfamiliarity with the language of educators, these participants emphasised the importance of sharing a common language and understandings, by writing with those with whom they could easily communicate:
It’s really important that for clinicians you have someone who really speaks exactly the same language…to translate between sort of the medical education, the educational theory and the clinical language that you’re very comfortable with. [INT1]
Rather than collegiality and professional knowledge, those who were not clinicians tended to rely on their understanding of the taught curriculum and of standard item writing processes such as blueprinting in their approach to item writing:
“So, reviewing through lecture slides, tutorial information, and thinking about ensuring that I – ensuring for the students that the information that’s being delivered is what is being assessed is really important, and to try and have a spread across the curriculum and across the two topics, so that one particular component isn’t more heavily assessed than another.” [INT11]