Having two researchers involved in the data analysis allowed discussions to take place around final themes, how data supported the themes, and if themes overlapped. Five students (S1-5) and seven examiners (E1-7) took part in the interviews. While themes overlapped between students and examiners these were experienced from different perspectives. The interview prompts also explored their recommendations for using a virtual assessment format in the future. The data is presented under three final themes: 1. Stress, 2. Holism and 3. Transferability, with direct quotes from students and examiners as relevant.
1. Stress
This theme represents participants’ references to feelings of uncertainty, feeling stressed or calm, and reading the situation as daunting. For most students, there was a balance of being calmer in their own home environment before the exam but worrying about how the physical examination part of the assessment would translate online:
it was very calming being in your own surroundings...got a coffee with my mum when it was over...you are more relaxed...sometimes the adrenaline of an exam helps you along… (S1)
However, this student also felt that ‘…it was a bit daunting to try a new form of examination for our final exam… (S1). Trying to figure out the transferability of the physical examination of the patient online was a challenge for all students e.g.
…another level of stress...so worried about how we were going to talk through the physical exam… (S3)
Because of Covid-19, the examination format was changed to online within two weeks of the date set, following approval from examiners. This was then communicated to students, so that preparation for the virtual assessment created uncertainty for the group as verbalised in this quote:
…what was challenging was there were a lot of unknowns up till the last point of how our exams would happen...I was not certain if I was saying or doing the right things… (S3)
On the other hand, some examiners expected there to be more stress for the student in a face-to-face exam, for example:
…it (long case) seems to generate a tremendous amount of anxiety, and it stems from this power differential. When you are in the room physically with two examiners and a student...it seems to generate a tremendous amount of anxiety (E7)
Yet, as an examiner, this person admitted’…I was nervous…I did not want to mess it up… It is different than just going to sit in a room and examine the student’ (E7)
For most examiners the stress during the virtual exam was related to internet connectivity concerns, with a feeling that there is ‘…always some technical difficulty...some technical glitches’ (E5). However, one examiner regarded it as ‘…an excellent platform...We were lucky it all went so smoothly (E6).
Transferring such a high stakes clinical examination online did bring its challenges such as the holistic assessment of the patient, discussed in the next theme.
2. Holism
This theme incorporates the perception that a physical examination makes for a more holistic assessment. The belief that the student needs to be able to read the patient and the view that there is a disconnect in the virtual experience as the patient acts as a trigger for the examination process.
One student described this as:
I find (the in-person examination) a better situation for me in terms of getting the information I want from the patient and being able to gauge their reactions and looking for different cues from them. (S4)
However, another student highlighted the disconnect between the patient’s presence face-to-face and the virtual environment, stating that it was not of any benefit as a graduating PA:
It was more comfortable for me, but I do not feel like it was beneficial for me as a clinician. I do not get anything from this. I mean, I can talk to you online, …you cannot see that my body language is completely different than for me to walk into a patient and do this. (S5)
Likewise, from the examiners’ perspectives:
…you will never fully replace a long case...the reading of things like body language...prefer real live patients...the relational stuff...communicating… (E3)
Including physical exam makes for more holistic assessment…a better rapport with people if it is in real life...you can kind of sense things more and there is a better flow...I always think in person is nicer (E1)
A number of examiners agreed that the face-to-face examination gives a ‘more holistic experience’ (E2; E3; E5; E6). Students concurred with this sentiment, overall, preferring a face-to-face exam. The following is typical of how students verbalised this sentiment:
…the non-verbal cues that you would get from a history from a patient...not as evident because you were just looking at a computer (S1).
I do prefer the patient interaction in person and being able to speak directly to the patient and having that closer interaction (S2).
On the other hand, students acknowledged that, while virtual clinics have a place in healthcare, they do not exclude the need for in-person appointments, as the following sample quote suggests
…eventually some patients do need to be seen in person...virtual clinics are the next way forward. During Covid, it was a safer thing not only for consultants and us, but also for the patient...safety comes first and it is just the safer way for everyone involved. (S1)
For an examiner the issue of spontaneity was an issue:
It is hard to be spontaneous...you have to wait until they finish talking so you can then interject something. (E7)
While another examiner felt that ‘you could give them a bit more direction if you were beside them...encourage them’. (E4)
These findings lead into the question of whether or not virtual clinical examinations are transferable or not, which is the final theme.
3. Transferability
While the long case examination was planned in such a way that all elements of the assessment, except for the student performing a physical examination of the patient, were followed as for a face-to-face examination with timing unchanged, the examiners agreed that the in-person format was not transferable to the virtual format. Responses to the prompt about using this assessment in the future included:
…it is not mimicking a real situation, or an inpatient scenario...not a true face to face interaction (E2).
…it is not the same being able to see both the student, your co examiner and the patient participant...I can’t read body language ... read the facial expression (E3).
While not recommending an online long case examination as a final assessment, this examiner acknowledged the convenience of it, and being able to examine someone if you worked in a different setting.
...it, it is handy to turn on your phone and examine someone if you are working somewhere else. So, it gives you more opportunity to have more of a range of examiners, for students as well. And it is good…if you have more of these exams opposed to less in-person ones, then you are getting more experience and training. (E1)
One student suggested that you could score higher on a virtual examination but would still not recommend it for future assessments
I feel the physical ones are more real, they are setting you up for the future and if you mess up on something, it can be corrected straight away. With long cases online, you could be doing so many things wrong and you may never get it correct. So, you are going to keep doing it. With an online long case, you are not examining physical skills at all. But it is easy for the students because you can score higher. (S5)
Likewise, this student did not see it as a substitute for the final long case examination but more as a supplemental type of examination, where students could practice being assessed.
I would recommend it as an addition, a way for students to be assessed more frequently by examiners, when in person things cannot be organised or there's not capacity for it or there is not the space, I think it does give you a chance to run through what a long case is meant to be. You know, you can take a history from your patients. Like I have mentioned, the physical exam part is really challenging. But the questions at the end, that is the same that you would get, even if you were in person. So, there is a space for it, but more like a supplemental type of examination. But, I would not think it should be the sole method of examination for long cases. (S3)
While beneficial for history taking and having a place for practice sessions S2 believes that an in-person examination is better:
You are better off to do physical exams in terms of OSCEs and long case in person, it makes sense. The examiner cannot see your technique online. They can't see that you know, where you are listening to, with your stethoscope. Yes, it is important anatomically to know what people are doing. (S2)
In summary, it seems that, in the context of the pandemic and the challenges which were faced by students and examiners, the virtual assessment met the needs at that time. In relation to the Product of the CIPP model, results for these students were comparable with results from their in-person long case examination in the previous semester.