Participant Characteristics
The student participants were predominantly female (n = 11, 73.3%) with a median age of 24 years (range 22-45 years). Most students were entering their third or fourth year of pharmacy school (n = 11, 73%), meaning they had experience working in a pharmacy practice setting through required clinical experiences. In addition, 13 students (87%) indicated working in a healthcare-related field outside of their coursework. Eight students (53%) reported working in a non-healthcare human services field with a one year of experience being the median (range 0-10 years). Eighty percent (n = 12) of students reported having training related to empathy; they most often cited coursework or classroom discussions regarding mental health and working with patients.
The pharmacists were predominantly female (n = 13, 86.6%) with a median age of 36 (range 29-51 years). All pharmacists were employed in a university hospital setting, working in various practice areas, with a median of 8 years of experience as a licensed pharmacist (range 6-23 years). Most completed residency training (n = 13, 87%) and were board certified (n = 11, 73%), indicating these individuals have extensive training in specialty areas and providing advanced patient care. Eleven pharmacists (73%) reported previously working in a non-healthcare human services field with a median of 4 years of experience (range 0-10 years) outside of pharmacy. Only 33% (n = 5) of pharmacists reported having training related to empathy; participants frequently cited exposure to material related to emotional intelligence or service recovery training specific to their institution.
All 30 participants completed the QCAE. Mean QCAE score was 91.8 (SD 6.1) and total scores ranged from 79 to 105. The mean CE and AE sub-scores were 57.1 (SD 5.4) and 34.7 (SD 3.8), respectively. Results of a Mann-Whitney test suggested non-significant differences (p > .05) between the median QCAE, CE, and AE scores for participants in the student and pharmacist groups. This finding implies that the pharmacy students and licensed pharmacists included in this study scored similarly and did not differ significantly in their levels of cognitive, affective, and overall empathy.
Average SJT score was 180.6 (SD 11.8) with a range of 142 points (59.2% correct) to 200 points (83.3% correct). Results of a Mann-Whitney test suggested non-significant differences (p > .05) between the median SJT scores for participants in the student and pharmacist groups; this corroborates findings from the QCAE that suggested similar student and practitioner empathy levels.
Proposed SJT Response Process Model
In this section, a model has been proposed that builds on the framework of Tourangeau and colleagues, who describe survey response processes as including four key components: comprehension, retrieval, judgment, and response selection.26 This framework was combined with features previously reported to be salient in the response process in addition to new features identified through this exploratory research.18,45,46
The model, provided in Figure 2, includes the four primary components connected to the features that are proposed to influence each step in this process. Features that are bolded are those that have substantial evidence from cognitive and think-aloud interviews to support their existence in SJT response processes (i.e., described in detail in the subsequent sections), whereas those that are not bolded have limited data to support their inclusion. All features evaluated in this exploratory analysis were included as there were references to all components at least once in the process; therefore, the significance of these relationships cannot be excluded. Due to the qualitative nature of this study, a larger sample size would be necessary to confirm if the minor features could be excluded in subsequent models. Within each box connected to the primary component, features are ordered in terms of their prevalence (i.e., features that are higher on the list were referenced more frequently and identified as having a notable influence on the response process).
Comprehension Component
Comprehension was considered an essential component of SJT response processes, as a participant must read the item to be able to answer it accordingly. This component also included references to how participants interpreted key elements of SJT scenarios, which is a significant component of the comprehension process. This research identified two features not previously described in the literature: (1) participants often identified a task or objective that needed to be completed and (2) participants made assumptions about the scenario. In addition, the comprehension component is connected to the ability to identify the construct as the examinee’s interpretation of the item can be related to the suspected construct.
First, participants often identified an objective that was to be achieved in the scenario. Provided response options were then evaluated—in the judgment process—based on predictions of how well that response would achieve the targeted objective, among other factors. The objectives identified by participants in the cognitive and think-aloud interviews were categorized based on the goal they described. A list of these categories, descriptions, and examples (i.e., excerpts from the pharmacists and students) are provided in Table 2; these categories are ordered from most to least prevalent.
The task objective most often referenced by participants was related to the exchange of information, which could include collecting or sharing information with another individual. The objective least often described by participants referred to modifying a relationship, often between a patient and the healthcare provider. Of note, many task objectives were broad and lacked a specific focus. For example, participants made general statements about something working well or not without any indication of an explicit goal. Based on the distribution of codes, participants attempted to identify the task objective during this SJT regardless of the item setting, empathy component being assessed, or participant type.
Next, in addition to identifying the objective of SJT scenarios, comprehension of SJT items also included the participant making key assumptions about the presented case. Throughout the cognitive and think-aloud interviews, participants made statements about how they interpreted information that was provided. These assumptions could be classified according to what the assumptions were about, which is summarized in Table 3 with descriptions and examples. The assumption categories are organized from most to least prevalent across all interviews. Based on the distribution of codes, there was some evidence to suggest that the type of component being assessed may contribute to varying uses of assumptions; however, the limited sample size precludes generalization.
In general, assumptions appeared to serve as a component of the response process for some participants when there were insufficient details provided in the scenario. As many of these scenarios were designed to exclude extraneous details, it was possible that this required more inferences by the participants. One participant best described this process by saying, “there’s a fair amount of projection” onto the scenario, depending on the elements that were provided. These data suggest that details about the scenario may be necessary if the use of assumptions in the comprehension process is not desirable. Overall, assumptions made up a small proportion of the total number of codes (3.1%), therefore, there is minimal evidence to suggest that assumptions are an overwhelmingly significant component of the SJT response process. It is evident, however, that assumptions can be used by participants to fill in the gaps and it may be advisable that SJT design includes explicit statements for examinees pertaining to assumptions about the setting or other features to avoid misinterpretation.
Retrieval Component
Retrieval was the next component of the SJT response process in which participants reflected on knowledge and experiences pertinent to the scenario while they formulated their response selection—this also included references to job-specific and general knowledge and experiences. Of note, in the proposed model, there is a bidirectional relationship between retrieval and judgment that differs somewhat from the original model presented by Tourangeau and colleagues (2000). The proposed model suggests that the response process is not always linear and can integrate various memories and judgments that build on each other prior to the final decision in the response selection, which was evident by participants who retrieved multiple experiences or knowledge elements when discussing SJT items.
Transcripts were analyzed to determine if there were consistent features of the experiences and knowledge referenced by participants that were retrieved during the SJT. References to job-specific and general experiences often included features related to the location, the actors in the scene, and the task or topic. In addition, the experiences could be classified on their similarity to the presented scenario, the specificity of the details provided, and the recency of the memory to the present moment. Features of knowledge references included information, a strategy, or a skill that was applicable to the scenario. Table 4 provides a description of these features and examples from the transcripts.
With respect to job-specific experiences and knowledge, pharmacists and students generally referenced these elements in very similar ways. Pharmacists often explicitly connected to their work experiences with few references to pharmacy school; whereas students included experiences from school, clinical rotations, and some work experiences. In addition, student experiences more often included observations of interactions in which they were not an active participant as well as shared stories, class discussions, and simulations. For example, one student participant (S10) discussed how they had “seen some pharmacists delivering sensitive information about what could happen with certain drugs”; a pharmacist, P13, when discussing the same test item instead thought “about a situation when [they] were practicing in the HIV clinic.” Another example was from S3 who stated, “I know we talked about a lot of different scenarios in class… especially diabetes patients” and S2 who shared, “we’ve talked about medication errors in class a lot and I’ve talked about it on some of my rotations.” The data suggests that students more often integrate job-specific experiences that relate to their education and training witnessed so far, which may not include their direct involvement in a similar scenario.
Components that were not discussed frequently included general experiences and knowledge. In general, the experiences tended to be vague and closely related to the presented SJT scenarios. The actors in these scenarios were often friends and family members and the discussion about these experiences occurred mostly when discussing items referring to non-healthcare settings. One notable feature was that examples from televisions shows were sometimes referenced as viable experiences. For example, when P15 was discussing the item related to a friend taking a medication to help them study their immediate response when asked about the question was “Jesse Spano – from Saved by the Bell.” One student, S13, also discussed “I think of experiences that a lot of times I watch on TV shows like Dateline.” General knowledge often included references to information such as “just thinking about social norms, you wouldn’t confront somebody in the grocery store”, as shared by S14. Overall, there is minimal evidence to suggest that general experiences and knowledge include particularly salient features that contributed to SJT response processes in this study.
Participants also included nondescript experiences as well as references to a lack of experience or knowledge; however, these references were limited. Examples included instances where P1 stated “this [question] is a tough one because I feel like this like a reality every day” and S14 who shared “this one felt familiar to me.” References to a lack of experience, however, were reviewed to determine if they were more prevalent in specific scenarios. Most participants made statements of unfamiliarity, such as “I don’t really have very much to draw on” (S3) or “this has never happened” (P14). One difference, however, was that pharmacists tended to be more specific when they considered whether they had experiences to draw from. For example, P6 stated “I haven’t had a particular scenario with regards to chemotherapy” whereas students discussing the same question would state more generally that they “haven’t been in a situation where a family member is that upset” (S3). The data suggest that practitioners may be more attentive to granular details compared to students when searching for similar experiences.
Judgments Component
Judgments represented the most prominent code and included comments about the decision-making process as well as any value statement made while assessing the response options. The analysis for this component was focused on factors of SJT frameworks that pertained to the judgments, such as emotional intelligence, self-perception, ability, and impressions management. In addition, three new sub-codes were identified during the analysis: perceptions, feelings about the test, and context.
One of the most prominent judgments included the use of emotional intelligence, which was defined as the capacity to be aware of, control, and express one’s emotions as well as the emotions of others (Cherry, Fletcher, O’Sullivan, & Dornan, 2014). This was not abnormal as this SJT was intended to measure participant empathy. Explicit references to affective and cognitive empathy separately, however, were relatively infrequent across interviews compared to other codes. Instead, broad references to empathy were exhibited more often. The remaining factors in SJT frameworks—self-perception, impressions management, and ability—were infrequently discussed among the interviews but were still included in the model as they pertained to judgments in SJT response processes and were consistent with theoretical frameworks about SJTs. Of these three codes, self-perception was the most common, whereas impression management and ability were rare.
Self-perceptions shared by participants often focused on either: (1) attributes of their personality, (2) their identity as a healthcare provider, friend, or family member, or (3) their comfort with a presented scenario. References to their participant personality often included comments such as, “I think I’m probably a little bit less aggressive” (P11) or “I’m not very confrontational” (S11). References to participant identity typically related to their status as a healthcare provider, such as P07 who stated, “I guess being a pharmacist though, it’s a little clearer”. These references also included their identities outside of work as well, such as when P03 shared that “as a new parent” there are differences in how they perceived some situations. Lastly, some participants were aware of their comfort with engaging in certain scenario. For example, S02 stated “I’d feel more comfortable talking about the error if it was something like food”. Each of these types of self-perceptions contributed to their judgements about the scenario and could impact their response selection; however, overall there was limited evidence to suggest their criticality in the process.
Impression management and ability were less frequent in the response process—most participants reported they forgot that they were told to imagine that the test was being used for selection into a health professions program or residency. For the participants who did not forget, they described a struggle with differentiating their answer choices on what they should do compared to what they would do as expected by the individual administering the test. For example, S12 shared they “kind of knew what the right answer was versus what [they] would actually do was harder to separate.” With regard to ability, participants most often made references to a lack of a knowledge of skill set that would allow them to operate best in the given scenario instead of affirmations about their abilities to succeed in a situation. For example, P07 stated that, “as a pharmacist, I’m not really trained to walk-through the risks and benefits in that case.” Overall, the few references to abilities limited the analysis; however, the factor was still retained within the model as there was some evidence to suggest ability (or the lack thereof) may be play a role in the response process in that some response options were ranked lower if the participant did not feel they had the skill set necessary to successfully carry out a response option.
Another new feature identified inductively was that participants made references to perceptions of factors weighed when evaluating response options. These perceptions were coded throughout the cognitive and think-aloud interviews, then categorized based on the features that were most salient. Table 5 includes a summary of the most prevalent categories, as well as a description and example for reference.
The most prevalent type of comment from participants was regarding the impact on how others would think about them if a certain response option was selected. Participants most frequently identified negative attributes about the impact on their image including thoughts that it could: “make you look like a jerk” (S10), “come off like accusing the patient” (S03), and “seem unprofessional” (P06). In general, there was a significant concern about how nice a response was or perceptions about the tone in which something was delivered, which could subsequently impact their image and response selection. Examples included comments about response options that “sounded really cold” (S15) or that “can come off a little harsh” (P05); these responses were then not as highly ranked. Similar to this was the perceived integrity of certain response options; for example, participants evaluated if the response was an honest reflection of the situation or if the response was legal. Each could potentially have implications for the image, but these focused specifically on an important element other than how professional or how nice they were coming across. Other perceptions included an awareness of what individuals would do in real-life scenarios, as well as a balance between perceptions of what participants believed individuals would want in the scenario along with what they would want in the scenario. Lastly, some individuals referenced their instincts in the scenarios and stated, “it just feels right” as their reasoning.
In addition, the setting and contextual features of the item appeared to have an effect on the response process. Interestingly, there were many uses of the phrase “it depends” (and other equivalents) by participants across the transcripts, which suggested the importance of contextual elements in SJT response processes. These factors were coded and classified into four categories: (1) factors pertaining to the participant or examinee, (2) factors pertaining to actors in the presented scenario, (3) factors pertaining to the relationship between the examinee and actors, and (4) factors pertaining to the situation.
Participants often cited multiple factors that influence their response process and that these factors could affect their response differently based on the scenario. For example, item AH1 asks how the participant should respond to a patient who is upset about the recent loss of a loved one. One pharmacist, P06, stated that “If it were a friend, I would have been more inclined to share my own personal experiences…I’d feel more comfortable sharing personal loss and talking about it on a more personal level.” In this case, the participant identified that the actor (e.g., a friend instead of a patient) has an impact on the response selection as well as the relationship (e.g., a personal instead of a professional relationship).
Participants commonly identified that relationships with friends and family members come with different expectations compared to relationships with work colleagues or patients. For example, student S10 shared that when trying to convince a patient about not taking a non-prescribed medication compared to convincing a friend, they thought “it’d be easier because you could come at it from the standpoint of I’ve had training in this… and there’s no evidence to back this up or that’s illegal.” In this case, factors such as the examinee’s training as well as the legality of the situation also contribute to the response process. Altering the question to exclude factors such as the illegality or the examinee’s position could alter their response. The impact of the setting on selection of responses was part of another research question of the larger study and is reported in greater detail elsewhere (Wolcott, 2018).
Response selection
The last component of SJT response processes is the response selection, which, for this study, included having the examinees rank their response option on the SJT. Response selection in this study included any reference to the final ranking assigned to any response option. A notable feature of the response selection in this research study was the integration of general strategies that participants reported using throughout the SJT.
Table 6 summarizes the different strategies that were used by participants in making their final selections. In general, most participants approached the response process in the way they were instructed to, which was to rank responses from most to least appropriate. Others, however, considered working backwards in some situations or identifying the extremes (most and least appropriate) first and then filling in the remaining ranks. Other strategies included comparing response options, guessing, and using a process of elimination. Some participants when reading questions aloud also rephrased the item by orienting themselves within the question. One pharmacist, for example, started each response option with “Do you…” when reading the item aloud despite this not being present in the written document. Students appeared to reference test taking strategies more often than pharmacists based on the distribution of codes. In summary, there was some evidence to suggest that general test taking strategies are a relevant feature in SJT response processes and the use of strategies may differ based on who is taking the test.