Participant Characteristics
The student participants were predominantly female (n = 11, 73.3%) with a median age of 24 years (range 22-45 years). Most students were entering their third or fourth year of pharmacy school (n = 11, 73%), meaning they had experience working in a pharmacy practice setting through required clinical experiences. In addition, 13 students (87%) indicated working in a healthcare-related field outside of their coursework. Eight students (53%) reported working in a non-healthcare human services field with one year of experience being the median (range 0-10 years). Eighty percent (n = 12) of students reported they completed training about empathy; they most often cited coursework or classroom discussions regarding mental health and working with patients.
The pharmacists were predominantly female (n = 13, 86.6%) with a median age of 36 (range 29-51 years). All pharmacists worked in a university hospital setting across various practice disciplines, and they had a median of 8 years of experience as a licensed pharmacist (range 6-23 years). Most completed residency training (n = 13, 87%) and were board-certified (n = 11, 73%), indicating these individuals have extensive training in specialty areas and providing advanced patient care. Eleven pharmacists (73%) reported previously working in a non-healthcare human services field with a median of 4 years of experience (range 0-10 years) outside of pharmacy. Only 33% (n = 5) of pharmacists reported having training about empathy; participants frequently cited exposure to material related to emotional intelligence or service recovery training specific to their institution. A summary of participant demographics and performance on the SJT is available in the supplemental appendix.
Proposed SJT Response Process Model
The study results build on the model proposed by Ployhart, which described the SJT response process with four stages: comprehension, retrieval, judgment, and response selection.18 The model, provided in Figure 2, includes the four stages and the factors believed to contribute to each. Factors that are bolded are those with substantial evidence from the cognitive interviews that support their existence (i.e., described in detail in the subsequent sections). The non-bolded factors have limited data to support their inclusion. The proposed model includes all factors identified at least once in the study due to the exploratory purpose; the team decided that a factor’s significance could not be excluded due to the small sample size. Within each box connected to the primary stage, factors are arranged by prevalence (i.e., factors higher on the list were referenced more frequently and had a notable presence).
Comprehension Stage
During comprehension, the individual reads an item, interprets it, and identifies the question.18,26 This research identified two features not previously described in the literature: participants often identify a task or objective, and participants made assumptions about the scenario. In addition, the comprehension stage includes the ability to identify the construct being assessed.29
Task Objective. Participants often identified an objective or task to accomplish in the scenario. They then evaluate provided SJT response options—in the judgment stage—based on predictions of how well that response would achieve the identified objective. Objectives could often be grouped based on their goals, such as exchanging information, emotional improvement, or problem resolution (Table 2). Of note, many task objectives were broad and lacked a specific focus. For example, participants made general statements about something working well or not without any indication of an explicit goal, such as the student (S15) who said, “that never ends well.”
Assumptions. Participants also made assumptions about how they interpreted the case. Assumptions often referred to the person, tone, severity, information accuracy, urgency, or positionality (Table 3). Participants appeared to share assumptions when they believed the scenario lacked sufficient details. Interestingly, SJT scenarios are frequently designed to exclude extraneous information to limited cognitive overload. One individual (P01) best described this by saying, “there’s a fair amount of projection” onto the scenario. These data suggest that details about the scenario may be necessary if assumptions in the comprehension process are not desirable.
Ability to Identify the Construct. Previous research suggests the examinee’s ability to identify the construct assessed may impact their interpretation and response process.29 In this study, few participants referenced what they believed the item was measuring—usually, it was statements such as, “I am not sure what I am expected to do here” (P06). Even when asked explicitly during the cognitive interview, participants had difficulty distinguishing empathy consistently.
Retrieval Stage
Retrieval includes selecting knowledge and experiences pertinent to the scenario when formulating a response.18,26 For SJTs, the theoretical understanding is the retrieval should consist of references to job-specific and general knowledge and experiences.28 This research also identified that examinees consider their lack of experience or knowledge during their response, which has not been previously described.
Job-Specific Experiences and Knowledge. References to job-specific and general experiences (Table 4) often described the location (e.g., the ICU or community pharmacy) and the actors in the scenario (e.g., patients, physicians, nurses). Experiences could also be classified on their similarity to the presented scenario (e.g., how similar or dissimilar to their memory), the specificity of the details provided (e.g., explicit details they recall), and the recency of the experience to the present moment (e.g., days or weeks ago). Knowledge references (Table 4) included information, strategies, or skills applied to the scenario, such as legal requirements, direct questions to ask, or broad communication techniques, respectively.
General Experiences and Knowledge. General experiences and knowledge (i.e., outside of a healthcare setting) were not referenced often by participants. If discussed, it often occurred with scenarios including friends or family members in a non-healthcare setting. Notable observations included references to television shows as relevant experiences. For example, when P15 discussed the scenario with a friend taking a medication to help them study, their immediate response included, “Jesse Spano – from Saved by the Bell.” One student, S13, discussed, “I think of experiences that a lot of times I watch on TV shows like Dateline.” General knowledge included references to information such as, “just thinking about social norms, you wouldn’t confront somebody in the grocery store,” as shared by S14. Overall, there was minimal evidence in this study suggesting general experiences and knowledge contributed extensively to SJT response processes.
Lack of and Nondescript Experiences. Participants also included nondescript experiences and references to a lack of experience or knowledge; however, these references were limited. Examples included instances where P1 stated, “this [question] is a tough one because I feel like this like a reality every day,” and S14 shared, “this one felt familiar to me.” Most participants made statements about broad unfamiliarity with a situation, such as “I don’t really have very much to draw on” (S3) or “this has never happened” (P14).
Judgment Stage
Judgment includes utterances about the decision-making process as well as any value statement made while assessing the response options. Factors relevant to this stage included references to emotional intelligence, self-awareness, ability, and impression management.18,26 Three new factors identified included: perceptions, feelings about the test, and item context.
Emotional Intelligence and Empathy. One of the most frequent references related to emotional intelligence defined as the capacity to be aware of, control, and express one’s emotions as well as the emotions of others.46 This was not abnormal as the SJT focused on measuring empathy. References to affective and cognitive empathy separately were relatively infrequent; instead, broad references to empathy, such as “putting myself in their shoes” or “this is so sad,” occurred more often and were stated by multiple participants.
Self-Awareness. Participant comments about themselves related to attributes of their personality, their identity, or their comfort with a scenario. For example, individuals shared the scenario did not resonate with their personality, such that, “I think I’m probably a little bit less aggressive” (P11) or “I’m not very confrontational” (S11). References to their identity were typically about their status as healthcare providers, such as P07, who stated, “I guess being a pharmacist, it’s a little clearer.” These references also included identities outside of work. For example, P03 shared that, “as a new parent,” there are differences in how they perceive some situations.
Impression management. Participants rarely described that they were intentionally modifying their responses for the person who would review their answers (i.e., impression management).28 Most participants reported they forgot to imagine that the test was for selection into a health professions program. The participants who did not forget described a struggle with differentiating their answer choices on what they should do compared to what the individual administering the test would expect them to do. For example, S12 shared they “kind of knew what the right answer was versus what I would actually do was harder to separate.”
Ability. Participants often referenced a lack of skills to complete the tasks instead of affirmations about their ability to succeed. For example, P07 stated that “as a pharmacist, I’m not really trained to walk-through the risks and benefits in that case.” Despite the limited number of ability references, the factor remained in the model as there was some evidence to suggest ability (or the lack thereof) may play a role in response processes. For example, some participants stated they ranked options lower if they did not feel they had the skills necessary to carry it out.
Perceptions. One new factor identified was that participants shared perceptions that influenced their evaluation of response options (Table 5). For example, participants described how others in the scenario would perceive them if they selected a specific response option. This code is different from impression management, which refers to how the assessor may view the examinee and whether their actions align with the job expectations. Participants often focused on negative impacts such as it could: “make you look like a jerk” (S10), “come off like accusing the patient” (S03), and “seem unprofessional” (P06). Participants also evaluated how the response would be delivered and perceptions about tone; for example, a response option that “sounded really cold” (S15) or that “can come off a little harsh” (P05)—they would then not rate the option as highly. Similar to this was the perceived integrity of response options; for example, participants evaluated if the response was an honest reflection of the situation or if the response was legal.
Setting. Another new factor was the role of the item setting—many participants supplemented their selections with “it depends” and other equivalents. Participants cited many factors, such as their role in the scenario, the role of the actors in the scenario, the relationships between themselves and the actors, and historical factors about the scenario. One pharmacist, P06, stated that “If it were a friend, I would have been more inclined to share my own personal experiences…I’d feel more comfortable sharing personal loss and talking about it on a more personal level.” The participant identified that the actor (e.g., friend or patient) and the relationship (e.g., a personal instead of a professional association) impacts the response. Participants also explained there are different expectations based on relationships with colleagues compared to patients. For example, one student (S10) shared it is easier to convince a patient (rather than a friend) not to take a non-prescribed medication “because you could come at it from the standpoint of I’ve had training in this.”
Response Selection Stage
The response selection stage included any reference to the final ranking assigned to a response option.18,26 Table 6 summarizes the different techniques used by participants in making their final selections.
Strategies. Most participants approached the response process in the way they were instructed to, which was to rank responses from most to least appropriate. However, some individuals worked backward (i.e., from least appropriate to most appropriate) in some situations, or they identified the extremes (i.e., most and least appropriate) first and then filled in the remaining ranks. Other strategies included comparing response options, guessing, and using a process of elimination. Some participants, when reading questions aloud, also rephrased the item by orienting themselves within the question. For example, one pharmacist started each response option with “Do you…” when reading the item aloud despite this not being present in the written document.