Trainee versus supervisor viewpoints of entrustment: using artificial intelligence language models to detect thematic differences and potential biases

doi:10.21203/rs.3.rs-3223749/v1

Download PDF

Research Article

Trainee versus supervisor viewpoints of entrustment: using artificial intelligence language models to detect thematic differences and potential biases

https://doi.org/10.21203/rs.3.rs-3223749/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

The entrustment framework redirects assessment from considering only trainees’ competence to decision-making about their readiness to perform clinical tasks independently. Since trainees’ and supervisors’ viewpoints both contribute to entrustment decisions, we examined how they may differentially prioritize factors that determine trust, and how biases may influence this process. Under a social constructivist paradigm, we developed large language model (LLM) based approach to examine feedback dialogs (N = 24187, each with an associated entrustment rating) between student trainees and their precepting supervisors. Using LLM-assisted factor analysis, we compared how trainees and supervisors documented similar types of clinical tasks by identifying factors correlated with entrustment ratings. Supervisors’ factors were dominated by skills related to patient presentations, while trainees’ factors captured a wider range of themes, including both clinical performance and personal qualities. To examine bias, we developed a gender-neutral LLM to measure sentiment in feedback narratives. On average, trainees used more negative language (5.3% lower probability of positive sentiment, p < 0.05) compared to supervisors, while assigning themselves a higher entrustment rating (+ 0.08 on a 1–4 scale, p < 0.05). Trainees’ documentation reflected more positive sentiment in the case of male trainees (+ 1.3%, p < 0.05) and of trainees underrepresented in medicine (+ 1.3%, p < 0.05). Importantly, entrustment ratings themselves did not appear to reflect these biases, neither when documented by the trainee nor supervisor. As such, bias appeared to affect trainee self-perceptions more than the degree of entrustment they experienced. Mitigating these biases is nonetheless important because they may affect trainees’ assimilation into their roles and formation of trusting relationships.

Entrustment

Clinical Supervision

Gender Bias

Natural Language Processing

Large Language Models

Artificial Intelligence

Within supervisor-trainee relationships, entrustment plays a key role in supporting learning and professional growth. Providing trainees with an optimal balance between supervision and autonomy can facilitate their development towards independent practice (ten Cate et al., 2016). Entrustment operationalizes this balance via decisions that rely on a supervisor’s trust in a trainee to perform clinical tasks with varying levels of independence, and encourages feedback on the competencies needed for progressive independence. While recent studies examined supervisor and trainee perspectives on entrustment separately, it would be useful to know how their interpretation of feedback on similar types of clinical tasks (in similar contexts) compares. Such feedback should reflect factors supervisors consider when making entrustment decisions, including both trainees’ competence and personal qualities (Dijksterhuis et al., 2009; Hauer et al., 2014; Kennedy et al., 2007; ten Cate & Chen, 2020). While trainees are aware that these factors influence their supervisors’ trust in them (Caro Monroig et al., 2021; Gin et al., 2021), it is less clear whether trainees regard the same factors as equally important as their supervisors do for earning clinical trust. Supervisors with a performance focus may base their assessments on how well trainees demonstrate clinical competencies, while trainees adapting to the clinical learning environment may be more attuned to how their developing roles and relationships can act as gatekeepers to participation (Castanelli et al., 2021, 2022; Hatala et al., 2022; Pugh & Hatala, 2016). Bridging these two distinct viewpoints of trust – and identifying potential biases that can shape them – are key to developing supervisor-trainee relationships that lead to assessment for learning (AfL) and ensuring equitable implementation of entrustment.

When framing entrustment from a social constructivist standpoint, it becomes paramount to examine both supervisor and trainee viewpoints. Trust within a supervisor-trainee relationship forms collaboratively, developing from verbal and nonverbal interactions between them (Hauer et al., 2014; Mitchell et al., 2018). Hence, entrustment is neither an isolated decision made solely by a supervisor, nor an assessment based only on their judgement of the trainee. The trainee plays a role in the co-construction of trust, and hence their actions and impressions contribute to the entrustment decision (Sklar, 2016). Prior qualitative studies based on retrospective interviews shed light on supervisor and trainee viewpoints separately (Gin et al., 2021; Hauer et al., 2015; Sheu et al., 2016; Sterkenburg et al., 2010). However, their retrospective nature may include hindsight bias and distortion of how those insights apply in moments of actual teaching and practice. Narratives documenting feedback dialogs about observed patient encounters provide a window that is slightly closer to entrustment decisions made in actual practice. Such feedback narratives are sometimes written by supervisors and sometimes by trainees. Comparing narratives written by supervisors and trainees about similar types of clinical tasks (in similar contexts) could provide insight into differences between their perceptions or viewpoints. These narratives can be analyzed for factors related to the selection of entrustment levels, and for the emotional tone expressed in their language (i.e. sentiment). Finally, such feedback may also reflect potential biases related to trainee gender and underrepresented in medicine (UIM) status (Rojek et al., 2019).

When considering similar types of clinical tasks, trainees and supervisors may prioritize different factors when making entrustment decisions. From the supervisor standpoint, studies have focused on factors influencing supervisors’ decisions to entrust trainees. Theoretical studies developed five factors that supervisors consider (trainee, supervisor, context, task, and relationship) (Dijksterhuis et al., 2009; Hauer et al., 2014; Holzhausen et al., 2017; Kennedy et al., 2007), which were supported by empiric studies primarily based on retrospective supervisor interviews (Hauer et al., 2015; Nelson et al., 2023; Sheu et al., 2016). More recently ten Cate and Chen developed a framework that summarizes trainee qualities for entrustment found in the literature (ten Cate & Chen, 2020). The trainee viewpoint of entrustment has focused on several manifestations of trust including self-trust (Sagasser et al., 2017; Sturman et al., 2021), supervisor trust in the trainee (Caro Monroig et al., 2021; Gin et al., 2021; Karp et al., 2019), trainee trust in the supervisor (Castanelli et al., 2021, 2022), and mutual trust (Bonnie et al., 2020). While these aspects of trust are interrelated, in this current study we focus on the trainee’s understanding of their supervisor’s trust, since it is most directly tied to the level of autonomy a trainee is granted (and supervision they are provided) to perform a given clinical task. From retrospective studies on trainees’ perceptions of supervisor trust, we have learned that they consider the same five factors that supervisors use to determine trust (Gin et al., 2021) but with an emphasis on the relationship and trainee factors (Caro Monroig et al., 2021). While these differences in supervisor and trainee viewpoints have been suggested by interviews reflecting aggregate experiences in retrospect, it remains of interest to assess whether they are reflected in actual supervisory decisions when they occur in practice.

The sentiment of feedback narratives may also reveal details around the negotiation of entrustment and acceptance of feedback. Prior work on feedback by Ginsberg, et al. utilized the lens of politeness theory to demonstrate that social pressures to maintain effective supervisor-trainee relationships led to a lack of directness in the language used by supervisors (Ginsburg et al., 2016). Feedback documented by trainees may not reflect such pressures when directed towards themselves. Trainees were found to make active decisions about whether to accept feedback, based on their judgement of the credibility of the feedback provider (van de Ridder et al., 2015). If a trainee were to document a supervisor’s feedback that they did not agree with, the language that the trainee uses may reflect ambiguity or a lack of agency, since the emotional content of language can reflect a trainee’s perceptions of competence and self-efficacy (Sagasser et al., 2017).

Given trust’s dependence on human judgement and instinct (often based upon little data or experience), these viewpoints are inevitably susceptible to bias (Hauer et al., 2023). Multiple junctures within entrustment are subject to bias, including: the entrustment ratings themselves, the content of the narratives, and the language (e.g. sentiment) used in the narratives. Recently, Padilla et al. examined entrustment ratings in a surgical residency context for gender bias (Padilla et al., 2022). They found no such bias in assessments submitted by faculty, but a negative bias in self-assessments submitted by female residents. Dayal et al. examined milestone ratings in an emergency medicine residency, finding a bias in the rate of milestone attainment that favored male residents (Dayal et al., 2017). With respect to content, Mamtani et al. performed a large qualitative study comparing feedback themes in narrative comments given to male and female residents in an emergency medicine setting, finding that female residents were more likely to be told they lacked confidence with procedural skills (Mamtani et al., 2022). Rojek et al. examined adjectives used in medical students’ clinical evaluations, finding biases related to both students’ gender and under-represented minority (URM) status (Rojek et al., 2019). Similar biases in feedback content and sentiment related to trainee gender have either been suggested (but found lacked statistical significance) (Minter et al., 2005) or found to be unlikely (Andrews et al., 2021).

While qualitative studies led to retrospective insights about supervisor and trainee viewpoints of entrustment, and quantitative studies found conflicting results on bias in different settings, systematic analysis of a large dataset of entrustment-associated narratives may allow us to perform both analyses simultaneously and assess how they interact. A large dataset taken from an institution-wide experience over several years may thus allow for identification of systematic differences in trainee and supervisor viewpoints, and examination of potential biases represented in the narratives. However, performing consistent text analysis of this nature across a large dataset would be difficult to do via manual coding, and may be prone to bias of the coders themselves. Recently, the development of large language models (LLMs) has facilitated innovations in natural language processing (NLP) and artificial intelligence (AI) that elevate the ability of NLP to characterize themes and emotions (Alaparthi & Mishra, 2021; Boscardin et al., 2023; Zhang et al., 2023). LLMs underly generative AI applications such as ChatGPT and Bard. LLMs are implemented via artificial neural networks that have been trained to represent language probabilistically, considering interrelationships of words in the context of sentences, paragraphs, bodies of text, and entire corpora. When applied to narrative excerpts, they can be used for many NLP applications, including: representing meaning numerically (i.e. via embeddings), producing specific output (i.e. measuring sentiment), and generating new next based on specific prompts (i.e. chatbots).

In this study we developed and utilized NLP tools based on LLMs to systematically compare the thematic content and sentiment of feedback narratives about observed clinical encounters written either by supervisors or trainees. We developed a gender-neutral sentiment analysis strategy to mitigate algorithmic bias. By examining how supervisors and trainees documented such feedback dialogs on an institution-wide scale over two years, we quantitatively compared how trainees and supervisors differentially: 1) prioritized factors tied to entrustment ratings, 2) used sentiment to convey trust, and 3) reflected potential sources of bias (based on students’ gender identity and UIM status).

Positionality and overview of AI-assisted document analysis

We approached feedback dialogs from a social constructivist perspective, considering them to be co-constructed through the interaction of trainees and their supervisors (Dudek et al., 2019; Telio et al., 2016). Such interactions included both the clinical observation, the clinical performance of the trainee, any corrective or reinforcing actions taken by the supervisor, and the feedback dialog occurring afterwards. While both parties participated in each of these steps, we hypothesized that documentation of these interactions would reveal differences in the perspectives of trainees and supervisors when written separately by either participant.

Thus, we adopted a document analysis approach to evaluate these feedback narratives (Cleland et al., 2023). Our analysis focused on the content (factors related to entrustment ratings), linguistic character (sentiment), and latent content (the trainee’s implied acceptance or rejection of feedback, and evidence of influence from sources of bias) of the documents.

We developed AI strategies as extensions of analytic procedures we would have performed manually using a reflexive thematic analysis approach (Braun & Clarke, 2021), had the dataset been orders of magnitude smaller.

For theme (factor) extraction

We developed a transfer learning[1] AI approach based on a previously-trained LLM to abstract the thematic content of each individual narrative numerically, and used principal component analysis (PCA) to segregate the space of relevant content into its most prominent factors. During this process, we employed a panel of expert coders to iteratively define and refine themes these factors represented, as would be done in traditional qualitative coding.

For sentiment analysis

We trained an LLM to classify narratives by their probability of having a negative or positive emotional valence.

For investigation of bias

To look for evidence of bias in the narrative dataset, we first had to assess the LLM’s own bias towards gender, and developed a strategy to mitigate this bias by removing gender-associated pronouns and nouns from both training, validation, and study datasets.

For statistical analysis

We performed our final statistical analysis using standard multilevel modeling to account for the nested structure of the data (multiple entries associated with each individual student), including investigation of important confounders and covariates.

We refined our data analysis, emerging theories, and algorithms iteratively during the process of data collection, but we did not alter the instrument or scope of data collection during the study. We will make our data analysis algorithms available as python scripts in a publicly accessible online repository.

Data, participants, and setting

The data consisted of feedback narratives generated from N = 24187 discrete feedback dialogs between 552 clerkship-year medical student trainees and their direct clinical supervisors (4926) that occurred following observed clinical tasks (physical exams, history-taking, procedures, note-writing, communications, oral presentations, and other) (Table 1). The narrative from each feedback dialog was documented either by the trainee or supervisor, but not by both. The documentation also included a rating (from 1–4, see Supplemental Table 1) of the level of supervision provided for the clinical encounter (based on the Modified O-SCORE scale), which we refer to here as the entrustment rating (ten Cate et al., 2020). While both the feedback narrative and entrustment rating prompts were adjacent to each other on the assessment instrument, the narrative prompt did not explicitly require elaboration on why a particular entrustment rating was chosen. The data were collected over two calendar years (January 2020 to December 2021) from all clerkship-year medical students in all required clinical clerkships at a 4-year post-baccalaureate medical school in the United States. Students were asked to complete two such observations weekly during their required clerkships (which included: pediatrics, internal medicine, OB/GYN, neurology/psychiatry, surgery, family and community medicine, and anesthesia). Identities of both students and supervisors were masked and replaced with random tokens.

Of note, the majority of feedback dialogs were documented by trainees (20535) while supervisors documented comparatively fewer (3652). To accommodate for this asymmetry in the data, as well the absence of simultaneous documentation by a trainee and supervisor of the same feedback dialog, we sought not to make comparisons of viewpoints from each dialog, but of dialogs representing similar task types in similar contexts. Thus, our multilevel statistical model included the task-related and contextual variables in Table 1 as covariates, and used student and supervisor identities to define clusters (described below).

AI algorithms: language models to identify and measure entrustment-related factors and sentiment

NLP-assisted theme (factor) extraction. We designed an NLP strategy to broadly characterize the set of themes (factors) in the overall narrative dataset (without considering which factors may or may not relate to entrustment), and then measured how strongly each factor was reflected in each feedback narrative. To characterize the factors, we utilized the Universal Sentence Encoder (USE) by Google, Inc. to represent the thematic content of each narrative numerically (Cer et al., 2018). The USE is a language model designed to compare the meaning of sentences and paragraphs by encoding them as a vector embedding – a 512-dimensional vector whose dimensions abstractly represent semantic meaning. We applied the USE to each feedback narrative in our dataset, generating a vector embedding representing each narrative. We then applied PCA to the set of standardized[2] vector embeddings generated by the USE from all narratives in our dataset to identify the strongest factors contributing to our dataset’s overall thematic variance (Joliffe & Morgan, 1992). We retained the first 17 principal components (factors), which represented 33% of the overall thematic variance.

As in typical factor analysis, the themes represented by the principal components need to be identified based on the subsets of narratives associated with each component. To accomplish this, we empaneled a group of human coders with backgrounds in medical education (authors BG, CB, PO’S, and OtC) to perform thematic analysis on each of the 17 principal components. Mirroring the PCA coding procedure we developed in our prior work (Gin et al., 2022), we identified the subsets of narratives that most strongly projected onto both directions of every principal component (i.e. the 99th and 1st percentiles), and found that they did indeed represent coherent themes. Thus, we were able to perform standard reflexive thematic analysis to code each subset of narratives separately (i.e. two subsets for each principal component). Each subset was coded independently by at least two coders, and we iterated until we reached consensus on the themes represented by the two directions of each principal component (Supplemental Table 2). Once the themes were verified, we conducted regression analysis to assess the correlation of these themes with entrustment rating (discussed below).

Thematic reflexivity and NLP algorithmic considerations. Our overall approach to coding could thus be viewed as a hybrid between NLP-assisted factor analysis combined with human-based coding of those factors. The reflexivity and positionality considerations discussed above are thus reflected in our coding of each factor (Cambo & Gergle, 2022; Gin, 2023). “Algorithmic reflexivity” would be represented here by the choice of factors representing the dataset. For example, use of a different algorithm than the USE (for instance, Facebook’s LASER, or a fine-tuned version of BERT for semantic similarity) may yield different factors. We chose to employ PCA here instead of other clustering techniques (such as gaussian clustering or HDBSCAN) after first testing those other algorithms (Malzer & Baum, 2019). The PCA analysis had an advantage over other techniques in producing coherent themes consistently. Finally, we opted not to use newer LLMs here such as GPT-4, LLaMA-65b, or Falcon-40b (which were available at time of writing) because even the largest of these models was restricted by a token limit (2¹⁵ tokens, or about 25,000 words for GPT-4) that would prevent it from considering the dataset in its entirety, as we could do with the stepwise USE-PCA strategy outlined here. Furthermore, use of shared cloud computing resources would have violated our institutional security policy on use of sensitive data. Conversely, we did not use more traditional NLP methodologies such as TF-IDF or other bag-of-words based techniques, since these strategies are based on word frequency only without consideration for the meaning of patterns or sequences of words (Agarwal & Nayak, 2020). The transformer-based artificial neural networks underlying the USE have the ability to represent meaning abstractly and form the basis of modern LLMs.

Approach to gender-neutral sentiment analysis. Sentiment analysis is the practice of assigning emotional valence to narrative data and is a field at the intersection of linguistics and machine learning with broad applicability to both academic, and educational applications (Nandwani & Verma, 2021). Sentiments may be as varied as multiple emotional axes or simply as positive versus negative. For our study, the goal of sentiment analysis was the latter – to estimate the probability that each feedback narrative had a positive emotional valence (compared to a negative one).

To perform our sentiment analysis, we started with BERT-PubMed – a specialized version of the LLM, BERT, which was trained on text from MEDLINE/PubMed (Devlin et al., 2019). We placed BERT-PubMed as the encoding layer in a LLM classifier whose output was a numerical probability (from 0-100%) that its input text reflected a positive versus negative sentiment of the writer. We then trained this sentiment classifier to predict sentiment using the Stanford Sentiment Treebank (SST-2) a collection of 11,855 sentences extracted from movie reviews annotated by human judges (Socher et al., 2013), or the Large Movie Review Dataset 1.0 (LMRD) a collection of 50,000 individually labeled (as positive or negative) movie reviews selected for their polarity (Maas et al., 2011). The annotated python source code used for training (including LLM training details) will be made available in an online repository. After training the LLM, we applied it to each narrative in our dataset, generating a probability for each narrative. We found that regardless of which training dataset we used (either the SST-2 or LMRD), there was a substantial negative bias in the sentiment probabilities when the pronouns were female (approximately − 5% probability of being positive) or gender-neutral (approximately − 10%), compared to male.

Mitigating algorithmic bias. In order to mitigate this apparent gender bias in our sentiment classifier[3], we replaced all gendered pronouns with their gender-neutral equivalents in the LMRD training dataset, and re-trained the LLM (we opted to train using only this modified LMRD dataset for the final analysis, since it represented a larger collection of narratives). We also replaced all gendered pronouns from our narrative dataset before analyzing it (Bhardwaj et al., 2021). We then applied the trained LLM to each gender-neutral feedback narrative in our dataset (excluding other variables such as entrustment rating or demographics), generating a probability for each narrative of its positive sentiment (thus, a probability 100% represents positive sentiment and 0% represents negative sentiment, while 50% represents neutral sentiment).

Computation, Data security, and Institutional Review Board procedures. We performed all AI modeling using TensorFlow 2.10 in Python 3.9 (Abadi et al., 2015), with all computation running entirely locally on a secured entry-level consumer computer with a discrete graphics processing unit which did not have any specialized capability. We performed all statistical analysis locally as well, using Stata 17.0 (Rabe-Hesketh & Skrondal, 2012). Thus, we maintained security of the data without exposing it to any cloud/online or shared computing resources. Our Institutional Review Board approved the study protocol (study ID 20-32478).

Statistical analysis

Examining the relationship between feedback themes (factors) and entrustment. In order to determine how strongly each factor related to the assignment of entrustment ratings, we performed a multilevel multivariable ordinal logistic regression using the entrustment rating as the dependent variable, and the 17 (standardized) PCA-derived components of each feedback narrative as the independent variables (using Stata’s meologit function). We included the following confounders/covariates in the model: clerkship rotation specialty, task type, number of semesters completed the student at the start of their rotations, student gender, student UIM status, sentiment of the narrative (as discussed above), and whether the feedback narrative was documented by the student or their supervisor. The multilevel structure accounted for non-independence (multiple feedback dialogs) of observations related to each student (i.e. feedback dialogs clustered by student), and the identities of the supervisors within each student cluster (e.g. a two-level multilevel model). The magnitude of the logistic regression coefficient between the entrustment rating and each factor’s associated PCA projection thus reflected how strongly that factor independently influenced the entrustment rating (positive coefficients related to the theme coded from the 99th percentile of narratives projecting onto the given principal component, while negative coefficients related to the theme coded from the 1st percentile of narratives) (Gin et al., 2022).

Comparing viewpoints and assessing for bias. To compare student and supervisor viewpoints of the identified factors, we added interaction terms (to the above regression) between the feedback writer (a binary variable indicating if the narrative was documented by the student or supervisor) and each factor’s representative PCA projection (a set of independent, or predictor variables) (Rabe-Hesketh & Skrondal, 2012). The correlation coefficients specific to the set of students and supervisors could thus be identified. Similarly, to assess for bias, we added interaction terms between the feedback writer and the student’s gender identity (male vs female only, as there was not enough data to draw conclusions from observations of students identifying as neither male nor female) and UIM status. To assess for differences between entrustment ratings based on writer role (supervisor vs. trainee), gender, and UIM status, we first conducted multilevel multivariable linear regressions for sentiment and entrustment rating (using the same covariates/confounders as the ordinal logistic regression described above) and then computed the estimated marginal means from the fully fitted models (using the xtmixed and margins functions, respectively). We assessed for statistically significant differences between groups using pairwise comparisons at the p < 0.05 level (using the pwcompare by group method).

We present our findings by our three research questions with our focus on understanding how trainee and supervisor perspectives differ when engaging in feedback about entrustment to perform a clinical task. First, we examine how trainees’ and supervisors’ documentation of the feedback dialog differed with respect to how feedback themes correlated with the entrustment rating, giving insight into factors trainees and supervisors viewed as important for entrustment. Secondly, we examine how the sentiment of the narratives differed between documentation written by supervisors and trainees, giving insight into how the language of feedback may reflect different attitudes of supervisors and trainees towards learning and improvement. Lastly, we examine how potential biases related to trainee gender and UIM may affect both the use of language and the assignment of entrustment levels, comparing documentation by supervisors and trainees to assess whether biases may differentially affect each perspective.

1. Trainees vs Supervisors: Which feedback themes correlate with entrustment ratings?

We identified and ordered feedback themes (factors identified from the PCA analysis) with a statistically significant correlation to the entrustment rating by the strength of that correlation (as measured by the logistic regression coefficient β and odds ratio), doing so separately for supervisors and trainees (Table 2). We included only the factors with coefficients β > 0.10 (i.e. an odds ratio > 1.1) and p-values < 0.02. Each factor in the table thus represents features of a trainee’s clinical performance that are associated with higher levels of entrustment, with the importance of those factors to entrustment reflected in their ordering from top to bottom.

Although supervisors and trainees were equally empowered to document their shared feedback dialog, only two out of the top six factors matched when comparing documentation by supervisors and trainees. Effective patient communications and oral presentations were the two factors that correlated with entrustment ratings in both supervisors’ and trainees’ documentation. In supervisors’ documentation, feedback factors correlating with increased entrustment were heavily dominated by the oral presentation’s structure, organization, length, and inclusion of relevant detail. In trainees’ documentation, feedback associated with higher entrustment appeared to encompass a wider variety of factors, including general praise, asking appropriate questions, physical exam skills, and suggestions for improving clinical reasoning.

2. Trainees vs Supervisors: How does the sentiment of feedback documentation differ?

Compared to supervisors, trainees tended to document feedback dialogs with language that utilized comparatively negative sentiment (-5.3%±1.4% probability, based on a 95% confidence interval, CI) (distribution shown in Fig. 1). Direct examination of the feedback narratives revealed that sentiment was often used by supervisors to balance constructive feedback with praise (i.e. the proverbial “feedback sandwich”) (Parkes et al., 2013), while trainees appeared to focus on the constructive feedback more directly. The LLM tended to associate praise with positive sentiment, while constructive comments tended to be measured as negative. For example, the following narrative from a supervisor documented constructive feedback between phrases of praise:

Good job doing a comprehensive history and physical examination on a patient with exacerbation of congestive heart failure. I was impressed with the level of detail and the thoroughness of the presentation. As we discussed, I would start to think about what information needs to be a part of an oral presentation, versus what important information can simply be recorded in your written note for reference. This will help to make presentations more concise and easier to follow. Great start!

In comparison, a trainee’s documentation of the same skill was more direct:

Overall, ***'s presentation of their patient during our coach-led preceptorship was comprehensive. Some points to work on: 1) non-pertinent information in HPI can be moved to ROS to streamline narrative and 2) organize medication list by diagnosis (will be helpful to listener and help them follow along).

We found the opposite trend when comparing average entrustment ratings between supervisors and trainees. Trainees appeared to recall a significantly higher level of trust (and lower level of supervision) when documenting feedback (0.08 ± 0.04 on a 1–4 entrustment scale) than supervisors did, which is consistent with results of other studies (Marty et al., 2021; Sterkenburg et al., 2010).

3. How do gender and UIM status influence sentiment, entrustment ratings, and feedback themes (factors)?

We found that potential sources of bias (related to trainee gender and UIM status) appeared to affect sentiment (Table 3). We found five differences when examining sentiment written by supervisors versus trainees: On average, feedback written about male trainees tended to employ higher (more positive) sentiment than feedback about female trainees (1.3%±1.2%, p < 0.05). However, there was no significant difference in entrustment rating between female and male trainees (in fact, the average entrustment rating was identical). We examined whether supervisors or trainees were more prone to gender bias by looking for interactions between writer and gender variables. In the subset of narratives written by supervisors, there was no significant gender difference in either sentiment or entrustment level (i.e. its confidence interval of 1.4%±2.0% contained zero), while in the subset written by trainees, we found a significant gender difference in sentiment (1.3%±1.0%, p < 0.05), but not in entrustment ratings.

Finally, we compared the mean sentiment and entrustment ratings in the groups defined by trainee UIM status. We found that sentiment was more positive for UIM trainees (1.2%±1.1%, p < 0.05). However, this trend was only significant in feedback documented by trainees (1.3%±1.2%, p < 0.05), not in feedback documented by supervisors. We found no significant differences in mean entrustment ratings related to trainee UIM status.

Given the evidence of potential bias related to gender and trainee UIM status, we examined whether feedback themes (factors) may also be influenced by gender and UIM status (Table 4). Again ordering these factors by their correlation with entrustment ratings, we found that four out of the top six factors were the identical when comparing feedback written about male and female trainees. Of the four factors differing between groups, effective communications and presentations with relevant details were associated with higher trust in feedback written about male trainees, while properly sectioned presentations and comprehensive/relevant physical exams were associated with higher trust in feedback written about female students. We did not find significant differences in feedback factors partitioned by trainee UIM status.

By employing an NLP strategy to perform thematic factor and sentiment analyses across a large dataset of documented feedback dialogs, we were able to identify trends in supervisor and trainee documentation that suggest differences in their perspectives of entrustment. Further, our findings suggest that potential sources of bias derived from trainee gender identity and UIM status appear to affect the sentiment of documentation more so than the assignment of entrustment ratings. The ability to detect these small but statistically significant differences relied on consistent interpretation of narratives across a large dataset, for which we relied on LLM-based algorithms to augment the ability of human coders. Additionally, detection of small but significant biases in the sentiment writers expressed depended on mitigation of algorithmic bias, which would have masked the biases we wanted to investigate. While further investigation will be needed to assess the generalizability of our findings, the methods we developed here could be utilized to quantitatively assess qualitative features of other large narrative datasets.

Using a transfer learning strategy to augment qualitative coding, we were able to investigate factors tied to entrustment ratings in feedback narratives documented by supervisors and trainees, giving insight into their thinking around entrustment decisions. Both supervisor and trainee perspectives emphasized the importance of delivering effective oral presentations in determining entrustment. Both perspectives documented narratives that linked not only reinforcing feedback to entrustment, but also constructive feedback. Constructive comments about improving the HPI and clinical reasoning were also correlated positively with the entrustment rating. This finding suggests that open dialog may have been more important to building trust than the particular level of competence the trainee may have displayed (Castanelli et al., 2022; Telio et al., 2015). In terms of differences, while supervisors’ documentation tended to focus on presentations, trainees’ documentation expanded upon a broader scope of clinical skills and personal qualities. While this discrepancy may reflect that patient presentations were supervisors’ primary interactions with trainees – i.e. that patient presentations acted as the “signature pedagogy” (Gardner & Shulman, 2005) – it also may tie to trainees’ developing comfort within their roles in the clinical learning environment (Gruppen et al., 2019; O’Brien et al., 2007). Supervisors may have to make entrustment decisions primarily based on trainees’ presentations, but trainees need to consider a broader range of skills to be effective clinicians. Given potentially limited interactions between supervisors and trainees apart from presentations, trainees may have had to rely on their self-perceptions to assess their broader skillset.

These trainee self-perceptions appeared to have been subject to significant biases, reflected in the language (sentiment) trainees used to document their own feedback. Our sentiment analysis suggested that female trainees appeared to use a more negative tone when documenting their own feedback, than their male counterparts. UIM students appeared to use a more positive tone when documenting feedback. These differences in tone may reflect trainees’ self-perceptions of efficacy (Nomura et al., 2010; Sagasser et al., 2017). For comparison, several studies have found conflicting evidence of gender bias in feedback, as related to sentiment, assessment ratings, or clinical content. With respect to sentiment, Andrews, et al. utilized NLP to analyze narratives in assessments of internal medicine residents, finding no significant difference between male and female subgroups (Andrews et al., 2021). While their results may have been limited by effect strength and sample size (N = 3864 total evaluations) compared to our larger dataset, our sentiment model may also have been more sensitive to potential nuances in language, being based on more sophisticated LLMs than word frequency techniques. In our study, neither finding (of gender or UIM-related bias) was replicated in supervisors’ documentation, which may reflect an institutional culture promoting faculty consideration of diversity, equity, and inclusion (DEI) at the site we studied (Lucey et al., 2020; Teherani et al., 2020). For example, Sarraf, et al. at their institution found significant gender bias in letters of recommendation written for general surgery residency candidates; their sample included letters from decades before significant DEI efforts are likely to have been made (Sarraf et al., 2021).

We found no gender bias in entrustment level, neither in documentation written by supervisors nor by trainees. This contradicts several studies performed in different settings. Minter, et al. found that female surgical residents may underestimate their performance when completing self-evaluations, but this finding was not statistically significant (Minter et al., 2005). Padilla, et al. found that female surgical residents’ self-assigned EPA ratings were lower than those of their male counterparts, but did not find gender bias in EPA ratings they received from their supervisors (Padilla et al., 2022). Dayal, et al. found that male emergency medicine residents were granted milestone attainment at a faster rate than their female counterparts (Dayal et al., 2017). Mamtani, et al. similarly found that male emergency medicine residents were more likely to receive feedback indicating they were above the expected level of competency than female residents (Mamtani et al., 2022). While direct comparison of trainees in our study (medical students in their clerkship year) and these examples (residents in emergency medicine) may not be indicated, it is interesting to speculate if entrustment ratings may be less prone to bias than ratings based on milestones or competencies.

Apart from sentiment and ratings, several studies have also examined whether the thematic content of feedback can be biased by trainee gender. Mamtani, et al. compared themes in feedback given to male and female residents in emergency medicine, finding multiple disparities in the frequency with which these themes were found (Mamtani et al., 2022). Female trainees were more likely to receive feedback related to their procedural confidence rather than competence. Their study examined the frequency of themes found in feedback, but did not consider how those themes were tied to a quantitative performance metric such as the entrustment rating. Here, we have considered not only the frequency of themes (factors), but also the degree to which they are correlated with the entrustment rating. This important difference in methodology is specific to our research question of identifying factors related to entrustment, rather than the general scope of feedback. With this consideration in mind, our results revealed that documentation of feedback about male and female trainees was more similar than it was different, which mirrors the results found by Andrews, et al. (of feedback topics describing internal medicine residents) (Andrews et al., 2021). We found that feedback about male and female trainees emphasized both performance competencies (i.e. qualities of the presentation, physical exam, and communications) as well as personal characteristics (i.e. asking appropriate questions). The main gender discrepancy we found was that feedback about male trainees included prioritization of communications skills, whereas for female trainees the physical exam was emphasized. Further investigation is needed to understand the significance of this difference. For comparison, Mamtani, et al. similarly found that their male subgroup received feedback about communication skills with a higher frequency.

This study has limitations. As previously discussed, there was asymmetry in the number of feedback dialogs documented by trainees compared to supervisors. Rather than comparing trainee and supervisor documentation of the same feedback dialog, we made comparisons of viewpoints of similar types of clinical tasks in similar contexts by designing a multilevel model accounting for these variables. This strategy does not account for the possibility that there was selection bias as to whether the trainee or supervisor documented the feedback (i.e. supervisors may be more likely to document feedback if it is positive). However, the entrustment ratings’ apparent invariance to the potential sources of bias we investigated suggests that the effect of such a selection bias on entrustment ratings could also be small. Further, the generalizability of our data remains to be investigated, since we focused on a single institution with medical students as trainees. A study during residency may emphasize different factors as important for entrustment since the opportunities for trainee independence would be greater and the risk of harm to patients from inappropriate entrustment would potentially be higher. The NLP techniques we developed here could be readily applied to feedback obtained in such a setting. Finally, we note a methodological limitation that involves the scope of the knowledge transferred from the trained LLMs we used. All LLMs transfer inferences from training sets to the analysis of study data; as such, they are susceptible to biases and lack of generalizability. We provided strategies here for mitigating such biases (i.e. the gender-neutral LLM design), and for improving generalizability: utilizing a panel of human coders to assess themes/factors derived from the LLM, and combining a movie-review based sentiment training set with an LLM trained on PubMed.

We conclude that while bias persists in workplace-based assessment, it appears to affect trainee self-perceptions more than the degree of entrustment they experience. Trainees considered a broader range of factors when rationalizing the level of entrustment they required, compared to supervisors who focused on trainees’ oral presentations. While these trainee perceptions of entrustment were biased by gender and UIM status, the biases did not make their way into the linked entrustment ratings, even when trainees documented those ratings. While the lack of bias in entrustment ratings may be due in part to DEI-related teaching interventions aimed at clinical supervisors, it is also somewhat reassuring to find that entrustment ratings appeared to be less susceptible. Perhaps, trainees may experience supervision/autonomy separately from their self-perceptions of competency. While entrustment frameworks, AfL, and DEI-related interventions may be improving parity in assessment, there is still much room to improve the inclusiveness of the clinical learning environment – as indicated by persistent biases reflected in trainees’ self-perceptions.

Table 1

**Characteristics of the assessment dataset**, collected from Jan 2020 to Dec 2021 of all clerkship-year medical students across a single US-based medical school.
Total number of students Male gender (%) Female gender (%) Neither male / female gender (%) Not UIM (%) UIM (%) Semesters completed at start of clerkships: 4 or less (%) More than 4 (%)	552 256 (46%) 293 (53%) 3 (0.5%) 356 (64%) 196 (36%) 408 (74%) 144 (26%)
Total number of supervisors	4926
Total number of observations By specialty: Anesthesia Family and Community Medicine Internal Medicine Neurology Obstetrics / Gynecology Pediatrics Psychiatry Surgery Other / Unspecified By task type Communication History Note Taking Oral Presentation Physical Exam Procedure Other	24187 1295 3376 4185 1846 3211 3258 1978 3866 1172 1908 3692 3707 6585 2785 2633 2877
Entrustment-Supervision (ES) level rating distribution (%) 1 – Student required complete guidance or was unprepared 2 – Student was able to perform some tasks but required repeated directions 3 – Student demonstrated some independence and only required intermittent prompting. 4 – Student functioned fairly independently and only needed assistance with nuances or complex situations.	119 (0.5%) 1674 (6.9%) 8694 (35.9%) 13700 (56.6%)
Mean Assessments per student [SD]	43.8 [33.9]
Mean Assessments per supervisor [SD]	4.9 [8.8]
Mean Words per feedback narrative [SD]	37 [21]
Mean Sentences per feedback narrative [SD]	2.3 [1.3]
Mean Entrustment-Supervision (ES) level [SD]	3.1 [0.6]
Number of assessments documented by: Student (%) Supervisor (%)	20535 (84.9%) 3652 (15.1%)

Table 2

**Feedback themes (factors) demonstrating the strongest association with entrustment ratings, documented either by supervisors or trainees**. Association strength is measured as the logistic regression coefficient β (with standard error given in the parenthesis, followed by the odds ratio, OR) for the entrustment rating as predicted by each factor’s standardized PCA components, over narratives in the entire dataset. The shaded cells represent pairs of factors of high relevance to trust for both supervisors and trainees. We included here only the factors with coefficients > 0.10 and p-values (not shown) < 0.02.
Supervisors	Trainees
Oral presentations were concise, thorough, and/or organized β = 0.31 (0.05), OR 1.37	Oral presentations were concise, thorough, and/or organized β = 0.47 (0.03), OR 1.60
Communications with patient were effective β = 0.25 (0.05), OR 1.29	Nonspecific praise β = 0.29 (0.03), OR 1.34
Presentations included relevant details β = 0.14 (0.04), OR 1.16	Communications with patient were effective β = 0.18 (0.03), OR 1.20
Presentations included proper sections β = 0.14 (0.05), OR 1.15	Suggestions for improving clinical reasoning β = 0.14 (0.02), OR 1.16
Assessments were thorough β = 0.13 (0.04), OR 1.14	Asked appropriate questions β = 0.14 (0.02), OR 1.15
Suggestions for improving HPI β = 0.12 (0.05), OR 1.12	Physical exams were comprehensive and relevant β = 0.13 (0.03), OR 1.14

Table 3

**Pairwise comparison of average sentiment probabilities and entrustment rating by group.** Estimated marginal means from the linear multilevel models were applied to different groups (narrative writer, trainee gender, and trainee underrepresented in medicine (UIM) status), with standard errors given in the parentheses. The “#” represents statistical interactions between the indicated groups. Means that are significantly different *within* each paired comparison are marked with a “*” (significance at the p < 0.05 level). Significant differences *between* pairs are not shown.
Group	Sentiment, % probability of positive sentiment (SE)	Entrustment rating, range 1–4 (SE)
Writer: Supervisor (Sup)	81.1% (0.6%) *	3.41 (0.02) *
Writer: Trainee (Tr)	75.8% (0.3%)	3.49 (0.01)
Trainee Gender: Male (M)	76.9% (0.3%) *	3.48 (0.02)
Trainee Gender: Female (F)	75.6% (0.5%)	3.48 (0.02)
Writer#Gender: Sup & M	81.9% (0.7%)	3.39 (0.03)
Writer#Gender: Sup & F	80.5% (0.7%)	3.42 (0.02)
Writer#Gender: Tr & M	76.4% (0.4%) *	3.49 (0.02)
Writer#Gender: Tr & F	75.1% (0.3%)	3.49 (0.02)
Trainee UIM: No	75.8% (0.3%) *	3.47 (0.02)
Trainee UIM: Yes	77.0% (0.5%)	3.50 (0.02)
Writer#UIM: Sup & No	81.0% (0.7%)	3.42 (0.02)
Writer#UIM: Sup & Yes	81.5% (0.8%)	3.38 (0.03)
Writer#UIM: Tr & No	75.4% (0.4%) *	3.48 (0.02)
Writer#UIM: Tr & Yes	76.7% (0.5%)	3.52 (0.02)

Table 4

**Feedback themes (factors) correlated with entrustment ratings by gender** (in narratives written by both supervisors and trainees). As in Table 2, factors in common between groups are shaded.
Male trainees	Female trainees
Oral presentations were concise, thorough, and/or organized β = 0.41 (0.04), OR 1.51	Oral presentations were concise, thorough, and/or organized β = 0.47 (0.03), OR 1.60
Nonspecific praise β = 0.27 (0.04), OR 1.31	Nonspecific praise β = 0.24 (0.04), OR 1.27
Communications with patient were effective β = 0.25 (0.04), OR 1.28	Asked appropriate questions β = 0.15 (0.03), OR 1.15
Suggestions for improving clinical reasoning β = 0.16 (0.03), OR 1.18	Suggestions for improving clinical reasoning β = 0.13 (0.03), OR 1.14
Asked appropriate questions β = 0.14 (0.03), OR 1.15	Physical exams were comprehensive and relevant β = 0.13 (0.03), OR 1.14
Presentations included relevant details β = 0.08 (0.03), OR 1.09	Presentations included proper sections β = 0.12 (0.03), OR 1.12

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jozefowicz, R., Jia, Y., Kaiser, L., Kudlur, M., … Zheng, X. (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. https://www.tensorflow.org/about/bib
Agarwal, B., & Nayak, R. (2020). Deep Learning-Based Approaches for Sentiment Analysis (B. Agarwal, R. Nayak, N. Mittal, & S. Patnaik, Eds.). Springer Singapore. https://doi.org/10.1007/978-981-15-1216-2
Alaparthi, S., & Mishra, M. (2021). BERT: a sentiment analysis odyssey. Journal of Marketing Analytics, 9(2), 118–126. https://doi.org/10.1057/s41270-021-00109-8
Andrews, J., Chartash, D., & Hay, S. (2021). Gender bias in resident evaluations: Natural language processing and competency evaluation. Medical Education, 55(12), 1383–1387. https://doi.org/10.1111/medu.14593
Bhardwaj, R., Majumder, N., & Poria, S. (2021). Investigating Gender Bias in BERT. Cognitive Computation, 13(4), 1008–1018. https://doi.org/10.1007/s12559-021-09881-2
Bonnie, L. H. A., Visser, M. R. M., Kramer, A. W. M., & van Dijk, N. (2020). Insight in the development of the mutual trust relationship between trainers and trainees in a workplace-based postgraduate medical training programme: a focus group study among trainers and trainees of the Dutch general practice training programme. BMJ Open, 10(4), e036593. https://doi.org/10.1136/bmjopen-2019-036593
Boscardin, C. K., Gin, B. C., Black Golde, P., & Hauer, K. E. (2023). ChatGPT and generative artificial intelligence for medical education: potential impact and opportunity. Academic Medicine, in press.
Braun, V., & Clarke, V. (2021). Can I use TA? Should I use TA? Should I not use TA? Comparing reflexive thematic analysis and other pattern‐based qualitative analytic approaches. Counselling and Psychotherapy Research, 21(1), 37–47. https://doi.org/10.1002/capr.12360
Cambo, S. A., & Gergle, D. (2022). Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science. CHI Conference on Human Factors in Computing Systems, 1–19. https://doi.org/10.1145/3491102.3501998
Caro Monroig, A. M., Chen, H. C., Carraccio, C., Richards, B. F., Ten Cate, O., & Balmer, D. F. (2021). Medical Students’ Perspectives on Entrustment Decision Making in an Entrustable Professional Activity Assessment Framework: A Secondary Data Analysis. Academic Medicine, 96(8), 1175–1181. https://doi.org/10.1097/ACM.0000000000003858
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2021). Trust, power and learning in workplace‐based assessment: The trainee perspective. Medical Education, August, 1–12. https://doi.org/10.1111/medu.14631
Castanelli, D. J., Weller, J. M., Molloy, E., & Bearman, M. (2022). How Trainees Come to Trust Supervisors in Workplace-Based Assessment: A Grounded Theory Study. Academic Medicine, 97(5), 704–710. https://doi.org/10.1097/ACM.0000000000004501
Cer, D., Yang, Y., Kong, S. yi, Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Céspedes, M., Yuan, S., Tar, C., Sung, Y. H., Strope, B., & Kurzweil, R. (2018). Universal sentence encoder for English. EMNLP 2018 - Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Proceedings, 169–174. https://doi.org/10.18653/v1/d18-2029
Cleland, J., MacLeod, A., & Ellaway, R. H. (2023). CARDA: Guiding document analyses in health professions education research. Medical Education, 57(5), 406–417. https://doi.org/10.1111/medu.14964
Dayal, A., O’Connor, D. M., Qadri, U., & Arora, V. M. (2017). Comparison of male vs female resident milestone evaluations by faculty during emergency medicine residency training. JAMA Internal Medicine. https://doi.org/10.1001/jamainternmed.2016.9616
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference.
Dijksterhuis, M. G. K., Voorhuis, M., Teunissen, P. W., Schuwirth, L. W. T., Ten Cate, O. T. J., Braat, D. D. M., & Scheele, F. (2009). Assessment of competence and progressive independence in postgraduate clinical training. Medical Education. https://doi.org/10.1111/j.1365-2923.2009.03509.x
Dudek, N., Gofton, W., Rekman, J., & McDougall, A. (2019). Faculty and Resident Perspectives on Using Entrustment Anchors for Workplace-Based Assessment. Journal of Graduate Medical Education, 11(3), 287–294. https://doi.org/10.4300/JGME-D-18-01003.1
Gardner, H., & Shulman, L. S. (2005). The professions in America today: crucial but fragile. Daedalus, 134(3), 13–18. https://doi.org/10.1162/0011526054622132
Gin, B. C. (2023). Evolving natural language processing towards a subjectivist inductive paradigm. Medical Education, 57(5), 384–387. https://doi.org/10.1111/medu.15024
Gin, B. C., Cate, O., O’Sullivan, P. S., Hauer, K. E., & Boscardin, C. (2022). Exploring how feedback reflects entrustment decisions using artificial intelligence. Medical Education, 56(3), 303–311. https://doi.org/10.1111/medu.14696
Gin, B. C., Tsoi, S., Sheu, L., & Hauer, K. E. (2021). How supervisor trust affects early residents’ learning and patient care: A qualitative study. Perspectives on Medical Education, 10(6), 327–333. https://doi.org/10.1007/s40037-021-00674-9
Ginsburg, S., van der Vleuten, C., Eva, K. W., & Lingard, L. (2016). Hedging to save face: A linguistic analysis of written comments on in-training evaluation reports. Advances in Health Sciences Education, 21(1), 175–188. https://doi.org/10.1007/s10459-015-9622-0
Gruppen, L. D., Irby, D. M., Durning, S. J., & Maggio, L. A. (2019). Conceptualizing Learning Environments in the Health Professions. Academic Medicine, 94(7), 969–974. https://doi.org/10.1097/ACM.0000000000002702
Hatala, R., Ginsburg, S., Gauthier, S., Melvin, L., Taylor, D., & Gingerich, A. (2022). Supervising the senior medical resident: Entrusting the role, supporting the tasks. Medical Education, 56(12), 1194–1202. https://doi.org/10.1111/medu.14883
Hauer, K. E., Oza, S. K., Kogan, J. R., Stankiewicz, C. A., Stenfors-Hayes, T., Cate, O. ten, Batt, J., & O’Sullivan, P. S. (2015). How clinical supervisors develop trust in their trainees: a qualitative study. Medical Education, 49(8), 783–795. https://doi.org/10.1111/medu.12745
Hauer, K. E., Park, Y. S., Bullock, J. L., & Tekian, A. (2023). “My Assessments Are Biased!” Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education. Academic Medicine, 98(8S), S16–S27. https://doi.org/10.1097/ACM.0000000000005245
Hauer, K. E., Ten Cate, O., Boscardin, C., Irby, D. M., Iobst, W., & O’Sullivan, P. S. (2014). Understanding trust as an essential element of trainee supervision and learning in the workplace. Advances in Health Sciences Education : Theory and Practice, 19(3), 435–456. https://doi.org/10.1007/s10459-013-9474-4
Holzhausen, Y., Maaz, A., Cianciolo, A. T., ten Cate, O., & Peters, H. (2017). Applying occupational and organizational psychology theory to entrustment decision-making about trainees in health care: a conceptual model. Perspectives on Medical Education, 6(2), 119–126. https://doi.org/10.1007/s40037-017-0336-2
Joliffe, I. T., & Morgan, B. (1992). Principal component analysis and exploratory factor analysis. Statistical Methods in Medical Research. https://doi.org/10.1177/096228029200100105
Karp, N. C., Hauer, K. E., & Sheu, L. (2019). Trusted to Learn: a Qualitative Study of Clerkship Students’ Perspectives on Trust in the Clinical Learning Environment. Journal of General Internal Medicine, 34(5), 662–668. https://doi.org/10.1007/s11606-019-04883-1
Kennedy, T. J. T., Lingard, L., Baker, G. R., Kitchen, L., & Regehr, G. (2007). Clinical oversight: Conceptualizing the relationship between supervision and safety. Journal of General Internal Medicine, 22, 1080–1085. https://doi.org/10.1007/s11606-007-0179-3
Lucey, C. R., Hauer, K. E., Boatright, D., & Fernandez, A. (2020). Medical Education’s Wicked Problem: Achieving Equity in Assessment for Medical Learners. Academic Medicine : Journal of the Association of American Medical Colleges. https://doi.org/10.1097/ACM.0000000000003717
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011). Learning word vectors for sentiment analysis. ACL-HLT 2011 - Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.
Malzer, C., & Baum, M. (2019). A Hybrid Approach To Hierarchical Density-based Cluster Selection. https://doi.org/10.1109/MFI49285.2020.9235263
Mamtani, M., Shofer, F., Scott, K., Kaminstein, D., Eriksen, W., Takacs, M., Hall, A. K., Weiss, A., Walter, L. A., Gallahue, F., Yarris, L., Abbuhl, S. B., & Aysola, J. (2022). Gender Differences in Emergency Medicine Attending Physician Comments to Residents: A Qualitative Analysis. JAMA Network Open, 5(11), e2243134. https://doi.org/10.1001/jamanetworkopen.2022.43134
Marty, A., Frick, S., Bruderer Enzler, H., & Zundel, S. (2021). An analysis of core EPAs reveals a gap between curricular expectations and medical school graduates’ self-perceived level of competence. BMC Medical Education, 21(1), 105. https://doi.org/10.1186/s12909-021-02534-w
Minter, R. M., Gruppen, L. D., Napolitano, K. S., & Gauger, P. G. (2005). Gender differences in the self-assessment of surgical residents. American Journal of Surgery. https://doi.org/10.1016/j.amjsurg.2004.11.035
Mitchell, R. M., Kensler, L., & Tschannen-Moran, M. (2018). Student trust in teachers and student perceptions of safety: Positive predictors of student identification with school. International Journal of Leadership in Education, 21(2), 135–154. https://doi.org/10.1080/2374068X.2016.1157211
Nandwani, P., & Verma, R. (2021). A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1), 81. https://doi.org/10.1007/s13278-021-00776-6
Nelson, K., McQuillan, S., Gingerich, A., & Regehr, G. (2023). Residents as supervisors: How senior residents make ad hoc entrustment decisions. Medical Education, January, 1–9. https://doi.org/10.1111/medu.15017
Nomura, K., Yano, E., & Fukui, T. (2010). Gender Differences in Clinical Confidence: A Nationwide Survey of Resident Physicians in Japan. Academic Medicine, 85(4), 647–653. https://doi.org/10.1097/ACM.0b013e3181d2a796
O’Brien, B., Cooke, M., & Irby, D. M. (2007). Perceptions and attributions of third-year student struggles in clerkships: do students and clerkship directors agree? Academic Medicine : Journal of the Association of American Medical Colleges, 82(10), 970–978. https://doi.org/10.1097/ACM.0b013e31814a4fd5
Padilla, E. P., Stahl, C. C., Jung, S. A., Rosser, A. A., Schwartz, P. B., Aiken, T., Acher, A. W., Abbott, D. E., Greenberg, J. A., & Minter, R. M. (2022). Gender Differences in Entrustable Professional Activity Evaluations of General Surgery Residents. Annals of Surgery, 275(2), 222–229. https://doi.org/10.1097/SLA.0000000000004905
Parkes, J., Abercrombie, S., & McCarty, T. (2013). Feedback sandwiches affect perceptions but not performance. Advances in Health Sciences Education, 18(3), 397–407. https://doi.org/10.1007/s10459-012-9377-9
Pugh, D., & Hatala, R. (2016). Being a good supervisor: It’s all about the relationship. Medical Education, 50(4), 395–397. https://doi.org/10.1111/medu.12952
Rabe-Hesketh, S., & Skrondal, A. (2012). Multilevel and longitudinal modeling using Stata - Volume I: Continious Responses. In Stata Press.
Rojek, A. E., Khanna, R., Yim, J. W. L., Gardner, R., Lisker, S., Hauer, K. E., Lucey, C., & Sarkar, U. (2019). Differences in Narrative Language in Evaluations of Medical Students by Gender and Under-represented Minority Status. Journal of General Internal Medicine, 34(5), 684–691. https://doi.org/10.1007/s11606-019-04889-9
Sagasser, M. H., Kramer, A. W. M., Fluit, C. R. M. G., van Weel, C., & van der Vleuten, C. P. M. (2017). Self-entrustment: how trainees’ self-regulated learning supports participation in the workplace. Advances in Health Sciences Education, 22(4), 931–949. https://doi.org/10.1007/s10459-016-9723-4
Sarraf, D., Vasiliu, V., Imberman, B., & Lindeman, B. (2021). Use of artificial intelligence for gender bias analysis in letters of recommendation for general surgery residency candidates. American Journal of Surgery, 222(6), 1051–1059. https://doi.org/10.1016/j.amjsurg.2021.09.034
Sheu, L., O’Sullivan, P. S., Aagaard, E. M., Tad-Y, D., Harrell, H. E., Kogan, J. R., Nixon, J., Hollander, H., & Hauer, K. E. (2016). How residents develop trust in interns: A multi-institutional mixed-methods study. Academic Medicine, 91(10), 1406–1415. https://doi.org/10.1097/ACM.0000000000001164
Sklar, D. P. (2016). Trust Is a Two-Way Street. Academic Medicine, 91(2), 155–158. https://doi.org/10.1097/ACM.0000000000001046
Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference.
Sterkenburg, A., Barach, P., Kalkman, C., Gielen, M., & ten Cate, O. (2010). When do supervising physicians decide to entrust residents with unsupervised tasks? Academic Medicine : Journal of the Association of American Medical Colleges, 85, 1408–1417. https://doi.org/10.1097/ACM.0b013e3181eab0ec
Sturman, N., Parker, M., & Jorm, C. (2021). Clinical supervision in general practice training: the interweaving of supervisor, trainee and patient entrustment with clinical oversight, patient safety and trainee learning. Advances in Health Sciences Education, 26(1), 297–311. https://doi.org/10.1007/s10459-020-09986-7
Teherani, A., Harleman, E., Hauer, K. E., & Lucey, C. (2020). Toward Creating Equity in Awards Received during Medical School: Strategic Changes at One Institution. Academic Medicine, 95(5), 724–729. https://doi.org/10.1097/ACM.0000000000003219
Telio, S., Ajjawi, R., & Regehr, G. (2015). The “educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Academic Medicine, 90(5), 609–614. https://doi.org/10.1097/ACM.0000000000000560
Telio, S., Regehr, G., & Ajjawi, R. (2016). Feedback and the educational alliance: examining credibility judgements and their consequences. Medical Education, 50(9), 933–942. https://doi.org/10.1111/medu.13063
ten Cate, O., & Chen, H. C. (2020). The ingredients of a rich entrustment decision. Medical Teacher, 42(12), 1413–1420. https://doi.org/10.1080/0142159X.2020.1817348
ten Cate, O., Hart, D., Ankel, F., Busari, J., Englander, R., Glasgow, N., Holmboe, E., Iobst, W., Lovell, E., Snell, L. S., Touchie, C., Van Melle, E., & Wycliffe-Jones, K. (2016). Entrustment Decision Making in Clinical Training. Academic Medicine, 91(2), 191–198. https://doi.org/10.1097/ACM.0000000000001044
ten Cate, O., Schwartz, A., & Chen, H. C. (2020). Assessing Trainees and Making Entrustment Decisions: On the Nature and Use of Entrustment-Supervision Scales. Academic Medicine, 95(11), 1662–1669. https://doi.org/10.1097/ACM.0000000000003427
van de Ridder, J. M. M., Peters, C. M. M., Stokking, K. M., de Ru, J. A., & ten Cate, O. T. J. (2015). Framing of feedback impacts student’s satisfaction, self-efficacy and performance. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-014-9567-8
Zhang, W., Deng, Y., Liu, B., Pan, S. J., & Bing, L. (2023). Sentiment Analysis in the Era of Large Language Models: A Reality Check. http://arxiv.org/abs/2305.15005

In AI/ML terminology, “transfer learning” refers to employing algorithms trained on datasets external to the dataset being analyzed.
Each dimension scaled and centered to have a standard deviation of 1 and mean of 0.
It is likely that the bias in the sentiment prediction derived from gender bias in the training datasets, but it is also possible that the underlying PubMed-trained BERT LLM also contained gender bias.

No competing interests reported.

SupplementaryMaterial.docx

Download PDF

Editorial decision: Major revision
01 Oct, 2023
Reviews received at journal
11 Sep, 2023
Reviewers agreed at journal
18 Aug, 2023
Reviewers invited by journal
18 Aug, 2023
Editor assigned by journal
02 Aug, 2023
Submission checks completed at journal
02 Aug, 2023
First submitted to journal
01 Aug, 2023

You are reading this latest preprint version

Trainee versus supervisor viewpoints of entrustment: using artificial intelligence language models to detect thematic differences and potential biases

Status:

Version 1

Abstract

Figures

Introduction

Methods

Positionality and overview of AI-assisted document analysis

Data, participants, and setting

AI algorithms: language models to identify and measure entrustment-related factors and sentiment

Statistical analysis

Results

1. Trainees vs Supervisors: Which feedback themes correlate with entrustment ratings?

2. Trainees vs Supervisors: How does the sentiment of feedback documentation differ?

3. How do gender and UIM status influence sentiment, entrustment ratings, and feedback themes (factors)?

Discussion

References

Footnotes

Additional Declarations

Supplementary Files

Status:

Version 1