Letters of recommendation continue to be a key component of the modern fellowship application (6). However, this current study of over 700 LORs reveals a significant variability in letter quality. Variability in letter quality appeared dependent on the relationship between the writer and applicant. Not surprisingly, PDs’ letter writing practice is consistent with their recommendations. This is demonstrated by the fact that PD letters had the highest proportion of high-quality letters (10.4% included all 5 components of the 5-point composite endpoint and 94% had 3 or more components), while letters from elective supervisors were of much lower quality, with 0% having all 5 components of the 5-point composite endpoint, and only 11.2% having 3 or more components.
Although there is existing literature examining LORs, the current study adds to this in several respects. Most studies to date have looked at trainees in the United States medical training system (10, 11, 17), while the current study examined letters from across Canada, thus adding the experience of another country and enriching the data pool. This study also highlights the relationship between letter quality and letter writer, leading to opportunity for possible faculty development with a goal of improving letter quality. Lastly, it brings forth the issue of elective rotations, and their role as auditions for position, and whether letters of recommendation should be a part of key outcomes of this effort, particularly in light of the poor quality of the letters that come from the experience.
The letter writer
Program directors practice what they preach. They understand the value of specific components of LORs and include these components with the highest frequency. This indicates that PDs are an untapped resource for providing faculty development to all clinical supervisors who are potential letter writers. In stark contrast to PD letters, LORs from elective rotation supervisors were of the lowest quality. One possible reason for this finding is that a short amount of time spent with an unknown trainee does not provide sufficient data to allow the supervisor to be comfortable in rating the trainee in key domains. As many residents will seek out elective rotations at other sites with the intention of not only introducing themselves, but also to obtain a LOR from this site, the overall poor quality of these letters calls into question at least one element of the perceived value of these rotations.
While letters from elective supervisors represent approximately 15% of overall letters in our sample, their utility appears very limited; a meagre 11.2% met the criteria for a good to high-quality letter by including 3 or more components of the 5-point composite endpoint. Questions raised by this include 1) why would trainees get these letters, and 2) did this study miss some important aspect of the letter that would make letters from electives important? While our data suggest elective letters are not of good quality, and thus seemingly not useful overall, they may be a necessity for a particular institution, regardless of how non-specific the information within them, as suggested by recent papers from the surgery field (18–19). These could be considered letters from an ‘audition elective’- a term used to describe a clinical elective taken by trainees to distinguish themselves from their peers in order to improve their chances of being selected by a residency program (20). Internal medicine residents will spend a great amount of time and expense completing sub-specialty audition electives to improve their chances of success in the subspecialty match, possibly via obtaining a strong reference letter from their elective supervisors. As we have shown the LORs from elective rotations to be of low quality, this calls into question the value of these rotations, given their associated costs. With the current travel restrictions related to the COVID-19 pandemic, completion of these away rotations may be even more difficult. This is consistent with other studies questioning the value of the ‘away rotation’ and that the value of such rotations should not depend solely on the obtaining of a LOR (21).
The current study does have certain limitations. While exposing key differences in letter quality according to the letter writer, further exploration of this data is somewhat limited. To ensure anonymity, the letters underwent redaction prior to release by CaRMS, with demographic data including age, gender, and candidate home institution name being removed. Moreover, we were unable to correlate these findings with subspecialty match or residency performance results, thus limiting us to examining the process rather than outcomes that may be of interest to a PD. We put a lot of weight on the opinions of internal medicine PDs, largely because of their experience in both writing and reading letters of recommendation. However, there are other members of selection committees, whose views and opinions of a quality letter may differ. In consideration of this, our composite endpoint was developed with input from the internal medicine PD survey, but also with input from a panel of academic internal medicine subspecialists who sit on various residency training selection committees. Lastly, the current study only looked at letters for application to the subspecialty internal medicine match and results may not be transferable to programs accepting medical students into their first year of residency training (PGY-1).
Should the Medicine Subspecialty Match (MSM) in Canada consider the adoption of a standardized LOR to reduce the current variable utility of LORs? The Association of Academic Internal Medicine (AAIM) currently has guidelines for a standardized LOR based on the CanMeds roles (22), which do not match the components of the 5-point composite endpoint developed here. Internal medicine program directors in Canada could lead the process in developing an equivalent guideline document for Canadian MSM applicants. We note that standardized LORs are not without concerns as the Best Evidence Medical Education (BEME) has highlighted the potential for grade inflation (23), thus limiting ability to discriminate between applicants. Therefore, although an attractive option to address the issues of LORs in the MSM, we may end up back where we started despite efforts to move forward.
LORs from elective rotations: We recommend strong consideration by PDs, trainees, and letter writers of a new approach to letters- perhaps 1) clear communication with trainees on electives that letters from these rotations are not mandatory for consideration of acceptance into a program, and 2) having a faculty wide approach/position that writers will not provide letters based solely on elective rotations. These two interventions would presumably lead to trainees asking for letters from other sources, likely faculty that know them better and thus can provide a higher quality letter.
Faculty development: Program directors are a potential resource for faculty development and can provide guidance to letter writers based on their experience and expertise in postgraduate medical education. Institutions should approach program directors for faculty development strategies that will lead to improved quality of LORs.
Letter writers: Clinical supervisors who are writing letters for applicants in the Medicine Subspecialty Match can use the components of the 5-point composite endpoint as guide for the inclusion of important components (relationship between the referee and candidate, duration of relationship, inclusion of In Training Evaluation Report (ITER) information, research involvement and comments on areas for improvement). Also, if a supervisor does not feel they can comment on one or more of these components, then it would be acceptable to decline a request for a LOR from an applicant.
Letter readers: Similarly, those reviewing applicants’ portfolios can use the components of the 5-point composite endpoint to judge the quality of the letter.
Applicants: Applicants should be aware of what program directors value in LORs and should direct their requests to supervisors who are appropriate and capable of commenting on the key components.