In this pilot, study we designed and implemented an OSCE and checklist to assess medical resident ability to take a patient-centered obesity-focused history using telehealth. This is the first obesity OSCE to assess resident performance using obesity competencies for medical education published by the Obesity Medicine Education Collaborative via telehalth.11 Given that overall agreement between SPs on our history taking, communication and professionalism checklists were 83.2% (κ = 0.63), 99.5% (κ = 0.72) and 97.8% (κ = 0.44), our checklists are moderately to substantially reliable13 for assessing overall performance, and can be used by other institutions to assess obesity competencies over telehealth. Our results also reveal several gaps in resident obesity-related history taking skills that must be addressed with curricular changes in medical education.
Although previous work has validated OSCEs for medical student and resident clinical skills,14–17 our OSCE is unique in its focus on obesity-related clinical skills via telehealth. Agreement between SPs on our checklists overall was 80-100%, however, agreement remained < 75% on several history taking checklist items following the post-OSCE debrief. After discussing these checklist items with the SPs and re-watching the recorded OSCE sessions, we suspect that this continued disagreement may have been related to discrepancies in SP OSCE performance. Specifically, during several encounters, SPs provided residents with answers to history taking questions without being specifically asked. SPs indicated that this created uncertainty regarding how to rate residents on these items. In our post-hoc analysis we found that agreement tended to be higher during encounters with SP1 who, qualitatively, offered more focused answers to resident questions than did SP2. SP agreement also tended to be worse on subjective items (ex. eliciting the patient’s perspective) compared with objective questions about patterns of weight gain and diet. These findings highlight the importance of pre-OSCE training that targets SP script and assessment interpretation.
For several items on our checklists, percent agreement was high (> 85%) while the kappa statistic was low (≤ 0). We suspect that this paradox may be related to the rare occurrence of a null response on these history taking items given that that the kappa statistic is associated with the prevalence of the finding and may not always be reliable for rare events.13 For the communication and professionalism checklists, these discrepancies may have resulted from a high expected agreement given the weighting of the kappa statistic to allow SP answers to vary by 1 point on the Likert scale.
Despite discrepancies in SP ratings, our history taking assessment revealed consistently poor performance on several checklist items. Residents asked 65% of items on the history taking checklist; less than 20% asked the patient’s highest and lowest weight, and less than 50% asked about beverage consumption, family support and the patient’s perspective on weight gain. These results are consistent with prior studies revealing poor performance on obesity, nutrition and physical activity knowledge assessments among residents.18 Interestingly, resident self-performance ratings tended to be higher than SP ratings. Prior work suggests that clinical self-performance ratings may be related to opinions regarding prior knowledge and abilities,19 highlighting the need for independent raters and further development of OSCEs for clinical assessment. Despite overall higher resident self-assessments, performance remained low overall on the history taking checklist. Given the rising prevalence of obesity in the United States,20 and the importance of taking on obesity-focused history,21 it is essential that physicians be better trained in these skills.
Interventions in medical education have shown promise in improving medical student and resident performance in obesity-related care. Participation in a multi-modal obesity counseling curricula involving case-studies, role-playing and practice with SPs improved the quality of obesity counseling among primary care residents.22 Similarly, medical student involvement in a weight management educational curriculum improved performance on a weight management OSCE and perceived weight management skills.9 More widespread, structured curricular changes are needed to improve obesity counseling among primary care physicians. In addition, since residents with more telehealth experience in our study tended to perform better on our obesity checklists, incorporating telehealth into obesity curricula may further prepare physicians to provide obesity-related care via telehealth.
Strengths of this study include the use of a telehealth platform to conduct the OSCE. The rise in telehealth during the coronavirus pandemic has required that physicians become more facile in delivering medical care over a virtual platform.12 Our OSCE offers an opportunity for residency programs to incorporate telehealth into medical training. In addition, conducting our OSCE over telehealth allowed for compliance with social distancing recommendations and, therefore, the continuation of medical resident education and assessment in the setting of a global pandemic. Furthermore, compared with conventional in-person OSCEs, telehealth assessments are less costly and time-consuming as they reduce travel time, staffing and equipment needs.23 This was also the first OSCE to use the Obesity Medicine Education Collaborative’s obesity related competencies to assess resident skills in obesity care.11
Limitations of this study include a small sample size of resident physicians and use of only two SPs for resident assessment. However, we were able to recruit a diverse sample of residents including up to one-third of PGY2-3s in a residency program during the coronavirus pandemic. In addition, there were discrepancies between our SPs in script interpretation and OSCE performance, which may have contributed to differences in SP checklist assessments. Additional SP training ensuring consistency in SP performance and resident assessment prior to OSCE implementation could improve the reliability of our checklists for future use. Future research is needed to validate our checklists at other residency programs using different SPs.