Impact of peer-delivered tuition/feedback on appointment to United Kingdom Higher Surgical Training

doi:10.21203/rs.3.rs-63538/v1

Download PDF

Research article

Impact of peer-delivered tuition/feedback on appointment to United Kingdom Higher Surgical Training

https://doi.org/10.21203/rs.3.rs-63538/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

In the United Kingdom, Core Surgical Trainees (or equivalent) undergo a rigorous National Selection (NS) process to secure a ST3 Training Post in General or Vascular Surgery. Despite a decrease in competition ratio from 3:1 in 2012 to 1.48:1 in 2017, many applicants were unable to secure ST3 Training Posts. We hypothesise that this is due to applicant performance anxiety rather than lack of clinical experience.

Methods

This is a prospective observational study following 27 delegates who have completed a 2-day course consisting of didactic lecture and interactive practice with a 1:1 faculty:delegate ratio. The course was delivered by surgical consultants and trainees. A specific mentor was nominated to oversee the delegate’s progress throughout and post course. Personalised feedback was given by the entire faculty throughout the course. Delegates were asked to complete evaluation forms to track their perceived confidence levels of success at NS at different timeframes, measured by a visual analogue scale. Paired t -test was performed to compare the mean confidence level pre- and post- course.

Results

Evaluation forms were received from all (n = 27) delegates. Delegate self-reported confidence increased significantly for all domains except Professional Communication. 23 (85%) of the participants were successful in the NS post course. There was no statistically significant difference between UK and non-UK graduates; and first time and not-first-time applicants.

Conclusions

Peer-delivered teaching, practice and feedback as a structured interview practice course can significantly improve applicants’ overall confidence levels in preparing for the UK ST3 General and Vascular Surgery NS. Improvement in confidence levels translates to high attrition rate of ST3 Training Posts. Communication skills require longer practice time extending beyond the duration of the 2-day course.

Educational Philosophy and Theory

General Surgery

UK Higher Surgical Training

interview course

peer-delivered tuition/feedback

mentoring

performance feedback

communication skills

Appointment to higher surgical training in the UK (Specialty Training (ST) from Core Surgical Training is competitive. Since 2011, eligible Core Surgical Trainees (or equivalent) wishing to obtain certification (Certificate of Complete Training (CCT)) in General or Vascular Surgery apply to undergo a 3-hour structured interview at a National Selection Centre. The interview is hosted by Health Education England (HEE) on behalf of the Vascular Surgery and General Surgery Specialty Advisory Committees (SAC) (1). The structure of the ST3 General and Vascular Surgery Interview puts emphasis on 6 different domains: (i) Clinical Scenario, (ii) Clinical Management, (iii) Professional Communication, (iv) Portfolio, (v) Academic Knowledge, (vi) Teaching and Technical Skills. Until 2018 Patient Communication and Leadership Skills were also included. Simulated cases based on each assessed domain are designed to closely mirror real life surgical practice. Candidates who have achieved an overall score which exceeds a defined threshold are deemed appointable. They are matched to training posts within the UK depending on availability. Despite a decrease in ST3 competition ratio from 1.68:1 in 2016 (2) to 1.48:1 in 2017 (3), many applicants were unable to secure a National Training Number (NTN). Whilst such observation may be explained by lack of real life clinical or leadership experience, some candidates were hampered by poor interview technique.

We hypothesised that poor interview techniques and the applicant’s own psychological barriers limit performance on the day of interview. We developed a structured peer-delivered interview preparation course to address these barriers. Peer-delivered learning was chosen as it has been demonstrated to improve confidence and effective (4).

The aim of this study is to evaluate the effectiveness of our course in improving the self-reported confidence of the applicants at the UK ST3 General and Vascular Surgery Interview. The secondary aim is to evaluate the impact of the course on non-UK medical graduates and not-first- time applicants, for whom there is anecdotal evidence of being disadvantaged.

Prospective applicants who intended to apply to the UK ST3 General and Vascular Interview applied voluntarily to attend the paid course between February 2017 and February 2018. The 2-day interview preparation course was designed with a maximum delegate number of ten per course (Fig. 1). It was run over two days in the Clinical Skills Centre, Ninewells Hospital, University of Dundee, United Kingdom. The number of delegates was kept to a maximum of ten which allowed for a 1:1 delegate to tutor ratio. Each delegate was assigned a mentor at the beginning of the course serving as a single point of contact over the entire course. The course faculty consisted of surgical consultants and trainees (ranging from ST3 to ST8), all of whom had successfully passed the same ST3 General and Vascular Surgery Interview as applicants and appointed into training programmes.

The first day began with didactic lectures focusing on each domain within the interview process: Portfolio, Clinical Skills, Clinical Management, Professional Communication skills, Academic, Technical Skills and Teaching. This was followed by moulage which allowed for practice and immediate feedback on performance. The second half of day one involved peer-delivered teaching and practice on selected technical skills as used for the interview process. Delegates worked in pairs, allowing them to observe each other’s performance. In addition to verbal feedback each delegate was scored individually using a structured scoring sheet with scores of 1–5. This score sheet mirrors the one used in the real-life interview.

Day two began with an Objective Structured Clinical Examination (OSCE) simulating the entire interview process. Each delegate rotated into stations representing each assessed domain. Each station was manned by one faculty member, who delivered verbal and written feedback at the end of the station.

Following review and discussion by the faculty, the delegate was individually called into a private meeting with the faculty. Led by the delegate’s mentor, constructive formative feedback was delivered with contributions from the rest of the faculty. Following this a further OSCE was conducted with written scoring and no verbal feedback to simulate the actual interview process. The delegate was given the opportunity to review all collated personalised feedback at the end of the course. Delegates were also encouraged to contact their individual mentors after the course for future support.

A pre-course evaluation form was used to collect delegate demographic data including current level of training, duration of preparation, methods of preparation, number of previous ST3 interview attempts as well as self-evaluation using a visual analog scale (VAS) on perceived confidence level of success in National Selection. VAS was chosen as a self-assessment tool due to its ease of use, reliability and the ability to assess change over time (5). The evaluation consisted of a subjective assessment of the participants overall confidence for performance in the interview as well as confidence at each of the specific components: Portfolio, Clinical Scenario, Management, Professional Communication, Academic, Technical and Teaching Skills. Delegates scored themselves by placing an ‘X’ along the VAS line to denote their perceived level of confidence, ranging from low to high (Fig. 2). Each line measured 10 cm with one end being “no marks/not confident” and the other end, “full marks/very confident”. Data on confidence were gathered using the VAS tool pre- and post course. Data was collated per delegate by measuring in centimeters to one decimal point to assess the level of confidence and compare progression of this parameter through the course. All delegates consented to their data being used in anonymised manner for the purpose of improving the course.

Statistical Analysis

Analysis was performed using the SPSS Statistical program for Mac version 26 (IBM, Chicago, Ill). Normality testing was performed in order to determine data distribution. As data showed a normal distribution. Continuous data is presented using mean and standard deviation and categorical data is presented using %. Paired samples t -test was performed to compare means before and after the course completion. Statistical significance was set at p value of < 0.05.

27 Delegates completed the full 2-day course and submitted evaluation. Demographic data is presented in Table 1. 44% of the course participants were at Core Trainee (CT) level, 30% were at level of Locum in Training (LAT) ST3, with the remaining participants (26%) above ST3 level (including Locum in Service (LAS), ST, clinical fellow posts or others). 16 (59%) participants have not attempted the interview before, 8 (30%) had one previous failed attempt, and 3 people (11%) had more than 1 failed attempts (2–3). Non-UK graduates accounted for 37% (n = 10) of the cohort.

Table 1

Participant Demographics
Participants	N	%
Level of training
CT2	12	44%
LAT ST3	8	30%
above of ST3 equivalent	7	26%
Number of previous attempts
0	16	59%
1	8	30%
> 1	3	11%

Following the course, delegate self-reported confidence increased significantly for the Portfolio station (5.35 ± 2.03 vs 6.60 ± 2.08, p = 0.008). Confidence score improved after completion of both Clinical and Management stations (6.54 ± 1.70 vs 7.43 ± 1.41, p = 0.004, and 6.64 ± 1.48 ± 7.43 ± 1.36, p = 0.012 respectively). Similarly, there were improvements in confidence in the Technical and Teaching Skills station (6.03 ± 2.04 vs 7.24 ± 1.92, p = 0.007) as well as the Academic station (4.93 ± 2.21 ± vs 6.66 ± 1.75, p = 0.004). While there was an overall increase in confidence in the Professional Communication station, this was not statistically significant (p = 0.063). Overall confidence of success at National Selection has increased significantly (4.32 ± 2.05 vs 6.43 ± 1.81, p < 0.001) (Table 2 and Fig. 3).

Table 2

Levels of confidence before and after the course.
Station	Before mean (SD)	After mean (SD	p
Portfolio	5.35 (2.03)	6.60 (2.08)	0.008
Clinical Scenario	6.54 (1.70)	7.43 (1.41)	0.004
Clinical Management	6.64 (1.48)	7.43 (1.36)	0.012
Professional Communication	6.12 (1.75)	7.10 (1.69)	0.063
Academic	4.93 (2.21)	6.66 (1.75)	0.004
Technical Skills & Teaching	6.03 (2.04)	7.24 (1.92)	0.007
Overall Confidence	4.32 (2.05)	6.43 (1.81)	< 0.001

The change in overall confidence did not differ between UK graduates and non- UK graduates (p = 0.538), nor between first time and not-first-time applicants (p = 0.893) (Fig. 4A and Fig. 4B). 23 (85%) of the course participants were successful in obtaining a national training number at the interview following the course. The success rates for non-UK graduates and non-first-time applicants were 70% (7/10) and 67% (8/12) respectively.

In this study, we believed that peer-delivered teaching, practice and feedback translated to increased confidence levels as perceived by the delegates. With the exception of Professional Communication, statistical significance was reached in all other stations. The biggest increase was observed in the Academic Station. Anecdotal feedback from delegates suggested this station represented the greatest psychological barrier due to its perceived complexity and knowledge gap. We have consistently received good qualitative feedback from the course delegates. Improved confidence achieved at the course translated to success at National Selection rate – 85% of course delegates obtained training posts in General or Vascular Surgery in the UK.

The concept for this course resulted from informal ‘interview practice’ sessions with many non-UK graduates and non-first-time applicants. There is a perception amongst these groups that they are disadvantaged by individual factors (6). Non-UK graduates struggle to communicate their knowledge effectively in a time-limited, pressurised environment (7). This is evident in the lower reported confidence scores by non-UK graduates in the Professional Communication domain. Our course achieved a 70% success rate for non-UK graduates. For most non-UK graduates, achieving a NTN provides the most reliable pathway to CCT and independent consultant practice. The current alternative to Specialist Registration (‘Article 14’/CESR) is widely known to be a frustrating process (8).

Not-first-time applicants are hampered by memories of their previous performance failure. They often find it difficult to part with their negative mentality in order to succeed. However, as we found that there was no statistically significant difference between first time and not-first-time applicants, this means that all groups responded well to the course number of applications. Our course achieved a 67% success rate for non-first-time applicants.

We note a decrease confidence was reported by UK graduates and first-time applicants in the Technical and Teaching domain. Under the Technical and Teaching stations, five technical skills were selected based on previous application in real life interview: excision of skin lesion, small bowel resection, basic laparoscopy, arteriotomy and vein patch. Assessment of these skills in the course, as in simulated or real-life training exercises, were carried out using Procedural Based Assessments (PBAs). Despite PBAs being readily accessible by surgical trainees on the Intercollegiate Surgical Curriculum Project (ISCP) portal, very few delegates took the effort to revise PBAs prior to the course. We believed that non-UK graduates and not-first-time applicants reported more experience in carrying out or assisting in these procedures, therefore putting them at a higher baseline for confidence and learning. In contrast, UK-graduates and first-time applicants may have felt overwhelmed and alienated by the knowledge and technical skills required to perform a simulated task in a limited timeframe for preparation.

Our course utilised OSCE style learning with minimal lecture based teaching. OSCEs have been used since the 1970s in medical education to prepare healthcare students for real clinical practice in a safe environment, allow application and reinforcement of theoretical learning and real-time feedback on areas for improvement (9). By emphasising a safe, non-judgemental learning environment, delegates reframe their mindset to overcome their anxiety barrier. This is due to the perception that “the stakes are not as high” in the mock environment (10).

Currently, numerous interview preparation courses of differing formats are available in the UK. Although most courses covered similar content, they lack the practical aspect of OSCE style interviews as well as one-to-one peer mentoring during and after the course. As these teaching methods are resource intensive, most of these courses run in a pure didactic lecture format. Such passive format fails to assess the candidates understanding with little personalized feedback. In addition, most if not all other courses are run by Consultants who have not experienced the National Selection Interview as applicants. Delegates therefore experience less empathy. In contrast, this course provides immediate verbal feedback after the OSCE style interview on the first day as well as written feedback throughout both days about the candidates’ performance. Faculty feedback on Day 2 to individual delegates ensures the process is holistic and unbiased. Delegates are given opportunities to reflect on their strengths and weakness throughout the course.

We did not demonstrate an improvement in confidence reaching statistical difference in Professional Communication. We believed that communication skills require longer practice time extending beyond the duration of the 2-day course. Some advocate that communication skills should be taught formally especially to healthcare professionals (11). In the future, we plan to expand the communication element in the course through OSCE style learning.

Limitations of our study include small number of cohort of 27. Confidence scores were only self-reported, with no objective scoring to correlate with the observed positive trend in confidence levels. The delegates were a self-selected group, suggesting a degree of insight and motivation in their preparation approach. Fatigue experiences by delegates during the 2-day course may also influence their self-reported scores. Finally, this study only assessed confidence which is subjective. Therefore, an improvement to the methodology would be to compare objective performance score provided by the interviewer in each station to the delegates confidence scores.

In the near future, we expect a move to a digital platform due to new social distancing measures resulting from the COVID-19 pandemic. This will significantly alter the way delegates experience the course.

Our study concluded that peer-delivered teaching, practice and feedback delivered as a structured interview practice course can significantly improve the applicants’ overall confidence levels in preparing for the UK ST3 General and Vascular Surgery National Selection. Improvement in confidence levels translates to high attrition rate of ST3 Training Posts. Communication skills require longer practice time extending beyond the duration of the 2-day course.

(2183 words)

ST: Specialty Training; CCT:Certificate of Complete Training; HEE:Health Education England; SAC:Surgery Specialty Advisory Committees; NTN:National Training Number

Acknowledgements

We would like to thank the Dundee Institute for Healthcare Simulation team and the course faculty members for their help in running the course. We would like to thank all of the course delegates who participated in this course for allowing us to use these data.

Authors’ contributions

WLC has provided substantial contributions to the conception and design of the work; and the acquisition, analysis, and interpretation of data for the work; drafted the work and revising it critically for important intellectual content; involved in the final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
MJ has provided substantial contributions to the conception and design of the work; and the acquisition, analysis, and interpretation of data for the work; drafted the work and revising it critically for important intellectual content; involved in the final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
MM has provided substantial contributions to the conception and design of the work; and the acquisition, analysis, and interpretation of data for the work; drafted the work and revising it critically for important intellectual content; involved in the final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
BT has provided substantial contributions to the conception and design of the work; and the acquisition, analysis, and interpretation of data for the work; drafted the work and revising it critically for important intellectual content; involved in the final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
BI has provided substantial contributions to the conception and design of the work; and the acquisition, analysis, and interpretation of data for the work; drafted the work and revising it critically for important intellectual content; involved in the final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved

Funding

There are no sources of funding required for this publication.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

Ethical approval for the delegates to participate in this study was obtained from the University Research Ethics Committee of the University of Dundee (Ref 15,240). It was advised by the ethic committee that the delegates should be informed clearly that participation in the study was purely voluntary. Verbal consent was given by the delegates to indicate that they fully understood the nature of this study and agreed to participate in this study.

Consent for publication

This manuscript does not contain any individual person’s data. Participation in this study included a statement of intent to publish.

Competing interests

There are no competing interests.

NHS Health Education England. Available from: https://specialtytraining.hee.nhs.uk [Accessed 14^th June 2020]
NHS Health Education England. Available from: https://specialtytraining.hee.nhs.uk/Portals/1/Content/Resource%20Bank/Competition%20Ratio%27s/Competition%20Ratios%202016.pdf [Accessed 14^th June 2020]
NHS Health Education England. Available from: https://specialtytraining.hee.nhs.uk/portals/1/Content/Resource%20Bank/Competition%20Ratio%27s/Competition%20Ratios%202017%20ST3.pdf [Accessed 14^th June 2020]
Siddiqui S, Siddiqui S, Mustafa Q, Rizvi AF, Hossain IT. The benefits of a peer-assisted mock PACES. Clin Teach. 2018; 15(3): 221‐ Available from: doi.org/10.1111/tct.12658
Sung, Y, Wu, J. The Visual Analogue Scale for Rating, Ranking and Paired-Comparison (VAS-RRP): A new technique for psychological measurement. Behav Res. 2018;50, 1694–171. Available from: doi.org/10.3758/s13428-018-1041-8
Pattinson J, Blow C, Sinha B, et al. Exploring reasons for differences in performance between UK and international medical graduates in the Membership of the Royal College of General Practitioners Applied Knowledge Test: a cognitive interview study. BMJ Open.2019; 9: Available from: doi.org/10.1136/bmjopen-2019-030341
Pilotto LS, Duncan GF, Anderson-Wurf J. Issues for clinicians training international medical graduates: a systematic review. Med J Aust. 2007; 187(4): 225-228. Available from:
White C. Many CESR and CEGPR decisions take longer than the statutory three months. BMJ 2012; 345: e5289
Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse Educ Today. 2007; 27(5): 481‐ Available from: doi.org/10.1016/j.nedt.2006.08.009
Brown, J. Preparation for objective structured clinical examination: a student perspective.J of Perioper Pract. 2019; 29(6), 179- 184. Available from: org/10.1177/1750458918808985
Ranjan P, Kumari A, & Chakrawarty A. How can doctors improve their communication skills? J of Clin Diagn Res. 2015; 9(3), JE01-4. Available from: doi.org/10.7860/JCDR/2015/12072.5712

Download PDF

Version 1

posted

You are reading this latest preprint version

Impact of peer-delivered tuition/feedback on appointment to United Kingdom Higher Surgical Training

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Statistical Analysis

Results

Discussion

Conclusion

Abbreviations

Declarations

References

Status:

Version 1