In the beginning the application was introduced at the department of orthopedics and traumatology as well as the department of general and visceral surgery of the Cantonal Hospital of Lucerne. 32 residents were registered as trainees, 33 consultants and senior consultants as educators. During the trial period between August 2019 and April 2020 a total of 510 App based assessments were generated, which is about 60 per month. Out of the pre-defined 40 relevant EPAs, 36 were assessed.
In comparison there were significantly less paper-based competency-based assessments done in the same number of months before the launch of the App. Since not all institutions keep statistics on what paper-based assessments were done, the exact number is hard to tell. Generally, there are an estimated 160 paper-based assessments done over the same amount of time. It is very likely that the minimum number of assessments required per year – in Switzerland 4 - is not met by most institutions. According to international consensus meetings, it is beneficial to have short assessments more frequent, instead of longer assessments at irregular intervals 17. Thus, the Surg-prEPAred App is an easily applicable and efficient tool for assessing trainees in the actual clinical setting.
Analysis of user-friendliness through pre- and post-trial questionnaires
Before the initiation of the App and after the 9-month trial period trainees and supervisors where both sent questionnaires to evaluate the user friendliness and effectiveness of the App. A total number of 15 trainees and 16 supervisors returned the questionnaires with the following results:
Trainees (n=15):
93% want to use the App often
93% say, the App is easy to use
100% claim they learned quickly how to use it
50% think assessments by App are better than the old paper-based assessments with
50% being neutral towards both
93% think the usability of the App is good to excellent
Out of those trainees that did not use the App, 29% claimed to have been too busy otherwise, 12% had second thoughts about data security, 29% were on an external rotation, and 18% claimed to not have the technical equipment to do so.
The free text fields mainly proved that excitement and approval for the App were dominant attitudes, that more detailed feedback is important and that a culture change for feedback conversations is imminent.
Supervisors (n=16):
93% want to use the App often
93% say, the App is easy to use
87% claim they learned quickly how to use it
66% think assessments by App are better than the old paper-based assessments with
27% being neutral towards both
93% think the usability of the App is good to excellent
Out of those supervisors that did not use the App, one person claimed to have been too busy to do so, another said no trainee asked to be assessed and a third person is a supervisor in the orthopedic department that has not yet been integrated into the project.
The free text fields showed constructive feedback such as that feedback should be made obligatory for each EPA assessment and some supervisors showed second thoughts concerning the low complexity of the assessments and wondered if they are detailed enough to decide about competence levels and necessary level of supervision.
Effect on feedback quality
Pre- and post-trial questionnaires showed an increase in feedback given in addition to the assessment by 20% with immediate feedback being given 20% more often than before. The specificity of the feedback increased by 11% and 31% agreed that the feedback includes an action plan of how to improve in the future. It is usually the supervisors that dominate the feedback rounds. The quality of the feedback increased by 24% towards high quality feedback. 17% of the trainees were more satisfied with the feedback given at the end of the 9-month trial period than before usage of the App.
Other key figures evaluated
Duration of rating
The average duration per rating with regard to complexity of the EPA and level of supervision without feedback took 4minutes and 19 seconds for trainees and 1minute and 42 seconds for supervisors. The average amount of time per feedback including the definition of new learning goals was 2 minutes and 27 seconds. A complete rating for supervisors thus took 6 minutes ad 46 seconds. 93% of all ratings were done completely.
These numbers show that the time effort for a rating via the present App is very little. Since it is the trainee who initiates the rating, the time effort for the supervisor is minimal and can be done in less than two minutes if they opt go without the feedback part. Thus, the App can be applied perfectly in the daily clinical setting. This is proven by the fact that only 9% of all initiated ratings were not finished.
Usage of feedback option
In 40% of all ratings feedback was given. The answer "No time for feedback" was chosen in 40% of all returned questionnaires. In 6% learning objectives were defined but no feedback was given. The surg-prEPAred App is designed for optimizing a conjoint evaluation by trainees and supervisors with respect to the complexity of the EPA and the level of supervision needed. This data by itself is highly valuable for both parties since it shows a clear picture of how one judges the situation. Each data set is a combination of self- and external assessment. The feedback option has been labeled as optional by intent and it is thus highly positive to see that in 50% of all ratings either feedback has been given or learning goals have been defined anyways.
Future improvements on the App
So far, our experience with the App has been positive. After our first go-live, though, a number of improvements were defined to be implemented in future versions of the App. There will be additional levels of supervision in accordance with the ten Cate scale 1. The App can generally be used in each medical specialty since EPAs can be defined individually according to the specialty's curriculum requirements. Each trainee can build up a mobile, individual and sustainable competency profile to be used at follow-up training institutions without losing competency when changing positions. Usability will be improved by new graphics for the EPA profile of each trainee, including new legends and management options for learning tools to be defined. In order to generate even more feedback, new feedback options by including pictures or videos are being tested. In future a pdf file of the profile will be generated that can be submitted to educational committees since EPAs are expected to be an integral and mandatory part of medical education. We used the original English version of the application till now, German, French and Italian translations of the interface are just created. Due to the very good first experience with the Surg-prEPAred App on a local level, the next goal should be to implement both EPAs and App into the specialty training documentation of the Swiss professional society.