PT has been used in some medical schools, primarily in Europe, to improve knowledge-based competencies [1]; however, the concept is becoming more widespread as competency-based curricula in medical education have become more widely known. The use of written examinations has been shown to be beneficial in providing feedback to trainees and educators, monitoring growth over time, and promoting deeper learning by shifting the focus from assessment of learning to assessment for learning [15]. However, when assessing many of the skills required from physicians, such as physical examinations and the ability to communicate with patients, the usefulness of written examinations is limited. Although OSCEs are expensive to administer, they are a viable means of assessing clinical skills [16], and their predictive validity for future performance has been studied [17].
As demonstrated in this study, OSCEs in undergraduate education can be used as PT. OSCE was found to be highly reliable in this study, showing higher scores and effect sizes for more experienced students. The analysis of individual stations showed that the mean test scores from all three rating scales had high inter-rater reliability. The scores on each rating scale were highly correlated. The IRS and TLRS scores differed between the training levels at all stations, suggesting that they were good at identifying levels of expertise across all content areas. yGRS scores, on the other hand, are useful for making complex multifaceted judgments, such as summative assessments. As each measure serves a different purpose, it would be beneficial to include all of them.
From the above, it was concluded that the TLRS could be used to provide individual feedback to students, showing their overall ranking and the IRS could be used to communicate the level of achievement of individual items. By contrast, the yGRS could be used for a summative assessment of test results. These characteristics are similar to those in a study conducted on postgraduate education [5]. Their study was conducted on residents in their first to fourth year of residency, whereas ours was conducted on students from two clinical years and therefore showed two levels of progress. Although shorter, we were able to measure progress in competencies similar to previous studies, suggesting that OSCE-PT is useful in undergraduate clinical education.
The results of the qualitative analysis showed that SDL can be facilitative, especially in terms of motivation. However, the results also suggest that it could inhibit self-management and self-monitoring. Much of this was owing to the fact that the curriculum and educational opportunities were not sufficiently aligned with the current testing opportunity. The original PT established its significance on a vertically integrated curriculum [18]. Students in this curriculum can be aware of their final competencies from the earliest stages and continue to engage in their learning. A trial of PT in a non-integrated curriculum revealed that the increased motivation of students through examinations did not necessarily lead to the establishment of self-directedness [19]. These results are consistent with those in the present study. Furthermore, our attempt is about performance training, which requires more readiness for learning than knowledge acquisition because it is more difficult to start learning independently.
Additionally, aspects of self-management and monitoring as well as competencies at the end of the clinical program are difficult to convey to students, despite the fact that these competencies were explained to them from the time they enrolled in the clinical clerkship. One of the characteristics that distinguishes the undergraduate clinical clerkship from postgraduate training is that it rotates through many departments, and the students are less independent in the undergraduate curriculum than in the residency. Strategies for encouraging consistent monitoring, management, and support training were considered to be inadequate. Planned opportunities must be incorporated into the clinical clerkship curriculum to enable students to train more thoroughly and regularly. Providing extra-curricular skill training is also helpful for motivated students. When introducing the OSCE-PT to undergraduates, a coordinated effort with ongoing support across the curriculum, and not only for the exam, was considered necessary.
Notably, we were able to measure progress in competencies as PTs in the issue of patient safety. Since the publication of the Patient Safety Curriculum Guide by the World Health Organization [20], patient safety education has received increasing attention in recent years. Education is necessary not only for post-accident measures but also for quality improvement; however, there are areas where the undergraduate curriculum does not adequately meet needs, such as diagnostic errors [21] and medication safety [22]. In addition, it is difficult to incorporate appropriate methods for assessing competencies into the curriculum. Self-reflection and portfolio are often used but sometimes susceptible to social desirability bias in issues where the ideal response is clear [23], such as patient safety. Also, in workplace-based assessments, consistency is a concern when dealing with patient safety events that are highly context-dependent. By contrast, OSCE is suitable for the summative assessment of patient safety because it can assess competencies collectively in a validity-controlled context. In the future, OSCE tasks for patient safety should be developed, and further utilization should be explored to use them as a PT.
Limitations
This study had several limitations. First, it was limited to a single comparison as an OSCE and did not compare PT with written test performance, although a study at a Japanese medical university suggested that PT may be beneficial in the knowledge domain [19].
Second, there were concerns regarding the reliability of the examiners. Although they had experience with existing OSCE assessments, they were not blinded to the examinees’ academic years, which may have anchored them to the differences in ability. However, given the correlation between the ratings and the actual academic years, we can conclude that the examiners were able to assess the students independently based on their actual academic years. They also rated the stations independently; therefore, we could not test for inter-rater errors or other rater characteristics. However, as many high-stakes OSCEs are administered by a single rater, we believe the impact was slight.
Third, this pilot study was conducted before the onset of the COVID-19 pandemic. After the pandemic, the situation regarding the OSCE changed slightly, and there has been a shift back to a workplace-based assessment [24]. However, workplace-based assessments have not been fully implemented yet [25]. In East Asian cultures, where students are aware of exceptionally reliable examinations, both faculty and students must establish assessments using standardized tasks rather than case-specific workplace assessments [26]. As it has been reported that even summative assessments can be effective in promoting learning if they are highly authentic, it is expected that the OSCE-PT will promote the learning of performance competencies.