Attainment of Core Competencies Milestones in Otolaryngology Residency Training in Taiwan: A Pilot Study

Background: Competency-based medical education has emerged as a mainstream method for educating and assessing the next generation of physicians. This study aims to initiate and examine the preliminary results of an integrated Traditional Chinese Otolaryngology Milestone Project in Taiwan. Methods: This prospective study enrolled 18 participants in an academic hospital setting from July 2017 to August 2019. Participants included twelve attending and six resident physicians. Using the Integrated Traditional Chinese Otolaryngology Milestones (ITCOM), five biannual evaluations involved independent self-assessments by the resident doctors, which the chief resident and attending physicians independently reevaluated. A Kruskal– Wallis test was used to compare the results of the five assessments. Results: The average scores of attained milestones for the five assessments were as follows for residents PGY1– PGY6: PGY1 (1.48 ± 0.24; 1.69 ± 0.10, P = 0.01), PGY2 ( 1.24 ± 0.12; 1.51 ± 0.23; 1.75 ± 0.06; 1.98 ± 0.21; 2.47 ± 0.18 , P < 0.0001), PGY3 (2.19 ± 0.24; 2.36 ± 0.25; 2.80 ± 0.19; 2.96 ± 0.24; 3.33 ± 0.23, P < 0.0001), PGY4 (2.68 ± 0.17; 2.96 ± 0.09; 3.35 ± 0.13; 3.58 ± 0.10; 4.17 ± 0.08, P < 0.0001), PGY5 (3.07 ± 0.24; 3.38 ± 0.12; 3.58 ± 0.10; 4.05 ± 0.09, P < 0.0001) and PGY6 ( 3.30 ± 0.27; 3.61 ± 0.28; 4.22 ± 0.20, P = 0.0001). The score results for patient care, medical knowledge, and professionalism were more likely to indicate heightened attainment of milestone levels as the program progressed. However, the curves of the score results for system-based practice, problem-based learning and improvement, and interpersonal and communication

participants and multiple institutions is warranted.

Background
Competency-based medical education (CBME) has emerged as a mainstream method for educating and assessing the next generation of physicians [1][2][3]. In 1998, the Accreditation Council for Graduate Medical Education (ACGME) started the "Outcome Project." The goal was to improve the ability of resident physicians to enhance the quality of medical care for patients in fostering specialists who could effectively perform medical work in contemporary and future health care systems [4]. In February 1999, the ACGME formally proposed the six core competencies required for training resident physicians in various specialties [5,6]. The six core competencies comprised patient care (PC), medical knowledge (MK), system-based practice (SBP), practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS). All specialties and subspecialties were required to incorporate the cultivation of six core competencies into resident training to improve and ensure the quality of residents' education [5,6].
In 2009, the ACGME began to construct an accreditation system based on the training effectiveness of the six core competencies, which is called the Next Accreditation System (NAS) [6]. The main purpose of this system is to strengthen the function of a peer review system to cultivate qualified doctors to be able to pursue their profession in the 21st century, as well as to improve the effectiveness of the residency training programs.
Additionally, it reduces the burden caused by on site evaluations. The key factor to the NAS's success is the effectiveness, completeness, objectivity, and consistency of the measurement tools. Among the six core competencies; four of them, namely SBP, PBLI, PROF, and ICS, have continually proven to be the most difficult to evaluate objectively.
Clinical preceptors often rely solely on generalized impressions to evaluate the performance of trainees. Therefore, in 2009, the ACGME and American Board of Medical Specialties selected and compared the advantages and disadvantages of various assessment tools. They jointly proposed a set of assessment models to help clinical teachers understand their students' learning effectiveness and, through the results and recommendations of this assessment, allow trainees to continuously improve their own weaknesses during their learning process rather than focus solely on test scores. This type of assessment model is called a "milestone project." The concept of milestones derives from a form of assessment called a rubric and is consistent with the concept of CBME [7]. CBME emphasizes core competencies as the main axis for the progression of learning. Each training course should focus on the quality of training results, should be learner-centered, and should less emphasize length of time as the basic requirement for training. The so-called "milestone" refers to "an observable and measurable performance statement for a specific competence [3]." These expressions of competent performance are developed in accordance with the model of the clinical practice of medical specialties and are based on the ACGME's six core competencies. At present, milestones are mainly applied in postgraduate residency training and comprise two major parts: the "subcompetencies" derived from the six core competencies and the five assessment levels within each subcompetency (within each level, one or more milestones are clearly stated). The design-according to which each subcompetency has five levels-mainly derives from the Dreyfus model of skill acquisition [8,9]. This model divides the process of acquiring professional skills into five stages, namely novice, progress, competent, proficient, and expert. The basic framework of "milestones" deconstructs the physicians' learning progress into several "subcompetencies", establishes "snapshot images" that are typical of each level of trainees, and then reconstructs them into a comprehensive clinical performance assessment [8].
In 2013, Swing et al. published a preliminary report on seven specialties in which the concept of milestones was developed for the first time to join the NAS [10]. Their results indicated that the numbers of subcompetencies and milestones determined by various medical specialties all differed. Today, the milestones developed by almost all medical specialties can be found at www.acgme.org, including for otolaryngology. In 2014, a steering group led by Tsue published the otolaryngology milestones in the United States [11]. Thus far, the ACGME has published 49 subcompetencies and 765 milestones in otolaryngology and related subspecialties. To date, there have been no reports in the English literature regarding the use of a comprehensive assessment framework in otolaryngology residency programs in Taiwan. We piloted a Milestone Project in an academic hospital by introducing the ACGME otolaryngology milestones to assess the learning progression of the resident physicians at the department of otolaryngology-head and neck surgery. The results of the first five biannual evaluations were demonstrated and analyzed.

Study Design and Participants
The institutional review board at Cardinal Tien Hospital approved this single-center prospective study (CTH-106-3-5-028). Written informed consent was obtained from all participants. To ensure that resident physicians were not adversely affected because of participating, the written consent clearly stated that the results of the assessment were referenced only for academic research and would not be used for certification or for the resident physicians' promotion based on their training year. Twelve otolaryngology specialists and six resident physicians (PGY1-PGY6, the number denotes the seniority of training year) participated in the pilot study.

Development of the Integrated Traditional Chinese
Otolaryngology Milestones (ITCOM) We obtained a formal license agreement from the ACGME in June 2017 to translate the otolaryngology milestones [12] (17 subcompetencies and 319 milestones), an appendix [13] with 6 subcompetencies and 121 milestones, and other related subspecialties, including neurotology [14] (13 subcompetencies and 152 milestones) and pediatric otolaryngology [15] (13 subcompetencies and 173 milestones), into the Traditional Chinese version. A total of 49 subcompetencies and 765 milestones were approved by the ACGME after back-translation. A working group of ten attending otolaryngology specialists was responsible for the designation and development of an integrated Traditional Chinese milestone project relevant to otolaryngology residency training in Taiwan. The preliminary plan was to select approximately 20 subcompetencies, of which 10-12 were subordinate to PC and MK, and the remaining 8-10 pertained to the other four core competencies, namely SBP, PBLI, PROF, and ICS. The members of the working group considered the completeness and balance of each subspecialty and rated each subcompetency as high, moderate, and low priority. After three rounds of voting, the Integrated Traditional Chinese Otolaryngology Milestones (ITCOM) was determined [16].

Consensus on Performance Rating
Each subcompetency had five levels, measured on a scale from 1 to 5 points. The points between one level and the other were recorded with a.5 numerical range (e.g.,.5, 1.5, 2.5, 3.5, and 4.5). In other words, when a resident has developed all milestones in a given level or at a lower level, he will provide a score at that level (e.g., 1, 2, 3, 4, and 5).
However, when a resident has actually attained all of the lower-level milestones and also some higher-level milestones, then he or she will check the score between the two levels (e.g., 1.5, 2.5, 3.5, and 4.5). If the resident has not reached all milestones required for the first level, then the score will be.5.

Assessment
The authors piloted the semiannual phase milestone assessments with six residents (PGY1-PGY6) in July 2017, January and July 2018, January and July 2019. The residents performed independent self-evaluations; subsequently, the chief resident and faculty members graded them independently. Before reassessing each resident, the chief resident and faculty members may have access to their profile, consisting of Ad hoc evaluation results such as case logs, medical reports in academic conferences, results of mini-clinical evaluation exercises (mini-CEX) and direct observation for procedural skills (DOPS), multirater feedback of patients and paramedical staff in the department, and the results of written exams. During these re-evaluations, the chief resident and attending physicians added or deleted certain milestones that the residents might or might not have attained and determined the re-evaluation scores.

Data Analysis
The time required to complete each assessment (self, chief resident, and attending) was recorded. The average score of the 21 subcompetencies was calculated for each resident physician. Mann-Whitney U test and Kruskal-Wallis test were conducted as appropriate to compare the time and results of the 2-5 re-evaluations of the 12 faculty members. The average score of attained milestones re-evaluated by the attendings for the six core competencies according to the training seniority was calculated. The degree of the linear relationship between the average score of each core competency and the postgraduate year level was described. Data were analyzed using SAS software (version 9.4; SAS Institute, Inc., Cary, North Carolina, USA). All reported P values were two-sided, and a Pvalue of <.05 indicated statistical significance.

Results
The ITCOM addresses 21 subcompetencies, covering all six core competencies [16]. Each subcompetency is divided into five levels, and each level comprises 1-6 milestones, constituting a total of 368 milestones. Of the subcompetencies, 13 are subordinate to PC and MK, consisting of three subcompetencies for the subspecialty of head and neck surgery, three for otology and neurotology, two for rhinology and reconstructive surgery, two for laryngology, one for pediatric care, and one for sleep medicine. Non-PC and non-MK comprise two SBP, two PBLI, one PROF, and three ICS (Table 1 and online supplement data). Three resident physicians (PGY2-PGY4) and 10-12 attending physicians completed all five consecutive assessments. One resident physician (PGY1) completed two evaluations. The other two resident physicians (PGY5-PGY6) completed 3-4 evaluations according to their seniority before they passed the otolaryngologic board examination.
The participation rate was 100%.  Figure 1 showed the relationship between the attainment of the average milestone levels during the five assessments by residents of post-graduate year 1-5. We observed a higher score with increasing training seniority in the five assessments.

Discussion
In the present study, the ITCOM was applied to conduct five consecutive assessments with an interval of 6 months to demonstrate the attainment of competencies of the otolaryngologic resident physicians at different levels in terms of the six core competences. The results demonstrate that the ITCOM can quantify the learning progression of the resident physicians on the six core competencies, particularly for those non-PC and non-MK competences. The time required for each assessment was approximately 15-25 minutes. The otolaryngologic residents attained more milestones evaluated by the attendings with increasing training seniority. Collinearity among selfevaluation and re-evaluations is better in PC, MK and ICS than in PROF, SBP and PBLI.
Additionally, the accumulated radar chart of the 21 subcompetencies of the individual resident was valuable for the "coaching feedback" conducted by the program director biannually.
Patients' demand for quality of health care has increased over time. Increasing attention to the balance between cost-effectiveness of the health care system and patient safety is emphasized. Therefore, continually improving the quality of resident training in all specialties is crucial to foster future doctors that meet the requirement of the society. In the cultivation of surgical specialists, a lack of effective assessment methods and outcome measurements has long inhibited the evaluation of the performance of trainees' learning progression and the determination of training standards [17]. A study by Maruthappu et al. confirmed that when residents from the surgical department received effective feedback during the training process, they improved their learning effectiveness and clinical performance [18]. The training process requested by the Taiwan Society of Otorhinolaryngology-Head and Neck Surgery (TOS), according to training seniority, currently adopts the "case logs" to record relevant clinical skills. The case logs method is also used to record the hours of continued medical education in areas such as basic and advanced surgeries, medical ethics, and to record academic publications; all of these are necessary conditions before applying for the board examination. However, only the date and the instructor's signature are registered in the "case logs." The learning progression and the residents' level of participation cannot be evaluated. Previously, the ACGME monitored the number of cases of online registration of all resident physicians once a year and found that the number of cases registered by different training institutions varied considerably [19]. An excessive number of cases easily affects the quality of learning. In contrast, the resident physicians in some training programs have not attained the minimal requirements of the ACGME [20]. Cadish et al. investigated multiple specialty residents in a single training institution and found that nearly two-thirds of resident physicians considered the "case logs" work to be highly arduous, which affected the willingness, frequency, and accuracy of registration [21]. Dermody et al. conducted an anonymous questionnaire survey of all otolaryngology-head and neck surgery residents and program directors (PDs) in the US; the survey revealed that most of the residents and PDs believed that accuracy was not high for online surgical case logs prescribed by the ACGME [22]. Therefore, whether these surgical case logs genuinely reflect the operative ability of residents is questionable [23]. By contrast, in Taiwan, only at the time of on-site evaluation and before applying for the board examination, TOS-appointed members check the contents of the "case logs" registration, which lack timeliness and the ability to reflect individual resident's training progress [24].
At the end of 2016, the ACGME reported the first results of the milestone assessment across all specialties in the US [25]. The data showed that the average level of milestones resident physicians attained do increase as the program progresses.Beeson et al. reported the assessment results of the emergency department, which first developed the milestones, and verified the reliability and validity of the emergency department milestones [26]. They confirmed that milestones can be used for the objective assessment of their specialty trainees' competencies. Meier et al. employed milestones to assess 44 residents from the department of general surgery and followed them for 2 years. They demonstrated that the residents' milestone scores significantly increased with the years of training. The scores (including total scores and scores in each area of the six core competencies) of the appraisers (including those involved in self-assessments) were highly consistent. The participation rates of residents and attending physicians were 93.9% and 92.9%, respectively, and most of the physicians regarded the milestone assessment positively [27]. In the present study, with a participation rate of 100%, we examined the correlation between the average score of the 21 subcompetencies of the ITCOM evaluated by the attendings and resident physicians of different post-graduate year. The results indicated that both the self-evaluations and re-evaluations exhibited an increase in levels as training seniority increased, which is consistent with the results from other specialties in the US. However, when average score of the six core competencies were further subdivided, we observed that the score results of three core competencies, namely PC, MK and PROF, were more likely to indicate a heightened attainment of milestone levels as the program progressed. By contrast, the curves of the scoring results of SBP, PBLI, and ICS were more horizontal, indicating that the milestones that represent these core competencies can indicate a higher level of performance even in a resident with low training seniority. These results confirm that some milestones should be presented by residents with low training seniority (early milestones), whereas others are expected to be reached after a certain number of years of training (late milestones). With this possible distinction, the PDs can easily detect unsuitable or ineligible trainees early and intervene timely [28,29].
The consistency among the self-evaluation and the results of the re-evaluations by chief resident and the attendings was examined. We observed a better agreement among the results of the self-evaluation and re-evaluations in the core competencies of PC, MK and ICS than those in the PROF, SBP and PBLI. Tichter et al. reported that in emergency residency training, the self-evaluation scores of resident physicians at all levels were higher than those of the re-evaluation results [30]. However, Lyle et al. compared the assessment results of milestones in a department of surgery, reporting that the self-rated scores of resident physicians were on average.5 points lower than the re-evaluation results, of which the gap between MK was most significant [31]. Yao et al. analyzed an assessment of milestones attained by residents in the plastic and reconstructive department, indicating that with the exception of those of the physicians in their sixth year of residence, the self-evaluation scores of resident physicians at all levels were higher than the re-evaluation scores by attending physicians [32]. They postulated that the differences in the results of these studies may be related to the degree of selfconfidence in individual resident physicians or some intrinsic culture among different specialties. In general, the advantages of applying milestones as a learning assessment approach for specialty trainees are believed to indicate that the Milestone Project clearly specifies the knowledge, skills, or attitudes that trainees should possess for each subcompetency. However, among the six core competencies, PROF, ICS, PBLI, and SBP are more difficult to evaluate objectively. Additionally, these non-PC and non-MK domains of milestones demonstrated a wide variation among specialties. In a multispecialty review, Edgar et al. reported that there were more than 230 different ways of describing PROF, 176 for ICS, 171 for PBLI, and 122 for SBP, in the 26 core specialties and the transitional year [33]. This made the comparison between specialties difficult. The ACGME addressed this issue in the ongoing revision process for Milestones 2.0 and developed cross-specialty "harmonized" milestones for PROF, SBP, PBLI, and ICS [34]. When these common set of subcompetencies and milestones become available, the non-PC and non-MK requirement of the trainees across specialties will be standardized.
The ACGME requires resident physicians to perform self-assessments before receiving milestone re-evaluations. However, whether the residents understand the purpose of the milestone assessment and are capable of self-evaluation according to the milestone assessment framework is unknown [35]. Also unknown is whether the attending physicians responsible for re-evaluations understand the content of these milestones to a certain extent. A consensus on the assessment criteria is the key to the success of an assessment method based on the milestones. In the present study, residents, the chief resident, and attending physicians were responsible for the translation of a part of the ACGME otolaryngology milestones. The chief resident and the correspondent author were responsible for the post-translation validation. Once the translation was complete, the articles were sent back to the ACGME, who translated the articles back into English and approved the Traditional Chinese version. All attending physicians reviewed the milestones in detail, considered the basic requirements for resident training and the comprehensiveness and balance of each subspecialty field, and participated in the selection of the subcompetencies for ITCOM. Prior to the grading, two consensus meetings were held. Therefore, all the participating residents and attending physicians were able to complete the assessment timely. According to Tsue et al., an average of 10-12 minutes is required to evaluate an otolaryngologic resident physician [11]. In this study, the residents' self-assessment required 15-30 minutes, the average time required by the attending physicians of the five biannual re-evaluations for each resident physicians was about 15-25 minutes, which were longer than the times noted by Tsue et al. This was possibly because in the US, the otolaryngology milestones only feature 17 subcompetencies and 319 milestones. However, the ITCOM comprised 21 subcompetencies and 368 milestones; thus, the assessment required longer time. Moreover, the comparison indicated that the time required for the re-evaluations by the attending physicians for each resident did not differ in the five assessments. This is reasonable because the attendings have to reconsider the performance of individual resident physician for each level of milestones during the re-evaluations.
In terms of the quantity of subcompetencies and milestones, no objective criteria currently exist; most are determined by experts in various specialties. Less number of subcompetencies and milestones would render completing the training overly easy, whereas too many subcompetencies and milestones would increase the difficulty in score attribution. Determining a balance in the milestone assessment requires extensive operational experience and poses a challenge to each specialty. We employed the ACGME milestones as a blueprint, which were selected, reviewed, and approved by ten faculty members from various otolaryngology subspecialties before the pilot assessment. The original concept of the milestone concept is to amend the rule that training seniority is the prerequisite for participating in the board exams [36]. However, in practice, because residents of different seniority differ in their clinical experience, different assessment approaches must be considered. For example, in terms of PC, senior residents may have encountered patients with different levels of complexity; therefore, their scores may vary in levels and even regress occasionally. Although continual periodic assessments can solve this problem, the seniority of the resident physicians should still be considered. The ACGME Milestone assessment assumes that residents can obtain their degrees once their training is complete; this objective is set at the fourth level, which is "proficiency." In this study, the average score of the re-evaluation of the two consecutive R5s before attending the board exam was 4.22 and 4.05, respectively. Further tracking and evaluations were required to determine whether R5 could attain the subcompetency milestone average score of 4 before the training program was over.
A "coaching feedback" based on the results of the milestone assessment is valuable. As shown in the radar chart in Figure 4, the score of PGY3 in PC1, PBLIN2, and PROF1 was lower in the second re-evaluation than in the first. The PD essentially relied on this result to provide substantive feedback, explore the causes to the results, and formulate methods of continual improvement. Thus, the same resident physician showed a great improvement in the performance of these perspectives in the following re-evaluations.
This study had several limitations. First, only one resident physician was evaluated in each training year. Therefore, using statistical methods to calculate whether the increase of milestone scoring (including self-and re-evaluations) exhibited a statistically significant difference among resident self-evaluations was not possible. In addition, the study involves single center with 12 faculty members and 6 trainees. Future study involving large-scale evaluations with multiple institutions and peer resident physicians is warranted. However, in this small-scale pilot, the attendings and the residents work together closely, and the re-evaluations were collected from all faculty members, thus, the results should have been more reliable. However, in a large training program, the residents will rotate in different subspecialties to learn with different preceptors. The clinical teachers might not be able to grade trainees effectively because of the limited opportunities for direct observation of subcompetency-related milestones in nonexpert fields [31]. According to the suggestion of Tsue et al.,, in this situation, resident physicians can be divided into groups, and the assessment can be conducted in more than one session in each group [11]. In the case of otology, trainees ought to be evaluated by relevant attending physicians on the PC and MK otology-related subcompetency milestones, as well as non-PC and non-MK subcompetencies. Individual resident physician will not be evaluated by all attendings. The workload of the clinical teachers will be reduced. However, the consistency between different groups of attendings may raise some concern.
Conclusions The ITCOM incorporated the ACGME six core competencies and represented the knowledge, skills, attitudes, and attributes that correspond to each milestone component and level in otolaryngology. This study provides an initial look at the adoption and implementation of the ITCOM assessments in individual otolaryngology training program in Taiwan. The results demonstrate that the ITCOM can document the learning progression of the resident physicians on the six core competencies, particularly for those non-PC and non-MK competences. Additionally, the assessment system can stimulate the active learning and self-development of residents. The time required for each assessment was reasonable. The otolaryngologic residents attained more milestones evaluated by the attendings with increasing training seniority. The slope of the regression lines between the milestone scoring and the training seniority in PC, MK and PROF were larger than those in SBP, PBLI and ICS. Collinearity among self-evaluation and re-evaluations is better in PC, MK and ICS than in PROF, SBP and PBLI. Early milestones included domains of SBP, PBLI and ICS, while late milestones comprised domains of PC, MK and PROF. Future research will include a multicenter evaluation using ITCOM, and the development of online assessment process with an automatic scoring algorithm to simplify the assessment process in a large institution.  Tables   Table 1 List