Differences in the Assessed Performance of Medical Interns in a Comparison of Competency- based and Subject-based Curricula: a Qualitative Study


 Background: An internship is a transitional training period a medical school graduate undertakes prior to attaining full registration as a medical doctor. Numerous studies have shone a light on the unpreparedness of medical school graduates for the realities of medical practice. Implementation of a competency-based curriculum (CBC) is thought to produce better prepared medical graduates because this curriculum is more structured and integrated; moreover, learning outcomes, the learning process, and assessment are better aligned in the cognitive, affective, and psychomotor domains. During 2006–2011, Indonesia conducted a nation-wide reform shifting to a CBC from a subject-based curriculum (SBC) and launched an internship program in 2013. This study aimed to compare the performances of medical interns training under these two different curricula. Methods: This study was conducted in Yogyakarta Province, involving six supervisors and six interns from six different types of district hospitals. Qualitative, semi-structured interview methods, and thematic analysis were applied. Results: In total, 1,296 initial codings were identified and grouped under the following topics: overall experience (327), effective communication (97), scientific foundation (47), clinical skills (48), information management (23), management of health problems (194), self-insight and professional development (351), and professionalism (209). Eighteen themes emerged, namely understanding the internship, authority, practice context, interprofessional communication, patient communication, cultural awareness in communication, gap and fulfillment in clinical skills, knowledge gap and enhanced knowledge, assumption of a General Practitioner role, progression to profiency, information literacy, internal motivation, learning opportunities, constructive feedback, transition, and working culture. There were irregular patterns of performance from both of the CBC and SBC interns. Interestingly, compared with the SBC group, the CBC group revealed less confidence and assertiveness in handling patients. This group also lacked communication skills when interacting with nurses and patients alike. Moreover, the SBC group demonstrated the ability to learn and adapt quickly to the working environment. Neither group paid much attention to information literacy. Conclusion: We cannot conclude that a CBC would yield better prepared medical graduates. The results of this study showed irregular patterns in eight areas. Hospital contexts and variation in the characters of interns might have contributed to this irregularity. More exposure to the working context and greater contributions from the healthcare system are recommended in the medical curriculum, whether it be a CBC or a SBC. WC:338


79
In many countries, especially those following the United Kingdom (UK) system, 80 internships are part of the training program medical students must undertake before they can 81 apply for registration as medical doctors. An internship is defined as the period of work-based 82 learning or practical training that follows medical school (1). In these countries, the internship 83 is perceived as the transition period from medical student to junior doctor. It can be 84 considered as formal or informal training, or part of postgraduate training, and is a critical 85 stage in the progression of a doctor's career (2). This transition is discerned as a major 86 challenge, particularly in terms of the increased responsibility and workload, as well as 87 increased professional contact with other healthcare workers and patients (3). 88 In this period, new medical graduates are confronted with the realities of the clinical 89 workplace, which require them to evolve from supervised interns to independent 90 practitioners. A number of studies have been conducted investigating this evolutionary 91 process from various perspectives. A study in Germany showed that many junior doctors 92 were not adequately prepared for their upcoming careers (4). This finding was confirmed by 93 research from various countries, indicating that this transition is frequently considered 94 stressful. In addition to the unpreparedness, poor support, and limited education, a number 95 of factors contributing to this stressful experience have been identified in new medical 96 graduates entering clinical practice (5). A study in the UK reported a higher prevalence of 97 burnout syndrome during the second semester of the internship. Students who conducted 98 their internships in public hospitals demonstrated higher scores on emotional fatigue and 99 depersonalization than those who performed their internships in private hospitals (6). 100 Another study drew the conclusion that the transition from medical student to clinician does 101 not necessarily have to be signaled by stress and mental fatigue but may provide a fruitful 102 opportunity for interns to grow into their roles as medical doctors if they are provided with 103 extensive support and guidance (7). 104 To overcome the unpreparedness of medical graduates, many medical schools have 105 reformed their curricula into competency-based or outcome-based curricula (8). The 106 University of New South Wales (UNSW) in Australia evaluated its medical graduates three 107 months into their internships. Medical graduates from a new outcome-based integrated 108 program rated themselves as having good clinical and procedural skills, with ratings that 109 indicated significantly greater capabilities than those of graduates of the previous UNSW 110 content-based program (9). Another study demonstrated the strength of an outcome-based 111 curriculum in its ability to produce competent students that are well prepared for their 112 internships (10). Other investigations into preparedness in terms of clinical skills showed no 113 significant differences between lecture-based and problem-based curricula in these metrics 114 (11,12). Problem-based graduates rated themselves as better prepared in their "awareness of 115 legal and ethical issues," and lecture-based graduates rated themselves as better prepared in 116 their "understanding of disease processes" (12). A study in Groningen Medical School in the 117 Netherlands on the effect of implementing undergraduate competency-based medical 118 education did not support the assumption that competency-based education results in 119 medical graduates who are better prepared for medical practice (13). 120 In 2013, the Indonesian government launched an internship program for new medical 121 graduates, which was decreed in the Medical Education Law, following the implementation 122 while all medical schools were shifting from a subject-based curriculum (SBC) to a 125 competency-based (CBC) curriculum. This was conducted under the assumption that the 126 medical internship would produce better prepared medical graduates, who eventually would 127 become proficient independent practitioners. This study aimed to determine the differences 128 in medical interns' performance based on their curricular background, namely CBC and SBC. 129 130 METHODS 131

Context 132
This study was conducted in Yogyakarta Special Province, because it was among the first of 133 the Indonesian provinces to adopt the medical internship program. At the provincial level, 134 the Health Office of the Yogyakarta Provincial Government organizes the program, which 135 consists of recruitments, placements to district hospitals and community health centers, as 136 well as an induction program for the medical interns. Most of the medical graduates who 137 apply in Yogyakarta Province come from the three medical schools located in Yogyakarta. In

Study Design 143
This was a qualitative study using semi-structured interviews aimed at evaluating the 144 performance of medical interns in seven areas of competency as described in the Standards of 145 investigation is the performance of interns as it reflects their transition from competency to 149 proficiency during the internship program as perceived by the supervisors and their interns. 150

Study Participants 151
To achieve maximum sampling variation, participants were recruited from six different types 152 of hospitals: three public district hospitals from three different districts, one district military 153 hospital, one district police hospital, and one private hospital. Six medical interns from the 154 Faculty of Medicine Universitas Gadjah Mada were recruited, from both CBC and non-CBC. 155 Six supervisors who supervised these medical interns were recruited from the six hospitals.

Data Analysis 163
Thematic analysis was applied in this research because it is especially suitable for 164 systematically identifying, organizing and suggesting insights into patterns of meaning 165 (themes) across a data set, which allows researchers to make sense of the collective meaning 166 and experiences. The purpose is to identify relevant themes to answer particular research 167 questions, which can be performed both inductively and deductively. A six-step approach to 168 thematic analysis was applied, namely (i) familiarization with the data, (ii) generation of 169 initial codes, (iii) search for themes, (iv) review of potential themes, (v) definition and naming 170 of themes, and (6) production of a report (14). 171 For the in-depth interviews, once completed, the responses were immediately 172 transcribed by the transcriber. The interviewer, who was part of the research team, would 173 check and identify the initial coding and themes. To strengthen the results' trustworthiness 174 and consistency, member-checking was applied. Three persons (namely, the interviewer, the 175 principal researcher, and the expert who was not member of the research team) conducted the 176 initial codings. Each person conducted the initial codings independently. Then, the principal 177 investigator thoroughly read the results of the initial codings and sorted them in accordance 178 with the interview questions using color codes. The initial codings with the same color codes 179 were assembled into a table to list the relevant topics. After reading the initial codings several 180 times and attempting to identify the underlying meanings, the principal investigator 181 constructed potential themes for each question from the CBC and SBC groups' responses. 182

184
There were 233 pages of transcripts from six supervisors and six medical interns, and 185 658 initial codings from supervisors and 734 initial codings from medical interns were 186 identified. The initial codings were grouped according to the interview topics. Table 1 shows 187 the sums of the initial codings for each of the topics from supervisors and medical interns 188 grouped into CBC and SBC. Potential themes were identified on the basis of initial codings 189 for each interview topic for supervisors and medical interns. After being reviewed and 190 contextualized in several iterations, potential themes with similar meanings or referring to the 191 same situations were further combined into a final theme, as demonstrated in Table 1. 192 Based on all initial codings identified from the supervisors and interns (total number 199 of responses = 1,296), 18 themes from all topics were developed, as depicted in Table 2. 200

Overall Experience 204
With regards to the overall experience, medical interns spoke more than supervisors. Medical 205 interns from SBC expressed their overall experience more often than those from CBC. Three 206 themes emerged from this topic: understanding the internship, authority, and practice 207 context. The CBC group perceived the internship as a period for new medical graduates to 208 become proficient and independent, whereas the SBC group considered the internship as an 209 opportunity to apply medical knowledge and skills directly to patients, but under 210 supervision. Concerning authority, the CBC group shared that medical interns were given full 211 authority to practice as General Practitioners (GPs). However, the SBC group judged that, 212 although interns already had medical degrees, they only had three-quarters of the 213 responsibility, and the rest was held by the supervisors. For the practice context, the CBC 214 group considered the total number of patients in a hospital important: If there is a small 215 number of patients, then that hospital cannot be used for the internship program. The SBC 216 group observed that, in practice, basic clinical procedures were conducted mostly by nurses, 217 and every doctor had the authority to prescribe medication. Interns only had to adhere to the 218 protocols. According to the response patterns, the SBC group perceived the internship as a 219 continuation of the clinical rotation with additional responsibilities, whereas the CBC group 220 understood the internship as a workplace where interns worked as GPs with full 221 responsibility. Table 3 shows some of the excerpts concerning the interns' overall experience 222 with the topic's three related themes. 223 was not sufficient to understand the diversity of the patients' cultural backgrounds, whereas 248 when the SBC interns were faced with cultural issues, they tended to be quiet at the beginning 249 while observing the behavior of other health professions and the house doctors. Gradually, 250 they adopted those behaviors. The following excerpts (Table 4) reflect these accounts. 251

Clinical Skills 254
Regarding clinical skills, two themes were identified, namely the clinical skills gap and 255 fulfillment. For the first, both CBC supervisors and interns admitted that medical graduates 256 needed to practice their clinical skills more because, in medical schools, they were at the point 257 of "only having practiced the required clinical skills once." The CBC interns admitted that in medical schools, they lacked the opportunities to apply their clinical skills directly to patients. 259 A SBC medical intern expressed that the stitching procedures used in the hospital were not 260 compliant with the standard procedures. 261 The second theme was fulfillment, indicating how medical interns could close this gap. 262 The CBC group explained that medical interns were given the authority to handle patients on 263 their own from the beginning, including anamnesis, physical examinations, and suggesting 264 supporting examinations. The SBC supervisors explained that some medical interns at first 265 were not able to perform wound stitching nor write medical prescriptions, but during the 266 internship, the supervisors taught them, and eventually they could perform both functions. 267 The SBC interns also mentioned that, because they were already a medical doctor, they were 268 more respected by other health professionals and house doctors, and thus they felt that they 269 had more freedom to apply their competencies to patients. This sentiment is echoed in the 270 following excerpts in Table 5. 271 292

Management of Health Problems 293
For management of health problems, two themes were also identified. The first was "to 294 assume a GP's role." Both supervisors from CBC and SBC explained that the medical interns 295 were given the responsibilities to assume a GP's role. They examined the patients from the 296 beginning and wrote down all the findings in their medical records, including the treatment 297 given. The interns had discretion to determine when they needed to consult with the 298 specialists. Interns from both groups confirmed this and added that they also had a 299 responsibility to deliver promotions during placement in community health centers. The 300 second theme was "progression toward proficiency." Interns from both groups confirmed 301 that the internship improved them with proficiency in patient management. Although, there 302 were some obstacles, such as new protocols in district hospitals, and basic medical 303 procedures, such as suturing, which are the tasks of the nurses as illustrated in the following 304 excerpts in Table 7. 305

RI2 page 167 paragraph 13
We did health promotion when we were placed in community health centre.
Our supervisors were very good. We felt comfortable. They gave discretion to us, but they still monitored and checked before the treatment was delivered.

R16 page 17 paragraph 46
We were given discretion to performce procedures, and we had to inform the patients about the risks.

R16 page 19 paragraph 52
One of our compulsory tasks is to do health promotion in the community health centre, we have to many villages.

RI1 page 4 paragraph 30-34
In the ward, we have to consult with the specialist. But in community health centre, we could handle ourselves.

RP3 page 23 paragraph 166
Writing prescription needs to be improved, including the dosage, and how to administer the drug….I asked them to learn again.

Information Management Skills 308
The one theme that emerged in this topic was medical record literacy, which describes the 309 extent to which interns understand how to complete medical records and how to use them. 310 Interns from both groups explained that they understood how to complete the medical 311 records but lacked perspectives on how to make use of them. 312

Self-Insight and Personal Development 313
With regards to this topic, three themes were identified, namely internal motivation, learning 314 opportunities, and constructive feedback. Both groups-interns and supervisors-agreed on 315 how internships would enhance the interns' competencies depending on their internal 316 motivations. Interns who were serious about using every opportunity to improve their 317 competencies would achieve more, but those who lacked the internal drive tended to be 318 passive and missed many chances to learn. The second theme was learning opportunities. 319 While interns are regarded as a full GP with their own professional discretion and 320 responsibilities, the hospitals, house doctors, and other health professionals offered them 321 unlimited learning opportunities. Both supervisors from CBC and SBC groups confessed that 322 they monitored how the interns performed from a distance. With regards to constructive 323 feedback, the CBC group explained that feedback was given both formally and informally. To 324 avoid embarrassment, constructive feedback was purposefully given after the intern had 325 finished interacting with the patient. SBC interns explained that they felt the logbook was not 326 useful. The supervisors were focused on giving feedback related to the cases, rather than on 327 the interns' performance (such as communication or clinical skills). CBC interns highlighted 328 that compatibility between interns and their supervisors affected the success of the 329 supervisory process. The following excerpts (Table 8) portray this topic and the three related 330 themes. 331 Table 8. Excerpts for self-insight and personal development 332

Competence-based curriculum Subject-based curriculum
Internal motivation RP1 page 18 paragraph 120 As supervisors, we could not force the interns. It is up to them how they self-assess their ability.

RP1 page 3 paragraph 68-71
For clinical skills, the interns must be eager to try and practice. At the beginning, they are scared to try…

RP4 page 8 paragraph 26
The seriousness, the willingness and the interest of each intern are different.

RP4 page 12 paragraph 44
To what extent interns make use of the opportunities to advance their competencies, knowledge and skill depend on interns' inisitiave and motivation.

RP1 page 1 paragraph 2
Interns are given full discretion to perform GP duties, however opportunities to increase their proficiency are wide open.

RP5 page 10 paragraph 58
In the hospitals, the basic clinical procedures are performed by the nurses; although the interns know the theories. So, they learn from the nurses to improve their proficiency

RI3 page 21 paragraph 98
Feedback from the supervisors…are mostly related to the cases, not directly about how is the interns' professionalism.

RI5 Page 20 paragraph 70
Feedback is given in a group. When an intern is dealing with the patients, the others are observing from the other side behind the curtain. The next day, we discuss.

RI5 page 19 paragraph 68
The supervisors provided feedback after we finished handling the patients. They did not embarrass us in front of patients…

333
Professionalism 334 There were two themes for the topic of professionalism, namely transition and working 335 culture. Both groups confirmed that, at the beginning, the interns were required to make 336 adjustments within the work environment in the hospitals. They needed to increase their 337 speed and accuracy when examining patients. After completing the internship, most interns 338 had increased their professionalism. However, CBC supervisors expressed their concerns that 339 their interns were frightened to handle the patients directly, lacked the ability to build rapport 340 with patients, and lacked confidence and prowess in decision making. Therefore, at the 341 beginning, CBC interns needed more intensive supervision, and gradually, by the end of the 342 program, they had gained confidence and courage in handling patients independently. SBC 343 supervisors did not report such situations. They observed that SBC interns demonstrated a 344 sense of responsibility in handling patients. 345 On the theme of working culture, CBC interns expressed worry about their ability to 346 build relationships with colleagues and other health professionals, as well as concerns over a 347 discrepancy between the ideal situation and the practical one, which imposed an ethical 348 dilemma. In contrast, the SBC interns explained that, in real work environments, emotions 349 were involved when dealing with patients as well as when building relationships with 350 colleagues. Such aspects were not found when they practiced laboratory skills or conducted 351 clinical rotations. Cultural differences also affected how they handled relationships with 352 patients and colleagues. Table 9 shows the excerpts for this topic and its two related themes. 353

RP3 page3 paragraph 14
Compared to the previous intern, they had the same level of knowledge. But, I was confused why the current interns are getting less confidence, less courageous, and less adeptness.

P 3 p paragraph 16
Ability for decision making of interns is good, the communication is also good.

RI3 P5 paragraph 28
During internship, professional behavior is challenging, because we handle the patients with all their emotions. As time goes by, we becomes more mature.

RI6 P 18 paragraph 50
When we were medical students in clinical rotation, our main target was to pass the exam. But now, our target is to heal the patients. Our sense of responsibility to the patients emerges.
Working culture

RP3 p20 paragraph 105
The interns must have a courage to take the risks, but they are not.

356
Our assumption that CBC would perform better in all aspects of areas of competence 357 are not supported by these qualitative findings. This is in line with the findings from a study 358 in Groningen in the Netherlands, which also claimed that their findings could not bear the 359 assumption that a CBC would yield better prepared graduates (13). In contrast, a study in the 360 In this study, we obtained surprising results on the topics of effective communcation, 366 clinical skills, health management, and professionalism. We expected that CBC interns would 367 demonstrate better performance in these areas because CBC has a more structured and better 368 designed curriculum for communication and clinical skills, as well as a new skills laboratory 369 provided with improved clinical skills equipment. However, the findings were to the 370

contrary. 371
A study in Kenya showed comparable results: they discovered a consensus across 372 subjects on deficiencies in interns' clinical skills and experience of handling clinical problems. 373 Supervisors in general provided critical comments regarding interns' competencies, whereas 374 interns showed more concerns about their weaknesses. Supervisors expected better 375 performance on surgical procedures than how interns anticipated to. Perception of medical 376 graduates' unpreparedness seemed to be caused by a failure to apply the apprenticeship 377 model in medical school and insufficient exposure to district hospitals prior to graduation 378 (15). Another study in the UK investigated the induction process as a solution to improving 379 interns' preparedness. It was discovered that the induction program was more useful when 380 conducted at the beginning than in later rotations, and longer inductions were preferable to 381 shorter ones. When interns were placed in clinical departments, medical interns lacked proper 382 inductions, particularly regarding their roles and responsibilities in particular departments, 383 including where to find equipment and documents, who to contact and how to contact them, 384 as well as local preferences, policies and procedures (16). In this study, the induction was 385 performed twice. The first time was in the District Health Office regarding the internship 386 program, and the second was in an orientation that each hospital gave to new interns. 387 However, the induction was performed as a one-way lecture. Therefore, some interns had 388 ethical dilemmas when they experienced that the suturing procedures were not the same as 389 the procedures they practiced in the skills laboratory. 390 Brennan et al. studied the transition of medical students to junior doctors at the 391 Peninsula Foundation School in the UK. They discovered that although curriculum reforms 392 had been conducted, most participants still perceived that the transition was pressured. 393 Coping with their freshly attained responsibility, surviving uncertainty, dealing with multi-394 professional teams, encountering the sudden death of patients, and feeling unsupported were 395 important themes. However, the stress of transition could be lessened by the level of clinical 396 experience obtained during their undergraduate years. They suggested that medical schools 397 need to ensure that students are given prior exposure to clinical environments that allow for 398 continuing "meaningful" contact with patients and increasing opportunities to assume a role 399 as junior doctor, even as students (17). This study confirms Brennan's findings. Also, in 400 Indonesia, the new CBC structured the clinical rotation into defined activities, such as clinical 401 tutorials, bedside teaching, etc. During their clinical rotation, medical students had fewer 402 opportunities to take full responsibility for patients, causing them to feel unprepared for their 403 responsibilities in their internships. The interns confirmed that they studied the patients in 404 fragmented sequences, and their major concern had been how to pass the exams, rather than 405 fully learning the process of patient management. increasing confidence and competence in this early period of career transition. However, a 410 number of organizational barriers were pinpointed, e.g., incomplete teams and shift work, 411 which produced problems in their integration (18). In this study, both interns from CBC and 412 SBC confirmed that team work among interns could run well, although there were variations 413 in interns' characters and habits. However, team work with nurses did not always run 414 smoothly. They found that different expectations and the dominance of nurses hindered the 415 teamwork. Concerning support from supervisors and the hospitals, they had positive 416 impressions. The learning opportunities given by the supervisors and the hospitals were 417 immense; some interns seriously responded to these opportunities, while others were more 418

relaxed. 419
For the management of health problems, CBC interns lacked confidence and courage 420 compared with SBC interns. Both interns believed themselves to have sufficient knowledge 421 and clinical skills, but CBC interns experienced more difficulties with patient management, 422 whereas SBC interns had more confidence handling patients. This might be due to a more 423 structured clinical rotation in CBC, which reduces the authenticity of the clinical environment. compromising key relationships; communicate concisely; understand team roles (including 432 their own status within hospital hierarchies); and negotiate conflict. Participants explained 433 that they experienced a decrease in self-confidence and difficulty maintaining self-care and 434 social relationships (19). These results substantiated the findings from this study that the CBC 435 interns also experienced decreasing confidence and difficulty in communication with nurses 436 and patients. 437 This study enriches the literature on the transition period for medical students to 438 become proficient independent practitioners. Some findings corroborate previous studies. 439 But, there were findings that are also contrary to expectations concerning CBC. Some 440 limitations of this study were the small number of subjects and only one province was 441 included. A larger scale study should be performed in the future, involving more provinces, 442 district hospitals, and community health centers with a wider context. 443