Group assessment of residents’ clinical skills using Case-based Discussions is feasible, highly valued and moreover fosters learning

Background Assessment of residents’ clinical skills using Case-based Discussion traditionally involves a one-to-one discussion with a supervisor. This entails a subjective, and maybe unequal, assessment of the resident, which can only be overcome with the use of multiple assessors. The purpose of this study was to explore residents’ and assessors’ perception of a group-assessment concept. Methods Eleven to fteen residents in a medical department met 4 times (every 3rd month) over 1 year for 5 hours each time. The residents took turns presenting clinical cases within a predened topic and discussed it with peers and assessors (diagnosis, differential diagnosis, ethical considerations etc.). Four assessors (specialists in internal medicine) participated in the discussion and together assessed the residents` clinical skills. An external consultant observed the group-assessment several times and conducted semi-structured interviews with the residents as well as the assessors. Notes from the observations and transcribed interviews were analyzed using an inductive approach looking for participants’ perceptions of the concept.


Background
Along with the implementation of competency-based education, workplace based assessment (WPBA) has been widely adopted (1,2). Although the validity and reliability of WPBA methods has been disputed, evidence for their utility seems well established for a number of assessment methods including Casebased Discussions (CbD) (1,(3)(4)(5). To enhance the e cacy of WPBAs, multiple factors should be taken into account.
The assessment should focus on competencies that are central to the activity being assessed, and the assessors who are best-placed to judge performance should be asked to do the assessments (6). To enhance learning, it is critical that provided feedback is consistent with the needs of the learner and focus on important aspects of the performance (while avoiding personal issues) (1,4,7). The reliability of WPBAs improves by increasing both the number of assessments and the number of assessors involved (1,6,8). The factors that enhance the utility and e cacy of WPBAs, have likewise been reported in relation to the use of CbDs (3)(4)(5)8). Furthermore it is suggested that the educational value and content of CbDs could be improved by providing ring-fenced time for these in both specialists' and residents' job plans, in order to provide su cient time for teaching and learning (4,(8)(9)(10).
A fairly new study exploring how CbDs were perceived by a UK cohort of medical trainees, demonstrated that most trainees appreciate the educational value of CbDs (11). However, more emphasis on planning these assessments is required and also on providing feedback that is both speci c and actionable (11). Similar conclusions has been made from other studies (4,10).
Based on knowledge regarding the requirements for successful performance of WPBAs and in particular CbDs, the idea for a new group-assessment concept took shape. The purpose of the group-assessment concept was to increase the quality of CbDs used for assessment of residents` clinical skills and to make sure that the assessments were performed in an operational manner, that could perhaps also be time and cost e cient.
In a medical department, the mandatory assessments through one-to-one CbDs were therefore experimentally replaced with group assessments.

Methods
The aim of this study was to evaluate the group-assessment concept and to explore the residents perception of the concept.

Setting of the study
In Denmark, where the study took place, outcome-based education with mandatory WPBA, based on the seven CanMEDS roles was implemented in 2004. In order to become a specialist, the physician must go through a 2-year internship followed by four or ve years of specialty training. All residents have an educational supervisor appointed for each rotation, who is responsible for the resident's educational program in that particular rotation. However, all specialists in a department are obliged to act as clinical supervisors in daily clinical practice, and may act as assessors of clinical skills.
This study took place in a large multidisciplinary medical department. Specialties represented are cardiology, endocrinology, rheumatology, gastroenterology, and pulmonology.

Study design
This study was based on observations of the group assessment sessions. All the residents and specialists, who participated in the sessions, were later invited for an interview (by a rst come rst served principle). Recruitment for interviews stopped when saturation of data was achieved. Six residents and four assessors were interviewed.

Set-up
The group-assessments were held every 3 rd month during a year, where every session was scheduled to last 5 hours. Participation in the group-assessments was mandatory for residents and scheduled in the doctors' job plans.
The schedule for the group-assessments was send out a year in advance and three months before every meeting, the residents were reminded about the 3-4 prede ned topics of the next session. The assessment group, who were medical specialists within the topic for the next session, were also invited to participate (Table 1). All residents at the medical department participated (eleven to fteen residents each time). The residents were all ve years or less from becoming medical specialists in rheumatology, endocrinology, cardiology, gastroenterology or pulmonology. Four assessors and a moderator who were all medical specialists also participated in each group-assessment. Author HS served as moderator at all meetings/sessions.

Data collection
An external consultant (an anthropologist) observed the group-assessment three times ( gure 1). The observations included observation of the structure and process, number of cases, the number of questions and comments, the placement of the participants, mutual conversations, use of mobile phones/ipads, length of breaks and so on. Field notes were taken.

Development of interview guide
From observation of the rst group-assessment, the anthropologist developed an interview guide [see Additional le 1].

Interviews
Thirty minutes semi-structured interviews using the interview guide were carried out. The interviews included topics such as experience with and attitude towards group assessment, comparison/experience with on-to-one assessment, structure of the group-assessment, preparation, outcome, case presentation and more.

Analysis
All the interviews and eld notes were transcribed verbatim. The data material was thematically categorized (13) according to the interview guide by the author RBN, who read the material and identi ed main themes and sub-themes. The categorization, themes and sub-themes were discussed with author HS until consensus was reached. Qualitative guidelines were followed to ensure transparency (14,15). Quotations that most accurately illustrated the sub-themes were selected and translated from Danish into English. Quotations are anonymized using pseudonyms. Table 2 shows the themes and sub-themes with examples of statements belonging to the themes. Theme Residents Assessors Overall perception of group assessment "I think it is a much better idea than the way it works other places. We have to have these signatures…." (resident 4) "I think it is a very big relief to have your competences approved in this manner, because the other way [one-to-one assessment] is chaotic and random." (resident 6) "After participation I felt very positive about the concept. I really think it was very rewarding in many ways. It was definitely something that gave new dimensions that I have not experienced in other places…" (assessor 1) Group assessment versus oneto-one assessment "There are advantages and disadvantages related to both types of assessment, but I think I would personally benefit most from the one-to-one assessment. There is nothing bad to say about the group assessment. I think [groupassessment] is a good way to have your competences approved." (resident 2) "It is academically very enriching that many fields of medicine are gathered. We come together and discuss which is the opposite of what my fellow residents will experience other places. I have earlier participated in one-to-one assessments where you just meet with your supervisor and look at a couple of medical records and then the supervisor says; "It looks good" and then there is no more discussion. ……So, I would definitely say that it is worthwhile to use 5 hours 4 times a year on group-assessments where there are many who provide me with feedback." (resident 4) "The one-to-one assessments are very shallow. It is something like; "Well, this looks good. Do you have any comments?" and "We have to have this signed". There was very little content in the oneto-one assessment and it was done because we had to, and not in order to achieve anything." (assessor 1) Outcome regarding learning, motivation and attitude "Well, the good thing is that we come from different fields of medicine, and the value of this is, that if you for instance present a case with a rheumatologic disease, then it is exciting to hear what the doctors in the field of rheumatology have to say about the case." (resident 5) "I look at it as a good opportunity to discuss common challenges. I mean certain types of patients and illnesses. So, I see it more as an opportunity for discussion, than an assessment of my competences." (resident 4) "….I am for instance a specialist in gastroenterology and I am a supervisor for a resident, but I also have to approve competences that are outside the field I normally deal with." (assessor 3) "I think it works well; you get around a lot of topics during the group-assessment…." (assessor 3) Context and structure regarding preparation, time used, case presentation and the moderator "The group-assessments are announced a long time ahead of time. You know what the topics are so you pay attention to them in your daily clinical work, if you encounter something exciting. So, you want to find a case with substance that you think is interesting for your colleagues to hear about." (resident 5) "I think the length of the session was very suitable. You can feel a bit tired in the end, but I don`t think it is a problem. I have tried to attend something in just five minutes where I have been more tired. It is after all a matter of how interesting it is." (resident 4) "There has been a tendency to pick good and exciting cases. However, in our field we could be better at presenting what is difficult or unclear, because that is what we often encounter in the daily clinical work…" "The second time I participated, it worked out really well. ……... The residents all came well prepared and we did too. Everyone was set on getting a really good discussion of the cases." (assessor 2) "I have been very positively surprised of how well prepared the residents have been, also in regard to background knowledge…" (assessor 4) (resident 1) "The cases that are to be presented, are cases that were challenging and difficult. …… It has to be a case where you were challenged while on night shift or at the ward round or something else, where you had a hard time solving the problem, and then you can present this issue." (resident 2) "The moderator makes sure that everyone gets to present and contribute to the discussion, and this assures that everyone is well-prepared for the session." (resident 4) "We talked a lot about how important it is to have someone to steer the meeting and that the moderator makes sure that everyone has contributed to the discussion, or when it is necessary to confront some of the residents who have been less active in the discussion." (assessor 4) "I think the whole concept depends on a really good moderator who can steer the meeting and make sure that the purpose of the groupassessment is met, which is that all residents get their competences approved." (assessor 4) Relation "When you come to a new department and join the groupassessment then it is also a good place for social networking, because you have five hours together…. you are in dialogue with the others and feel safe." (resident 1)

Results
Observations of the group-assessment by the external consultant are presented, followed by interview data, split into relevant sub-themes.
Observation of the group-assessment On the day of the group-assessment, each resident took turns presenting a clinical case within each of the day's topics (approximately 1 hour discussion for each topic). PowerPoint were not used, but patient data, such as blood samples (from the electronic journal), was sometimes presented on screen.
The cases were discussed with both peers and the assessors. The moderator kept track of time and made sure that all residents participated in the group discussion and the presentation of cases. The four assessors and the moderator together assessed the residents` skills and feedback was given during the discussion of the case. Notes and mandatory standard assessment forms were used for the assessment [see Additional le 2].
The external consultant was in no position to evaluate the medical content of the discussion, but it was evident that a case could result in discussion of many aspects such as social background, ethics, and collaboration besides core medical content. This generated numerous inputs from both residents and assessors. Many of the cases presented were known by many of the gathered doctors, increasing the possibility of viewing the case from different perspectives. There was a great variation in the number of doctors who took part in the discussion after each case presentation, and also a great variation in the amount of time used on each case discussion (from 2 to 20 minutes). There was also a great variety in how active the assessors were, especially in regards to follow-up questions for assessment of satisfactory clinical competences of the resident, suggesting different understanding of the role as an assessor, which was also con rmed during the interviews.
During the CbDs it was noticed that the moderator played a powerful role in assuring that all topics were discussed and that all residents participated in the case presentations and following discussions.
After each topic, the assessors and moderator, in privacy, discussed the performance of the residents, and came to a consensus on whether the competence should be approved. If a resident did not perform su ciently, a one-to-one assessment together with the educational supervisor would be arranged. The results of the assessment were communicated to both the resident and the educational supervisor at a later time in order to avoid humiliating the resident in a group forum.

Interviews
Preparing for the group-assessment and case presentation As the topics and dates for the group-assessments were send out a year in advance, with a reminder three month prior to the next group-assessment, the residents felt that they had plenty of time to gather and prepare cases; "You know what the topics are, so you pay attention if you encounter an interesting patient. You want to nd a case with many interesting aspects that you think will be of interest to all your colleagues." (resident 5) The residents remembered the cases by making notes about them and by reading the notes again directly before the group-assessment.
The assessors were a bit nervous about what to expect prior to the rst group-assessment, but found there was no need for preparation; "The rst time I was unsure about what to expect, but I think it went well… I found out that I knew the things I should know, and it did not require any great preparation." (assessor 3) The residents` themselves picked out the cases they wanted to present. Therefore, as some pointed out, there could be a tendency to pick out cases where they felt con dent of own knowledge. Several residents stated that there might be a tendency to pick out cases that were rare and intriguing; "There has been a tendency to pick good and exciting cases. However, in our eld we could be better at presenting what is di cult or unclear, because that is what we often encounter in the daily clinical work….." (resident 1) Other residents, on the contrary, stated that they were more likely to present di cult cases where they had been in doubt of what to do. Yet another resident stated that she chose to present a case within her own specialty that she thought it was important for all doctors to have a minimum of knowledge about.

Assessment, resources, and approval of competences
A repeated statement was that it could be di cult to assess all the residents at the same time. One might hide in a group, not necessarily because he/she did not have the required competences. This could be due to personality, as some are more introvert and timid/reserved, while others are extrovert and like to be heard; "You are in a forum, where one might feel a bit exposed. You are together with specialists from the department, and everybody has that basic fear. One would not want to present oneself as professionally ignorant." (resident 1) Many doctors acknowledged the ability of the moderator to steer the meeting, and in making sure everyone was heard. The length of the group-assessment was regarded as suitable and many emphasized that the meetings were relaxed with a good and safe environment, which was not exam-like. Since there were several assessors, many thought that the assessment was more consistent.
All though it took time to plan the group-assessments, both the residents and assessors felt that they were more resource-e cient and manageable than the one-to-one assessments.
As the meetings were scheduled during working hours it was highly appreciated that the groupassessment concept made it simple and straightforward to have the competences approved. A resident pointed out that formerly having competences approved often became a hunt for signatures, often without proper assessment and feedback; "I think this is a much better idea, than the way it works in other places. We have to have these signatures, there is a terrible lot of signatures…So it is a bit of a hunt for signatures, without anyone really going into the depth with the different things." (resident 4)

Professionalism, learning and interdisciplinarity
The residents especially appreciated the professional discussions in an interdisciplinary environment. It was found to be professionally enriching and educational to meet with doctors from other specialties to discuss relevant common relevant challenges. It was also mentioned by the residents that the groupassessments gave an insight into of the other doctors` skills, which could be used as an inspiration to become just as skilled; "You get a better insight into the other doctors skills, and this may be used as an inspiration to become just as skilled as the others." (resident 5) For the assessors, the assessment process had a greater focus than the professional discussions, but they also acknowledged that all doctors gained more knowledge from the group-assessments compared to the one-to-one assessments. The extra time to discuss cases was highly appreciated. As one assessor put it; "…. I think we all felt it was a luxury to have the time for discussion. Because we all learn from each other, no matter if you are a resident or an expert. But the expert is not an expert in all elds of medicine, so in that way it is of mutually bene t, although the assessment is about them [the residents]. In that way I think they get a wider knowledge compared to the one-to-one assessments." (assessor 2) Both residents and assessors thought that the group-assessments were rewarding for all. Most participants looked forward to the next group-assessment, because they gained new knowledge and insight during the meetings. The group-assessments made it possible to discuss more cases (around 24 cases in a session) compared to the number of cases covered at the one-to-one assessments (usually three to four cases). Furthermore, the presence of both residents and assessors from different specialties provided more perspectives and interdisciplinary knowledge. One resident thought the concept was a stroke of genius; "The fundamental idea to meet four times a year and have the competences assessed in a forum where residents and specialists are gathered, is a stroke of genius. Because it shows that this is something that needs to be done, and there is a setting and a deadline for the assessment." (resident 1) Overall perspective of the group-assessment concept Eight out of the ten doctors who were interviewed, preferred the group-assessment concept prior to the one-to-one CbDs, while two residents preferred the individual one-to-one assessments, given that there was enough time for the case discussion; "I think I would personally bene t most from the one-to-one assessment. But it [group-asssessment] is a good way to have your competences assessed." (resident 2) The residents acknowledged the value of individual supervision with their educational supervisor, but for the approval of core internal medicine competences, the group-assessment concept was found superior by most.
Many of the doctors therefore suggested that the group-assessment concept should be tried out and subsequently maybe implemented at other departments, as a replacement for mandatory one-to-one CbD assessments of key competences.

Discussion
By creating a group-assessment concept it seems that many of the obstacles associated with one-to-one CbDs for assessment of clinical competences can be eliminated. The group-assessment concept provides ringfenced time for the assessment process and by including many assessors, subjectivity and bias in the assessment process is likely to be reduced. Furthermore, the gathering of assessors and residents from different specialties, raises the level of discussion, and both residents and assessors nd that the additional learning process in the assessment is improved.
Dedicating 4 × 5 hours for group assessment with the gathering of many of the doctors in the department, may seem like an unaffordable solution. It requires planning ahead of time to make sure the residents have the time to nd cases, but also to make sure that the residents and specialists are not scheduled to other tasks. This de nitely requires an open-minded head of the department, willing to invest the time on education. However, the investment seems to be worthwhile since both residents and specialists in the department report increased learning when using this new concept in assessment. Thus, the introduction and implementation of this group-assessment concept is an example of how complex changes in health care succeed if you accept local solutions. Very recently, Dagnone et al (16) called for more freedom in the interpretation and implementation of competency-based medical education to make it t locally. The group-assessment concept presented here might be an inspiration to other departments and countries.
By participation of multiple assessors in the group-assessments, the risk of bias is likely to be reduced, due to the reduced in uence of personal relations (17). Furthermore, assessors from different elds of internal medicine, emphasize different aspects of cases. For instance, the emphasis may be quite different when it comes to a discussion of "dizziness", whether you are a cardiologist or a geriatrician. A broader view of the cases and the required competences is therefore gained.
The assessors understood their role as assessors differently, and perhaps the use of assessors who were already familiar with the concept, could further improve the group-assessment (18). However, the mutual discussions among the assessors regarding whether or not a resident had the necessary competence to get approval in it-self contributed to the assessors knowledge and skills as assessors and lead to a common understanding of the concept of CbD, and how it is to be used in this speci c department. The need to be trained as an assessor has been recommended by several authors (19)(20)(21). The introduction of this group-assessment concept thus provided a learning opportunity for the assessors regarding the use of CbD. Whether this led to an increase in the quality of other assessments in the department was beyond the scope of this study. However, the general agreement on pass/fail level obtained through the discussions among the assessors does lead to a higher uniformity and probably makes the assessments more fair.
The residents themselves chose the cases that they wanted to present at the group-assessment.
Therefore, as pointed out by the residents, they might present cases, where they had performed well, and therefore the CbDs might not alone give a reliable picture of the residents competences.
One of the di culties reported by the assessors was the variation in the residents' contribution to the discussion, which might lead to the more introvert residents being overlooked. All assessors stressed the important role of the moderator in making sure all residents were active. Besides this, the moderator should be able to steer the conversation and ensure group effectiveness. These are all competences ascribed to a good facilitator (22). In the opinion of the assessors the moderator plays a crucial role for the success of the concept.
To our knowledge similar studies regarding group-assessment of residents` clinical skills has not been made, but the concept may resemble practice-based small group learning (PBSGL) where groups of doctors gather to discuss cases from daily clinical work, and where case presentations are often followed by topic review and discussion of the related evidence-based medicine articles to identify implications for practice changes (23). The concept is widely used by general practitioners and seems to be a promising method of continuing professional development (23). In this study it was found that the groupassessment concept provided an increased learning experience for both residents and assessors and perhaps similarly also contributed to a continued professional development of the assessors.
Limitations A limitation to the study was that residents and assessors volunteered for the interview, and therefore doctors who were either very fond of or dissatis ed with the group-assessment, might be more likely to volunteer. This might have given either more positive or more negative results. However, in the interviews both positive and negative evaluations of the concept appeared. Thus the results seem to re ect reality.
Another limitation is that only one of the authors of this paper performed the data analysis. However, the categories and themes were discussed and agreed upon by two of the authors (HS, RBN), who also attended all the group-assessment sessions.
Furthermore, it is di cult to consider the generalizability of the results from this study, since only one department participated. It might not be possible to implement group-assessment in all departments or in all specialties. However, most specialties have general competences and many specialties use CbD in their assessment program, therefore it might be interesting to try out the concept in other specialties and departments.

Conclusion
The group-assessment concept offers an acceptable, feasible and e cient model for CbD used as a formative assessment of residents´ competences in internal medicine. It reduces the effect of interpersonal issues between resident and supervisor and thereby minimizing bias. It provides the busy clinicians with protected time to engage in teaching/assessment activities. The amount of knowledge, skills, input and inspiration grows with the number of residents and medical specialists. Thus, groupassessment serves as a tool to assess clinical skills, but also provides learning for all the participating doctors in the department.
The group-assessment concept with the goal of assessment of residents` competences, alongside mutual learning, could serve as an inspiration for other departments and specialties.

Abbreviations
CbD: Case-based Discussion WPBA: Workplace based assessment