Fitness-for-purpose of the CanMEDS competencies for workplace-based assessment in General Practitioner’s Training: a Delphi study

doi:10.21203/rs.3.rs-1652676/v1

Download PDF

Research Article

Fitness-for-purpose of the CanMEDS competencies for workplace-based assessment in General Practitioner’s Training: a Delphi study

https://doi.org/10.21203/rs.3.rs-1652676/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 01 Apr, 2023

Read the published version in BMC Medical Education →

You are reading this latest preprint version

Background

In view of the exponential use of the CanMEDS framework along with the lack of rigorous evidence about its applicability in workplace-based medical trainings, further exploring is necessary before accepting the framework as valid and reliable competency outcomes for postgraduate medical trainings. Therefore, this study investigated whether the CanMEDS key competencies could be used as outcome measures for assessing trainees’ competence and for supporting competency growth across training phases in a workplace-based General Practitioner’s (GP) Training setting.

Methods

A web-based Delphi study was employed to collect validity evidence. In three Delphi rounds, a panel of experts (n = 25–43) was asked to rate on a 5-point Likert scale whether the CanMEDS key competencies were feasible for workplace-based assessment, and consistent for assessment across different training phases. Comments on each CanMEDS key competency were encouraged. Descriptive statistics of the ratings were calculated, while content analysis was used to categorise panellists’ comments.

Results

Out of twenty-seven CanMEDS key competencies, 21 were scored as feasible, and 16 as consistent across training phases for assessment in the workplace. All the key competencies under the “Medical Expert”, “Communicator”, and “Collaborator” roles reached consensus for feasibility of assessment. The panel also agreed on one out of four key competencies under the role “Leader”, one out of two competencies under the role “Health Advocate”, three out of four competencies under the role “Scholar”, and three out of four competencies for the role “Professional”. Regarding consistency of assessment in the workplace, consensus was achieved for four out of five competencies under “Medical Expert”, three out of five competencies under “Communicator”, two out of three competencies under “Collaborator”, one competency out of two under “Health Advocate”, three out of four competencies under “Scholar”, three out of four competencies under “Professional”. No competency under the role “Leader” deemed to be consistently assessed across training phases.

Conclusions

The findings indicate a mismatch between the initial intent of the CanMEDS framework and its applicability in the context of workplace-based assessment. Although the CanMEDS framework could offer starting points, further contextualization of the framework is required before implementing in postgraduate medical trainings.

competency framework

CanMEDS

Delphi methodology

General Practice

postgraduate training

In an era of greater accountability, the need for reassurance of medical competence among health professionals and trainees has revived the attention to competency-based medical education (CBME).[1] This outcome-based approach provides fruitful ground for individual, programmatic, and institutional changes in order to align medical curricula with societal and patient expectations.[2] Implementation of CBME could offer guidance and direction to learners, and, at the same time, transparency and accountability to patients and to the general public. Following the International CBME Collaborators, competence designates an array of physicians’ abilities, whereas competency defines an observable and measurable physicians’ ability.[1] In CBME, predefined competencies and outcomes that go beyond knowledge and clinical reasoning are a prerequisite to meet high quality of patient care.[3]

Medical educators have repeatedly attempted to define competencies for CBME. The result of these attempts is the creation of different competency frameworks for postgraduate medical education, like the Canadian Medical Education Directives for Specialists (CanMEDS), which is currently the most widely accepted framework.[4, 5] The CanMEDS framework identifies and describes different outcomes as competencies that physicians should acquire to follow patient-centred care. These competencies are thematically grouped under seven different roles: medical expert, communicator, collaborator, leader, health advocate, scholar, and professional.[4] The framework also splits competencies into two levels, level one contains the key competencies and level two the enabling competencies. The two levels together provide a multifaceted sum of descriptors to comprehend of what is expected by future physicians.[6] By encompassing various aspects of outcomes, the CanMEDS roles provide a comprehensive analytic framework for learning, teaching, and assessment in medical curricula.[6]

In a workplace-based curriculum, the CanMEDS competencies could be captured through different workplace-based assessment tools.[7] However, to increase implementation and acceptance of CBME in postgraduate medical training, the CanMEDS framework not only need to align with and accommodate the purposes of workplace-based assessment, but also to support competency growth over time.[3, 8] Although context dependency and relevance have been demonstrated in the literature before, there is a lack of evidence about the extent to which the CanMEDS key competencies could be applied as valid and reliable outcome measures in a workplace-based postgraduate medical training. [9, 10, 11] Therefore, this study aimed to investigate, firstly, whether the CanMEDS key competencies could be assessed in the workplace, and, subsequently, whether they could be assessed in a consistent way across training phases to support growth. We collected evidence about content validity the CanMEDS competencies in the Flemish General Practitioner’s (GP) Training, in Belgium.[12]

We employed a web-based Delphi survey to gather validity evidence based on consensus ratings on which CanMEDS key competencies could be feasible and consistently assessed for workplace-based assessments in the GP Training.[13-15] Based on available literature, we discussed and decided on the necessary steps to ensure methodological rigor. Table 1 provides an overview of designing steps, based on the Guidance on Conducting and REporting DElphi Studies (CREDES) guideline.[16] We further elaborate on our methodological decisions considering the CREDES design steps.

Table 1 Steps for designing a Delphi study based on the CREDES guideline.

• Defining the purpose of the Delphi study

• Definition of Delphi round

• Definition of (non) consensus

• Selection of expert panel

• Development and pilot of Delphi instrument

• Guidelines on interpreting results and proceeding between the rounds including informational input for experts

• Role of research team to prevent bias

• Strategies to improve response rate

Study design

We chose to employ an e-Delphi to recruit panellists from diverse geographic locations within Flanders and to reach a larger group in a cost-efficient way(Gill, Leslie, Grech, & Latour, 2013; Graham, Regehr, & Wright, 2003). The online form was also preferred since this study took place during the COVID-19 pandemic. We defined feasibility as what can be measured in the workplace and whether the competency formulation is suitable for workplace-based assessment. We defined consistency as what can be consistently measured across different training phases in the workplace (Figure 1).[13-15] Consensus was defined as 70% of respondents agreed or strongly agreed that an item was feasible and consistent for assessment in the workplace.[17] Non consensus was defined as no major change in consensus ratings nor any suggestions for change by the panel after 2 rounds.

To guarantee the reiterative nature of our study, we decided to set a minimum number of three rounds.[18] After each Delphi round, when consensus was achieved for a CanMEDS key competency, the latter was no longer offered for evaluation. Although the traditional Delphi methodology commences with an unstructured round, we chose to follow a semi-structured approach, since our main research goal was to validate the predefined CanMEDS framework.[4] Therefore, we used a combination of closed and open-ended questions.[19]

In the first round, panellists were asked to evaluate the CanMEDS key competencies as feasible and as consistent based on a 5-point Likert scale. They were also able to provide qualitative comments for each key competency.[7, 14] In the second round, we informed the panellists about the consensus ratings of round 1. In this round, panellists were asked to formulate concrete suggestions for adjustments and rate the two research criteria. A document was also added addressing the issues that arose in round 1 based on qualitative remarks. To provide some clarity about the formulation of the competencies, the CanMEDS enabling competencies of each key competency were provided to assist the panel with their suggestions.

To answer to some of the remarks, we added the enabling competencies that corresponded to each key competency. Additionally, we listed and categorized the most frequent qualitative comments providing an overview. Decisions about modifications on key competencies were clearly communicated. We asked the panel again to evaluate the CanMEDS key competencies as feasible and as consistent for workplace-based assessment based on 5-point Likert scale. In the third round, we provided summaries of the ratings from the previous rounds. After panellists’ request, we included a list of examples of how each CanMEDS key competency would transfer to the workplace. In this final round, we asked the panellists whether they agree or not that a CanMEDS key competency was feasible and consistent for assessment in the workplace. If not, they were required to specify the reasons for abstaining consensus.[15] Figure 2 shows an overview of each Delphi round.

Study setting

To create a coherent approach across Flanders, four Flemish universities (KU Leuven, University of Ghent, University of Antwerp, and the Flemish Free University of Brussels) have created an interuniversity curriculum for the GP Training which consists of three phases. Practical coordination and decision-making regarding the curriculum are the responsibility of the Interuniversity Centre for GP Training (ICGPT). The ICGPT is responsible among others for allocating clinical internships, organizing exams, arranging fortnightly meetings of GP trainees with tutors, and handling trainees’ learning portfolios where evaluation of competencies is registered.

Selection of panel

To select panellists, we followed a purposive sampling.[13, 20] We set three selection criteria: 1) sufficient experience as a GP (>3 years of experience), 2) experience in mentoring and assessing trainees in the workplace, 3) sufficient time and willingness to participate.[7, 21] To incorporate a wide range of opinions, the panel consisted of both GP trainers and GP tutors.[22] GP trainers were workplace-based trainers assisting trainees during their internship, while GP tutors were associated with a university providing guidance and facilitating peer learning and support (10-15 trainees per group) twice monthly. Both groups were responsible for assessing trainees in the workplace. Panellists resided in different provinces of Flanders to minimize converging ideas and to ensure reliability.[13, 22] Although there is no consensus about an appropriate sample size for a Delphi design, a number of 15-30 panellists could yield reliable results.[22, 23] In our study, we selected panellists that had received the same medical background and hold general understanding in the field of interest. In addition, to determine sample size, we took into consideration feasibility parameters to obtain a good response rate, such as providing large timespans for each Delphi round and reasonable required time to completion.

Development and pilot of Delphi survey

The 27 CanMEDS key competencies were translated from English to Dutch, because the panel was Dutch speaking. Figure 3 graphically illustrates how the Delphi survey was constructed. First, the CanMEDS competencies were translated by five researchers separately.[24] After discussing and evaluating all translations, we decided to keep the Dutch translation as close as possible to the original English framework. Secondly, to validate the translation and pilot the instrument, we sent it to a group of medical professionals to comment on it. Thirdly, once feedback was received and the Dutch translation was finalized, the Dutch version of the framework was back translated to English to confirm the accuracy of the translation.[25]

Every Delphi round consisted of an introductory part, the CanMEDS key competencies, and an ending section. In the introduction, the purpose of each round was explained, and decision rules were communicated. We added the ending section to provide space to the panel for communication and feedback not related to the CanMEDS key competencies (e.g., necessary time to completion, remarks about layout). To avoid confusion among the different CanMEDS roles, the key competencies were grouped per role. Figure 4 illustrates how the survey items were displayed prior to any consensus had been reached.

Data collection and analysis

To collect our data, we used the Qualtrics XM Platform. This online tool allowed for maintaining anonymity among the panellists.[26] A personal link was sent via email to each panellist. This allowed following-up response rates and sending reminders to specific members. Due to the workload caused by the COVID-19 pandemic, each round lasted four weeks. We opted for a flexible approach towards the panellists to increase the response rate of each round. Reminders were weekly sent to members that had not completed the survey.[26] Data collection took place between October 2020 and February 2021. For analysing quantitative data, we calculated descriptive statistics of every item using SPSS 27 (IBM SPSS Statistics 27). We used Microsoft Excel to list and categorize qualitative data. Panellists’ comments were anonymously and literally registered. For analysing qualitative data, we used content analysis.[27]

Role of research team to prevent bias

Methodological decisions made by the research team were in line with the available literature. We predefined and stipulated methodological steps before commencing the study. We applied, monitored, and evaluated these steps during the study. The results of each round were discussed by the research team, while qualitative data were interpreted by two researchers for researcher triangulation.[28]

Initially, 53 GP trainers and GP tutors chose to participate in the panel. Forty-three out of 53 responded in the first Delphi round (response rate of 89.4%). Twenty-five out of 43 were female, while 18 were male (Table 2). Most of the panellists (n=27) were 36 years old or older and mainly GP trainers (n=31) with at least 5 years of experience (n=27) in guiding GP trainees in the workplace, and at least 5 years of experience (n=26) in assessing GP trainees in the workplace. The second Delphi round had a response rate of 88.9% (n=33), while the third round had a response rate of 76.12% (n=24). Panellists who chose to withdraw their participation attributed this to the COVID-19 related increase of workload in their clinical practice. Table 3 and Table 4 provide an overview of the level of consensus per Delphi round for feasibility and consistency of assessment respectively. An additional file presents the number of comments on feasibility and consistency of assessment per CanMEDS key competency, per role, and per Delphi round [see Additional file 1].

Table 2 Composition of panel

	N
	43
Sex
Male	18
Female	25
Age
25-35 years old	4
36-45 years old	12
46-55 years old	14
56 years old or older	13
Role in the GP Training
GP trainer	31
GP supervisor	6
Both	6
Flemish region
West Flanders	7
East Flanders	13
Brussels	2
Antwerp	10
Flemish Brabant	6
Limburg	5
Years as GP
Between 3 and 5 years	1
Between 6 and 10 years	6
More than 10 years	36
Years of experience in guiding GP trainees
Less than 3 years	4
Between 3 and 5 years	12
Between 6 and 10 years	12
More than 10 years	15
Years of experience in assessing GP trainees in the workplace
Less than 3 years	5
Between 3 and 5 years	13
Between 6 and 10 years	11
More 10 years	14

Delphi round 1

Five out of 27 CanMEDS competencies reached a positive 70% consensus rate for feasibility and 3 competencies for consistency (Table 3 and Table 4 respectively). In total, the panellists gave 154 qualitative comments, 130 comment about feasibility and 24 about consistency of assessment. Those comments were clustered into six main categories: 1. CanMEDS key competency not suitable for workplace-based assessment via a portfolio (n=46), 2. Vague formulation of CanMEDS key competency (n=19), 3. Assessment of CanMEDS key competency dependent on phase in GP Training (n=16), 4. CanMEDS key competency not applicable during GP Training (n=15), 5. Assessment of some CanMEDS key competencies dependent on location of clinical practice and patient population (n=12), and 6. Assessment of CanMEDS key competency dependent on trainer and/or trainee (n=12). The pie charts (Figure 5) display the distribution of the panellists’ remarks in Delphi round 1 per research criterion.

Table 3 Consensus ratings on feasibility of the CanMEDS competencies per Delphi round

CanMEDS Roles	CanMEDS competencies	Level of consensus: Is this CanMEDS competence feasible for assessment in the workplace?
CanMEDS Roles	CanMEDS competencies	No1	No2	No3
MEDICAL EXPERT The GP trainee is able to:	1. Practice medicine within their defined scope of practice and expertise	71.4%
	2. Perform a patient-centred clinical assessment and establish a management plan.	71.4%
	3. Plan and perform procedures and therapies for the purpose of assessment and/or management.	71.4%
	4. Establish plans for ongoing care and, when appropriate, timely consultation.	57.1%	100%
	5. Actively contribute, as an individual and as a member of a team providing care, to the continuous improvement of health care quality and patient safety.	40%	56%	72.2%
COMMUNICATOR The GP trainee is able to:	1. Establish professional therapeutic relationships with patients and their families.	38.2%	87.5%
	2. Elicit and synthesize accurate and relevant information, incorporating the perspectives of patients and their families.	51.5%	66.7%	83.3%
	3. Share health care information and plans with patients and their families.	53%	91.7%
	4. Engage patients and their families in developing plans that reflect the patient’s health care needs and goals.	41.2%	52.1%	77.8%
	5. Document and share written and electronic information about the medical encounter to optimize clinical decision-making, patient safety, confidentiality, and privacy.	54.3%	79.1%
COLLABORATOR The GP trainee is able to:	1. Work effectively with physicians and other colleagues in the health care professions.	45.7%	87.5%
	2. Work with physicians and other colleagues in the health care professions to promote understanding, manage differences, and resolve conflicts.	34.3%	54.2%	88.9%
	3. Hand over the care of a patient to another health care professional to facilitate continuity of safe patient care.	48.5%	62.5%	83.4%
LEADER The GP trainee is able to:	1. Contribute to the improvement of health care delivery in teams, organizations, and systems.	34.3%	45.9%	55.6%
	2. Engage in the stewardship of health care resources.	33.9%	50%	72.2%
	3. Demonstrate leadership in professional practice.	42.8%	50%	50%
	4. Manage career planning, finances, and health human resources in a practice.	28.6%	25%	50%
HEALTH ADVOCATE The GP trainee is able to:	1. Respond to an individual patient’s health needs by advocating with the patient within and beyond the clinical environment.	38.9%	70.8%
HEALTH ADVOCATE The GP trainee is able to:	2. Respond to the needs of the communities or populations they serve by advocating with them for system-level change in a socially accountable manner.	16.7%	25%	44.5%
SCHOLAR The GP trainee is able to:	1. Engage in the continuous enhancement of their professional activities through ongoing learning.	80.6%
	2. Teach students, residents, the public, and other health care professionals.	36.1%	54.2%	61.1%
	3. Integrate best available evidence into practice.	77.7%
	4. Contribute to the creation and dissemination of knowledge and practices applicable to health.	52.7%	58.30%	77.8%
PROFESSIONAL The GP trainee is able to:	1. Demonstrate a commitment to patients by applying best practices and adhering to high ethical standards.	63.9%	70.9%
	2. Demonstrate a commitment to society by recognizing and responding to societal expectations in health care.	31.5%	33.3%	44.5%
	3. Demonstrate a commitment to the profession by adhering to standards and participating in physician-led regulation.	50%	78.2%
	4. Demonstrate a commitment to physician health and well-being to foster optimal patient care.	38.9%	43.5%	88.9%

Table 4 Consensus ratings on consistency of the CanMEDS competencies per Delphi round

CanMEDS Roles	CanMEDS competencies	Level of consensus: Is this CanMEDS competence consistent for assessment in the workplace?
		Round 1	Round 2	Round 3
MEDICAL EXPERT The GP trainee is able to:	1. Practice medicine within their defined scope of practice and expertise.	65.7%	84%
	2. Perform a patient-centred clinical assessment and establish a management plan.	71.4%
	3. Plan and perform procedures and therapies for the purpose of assessment and/or management.	77.2%
	4. Establish plans for ongoing care and, when appropriate, timely consultation.	54.3%	88%
	5. Actively contribute, as an individual and as a member of a team providing care, to the continuous improvement of health care quality and patient safety.	48.5%	60%	55.6%
COMMUNICATOR The GP trainee is able to:	1. Establish professional therapeutic relationships with patients and their families.	51.5%	75%
	2. Elicit and synthesize accurate and relevant information, incorporating the perspectives of patients and their families.	57.6%	50%	66.7%
	3. Share health care information and plans with patients and their families.	52.9%	79.2%
	4. Engage patients and their families in developing plans that reflect the patient’s health care needs and goals.	41.2%	39.1%	44.4%
	5. Document and share written and electronic information about the medical encounter to optimize clinical decision-making, patient safety, confidentiality, and privacy.	54.2%	79.2%
COLLABORATOR The GP trainee is able to:	1. Work effectively with physicians and other colleagues in the health care professions.	51.4%	87.5%
	2. Work with physicians and other colleagues in the health care professions to promote understanding, manage differences, and resolve conflicts.	40%	54.1%	66.7%
	3. Hand over the care of a patient to another health care professional to facilitate continuity of safe patient care.	54.3%	62.5%	83.3%
LEADER The GP trainee is able to:	1. Contribute to the improvement of health care delivery in teams, organizations, and systems.	42.8%	37.5%	55.6%
	2. Engage in the stewardship of health care resources.	37.2%	45.8%	66.6%
	3. Demonstrate leadership in professional practice.	45.7%	54.2%	54.2%
	4. Manage career planning, finances, and health human resources in a practice.	48.6%	20.9%	50%
HEALTH ADVOCATE The GP trainee is able to:	1. Respond to an individual patient’s health needs by advocating with the patient within and beyond the clinical environment.	42.9%	58.3%	77.8%
	2. Respond to the needs of the communities or populations they serve by advocating with them for system-level change in a socially accountable manner.	13.9%	25%	38.9%
SCHOLAR The GP trainee is able to:	1. Engage in the continuous enhancement of their professional activities through ongoing learning.	77.7%
	2. Teach students, residents, the public, and other health care professionals.	36.1%	50%	72.2%
	3. Integrate best available evidence into practice.	83.4%
	4. Contribute to the creation and dissemination of knowledge and practices applicable to health.	47.2%	54.1%	61.1%
PROFESSIONAL The GP trainee is able to:	1. Demonstrate a commitment to patients by applying best practices and adhering to high ethical standards.	72.2%
	2. Demonstrate a commitment to society by recognizing and responding to societal expectations in health care.	28.6%	30.4%	44.5%
	3. Demonstrate a commitment to the profession by adhering to standards and participating in physician-led regulation.	52.8%	78.2%
	4. Demonstrate a commitment to physician health and well-being to foster optimal patient care.	38.8%	43.5%	83.4%

Delphi round 2

In the second Delphi round, the panel was asked to rate 22 CanMEDS key competencies for feasibility and 23 for consistency of assessment in the workplace. As seen in Table 3 and in Table 4, the panel reached consensus for 8 out of 22 CanMEDS key competencies for feasibility and for 8 out of 23 for consistency of assessment. Overall, the consensus rates increased. However, a key competency (n°4) under the role “Leader” had lower scores on both feasibility and consistency comparing to the ones from round one (Table 3 and Table 4). Additionally, key competency n°1 under “Leader” scored lower on consistency than in round 1, while consensus rates on key competency n°2 under “Communicator” for consistency also decreased in round 2. No adjustments were made to the formulation since no panellists’ comments suggested any alterations.

In total, 117 remarks were given by the panel that were listed in four main clusters: 1. Vague formulation (n=48), 2. CanMEDS key competency not applicable in GP Training (n=13), 3. Suggestions for operationalization (n=14), and 4. CanMEDS key competency could only be assessed by GP trainer (n=10). Two comments were about the fact that assessment of some key competencies was dependent on the location of clinical practice or the context, one comment was about dependency of assessment on training phase, while two comments suggested some overlap between key competencies n°3 and n°4 under the “Medical Expert” role. Fifteen comments were deemed as not relevant and, therefore, categorized as others. Figure 6 depicts the distribution of the panellists’ comments in Delphi round 2.

At this point, despite the lack of concrete suggestions, low consensus levels on the key competencies under the “Leader” role led to adjusting three out of four key competencies. More adjustments were made to key competency n°2 under “Health Advocate” to make it more comprehensible, while key competency n°2 under “Scholar” was also modified to emphasize the importance of educating colleagues in the same discipline. Additional file 2 illustrates in detail the CanMEDS competencies that were adjusted [see Additional file 2]. These adjustments were based on all the data after two Delphi rounds and after discussions within the research team.

Delphi round 3

The third Delphi round included 12 CanMEDS key competencies to be rated for feasibility of assessment, and 15 to be rated for consistency of assessment in the workplace. Out of 12 competencies included in this round, 6 deemed feasible, while 4 out 15 deemed consistent. Panellists made 39 comments about the importance of the CanMEDS competencies for the workplace and suggestions about the way of assessing these competencies in the clinical workplace. Panellists did not agree on 6 CanMEDS key competencies about feasibility and on 11 key competencies about consistency. No further round deemed necessary since panellists’ comments did not provide any insights for further modifications.

The aim of this study was to collect evidence about the content validity of the CanMEDS key competencies when utilized for workplace-based assessment by employing an e-Delphi study. The CanMEDS competency framework is well-known globally and has increasingly been used in postgraduate medical education.[6] However, its applicability seems to be dependent on the context and on medical specialty.[9, 29] This study focused on validity evidence based on two criteria, feasibility of assessment and consistency of assessment across training phases embedded in a workplace-based postgraduate GP training. Our findings also show issues related to the fitness-for- purpose of the CanMEDS key competencies, and to the extent that they can be used for assessment purposes throughout the course of a postgraduate training.

Regarding the assessment feasibility, our research indicates that not all CanMEDS key competencies could be clearly related to observable and measurable behaviour. Some CanMEDS key competencies under the “Leader”, “Health Advocate”, and “Professional” role got notably lower scores (< 50%) compared to competencies under other CanMEDS roles, such as “Medical Expert”, “Communicator”, and “Scholar”. The big discrepancies in the rating scores might imply that the panel experienced difficulties in associating how those CanMEDS competencies could be transferred and translated into assessment activities in the workplace. This is in line with previous research showing difficulties in applying and using the roles of “Health Advocate and “Leader” in undergraduate medical curricula.[7, 30] Panellists seem to be more familiar with competencies focusing on medical knowledge, clinical and communication skills rather than with competencies with societal and public engagement. Clearly, the predominance of clinical and communication skills apparently reflects the role of a GP in primary health.[31] However, medical educators should consider that public education and engagement is a prerequisite for a more equitable and patient-centred healthcare system.[32]

Our exploration of consistency across training phases of the CanMEDS key competencies yielded unanticipated results as well. Similar patterns as in the ratings of assessment feasibility were also found for those regarding assessment consistency of the CanMEDS key competencies. Strikingly, almost half of the CanMEDS key competencies seemed unsuitable for being assessed in a consistent way in the clinical workplace. Remarkably, none of the key competencies under the “Leader” role was deemed as consistent. It may be possible that the panel may have reported consistency based on their role in the GP Training. They seemed to pay attention on which conditions (e.g., comments under “Practice/context dependent”) and by whom (e.g., comments under “only by trainer”) each competency could be assessed. Our results suggest a potential mismatch between the CanMEDS key competencies and specific training phases. This problematic might indicate a dissociation between the CanMEDS framework and supporting competency growth across different training phases.

Our findings suggest that there is a gap between the initial intent of the CanMEDS competency framework and its applicability when it comes to workplace-based assessment.[11] Although the importance of the framework was reinstated by the panel, the difficulty of reaching consensus throughout the three Delphi rounds elucidates issues regarding applicability of the CanMEDS in the clinical workplace. The CanMEDS key competencies might offer a starting point for structuring assessment in the workplace and capturing medical competence. Nevertheless, further refinement and contextualisation of the framework is necessary to assist observations and measurements of trainees’ behaviour involving all seven CanMEDS roles during clinical practice. Future research should explore applicability issues in different health care contexts and settings to gather more evidence on the content validity of the CanMEDS framework.

Limitations

We acknowledge several limitations of this e-Delphi study. Notably the fact that the CanMEDS framework was translated in Dutch might have caused some degree of linguistic bias. Since language arguably carries cultural associations and meanings, an accurate translation of the original framework could presumably miss cultural elements inherent to the Flemish GP Training. However, choosing to stay as close to the original framework as possible implies that our findings could be generalized and used in other international contexts. Another limitation is that the panel might have kept in mind a specific interpretation of the CanMEDS framework, for example how they understand its applicability in the clinical workplace. A final limitation that needs to be acknowledged is the high dropout rate, since 43 panellists had initially agreed to participate. Unfortunately, the begin of our study coincided with the onset of a COVID-19 wave. Consequently, the increased workload of clinical practice did not allow some panellists to further participate in this study.

This study aimed to gather evidence on the content validity of the CanMEDS key competencies for workplace-based assessment purposes. Given that CBME is increasingly implemented across the globe, the findings provide some insight into the applicability of the CanMEDS framework and its fitness-for-purpose for assessment in the workplace. Drawing on the results of this e-Delphi study, adapting and adopting the current CanMEDS key competencies should be considered before implementation. The challenge for medical educators relates to how to encompass and capture in a workplace-based learning environment aspects of medical competence besides medical knowledge, and clinical and communication skills. Lastly, for CBME to pertain, more attention should be paid on how the CanMEDS competencies could be used as educational outcomes across different training phases. There is a need for rigorous evidence on how the CanMEDS framework can enable and support competency growth in the workplace.

GP: General Practitioner,

CBME: competency-based medical education,

CanMEDS: Canadian Medical Education Directives for Specialists,

CREDES: Guidance on Conducting and REporting DElphi Studies guideline

ICGPT: Interuniversity Centre for GP Training

Ethics approval and consent to participate

All participants had to actively consent by signing an informed consent form before participating in the study. Additionally, this study has been approved by the Social and Societal Ethics Committee (SMEC) KU Leuven (G-2020-1726-R2(MIN)).

Consent for publication

Participants actively consented for data publication when signing the informed consent form

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

This study is a part of the SBO SCAFFOLD project (www.sbo-scaffold.com) funded by Research Foundation Flanders (FWO). The funding organisation was not involved in any stage of this study.

Authors’ contributions

All authors (VA, SP, JE, ME, NM, BS) contributed to designing the study. VA collected the data, led the data analysis, and wrote this manuscript. All authors contributed to the critical revision of the paper. All authors have read and approved the manuscript.

Acknowledgements

The authors would like to acknowledge the contribution of Mrs. Oona Janssens, Mrs. Marieke Robbrecht, and Ms. Sofie Van Ostaeyen that assisted in the design of the study. We would also like to thank and acknowledge Prof. dr. Martin Valcke for facilitating this study through the SBO SCAFFOLD project(www.sbo-scaffold.com). Additionally, we would like to thank Mrs. An Stockmans and Mr. Guy Gielis from the Interuniversity Centre for GP Training for assisting with recruiting panellists. Lastly, we would like to thank our participants who accepted to participate in this study during the COVID-19 pandemic.

Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–45.
Caverzagie KJ, Nousiainen MT, Ferguson PC, et al. Overarching challenges to the implementation of competency-based medical education. Med Teach. 2017;39(6):588–93.
Ferguson PC, Caverzagie KJ, Nousiainen MT, et al. Changing the culture of medical training: An important step toward the implementation of competency-based medical education. Med Teach. 2017;39(6):599–602.
Snell L, Frank JR, Sherbino J. CanMEDS 2015 Physician Competency Framework. Royal College of Physicians & Surgeons of Canada; 2015.
Norman G. CanMEDS and other outcomes. Adv Health Sci Educ. 2011;16(5):547–51.
Pangaro L, ten Cate O. Frameworks for learner assessment in medicine: AMEE Guide No. 78. Med Teach. 2013;35(6):e1197-210.
Michels NR, Denekens J, Driessen EW, et al. A Delphi study to construct a CanMEDS competence based inventory applicable for workplace assessment. BMC Med Educ. 2012;12:86.
Nousiainen MT, Caverzagie KJ, Ferguson PC, et al. Implementing competency-based medical education: What changes in curricular structure and processes are needed? Med Teach. 2017;39(6):594–8.
Stutsky BJ, Singer M, Renaud R. Determining the weighting and relative importance of CanMEDS roles and competencies. BMC Res Notes. 2012;5:354.
van der Lee N, Fokkema JP, Westerman M, et al The CanMEDS framework: relevant but not quite the whole story. Med Teach. 2013;35(11):949 – 55,Anderson HL, Kurtz J, West DC. Implementation and Use of Workplace-Based Assessment in Clinical Learning Environments: A Scoping Review. Academic Medicine. 2021;96(11S).
Campbell C, Hendry P, Delva D, et al. Implementing Competency-Based Medical Education in Family Medicine: A Scoping Review on Residency Programs and Family Practices in Canada and the United States. 2022.
Cook DA, Brydges R, Ginsburg S, et al A contemporary approach to validity arguments: a practical guide to Kane's framework. Med Educ. 2015;49(6):560 – 75,Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Advances in Simulation. 2016;1(1):31.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.
Embo M, Helsloot K, Michels N, et al. A Delphi study to validate competency-based criteria to assess undergraduate midwifery students' competencies in the maternity ward. Midwifery. 2017;53:1–8.
Chia-Chien H, Brian AS. The Delphi Technique: Making Sense of Consensus. Practical Assess Res Evaluation. 2007;12(10):1–8.
Jünger S, Payne SA, Brine J, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706.
Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach. 2017;39(1):14–9.
de Loë RC, Melnychuk N, Murray D, et al Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities. Technological Forecasting and Social Change. 2016;104:78–88,Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401-9.
Foth T, Efstathiou N, Vanderspank-Wright B, et al. The use of Delphi and Nominal Group Technique in nursing education: a review. Int J Nurs Stud. 2016;60:112–20.
Patton MQ, Fund RECM. Qualitative Research & Evaluation Methods: SAGE Publications; 2002.
Skulmoski GJ, Hartman FT, Krahn J. The Delphi Method for Graduate Research. J Inform Technol Educ. 2007;6:1.
de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639–43.
Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37-.
Coussens A, Dh M, De Bruyne S, et al. Methodisch werken in de gezondheidszorg. Antwerpen – Apeldoorn: Garant; 2020.
Ozolins U, Hale S, Cheng X, et al. Translation and back-translation methodology in health research - a critique. Expert Rev Pharmacoecon Outcomes Res. 2020;20(1):69–77.
Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach. 2017;39(1):14–9.
Savin-Baden MMCH. Qualitative research: the essential guide to theory and practice. London; New York: Routledge; 2013.
Carter N, Bryant-Lukosius D, DiCenso A, et al. The use of triangulation in qualitative research. Oncol Nurs Forum. 2014;41(5):545–7.
Ringsted C, Hansen TL, Davis D, et al. Are some of the challenging aspects of the CanMEDS roles valid outside Canada? Med Educ. 2006;40(8):807–15.
Kalén S, Lachmann H, Varttinen M, et al. Medical students' experiences of their own professional development during three clinical terms: a prospective follow-up study. BMC Med Educ. 2017;17(1):47.
Tavakoly Sany SB, Behzhad F, Ferns G, et al. Communication skills training for physicians improves health literacy and medical outcomes among patients with hypertension: a randomized controlled trial. BMC Health Serv Res. 2020;20(1):60.
Organization WH. Towards people-centred health systems: An innovative approach for better health outcomes. In: Europe, ROf, editors. Denmark: World Health Organization; 2013.

Download PDF

Journal Publication

published 01 Apr, 2023

Read the published version in BMC Medical Education →

Reviewers agreed at journal
25 Aug, 2022
Reviewers invited by journal
13 Jun, 2022
Editor invited by journal
13 Jun, 2022
Submission checks completed at journal
12 Jun, 2022
Editor assigned by journal
16 May, 2022
First submitted to journal
16 May, 2022

You are reading this latest preprint version

Fitness-for-purpose of the CanMEDS competencies for workplace-based assessment in General Practitioner’s Training: a Delphi study

Status:

Journal Publication

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Background

Methods

Study design

Study setting

Selection of panel

Development and pilot of Delphi survey

Data collection and analysis

Role of research team to prevent bias

Results

Delphi round 1

Delphi round 2

Delphi round 3

Discussion

Conclusions

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors’ contributions

Acknowledgements

References

Supplementary Files

Status:

Journal Publication

Version 1