Context
Our curriculum is hybrid and problem-based, and it takes students over 5 years to complete.
The fourth year consists of eight clinical courses that range between 5 and 9 weeks in duration, including primary health care, ENT/OPHTA, OB/GYN, orthopaedics, dermatology, and psychiatry and anaesthesia courses.
The EBM module was first introduced as part of the primary health care course among the first cohort in 2015/16, and it has continued to be part of the 4th year curriculum since then. The primary health care course was the first clinical rotation in the fourth year. It introduced the students to not only EBM but also communication and consultation skills, which are important foundations for the medical profession. The EBM module was included in the fourth year, which is taken after the medical research/epidemiology course that is offered in the third year. The placement of the module was deliberate to ensure that students would have a foundation in study design, research methodology and basic biostatistics prior to taking the EBM module. The objectives of the module included the following: to develop relevant knowledge and skills in framing questions in the PICO format, conducting database searches and critically appraising findings and be able to demonstrate these steps of EBM. That is, by the end of the module, the students were expected to be able formulate a PICO question based on a clinical scenario, search online for the relevant studies, critically appraise the findings for their validity and appropriateness and discuss their applicability. These skills were assessed with the use of a graded rubric for both written (report) and verbal (presentation) assignments. The module was taught by faculty members who specialized in family medicine (2), community medicine (1) and public health. (1) All faculty were trained in EBM teaching either in the Centre for Evidence-Based Medicine at the University of Oxford or at McMaster University and had been trainers at the National & Gulf Centre for Evidence-Based Health Practice in Riyadh, Saudi Arabia.
The content of the module is shown in Table 1. The module included the following elements. In week 1, 50- to 100-minute lectures were delivered on one to three mornings throughout the week. Additionally, 50–100-minute interactive sessions were used for online database searching. During week 2, 75–150-minute small-group workshops were conducted on critical appraisal skills using therapy and diagnosis papers that were chosen by the trainers. In weeks 3 and 4, students were assigned to small groups supervised by faculty to work on their own assignments. In week 5, each student’s skills in presenting an EBM topic were assessed; each student submitted a detailed report that was standardized and formatted to cover the EBM steps (Ask, Acquire, Appraise, Apply and Assess). Students were asked to submit the full-text article used in the assignment as well as the critical appraisal sheet.
Table 1. Content of the EBM module
Topic
|
Learning Objectives
|
Teaching Method/Duration
|
Timing
|
EBM I: Introduction
|
- Define EBM
- List the EBM steps
- Formulate answerable clinical questions (PICO)
|
Lecture/100 min
|
Once/Week 1
|
EBM II: Literature search
|
- Translate PICO questions into a search strategy
- Demonstrate the EBM resources search
|
Interactive tutorial/50–100 min
|
Once/Week 1
|
Critical Appraisal Skills I
|
- Understand the concepts of critical appraisal (diagnosis and therapy)
|
Lecture/50 min
|
3 times/Week 1
|
Critical Appraisal Skills II
|
Lecture/50 min
|
Critical Appraisal Skills III
|
Lecture/50 min
|
EBM Workshop I
|
- Critically appraise an article on therapy (RCT–SR&MA) using a McMaster worksheet
|
Hands-on workshop/150 min
|
Once/Week 2
|
EBM Workshop II
|
- Critically appraise an article on diagnosis using a McMaster worksheet
|
Hands-on workshop/75 min
|
Once/Week 2
|
EBM Assignment
|
- Demonstrate EBM steps for clinical questions selected by the student under faculty supervision
|
EBM Presentation and report
|
SDL*/Week 3 - 4
|
*SDL: self-directed learning
In this study, we did not test the validity, inter-rater agreement, internal consistency, acceptability and feasibility due to the small sample size. However, we collected the relevant data for consideration for future publication about adapting authentic assignment as a method of EBM skills assessment. Moreover, since we started teaching the EBM module, it has evolved with time and has undergone modifications, whether in teaching strategy or assessment method. Recently, we added two small group workshops wherein students choose their own clinical question until they retrieve articles they have chosen for their assignment and appraise them; then, the students are distributed into pairs to enhance peer learning. Currently, with the Covid-19 pandemic, we have been conducting a virtual module that is different and interesting.
Study design
No control group was included in this before-and-after study, as the participants were the first cohort who received the educational intervention. This was our first experience teaching EBM in our curriculum. Moreover, the EBM module has evolved over time, making it difficult to compare between cohorts.
Sample size
All fourth-year medical students were included (n=52).
Sampling technique
Not applicable
Data collection methods, instruments used, measurements
A before-and-after study design was used. A 5-week short EBM module including lectures, workshops, and online search sessions was conducted with fifty-two fourth-year medical students during their primary healthcare course at the College of Medicine, PNU, from the first of September until mid-December 2016. We used the Fresno test to assess the effectiveness of a hands-on workshop on EBM knowledge and skills gain. Overall, students’ performance was assessed by EBM assignment at the end of the module. The pre-test was administered in the second week of the module after ending the theoretical part and immediately before the critical appraisal workshops. This timing was chosen because EBM is a relatively new subject for students, and on the other hand, we would like to assess whether workshops will improve their skills gain or not. The post-test was administered during week 3 after the teaching of the module, including the workshops, was completed. Each student had a code number that replaced her name on the pre- and post-test papers to ensure that the students’ identities were protected. Moreover, to avoid inter-rater bias, an experienced faculty member in EBM was responsible for grading both the pre- and post-test papers. However, none of the scores (pre/post) were counted as part of the students’ assessment or final grade. In weeks 3 to 5, students were asked to work on their EBM assignment under faculty supervision. Therefore, each student was expected to come up with a clinical question/scenario from either their observation or interest and then work on EBM steps as they have been taught during the workshops. They may also visit or contact their supervisors if they face difficulties on this task. By the end of the course, i.e., week 5, all students should submit their work in report format and present their topic in front of the group. One faculty member was responsible for marking the EBM reports using a standardized format. Two other faculty members were invited to assess students’ EBM presentation skills using a standardized rubric. Their performance in the assignment accounted for 10% of their primary health care course grade.
Statistical analysis
Statistical analysis was performed using SPSS statistical software (version 20, SPSS Inc., Chicago, US). The effectiveness of the EBM workshop was assessed based on the differences in the total and subtotal pre- and post-training Fresno test scores, which were the primary outcome. Parametric tests were used to test the study hypotheses, such as the mean, standard deviation, and confidence interval (CI %). Comparison of the total Fresno pre- and post-test scores was conducted using Student’s paired t-test to determine whether there was a statistically significant difference in the students’ performance before and after the intervention. A P value .05 was considered significant.
Reliability of the Fresno test
The Fresno test is an objective, comprehensive tool that consists of 12 items that cover basic knowledge and skills in EBM [18]. The test includes two clinical scenarios with open-ended questions. Participants are required to complete the four key steps of the EBP process to adequately answer the open-ended questions related to the clinical scenarios. In addition, two questions are related to statistics for observational and experimental studies. We omitted some items in the calculation section because 98% of students left them blank on the pre-test. Therefore, the total Fresno score in this study ranged from 0–204.
Cronbach’s alpha and the item-total correlation were used to determine internal reliability. Cronbach’s alpha was used as the index of internal consistency of the test, with an acceptable range from .7 to .95. The item-total correlation was used to determine the reliability of each scale, with an acceptable value of 0.2 or higher. The lowest item-total correlation was .263, which is above the acceptable value (item 9). As shown in Table (2), Cronbach’s alpha was .775, which is statistically acceptable.
Table 2. Reliability results and item-total statistics
|
Corrected Item-Total Correlation
|
Cronbach's Alpha If Item Deleted
|
Item 1A
|
.415
|
.763
|
Item 1B
|
.401
|
.763
|
Item 2
|
.455
|
.760
|
Item 3
|
.443
|
.757
|
Item 4
|
.523
|
.747
|
Item 5
|
.483
|
.754
|
Item 6
|
.497
|
.755
|
Item 7
|
.556
|
.745
|
Item 8
|
.500
|
.752
|
Item 9
|
.263
|
.775
|
Item 10
|
.268
|
.774
|
Item 11
|
.369
|
.768
|
Item 12
|
.351
|
.768
|
Total items
|
|
.775
|