Combatting with Challenges in Medical Education During COVID-19 Pandemic: A Mixed-Methods Study Identifying the Effectiveness of Using E-Learning in Pediatric Clinical Clerkship of A University Hospital, Pakistan

Background The emergence of COVID-19 raises the opportunity to reimagine medical education. One way of attempting this is online classes, also known as e-learning, through recordings and or live streaming. The purpose of this research is to ascertain the effectiveness of using the e-learning instructional methodology for a Pediatric module with the fourth year MBBS students at the Aga Khan University Hospital, Karachi. Methods It was a sequential (Quantitative-Qualitative) mixed-method study. The quantitative component of the study consisted of pre and post-tests, as well as feedback on each session. The qualitative component was composed of focused-group discussions to explore students' experiences. Statistical analysis was performed using SPSS 20.0. Mean ±SD was reported for quantitative variables, and frequency and percentages were calculated for nominal variables. The pre and post-test scores were compared using a paired t-test. Pre and post mean test scores were analyzed in comparison to the level of student groups (Experts, Semi-experts, and Novice) by one-way ANOVA. For qualitative content analysis, categories were clumped together to yield subthemes that were further merged into themes.


Background
The novel Coronavirus (COVID- 19) was declared a pandemic, and it threw the world into disarray. 1 Every sector began grappling with this global phenomenon to the best of their abilities. 2 Beyond the struggle of health-care facilitation, like many other aspects of everyday life, the pandemic also impacted education and in-person learning. 3 To contain the transmission of COVID-19, social distancing measures called for educational institutions, including medical colleges, to be closed temporarily. Any gathering is considered an invitation for the spread of this virus, but medical students are at a particularly high risk of exposure. Not only would they be more vulnerable, but they also pose a threat to be potential vectors of the virus due to their frequent rotations between departments and hospitals and their interactions with patients and other health-care workers. 4 Hence the closure of medical schools deemed necessary and caused an unprecedented halt in the medical education system.
A major component of medical education is based on training students through direct observation and inperson sessions and interactions with residents and faculty. The sudden suspension of these proceedings has been a genuine concern for medical students and teaching hospitals. 5,6 The emergence of COVID-19 raises the opportunity to question our learning priorities and reimagine medical education for future physicians and surgeons. 7 Medical institutions are employing alternative strategies to consolidate the educational processes without hampering the continuum of assessments, rotations, and electives. One way of attempting this is online classes, also known as e-learning, through recordings or live streaming. 8 The use of e-learning in health sciences education (including medical education) has been increasing over the years, but the role of e-learning and its effect on the student's performance has not been looked into thoroughly. 9,10 To facilitate interactions in these classes, it is imperative to record and study the impact of this innovation, as it may transform the way medical education has been delivered traditionally.
The Aga Khan University Hospital (AKUH) in Karachi is one of the leading medical institutions in Pakistan. Soon after the rst COVID-19 positive patient was detected in Pakistan, the provincial government announced lockdown and closure of all educational institutions by 13th March 2020. 33 The medical students of AKUH were sent home and all academic connections were altered. In response to this new change, while the university leadership was working on a contingency plan, the Hospital Pediatrics clerkship stakeholders were also working on re-building the connection with the medical students.
The pediatrics clerkship is a two-month rotation in the fourth academic year of the MBBS program at AKUH. Considering it is a clinical rotation, the students rotate in ve batches (around 20 students per batch) throughout the academic year. The basic instructional methods being used in this rotation include bedside teaching, simulation-based teaching, tutorials, case-based learning, and faculty sessions. In the given circumstances, the clerkship was revised and re-directed towards e-learning as the main form of teaching. The key purpose of this research is to ascertain the effectiveness of using the e-learning instructional methodology in delivering the key learning objectives to the fourth-year MBBS students at the Aga Khan University Hospital.

Study questions
1-What is the effect of using e-learning methodology on the knowledge of the fourth-year MBBS students who have been through the Pediatric eLearning module? 2-What is the perceived effectiveness of e-learning is as experienced by the fourth-year MBBS students who have been through the Pediatric eLearning module? Objectives 1-To determine the effect of using e-learning on the knowledge of fourth-year MBBS students who have been through the Pediatric e-learning module 2-To explore the experiences of fourth-year MBBS students who have been through the Pediatric e-learning module.

Study design
It was a sequential (Quantitative-Qualitative) mixed method study. The quantitative component of the study was of a quasi-experimental pre-post design. The qualitative component was used to gather the experiences in detail to come to accurate conclusions. (Figure 1) Participants All medical students in the fourth year at the Aga Khan University, attending the online module of Pediatrics, were invited to participate in the study (n = 102).

Study Setting
The study took place in the Department of Pediatrics & Child Health of the Aga Khan University, a teaching hospital in Karachi, Pakistan. The department is responsible for organizing the Pediatrics clerkship for fourthyear medical students. The clerkship usually consists of an 8-week rotation of in-hospital Pediatrics, but this year it was shifted to a 5-week online module. Ethical approval was acquired through AKUH's Ethical Review Committee (AKU-ERC). The participants were invited for voluntary participation, and informed consent was obtained. Data was collected using anonymous tools. The participants were given a unique identity to ensure the anonymity of the data.

Pediatric e-learning module (Hospital Pediatrics Online Module-HoPOM)
The total duration of the module was ve weeks. There was one e-learning session held each week. These sessions were developed by keeping in mind the three tiers of learners: 1. Expert (students who had already completed the rotation in the pediatrics clerkship) 2. Semi-expert (students who were in the middle of the rotation at the time of lockdown) 3. Novice (students who will be having their rotation subsequently) These sessions were each of total one-hour duration and based on the themes which are the crux of pediatrics.
A session plan was devised to not only provide students with maximum learning but also keep them engaged (Annex 1). Pre and post-tests were developed for each of the topics discussed in the sessions. The tests were offered via "Virtual Learning Environment" (an online software available to students and faculty for the provision of all study material and tests) and comprised of 15-20 questions.

Quantitative data
Pre and post-tests: The students' knowledge regarding the planned session was tested every week using 15-20 questions. The pretest was administered three days before the session, and the post-test was two days after each session. The tests consisted of True or False and Best Choice Questions (BCQs). The tests were available on the Virtual Learning Environment (VLE) platform. Both the test scores were shared with the students in the following week for them to assess their performances.
Session feedback: Each session was evaluated using an anonymous online tool: Menti-meter. The online tool was displayed in the last 2 minutes of each session and responses were obtained from the students. The online tool contained vequestion items asked to give feedback regarding the gain in new knowledge, amount of opportunity to interact with faculty, presence of engagement during the session, how well-prepared the faculty was, and if they would want more sessions in the same format. Each of these questions was rated on a Likert scale of 1 to 5 [1= strongly disagree to 5=strongly agree]. It also included one open-ended question for suggesting areas needing improvement.

Qualitative Data
Focused group discussions (FGDs) The focused group method was utilized to explore the experiences of our medical students in-depth. It was a form of group interview that emphasized communication between students and researchers to generate holistic data. These sessions were held at the end of the entire online module. Three FGDs were organized according to the group levels, Novice, Semi-expert, and Expert, and composed of 5-7 medical students each. A set of guiding questions were used to conduct the discussion in a streamlined manner.

Data Analysis
Data underwent descriptive statistical analysis using SPSS 20.0. For descriptive statistics, mean ±SD or Median ±IQR was reported for quantitative variables like test scores, the number of the feedback given, e.t.c., depending upon the data distribution. For nominal variables like gender and session evaluations, frequency and percentages were calculated. The pre and post-test scores were compared using a paired t-test. P-value of <0.05 was considered signi cant. Pre and post mean test scores were analyzed in comparison to the level of student groups (Novice, Semi-experts, and Expert) by one-way ANOVA.
The generic guide proposed by Creswell (1998) was utilized for the purpose of content analysis. 11 Qualitative data analysis is a method of analysing and interpreting to provide greater meaning to the data. 12 Data that is in the form of text is systematically coded into categories and themes that can be identi ed easily. The deeper meaning of these themes is then interpreted. 13 The transcriptions were generated from the audio recordings within 24-48 hours of data collection. We used whole sentences as a unit of analysis and subjected them to condensation and abstraction to develop categories. Categories were clumped to yield subthemes, and similar subthemes were merged and emerged as themes. "I just wanted to say that there was not much of a practical approach because we were obviously not exposed to patients. So just a suggestion, maybe we can have a simulated patient like a patient and a mother or something and maybe that can help us-like say take History from them." With this online module of Pediatrics as pedagogical support, students were able to continue their medical education while experiencing an increased engagement in their learning due to the appropriate structure and content of the online sessions.

Results
The students could access the online class through their laptops and mobile devices in their homes, interact with faculty for an ample amount of time, take tests on the Virtual Learning Environment (VLE) portal, as well as view the old sessions again for any clari cation. This exibility gave them more control over their learning, to test themselves, re ect, ask questions, take notes, and not lose focus. They were satis ed with the choice of topics and with the efforts of the rotation coordinator. The students agreed that this way of continuing their medical education was appropriate, as noted by one student: "I really like that we started off right from the beginning, History taking and Physical Exam and everything..." The majority of students shared the sentiment that the online module was interactive (Table 1). Students were called out by name to answer questions, take part in quizzes during the sessions, and prepare for the pre-test and the post-test. Altogether this orchestrated a heavily engaging module for the students. It was an attempt to bridge the gap between faculty and student interaction, and it turned out to be quite popular among the students. Some did nd it bothersome to be called out in the beginning, but it triggered their interest and created healthy competition amongst their peers.

Theme 2: Barriers and challenges of online learning
There are always challenges faced when preparing any lesson plan, and this curriculum made for the rst time for an online module was no exception.
Apart from the expected issues of internet connectivity and occasional connection problems, the students believed the di culty level of the tests was not standardized (Table 1). Some students went on to explain that there were some issues faced in reviewing their tests and that not having the answer key was a problem. They also asked for more exible timing as to when they could take the test because of time lost when trying to log in or connect to the internet. Another issue raised was the gap between the session and the test, causing decay in knowledge, and so they had to re-visit the lecture. Some students felt that the tests and the sessions did not complement one another, so they had a hard time guring out the correct answers.
As the students were divided into three groups, trying to address all groups in one session was bound to lead to some critic. In the view of the Expert and Semi-expert students, they found it "redundant", and it was only when they were separated from the others that they felt more comfortable. Whereas Novice students found that some concepts were harder to grasp and assumed it would only possible after being through the practical pediatric rotation.
Theme 3: Future goals to enhance online learning All students were requested to share their thoughts on how improvements could be made to this online module, speci cally if we have to prepare it again in the future.
Some suggested that, because the clinical application of knowledge was missing, virtual patients could help x that problem, while others focused on bringing changes to the tests ( Table 1). The Novice and Semi-expert students were concerned that they were missing out on the learning opportunities that their Expert peers had, and simulated patients could be the solution to cater to that pertinent issue. Other suggestions also included letting students rotate with faculty doing "tele-clinics" so that they could observe or practice their history taking skills at the least, and then discuss diagnoses and management plans.
Because of the issues raised with the test, the students made some suggestions that could help tweak them and enhance their power as a learning and engagement tool. Many students recommended for problem-based learning to be incorporated into the online sessions or tests. Others suggested for the facilitator of the session to speci cally make the questions themself before the presentation and then have the presentation addressing those questions so that there is some connection established between the presentation and the questions in the pre-and post-tests. Finally, the majority of students asked for the release of answers to their tests to learn from their mistakes. One student pointed out, "I feel that there is no point in learning when you don't know which questions you did wrong in the test and which you did right."

Quantitative
Out of 102 eligible fourth-year medical students, all participated and enrolled in this study. Fifty-nine participants (68.8%) were female. All the students belonged to Novice 41(40.2%), Semi-expert 21 (19.6%), and Expert 40 (39.2%) groups (Table 2). We received a total of 188 responses of feedback (Table 3) and suggestions (Fig. 2) from the students for the online teaching method. From the responses collected, it can be deduced that the students showed maximum agreement for gain in knowledge during the Common Newborn Illnesses session (n = 12/13, 92.3%), but it was also the least engaging compared to others (n = 2/13, 15.4%).
For the Developmental Delay session, 60.0% (n = 12/20) strongly agreed with an increased opportunity to interact with faculty, and 100% ( n = 20/20) believed the faculty to be well prepared. Regarding future improvements, the majority of students (n = 64, 47.0%) responded that no changes were necessary.   Using paired t-test analysis between pre and post-test scores of each session, it was discerned that there was indeed a signi cant effect (p < 0.005) on knowledge enhancement during each session, depicted by the improvement in test scores (Table 4). It was also noted that the correlation of each session (r = 0.242 to 0.595) displayed a positive association between the gain in knowledge and the session held, supporting the paired ttest results. To test the hypothesis that the level of student (Novice, Semi-Expert, or Expert) affected the experience in this online module, a between-groups ANOVA was carried out for each session. Table 4 displays that the ANOVA yielded no statistical signi cance between the knowledge gained among the three group levels, so the null hypothesis that there is no difference between the groups was accepted, denoting that our online module had been proven successful in achieving the same learning goals as an in-person rotation.

Discussion
Extensive literature exists on the advantages of the closure of educational institutions to decrease transmission of highly infectious diseases, so there was no debate necessary when opting to shut down universities. 14,15,16 This ultimately shifted all institutions onto the platform of online education or e-learning. Our study aimed to review the effectiveness of using e-learning to incorporate knowledge in medical students of the Aga Khan University and to explore the experience of these students as they venture on this new journey.
The department of Pediatrics at Aga Khan University, together with the department of Educational Development, devised a plan to proceed with the newly faced challenge of an online module. Online learning is de ned as learning experiences in a simulated environment using different devices with internet access and facilitate interactions with instructors and other students. 17,18 We set up Zoom meetings of Pediatric faculty and students, included a comprehensive lecture, short "Kahoot" quizzes, pre and post-tests, and compared the knowledge of those who had gone through the rotation already with those only going through it online. It gave us a complete picture of how successful the online module had been in providing the appropriate knowledge and in keeping the students engaged. Though the online system of education is assumed to be a relatively new concept, if it is delivered in the right ways, it will be just as effective as in-person learning. 19,20 Our study showed that the medical students were quite relieved that they would be able to continue their learning, and those that had already gone through the rotation agreed that topics were chosen following the normal clerkship. Our quantitative analysis showed that the majority of students were satis ed with the structure of the sessions as it was. Similarly, our qualitative focused group discussions with individuals from each group level of Novice, Semi-expert, and Expert, also approved of the format of the lectures as effective in solidifying concepts. A recent study was done in China that also showed that students perceived online-learning, especially during the pandemic, as useful and highly effective in acquiring relevant knowledge. 21 Our results further show that students discovered the greatest advantage of this online module was the engagement and encouraged participation. This level of engagement is not usually present in usual in-person classes or rotations because here, students were called out by name in video-call sessions to partake in discussions and then followed with surprise quizzes. When discussing this in the focused group, students stated that initially, they did not like the level of interaction, as it was out of the norm and made some of them anxious. But as the lesson plan proceeded, a majority agreed that it was the best way to stay attentive during the sessions. Our quantitative results correspond with this same aspect, as a majority of students chose 'agreed' for being engaged during the sessions. Results from a similar study emphasized that for e-learning to have an educational impact, importance has to be given to student engagement and interaction. 22 Other studies also stated, to be successful, online programs should be designed in a way that they are studentcentered and group-based. 23,24 We also implemented this as we divided the students into groups according to their level of expertise. Furthermore, the students also exhibited annoyance at being in the same sessions as their experienced or inexperienced counterparts. Group-based learning proved to be more popular.
A positive impact of our online module was the dedicated and ample time received from faculty. Our quantitative feedback shows that a greater number of students agreed that ample time of interaction was given, and during our qualitative group discussion, students saw it as an advantage to be given such unique exposure to faculty. Under normal circumstances, the students would have exposure to different faculty at different times, and there would be a bias created, as every faculty would have a different way of facilitating or teaching. But through this online format that bias was removed, and all students were exposed to the same teacher. Furthermore, these lectures were recorded, and students were able to go back to see them any number of times and at any given time, but routinely one cannot always bother an attending to review a lesson. A study from India agreed that the "Anywhere-Anytime feature of e-learning" is bene cial for students in all elds. 25 Besides the fact that the students had appropriate resources and timing to revise, the bene t was perhaps twofold, as the facilitators also were better prepared. An overwhelming majority of our students agreed that the facilitators were well-prepared and had a substantial amount of knowledge of the topic they taught.
Regarding our data collected on the test scores, a trend of consistent improvement could be seen among the students in their post-test results compared to the pre-test results. Many students agreed that the tests were the main reason the sessions were successful. It gave them a chance to view their gap in knowledge and ll it with their online class and then re-evaluate themself afterward. This makes the students proactive in reading up before lectures, being diligent during the class, making notes, and then revising for the post-test. Other studies have also shown that testing students' knowledge before and after online sessions is a very effective way of learning, and most students will always show some improvement in the post-tests. 26,27 However, a major error in the tests was the unavailability of answers to the quizzes. Providing students with their answers could help them learn from their mistakes and ultimately enhance the power of the tests as a tool of education. Beyond just for learning, the pre and post-tests results can also be used to assess the competency of the lectures. 3 From our results we can judge whether the lecture content was not known to the students before, based on their pre-test scores, and then if it got known or not after the lecture, based on their post-test scores. Since our mean data showed an improvement in all the scores, we can conclude the lectures were useful in providing them the required knowledge.
Additionally, a major concern was if the Expert students have an unfair advantage in this online module, over the other groups, since they had already rotated through the Pediatric clerkship before the pandemic began. But through our ANOVA analysis, this concern was addressed, and it was surprising to note that there was no signi cant difference between Expert student learning and the learning of the Novice and Semi-expert groups.
Hence, we can conclude that all the students got an equal opportunity to learn and that the online module was successful in providing the theoretical knowledge normally received during the Pediatric clerkship. Similar studies in the past have also shown that medical students have accepted online training as an equal or better learning experience than the classroom sessions. 28 In the future we can aim to further improve on such online content and be somewhat con dent that students will not miss out on any essential lecture teachings they receive in-person.
However, lectures and discussions compose only a minor component of the existing curriculum in medical schools, so just converting it to an online platform is not enough to satisfy a full clerkship. Online content is all theoretical and does not let students apply or practice their learning in real-life situations. When asking our students for suggestions on how to make this online-module more effective, a few suggestions of virtual patients and "tele-clinics" were put forward. During their normal rotation, students would have clinical examination practice sessions and Problem-based-learning group discussions, that were now missing. Finding ways of re-incorporating these into the online module could have proved to be more successful and relevant.
Technology has become very advanced and virtual patients, simulations, video triggers, and voice-covers can be easily used to implement real-world scenarios. 29 In a meta-analysis done on articles from 2000-2017, to differentiate between online and o ine learning via pre and post-tests, 7 articles reported no signi cant difference between the two teaching methods, and 9 articles reported signi cant improvement in the online learning groups. 30 It might be worthwhile to combine the advantages of online and o ine teaching methods and adopt a blended learning technique for future modules. 31,32 After all, these students are being trained to become the healthcare workforce of tomorrow and need to be able to deal with real patients, as well as navigate through the latest technology, so that we can better prepare them to face the post-pandemic world.

Limitations
This study only applies to one medical student rotation and equal number of student participation was not present in each session. We had a varied number of student feedback in each session and a low response for suggestions of improvement.
Conclusion E-learning is an effective way of continuing the process of delivering medical education, especially in unprecedented times, such as the COVID-19 pandemic. Technological enhancements will help carry the impact forward as a blended-learning pedagogical approach in undergraduate medical education. Further studies on incorporation of e-learning and improvements of medical education need to be pursued, so that medical institutions are better prepared. The participants were invited for voluntary participation and informed consent was obtained.

Consent for publication
Not applicable Availability of data and materials The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. All methods were carried out in accordance with relevant guidelines and regulations.

Competing interests
The authors declare that they have no competing interests.

Funding
Not applicable Authors' contributions MR was the major contributor in writing the manuscript and did the relevant literature searches. SK collected and organized the Qualitative data. SQ analyzed and interpreted the Quantitative data. NA conceptualized and over-looked the entire study. AM collected data from all the students. SS conceptualized, edited, and reviewed the nal manuscript and over-looked the entire study. All authors read and approved the nal manuscript.