A National Virtual Research Training Workshop for 3800 Medical Students: A Mixed-Methods Evaluation of A Novel Research Education Program

Research training is necessary to ll the burgeoning global decline in physician-scientists, which is further pronounced in developing countries. This has generated much interest within institutions across the world to increase research exposure for students as part of medical school curricula. In order to produce a cadre of aspiring physician-scientists, the Student Research Forum (SRF) at Aga Khan University conducted a series of online research workshops leveraging innovative technological solutions during the COVID-19 pandemic. Our study aims to explore students’ perceptions of engaging undergraduate medical students in online research training workshops and to evaluate the effectiveness of such interventions in improving the knowledge and skillset of participating students.


Abstract Background
Research training is necessary to ll the burgeoning global decline in physician-scientists, which is further pronounced in developing countries. This has generated much interest within institutions across the world to increase research exposure for students as part of medical school curricula. In order to produce a cadre of aspiring physician-scientists, the Student Research Forum (SRF) at Aga Khan University conducted a series of online research workshops leveraging innovative technological solutions during the COVID-19 pandemic. Our study aims to explore students' perceptions of engaging undergraduate medical students in online research training workshops and to evaluate the effectiveness of such interventions in improving the knowledge and skillset of participating students.
Methods 'A Beginners Guide to Research' was a nation-wide virtual research workshop series conducted for medical students across Pakistan in June 2020. A mixed-methods study design was employed to evaluate course e cacy in terms of course content, duration and strategy. Dissemination of quantitative feedback forms alongside simultaneous conduction of qualitative interviews were used to receive feedback for the workshop and obtain the perspectives of medical students on the effectiveness of educational interventions in improving research-related knowledge.

Results
A total of 3,862 medical students enrolled in the national workshop series. The feedback form was completed by 2,754 participants, hailing from 113 universities across 37 cities in Pakistan. 2,311 (84.0%) participants stated that the course helped them improve their overall research knowledge and 2,333 (84.7%) felt con dent that the skills learnt would help them in future research projects. 40 participants from 25 institutions across Pakistan participated in the qualitative interviews. 24 (60.0%) participants had no prior knowledge of research, 23 (57.5%) found virtual screen-sharing to be the most useful learning strategy and 22 (55.0%) students expressed interest in initiating/expanding their institutional research platforms.

Conclusion
This nation-wide series presented a way in which alternative pedagogies can be used for the dissemination of research knowledge and skills to medical students with a strong inclination towards research. Efforts must be made by institutions to provide opportunities to medical students for engaging in meaningful scienti c research during the COVID-19 pandemic and beyond.
Background Page 4/20 The declining numbers of physician-scientists globally is a challenge that requires urgent attention. This de cit is coupled with an increasing lack of enthusiasm demonstrated by budding physicians to partake in meaningful scienti c research. The dearth of trained physician-scientists has necessitated the need to implement coordinated efforts to reverse this trend [1].
Various reasons have been cited by medical students to explain a lack of interest in research such as preference for direct patient care, absence of mentorship in research-related activities, decreased desire to prolong academic careers, accrual of student debt during medical school training, striking an appropriate work/life balance and lack of funding opportunities for research [2][3][4]. In response to this physicianscientist shortage, a multitude of solutions have been proposed by countries around the world to give medical students a holistic exposure to scienti c research and to develop the prerequisite skills needed for a successful career in academic research [1]. Some notable programs include development of nationwide research programs in Norway and introduction of online research courses by the United States [5,6]. Other strategies include research methodology workshops (Saudi Arabia), innovative research fairs (Germany) and medical student research groups (Egypt) to increase medical students' interest in research [7][8][9].
Formal research training is not being given the due importance as it still remains non-uniformly incorporated into medical school curricula. A vast majority of students felt that medical research was often viewed as an extra-curricular activity in medical school, with a huge focus directed on basic and clinical sciences subjects. 6 Recent initiatives in establishing better research opportunities for students within the medical curricula in developed countries have been a responding action to this demand [6,7].
In Pakistan, several studies have shown the undeniable research interest that prevails within the medical student community [10,11], however, there is a continuous lack of research initiatives within the country, highlighting a huge gap between the demand and production of trained physician-scientists. In response to this situation, the Student Research Forum at the Aga Khan University conducted the rst national research workshop to increase the knowledge and research skills of medical students across Pakistan. Due to limitations imposed by COVID-19, a novel virtual platform and latest technological solutions were used to deploy this nation-wide initiative. This use of technological modalities to ensure stability within medical education has opened a plethora of alternatives that can simultaneously be used to reach a larger audience of medical students and provide them with the research skillset required to become a competent academic clinician.

Methods
Overall Study Design: The Student Research Forum (SRF) at Aga Khan University in Karachi, Pakistan employed a quasiexperimental design to conduct a workshop series for medical students across the country. The course content for this series was developed a priori with expert input and in concordance with the participants' level of comprehension and research knowledge. A mixed-methods approach was used to gauge the impact of the workshop; with online feedback survey forms for quantitative and focused group discussions (FGDs) for in-depth qualitative evaluation. All participants included within the study provided informed consent by completing a digital consent form. All methods were carried out in accordance with relevant guidelines and regulations, ie, the SRQR guidelines for qualitative research.
Study Setting and Duration: The virtual research workshop series, titled as "A Beginner's Guide to Research", was advertised through various social media platforms, namely Facebook, Instagram and Twitter, enrolling medical students from across the country. The workshop included an orientation and four interactive live sessions which took place using an online platform on alternate days within the span of one week from June 22 nd , 2020 to June 27 th 2020, each with a duration of approximately one hour. Figure 1 shows a comprehensive outline of the entire workshop series content along with key elements of the study methodology.
All the workshops were conducted on Zoom application Version: 5.0.4 (25694.0524) and were broadcast as a live-streaming webinar through the Video IBM Channel to increase national accessibility of the sessions. At the end of each workshop, a Frequently Asked Questions (FAQ) document was circulated amongst the participants, including the key points pertaining to the speci c topic, for further clarity and review.
To assess the impact and gain participants' perception of the workshop series, a feedback form was disseminated to all participants and qualitative interviews were conducted at the summit of the series, which largely constituted a FGD format. Each FGD consisted of 7-9 students and was conducted virtually on Zoom Application Version: 5.0.4 (25694.0524).

Study participants, sampling and enrollment:
Inclusion criteria for the enrollment of participants within the series required them to be currently enrolled in medical school. Participants who originally registered for the workshop but did not follow through till the end were excluded from the overall analysis. The quantitative survey form was disseminated to all participants who enrolled in the series. In parallel, convenience sampling was used for recruiting participants for the FGD. Qualitative data analysis occurred in conjunction with the FGDs, allowing the research team to continue recruiting participants until no new themes emerged. Thematic saturation was achieved after 5 FGDs.
Digital informed consent was procured from all participants prior to the start of the workshop series. A De-Identi cation code (D-ID) was assigned to each participant to maintain con dentiality and ensure privacy of data.

Study instrument 1 -Feedback Survey:
The quantitative survey form solicited information on the objectives, content, delivery and duration of the course along with overall strengths and limitations of the series, using a standard 5-point Likert scale, asking respondents to indicate their level of agreement with prespeci ed sub-themes. Additionally, participants were asked to overall rate the workshop series using a rubric, identify its strongest feature and the areas requiring further improvement (See Additional File 1).

Study instrument 2 -Focused Group Discussions:
For the FGDs, a semi-structured qualitative interview guide (See Additional File 2) was developed through expert input and adaptation of previous instruments [12]. This guide was divided into three components and sub-components to establish standardization of the discussion. The developed questionnaire covered the following 3 major themes: 1. Participant Expectations: This theme allowed the participants to discuss their inspiration and motivation for registering in the workshop and whether their expectations were appropriately met.
2. Course Content: The satisfaction of participants with regards to the course content, duration and classroom format was discussed within this section, in addition to the strengths and limitations.
3. Impact Assessment: This theme allowed the participants to discuss any improvements that they felt regarding their research knowledge after the series and evaluated their con dence in independently conducting research in their respective institutions.
In each component, the leading question was left open-ended, followed by speci c close-ended questions. Interviewers were also trained a priori and a pilot run was done to ensure identi cation and recti cation of any obstacles. Each FGD was led by 3 members of the research team and a clinical research fellow, with 1 moderator and 2 other members for audio recording. Data collected through audio recordings was deidenti ed and transcribed verbatim prior to analysis. Data from the FGDs was analyzed using NVivo software (version 12.6.0.959), according to the Braun and Clarke six-step process for thematic analysis [13]. An iterative coding process was used, where 4 members of the research team individually identi ed themes, sub-themes and codes using the transcribed verbatim. A consolidated codebook of 51 unique codes was generated. 2 members independently used NVivo to pair each transcribed statement with its equivalent code. The results were compared and a consensus was reached for each code, producing the results of the qualitative discussion as frequencies.

Quantitative Survey Results
The feedback form was completed by 2,754 participants (response rate = 71.3%). The participants had a mean (SD) age of 21.2 ± 2.1, with 27.5% males and 71.9% females. Additional demographic data for the participants can be found in Table 1. helped them in improving their overall research knowledge and 2,333 (84.7%) reported feeling con dent that the skillset learnt through the series would help them in future research projects. The ndings also revealed a high interest in 2,498 (90.8%) participants for a future advanced research series.
Participants were also asked to nominate the area of the workshop that they felt was the strongest feature and the area that needed the most improvement. The two key areas that were identi ed by the highest number of participants as the main strength were course content (n=853, 31.0%) and delivery method and approach (n=693, 25.2%). Likewise, the two key areas that were identi ed to be the major limitation were course duration (n=636, 23.1%) and course video and sound quality (n=478, 17.4%).

Thematic analysis of Focused Group Discussions (FGDs):
A total of 40 participants took part in the FGDs, consisting of 13 (32.5%) males and 27 (67.5%) females, with a mean age of 20.9 ± 2.0 ( Table 1). Thematic analysis ( Table 2) exhibited the following noteworthy ndings, extrapolated from the opinions shared by the FGD participants (Table 3). Table 2 Thematic analysis of the ISGMs -themes, subthemes, codes and frequencies  register for the series and particularly learn basic research skills (n=36, 90.0%). For example, a secondyear medical student commented, "We didn't have any orientation related to the concept of research; neither in rst-year, nor in the second-year -so we wanted to learn about research." Similarly, another rstyear medical student was quoted as saying "…Wanted to start from scratch so really needed a basic course and a platform where people could tell me how research is done…" Other factors that inspired registrations included the provision of certi cates on the completion of the course (n=21, 52.5%), the possibility of developing an academically sound career (n=21, 52.5%) and the opportunity to learn from research experts a liated with a leading research institute in the country (n=13, 32.5%). When participants were asked about any logistical issues encountered during the workshop, most of the students (60.0%) did not face any challenges. However, 10 (25.0%) respondents experienced internet connectivity issues and 3 (7.5%) students reported interrupted livestreaming.

Theme 3: Workshop Feedback and Evaluation
27 (67.5%) participants stated that the major strength of the workshop was the ful llment of all stated objectives, while 16 (40.0%) students felt that the course content was appropriate for beginners. One second-year medical student appreciated the "synchronization of content with objectives," and another stated: "I am satis ed with the content and objectives and how they were covered. It was pretty good and comprehensive for beginners." Similar to the limitations highlighted in the feedback form, the duration of the sessions was considered a major shortcoming by 22 (55.5%) participants. Furthermore, a signi cant variability was seen in the comprehension of the course content by students, which led some to nd the course too basic (n=13, 32.5%), while others found it to be too complicated (n=9, 22.5%).
Theme 4: Impact Assessment 23 (57.5%) participants stated their con dence in leading a research project after the course, along with 22 (55.0%) students who showed interest in expanding a research platform within their institutions through dissemination of the learnt knowledge, as stated by a rst-year medical student, "…. I can teach and deliver the skills to students in my institution, at least the ones I learnt".
The possibility of an advanced workshop series was exciting for many participants, with 29 (72.5%) attendees showing an interest in attending a future workshop series.

Discussion
Effective research in medicine provides an impetus for novel and innovative developments that ultimately allow us to nd solutions to challenging and complex medical problems. This highlights the core principle behind conducting educational interventions to increase the knowledge and awareness of medical students regarding scienti c research. The COVID-19 pandemic has brought about a major paradigm shift in teaching all over the world, with increased adoption of online means of instruction using various softwares such as video conferencing tools and webinars. Comparison studies including systematic reviews have tried to evaluate the e cacy of online-teaching pedagogies with the traditional in-person teaching methods, concluding a similar impression of e cacy between the two [14,15].
An online format using Zoom was a strategy adopted due to social distancing restrictions posed by the COVID-19 pandemic and has proved to be an effective tool which allows researchers to qualitatively assess participant responses across the themes of discussion, increasing accessibility through the online nature of the medical students and young graduates in Karachi, Pakistan showed that majority of the participants (n=225, 59.0%) considered their research exposure inadequate, even though they were keenly interested in medical research [10]. Several factors have shown to contribute to a medical student's interest in research, namely a good role model coupled with a strong research knowledge [17], which were also evident inspirations for registrations in our study.
The popular opinion within participants of using live demonstrations through screen sharing was consistent with literature that shows how case studies and video demonstrations were the most effective instructional strategies in an online course [18]. In addition, ensuring the ful llment of all objectives and provision of supplementary reading material after virtual sessions allow increased information retention within the students. A systematic review has highlighted how web-based learning interventions can lead to ample gains in knowledge among medical students [19].
However, virtual teaching is still hindered by its limitations of decreased interaction between the speakers and participants, coupled with the issue of standardizing the course content for a diverse cohort. The latter has also been experienced within in-person sessions where the presence of diversity within the participants led to variable comprehension of the content being taught between them [20]. A study reported teaching-related factors as the main concern of students for video lectures which include lecture content, speaker, and delivery style rather than the technology employed [21]. This highlights the importance of paying attention to these logistical details for any teaching session, as was employed in our study to ensure quality transference of knowledge.
The e cacy of the series is showcased through the enthusiasm of 23 (57.5%) participants who stated their con dence in leading research projects and workshops in their institutions. Ultimately, this can prove to be fruitful in the efforts to increase the number of publications from institutions in Pakistan, which is currently falling behind the rest of the world by a substantial margin [22].
The interest shown by 29 (72.5%) participants towards an advanced research series is in accordance with the ndings of a study which concluded online learning to be more convenient for students, allowing them to balance their studies, family, and extra-curricular activities [23]. This ignited interest can play a pivotal role in propagating research knowledge and may marginally compensate for the lack of a robust research curriculum in Pakistani medical schools, leading to the increased numbers of Pakistani physician-scientists in the years to come.
To the best of our knowledge, this was the rst ever national research workshop conducted in Pakistan, using a virtual platform and made accessible to all medical students across the nation. The series ensured as much accessibility as possible to all medical students, regardless of their age, gender, geographical district, or nancial status. All participant registrations were taken into consideration to keep the cohort diverse and allow for wider dissemination of research knowledge. Ensuring a safe space within the FGDs allowed participants to share their feedback candidly, without any hesitation.
A major limitation highlighted during the series was the inadequate duration of the sessions, which affected the quality of content delivery to the participants. This also resulted in the inability to ask all the questions provided by the participants through the email during the Questions and Answer (Q&A) Section at the end. However, providing an FAQ document at the end of the sessions, aided participants to easily navigate through the workshop series.
In addition, connectivity issues and power outages often led the students to miss some sessions, but the FAQ documents helped in covering the gaps faced by them.

Conclusions
This rst-ever national virtual research workshop series for undergraduate medical students paves the way for future similar series to be conducted in a manner more accessible and acceptable to a diverse cohort of students interested in learning the basics of research. The initial overwhelming response, with continuously high attendance in all the workshop sessions indicates the presence of a high research interest within undergraduate students paired with a lack of research opportunities within their teaching institutions. Hence, there is a need to further spread the current research interest and passion using similar platforms, standardized for students all over the nation with the provision of knowledge and skills at the undergraduate level where it can be well-received and appreciated.