A Guide to Best Practice in Online Learning in Medical Education: A Middle East Qualitative Reflective Analysis


 Background: With the strike of Covid-19, an unprecedented rapid shift to remote learning happened worldwide with a paradigm shift to online learning from an institutional adjuvant luxury package and learner choice into a forced solo choice. This sheds the light onto quality assurance. While some groups have already established standards for online courses, teaching and programs yet very little information is included on methodology of their development and very little emphasis is placed on the online learning experience. Nevertheless, no work has been done specifically for medical education institutions.To develop a set of descriptors for best practice in online learning in medical education utilizing existing expertise and needs.Methods: This work utilizes a qualitative multistage approach to identify the descriptors of best practice in online learning starting with a question guided focus group, thematic analysis, Delphi technique and an expert consensus session done simultaneously for triangulationResults: This materialized into the development of a set of standards, indicators, and development of a checklist for each standard area. The standard areas identified were organizational capacity, educational effectiveness, and human resources each of which listed a number of standards. Expert consensus sessions identified the need for qualification of data and thus the development of indicators for best practice.Conclusion: Standards are needed for online learning experience and their development and redesign is situational and needs to be enhanced methodologically in axes that are pertaining to the needs of the education community. Taking such axes into consideration by educators and institutions will lead to planning and implementing successful online learning activities, while taking them into consideration by the evaluators will help them conduct comprehensive audits and provide stakeholders with highly informative evaluation reports.


Background
There is an increasing interest in the use of online learning in medical education from early undergraduate years through residency and fellowship training, and in continuing medical education (CME) (1).
This "disruptive innovation" of educational format is constantly evolving with different attempts and experiences to design, implement, assess, monitor and evaluate (1). With the strike of Covid-19, an unprecedented rapid change to remote learning happened worldwide with a paradigm shift to online learning from an educational option to the only existing solution (2,3).
This raises the issue of quality assurance since the conversation is turning from remote instruction as an emergency x to a new ongoing reality in education.
Recently, there have been efforts to standardize the necessary criteria for developing new content, adopting e cient teaching methods for online learning and establishing resources, yet no speci c attempts have been done within the context of medical education (4). One of the professional groups that have been working to reach this goal is the International Organization for Standardization (ISO) (5). Bari and Djouab (6) found that the existing standards or frameworks need to be modi ed to t with the context of speci c institutions.
There remains a need to generate operational guidelines to guide execution of online education focusing on building and evaluating the experience in medical education settings.
Thus, the purpose of this work is to address the gap in the online learning quality to shift from remote lecturing modes to high-quality online courses/programs. These standards for online learning experiences, as a comprehensive set of criteria are important to establish con dence in online learning among stakeholders and to facilitate structured and objective comparisons between various offered choices (7). Such standards and their indicators can also support the design of a rubric for peerreviewing, accreditation, and self-assessment for further improvement of online education in general.

Methods
Under the interpretivist paradigm, we used a deductive qualitative grounded theory approach aiming at creating a deeper understanding of the perceptions of medical educators and to explore the essence of their online experiences. This work applies the ground theory in a longitudinal approach through four phases.
Phase one (virtual focus groups) Two virtual focus group discussions were conducted. A convenient non-probability sample of faculty members in the regional medical schools were o cially invited to participate. They varied in gender, specialty, academic rank, and a liation. Precautions were taken to guarantee both the anonymity of the participants and the con dentiality of their contributions to the discussions.
Thirty faculty members attended each focus group discussion. This discussion was split into 5 groups and lasted for 90 minutes. Each group consisted of 6 faculty members and was moderated by one of the authors. The focus group discussions followed a deductive approach to test the hypotheses and themes had been derived from the data and saturation was reached after the 2 nd 5 focused discussions. In the focus groups, the moderators used a developed focus group guide that included questions aiming at exploring participants' views on what makes a good online learning experience re ecting on the transition phase they already went through post COVID 19.
Questions in the focus group guide covered three major themes: organizational capacity, effective learning and assessment, and educational effectiveness in online learning.

Page 4/28
The kickoff of the focus group was in the form of leading sentences and questions: From your previous experience, what is the role of institutional leadership in the success of online learning?
What kind of resources did you feel were necessary to conduct a successful online learning experience in your school?
What elements in the organizational bylaws need to be added or adapted to cope with the shift to online learning?
What are the essentials for shifting face to face programs into online format?
What are the elements required to create a motivating environment in online learning that ensures students engagement?
In your opinion, what are the criteria of choosing effective online assessment?
What are the most important criteria that are speci c to online learning evaluation rather than face-toface learning?
What are the needed attributes/capabilities needed in online learning required for faculty, students, and administration?
Phase two (Generation of standards and indicators) The focus group data was analyzed thematically and interpreted to identify descriptors of best practices in online learning. This was done by the authors. The focus group data was analyzed thematically, and saturation was con rmed by the authors. The themes were interpreted to identify descriptors of best practices in online learning.

Phase three (Delphi Technique)
To get the consensus of the experts on developed indicators and descriptors of best practice, a survey was developed based on the focus group discussion ndings and was pilot tested on a group of 5 respondents.
The survey was distributed using the Delphi approach. A two round online Delphi survey was conducted from October 2020 to January 2020.The Delphi panel consisted of 15 regional medical educators purposely selected based on their experience in online teaching and in managing quality standards. Over subsequent rounds of the Delphi, participants were invited to approve the quality area with its suitable indicators. This technique was repeated until 92% of the standards were approved by 100% of the panel.
Phase four (Expert opinion consensus session) A purposive sample of national and international experts in online learning were invited to participate in an expert opinion consensus session. An 'expert' was de ned as a person having different experiences and expertise in online learning. Experts were short-listed by members of the research team. The experts' opinions regarding the descriptors were aggregated and summarized. Forty-one experts were formally invited to attend a 1 day expert virtual panel meeting. The goal was to reduce the range of responses and arrive at something closer to expert consensus. Before their attendance, experts were given clear, written guidance on the objectives of the meeting and the required output of the expert panel -to review the standards of quality in online learning and its related indicators.
The credibility of this study was established through "analyst triangulation" to con rm and verify the conclusions drawn from the analysis which requires an external reviewer (Experienced medical educators with varied backgrounds) who worked together to analyze the transcripts until consensus was reached, as this analysis process helps to facilitate discussion and clarify possible blind spots. So phase 3 and phase 4 were conducted in parallel to achieve that triangulation Data collection and analysis All focus groups were audio recorded and transcribed by the authors. The transcripts were checked for accuracy and unclear data identi ed was excluded. The moderator eld notes and re ections on the transcribed data were attached to provide context. Two of the authors; NW and EA independently reviewed and analyzed the focus groups transcripts using a deductive approach. Thematic analysis was applied to identify common themes. Delphi technique analysis was done by calculating the percentage of consensus of each standard and repeated until there was no signi cant difference between percentages calculated.

Results
The results of this qualitative study will be presented into three main sections as following: Step 1: The thematic analysis of the focus group contributions A number of themes emerged from analyzing participant contributions in the focus group as follows:

The role and attributes of the leaders for successful online learning
Most of the participants agreed that leaders should set rules and be decisive especially in time of crisis. Encouraging teamwork and including everyone are the cornerstones for achieving goals. Another important attribute is exibility and how to cope with different personalities and work requirements.
Visionary leaders who predict the future and act proactively to suggest solutions, crisis management, support and sustainability were added as important roles of the leaders. One of the participants added that 'effective leadership requires receiving feedback from signi cant stakeholders including the students and they must be seen and heard for highly relevant feedback'. Proper communication can help in overcoming several obstacles as one participant mentioned 'Leaders should provide early and continuous student and faculty orientation about milestone in the learning experience.'

Resources needed to conduct a successful online learning.
There was a collective agreement among participants on the importance of resource allocation for online learning. These resources include user friendly learning management systems (LMS), internet services, ready-made or self-generated digital tools and equipment to support the online learning. Moreover, an important resource are the personnel involved in the online learning. They also added the importance of having a supportive IT team and conducting a well-organized faculty development program.

Institutional bylaws
The participants' responses varied regarding the modi cations of the bylaws. Some of the participants highlighted the importance of revising the institutional mission and vision. Others con rmed that the mission and vision will not be changed for the use of a new mode of learning. Important points should be added to the bylaws, as per participants, including the ratio of online to face-to-face learning, the assessment plan and attendance ratio in addition to a clear description of online learning competencies and required staff and faculty quali cations.
1.4 Key points to consider while shifting face to face programs into online format.
Participants agreed that institutions have to start with development of the skills and knowledge of their faculty. Faculty should understand the difference between face to face and online learning. The role of faculty, the nature of the content and instructional methods will change with this shift. Also, faculty and students needed to develop some essential skills to cope with this transformation. Materials should be simpli ed, interactive and motivating. Proper platforms or LMSs are important to conduct successful online learning since it is the 'vehicle for all the activities. Use of all the available tools to engage students and create interactive activities such as whiteboard, share-screen, assignments, e-portfolio, online quizzes, online discussion forums. Still there are di culties with conducting practical and clinical sessions. Virtual reality and simulation may help in this point, but funding will remain the main obstacle.
Guidance is required in a comprehensive way in online learning to guide the students and avoid isolation. It is important to provide the students with different alternatives that facilitate and ensure their engagement and participation even with poor internet connections particularly in rural areas. In online learning, mentorship and coaching are needed even more than in face-to-face learning. Finally, nding alternatives for clinical and practical skills teaching is a big challenge. Formative assessment and feedback are also critical points to consider.
1.5 Creating a motivating/engaging environment in online learning. Student engagement was a major problem that faced most of the universities last year as mentioned by most of the participants. Therefore, participants highlighted some important practices that should be considered while implementing online learning. One participant added the need for the use of more formative assessment to keep the students engaged: "Students become more engaged when they are about to have exams…". However, there is a need for redesigning and adapting teaching and learning materials to t the new learning environment as one participant reported that "We need to redesign our lectures to adapt to the new era of online learning…" Most of the participants linked the interesting content with student engagement. Therefore, they recommended the use of gami cations, quizzes, and the use of Multimedia learning principles "If we followed multimedia learning principles, that would help in both instruction & assessment…". Though it is still of high importance to select the suitable platform.
According to the participants, selecting the best model for learning may help the students' development of clinical reasoning skills with the help of scenarios, interactive diagnostic reasoning softwares and virtual simulation.
Student centered approaches and methods can be of great bene t especially in online learning. According to one of the participants "It will allow the students to lead and this may help them to feel secure". Another participant added "Engaging the students with a student-centered activity will get them out of isolation and will help the faculty to detect any student that was left behind". Additionally, the use of group work learning/teaching methods as online TBL may foster the development of a collaborative environment.
1.6 Criteria of effective online assessment Assessment becomes one of the most important dilemmas when shifting to online learning. Ensuring validity and reliability of the exam is a challenge when examinations are done at a distance. "Student Assessment in online learning should be innovative, secure, out of the box, creative and aligned with the teaching methodology." There is also the concept of accessibility and how the exam is made available to students. This requires availability of alternatives and ensuring exibility of format. "Assessment methods used in online learning should be open-book exam, case-based scenarios, single best answer, assignments, virtual OSCE, pattern recognition sessions e.g., histopathological slides, X-rays identi cation, clinical signs. Choosing the suitable online assessment method depends on the nature of the course, the available resources, student number and student staff ratio".

Criteria of online learning evaluation
Types of program evaluation that seemed popular among participants were process and outcome evaluation. However, participants emphasized the need for a comprehensive model of evaluation as the CIPP model because of the complexity of the online learning. "Merging more than one model of evaluation is indicated and highly important in online learning evaluation" was added by one of the participants.
When the participants were asked about the differences between face-to-face course evaluation and online course evaluation, one of the participants mentioned "Modifying and updating the traditional course evaluation surveys to include evaluation of learning management systems (LMS), connectivity and technical support".
"Collecting the contact details of the registered students is an important step to facilitate online courses evaluation" was added by one of the participants.
Using different data collection tools in online formats was recommended by participants, surveys and students quizzes are preferred.
Participants agreed that student engagement in online learning should be evaluated in terms of student interaction, performance and assessment. "Learning management systems (LMS) analytics such as submission of assignments, synchronous sessions attendance and dropout rate are indicators for students engagement". Finally, nding the suitable benchmark program and logic model for evaluation is of high importance as well as the need for external peer review to validate the evaluation process.

Important Evaluation Questions
Participants agreed on some areas that should be covered in evaluation questions e.g. evidence to prove learning (prove of learning), student satisfaction, management system analytics, faculty performance, student interaction, and student performance including the analysis of quiz grades, drop out rate, delay in assignment submission, discussion participation.
Examples of evaluations questions as suggested by most of the participants included: -If this course/program is optional, would students apply for it? -Are the staff satis ed with the online learning experience? Do they prefer face to face learning? -Does the program/course help students to learn and grow? -Does online learning help tutors/faculty to be better teachers?
1.9 Faculty, Students, Administration attributes in online learning.
Most of the participants discussed the competencies faculty members should have in online learning. They agreed that all faculty members should be skilled in using technology and online platforms and show creativity and innovation. Faculty also should show pro ciency in online communication, course design, online assessment, time management and in engaging students in an online learning environment. The participants also stressed on the importance of student-centered strategies to be adopted by faculty members.
Beside the proper use of technology in learning, engagement, critical thinking, collaboration, teamwork and communication, students should also know how to manage learning in an online context. To master such a skill, they should be familiar and trained on self-regulated learning, time management, setting goals and how and when to seek help.
The role of administration in assuring quality of online learning is an integral one. Participants nominated different attributes that will help administration maintain quality online learning including management skills, technical skills, strategic planning attributes and risk management, decision making, ethics and professionalism, communication, monitoring and evaluation.
Step 2: Formulation of descriptors of best practice The focus group contributions were analysed and reformulated by the authors into quality standards and indicators. Three main quality areas were identi ed: Organizational capacity, Learning and assessment and Human resources. The standards were designed as follows: 1. ORGANIZATIONAL CAPACITY

Governance
School Leadership is accountable and committed to support and lead the institution to delivering quality online education.

Indicators:
1. Leadership encourages a collaborative environment to plan, implement and monitor the quality of online learning activities.
2. Leadership shares and cements the values, beliefs, and the operational expectations for a quality online learning.
3. Leadership holds themselves accountable to disclose accurate information about the recruitment process, policy, fees, courses/programs, and reports.
4. Leadership demonstrates proactive understanding and analyzing organizational needs to deliver effective online education.
5. Leadership creates a culture of acceptance and encouragement for online learning.
6. Leadership delegates responsibility to multidisciplinary teams and facilitates their work to implement and monitor online learning activities.

Resources:
Resources for the online learning are allocated in a fair, reasonable manner that responds to the identi ed needs Indicators: 1. Presence of a learning management system (LMS) that ensures user-friendly and secure online environment.

Presence of accessible Internet services
3. Presence of digital tools that are aligned with the educational needs of learners.
4. Presence of the equipment that support successful online learning.

5.
Presence of trained technical support team.
6. Financial resources allocated to online learning.
7. Provisional needs documents are available designed annually and approved by proper authorities.
8. Budget is well-managed in a transparent and documented way.

Organizational bylaws
Bylaws clearly de ne the administrative issues, credit points calculation and the roles and responsibilities of team members Indicators:

Presence of written policies & procedures or all online courses
2. There is a de ned and documented process related to the online programs 3. There is a documented clear policy governing the ongoing training and support to the working staff.
4. All students have equitable access to the online learning resources.

Educational Program
The program has a clear robust design that respects the school vision, mission, and values and that demonstrates a clear understanding of the nature of the required graduate attributes. Indicators: 1. There is an approved, updated and well-constructed longitudinal online education plan that includes su cient data to support decisions and is aligned with the educational program.
2. There are aligned and cascaded goals: strategic, long term, intermediate and short-term goals.
3. There is clear identi cation of the required resources to ensure sustainability of the online programs and courses. 4. The instructional materials contribute to the achievement of the stated learning objectives or competencies.

5.
Online instructional methods and tools support active learning, student involvement, support interaction amongst students and between instructors and students and are based on recent best practices.
6. Online instructional methods are variable and support development of higher order thinking.
7. The relationship between learning objectives or competencies and course activities is clearly stated.
8. There is use of digital tools that best support students' involvement and better understanding of the learning material.
9. Learning and assessment schedules are clear, applicable, and fair for all students.
10. Online student Assessment methods planned are clear, fair for all students and include frequent formative assessment with feedback and summative assessment with clear and transparent reporting.

Course delivery
Courses should be delivered in the safest most accessible way providing standardized learning opportunities. Indicators: 1. There is a plan for frequent evaluation that is approved and implemented with identi ed data collection methods eg. observation, questionnaires, focus group.
2. There is a well-organized plan for delivery with a backup.
3. A troubleshooting and complaint policy and procedure exists and is announced and used by learners.
4. Designed learning activities are implemented with minimal deviation from plans.
5. Technologies required in the course are readily obtainable.

Student assessment
Student assessment to measure student achievement using multiple assessment methods that align with the learning objectives and the instructional methods. Data from assessment is evaluated and feeds into educational decision making.

Indicators
1. Digital tools are used to ensure secure, fair, valid, and applicable assessment.
2. There is use of multiple assessment methods to measure cognition, skills, and attitude of the students.
3. There is use of frequent formative assessment with feedback for better learning.
4. There are clear reports after the summative assessment.
5. There is a plan for academic counseling that is clear, manageable and is executed.

Evaluation
Educational monitoring and evaluation plans are available with clearly assigned evaluation questions, key performance indicators and assigned personnel. The plan is implemented and the information it generates feeds into the educational replanning. 2. There is documented periodic evaluation by external reviewers to validate the internal evaluation process and assess the goal achievement.
3. There is disclosure of the evaluation results with the stakeholders.
4. Data is used to drive decisions for continuous improvement.

III. Human Resources
The organization has personnel who can manage the educational process effectively and who are under continuous monitoring and development. Indicators: 1-Faculty 1. There is a wide variety of professional development activities for the faculty pertaining to skills needed for online education.
2. There is timely and effective technical support to the faculty.
3. There is timely, frequent, and constructive feedback about instructor performance.
4. Faculty have an opportunity to add to their professional portfolio within online learning in the school.
5. The number of assigned faculty is reasonable, su cient, and aligned with the student number and educational activities.
6. There is a clear de nition of faculty roles and responsibilities.

2-Students
1. There is brie ng and orienting the students about the accessibility and availability of the online learning resources and digital tools.
2. Equity and accessibility to technology to all students is ensured.
3. There is timely and effective technical support to students to overcome limitations of technology & computer literacy.
4. There is timely and constructive feedback to students.
5. There are guidelines for student-teacher and student-student communication.

3-Administration
1. There exists a supporting administration team that is reasonable and aligned with the educational processes, number of students, number of faculty etc.
2. There is a solid development plan for administration of the online learning program.
3. There is a clear role de nition for administration.
4. There is a de nite pathway for troubleshooting and for complaints for administrators in the program.
Step 3: Expert Consensus session response and Delphi Technique The above-mentioned descriptors of online learning were handled by experts the following results were achieved: A-All suggested standard areas were agreed upon with no further additions or amendments by 100% of experts except: B-The following standards were amended: 1. Digital tools are used to ensure secure, fair, valid, and applicable assessment.
Suggested amendment: Digital tools need to be more speci ed into KPI e.g., number of trained faculty on the digital tools of assessment, blueprinting to ensure content validity.
1. There is a plan for academic counseling that is clear, manageable and is executed.

Suggestion addition of:
Supported by the administration.
C-The following standards were proposed to be added (Table 1).
Based on the above ndings and recommendations a set of checklists were developed (table 2-4).

Discussion
The use of multi-level analysis to achieve a consensus report is not new to the scienti c community and was used by Gorard, 2003;Pahor and Novak, 2017; Shively and Smith, 2019 (11,12,13). The main focus of this work was to establish a regional consensus statement around what constitutes effective online teaching practice and thus develop guides to be used when evaluating an online teaching experience. The qualitative nature of the work cherishes the experience of individuals who have utilized crisis management mode post COVID-19 to achieve the most possible educational effectiveness. This experience came with trials and errors and thus lessons that need to be documented and acknowledged. The reliance on expert consensus was used before by Minas and Jorm (14) and Kern (15).
There are three areas of particular concern in developing standards representing best practice in online learning. These were organizational capacity, effective learning and assessment, and human resources.
This is in agreement with the developed standards by Kennedy (16), Dawson et al. (17), and Skiba (18).
This categorization laid signi cant importance on the human resource factor and identi ed it as a separate standard area in itself. This could be due to the large transition performed by faculty and administration at an unexpected pace after COVID-19. With this transition a lot of human resource adaptation and development was required (19)(20).
Much emphasis was placed on the use and the development of the learning management systems and developing tools to ensure its reliability and user friendliness. This is in agreement with Radwan et al. When experts handled the standards, it was obvious that there was an inherent need for quanti cation in order to set benchmarks. This is supported by the work of many other researchers (24,25,(27)(28)(29)(30)(31)(32)(33)(34)(35)(36)(37). This fact alone is important to highlight the need for regional standards that adapt easily to the needs of schools in different areas. These standards are thus intended to guide country adaptations and understanding of standards to suit their own practice line.

Conclusion
Ensuring the quality of online learning is of utmost importance especially during the times of crises (9) and dependence totally or partially on online learning. Effort was exerted into experience to apply scienti c methodology in identifying the different aspects of best practice descriptors and their success indicators. This included all elements related to online learning environments and processes and all stakeholders involved.
This work provides educators, institutions, and evaluators of educational practices with comprehensive recommendations that address three important axes, which are: a) institutional capacity, b) effective learning and assessment, and c) human resources. Taking such axes into consideration by educators and institutions will lead to planning and implementing successful online learning activities, while taking them into consideration by the evaluators will help them conduct comprehensive audits and provide stakeholders with highly informative evaluation reports.
Abbreviations CME continuing medical education (CME)

Declarations
Ethics approval and consent to participate: All methods were performed in accordance with the relevant guidelines and regulations. The work was approved by the Research and Ethics Committee (REC) of Ain Shams University under number FX2002-8/20. Informed written consent was obtained from the participants.The participants were informed about the purpose of the study and its relevance to the eld of medical education. Only those who signed a written consent to be involved in the study were included under the reassurance that participant names and a liation were to remain highly con dential.
Consent for publication: All authors approve to publish the work.
Availability of data and materials :The datasets used and/or analyzed during the current study are available at : https://dataverse.harvard.edu/dataverse/online_learning_guide Competing interests: The authors declare that they have no competing interests.