It’s what you do, Not the way that you do it – Online Versus Face to Face Small Group Teaching in First Year Medicine.

doi:10.21203/rs.3.rs-701211/v1

Background

Major disruptions imposed on medical education by the COVID-19 pandemic and the rapid shift to online teaching in medical programs, necessitated need for evaluation of this format. In this study we directly compared knowledge outcomes, social outcomes, and wellbeing of first year student small group teaching in either face to face (f2f) or online format.

Methods

At the end of the first course of our medical program, students were invited to participate in an online questionnaire with 10 quantitative items and 1 qualitative item. These were analysed using Factor Analysis Pattern Matrix and linear regression to group items and assess relatedness. Qualitative responses were thematized using Qualtrics software (Qualtrics, Provo, UT, USA). Summative assessment results were compared, both between current cohorts to historical cohorts.

Results

From a cohort of 298 students there was a 77% response rate. Overall, there were no differences in knowledge gains, either between groups or when compared to historical cohorts. Questionnaire items fell reliably into groups that related to either learning outcomes, social outcomes, or wellbeing. Independent T tests showed that format for teaching (online versus f2f) had an impact on social outcomes but no direct impact on learning outcomes. Linear regression revealed that the social outcomes have a direct impact on wellbeing and almost the double the impact on learning outcomes than mode of learning ie. F2f or online (β=.448 and β=.232 respectively).

Conclusion

In this study, we were able to show with statistical strength that social outcomes for students such as engaging with peers and facilitator, contributing to the group, and making friends have a direct impact on wellbeing and indirectly impact learning outcomes (such as motivation, satisfaction, integration of knowledge). In a rapidly changing educational landscape, it is vital that these aspects are a focus of design and delivery of medical education. The data from this study supports the notion that activity design and the expertise of the teacher in facilitating the small group activities, is more important than the mode of educational delivery itself.

Educational Philosophy and Theory

Educational Psychology

Special Education

Medical education

online learning

educational design

engagement

wellbeing

social outcomes

Like many other medical faculties, during the COVID-19 pandemic, campus-based courses in the early years of our medical program were transitioned online. Then, as our campus re-opened, these changes were reviewed and the cornerstone learning unit of junior years of our medical program, our scenario learning groups (1), were re-established as face to face activities for all students, except those that were still offshore due to ongoing international travel restrictions. It is likely that more disruptions will occur throughout the ebbs and flows of this pandemic. To minimise disruption of medical education, educators will need to continue to build resilience into the way they deliver medical education, evaluate, and produce data around contemporary ‘best practice’ and how this looks in a pandemic-constrained environment (2).

During the first two years of our (undergraduate) medical program, the courses are all inter-disciplinary and contain a variety of learning activities including lectures, practical classes, clinical and communication skills sessions, and scenario-group sessions (SGS). The SGS have a variety of names such as problem-based learning this approach is commonly used in medical programs (3). SGs are considered to be key learning sessions in medical training programs for many reasons, largely because the students are developing skills in small groups for self-directed and lifelong learning which include problem solving, critical thinking, team-based learning and reflection, as well as delivery of content (4, 5). This session is thought to be the most important in the course both for the development of skills such as these learning skills and also for the establishment of relationships with other students, friendships and relationships with teachers (6, 7). It is during these SG sessions that students generally have the most opportunity to develop learning skills and also for collaborative learning, sharing knowledge and what can be called communal ‘constructivism’ in relation to learning (8). For this reason, we undertook this study to assess the impact of these SG sessions being taught online as opposed to face to face (f2f).

In 2021, almost one third of our first-year student cohort were still offshore, unable to physically enter Australia due to international travel restrictions. All educational activities in the first course of the medical program were still being conducted online at that time, apart from the SGS. This allowed us to directly compare impact of the two formats of this key learning activity. We conducted a study to examine the impact of the mode of delivery (online versus f2f) on learning outcomes, learning outcomes, social outcomes, and wellbeing.

At the end of the Foundations course in the undergraduate medical program, all enrolled students were invited to participate in a web-based survey (Appendix A) which was created using Qualtrics software (Qualtrics, Provo, Utah, United States) and distributed to all students via internal notification in April 2021. The survey featured Likert scale questions about students’ experience of learning outcomes, social engagement, and wellbeing. There were 9 quantitative items in total, which could be grouped into categories of either learning outcomes or social outcomes and one on wellbeing (summarised in Table 1). Qualitative data was also collected about the best features of these learning sessions. The responses were then filtered according to whether the student was in a f2f or online group and responses of the two cohorts were compared. Objective knowledge gains of students were assessed by summative end of course assessments were compared between the two current cohorts and compared to historical cohorts 2019 and 2020).

Statistical analysis

Exploratory Factor analysis (Maximum Likelihood, using Oblimin Rotation with Kaiser Normalization) was employed to identify the underlying factors of student perceptions. Independent t-tests were used to measure the differences in student perceptions between the f2f and online groups. Multiple linear regression model was used to identify the predictors of learning outcomes, and path analysis, using structural equation modelling, was used to identify direct and indirect impacts on student wellbeing. Analyses were conducted by SPSS v22 and R (9–11).

From a cohort of 298 students there were 213 respondents (77% response rate) with a total completion rate of 97.7%. Of participants that completed the survey, 143 were f2f and 65 were online which represented very good and equitable sampling across both groups.

Summative assessment of knowledge gains

No differences were found in relation to performance at end of course exams between offshore (international students), onshore international students and domestic students in 2021. There was also no difference when compared with historical cohorts (2020 was taught entirely online and 2019 was taught entirely face to face).

Impact of f2f versus online learning on learning and social outcomes

An exploratory factor analysis was done to assess the contribution of the factors to learning outcomes and social outcomes (Table 1). This shows the significant groupings of questionnaire items into those that relate to the learning outcomes (light grey) and those that relate to the social outcomes (dark grey). We were able to use these groups for factor analysis and assess their impact on each other. The reliability of each factor is factor high, apart from items 2 and 10 in this analysis (unshaded).

Table 1

Factor Analysis Pattern Matrix^a: Student perception of their learning experiences
Questionnaire item	Factor
	1	2
Q4 Activities Enhanced Learning	.969	− .126
Q5 Useful for Integrating Knowledge	.921	− .101
Q3 Enhanced Motivation to Learn	.752	.136
Q1 Satisfied with SG Sessions	.630	.280
Q10 Enhanced Wellbeing	.476	.433
Q9 Easy to Make Friends	− .227	.928
Q8 Easy to engage with other students	− .062	.895
Q6 Easy to contribute to Group	.124	.588
Q7 Easy to Engage with Facilitator	.144	.584
Q2 Enjoyed Learning Format	.235	.428
Reliability Cronbach's alpha	.905	.845
Extraction Method: Maximum Likelihood. Rotation Method: Oblimin with Kaiser Normalization. a. Rotation converged in 6 iterations.

An independent T test was performed to see if the format for teaching (online versus f2f) had an impact on these outcomes. They show a significant impact on the social outcomes but not the learning outcomes as shown in Table 2.

Table 2

Independent Samples Test: Impact of teaching format on learning and social outcomes
		Levene's Test for Equality of Variances		t-test for Equality of Means
		F	Sig.	t	df	Sig. (2-tailed)	Mean Difference	Std. Error Difference	95% Confidence Interval of the Difference
		F	Sig.	t	df	Sig. (2-tailed)	Mean Difference	Std. Error Difference	Lower	Upper
Learning Outcomes	Equal variances assumed	1.995	.159	-1.629	206	.105	− .22634	.13898	− .50035	.04767
Learning Outcomes	Equal variances not assumed			-1.806	160.741	.073	− .22634	.12530	− .47380	.02111
Social Outcomes	Equal variances assumed	19.313	.000	3.966	206	.000	.48054	.12117	.24164	.71943
Social Outcomes	Equal variances not assumed			3.279	84.109	.002	.48054	.14653	.18914	.77193

This is also shown by Fig. 1 on learning outcomes and social outcomes, which graphically shows the better social outcomes in students who had face to face scenario group teaching, with no significant difference between the groups in terms of their perceived learning outcomes.

When we used linear regression to examine the impact of both mode of learning and social outcomes on the learning outcomes, there was a high degree of reliability (as shown by the Pattern Matrix in Table 1). Linear regression revealed that the social outcomes have almost the double the impact on learning outcomes than mode of learning ie. f2f or online (Table 3, β = .448 and β = .232 respectively).

Table 3

Linear regression analysis of impact of mode of learning and social outcomes on perceived learning outcomes
		Coefficients^a
Model		Unstandardized Coefficients		Standardized Coefficients	t	Sig.	95.0% Confidence Interval for B
Model		B	Std. Error	Beta	t	Sig.	Lower Bound	Upper Bound
1	(Constant)	2.202	.393		5.601	.000	1.427	2.977
	Social outcomes	.498	.072	.448	6.902	.000	.356	.640
	Mode of Learning	.466	.130	.232	3.576	.000	.209	.722
a. Dependent Variable: Learning Outcomes ; R2 = 0.199

Impact on Wellbeing

In order to assess the impact of these factors (learning outcomes, social outcomes and mode of learning) on student wellbeing, we looked at both the direct and indirect impact of these variables using path analysis is illustrated in Fig. 2 (9).

A positive rating of the learning outcomes had the most significant positive correlation with ratings of wellbeing. Importantly, mode of learning did not have any direct impact on student wellbeing (Table 4). The impact of the mode of learning on wellbeing was insignificant.

Table 4

Effect of variables (factors) on Wellbeing (Path Analysis)
Direct, Indirect and Total Effects
				95% C.I. (a)
Type	Effect	Estimate	SE	Lo	Hi	β	z	p
Indirect	Mode of Learning -> Learning Outcomes -> Enhanced Wellbeing	0.292	0.0865	0.122	0.461	0.123	3.369	< .001
	Social Outcomes -> Learning Outcomes -> Enhanced Wellbeing	0.313	0.0547	0.206	0.420	0.240	5.720	< .001
Component	Mode of Learning -> Learning Outcomes	0.463	0.1297	0.209	0.717	0.231	3.573	< .001
	Learning Outcomes -> Enhanced Wellbeing	0.629	0.0622	0.507	0.751	0.535	10.123	< .001
	Social Outcomes -> Learning Outcomes	0.498	0.0718	0.357	0.638	0.448	6.934	< .001
Direct	Mode of Learning -> Enhanced Wellbeing	-0.020	0.1195	-0.254	0.215	-0.008	-0.164	0.87
	Social Outcomes -> Enhanced Wellbeing	0.437	0.0713	0.297	0.577	0.334	6.131	< .001
Total	Mode of Learning -> Enhanced Wellbeing	0.272	0.1421	-0.007	0.550	0.115	1.913	0.056
	Social Outcomes -> Enhanced Wellbeing	0.750	0.0787	0.596	0.904	0.574	9.533	< .001
Note. Confidence intervals computed with method: Standard (Delta method)

Qualitative Results

Thematic text extracts were automatically produced by Qualtrics with inherent sentiment analysis and were similar across both groups in relation to the themes and best aspects of the SGS. This showed us that the best thing reported by the most students were the facilitators (teachers) of the groups. Other positive themes that emerged included: learning in small groups, revisiting lecture content, creation of a safe environment, interacting with peers, sharing ideas and making friends.

As medical educators reflect on and evaluate the rapid changes that occurred out of necessity during 2020 (12-17), it will be important to evaluate what changes can and should remain, what the impact of various changes is on student learning and how these changes affect other parts of the student experience, such as wellbeing and engagement. This is particularly relevant to the teaching in the early campus-based years of medical programs, where there can be many benefits of shifting educational activities to an online format. These include flexibility (geographical and time-related), scalability, interactivity, and quality control (12, 18). Students have given us feedback that all these things are better in the online environment, if appropriate effort is put into the design of educational resources (12, 19). Online teaching in medical programs is not new, but has been expedited during COVID-19 (20). One very important finding in our data set is that the mode of learning (online versus f2f) did not actually have an impact on learning outcomes of students either from the student’s perception or objectively measured via the summative end-of-course examinations. This major finding has also been reported elsewhere (21, 22). Two meta-analyses that have examined learning knowledge gains, skill gains and satisfaction of medical students have also found no difference between learning in a f2f environment and slightly better performance in the online environment for some parametres (23, 24).

Our study found that the mode of teaching was much less important than the social outcomes (engagement with peers and facilitators, making friends) on both learning outcomes and student wellbeing. Learning outcomes and knowledge gains can be maintained and optimised if appropriate work is put into educational design (24). In fact, the opportunity (and need) to develop learning materials for online delivery of healthcare education, may have enhanced these activities by supporting instructional design and embedding opportunity for rapid feedback.(25, 26) One of the challenges in e-learning is that not all activities work well in the online environment and this needs to be taken into consideration, when designing the activities (18).

Our study found that the social processes which include engagement with peers and facilitators and making friends was affected by the mode of delivery. This is not surprising, given that the small group learning such as SGS are the main learning activity during which these social processes can occur for the students. The group work arising out of SGS often require students to organise study sessions in addition to formal timetabled ones, allowing friendships to flourish and more informal communication. This is more constrained when all these activities must occur online (27).

Wellbeing was affected by both learning outcomes and social outcomes but not independently by mode of learning. The impact of social outcomes such as making friends and engaging with peers and facilitators had double the impact of the learning outcomes. It has also been shown elsewhere that the development of supportive relationships with peers and staff is very important for student wellbeing.(28, 29) Issues which impact the building of peer relationships and friendships, such as the constraints of an entirely online environment, have a negative impact on wellbeing (12, 18, 28). Other studies have also shown that stress relating to academic performance has also been shown to have a negative impact on mental health in medical students and many medical programs are now being proactive in efforts to address this (28, 30).

Our qualitative data also tells us that in both environments, the impact of the facilitator on the student experience is very important. In the online environment, the facilitator has an even greater role in keeping students engaged and focused (31). It is also likely that they benefit from specific training on how to best teach in this environment, as there are issues to manage that do not arise in the f2f environment, such as the online etiquette, building physically-distanced community for collaborative learning, and the use of specific online tools (14, 32, 33).

Limitations of this study

It was beyond the scope of this study to assess the impact of mode of delivery on the development of a range of individual skills and competencies taught across courses in medical programs and gradually developed by students, that may be more affected by mode of learning such as teamwork, communication, professionalism, and cultural safety. As mentioned in the meta-analysis by Pei (21) there are many items and factors that could be used in the analysis of online learning, in this study we have only looked at a limited number of items.

Ultimately the findings of our study support the idea that educational design and delivery is much more important than the mode of delivery on both learning outcomes and social processes. Social processes are more impacted than the learning ones by mode and this can have an impact on student wellbeing. This study highlights the need for educators and students to put additional thought into how to address these aspects in the online learning environment. This data emphasises the need for high quality educational design and iterative improvement whatever format is used (24, 34) as well as support and development of highly competent educators who are environment ambidextrous, so that student engagement is optimised (35). Putting energy into the development of contemporary educational resources and the development of our facilitators also builds resilience into our programs, our staff and our students, improves equity and will set medical programs up well against future disruptions.

Face to face: f2f

Scenario group: SG

SARS-CoV-2 virus: COVID-19

Ethics approval and consent to participate

- This study approval ethics approval from the UNSW Sydney Human Research Ethics Committee on , Ethics approval: UNSW HC210117 on 9/3/21. This included approval of the participant information and consent forms. Consent from study participants was in written form. All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

– Not applicable

Availability of data and materials

- The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

- The authors declare that they have no competing interests

Funding

– There was no funding for this research

Authors' contributions

– AT was responsible for the study concept, design, collection and writing of the manuscript; BS was responsible for the statistical approach, analysis and writing of the manuscript. All authors have read and approved the manuscript

Acknowledgements

– not applicaAble

McNeil HP, Hughes CS, Toohey SM, Dowton SB. An innovative outcomes-based medical education program built on adult learning principles. Med Teach. 2006;28(6):527–34.
Ashokka B, Ong SY, Tay KH, Loh NHW, Gee CF, Samarasekera DD. Coordinated responses of academic medical centres to pandemics: Sustaining medical education during COVID-19. Med Teach. 2020;42(7):762–71.
Aarnio M, Nieminen J, Pyörälä E, Lindblom-Ylänne S. Motivating medical students to learn teamwork skills. Med Teach. 2010;32(4):e199-204.
Neville AJ. Problem-based learning and medical education forty years on. A review of its effects on knowledge and clinical performance. Med Princ Pract. 2009;18(1):1–9.
Thistlethwaite JE, Davies D, Ekeocha S, Kidd JM, MacDougall C, Matthews P, et al. The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide No. 23. Med Teach. 2012;34(6):e421-44.
Schmidt HG, Rotgans JI, Yew EH. The process of problem-based learning: what works and why. Med Educ. 2011;45(8):792–806.
Tolsgaard MG, Kulasegaram KM, Ringsted CV. Collaborative learning of clinical skills in health professions education: the why, how, when and for whom. Med Educ. 2016;50(1):69–78.
Servant-Miklos VFC, Woods NN, Dolmans D. Celebrating 50 years of problem-based learning: progress, pitfalls and possibilities. Adv Health Sci Educ Theory Pract. 2019;24(5):849–51.
2020-08-24). RCTRfhcr-poRprfMs. R: A Language and environment for statistical computing. (Version 4.0) [Computer software].. (R packages retrieved from MRAN snapshot 2020-08-24).
Rosseel Y. Lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software,. 3012;48(2):1–36.
Field A. Discovering Statistics Using IBM SPSS Statistics. 5th ed2018.
Torda A. How COVID-19 has pushed us into a medical education revolution. Intern Med J. 2020;50(9):1150–3.
Alsoufi A, Alsuyihili A, Msherghi A, Elhadi A, Atiyah H, Ashini A, et al. Impact of the COVID-19 pandemic on medical education: Medical students' knowledge, attitudes, and practices regarding electronic learning. PLoS One. 2020;15(11):e0242905.
Cridland C, Drape T, Marine S, Gillaspy G. Ten best practices for taking experiential learning online. Biochem Mol Biol Educ. 2021;49(1):9–14.
Gordon M, Patricio M, Horne L, Muston A, Alston SR, Pammi M, et al. Developments in medical education in response to the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63. Med Teach. 2020;42(11):1202–15.
Kim JW, Myung SJ, Yoon HB, Moon SH, Ryu H, Yim JJ. How medical education survives and evolves during COVID-19: Our experience and future direction. PLoS One. 2020;15(12):e0243958.
Li W, Gillies R, He M, Wu C, Liu S, Gong Z, et al. Barriers and facilitators to online medical and nursing education during the COVID-19 pandemic: perspectives from international students from low- and middle-income countries and their teaching staff. Hum Resour Health. 2021;19(1):64.
Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair L. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11):e042378.
Torda AJ, Velan G, Perkovic V. The impact of the COVID-19 pandemic on medical education. Med J Aust. 2020;213(4):188-.e1.
Cheng SO, Liu A. Using online medical education beyond the COVID-19 pandemic - A commentary on "The coronavirus (COVID-19) pandemic: Adaptations in medical education". Int J Surg. 2020;84:159–60.
Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online. 2019;24(1):1666538.
Kronenfeld JP, Ryon EL, Kronenfeld DS, Hui VW, Rodgers SE, Thorson CM, et al. Medical Student Education During COVID-19: Electronic Education Does Not Decrease Examination Scores. Am Surg. 2020:3134820983194.
George PP, Papachristou N, Belisario JM, Wang W, Wark PA, Cotic Z, et al. Online eLearning for undergraduates in health professions: A systematic review of the impact on knowledge, skills, attitudes and satisfaction. J Glob Health. 2014;4(1):010406.
Sandars J, Patel R. The challenge of online learning for medical education during the COVID-19 pandemic. Int J Med Educ. 2020;11:169–70.
Lewis KO, Cidon MJ, Seto TL, Chen H, Mahan JD. Leveraging e-learning in medical education. Curr Probl Pediatr Adolesc Health Care. 2014;44(6):150–63.
McCutcheon K, Lohan M, Traynor M, Martin D. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. 2015;71(2):255–70.
Takizawa PA, Honan L, Brissette D, Wu BJ, Wilkins KM. Teamwork in the Time of COVID-19. FASEB Bioadv. 2020;3(3):175–81.
Byrnes C, Ganapathy VA, Lam M, Mogensen L, Hu W. Medical student perceptions of curricular influences on their wellbeing: a qualitative study. BMC Med Educ. 2020;20(1):288.
McNeill KG, Kerr A, Mavor KI. Identity and norms: the role of group membership in medical student wellbeing. Perspect Med Educ. 2014;3(2):101–12.
Hill MR, Goicochea S, Merlo LJ. In their own words: stressors facing medical students in the millennial generation. Med Educ Online. 2018;23(1):1530558.
Coiado OC, Yodh J, Galvez R, Ahmad K. How COVID-19 Transformed Problem-Based Learning at Carle Illinois College of Medicine. Med Sci Educ. 2020:1–2.
de Nooijer J, Schneider F, Verstegen DM. Optimizing collaborative learning in online courses. Clin Teach. 2021;18(1):19–23.
Abbasi MS, Ahmed N, Sajjad B, Alshahrani A, Saeed S, Sarfaraz S, et al. E-Learning perception and satisfaction among health sciences students amid the COVID-19 pandemic. Work. 2020;67(3):549–56.
Loda T, Löffler T, Erschens R, Zipfel S, Herrmann-Werner A. Medical education in times of COVID-19: German students' expectations - A cross-sectional study. PLoS One. 2020;15(11):e0241660.
Jiang Z, Wu H, Cheng H, Wang W, Xie A, Fitzgerald SR. Twelve tips for teaching medical students online under COVID-19. Med Educ Online. 2021;26(1):1854066.

No competing interests reported.

FoundationsmanuscriptAppendixA.docx

It’s what you do, Not the way that you do it – Online Versus Face to Face Small Group Teaching in First Year Medicine.

Status:

Version 1

Abstract

Figures

Background

Methods

Statistical analysis

Results

Summative assessment of knowledge gains

Impact of f2f versus online learning on learning and social outcomes

Impact on Wellbeing

Qualitative Results

Discussion

Limitations of this study

Conclusion

Abbreviations

Declarations

Ethics approval and consent to participate

Consent for publication

Availability of data and materials

Competing interests

Funding

Authors' contributions

Acknowledgements

References

Additional Declarations

Supplementary Files

Status:

Version 1