The Learning Effect of Online Discussions: New Active Form of Learning under COVID-19

Background: COVID-19 created confusion in all areas of society and is no exception in education. In line with the demands of the times, we investigated the applicability of active learning online for the health professional. Methods: This study was conducted in the school of dentistry, a selective university. Eighty-two dentistry students participated in this study in the second semester of 2020. The students were randomly assigned to four different groups and learned according to given experimental procedures. Their performance in the nal test scores was analyzed using ANOVAs. Subsequently, we systematically analyzed past oine active learning and online experiments. Results: As with past oine experiments, self-study before discussion groups have achieved high learning outcomes in verbatim type and transfer type items. Online self-study groups scored slightly lower than oine self-study groups, but online self-study groups scored higher than oine and online lecture groups. Conclusions: Our ndings denote that active learning for health professionals was applicable online. When self-study precedes the discussion, it becomes richer, thereby increasing students' learning outcomes.

made changes in response to contemporary needs and have attempted to create better, more effective learning methods for their students. For decades, these educators have been working to transform their pedagogy by reducing or even eliminating lectures [9,10] and by implementing team-based, active, and self-directed learning [11,12]. Within the past few years, medical educators have been converting classroom-style learning to one-way lectures and providing instruction for asynchronous learning [13]. Similarly, an alternative to lectures is having discussions, a student-centered learning method. Evidence that discussion facilitates learning was con rmed in experimental studies comparing the effectiveness of learning after watching one-way lectures and after watching discussion-style videos online [14,15]. The results of this experiment showed that students who watched the discussion-style videos, and those who watched them while discussing the learning materials, had greater performance. However, many students majoring in health professions still study in monologue-style lecture sessions, laboratory sessions, and simulations [9].
In addition, there has been a lack of quantitative and empirical research on students' performance in online education under COVID-19 from the medical eld, including medical education and other elds related to health science (e.g., dentistry) [16,17,18]. It is especially rare to nd a study that evaluates student achievement by conducting active learning online. In this study, we conducted a study online that provided active learning for health professionals before the spread of COVID-19 [10]. In previous research, Lim and colleagues explored the ICAP framework, in which active pre-learning activities should be preceded by the discussion, and whether the framework can be utilized more effectively to train health professionals [10,19,20]. They proved the effectiveness of discussion on student's learning within health-related subjects including medicine, dentistry, veterinary medicine, and nursing and concluded when discussion was most effective. Therefore, based on the study of desirable active learning in health professions, while effectively replicating past experiments, we seek to investigate the direction of active learning online that ensures the quality of learner-centered classes that can be effectively applied in future pandemics and other crises.

Methods
For this experiment, we predetermined sample sizes of at least 19 participants per group in order to compare with the o ine experiment before COVID-19 [10]. In addition, we conducted power analyses using the G*Power to determine whether the design in this experiment had enough power [21]. Groups of 19 participants provided 80% power to detect an effect size of 0.87 (alpha =.05).

Participants
Participants (N = 82) were recruited from school of dentistry at Seoul National University for this study.
The study was reviewed and approved by the Institutional Review Board (IRB) of Seoul National University School of Dentistry (approval No. S-D20200053).

Survey on Prior Knowledge
We used a survey to check participants' prior knowledge or interest on the stimulus topic in order to minimize the effect of prior knowledge on the experiment [22,23]. Six survey items were assessed using a 7-point Likert scale ranging from "do not know about it at all (1)" to "know very well about it (7)." The survey included six items, two of which were related to the topic of the learning material (the criminal procedure code and the accusation and charge) and four that were unrelated (the genome project, civil law, the legalization of same-sex marriage, and the Special Acton Sexual Violence).

Video Lecture and Learning Material
The video lecture used in this study showcased a monologue-style lecture. The lecture was on the topic of law, particularly concerning the criminal procedure code, accusation, and charge. This topic was chosen because it is not typically taught in undergraduate courses. This is an especially unfamiliar topic for students studying health. This topic was also chosen because its assessments include a nite set of answers for transfer type items used to evaluate students' learning outcomes. The video lecture was 18 minutes in length.
Learning materials derived from the contents of the video lecture were seven pages long. These materials also covered information on the same topic: the criminal procedure code, accusation, and charge.

Final Test Items
Items for the nal test were comprised of verbatim type and transfer type items. Verbatim type items were comprised of 10 questions with answers that were directly inferable from the learning materials. These questions consisted of short answer or multiple choice questions, with a total of 10 questions worth one point each. For example, one question states, "Given that there is no one who led an accusation against a crime subject for prosecution, prosecutors must designate a person who can le the complaint within ( ) days upon the request of the stakeholders." Students who lled in a correct answer would have written "10" in the blank space given.
Transfer type items consisted of four questions, each requiring total comprehension of the given materials and application of these materials to new situations. These items were 15 points each; thus, a perfect score on these test items would total 25 points. An example of these questions is as follows: "The under-aged victim (V) accused the defendant (D) of rape, but eventually withdrew her accusation on February 1st, 2017. Afterwards, V's father (F), legal representative of V, accused D on February 10th, 2017. D was charged with rape and was convicted of the crime at the rst trial. However, D made an appeal claiming that F's complaint should not take effect since V had already withdrawn her complaint, and therefore, the prosecutor's indictment was against the provisions of the law. Will the Court of Appeals accept D's claim?" Procedure This experiment was designed to replicate and build off of the study conducted in the past that used an o ine lecture and set of learning materials [10]. At the same time, this experiment was designed to determine whether the same extent of active learning is possible through online discussion. Thus, the overall experimental methods and procedures were largely the same, excluding one variable that posed questions on the learning processes, which did not prove effective in the past experiment. The use of worksheets alongside instruction by the lecturer or professor was added as a new variable.
All students participated in this experiment through Zoom, an online meeting platform. Participants were instructed to ll out the background knowledge survey to determine their level of prior knowledge and interest. Participants were randomly assigned to groups of two different study conditions: lecture and self-study. Through the breakout rooms function within Zoom, students were randomly assigned to groups of three or four. In the lecture groups, participants were instructed to listen to a video lecture while looking at the learning materials for 18 minutes. In the self-study group, participants were instructed to study written materials by themselves for 18 minutes. Subsequently, participants from each of the two categories were divided into four different groups of three or four participants prior to discussions. Worksheets were provided for some groups during discussion, and not provided for others. The discussion entailed participants talking freely with peers about what they had studied for 20 minutes.
Two of the groups received worksheets from professors to discuss with their peers. Finally, students were given 20 minutes to complete a nal test.
To sum up, depending on our experimental design, students were randomly assigned to each of the four groups: two study conditions (lecture versus self-study) and two intervention conditions (discussion with worksheet versus free discussions). Speci cally, there were four groups: (1) the lecture and discussion (LD) group, (2) the lecture and discussion with worksheet (LW) group, (3) the self-study and discussion (SD) group, and (4) the self-study and discussion with worksheet (SW) group.

Data Analyses
Data analysis was carried out using SPSS version 23 (SPSS Inc., Chicago, USA). The data was analyzed using descriptive statistics (means, standard deviations (SDs), minimums, maximums). One-way or twoway analysis of variance (ANOVA) was conducted to compare each of four groups. P value < 0.05 was considered to indicate a statistically signi cant difference.

Learning Outcomes of Active Learning Online
Regardless of which group students were in, all students completed the given prior knowledge survey. The learning materials used in this experiment were law-related and deal with accusation, charge, and recognition of criminal procedure code. First, we analyzed survey items by topic: criminal procedure code Sexual Violence (M = 2.64, SD = 1.47). Among these items, scores of topic-relevant items (criminal procedure code and accusation and charge) were signi cantly lower than that of the genome project, which was the highest, t(164) = 7.29, P < 0.001, t(164) = 5.34, P < 0.001, respectively. There was also no difference between the four groups in their background knowledge of the learning material.
The mean values and standard deviations of the scores for total, verbatim type, and transfer type items are presented in Table 1. In order to rule out the subjectivity of scoring as much as possible, 15 nal test papers were randomly selected and were marked by two other raters (20% of the total). For all types of question items, the agreement measured by Intraclass correlation was 0.89 for the analysis {ICC (3.k)}.
Accordingly, the agreement measured by ICC was good, so the remaining nal test paper was marked by the rst author.

Comparison of Active Learning O ine and Online
We intended to replicate the results of the past experiment [10] and at the same time determine whether the same extent of active learning is possible through online discussion. Thus, we investigated whether the online discussion improved the student's learning outcomes, especially the greater the transfer.
The mean values and standard deviations of all groups are provided in Fig. 2. We compared o ine study [10] with online study, separating groups according to lecture or self-study, which were the learning activities prior to the discussion. The four groups were lecture and discussion o ine (LD O ine, n = 50), lecture and discussion online (LD Online, n = 40), self-study and discussion o ine (SD O ine, n = 58), and self-study and discussion online (SD Online, n = 42). The four groups had signi cant differences in total scores, verbatim types items and transfer type items, F ( . Among these items, scores of topic-relevant items (criminal procedure code and accusation and charge) were signi cantly lower than that of the genome project, which was the highest, t(164) = 7.29, P < 0.001, t(164) = 5.34, P < 0.001, respectively. There was also no difference between the four groups in their background knowledge of the learning material.
The mean values and standard deviations of the scores for total, verbatim type, and transfer type items are presented in Table 1. In order to rule out the subjectivity of scoring as much as possible, 15 nal test papers were randomly selected and were marked by two other raters (20% of the total). For all types of question items, the agreement measured by Intraclass correlation was 0.89 for the analysis {ICC (3.k)}. Accordingly, the agreement measured by ICC was good, so the remaining nal test paper was marked by the rst author.
For verbatim type items, the SD group had signi cantly higher scores than those of the SW group, the LD group, and the LW group, t(78) = 2.37, P = 0.020, d = 0.995; t(78) = 2.19, P = 0.032, d = 0.699; t(78) = 3.90, P < 0.001, d = 1.298, respectively. However, there were no signi cant differences between the LD group and the LW group, between the LW group and the SW group, and between the LD group and the SW group For transfer type items, the SD group was signi cantly higher than the SW group, the LD group, and the LW group, t(78) = 3.26, P = 0.002, d = 1.088; t(78) = 5.69, P < 0.001, d = 1.828; t(78) = 4.04, P < 0.001, d = 1.338, respectively. The SW group had signi cantly higher scores than those of the LD group, t(78) = 2.33, P = 0.022, d = 0.770, while there were no signi cant differences between the LD and the LW groups, and between the LW and the SW groups, t(78) = 1.02, P = 0.312, d = 0.300; t(78) = 1.27, P = 0.208, d = 0.386.

Comparison of Active Learning O ine and Online
We intended to replicate the results of the past experiment [10] and at the same time determine whether the same extent of active learning is possible through online discussion. Thus, we investigated whether the online discussion improved the student's learning outcomes, especially the greater the transfer.
The mean values and standard deviations of all groups are provided in Fig. 2. We compared o ine study [10] with online study, separating groups according to lecture or self-study, which were the learning activities prior to the discussion. The four groups were lecture and discussion o ine (LD O ine, n = 50), lecture and discussion online (LD Online, n = 40), self-study and discussion o ine (SD O ine, n = 58), and self-study and discussion online (SD Online, n = 42). The four groups had signi cant differences in total scores, verbatim types items and transfer type items, F (

Discussion
Although there has been a focused attention on the need to care for patients and communities, there has not been su cient focus to address the effect of the advent of COVID-19 on medical education. COVID-19 has effected educational institutions worldwide; in the country alone, thousands of schools and universities faced closure in given the limited time span to enforce social distancing [7]. While in the midst of the pandemic crisis, it is crucial to see that this situation offers the opportunity to rethink modern educational content, purposes, and systems. The profound effects of this disease call for changes in modern education.
Amidst the pandemic, as an alternative to existing teaching methods many countries around the world have adopted online learning such as web-based learning and e-learning. Although online learning has existed before, it has become more sophisticated and diverse through the development of information technology and the needs of the times. Accordingly, health-related education including medical education began to change existing learning methods to include online formats. In addition, educators tried to minimize or eliminate one-way lectures and improve student-centered learning methods [9,10]. As a good alternative to these approaches, Lim and colleagues proposed an active learning method for health professionals [10]. They con rmed the effectiveness of discussions in learning, while exploring which preliminary activities before the discussions were most effective. The experiment showed that the more active the students were in preliminary activities before discussions, the greater the effect of the discussion, which was shown to be the students' high learning performance. Thus, the present study was conducted to add additional evidence on the effectiveness of discussions with dentistry students by replicating the previous study and to investigate whether active learning could be applied online.
The results of this study obtained similar results as those of the past [10]. The main effect of the study condition (lecture versus self-study) was signi cant while the main effect of the discussion condition had no signi cance (discussions with worksheet versus free discussions). However, the interaction effect of these two factors was signi cant. In detail, the SD group had the highest performance when comparing total scores, for both verbatim and transfer type items of the nal test. The other three groups had no signi cant differences, but the LW group showed the lowest score. For scores on verbatim type items, the SD group had the highest scores among the four groups. It was not signi cant, but after the SD group the scores were highest in order of the LD group, the SW group, and the LW group. For transfer type items, the scores were highest in the order of the SD group, the SW group, and the two lecture groups, with no signi cant differences between the two lecture groups. These difference in scores and patterns between the groups were similar for all types of items. Thus, students who studied by themselves before the discussions had better outcomes compared to those who listened to a lecture. This suggests that student participation in more active learning activities rather than simply listening to lectures enhances the effect of discussions, which leads to higher learning outcomes.
In addition, structured discussions, such as those including the instructors' intervention (e.g., a worksheet), did not work well. The LW group that received the worksheet had a lower score than the LD group that did not, and the SW group that received the worksheet had a lower score than the SD group who did not. Interestingly, in the two groups who listened to the lecture, the instructors' intervention had some effect on improving the students' scores, but in the two groups who conducted the self-study, the instructors' intervention actually lowered the students' scores. These results suggest that free discussions, such as solving each other's questions or discussing topics in more depth, are effective discussions for self-study groups. On the other hand, the lecture groups in which students participated in more passive learning seemed to allow for deeper learning through the intervention of instructors. These are results that support past studies that show that one-way lectures cannot induce students to learn as deeply as other methods do [24,25,26].
Next, we compared the o ine experiment [10] with the current online experiment. As a result of the analysis, the SD O ine group was the highest of the four groups in total scores, followed by the SD O ine group. The LD O ine group had no signi cant difference from the LD Online group, and the LD Online group had the lowest score. For verbatim type items, the SD O ine group had the highest score followed by the SD Online group, and there was no signi cant difference between the LD O ine group and the LD Online group. Lastly, the SD O ine group had the highest score for transfer type items, and the LD Online group had the lowest score. In summary, regardless of whether learning was o ine or online, the group that had discussions and that was preceded by self-study scored higher. The important point is that in these groups, students participate in the learning process more actively and try to understand the learning materials themselves. However, online groups generally scored lower than o ine groups. This is because students were not only awkward with the online platform, but were also unfamiliar with the progress of active learning online. In the future, if students experience online learning, results are expected to be similar to those of o ine learning. Nevertheless, it is important that patterns in the results of o ine learning experiments and online learning experiments were similar.
The main results of our experiment support the ICAP framework [19,20]. In parallel with the framework that the engagement of students promotes greater learning outcomes, the self-study groups (e.g., the SD, SW, SD O ine, and SD Online groups) scored much higher than the lecture groups (e.g., the LD, LW, LD O ine, and LD Online groups). That is, the results are consistent with Chi and colleagues' ICAP framework showing the learning bene ts of active learning. Moreover, based on the results of this study, active learning online, including discussions, is expected to be applied to students with healthrelated majors, including medical and dentistry majors (e.g., Team-based learning online, Project-based learning online). However, future studies are needed to expand and generalize our ndings. One limitation of the present study is that the experimental group was limited to dentistry students. Thus, future studies need to recruit students from other health professions (e.g., medical, veterinary etc.) to conduct comparative studies. The second limitation is that past o ine experiments and current experiments did not entirely follow the same process. O ine experiments [10] used a question-generation condition, but we added a new variable called instructors' intervention (worksheets) during the discussion. Since the question-generation variable did not have a signi cant main effect in past experiments, the new variable of instructors' intervention during discussion is meaningful in that we explored new factors in active learning. Moreover, careful caution is needed since the intervention in a discussion may be good or bad depending on the situation.
This study explores effective learning methods for health professionals. Students of health professions have thus far pursued large amounts of knowledge, but they require effective learning methods to better remember, apply, or utilize this knowledge in new situations [27,28,29]. In this regard, this study presents a learning method that can improve their learning, which is a discussion after selfstudy. More importantly, in accordance with the demands of the times of COVID-19, this study explored a new form of active learning online and assessed whether the active learning method of discussion can be applied online. Possible follow-up studies centered on some of the above discussions, as well as active learning studies online, will create new changes in future education.

Conclusion
The pandemic changes the practical and logistical schemes of educations for health professions. In response to COVID-19, online or non-face-to-face learning methods are required. Thus, our experiment is timely and can be appropriate to cope with these times in the context of health education. Our ndings support the ICAP framework and provide practical implications for education for health professionals. It is better for students to study by themselves before discussions than to listen to lectures o ine or online. In particular, when students participate actively in learning activities before discussions, they increase their transfer of learning. Thus, through new approaches such as developing and applying existing methods online, we look forward to creating more effective and diverse educational methods for health professionals.
Abbreviations lecture and discussion (LD) group; lecture and discussions with worksheet (LW) group; self-study and discussion (SD) group; self-study and discussions with worksheet (SW) group; lecture and discussion o ine (LD O ine) group, lecture and discussion online (LD Online) group, self-study and discussion o ine (SD O ine) group, and self-study and discussion online (SD Online) group.

Declarations
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate
The study was approved by the Institutional Review Board (IRB) of Seoul National University School of Dentistry (IRB No. S-D20200053). All the experiment protocol for involving human participants was in accordance to guidelines of national/international/institutional ethical standards or Declaration of Helsinki. All participants were aware that they were taking part in this research and gave informed consent in addition to con rming that they would allow us to use their collected data anonymously for publication. All the data were anonymously collected and analyzed.

Consent for publication
Not Applicable. Note. Data are shown as mean values (standard deviations). LD = lecture and discussion, LW = lecture and discussion with worksheet, SD = self-study and discussion, and SW = self-study and discussion with worksheet. For each LD, LW, SD, and SW group, total, verbatim type, and transfer type items scores are given. Note. Data are shown as mean values (standard deviations). LD = lecture and discussion, SD = self-study and discussion. For each LD and SD group, total, verbatim type, and transfer type items scores are given. Figure 1