A meta-analysis of Twitter assisted learning outcomes during the COVID-19 pandemic

DOI: https://doi.org/10.21203/rs.3.rs-502081/v1

Abstract

The use of social media such as Twitter has been widely used in education during the COVID-19 pandemic due to their positive learning outcomes. To summarize the effect of Twitter use on educational outcomes, this study included 22 high-quality peer-reviewed journal articles for meta-analysis. It is concluded that the Twitter assisted learning approach can lead to significantly higher learning outcomes than the non-Twitter assisted but the former cannot lead to significantly higher teaching effectiveness than the latter. The use of Twitter can improve learning outcomes in the USA and Sweden rather than Spain. Twitter can significantly improve learning engagement in the USA but not in Greece. Males tend to show significantly higher learning outcomes than females in the Twitter assisted learning context. Whether Twitter is used as a supplementary or an integrated tool can significantly improve learning outcomes. Future research may adopt more interdisciplinary methods and include more literature to summarize the effect of Twitter use on educational outcomes especially during the COVID-19 pandemic.

Introduction

Positive findings

The COVID-19 pandemic has forced people to receive education through social media such as Twitter and Facebook (Yu, 2021). Twitter assisted learning could improve learners’ engagement in learning activities and communities. Active engagement in learning could improve learning outcomes although the link between them proved weak. Tips through Twitter were useful to learners in Spain because they deem Twitter a tool for educational purposes (Fouz-González, 2017). With Twitter, Spanish students could actively participate in learning activities although they could hardly integrate the use of Twitter into learning interactions. Guidance should, therefore, be provided for students to improve the interaction. The restriction on the length of a message could encourage users to think and decide what to type next. The training schedule could focus on the design of learning activities (Feliz, Ricoy, & Feliz, 2013). With social media such as Twitter, American students, who consider Twitter a useful tool, could actively join large learning communities and enhance their learning interest (Hitchcock & Young, 2016). Greek Students’ learning attitudes towards Twitter were positive, leading to their active engagement in learning activities (Katrimpouza, Tselios, & Kasimati, 2019). 

The use of Twitter could improve users’ academic achievements due to various factors. Integrating Twitter into a course coupled with teacher participation could greatly improve students’ academic achievements in the USA (Junco, Michael Elavsky, & Heiberger, 2013). Twitter assisted learning approaches could greatly improve students’ academic achievements by enhancing their engagement (Junco, Heiberger, & Loken, 2010). Twitter use could greatly improve Greek students’ laboratory performance although it is not found significantly correlated with their personality traits. The use of Twitter could improve Greek students’ academic achievements and social presence and enhance their self-efficacy (Loutou, Tselios, & Altanopoulou, 2018). In a classroom, American students who used Twitter frequently obtained significantly higher academic achievements than those who seldom used Twitter (Webb, Dugan, & Burchett et al., 2015). Broadcast journalism majors could perform excellently assisted with Twitter in terms of the degree of learning activity engagement and interactivity in the USA (Cozma & Hallaq, 2019). Other smart learning tools such as interactive webinars and podcasts may also improve academic achievements (Campi, Amparore, Checcucci, Claps, & Zhuang, 2021).

The use of Twitter could positively improve users’ psychological status such as engagement and memory. Engagement Twitter Support could improve American breast cancer patients’ gain of knowledge by reducing their anxiety (Attai, Cowher, & Al-Hamadani, Schoger, Staley, & Landercasper, 2015). Students were in general satisfied with the mobile learning system assisted with Twitter since it could improve knowledge gain and provide cooperative opportunities in Sri Lanka (Dissanayeke, Hewagamage, Ramberg, & Wikramanayake, 2016). Caring through Twitter dialogues rather than one-way communication could improve Swedish Twitter users’ learning attitudes and intentions (Colliander, Dahlén, & Modig, 2015). American students held positive learning attitudes toward the Twitter integrated classroom and actively participated in the learning activities (Luo & Xie, 2018). American students positively evaluated Twitter use in a classroom and reported increased social presence with peers although Twitter use might not improve their memory of academic contents (Smith & Tirumala, 2012). Social media tools such as Twitter could improve information retention through their visual input function, especially in American post-secondary education (Arceneaux & Dinu, 2018). The use of Twitter could improve American students’ memory and enables them to keep more important concepts in their mind (Blessing, Blessing, & Fleck, 2012). 

Some studies reported gender differences in the use of Twitter. For example, American men were more likely to examine tweets to share resources and criticize other tweets while women tended to write tweets and positively evaluated other tweets. This indicates that the teacher could use different teaching strategies toward different genders in the USA (Kerr & Schmeichel, 2018). Females’ contributions to Twitter seemed significantly larger than males (Davidson-Shivers, Muilenburg, & Tanner, 2001). Males could produce significantly more voice messages than females (McConnell, 1997).

Negative findings

Despite many studies revealed positive findings regarding the educational effect of Twitter use, there are also many studies reporting negative results. For example, it was reported that the use of Twitter could not improve learning outcomes in the USA (Al-Bahrani, Patel, & Sheridan, 2017). Even Twitter messages were related to lecture notes, they failed to improve American users’ academic achievements in terms of multiple-choice grades and free-recall performance. Worse, when learners create and send tweets frequently, the quality of their lecture notes will be reduced (Kuznekoff, Munz, & Titsworth, 2015). The use of Twitter in class could not significantly influence American users’ interest in politics and news reading although it might positively influence their learning outcomes (Feezell, 2019).

We should try to see both sides of a coin when exploring the use of Twitter in education. Professional contents on the Twitter platform, as well as American students’ learning attitudes, could greatly influence the teachers’ credibility. Twitter could be both an advantage of and an obstacle to learning and teaching (DeGroot, Young & VanSlette, 2015). There were several drawbacks, e.g technical issues, limitation to higher-order skill acquisition, and limited learning opportunities, in the Twitter integrated mobile learning system in Sri Lanka (Dissanayeke, Hewagamage, Ramberg, & Wikramanayake, 2016).

It was also found that Twitter may not be suitable for educational use. American Twitter users are younger and wealthier than those in other regions, leading to under-representativeness of Twitter population. This causes the claim that Twitter may be more properly used by corporations than by social science researchers in the USA (Blank, 2016). This argument may dampen the enthusiasm of Twitter use for educational purposes. Twitter may be not appropriate for educational purposes since tweets have numerous problems, e.g. lack of relational cues in tweets, limitation to contents, no hint of communicative behaviour, and insufficient timeline (Yoshida, 2021).

A research gap

Little is known about the effect of different Twitter usages on learning outcomes. Some studies adopted Twitter as an integrated learning tool (e.g. Feezell, 2019; Loutou et al., 2018; Feliz et al., 2013; Dissanayeke et al., 2016), while others as a supplemental learning tool (e.g. Al-Bahrani et al., 2017; Arceneaux & Dinu, 2018; Blank, 2016; Blessing et al., 2012). However, very few studies have been committed to the differences in the effect of both usages. It is thus meaningful to examine the effect of Twitter usages on learning outcomes.

Research questions

Considering the research gap and inconsistent findings regarding the use of Twitter for educational purposes, we proposed several research questions as follows.

RQ1: Can Twitter assisted approaches improve learning outcomes such as learning attitudes, academic achievements, learning engagement, and teaching effectiveness?

RQ2: Can Twitter assisted approaches improve learning outcomes in different countries?

RQ3: Are there any gender differences in Twitter assisted learning outcomes in different countries?

RQ4: Can different usages of Twitter lead to different learning outcomes?

Methods

Protocol and registration

This study focuses on Twitter assisted learning outcomes in terms of country, gender, and usage, which is not required to be registered. It is conducted based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (Moher, Liberati, Tetzlaff, Altman, & Group, 2009) and approved by the institutional review board which waives a review protocol.

Eligibility criteria

We selected and excluded studies based on inclusion and exclusion criteria. The studies will be included if they (1) focus on Twitter assisted learning outcomes in terms of country, gender, and Twitter usage; (2) are of higher quality; (3) divide participants into both control and treatment groups for a comparative analysis; (4) adopt a randomized controlled design through a comparative analysis between both control and treatment groups; and (5) are peer-reviewed academic works.

The studies will be excluded if they (1) focus on Twitter technology itself rather than Twitter use in education; (2) belong to reviews rather than empirical studies; (3) cannot provide enough information for meta-analysis even after contacting authors; or (4) are written in a language other than English.

Information sources

The information sources from multiple databases, e.g. EBSCOhost, Taylor & Francis Online, Wiley Online Library, and Sage. We also obtained information through corresponding with the authors in case the full-text does not provide enough data for meta-analysis. We searched the above databases on January 27, 2021. The coverage of dates ranges from the commencement year until January 2021.

Literature search and selection

The literature search and selection involve four steps based on the PRISMA flowchart (Yang, Yen, McGowan, Chen, Chiang, & Mancini, et al., 2012). All the literature was searched from online databases, i.e. EBSCOhost, Taylor & Francis Online, Wiley Online Library, and Sage. In the first step, we searched the online databases using corresponding terms to obtain literature before we removed the duplicated results. In the second step, we invited two independent reviewers to screen the irrelevant results by perusing abstracts and titles. In the third step, two independent reviewers evaluated the eligibility of the full texts. In the fourth step, both reviewers presented the results of evaluation and negotiated on any disagreements. A third experienced reviewer would join and determine the selection if both reviewers could not persuade each other on any specific literature selection. After the four-step selection process, we finally determined 22 peer-reviewed journal articles for the meta-analysis (Figure 1).

Quality assessment

We evaluated the quality of the selected studies based on the University of West England Framework for Critically Appraising Research Articles (Moule et al., 2003) (Appendix A). The selected articles are evaluated in terms of each section, i.e. The Introduction, The Methods Section, Data Collection and Analysis, The Results/Findings, and The Conclusions. Each section is assessed based on a certain criterion. For example, the introduction section should include a clear statement about the topic being investigated and a clear rationale for the research. The research design should be clearly described and the research methods should be appropriate for the topic being investigated. Two reviewers scored each selected article. The average score will be considered. We selected the top-scored 22 articles. The Inter-rater Cohen's kappa coefficient is 0.82.

Data extraction

The data were extracted by both reviewers from the eligible studies. To fulfill the meta-analysis, they extracted data such as authors, publication years, the total number of participants, means, and standard deviations of both groups.

They also extracted data regarding country, e.g. USA, UK, Sri Lanka, Sweden, Spain, and Greece, learning outcomes, and Twitter usage. Learning outcomes are classified into academic achievements, attitude, gender differences in learning outcomes, engagement, and teaching effectiveness. Twitter usage is classified into Twitter as a supplemental learning tool and Twitter as an integrated learning tool.

Similarly, both reviewers met together to decide on the final extracted data after they finished data extraction. In case they could not reach an agreement on any data, a third reviewer would be invited to make a decision. 

Statistical analysis

Stata MP/14.0 was used to analyze the extracted data and fulfill the meta-analysis. Z-statistics were adopted to analyze the effect of meta-analytical outcomes. Since the data type is continuous, we entered the numbers of participants, means, and standard deviations of both groups into Stata MP/14.0, labeled the data by authors and publication years, and selected country, learning outcome, and Twitter usage as a variable. The pooling model was either random (I-V heterogeneity) or fixed (Inverse Variance). The effect size was expressed as Cohen d or standardized mean differences (SMD). d is considered very small if the value is 0.1, small if 0.2, medium if 0.5, large if 0.8, very large if 1.2, and huge if the value is 2.0. (Sawilowsky, 2009). SMD was calculated through the formula: SMD = difference in mean outcome between groups/standard deviation of outcome among participants.

Due to various situations, different characteristics of participants, and different interventions, different studies tend to cause different effect sizes. Statistical heterogeneity occurs immensely and is unavoidable (Higgins & Green, 2011). We, therefore, quantify heterogeneity using I2 through the formula below:

where Q indicates the chi-squared result and df means the degree of freedom (Higgins & Green, 2011).  The heterogeneity will be considered not important if I2 = 0% - 40%, moderate if 30% - 60%, substantial if 50% - 90%, considerable if 75% to 100%. Roughly, we will use a random-effect model to conduct the meta-analysis if I2 is larger than 50%, and a fixed-effect model if it is smaller than 50%. If the value of I2 is large, we will test the sensitivity using the influence analysis in Stata MP/14.0. We will also test the publication bias using Begg’s (Begg & Mazumdar, 1994) and Egger’s (Egger, Smith, Schneider, & Minder, 1997) tests.

Results

Characteristics of included studies

Table 1 summarizes the characteristics of included studies, involving author, publication year, sample size, country, learning outcome, and Twitter usage. The specific data such as means and standard deviations of both control and treatment groups are provided in the data file.

Risk of bias within studies

Both Egger’s and Begg’s tests were used to detect the publication bias. The data input format theta se_theta was assumed for both tests. Figure 2 presents an Egger’s plot of publication bias. A dot in Figure 2 indicates an individual study. The uneven distribution of the dots along either side of the middle no-effect line indicates the presence of publication bias (t = 4.70, p < .01). Begg's test examines the publication bias through rank correlation between standardized intervention effect and its standard error, which also indicate the presence of publication bias (z = 3.15, p = 0.002).

Results for individual and synthetic analysis

This result section presents the findings regarding pooled learning outcomes, their differences in different countries and, different usages of Twitter.

Results for learning outcomes

To determine whether a random-effect or fixed-effect model we should adopt, we tested the heterogeneity of the effect sizes for academic achievements. The result shows that the effect sizes are heterogeneous (Q = 2129.93, df = 73, p <.01, I2 = 96.6) (Table 2). We, therefore, adopted a random-effect model to conduct the meta-analysis of academic achievements.

Through Stata MP/14.0, we obtained 74 effect sizes (SMD) in terms of academic achievements, so we analyze academic achievements independently to show them clearer in a forest plot (Figure 3). As shown in Figure 3, the horizontal line indicates the 95% confidence interval. The middle line is referred to as a no-effect line because if the horizontal line crosses it, the effect will be considered not significant. The diamond indicates the pooled result of effect sizes. If the diamond crosses the no-effect line, the result will be considered not significant. Since the diamond does not cross the no-effect line and is located to the right of the no-effect line, we conclude that Twitter assisted academic achievements are significantly higher than the non-Twitter assisted (d = 0.380,         95% CI: 0.16-0.60, z = 3.31, p = 0.001) (Table 2).

The effect sizes are also heterogeneous for other learning outcomes such as attitude (Q = 518.43, df = 17, p <.01, I2 = 96.7), gender difference (Q = 5965.96, df = 7, p <.01, I2 = 99.9), teaching effectiveness (Q = 15.13, df = 5, p = .01, I2 = 67), and overall learning outcomes (Q = 14131.53, df = 111, p <.01, I2 = 99.2), but not for engagement (Q = 1.03, df = 5, p = 0.96, I2 = 0) (Table 2). To keep the analysis method consistent, we adopted a random-effect model to conduct the meta-analysis regarding other learning outcomes.

As shown in Figure 4, the diamonds for attitude, engagement, gender differences, and overall learning outcomes do not cross the no-effect line and they are all located to the right of the no-effect line. We, therefore, report that Twitter assisted learning attitude (d = 0.302, 95% CI: 0.11-0.50, z = 3.08, p = 0.002), engagement (d = 0.408, 95% CI: 0.18-0.64, z = 3.52, p < .01), and overall learning outcomes (d = 0.48, 95% CI: 0.27 - 0.70, z = 4.43, p < .01) are significantly larger than the non-Twitter assisted (Table 2). Males obtain significantly better learning outcomes than females in the context of Twitter assisted learning (d = 2.17, 95% CI: 0.84 - 3.51, z = 3.20, p = .001) (Table 2). However, the diamond for teaching effectiveness (Figure 5) crosses the no-effect line (d = -0.14, 95% CI: -0.52 - 0.24, z = 0.72, p = 0.47) (Table 2). Thus, we report that there are no significant differences in teaching effectiveness between the Twitter and non-Twitter assisted learning approaches. 

Results for learning outcomes in different countries

To determine the type of analysis model, we measure the heterogeneity of effect sizes. The values of heterogeneity are all significant in the USA (Q = 12740.86, df =            83, p < .01, I2 = 99.3), Greece (Q = 41.54, df =      8, p < .01, I2 = 80.7), Spain (Q = 74.17, df = 7, p < .01, I2 = 90.6), and Sweden (Q = 12.48, df = 5, p = .029, I2 = 59.9) (Table 3). We, therefore, adopted a random-effect model to conduct the meta-analysis regarding learning outcomes in different countries.     

As shown in Table 3, Twitter assisted learning outcomes in the USA (d = 0.488, 95% CI: 0.218 - 0.758, z = 3.54, p < .01), Greece (d = 0.716, 95% CI: 0.361 - 1.071, z = 3.95, p < .01), Sweden (d = 0.404, 95% CI: 0.194 - 0.614, z = 3.77, p < .01), and overall regions (d = 0.489, 95% CI: 0.269 - 0.697, z = 4.43, p < .01) are significantly higher than the non-Twitter assisted. However, in Spain, there are no significant differences between Twitter and non-Twitter assisted learning outcomes (d = -0.16, 95% CI: -0.589 - 0.269, z = 0.73, p = 0.465). We, therefore, conclude that Twitter assisted learning can lead to significantly higher learning outcomes than non-Twitter assisted learning in the USA, Greece, and Sweden but no significant difference is revealed in Spain.

Results for academic achievements in different countries

To meta-analytically examine academic achievements in different countries, we obtained a total of 73 effect sizes. We finally selected 69 effect sizes after removing 4 results due to the confusing research venue (USA and UK mixed). 

As shown in Table 4, the effect sizes in the USA (Q = 1976.39, p < .01, I2 = 96.9), Greece (Q = 41.38, p < .01, I2 = 83.1), and overall regions (Q = 2129.93, p < .01, I2 = 96.6) are significantly heterogeneous although the effect size in Spain is not significantly heterogeneous (Q = 1.73, p = 0.189, I2 = 42.1). To keep analytical methods consistent, we adopted random-effect models to analyze different academic achievement in different countries.

As for academic achievements, Twitter assisted learning results are significantly higher than the non-Twitter assisted in the USA (d = 0.305, 95%CI = 0.052 - 0.558, z= 2.36, p = 0.018), Greece (d = 0.739, 95%CI = 0.353 - 1.126, z = 3.75, p < .01), and overall regions (d = 0.380, 95%CI = 0.155 - 0.604, z = 3.31, p = 0.001). However, it is not significantly different in Spain (d = -0.236, 95%CI = -0.701 - 0.230, z = 0.99, p = 0.321). Thus, we conclude that Twitter assisted learning could lead to significantly higher academic achievements in the USA and Greece than the non-Twitter assisted but no significant difference is found in Spain.

Results for learning attitudes in different countries

We obtained 6 effect sizes for learning attitudes in Sweden and 10 in the USA. A total of 16 effect sizes are summarized for the analysis of learning attitudes in different countries. 

As shown in Figure 6, the effect sizes in Sweden (I2 = 59.9, p = .029) and the USA (I2 = 95.8, p < .01) are all significantly heterogeneous. We, therefore, adopted a random-effect model to conduct the meta-analysis. In Sweden, students hold significantly more positive learning attitudes toward Twitter assisted learning than the non-Twitter assisted (d = 0.40, 95%CI = 0.19 - 0.61) (Table 3) since the diamond is located to the right of the no-effect line. In the USA, there are no significant differences in learning attitudes towards both Twitter and non-Twitter assisted approaches (d = 0.13, 95%CI = -0.07 - 0.34) (Table 3) since the diamond crosses the no-effect line. The overall results indicate that students hold significantly more positive learning attitudes towards Twitter assisted learning than the non-Twitter assisted (d = 0.29, 95%CI = 0.10 - 0.49) (Table 3) since the diamond is located on the right side of the no-effect line. Therefore, we conclude that in Sweden users hold significantly positive attitudes towards the use of Twitter in education but no significant difference is found in the USA.

Results for learning engagement in different countries

We retrieved 1 effect size in Greece and 5 effect sizes in the USA and adopted a fixed-effect model to conduct the meta-analysis since the results are not significantly heterogeneous (I2 = 0, p = 0.942 for USA and no result for Greece). One effect size in Greece and 5 effect sizes in the USA were obtained for the meta-analysis of learning engagement in different countries (Figure 7). As shown in Figure 7, the diamond regarding the effect size in Greece crosses the no-effect line. We, therefore, conclude that there is no significant difference in learning engagement between Twitter and non-Twitter assisted learning (d = 0.58, 95%CI = -0.11 - 1.26). On the contrary, in the USA, the pooled diamond does not cross the no-effect line and is located on the right side of the no-effect line, so we report that Twitter assisted learning can cause significantly more engagement than non-Twitter assisted in the USA (d = 0.39, 95%CI = 0.15 - 0.63). 

Results for teaching effectiveness in different countries

We retrieved 5 effect sizes for meta-analysis of teaching effectiveness in Spain and 1 effect size in Sri Lanka. Considering the significantly heterogeneous result in Sri Lanka (I2 = 57.0%, p = 0.01) and the insignificantly heterogeneous results in Spain (I2 = 29.5%, p = 0.225), we adopt the same random-effect model to conduct the meta-analysis. We report that teaching effectiveness of the Twitter assisted approach is significantly (d = -0.30, 95%CI = -0.56 - -0.03) (Figure 5) lower than the non-Twitter assisted since the pooled diamond does not cross the no-effect line and located to the left of it in Spain. In Sri Lanka, Twitter assisted teaching effectiveness is significantly higher than the non-Twitter assisted (d = 1.05, 95%CI = 0.22 - 1.87) (Figure 5). The overall effect reports that there is no significant difference in teaching effectiveness between the Twitter and non-Twitter assisted approaches (d = -0.14, 95%CI = -0.52 - 0.24) (Figure 5) because the pooled diamond crosses the no-effect line.

Results for gender differences in different countries

To determine gender differences in learning outcomes in different countries, we retrieved 7 effect sizes in the USA and 1 effect size in Spain. The results in the USA are significantly heterogeneous (I2 = 99.9%, p < .01, no result for Spain). We, therefore, adopted a random-effect model to conduct the meta-analysis. Male learners have significantly higher learning outcomes than females in both the USA (d = 2.41, 95%CI = 0.53 - 4.25) and Spain (d = 0.54, 95%CI = 0.49 - 0.59) because neither of the pooled diamonds in both countries crosses the no-effect line and they both are located to the right of the no-effect line (Figure 8). 

Results for learning outcomes of different Twitter usages

We firstly measured the heterogeneity of effect sizes of different Twitter usages. Effect sizes of learning outcomes via different Twitter are also highly heterogeneous in both Twitter as a supplementary tool (Q = 6995.91, df = 66, p < .01, I2 = 99.1) and Twitter as an integrated tool (Q = 14775.03, df = 44, p < .01, I2 = 99.1). We, therefore, adopted a random-effect model to conduct the meta-analysis. 

As shown in Table 5, either when Twitter is used as a supplementary tool (d = 0.501, 95% CI: 0.095 - 0.907, z = 2.42, p = 0.015) or as an integrated tool (d = 0.459, 95% CI: 0.216 - 0.702, z = 3.70, p < .01), the learning outcomes are significantly improved than the non-Twitter assisted learning approach. Thus, we concluded that Twitter assisted learning could lead to significantly higher learning outcomes than the non-Twitter assisted whether it is used as a supplementary or an integrated tool. 

A sensitivity analysis

To determine whether or not the results were stable, we conducted a sensitivity analysis through Stata MP/14.0. Extreme estimates of effect sizes tend to skew the average estimated values and remain out of the confidence interval (Borenstein, Hedges, Higgins & Rothstein, 2009). Consequently, it is necessary to determine whether any estimate of an individual study can greatly influence the pooled result (Borenstein, Hedges, Higgins & Rothstein, 2009). As shown in Figure 9, a dot indicates an individual study. The lower confidence interval limit is 0.27, and the upper confidence interval limit is 0.70. All the meta-analysis estimates remain between the upper and lower confidence interval given a named study is omitted. We, therefore, conclude that the meta-analytical results are stable.

Discussion

Summary of evidence

Generally, positive evidence has been revealed regarding the use of Twitter in education. This meta-analysis reports that the Twitter assisted learning approach can lead to significantly higher learning outcomes than the non-Twitter assisted in terms of academic achievements, learning attitudes, and learning engagement. But the former cannot lead to significantly higher teaching effectiveness than the latter. 

The use of Twitter can improve learning outcomes in the USA and Sweden rather than Spain. In Sweden, Twitter assisted learning is more positively evaluated than the non-Twitter assisted while no difference was found in the USA. Twitter can significantly improve learning engagement in the USA but not in Greece. Surprisingly, Twitter can decrease the teaching effectiveness in the USA but significantly improve it in Sri Lanka. Males tend to show significantly higher learning outcomes than females in the Twitter assisted learning context. Males assisted with Twitter perform significantly better than females in both the USA and Spain. Whether Twitter is used as a supplementary or as an integrated tool can significantly improve learning outcomes including academic achievements, learning attitudes, learning engagement, and teaching effectiveness. 

The findings are consistent with previous studies. For example, a Twitter-based mobile approach was used in an Agriculture knowledge pedagogy, where students generally obtained satisfactory learning outcomes. However, the teaching effectiveness, as well as the higher-order skills, was not highly evaluated due to limitations of Tweet language on the Twitter platform such as the word number limit and unfamiliar English technical terms for Sri Lankan students (Dissanayeke, Hewagamage, Ramberg, & Wikramanayake, 2016). 

Another support regarding the lower teaching effectiveness sources from Spain. In the teaching period, Tweets’ remaining tractable for students to review may cause the difficulty of checking whether students finish the assignment in time or not. It also becomes difficult to obtain the information regarding the time and its span students have learned based on the tweets. The limitation to training stimuli, merely 22 tweets, may have exacerbated Twitter assisted teaching effectiveness. Rewarding the participants may have also drawn some participants who are interested in rewards rather than learning, leading to poor teaching effectiveness (Fouz-González, 2017). Consequently, Twitter assisted teaching effectiveness may have been weakened, which requires a rigid teaching design based on Twitter. Due to confusion between Spanish and English, Spanish learners of English tend to insert epenthetic vowels prior to consonant clusters (Fouz-González, 2017), which may have negatively influenced their learning outcomes.

Previous studies also support that men tend to achieve more success in Twitter assisted learning. For example, males tend to produce slightly more tweets than females (Feliz, Ricoy, & Feliz, 2013). Females tend to be less engaged in Twitter assisted learning compared with males, who may be more interested in technology-based learning. Consequently, females obtained significantly lower academic achievements than males in terms of the final course grade, gap-closing measure, and post- and pre-test scores (Al-Bahrani, Patel, & Sheridan, 2017). Men seem to more readily share resources with peers through tweets, where they like to show off their academic, professional, or other personal success via critical languages while women tend to affirm other tweets (Kerr & Schmeichel, 2018). The different language style requires men to invest more time and energy in tweet production, while it requires women to invest more time in message browsing. Obviously, production needs more knowledge and creativeness than browsing. Unsurprisingly, men tend to obtain more knowledge through Twitter assisted learning compared with women. 

It is also found that Twitter can greatly improve students’ learning enthusiasm, promote students’ interaction and reflection, train students’ skills of producing concise sentences, and build a real learning environment through learning feedback (Zhu, 2010). In Twitter assisted teaching, teachers can record students’ feedback, trace students’ learning progress, and obtain their learning performance. In Twitter assisted learning, students can discuss with peers to solve difficult problems, share their own opinions, and retrieve a sea of information from the platform. This will definitely improve their engagement in learning, intensify their interest, cultivate their positive attitudes toward Twitter use in learning, and finally greatly improve students’ academic achievements and enhance the teaching effectiveness. 

However, Twitter use in education may also bring about negative results. Plentiful information and advertisements on the Twitter platform may distract students and teachers. They may also be indulged in non-academic information. They may also feel it hard to concentrate on a given topic in case they are confronted with excessive information. They may pursue the effect of entertainment rather than knowledge acquisition. Pieces of knowledge carried by Twitter may not benefit the higher-order thinking skills and the organization of structured knowledge, resulting in easy attrition of acquired knowledge and disorganized knowledge stored in learners’ brain. Teachers may be reluctant to change their teaching styles and methods when they are required to teach via Twitter. They may also feel it awkward to deliver knowledge and difficult to focus on a topic if they are passively required to apply Twitter to their teaching practice. This may be an important reason for the decreased teaching effectiveness in some studies (e.g. Fouz-González, 2017). 

Whether Twitter is used as a supplemental or as an integrated learning tool, it can exert a positive influence on learning outcomes to a certain degree. Besides, teachers and students should attempt to use Twitter to (1) build up a powerful learning community, (2) facilitate communication and discussion, (3) present students’ opinions and creativeness, (4) promote collaboration between peers and teachers, and (5) arrange and organize various kinds of learning activities. As learning activity organizers, teachers may carefully design Twitter assisted learning activities to meet these goals. As technology developers, Twitter engineers may pay much attention to educational purposes when designing the functions of Twitter.

Limitations

There are several limitations to this study. Firstly, both Egger’s and Begg’s tests indicate the presence of publication bias, which may have caused result bias. Future research may include more studies of variety. Secondly, the study may not include all studies in the meta-analysis. Those written in a language other than English are excluded. Thirdly, the included studies themselves may have limitations. Fourthly, we may not include all eligible studies due to the limitation to library resources. 

Conclusions

In general, Twitter use in education has been widely accepted and produced positive learning outcomes although there are still controversies in some countries regarding some aspects. Males tend to obtain more positive learning outcomes than females. Future research may adopt more interdisciplinary methods and include more literature to summarize the effect of Twitter on educational outcomes especially during the COVID-19 pandemic.

References

Al-Bahrani, A., Patel, D., & Sheridan, B. J. (2017). Evaluating Twitter and its impact on student learning in principles of economics courses. The Journal of Economic Education, 48(4), 243-253. https://doi.org/10.1080/00220485.2017.1353934 

Arceneaux, P.C., & Dinu, L.F. (2018). The social mediated age of information: Twitter and Instagram as tools for information dissemination in higher education. New media & society, 20(11), 4155-4176. https://doi.org/10.1177/1461444818768259

Attai, D. J., Cowher, M. S., & Al-Hamadani, M., Schoger, J.M., Staley, A.C., & Landercasper, J. (2015). Twitter social media is an effective tool for breast cancer patient education and support: patient-reported outcomes by survey. Journal of Medical Internet Research, 17(7), e188. https://doi.org/10.2196/jmir.4721

Begg, C.B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.

Blessing, S. B., Blessing, J. S., & Fleck, B. K. B. (2012). Using Twitter to Reinforce Classroom Concepts. Teaching of Psychology, 39(4) 268-271. https://doi.org/10.1177/0098628312461484

Borenstein, M., Hedges, L., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta-Analysis. John Wiley and Sons, West Sussex. https://doi.org/10.1002/9780470743386

Campi, R., Amparore, D., Checcucci, E., Claps, F., & Zhuang, J. (2021). Exploring the residents’ perspective on smart learning modalities and contents for virtual urology education: lesson learned during the COVID-19 pandemic. Actas Urológicas Espaolas (English Edition), 45(1), 39-48.

Colliander, J., Dahlén, M., & Modig, E. (2015). Twitter for two: investigating the effects of dialogue with customers in social media, International Journal of Advertising, 34(2), 181-194. https://doi.org/10.1080/02650487.2014.996197 

Cozma, R., & Hallaq, T. (2019). Digital Natives as Budding Journalists: College TV Stations’ Uses of Twitter. Journalism & Mass Communication Educator, 74(3), 306-317. https://doi.org/10.1177/1077695818805899

Davidson-Shivers, G. V., Muilenburg, L., & Tanner, E. (2001). How do student participate in synchronous and asynchronous online discussions? Journal of Educational Computing Research, 25, 351-366.

Dissanayeke, U., Hewagamage, K. P., Ramberg, R., & Wikramanayake, G. (2016). Developing and testing an m-learning tool to facilitate guided-informal learning in agriculture. International Journal on Advances in ICT for Emerging Regions, 8(3), 12. https://doi.org/10.4038/icter.v8i3.7165

Egger, M., Smith, G.D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629–634.

Feezell, J. T. (2019). An Experimental Test of Using Digital Media Literacy Education and Twitter to Promote Political Interest and Learning in American Politics Courses. Journal of Political Science Education. https://doi.org/10.1080/15512169.2019.1694531

Feliz, T., Ricoy, C., & Feliz, S. (2013). Analysis of the use of Twitter as a learning strategy in master’s studies. Open Learning: The Journal of Open, Distance and e-Learning, 28(3), 201-215. https://doi.org/10.1080/02680513.2013.870029

Fouz-González, J. (2017). Pronunciation instruction through Twitter: the case of commonly mispronounced words. Computer Assisted Language Learning, 30(7), 631-663. https://doi.org/10.1080/09588221.2017.1340309 

Higgins, J.P.T., & Green, S. (2011). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. 2011 Available from www.handbook.cochrane.org.

Hitchcock, L. I., & Battista, A. (2013). Social media for professional practice: Integrating Twitter with social work pedagogy. The Journal of Baccalaureate Social Work, 18(Special issue), 33–45.

Hitchcock, L. I., & Young, J. A. (2016). Tweet, Tweet!: Using Live Twitter Chats in Social Work Education. Social Work Education, 35(4), 457-468. http://dx.doi.org/10.1080/02615479.2015.1136273.

Junco, R., Heiberger, G., & Loken, E. (2010). The effect of Twitter on college student engagement and grades. Journal of computer assisted learning, 387, 1-14. https://doi.org/10.1111/j.1365-2729.2010.00387.x

Junco, R., Michael Elavsky, C., & Heiberger, G. (2013).Putting twitter to the test: Assessing outcomes for student collaboration, engagement and success. British Journal of Educational Technology, 44(2), 273–287. https://doi.org/10.1111/j.1467-8535.2012.01284.x

Katrimpouza, A., Tselios, N., & Kasimati, M. C. (2019).  Twitter adoption, students’ perceptions, Big Five personality traits and learning outcome: Lessons learned from 3 case studies, Innovations in Education and Teaching International, 56(1), 25-35. https://doi.org/10.1080/14703297.2017.1392890 

Kerr, S. L., & Schmeichel, M. J. (2018). Teacher Twitter Chats: Gender Differences in Participants' Contributions, Journal of Research on Technology in Education, 50(3), 241-252. https://doi.org/10.1080/15391523.2018.1458260

Kuznekoff, J. H., Munz, S., & Titsworth, S. (2015). Mobile Phones in the Classroom: Examining the Effects of Texting, Twitter, and Message Content on Student Learning. Communication Education, 64(3), 344-365. https://doi.org/10.1080/03634523.2015.1038727 

Loutou, S., Tselios, N., & Altanopoulou, P. (2018). The effect of twitter-mediated activities on learning outcome and student engagement: A case study. Computers and society. https://doi.org/10.13140/RG.2.2.12783.02724. https://arxiv.org/abs/1803.01343 retrieved on January 23, 2021

Luo, T., & Xie, Q. (2018). Using Twitter as a pedagogical tool in two classrooms: a comparative case study between an education and a communication class. Journal of Computing in Higher Education, 31, 81-104. https://doi.org/10.1007/s12528-018-9192-2

McConnell, D. (1997). Interaction patterns of mixed sex groups in educational computer conferences. Part I-empirical findings. Gender and Education, 9, 345-363.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7): e1000097. https://doi.org/10.1371/journal.pmed.1000097

Moule, P. Pontin, D. Gilchrist. M. Ingram, R. (2003). Critical appraisal framework. Retrieved Jan. 27, 2021, http://learntech.uwe.ac.uk/da/Default.aspx?pageid=1445

Sawilowsky, S.S. (2009). New effect size rules of thumb. Journal of modern applied statistical methods, 8(2), 597-599. https://doi.org/10.22237/jmasm/1257035100

Smith, J.E. & Tirumala, L. N. (2012). Twitter’s Effects on Student Learning and Social Presence Perceptions. Teaching Journalism and Mass Communication, 2(1), 21-31. 

Webb, A. L., Dugan, A., & Burchett, W., et al. (2015). Effect of a Novel Engagement Strategy Using Twitter on Test Performance. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 16(6). https://doi.org/10.5811/westjem.2015.10.28869

Yang, C.W., Yen, Z.S., McGowan, J.E., Chen, H.C., Chiang, W.C., & Mancini, M.E., et al. (2012). A systematic review of retention of adult advanced life support knowledge and skills in healthcare providers. Resuscitation, 83, 1055-1060.

Yoshida, M. (2021). Investigation of university students' behaviour in a heterarchical twitter community. Education and Information Technologies (1). https://doi.org/10.1007/s10639-020-10402-1

Yu, Z. (2021). The effects of gender, educational level, and personality on online learning outcomes during the COVID-19 pandemic. International Journal of Educational Technology in Higher Education, 18(14), 1-17. https://doi.org/10.1186/s41239-021-00252-3 

Zhu, Y. (2010). The application of twitter in Teaching. Chinese society of artificial intelligence & Chinese society of computer aided education. Proceedings of the 14th National Society of Computer-aided Education Annual Meeting, pp. 744-747.

Tables

Table 1. Characteristics of included studies

N

Author/year

Sample size

Country

Learning outcome

Twitter usage

Treatment

Control

1

Al-Bahrani et al., 2017

130

130

USA

academic achievements and their gender differences

Twitter as a supplemental learning activity

2

Arceneaux & Dinu, 2018

390

390

USA

academic achievements

Twitter as a supplemental learning activity

3

Attai et al., 2015

189

189

USA

academic achievements

Twitter integrated learning

4

Blank, 2016

94

78

USA/UK

academic achievements and attitude

Twitter as a supplemental learning activity

5

Blessing et al., 2012

30

33

USA

academic achievements

Twitter as a supplemental learning activity

6

Colliander et al., 2015

66

92

Sweden

attitude

Twitter integrated learning

7

Cozma & Hallaq, 2019

18110

18110

USA

attitude

Twitter integrated learning

8

DeGroot et al., 2015

16

22

USA

attitude

Twitter integrated learning

9

Dissanayeke et al., 2016

13

13

Sri Lanka

teaching effectiveness

Twitter integrated learning

10

Feezell, 2019

71

71

USA

attitude

Twitter integrated learning

11

Feliz et al., 2013

3026

3026

Spain

male-female

Twitter integrated learning

12

Fouz-González, 2017

44

24

Spain

academic achievements and teaching effectiveness

Twitter integrated learning

13

Hitchcock &    Young, 2016

30

30

USA

academic achievements

Twitter as a supplemental learning activity

14

Junco et al., 2010

19

53

USA

academic achievements and engagement

Twitter as a supplemental learning activity

15

Junco et al., 2013

65

53

USA

academic achievements

Twitter as a supplemental learning activity

16

Katrimpouza et al., 2019

80

152

Greece

academic achievements

Twitter as a supplemental learning activity

17

Kerr & Schmeichel,    2018

657

696

USA

Gender differences in learning outcomes

Twitter as a supplemental learning activity

18

Kuznekoff et al., 2015

30

32

USA

academic achievements

Twitter as a supplemental learning activity

19

Loutou et al., 2018

17

17

Greece

academic achievements and engagement

Twitter integrated learning

20

Luo & Xie, 2019

22

16

USA

academic achievements

Twitter integrated learning

21

Smith & Tirumala, 2012

41

35

USA

attitude

Twitter as a supplemental learning activity

22

Webb et al., 2015

2

66

USA

academic achievements

Twitter integrated learning

 

Table 2. Meta-analysis results for Twitter assisted learning outcomes

Learning outcomes

SMD

95% CI

%

weight

Heterogeneity statistic

df

p

I2(%)

z

p

Attitude

0.302

0.110

0.495

16.02

518.43

17

<.01

96.7

3.08

0.002

Academic achievement

0.380

0.155

0.604

66.11

2129.93

73

<.01

96.6

3.31

0.001

Gender difference

2.174

0.842

3.505

7.41

5965.96

7

<.01

99.9

3.20

0.001

Engagement

0.408

0.181

0.635

5.20

1.03

5

0.96

0

3.52

<.01

Teaching effectiveness

-0.140

-0.523

0.243

5.27

15.13

5

0.01

67.0

0.72

0.474

Overall

0.483

0.269

0.697

100

14131.53

111

<.01

99.2

4.43

<.01

 

Table 3. Meta-analysis results in different countries

Region

SMD

95% CI

%weight

Heterogeneity statistic

df

p

I2(%)

z

p

USA

0.488

0.218

0.758

74.99

12740.86

83

<.01

99.3

3.54

<.01

Greece

0.716

0.361

1.071

7.87

41.54

8

<.01

80.7

3.95

<.01

Spain

-0.160

-0.589

0.269

7.16

74.17

7

<.01

90.6

0.73

0.465

Sweden

0.404

0.194

0.614

5.49

12.48

5

0.029

59.9

3.77

<.01

Overall

0.489

0.269

0.697

100

 

 

<.01

 

4.43

<.01

 

Table 4. Meta-analytical results for academic achievements in different countries

Country

SMD

95% CI

%weight

Heterogeneity statistic

df

p

I2(%)

z

p

USA

0.305

0.052

0.558

83.97

1976.39

61

<.01

96.9

2.36

0.018

Greece

0.739

0.353

1.126

10.54

41.38

7

<.01

83.1

3.75

<.01

Spain

-0.236

-0.701

0.230

2.69

1.73

1

0.189

42.1

0.99

0.321

Overall

0.380

0.155

0.604

100.00

2129.93

73

<.01

96.6

3.31

0.001

 

Table 5. Meta-analysis results for different usages of Twitter

Twitter usage

SMD

95% CI

%weight

Heterogeneity statistic

df

p

I2(%)

z

p

As a supplementary tool

0.501

0.095

0.907

59.90

6995.91

66

<.01

99.1

2.42

0.015

As an integrated tool

0.459

0.216

0.702

40.10

14775.03

44

<.01

99.1

3.70

<.01

Overall

0.483

0.269

0.697

100.00

14131.53

111

<.01

99.2

4.43

<.01