A meta-analysis of the effect of Kahoot! on academic achievements and student performance

doi:10.21203/rs.3.rs-842089/v1

Kahoot!-based serious games have received international attention and obtained more than two billion players across over 200 countries and areas (Kahoot!, 2021). However, scanty studies have systematically examined their effect on educational outcomes using a meta-analysis. Through a meta-analysis using Stata/MP 14.0, this study concludes that Kahoot!-based serious games could significantly improve academic achievements and student performance, that the educational level of participants could not predict Kahoot!-assisted academic achievements and student performance, and that the difference in countries or areas could not predict Kahoot!-assisted academic achievements and student performance. More components such as cognitive loads, satisfaction, or anxiety in Kahoot!-based serious games could also be meta-analytically reviewed in the future.

Computational Biology

Bioinformatics

Psychology

Kahoot!

Serious game

Academic achievements

student performance

Meta-analysis

Kahoot!, integrated with serious games, could improve learning engagement, academic achievements, and student performance. It changes the physically dull classroom into an interesting game-assisted learning and teaching environment. Teachers could operate the game and encourage students to join as competitors. Teachers could project the questions to a large screen, requiring students to answer on their smart devices such as mobile phones and iPods. Teachers could then decide the proper pedagogical progress based on student performance shown on the large screen, where a chart could show either correct or wrong answers. If most students could not answer correctly, the teacher would then stop to further explain the questions. On the contrary, if most students could give correct answers, the teacher would move on without delay. The teacher could require students to answer questions either anonymously or with real names. In this way, the teacher could check students’ attendance and supervise their learning engagement (Wang, 2015).

Kahoot! is conducive to educational outcomes due to its nature of serious games. It is widely accepted that losing students’ attention poses a great threat to classroom-based learning, which may negatively influence academic achievements and student performance. How to sustain students’ attention, improve their motivation, and stimulate their learning interest are common problems awaiting solutions where serious games may play an important role. As a technology integrated with serious games, Kahoot!, different from the traditional approach, provides a new way to draw students’ attention, enhance their mastery of knowledge, improve their performance, and better their academic achievements.

Kahoot! could exert a positive influence on learning outcomes. Kahoot! has gained popularity since its pilot use in education in 2013 (Wang & Tahir, 2020). Since its birth in Norwegian University of Science and Technology in 2006, Kahoot! has been developed from a mere response system where teachers operate lecture quizzes for students to join and respond (Wang, 2015) to a platform with multiple functions such as quizzes, blank filling, multiple choices, and other serious games (Wang et al., 2007). While a great number of researches (e.g. Lauermann & Barbosa, 2018; Alawadhi & Abu-Ayyash, 2021; Chen, 2021) have been committed to its effectiveness and efficiency since then, scanty studies have meta-analytically or systematically examined its use in education. It is thus necessary to implement a meta-analysis of the effect of Kahoot! on educational outcomes to propose suggestions for future research and practice.

The subgroup in this meta-analysis is divided into both academic achievement and student performance including numerous components.

The components of academic achievements and student performance

The academic achievement is made of numerous variables of the same nature based on previous studies. Academic achievement tends to be determined by measuring the knowledge level acquired in either formal or informal educational contexts (Soares et al., 2015). It is also considered closely related to the broad conception of crystallized intelligence, which is deemed as the degree of knowledge acquired in a given context (Schneider & McGrew, 2012: 122). Later, academic achievement is used to measure the quality of learning including learner engagement, motivation, and learning experience (Lee & Shute, 2010). Therefore, academic achievement in this study includes homogeneous variables such as motivation, achievement tests, English vocabulary acquisition, English reading comprehension, final marks of a chemistry academic course, a formative assessment test of an Arabic language grammar course, e-learning susceptibility scores, e-learning avoidance scores, knowledge tests, final exam scores for nursing students, reading scores of EFL, student engagement, scores in knowledge, English vocabulary scores, students’ mathematics test scores, intrinsic motivation, scores of Pharmacology and Medicinal Chemistry, academic vocabulary scores, achievement scores, and test results of EFL pronunciation exams.

The student performance is also made of several variables based on previous studies. The internal conception of performance is multifaceted and developed based on the measurement of knowledge. The multifaceted dimensions of performance could be measured based on learner engagement and various competencies (Frank, 2005). The external component of performance is closely related to learning behaviors and learner actions (Watling et al., 2014). Thus, student performance in this study includes numerous dimensions such as diagnostic accuracy, sensitivity, and specificity, simulation performance, skill performance, vocabulary correct answers, delayed learning performance test score, exam performance, retention, concentration, homework, satisfaction, cooperation, involvement, perceived competency before and after the online faculty development program, the interaction between teachers and students, learning process management that emphasizes student participation, using media and learning resources, organizing the learning process, evaluation during teaching, teaching and personality, and encouraging learners to demonstrate proper behavior.

Academic achievements

Most studies have revealed positive effects of Kahoot! on academic achievements in Turkey. Kahoot!-based learning could significantly improve academic achievements and enhance learning engagement but reduce the anxiety degree of tests among middle school students in Turkey (Turan & Meral, 2018). The Kahoot!-based platform plays an important role in improving the learning and teaching effectiveness in online learning environments among prospective teachers in Turkey (Gokbulut, 2020). The Kahoot!-based platform could motivate Turkish EFL students to learn and improve reading comprehension of Turkish university students (Korkmaz & Öz, 2021). Pre-service teachers in Turkey held positive attitudes towards the use of Kahoot!-based serious games due to improvements in their perceived motivation, entertainment, experience, and competition (Korkmaz & Öz, 2021; Orhan Göksün & Gürsoy, 2019). The Kahoot!-based serious game could improve university nursing students’ knowledge acquisition and enhance their injection skills in Turkey (Öz & Ordu, 2021). The use of Kahoot! application could improve university students’ EFL pronunciation skills at various learning stages in Turkey (Yürük, 2020).

Many studies concluded that the use of Kahoot! could lead to positive academic achievements in Asia. Kahoot!-assisted learning could improve Arabic language acquisition of university students in the United Arab Emirates and significantly enhance their learning motivation (ElTahir et al., 2021). Kahoot!-based questions initiated by students in Taiwan could significantly improve their English skills and increase their mental efforts (Chen & Yeh, 2019). Kahoot!-assisted instant response system could improve intrinsic motivation of middle school students in Taiwan (Shyr et al., 2021). Kahoot!-based serious games, encouraging Taiwanese students to learn actively and allowing them to learn by playing anonymously, could improve academic achievements such as intrinsic motivation, learning engagement, and perceptions of learning activities (Wang et al., 2008). The Kahoot!-based serious game could facilitate tertiary students’ vocabulary learning in Malaysia (Tan & Goh, 2020).

Kahoot! was also proved effective in terms of academic achievements in education in Europe and America. Kahoot!-assisted learning could improve the acquisition of English vocabulary knowledge of middle school students in Chile (Quiroz et al., 2021). The Kahoot!-based serious game could improve learning engagement, enhance focus on academic activities, and extend the learning span of university students in the UK (Holbrey, 2020). Kahoot!-assisted learning could improve Spanish university students’ learning process, raise their test scores, and improve their engagement in learning activities (Díez-Pascual & García Díaz, 2020). The Kahoot!-based serious game could significantly improve the scores of final exams of university students in the USA (Kinder & Kurz, 2018).

However, Kahoot!-assisted learning also led to negative results in terms of academic achievements. Kahoot!-assisted learning could neither significantly improve tertiary nursing students’ academic achievements nor significantly enhance their motivational degree among university students in Turkey (Aras & Çiftçi, 2021). The Kahoot!-based serious game is limited to the expense of Internet connection, information system, small-sized screens, and space of word entry for prospective teachers in Turkey (Gokbulut, 2020).

Student performance

Many studies concluded that Kahoot! could improve student performance in Asia. The use of Kahoot! platform in Taiwan could improve rural secondary students’ learning motivation, draw their attention, encourage them to learn, and improve their learning efficiency (Lee et al., 2019). The Kahoot!-based response system could improve middle school students’ performance in mathematics learning, improve their learning interest, and enhance their self-efficacy in Taiwan (Shyr et al., 2021). Kahoot!-based serious games could improve Indonesian university students’ performance and mastery of vocabulary in interpersonal communication (Sartini, 2020). The Kahoot!-based program could improve Indian university students’ perception skills of teachers in delivering lecture notes during the pandemic. (Swaminathan et al., 2021). Students in a Thai university could obtain significantly higher exam scores in the Kahoot!-based group than the control group (Yuenyongviwat & Bvonpanttarananon, 2021). The Kahoot!-based serious games failed to improve the Thai university students’ satisfaction with the learning environment, learning management, and teacher performance (Yuenyongviwat & Bvonpanttarananon, 2021). Pre-service Turkish teachers perceived that the Kahoot!-based approach could improve their cooperation, engagement, and task completion, and reduce the difficulty levels of learning tasks (Özüdoğru, 2020).

Numerous studies revealed that Kahoot! could improve student performance in both Europe and America. Integrating Kahoot! into a pedagogical approach could facilitate Greek secondary students’ perceptions of conceptual knowledge of electric circuits, encourage their active engagement in class, improve their learning motivation, and cultivate a favorable learning atmosphere (Tsihouridis et al., 2018). The use of Kahoot! could improve diagnostic skills and it is thus a worthwhile educational tool for university students in Brazil (De Oliveira et al., 2019). Kahoot!-integrated serious games could promote British university students’ participation in learning and cultivate an interactive learning environment via their timely feedback and shared discussion on the forum (Holbrey, 2020). Pharmacy school students in an American university performed significantly better in the Kahoot!-based method than the traditional one (Sumanasekera et al., 2020).

Although there are many positive findings regarding the effect of Kahoot! on student performance, several studies argued that Kahoot! might play a negative role in student performance. While Singaporean university students held positive attitudes toward Kahoot!-integrated learning methods, their knowledge levels, and performance test scores were not significantly improved (Ignacio & Chen, 2020). Kahoot!-based learning could not improve students’ satisfaction and their cohesiveness in the classroom context (Özüdoğru, 2020). While Kahoot!-based serious games were considered an effective and worthy pedagogical approach, videos and exercises obtained significantly more learning success in terms of pharmacology conception, which was measured by their performance in the exams (Sumanasekera et al., 2020).

We observe that previous studies have revealed both negative and positive effects of Kahoot! on academic achievements and student performance across the world. The participants are generally either university students or middle school students. Considering the inconsistent previous findings, the research question proposed is: could Kahoot!-based serious games, educational levels, and different countries or areas significantly improve academic achievements and student performance? We thus propose research hypotheses as follows:

H1. Kahoot!-based serious games could significantly improve academic achievements and student performance.

H2. The educational level of participants could predict Kahoot!-assisted academic achievements and student performance.

H3. The difference in countries or areas could predict Kahoot!-assisted academic achievements and student performance.

This meta-analytical study is conducted based on the specific procedure and methods suggested by Moher et al. (2009) in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).

Literature search and inclusion

We searched the literature via online databases such as Web of Science, Sage, Wiley Online Library, Taylor & Francis, Elsevier Science Direct, EBSCO Academic Search Premier, and Springer Nature by entering keywords according to their specific syntactic rules. The keywords to search literature include Kahoot!, serious game, academic achievement, performance, education, control group, experimental group, treatment group, invention group, and effect, etc.

Generally, we finally obtained a total of 2752 results. Specifically, we obtained 2,674 results from Web of Science by keying in search terms: Kahoot* OR "serious game*" (Title) and "serious game*" OR "academic achievement" OR "perform*" OR "education" OR "control group" OR "experimental group" OR "treatment group" OR "invention group" OR effect (Abstract). We obtained 5 results from Elsevier Science Direct by keying in search terms: "control group" OR "experimental group" OR "treatment group" OR "invention group" AND title “Kahoot”. We obtained 4 results from Wiley Online Library for "Kahoot" in Title and "serious game*" OR "academic achievement" OR "perform*" OR "education" OR "control group" OR "experimental group" OR "treatment group" OR "invention group" OR effect" in Abstract. We obtained 3 results for [Publication Title: kahoot] AND [[Abstract: "serious game*"] OR [Abstract: "academic achievement"] OR [Abstract: "perform*"] OR [Abstract: "education"] OR [Abstract: "control group"] OR [Abstract: "experimental group"] OR [Abstract: "treatment group"] OR [Abstract: "invention group"] OR [Abstract: effect]].

Additionally, we obtained 62 results from EBSCO Academic Search Premier for “TI kahoot AND AB ("serious game*" OR "academic achievement" OR "perform*" OR "education" OR "control group" OR "experimental group" OR "treatment group" OR "invention group" OR effect)”. We obtained 2 results from Sage journals for [Title kahoot] AND [[Abstract "serious game*"] OR [Abstract "academic achievement"] OR [Abstract "perform*"] OR [Abstract "education"] OR [Abstract "control group"] OR [Abstract "experimental group"] OR [Abstract "treatment group"] OR [Abstract "invention group"] OR [Abstract effect]]. We obtained 2 results from Springer Nature for “Kahoot in title and control group as an exact phrase”. The search time period of all the databases ranges from their initiation to August 4, 2021.

We finally included 24 results for the meta-analysis after removing the duplicated results, screening the irrelevant results, excluding those ineligible, and deleting those without full texts or enough data information (Figure 1). The inclusion and exclusion criteria are established based on researchers’ rich experiences.

We will include the studies in case if they (1) are closely related to the use of Kahoot! in education, (2) are of higher quality based on The University of West England Framework for Critically Appraising Research Articles (Moule et al., 2003), (3) can report full texts and enough quantitative data for the meta-analysis, (4) are committed to the effect of Kahoot! on academic achievements and student performance, and (5) are controlled randomized trials. We will exclude the studies in case they (1) are not devoted to the use of Kahoot! in education but to its technological issues, (2) do not conduct the study based on control and experimental groups, (3) are themselves review studies, or (4) are of lower quality based on The University of West England Framework for Critically Appraising Research Articles (Moule et al., 2003).

Data extracting and literature coding

Two professors solicited and coded data from the included studies (Table 1). The data included author names, publication years, participants’ numbers, means, and standard deviations across both experimental and control groups. We coded a subgroup by dividing the findings into either academic achievements or student performance. We extracted the data regarding the place of research and grouped them into Asia, America, and Europe. Educational levels of the participants were also extracted to examine their relationship with academic achievements and student performance. If two researchers could not reach an agreement on any issue, a third reviewer would be invited to make a decision. Generally, two researchers reached a satisfactory level of the inter-rater reliability (Kappa = 0.83).

Data analyses

We first tested the heterogeneity of outcomes of the included studies using I², Heterogeneity statistics, and values of z and p. The variance of the study outcomes will be considered unimportant if 0 ≦ I²≦ 40%, moderate if “30% ≦ I²≦ 60%”, substantial if “50% ≦ I²≦ 90%”, and considerable if “75% ≦ I²≦ 100%” (Higgins & Green, 2021). A random-effect model will be adopted in the meta-analysis if I²≧ 50%, and a fixed-effect model will be in use if I²≦ 50%.

We calculated effect sizes using standard mean differences (SMD), namely Cohen d. The formula to calculate d is: d = (M1-M2)/SD_pooled, where M1 refers to the mean of the experimental or intervention group, while M2 refers to the mean of the control group. SD_pooled indicates the value of standard deviation across both experimental and control groups (Sedgwick & Marston, 2013). The effect size will be considered very small if d ≈ 0.1, small if d ≈ 0.2, medium if d ≈ 0.5, large if d ≈ 0.8, very large if d ≈ 1.2, and huge if d ≈ 2 (Sawilowsky, 2009).

We will test the publication bias and stability of the estimated results. We will test the publication bias of the included studies using both Begg’s (Begg & Mazumdar, 1994) and Egger’s tests (Egger, Smith, Schneider, & Minder, 1997). We will also test the stability of the estimated results using the program sensitivity or influence analysis in Stata/MP 14.0.

This section will present the tests of publication bias, sensitivity analysis, the heterogeneity tests, meta-analyses of academic achievements and student performance, and the regression analyses.

Testing publication bias

We tested the publication bias of the selected literature by entering “metabias _ES _seES, by (Subgroup1) graph (begg)”into Stata/MP 14.0 as the command. We then obtained a funnel plot (Figure 2).

The results indicate the absence of the publication bias. As shown in Figure 2, the x-axis indicates the standard errors of standardized mean differences. The y-axis refers to the standardized mean differences. The middle line is the no-effect line, indicating that if an individual study crosses it, there will be no significant effect of this study on the publication bias. A dot refers to a specific study. The dots are nearly symmetrically arranged along both sides of the middle line. It indicates the absence of the publication bias. Begg’s tests indicate no presence of the publication bias in terms of both academic achievement (z = 1.40, p = 0.161) and student performance (z = 1.37, p = 0.17). Egger’s tests also indicate the absence of the publication bias of both academic achievement (z = 6.12, p = 0.062) and student performance (z = 3.45, p = 0.266).

Testing sensitivity of the analysis

We entered “metaninf N M SD N0 M0 SD0, label (namevar=Author, yearvar=Year) random cohen” into Stata/MP 14.0 to test sensitivity of the analysis. Then we obtained a plot (Figure 3), where the left column shows both author names and publication years, and the right column shows the 95% confidence intervals including the lower and upper confidence interval limits. The dots indicate the values of SMDs.

All the estimates of SMDs are located within the upper and lower confidence interval limits given a named study is omitted (Figure 3). This indicates that the meta-analytical estimates are stable.

Testing the heterogeneity

We used I² and Heterogeneity statistics to determine either fixed-effect model or random-effect model could be used to conduct the meta-analysis. The results indicate that the percentage of variance of the heterogeneity of the estimates is statistically significant rather than due to random errors in both academic achievement (Q = 165.21, I2 = 89.7%, z = 4.35, p < .01) and student performance (Q = 124.71, I2 = 91.2%, z = 4.33, p < .01). We, therefore, conducted the meta-analysis using the random-effect model.

Meta-analyses of academic achievements and student performance

We adopted the random-effect model to implement the meta-analysis by entering “metan N M SD N0 M0 SD0, label (namevar=Author, yearvar=Year) by (Subgroup1) random cohen” into Stata/MP 14.0. We then obtained a forest plot (Figure 4), where the left column indicates the author names and publication years; the right column indicates the standardized mean differences and 95% confidence intervals; and the middle column shows the forest plot. In the forest plot, the middle line is referred to as the no-effect line. A box crossed by a line with a dot in the middle indicates an SMD and a 95% confidence interval. The diamonds at the bottom indicate the pooled results. If the line or the diamond crosses the no-effect line, we will consider that there is no significant effect regarding the differences between the experimental and control groups.

As shown in Figure 4, the diamonds of both academic achievements and student performance are located to the right of the no-effect line without touching it. This indicates that both academic achievements and student performance are significantly higher in the experimental group than in the control group. We, therefore, accept the research hypothesis that Kahoot!-based serious games could significantly improve academic achievements (d = 0.65, 95%CI = 0.36-0.94) and student performance (d = 0.99, 95%CI = 0.54-1.43).

The regression analyses

We test the effect of educational levels of participants on Kahoot!-assisted academic achievements and student performance using a meta-regression analysis by entering into Stata/MP 14.0 the command “metareg _ES EdL, wsse (_seES) bsest(reml)”. The meta-regression analysis shows no significant correlation between the educational level, academic achievement, and student performance (coefficient = .334, SE = .618, z = .54, p = .589, 95%CI = -.878-1.546). Therefore, we reject the research hypothesis that the educational level of participants could predict Kahoot!-assisted academic achievements and student performance.

We also determined the effect of countries or areas on both academic achievements and student performance using a meta-regression analysis by entering the command “metareg _ES Countryarea, wsse (_seES) bsest (reml)”. The result does not show significant relationship between countries or areas, academic achievements, and student performance (coefficient = .324, SE = .522, z = .62, p = .534, 95%CI = -.699-1.348). We thus reject the research hypothesis that the difference in countries or areas could predict Kahoot!-assisted academic achievements and student performance.

It is reasonable to arrive at the conclusion that Kahoot!-based serious games could significantly improve student performance. Kahoot! allows students to answer questions in various forms such as quizzes, multiple choices, polling, blank-filling, or discussion. Students might answer questions through voices, videos, or pictures. They could also attend the online classes assisted with Kahoot!. This brings about great convenience for both teachers and students especially during this special pandemic time (Díaz-Sainz et al., 2021). Due to its multiple functions, students and teachers do not need to attend the physical classes that require close physical face-to-face contacts. They can sit at home to conveniently attend the class at their will. This could doubtlessly encourage students to join the class actively and perform well. Kahoot! could smoothen the learning process and enlarge students’ capacity of memory (Díaz-Sainz et al., 2021), which might thus improve student performance. In Kahoot!-based serious games, teachers often provided badges for students, which could motivate them and simulate their learning interest, leading to better performance (Lister, 2015).

The rationales for students to obtain satisfactory academic achievements may be of variety in the classes based on Kahoot!. The serious game provides gamification elements for students to play, encouraging them to finish learning task when playing games. Students feel fun in the gameplay and attempt to obtain favorable results when playing. This definitely initiates them to complete learning goals efficiently and effectively. They could also join the gameplay anonymously, which may relax them and improve their academic achievements. The open achievements on the large screen may inform that they have numerous competitors in the gameplay. They are thus encouraged to learn and play actively to obtain better academic achievements in the competitive and entertaining environments. Kahoot!-based questions could improve students’ attention, motivation, and entertainment (Parra-Santos et al., 2018), leading to improved academic achievements. Kahoot! could improve in-class interactions and cooperation, and could thus exert a positive influence on student attitudes toward learning, performance, and learning atmosphere, contributing to enhanced students’ academic achievements (Wang & Tahir, 2020).

It is reasonable to conclude that educational levels could not predict Kahoot!-based academic achievements and student performance. With the development of information science and technologies, Kahoot!-based serious games have gained great popularity in education regardless of educational levels. The primary use in pre-secondary schools has been extended to various levels of education such as higher education and secondary levels. Learning technologies such as Kahoot! could be used to enhance learning outcomes due to their interactions and collaboration features across educational levels (Zucker & Fisch, 2019). Gamification elements provided by Kahoot! could promote learning success, improve teaching efficiency, and provide benefits for teachers and students at every educational level (Pertegal-Felices et al., 2020). Gamification elements have been widely accepted and can improve learning outcomes at all educational levels (Subhash & Cudney, 2018).

It is also reasonable to reveal that the difference in countries or areas could not predict Kahoot!-assisted academic achievements and student performance. Kahoot! has received international attention and obtained more than two billion players across over 200 countries and areas (Kahoot!, 2021). It has been proved beneficial to education outcomes in many countries such as Turkey (Aras & Çiftçi, 2021), Spain (Díez-Pascual & García Díaz, 2020), the United Arab Emirates (ElTahir et al., 2021), the UK (Holbrey, 2020), Indonesia (Sartini, 2020), Singapore (Ignacio & Chen, 2020), and the USA (Sumanasekera et al., 2020). Kahoot! may not have gained popularity in some countries due to the limitations to accessibility or socio-economic issues. But this limitation does not indicate the significant effect of countries or areas on Kahoot!-based academic achievements and student performance.

This concluding section will summarize the major findings, the limitations, and future research directions.

Major findings

Major findings of this study are generally consistent with previous findings. It concludes that Kahoot!-based serious games could significantly improve academic achievements and student performance, that the educational level of participants could not predict Kahoot!-assisted academic achievements and student performance, and that the difference in countries or areas could not predict Kahoot!-assisted academic achievements and student performance.

Limitations

Several limitations could be reported to this study. Firstly, the study cannot include all the qualified studies because of limited library sources. Secondly, the study outcomes are heterogeneous, which may have influenced the reliabilities of the pooled effect. Lastly, the study cannot include all the countries or areas across the world, which may influence the reliability of the study.

Future research directions

Future research could expand to a wider scope of literature and examine studies in more countries or areas. More components such as cognitive loads, satisfaction, or anxiety in Kahoot!-based serious games could also be meta-analytically reviewed in the future.

Alawadhi, A., & Abu-Ayyash, E. A. S. (2021). Students’ perceptions of Kahoot!: An exploratory mixed-method study in EFL undergraduate classrooms in the UAE. Education and Information Technologies. https://doi.org/10.1007/s10639-020-10425-8

Aras, G. N., & Çiftçi, B. (2021). Comparison of the effect of reinforcement with question-answer and kahoot method on the success and motivation levels of nursing students: a quasi-experimental review. Nurse Education Today, 102(2), 104930. https://doi.org/10.1016/j.nedt.2021.104930

Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101.

Chen, C. H., & Yeh, H. C. (2019). Effects of integrating a questioning strategy with game-based learning on students’ language learning performances in flipped classrooms. Technology, Pedagogy and Education, 1–15. https://doi.org/10.1080/1475939x.2019.1618901

Chen, Y. M. (2021). Understanding foreign language learners’ perceptions of teachers' practice with educational technology with specific reference to Kahoot! and Padlet: A case from China. Education and Information Technologies. https://doi.org/10.1007/s10639-021-10649-2

De Oliveira, M. L. B., Verner, F. S., Kamburoğlu, K., Silva, J. N. N., & Junqueira, R. B. (2019). Effectiveness of Using a Mobile App to Improve Dental Students’ Ability to Identify Endodontic Complications from Periapical Radiographs. Journal of Dental Education, 83(9), 1092–1099. https://doi.org/10.21815/jde.019.099

Díaz-Sainz, G Pérez, L Gómez-Coma, VM Ortiz-Martínez, & Rivero, M. J. . (2021). Mobile learning in chemical engineering: an outlook based on case studies. Education for Chemical Engineers, 35, 132–145. https://doi.org/10.1016/j.ece.2021.01.013

Díez-Pascual, A. M., & García Díaz, M. P. G. (2020). Audience Response Software as a Learning Tool in University Courses. Education Sciences, 10(12), 350. https://doi.org/10.3390/educsci10120350

Egger, M., Smith, G.D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British medical journal, 315, 629–634. https://www.jstor.org/stable/25175671

ElTahir, M. E., Alsalhi, N. R., Al-Qatawneh, S., Alqudah, H. A., & Jaradat, M. (2021). The impact of game-based learning (GBL) on students’ motivation, engagement and academic performance on an Arabic language grammar course in higher education. Education and Information Technologies, 26(3), 3251–3278. https://doi.org/10.1007/s10639-020-10396-w

Frank, J.R. (2005). The CanMEDS 2005 Physician Competency Framework. Better standards. Better physicians. Better care. Ottawa, ON: Royal College of Physicians and Surgeons of Canada.

Gokbulut, B. (2020). The effect of Mentimeter and Kahoot applications on university students’ e-learning. World Journal on Educational Technology: Current Issues, 12(2), 107–116. https://doi.org/10.18844/wjet.v12i2.4814

Higgins, J.P.T., & Green, S. (2021). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. 2021 Available from https://handbook-5-1.cochrane.org/

Holbrey, C.E. (2020). Kahoot! Using a game-based approach to blended learning to support effective learning environments and student engagement in traditional lecture theatres, Technology, Pedagogy and Education, 29(2), 191-202. https://doi.org/10.1080/1475939X.2020.1737568

Ignacio, J. , & Chen, H. C. (2020). The use of web-based classroom gaming to facilitate cognitive integration in undergraduate nursing students: a mixed methods study. Nurse Education in Practice, 46, 102820. https://doi.org/10.1016/j.nepr.2020.102820

Kahoot!. (2021). Research articles on Kahoot!. Oslo, Norway. URL: https://kahoot.com/research-articles/ [accessed August 5, 2021].

Kinder, F. D., & Kurz, J. M. (2018). Gaming Strategies in Nursing Education. Teaching and Learning in Nursing, 13(4), 212–214. https://doi.org/10.1016/j.teln.2018.05.001

Korkmaz, S., Öz, H. (2021). Using Kahoot to improve reading comprehension of English as a foreign language learner. International Online Journal of Education and Teaching (IOJET), 8(2). 1138-1150.

Lauermann, V., & Barbosa, D. N. F. (2018). Digital Technologies Applied to Textual Reading and Understanding in English: A Practical Approach Using Kahoot. In Communications in Computer and Information Science (pp. 70–79). Communications in Computer and Information Science. https://doi.org/10.1007/978-3-319-95522-3_7

Lee, C. C. , Hao, Y. , Lee, K. S. , Sim, S. C. , & Huang, C. C. . (2019). Investigation of the effects of an online instant response system on students in a middle school of a rural area. Computers in Human Behavior, 95(6), 217-223. https://doi.org/10.1016/j.chb.2018.11.034

Lee, J., & Shute, V. J. (2010). Personal and social-contextual factors in K–12 academic performance: An integrative perspective on student learning. Educational Psychologist, 45(3), 185–202. https://doi.org/10.1080/00461520.2010.493471

Lister, M. (2015). Gamification: The effect on student motivation and performance at the post-secondary level. Issues and Trends in Educational Technology, 3(2). https://doi.org/10.2458/azu_itet_v3i2_lister

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., & The PRISMA Group (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med, 6(7): e1000097. https://doi.org/10.1371/journal.pmed.1000097

Moule, P. Pontin, D. Gilchrist. M. Ingram, R. (2003). Critical appraisal framework. Retrieved August 4, 2021, http://learntech.uwe.ac.uk/da/Default.aspx?pageid=1445

Orhan Göksün, D., & Gürsoy, G. (2019). Comparing success and engagement in gamified learning experiences via Kahoot and Quizizz. Computers & Education, 135, 15–29. https://doi.org/10.1016/j.compedu.2019.02.015

Öz, G. Ö., & Ordu, Y. (2021). The effects of web based education and Kahoot usage in evaluation of the knowledge and skills regarding intramuscular injection among nursing students. Nurse Education Today, 103, 104910. https://doi.org/10.1016/j.nedt.2021.104910

Özüdoğru, M. (2020). The Use of a Student Response System in Teacher Training Classrooms and Its Effect on Classroom Environment. Acta Didactica Napocensia, 13(1), 29-42. https://doi.org/10.24193/adn.13.1.4

Parra-Santos, T., Molina-Jordá, J. M., Maiorano-Lauria, L. P., Casanova-Pastor, G. (2018). Gamifification for formative assessment in the framework of engineering learning. ACM International Conference Proceeding Series, http://dx.doi.org/10.1145/3284179.3284193, pp. 61–65.

Pertegal-Felices, M. L., Jimeno-Morenilla, A., Sánchez-Romero, J. L., & Mora-Mora, H. (2020). Comparison of the Effects of the Kahoot Tool on Teacher Training and Computer Engineering Students for Sustainable Education. Sustainability, 12(11), 4778. https://doi.org/10.3390/su12114778

Quiroz, M. F. , R Gutiérrez, Rocha, F. , Valenzuela, P. , & Vilches, C. . (2021). Improving English Vocabulary Learning through Kahoot!: A Quasi-Experimental High School Experience. Teaching English with Technology, 21(2), 3-13. http://www.tewtjournal.org

Sartini, S. (2020). Kahoot in maritime English teaching: Its impact on nautical science cadet’s oral reproduction and vocabulary. English Language Teaching Educational Journal, 3(1), 41-51. https://doi.org/10.12928/eltej.v3i1.1667

Sawilowsky, S.S. (2009). New effect size rules of thumb. Journal of modern applied statistical methods, 8(2), 597-599. https://doi.org/10.22237/jmasm/1257035100

Schneider, W. J., & McGrew, K. (2012). The Cattell–Horn–Carroll model of intelligence. In D. Flanagan, & P. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 99–144). New York: The Guilford.

Shyr, W. J., Hsieh, Y. M., & Chen, C. H. (2021). The Effects of Peer-Based Instant Response System to Promote Learning Performance, Intrinsic Motivation and Self-Efficacy. Sustainability, 13(8), 4320. https://doi.org/10.3390/su13084320

Soares, D. L., Lemos, G. C., Primi, R., & Almeida, L. S. (2015). The relationship between intelligence and academic achievement throughout middle school: The role of students' prior academic performance. Learning and Individual Differences, 41, 73–78. https://doi.org/10.1016/j.lindif.2015.02.005

Subhash, S., & Cudney, E.A. (2018). Gamified learning in higher education: A systematic review of the literature. Computers in human behavior, 87, 192–206. https://doi.org/10.1016/j.chb.2018.05.028

Sumanasekera, W. , Turner, C. , Ly, K. , Hoang, P. , & Sumanasekera, T. . (2020). Evaluation of multiple active learning strategies in a pharmacology course. Currents in Pharmacy Teaching and Learning, 12, 88-94. https://doi.org/10.1016/j.cptl.2019.10.016

Swaminathan, N. , Govindharaj, P. , Jagadeesh, N. S. , & Ravichandran, L. . (2021). Evaluating the effectiveness of an online faculty development programme for nurse educators about remote teaching during covid-19. Journal of Taibah University Medical Sciences, 16(2), 268-273. https://doi.org/10.1016/j.jtumed.2020.11.003

Tan, A.W.L., & Goh, L.H. (2020). Comparing the Effectiveness of Direct Vocabulary Instruction and Incidental Vocabulary Learning in Improving the Academic Vocabulary of Malaysian Tertiary Students. Pertanika journal of social sciences & humanities, 28 (S2), 263 – 279.

Tsihouridis, C., Vavougios, D., & Ioannidis, G. S. (2018). Assessing the Learning Process Playing with Kahoot – A Study with Upper Secondary School Pupils Learning Electrical Circuits. In Advances in Intelligent Systems and Computing (pp. 602–612). https://doi.org/10.1007/978-3-319-73210-7_70

Wang, A. I. (2015). The wear out effect of a game-based student response system. Computers & Education, 82, 217-227. http://dx.doi.org/10.1016/j.compedu.2014.11.004.

Wang, A. I., & Tahir, R. (2020). The effect of using kahoot! for learning – a literature review. Computers & Education, 149, 103818. https://doi.org/10.1016/j.compedu.2020.103818

Wang, A. I., Øfsdal, T., & Mørch-Storstein, O. K. (2007). Lecture quiz - a mobile game concept for lectures. In IASTED international conference on software engineering and application (SEA 2007) (p. 6). Cambridge, MA, USA: Acta Press.

Wang, A. I., Øfsdal, T., & Mørch-Storstein, O. K. (2008). An evaluation of a mobile game concept for lectures. In Proceedings of the 2008 21st conference on software engineering education and training. IEEE Computer Society.

Watling, C. , Driessen, E. , Van, d. V. C. P. M. , & Lingard, L. . (2014). Learning culture and feedback: an international study of medical athletes and musicians. Medical Education, 48(7), 713-723. https://doi.org/10.1111/medu.12407

Yuenyongviwat, V., & Bvonpanttarananon, J. (2021). Using a Web-Based Quiz Game as a Tool to Summarize Essential Content in Medical School Classes: Retrospective Comparative Study. JMIR Medical Education, 7(2), e22992. https://doi.org/10.2196/22992

Yürük, N. (2020). Using Kahoot as a skill improvement technique in pronunciation. Journal of Language and Linguistic Studies, 16(1), 137-153. Https://doi.org/ 10.17263/jlls.712669

Zucker, L., & Fisch, A.A. (2019). Play and Learning with KAHOOT!: Enhancing Collaboration and Engagement in Grades 9-16 through Digital Games. Journal of language and literacy education, 15(1), 1-15.

Table 1. A brief summary of included studies

N	Author	Year	Focus	Country/ area	Level
1	Aras & Çiftçi	2021	Success and motivation levels of nursing students	Turkey	university
2	Chen & Yeh	2019	language learning performances	Taiwan	university
3	De Oliveira et al	2019	Dental students’ ability	Brazil	university
4	Díez-Pascual & García Díaz	2020	Marks of chemistry academic course	Spain	university
5	ElTahir et al	2021	formative assessment test of Arabic language grammar course	The United Arab Emirates	university
6	Gokbulut	2020	e-learning scores	Turkey	university
7	Holbrey	2020	Engagement, retention, and concentration	UK	university
8	Ignacio & Chen	2020	Knowledge and simulation performance	Singapore	university
9	Kinder & Kurz	2018	Exam scores for nursing students	USA	university
10	Korkmaz & Öz	2021	reading scores of EFL	Turkey	university
11	Lee et al	2019	Exam and homework	Taiwan	secondary
12	Orhan Göksün & Gürsoy	2019	Achievement and engagement	Turkey	university
13	Öz & Ordu	2021	Knowledge and skill performance	Turkey	university
14	Özüdoğru	2020	Satisfaction, cooperation, and involvement	Turkey	university
15	Quiroz et al	2021	English vocabulary score	Chile	secondary
16	Sartini	2020	Frequent number of vocabulary correct Answers	Indonesia	university
17	Shyr et al	2021	Mathematics scores and intrinsic motivation	Taiwan	secondary
18	Sumanasekera et al.	2020	Pharmacology and Medicinal Chemistry scores	USA	university
19	Swaminathan et al	2021	perceived competency	India	university
20	Tan & Goh	2020	academic vocabulary scores	Malaysia	university
21	Tsihouridis et al	2018	performance	Greece	secondary
22	Turan & Meral	2018	Achievement and engagement	Turkey	secondary
23	Yuenyongviwat & Bvonpanttarananon	2021	Essential Content in Medical School Classes	Thailand	university
24	Yürük	2020	Test results of EFL pronunciation	Turkey	university

No competing interests reported.

A meta-analysis of the effect of Kahoot! on academic achievements and student performance

Status:

Version 1

Abstract

Figures

Introduction

Literature Review

Methods

Results

Discussion

Conclusion

References

Table 1

Additional Declarations

Status:

Version 1