Although Altmetrics was launched in 2011, some 728 of the papers published before this year had Altmetrics data (representing 31.2% of all the papers with Altmetrics data, which makes around 8.7% of the papers published to that date). Of all Altmetrics sources, Mendeley had the highest coverage. The average number of tweets per article in the pre-Altmetrics era (before 2011) was 0.39 compared to 3.98 in the post-Altmetrics era; such a large difference was statistically significant t(1715.89) = 11.81, p < .001). The average number of Mendeley readers for the pre-Altmetrics era was 37.7, compared to 45.6 in the post-Altmetrics era and the difference was statistically insignificant. The descriptive statistics for papers with Altmetrics data are presented in Table 1. As the table shows, except for Mendeley and Twitter mentions, the numbers were very low. The average number of Twitter mentions was 2.87 on average (SD = 10.11, Median = 1.00). News mentions were low (mean = 0.07, SD = 0.64, Median = 0.00) and so were blog mentions (mean = 0.06, SD = 0.29, Median = 0.00). The mean number of Mendeley readers was 43.17 (SD = 91.64, Median = 21). Wikipedia mentions (mean = 0.09, SD = 0.42, Median = 0.00), and Facebook mentions (mean = 0.03, SD = 0.21, Median = 0.00) were also very low.
Table 1
Mention statistics per source (n = 2336)
| Median | Mean | Std. Deviation | Minimum | Maximum |
Twitter mentions | | 1.00 | | 2.87 | | 10.11 | | 0 | | 313 | | |
Number of Mendeley readers | 21.00 | | 43.17 | | 91.64 | | 0 | | 1836 | | |
Facebook mentions | | 0.00 | | 0.03 | | 0.21 | | 0 | | 4 | | |
Blog mentions | | 0.00 | | 0.06 | | 0.29 | | 0 | | 6 | | |
News mentions | | 0.00 | | 0.07 | | 0.64 | | 0 | | 16 | | |
Patent mentions | | 0.00 | | 0.19 | | 2.62 | | 0 | | 118 | | |
Policy mentions | | 0.00 | | 0.03 | | 0.18 | | 0 | | 2 | | |
Q&A mentions | | 0.00 | | 0.01 | | 0.08 | | 0 | | 2 | | |
Video mentions | | 0.000 | | 0.004 | | 0.062 | | 0.000 | | 1.000 | | |
Wikipedia mentions | | 0.000 | | 0.086 | | 0.422 | | 0.000 | | 6.000 | | |
The trend of growth of Altmetrics data in Fig. 1 shows that only Twitter mentions are up-trending while all other trends are irregular or trending down. It is noteworthy to mention that each service has different time dynamics. For instance, in Mendeley, it is conceivable that the older a paper is, the more likely it is to be read by more readers and therefore, the trend should not be interpreted as a decreasing number of readers, but rather that older papers have been read more times overall. Similar to Mendeley, the patents, policy and Wikipedia mentions are expected to cite highly regarded or well-established papers. On the other hand, in news and blogs, recent papers are expected to make it to the news or be blogged about by authors. Similarly, Twitter mentions are expected to grow with time as researchers turn to Twitter to discuss the emerging research. Yet, it is hard to infer any future trends from other services, as the number of paper mentions is small and current trends are irregular.
Twitter mentions
The oldest paper that has Altmetrics data in our dataset was published in 1968 (Kuno and Oettinger 1968). The paper was tweeted by the account of Teaching NLP Workshop “How long have folks been thinking about #TeachingNLP? Here's a paper from more than 50 years ago by Susumu Kuno and Anthony G. Oettinger (CACM 1968).” (Teaching NLP Workshop @NAACL2021 2021). Articles with Twitter mentions were 1391 (59.5% of all articles with Altmetrics data, 15.3% of all articles in the post-Altmetrics era and 8.3% of all the articles in the dataset). The average number of Twitter mentions for any article in the whole dataset (16838 articles) was 0.39, while the average number of tweets for the articles was 4.81 (SD = 12.75, Median = 2, range: [1, 313]). The average age of articles that have received Twitter mentions was 5.8 years, compared to 15.8 in the non-mentioned articles. Such difference was statistically significant, and large (difference = 9.97, 95% CI [9.61, 10.33], t(2915.89) = 54.74, p < .001; Cohen's d = 2.03, 95% CI [1.94, 2.12]). Similarly, the tweeted articles received an average of 2.9 citations/article/year compared to 0.7 in the non-tweeted articles, the difference was statistically significant, and large (difference = -2.19, 95% CI [-2.45, -1.93], t(1410.09) = -16.56, p < .001; Cohen's d = -0.88, 95% CI [-1.12, -0.77]). Articles with Twitter mentions were more likely to have more Mendeley readers (mean = 48.57) compared to 35.22 for the articles with no Twitter coverage. This difference was statistically significant and the effect size was small t(1779.40) = -3.35, p < .001; Cohen's d = -0.16, 95% CI [-0.25, -0.07]).
The correlation between the number of Twitter mentions and total citations was statistically insignificant, while the correlation between Twitter mentions and number of citations per article per year was weak r = 0.17, p < 0.001. Furthermore, the correlation between Twitter mentions and number of Mendeley readers, policy mentions, Wikipedia mentions, patent mentions, and Facebook mentions were either trivial or statistically insignificant.
In summary, articles on Twitter tended to be more recent, with slightly more citations per article per year, as well as more Mendeley readers. It is important here to emphasize that we make no assumptions of any causal relationship, i.e., we do not imply that Twitter mentions increased the citations or readership. In fact, it is possible that the mechanism that made the article receive more citations (e.g., interesting, or novel findings) caused both Twitter mentions and citations. In all cases, such differences were very small.
Table 2
Correlation between Twitter mentions and other social media
Source | r | | p |
Number of Mendeley readers | | 0.163 | | < .001 |
Policy mentions | | 0.106 | | < .001 |
Q&A mentions | | 0.052 | | 0.055 |
Wikipedia mentions | | 0.069 | | 0.010 |
Patent mentions | | -0.048 | | 0.071 |
Facebook mentions | | 0.099 | | < .001 |
Total Citations | | 0.033 | | 0.216 |
Total Citations per year | | 0.167 | | < .001 |
Age of publication | | -0.252 | | < .001 |
The most mentioned topic on Twitter was computational thinking which garnered 2,384 mentions (9.7% of all Twitter mentions), followed by computational theory (1,943 mentions, 8%), programming (1861, 7.6%), introductory courses (1443, 5.9%) and pedagogy (1416, 5.7%) and education psychology (1,347, 5.5%). The order of the most mentioned topics and the timeline of tweets per year of publication is shown in Fig. 2. We see that topic of pedagogy, assessment, and introductory courses as well as games were early mentioned on Twitter. As the graph shows, there is no certain pattern that we can discern from the graph, and the timeline looks rather irregular. We also see that the year 2022 had witnessed a large increase in Twitter mentions for the first five topics (computational thinking, computational theory, introductory courses, pedagogy, and education psychology).
The top articles mentioned on Twitter in Table 3 come from different themes, e.g., ethics, programming education, introductory courses as well as computational thinking and inclusion. The top cited article in the list discusses the state of ethics education in computer science education. The authors claim that the field has an “ethics crisis” that needs to be addressed to avoid what they call “exclusionary pedagogy” where there is lack of interdisciplinarity and collaboration with other fields to improve the ethics curricula (Raji, Scheuerman, and Amironesei 2021). Five other papers addressed programming education discussing diverse topics. McGowan et al. (2017) reported a positive correlation between seating in the front row during programming classes and performance. Stefik and Siebert (2013) investigated the intuitiveness of the syntax of different programming languages. Drake and Sung (2011) used board games to introduce computer science topics to university students. Salac and Franklin (2020) found a weak correlation between performance and quality indicators calculated from school children’s Scratch artifacts. In the same token, Chen et al. (2019) found a positive correlation between prior programming experience and attitudes towards programming as well as academic achievement, and concluded that it is more effective to teach young students using a graphical language than a text-based one. Two articles among the top mentioned articles discussed political aspects of computer science education (Williamson 2016; Malazita and Resetar 2019). The last article in our list (Kemp, Wong, and Berry 2020) discusses female participation and attainment in CS; where the findings indicate that females score higher than their male peers but lower than their average score in other courses. The article also argues that the introduction of CS into the national curriculum might “decrease the number of girls choosing further computing qualifications or pursuing computing as a career”.
It is worth noting that six of the top Twitter mentioned articles have received less than 5 citations, emphasizing the discord between Twitter publicity and academic interest (as measured by citation count). Nonetheless, it is not difficult to discern where there has been a conversation about these articles on Twitter. For instance, two articles' titles have chosen thought provoking titles “You can't sit with us” and “choose your lecture seat carefully”. One article addresses the programming language war, and two articles address policy and politics, and an article cautions against introducing CS into female education. Lastly, the remaining of these articles addresses graduate students' dissertations which would be expected to be shared among doctoral students who are social media users.
Table 3
Top mentioned articles in Twitter
Title | Year | Twitter mentions | Citations |
"You Can't Sit with Us": Exclusionary Pedagogy in AI Ethics Education | 2021 | 313 | 3 |
Learning to Program - Choose your Lecture Seat Carefully! | 2017 | 139 | 3 |
An Empirical Investigation into Programming Language Syntax | 2013 | 123 | 139 |
Teaching Introductory Programming with Popular Board Games | 2011 | 115 | 20 |
The Organization and Content of Informatics Doctoral Dissertations | 2016 | 104 | 1 |
Infrastructures of Abstraction: How Computer Science Education Produces Anti-Political Subjects | 2019 | 103 | 4 |
Political Computational Thinking: Policy Networks, Digital Governance and ‘Learning to Code’ | 2016 | 67 | 31 |
If they Build it, Will they Understand it? Exploring the Relationship between Student Code and Performance | 2020 | 67 | 4 |
The Effects of First Programming Language on College Students’ Computing Attitude and Achievement: A Comparison of Graphical and Textual Languages | 2019 | 65 | 23 |
Female Performance and Participation in Computer Science: A National Picture | 2019 | 61 | 4 |
A total of 3044 authors had at least a paper mentioned on Twitter: 75.6% of them had a single paper and around 97.2% of the authors had five papers or less. The median year of the first publication of the authors with Twitter mentions was 2016 (compared to 2011) which reflects the recency of the Altmetrics and Twitter. There was a weak correlation between the mean number of tweets an author has and the mean number of citations their article gets; Spearman's rank correlation was statistically significant and medium (r = 0.21, p < .001). Yet, while the correlation is weak, it should not be interpreted as causation.
The top authors with Twitter mentions were not the most cited or the most productive authors. Nonetheless, they were mostly among the top 50 authors. On top of the list was Brett A. Becker, an assistant professor at the School of Computer Science at University College Dublin who had 30 of his papers mentioned on Twitter, each receiving an average of four mentions. Aman Yadav, a professor of educational psychology & educational technology at Michigan State University, had 24 of his papers discussed on Twitter, with a total of 202 mentions and an average of 8.4 mentions per paper. Amy J. Ko, professor of informatics at University of Washington, Seattle, and the Editor-in-Chief of TOCE had 22 of her articles discussed on Twitter, with an average of 6.2 mentions per paper. Table 4 has the full list of authors with most discussed papers on Twitter.
Table 4
Top mentioned authors on Twitter
Author | Oldest | # of papers | Proportion | Rank | Mean mentions | Mean citations |
Becker BA | 2016 | 30 | 0.682 | 36 | 4.1 | 17.2 |
Yadav A | 2011 | 24 | 0.667 | 64 | 8.4 | 26.75 |
Ko AI | 2009 | 22 | 0.611 | 61 | 6.2 | 22.36 |
Petersen A | 2011 | 20 | 0.4 | 23 | 3.5 | 19.95 |
Porter I | 2010 | 19 | 0.333 | 14 | 2.2 | 20 |
Franklin D | 2011 | 18 | 0.409 | 38 | 9.9 | 11.67 |
Cutts Q | 2007 | 16 | 0.432 | 55 | 11.6 | 9.62 |
Hellas A | 2016 | 16 | 0.356 | 35 | 4.7 | 10.19 |
Sentance S | 2011 | 16 | 0.364 | 39 | 11.6 | 14 |
Falkner K | 2009 | 15 | 0.375 | 47 | 5.4 | 13.93 |
Guzdial M | 1994 | 15 | 0.172 | 1 | 1.5 | 23.6 |
Simon | 1997 | 15 | 0.174 | 2 | 3 | 23.4 |
Luxton-Reilly A | 2005 | 14 | 0.2 | 6 | 3.4 | 27.36 |
Kafai YB | 2008 | 13 | 0.325 | 48 | 3.3 | 23.23 |
McGill MM | 2009 | 13 | 0.277 | 32 | 4.6 | 11 |
Mendeley
Some 2,268 articles had Mendeley data, which is 97% of all articles with Altmetrics data and 13.46% of all the articles in the dataset. The mean number of mentions was 43.17 (SD = 91.64, Median = 21), and the mean age of publication (time since published) was 9.21 (SD = 8.37, Median = 7.00) which is older than the mean age on Twitter. The presence of Mendeley data and the number of readers per article are well known to correlate with the number of citations across several studies (e.g., Erdt et al. 2016), which was the case in our study. Articles with Mendeley data were more likely to be cited with a mean of 19.51 citations compared to 6.00 in articles without, the difference was statistically significant and medium (difference = -13.51, 95% CI [-15.39, -11.63], t(2331.86) = -14.08, p < .001; Cohen's d = -0.58, 95% CI [-0.76, -0.50]). Within articles with Mendeley data, correlation between number of readers per article and citation count was statistically significant, positive and very large (r = 0.64, 95% CI [0.61, 0.66], t(2266) = 39.14, p < .001). There was also a weak correlation between the number of Mendeley readers and policy, news, blogs, Facebook, or Wikipedia mentions.
Regarding authors, the number of articles with Mendeley readers was correlated with citation count, which was statistically significant, and effect size was very large (r = 0.68, 95% CI [0.66, 0.69], t(4327) = 60.86, p < .001). Similarly, the number of Mendeley readers was highly correlated with citation counts which was statistically significant with a very large effect size(r = 0.78, 95% CI [0.77, 0.80], t(4327) = 83.35, p < .001). In summary, there is a very strong association between Mendeley mentions and citation counts for either papers or authors, which reflects the paper importance or impact rather than has caused the citation.
The top CSE papers with the highest number of readers (Table 5) are expected to also reflect highly cited papers since we have established that correlation was high. Our top papers include seven papers that address issues related to computational thinking of CSE in schools. The other three papers address game-based learning, an instructional computer laboratory and computer curriculum. Most of the top read Mendeley papers are highly cited with six papers having over 100 citations. The top read paper about game-based learning (Papastergiou 2009) is also the top cited paper in our complete dataset; the fifth paper about bringing computational thinking to K-12 (Barr and Stephenson 2011) is the second most cited paper, and the seventh top read paper about Scratch is the third most cited paper (Maloney et al. 2010).
Table 5
Top read papers in Mendeley
Title | Year | Mendeley readers | Citations |
Digital Game-based learning in High School Computer Science Education: Impact on Educational Effectiveness and Student Motivation | 2009 | 1836 | 984 |
Progress Report: Brown University Instructional Computing Laboratory | 1984 | 1763 | 16 |
Computational Thinking | 2007 | 1509 | 63 |
Design Patterns: An Essential Component of CS Curricula | 1998 | 1466 | 42 |
Bringing Computational Thinking to K-12: What is Involved and What is the Role of the Computer Science Education Community? | 2011 | 817 | 658 |
Which Cognitive Abilities Underlie Computational Thinking? Criterion Validity of the Computational Thinking Test | 2017 | 786 | 215 |
The Scratch Programming Language and Environment | 2010 | 727 | 640 |
Computational Thinking in Elementary and Secondary Teacher Education | 2014 | 613 | 191 |
Computational Thinking for All: Pedagogical Approaches to Embedding 21st Century Problem Solving in K-12 Classrooms | 2016 | 587 | 133 |
Constructivism in Computer Science Education | 1998 | 584 | 140 |
The list of authors with a high number of articles in Mendeley (Table 6) show interesting findings that are different from those of Twitter. Most of the authors in the list are among the top publishing authors in the general dataset. The top authors were also well-established authors who started their careers during the last century or in the early 2000s, the mean years in publishing about CSE was 19.93 [6, 32]. Each of the top authors had an average of 53.41 Mendeley readers [18.83, 118.72]; each of their papers received a mean number of citations per paper of 27.26 [11.47, 48.92]. Such numbers were not much different from other authors who are not on the top list.
Table 6
Top read authors on Mendeley
Author | Oldest | # of papers | Proportion | Rank | Mean mentions | Mean citations |
Guzdial M | 1994 | 49 | 0.56 | 1 | 38.06 | 38.37 |
Rodger SH | 1993 | 42 | 0.78 | 16 | 18.83 | 18.48 |
Astrachan O | 1990 | 34 | 0.65 | 17 | 55.74 | 11.47 |
Becker BA | 2016 | 33 | 0.75 | 36 | 37.36 | 17.12 |
Simon | 1997 | 33 | 0.38 | 2 | 36.09 | 17.30 |
Ben-Ari M | 1996 | 28 | 0.57 | 26 | 76.36 | 38.18 |
Ko AI | 2009 | 27 | 0.75 | 61 | 55.04 | 23.00 |
Lister R | 2000 | 25 | 0.37 | 9 | 53.88 | 48.92 |
Sheard J | 1997 | 25 | 0.33 | 4 | 52.88 | 27.16 |
Yadav A | 2011 | 25 | 0.69 | 64 | 118.72 | 30.48 |
Edwards S | 1998 | 24 | 0.35 | 8 | 47.71 | 36.04 |
Porter I | 2010 | 24 | 0.42 | 14 | 32.08 | 20.50 |
Luxton-Reilly A | 2005 | 23 | 0.33 | 6 | 55.04 | 29.65 |
Petersen A | 2011 | 23 | 0.46 | 23 | 51.22 | 20.91 |
Armoni M | 2004 | 22 | 0.49 | 34 | 72.14 | 31.36 |
News and blogs
Computing education research has appeared rarely in the news where only 79 (0.46%) articles were mentioned across the whole dataset, with a total of 136 mentions in total. The most mentioned articles by the news (Table 7) seem to address diversity and gender issues, which made more than half of the news mentions. The top article in the list was discussed by, e.g., Scientific American, Los Angeles Times, Christian Science Monitor, Houston Chronicle and SF Gate. Christian Science Monitor presented the article and concluded “Children need to be engaged in STEM before they start to lose interest. The image of STEM as solitary and isolating is strong in our culture. If we make STEM social, we can help inspire more students to discover their interest in STEM” (Master 2016). The second article with a significant number of news mentions has also discussed gender diversity and was mentioned by Business Insider, World Economic Forum, and The National Interest. For instance, Business Insider titled their story “Women are just as capable as men in computing skills —but they're not as confident. Here's how that's contributing to the gender gap in tech” (The Conversation 2021). The authors of the paper concluded that “many have made the case that companies need better participation of women in the STEM workforce for greater innovation and productivity. These efforts have had some success, but other avenues are needed to promote STEM careers to women and help them to believe in their abilities.” The World Economic Forum presented a similar story with the title “Computing has a gender problem – and isn’t about talent.”
Table 7
Top mentioned papers in the news
Title | Year | Citations | News mentions |
Computing Whether She Belongs: Stereotypes Undermine Girls' Interest and Sense of Belonging in Computer Science | 2016 | 173 | 16 |
Fostering Gender Diversity in Computing | 2013 | 13 | 16 |
Multiple Case Study of Nerd Identity in a CS Class | 2014 | 5 | 9 |
They Can't Find Us: The Search for Informal CS Education | 2014 | 22 | 9 |
Gender Neutrality Improved Completion Rate for All | 2016 | 1 | 6 |
What Is AI Literacy? Competencies And Design Considerations | 2020 | 51 | 6 |
Computer Science Trends and Trade-Offs in California High Schools | 2021 | 1 | 6 |
History of Logo | 2020 | 9 | 5 |
A Growth Mind-Set Intervention Improves Interest but not Academic Performance in the Field of Computer Science | 2020 | 23 | 5 |
Collaborative Strategic Board Games as a Site for Distributed Computational Thinking | 2011 | 99 | 4 |
Some 134 papers (0.8%) had blog mentions with 150 blog appearances in total. The highest blogged about paper (six times) was also the paper that received the top news mentions and addressed the stereotypes about girls interest in computer science (Master, Cheryan, and Meltzoff 2016). The blog named “Scienceblog” published a blog post titled “To Get Girls More Interested In Computer Science, Make Classrooms Less ‘Geeky’”. All other blog mentions were two mentions or less and therefore, will not be discussed in detail here.
Patents
A total of 131 (0.78%) articles received patent mentions and received a total of 434 mentions combined. Table 8 shows the articles with the most mentions. The article that received almost one third of all patent mentions describes an online computerized testing system called “QUIZIT” which supports adaptive and standard testing, automatic grading, and storage of results (Tinoco, Barnette, and Fox 1997). The paper was mentioned by several patents across a wide range of applications that include systems and methods for automatic scheduling of a workforce, discovering customer center information, recording audio as well as by a web service for student information and course management systems. The next article on the list discusses the development of a programming project where Java applets can be dynamically updated in an undergraduate programming course (Yang, Linn, and Quadrato 1998). The paper was mentioned by several patents (30) mostly covering access to database and software design. The remaining papers with patent mentions revolve around the same themes, i.e., either enhancement to an online teaching system or teaching programming.
Table 8. Top papers mentioned by patents
Title
|
Year
|
Citations
|
Patent mentions
|
Online Evaluation in WWW-Based Courseware
|
1997
|
19
|
118
|
Developing Integrated Web and Database Applications Using Java Applets and JDBC Drivers
|
1998
|
5
|
30
|
A Reusable Graphical User Interface for Manipulating Object-Oriented Databases Using Java and XML
|
2001
|
6
|
11
|
A Constructivist Approach to Object-Oriented Design and Programming
|
1999
|
30
|
11
|
The KScalar Simulator
|
2002
|
9
|
11
|
Interactive Hypermedia Courseware for the World Wide Web
|
1996
|
6
|
9
|
On-Line Programming Examinations Using WebToTeach
|
1999
|
9
|
9
|
Teaching Web Development Technologies In CS/IS Curricula
|
1998
|
8
|
8
|
Using a Model Railroad to Teach Digital Process Control
|
1988
|
9
|
8
|
Using Java to Teach Networking Concepts with a Programmable Network Sniffer
|
2003
|
10
|
8
|
Other Altmetrics sources
Other services had very few mentions. Only two articles had six mentions by Wikipedia (Mounier-Kuhn 2012; Osborne and Yurcik 2003), where the first discussed computer science education in French Universities and the second discussed visual simulation. Facebook mentions were also very scarce: the highest mentioned article received only four mentions and discussed computational thinking (Yadav, Hong, and Stephenson 2016). On the questions and answers website Stack Exchange, the mentions were even fewer, with only a single article mentioned two times (Liberal Arts Computer Science Conso 2007). The article was mentioned as a reply to the question “Which math classes should be included in an undergraduate computer science program?”.