The protocol for this study was registered within the Open Science Framework database
(https://osf.io/vxdhf/?view_only=313fd05399664b94bc7a9042aa225be3) before data collection began. This was a case-control study to retrospectively examine factors associated with mega peer reviewers as compared to a control group of peer reviewers. Mega reviewers were defined as individuals that completed peer reviews for 100 or more unique articles from January 2018 to December 2018. All aspects of this study were reported in accordance with the Strengthening (STROBE) reporting guidelines to facilitate the complete and transparent reporting of this work.9
Case Control Data
We gathered information from the Publons database. Publons tracks and publicizes peer-reviewer activities for individuals that create an account and connect their research activities to their profile. Individuals can download their peer-review, author, and editor metrics and this information can also be made public. Using the Publons database, two groups of individuals were of interest for this study, including (1) mega peer reviewers: all individuals that completed peer reviews for 100 or more unique articles from January 2018 to December 2018, inclusive (case group) (i.e., individuals completing approximately two peer reviews every week) and (2) a control group of individuals completing at least one peer review and less than 18 peer reviews over the same time period (i.e., individuals completing up to 1 peer review every 3 weeks). A random sample of controls were selected from Publons database.
A data scientist from Publons exported the following variables into a csv Microsoft excel file: peer reviewer characteristics [i.e., name, publons ID, institution, country of institution, number of publications based on Publons data, publications in Web of Science, publications in 2018, citations in 2018, total number of citations, h-index, presence of a Publons reviewer award (top 1% of reviewers in 22 research areas, top quality reviews based on editor rated evaluations)], review characteristics based on Publons data (i.e., number of unique manuscripts peer-reviewed in 2018, number of unique manuscripts reviewed each month, average number of words per review, average number of words per review at reviewer’s institution). Sex was not available on Publons. As such, sex was estimated by using the Genderize data base, which uses data collected from countries to assess the probability of the sex being associated with a given name (https://genderize.io/). For any sex that could not be estimated with more than 80% certainty, this was marked as missing data.
Sample Size Calculation
The mega peer-reviewers sample size was based on the number of peer-reviewers on the Publons website that met our inclusion criteria (i.e., greater than 100 peer-reviews 2018). For the control group, a sample size calculation based on the total number of reviewers that met the control group requirements (i.e., completing at least one review and less than 18 peer-reviews in 2018) was conducted using the standard deviation of the average word count which was estimated using preliminary data from Publons. A sample size calculation was conducted in R package (pwr) for a two sample t-test comparing mega peer reviewers and the control group. The pooled standard deviation was calculated, and a minimum sample size of 1167 was estimated (see Appendix 1). To determine the number of peer reviewers needed for the control group, a 1:1 random sample was selected and the standard deviation of the average word count of peer review quality was determined.
Primary data analysis calculated descriptive characteristics of both samples of reviewers. The secondary data analysis involved conducting a logistic regression to compare the mega peer reviewer characteristics to the control group, treating mega-reviewing as a binary outcome. Given the exploratory nature of this study, the association between peer reviewer characteristics (i.e., sex, country of institution, number of publications in Web of Science, publications in 2018, citations in 2018, total number of citations, h-index, presence of Publons reviewer award) and review characteristics (i.e., average number of words per review, average number of words per review at reviewer’s institution) (independent variables) were compared in the two groups of reviewers (mega-reviewers and control group of reviews) within the regression model. Prior to conducting regression analyses, preliminary tests were performed to determine the appropriateness of analyses based on any violations of regression assumptions. Tests of multicollinearity were conducted, and independent variable tolerance values were reviewed. Four variables did not reach the recommended cut-off values for collinearity statistics,10 suggesting that variables were highly correlated with one another [tolerance values less than 0.1 for publications based on Publons (0.04), publications based on Web of Science (0.04), citations in 2018. (0.06), total citations (0.06). As such, independent samples t-tests for continuous data and chi-square analyses for categorical data were used. Analyses were conducted using SPSS version 27.0, and statistical tests were two sided with a significance value of P < .05