Scientometers and sociologists of science have spilled much ink on the topic of peer review over the past twenty years given its primordial role in a context marked by the exponential growth of scientific production and the proliferation of predatory journals. Although the topic is addressed under different prisms, few studies have empirically analyzed to what extent it can affect the quality of publications. Here we study (1) the link between the length of reviewers' reports and the citations received by publications, (2) The threshold (expressed in number of words in reviewers' reports) at which the effect on citations becomes statistically significant. To do this, we used data from the Publons database (58,093 peer review reports). We have adjusted this sample to match the WoS database structure. Our regression results show that peer review positively affects the quality of publications. In other words, the more in-depth (longer) the referees' reports are, the greater the publication improvements will be, resulting in an increase in citations received. This result is important from both the point of view of reviewers and that of journal's chiefs-editors. Even if it is not a remunerated activity, it is important that it be more valued at least within the framework of research evaluation exercises, given its positive impact on science. On the thresholds, the results differ between the types of regression used. The threshold is 1250 words (about one page) for Negative Binomial regression, 1400 words for Poisson regression and 2400 words for GLM regression. These results suggest that a report that brings the most value to the publication is a deep report, which naturally requires more time to produce. This is an important result in a context where the deadlines granted to reviewers tend to decrease for some publishers.
Article
On The Peer Review Reports: Does Size Matter?
https://doi.org/10.21203/rs.3.rs-2410967/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Research evaluation
length review reports
peer-review
citation impact
research funding
open peer review
bibliometrics
The number of scientific journals has grown exponentially over the past twenty years that it is "now impossible for a researcher to apprehend all of the published literature in his field" 1. This provides fertile ground for the development of predators seeking to take advantage of the situation. The number of predatory journals is currently estimated at 13,000 journals with more than hundred thousand published articles 2,3.
3 provided a detailed description of the predation phenomenon in the scholarly publishing. Predatory journals are characterized by three main strategies: all predatory journals are Open Access (OA) journals based on a "author-pays" business model, in which peer review is not rigorous (or even absent) and displays "standard markers of scientific publications" (e.g. ISSN, DOI, etc.).
As much of the literature has shown, the publishing world cannot be categorized solely as black or white 4,5. Sometimes the line between what a predatory journal and a real scientific journal is not clear. One of the criteria that allows deciding is the quality of the peer review practiced within journals. In particular if, all the other administrative characteristics surrounding the publication are not false or usurped (e.g. ISSN, DOI). This is the case, for example, of newly created journals, little known, with low bibliometric indicators (e.g. low impact factor). To grow and exist in the current oligopolistic market of scholarly communication, with many barriers to entry, these journals sometimes adopt aggressive commercial strategies. Therefore, they may look like predatory journals without actually being.
This issue creates a certain anxiety among the scientific community who fear for their reputation if they happen to publish in journals, which prove to be predatory, or without scientific basis 6,7. Some have described a kind of omerta within the scientific community 8. In this regard, 9 stressed that the OA status of journals should not be a reason for distrust for researchers, since not all OA journals are predatory. (Björk and Solomon, 2012) have also emphasized that researchers should, on the other hand, carefully check the quality standards of journals before submitting their work.
In this context, the role of peer review is paramount and deserves special attention. How can we "assess" peer review? Does the length of reviewers’ reports improve the quality of manuscripts? Alternatively, is it only a rhetorical instrument intended for publishers and authors? The analysis carried out by Publons team is the only one to our knowledge that has investigated the link between the size of reviewers’ reports and the citation impact on a large sample of over 378,000. 10 showed the existence of a positive correlation between the average number of words in evaluation reports and the impact of journals. Nevertheless, the study concluded that it is not possible to say with certainty that longer reviews are better or worse than shorter ones: “Great reviews can be short and concise. Poor reviews can be long and in-depth, or vice-versa”. The analysis carried out by Publons remains quite descriptive and does not indeed allow confirming the positive link observed between the two variables. For this, an econometric model is necessary to neutralize the effects of the variables affecting citation impact on the one hand, and the length of the reviewers’ reports on the other.
In this study, we hypothesize that reviewers’ reports actually improve the quality of publications. The interest they have received within the scientific community and results in a high number of citations reflects to some extent their quality. Thus, the length of reviewers’ reports is representative of the number of modifications / improvements that authors must make to their manuscript for an eventual acceptance for publication. In other words, there should be a positive relationship between the length of reviewers’ reports and the quality of scientific publications.
To test this hypothesis, we use on the one hand the data provided by Publons (https://publons.com/) for the length of reviewer’ comments, and on the other hand on the data from the Web of Science (WoS) database for the calculation of bibliometric indicators. We performed an econometric model to analyze the link between the two variables: size of reviewers’ reports and citation impact.
The article's plan is the following: First, we present a literature review on the link between peer review (in general) and the quality of publications. Then, we present some descriptive statistics on the database used (Publons), followed by the results section. Finally, the conclusion and the discussion return to the results obtained and their scope and implications.
Literature review
Peer review is a topic that has garnered much ink in the literature in Scientometrics and the sociology of science. We will only present here the work related to our research question, which is the impact of the peer review process on the quality of publications, approached by citations indicators.
Peer review processes are generally long, with disparities across disciplines 11–14. Especially for accepted publications 15. Over the past ten years, the duration has lengthened because of the inflation in the number of publications submitted to journals. On the other hand, the acceptance rate has increased by around 50% 16. In addition to the constraints related to the number of publications submitted each week to journals, the fact that referees are not paid for their review activity can contribute to slowing down the process 17–19. The increase in the technicality of publications can also increase the evaluation time in certain disciplines (Azar, 2007).
Whatever the reasons that may affect the evaluation times of scientific publications, the final objectives of this are multiple and revolve around improving the effective quality of publications. 20 outlined eight purposes of peer review: (1) assess the contribution and originality of a manuscript, (2) perform quality control, (3) improve manuscripts, (4) assess the suitability of manuscripts for the topics of the journal, (5) provide a decision-making tool for editors, (6) provide comments and peer feedback, (7) strengthen the organization from the scientific community, and (8) provide some sort of accreditation for published papers.
Studies that assess the quality of the review process, from the perspective of authors or editors, are quite rare. 14 analyzed data from 3,500 experiences and author reviews published on SciRev.sc (https://scirev.org/). The SciRev.sc interface groups together author reviews and ratings assigned to journals based on the quality (perceived by them) of the review process. 14 have shown that, unsurprisingly, reviews with short response times tend to score higher. The same goes for experiences that resulted in a positive response from the journal. 21 arrived at similar results on a sample of 193 authors. Furthermore, 14 have shown that journals in disciplines where the evaluation processes are relatively long are on average better rated than journals in disciplines where the processes are short. In a more recent study, 22 analyzed the perception of authors and editors of the quality of peer reviews in 12 journals: 809 manuscripts and 313 opinions and recommendations. 22 found that authors give high ratings and positive perception when reviewers' comments recommend acceptance, unlike comments that recommend rejection (which is to be expected). On the other hand, the evaluations recommending a revision, are of better quality according to the indicator used (Review Quality Instrument - RQI). In addition, 22 have shown a strong association between the recommendation of referees and the publication decision of editors.
Studies that cross the length of the peer review process with the impact of journals or publications are even rarer. The study of 23 on 22 ecological / interdisciplinary journals, showed the existence of an inverse relationship between the acceptance delay and the impact factor of the journals. These same journals, however, have relatively low acceptance rates. The analysis of 24 on the three journals (Nature, Science and Cell), over the period 2005-09, arrived at different conclusions from those of 23. For the three journals studied, the authors observed an inverse relationship between editorial time and the number of citations received.
There is room for improvement in the existing literature on assessing the length of the peer review process. The analyzes of 14, of 21 and of 22 can be criticized insofar as they assess the perceived quality of the evaluation and not the intrinsic quality. The perceived quality depends very much on the final decision and the length of the revisions requested by the referees; it is therefore very subjective. Hence, a thorough analysis of referees can result in a lot of criticism and be a source of improvement of the publications, even if the authors perceive it negatively, because it delays the publication (and adds a lot of additional work). To measure intrinsic quality, it is essential to cross-reference the performance indicators (eg citations received) of journals or articles.
Likewise, the study by (Pautasso and Schäfer, 2010) which showed that high impact journals have short turnaround times ignore the type of decision made by the journals. In general, the number of articles submitted to high-impact journals is important, as is the rejection rate. The rejection decision usually comes a few days / weeks after submission, reducing the average processing time for the entire review. Taking into account the time to get the first response and the type of response are important elements in the analysis. Thus, it can be assumed that lead times for accepted papers are higher in high-impact journals.
Data
Web of Science data
The data about citations scores and disciplinary assignation of publications has been extracted from the “Observatoire des Sciences et Techniques” (OST) in-house database. It includes five indexes of WoS available from Clarivate Analytics (SCIE, SSCI, AHCI, CPCI-SSH and CPCI-S. for more information see: https://clarivate.com/webofsciencegroup/solutions/webofscience-platform/) and corresponds to WoS content indexed through the end of November 2020. We have limited the analysis only to the original contributions; i.e. the following documents types: "Article", "Conference proceedings" and "Review".
Publons data
Publons is an interface created in 2012 specializing in peer review issues. In 2017, the supplier of the WoS database, Clarivate Analytics, acquired Publons. Currently, this database index over 2 million researcher profiles who share their peer review experiences. Publons offers a free service for researchers to promote their contributions. Publons also made available more than 300,000 items designating the characteristics of referee evaluation reports in journals: average word count, country of referee and journal. As part of this study, Publons made its database available to us by adding article identifiers (WoS UT). This allowed us to easily match Publons data with that of the WoS database and add other variables. Our final dataset contains 58,093 distinct publications reviewed by 62,152 reviewers (Table 1). Publons database provides word count per review.
|
#Reports by publication |
#Publications |
Freq. (%) |
#Reports |
Freq. (%) |
|---|---|---|---|---|
|
1 |
54,296 |
93.464 |
54296 |
87.360 |
|
2 |
3552 |
6.114 |
7104 |
11.430 |
|
3 |
229 |
0.394 |
687 |
1.105 |
|
4 |
15 |
0.026 |
60 |
0.097 |
|
5 |
1 |
0.002 |
5 |
0.008 |
|
Total |
58,093 |
100.000 |
62,152 |
100.000 |
Adjustment of Publons data
The data extracted from Publons database represents a sample of the global WoS database, which we consider in this work as the population. When working with samples (e.g. public opinion polls), it is important to adjust data for population parameters. Thus, to study whether the conclusions drawn on the Publons database can be extended to the WoS database, we analyzed whether the respective structures are similar, based on a certain number of variables considered significant in explaining citation scores. Given that there are statistically significant differences between the Publons structure and that of the WoS, we adjusted the Publons sample using the procedure called “raking ratio”25,26, which provides adjustment weights. The variables used for the adjustment of the Publons sample:
-
Publication year: dummy variables;
-
Open access: binary variable (Yes/No);
-
Financing: one, two, three, four or more;
-
Number of countries: one, two, three, four or more;
-
Scientific classification: ERC codes (dummy variable).
-
The impact of journal (5 classes : <0.8, [0.8, 1.2[, [1.2, 1.8[, [1.8, 2.2[, >=2.2), for the calculation method see: 27.
Figure 1 summarizes the Publons data adjustment procedure. Appendix A.1 and A.2 show the structure of the Publons sample and of the relevant WoS population, before and after adjustment.
Regression analysis
Before performing the regressions, some processing is necessary to harmonize the data. To simplify the statistical and econometric analysis, given the fact that nearly 93.5% of publications present a single report (see Table 1), we proceeded to a random draw to build a dataset with a single reviewer' report per document. This solution seems optimal because the Publons database does not provide the number of reviewers (number of reports) by document and, consequently, the use of procedures such as the clustering of residues seems to be not necessary in this case.
|
Lower bound |
Upper bound |
Effectif |
Freq. |
|---|---|---|---|
|
0 |
200 |
20939 |
36,0% |
|
200 |
400 |
14141 |
24,3% |
|
400 |
600 |
9160 |
15,8% |
|
600 |
800 |
5334 |
9,2% |
|
800 |
1000 |
3162 |
5,4% |
|
1000 |
1200 |
1998 |
3,4% |
|
1200 |
1400 |
1137 |
2,0% |
|
1400 |
1600 |
701 |
1,2% |
|
1600 |
1800 |
427 |
0,7% |
|
1800 |
2000 |
317 |
0,5% |
|
2000 |
2200 |
218 |
0,4% |
|
2200 |
2400 |
144 |
0,2% |
|
2400 |
2600 |
106 |
0,2% |
|
2600 |
2800 |
69 |
0,1% |
|
2800 |
3000 |
58 |
0,1% |
|
3000 |
13671 |
182 |
0,3% |
|
Total |
58093 |
100,0% |
|
|
Number of observations - total |
58093 |
||
|
Number of observations - sample |
36972 |
||
|
Average - total |
431 |
||
|
Average - sample |
610 |
||
|
Median - total |
306 |
||
|
Median - sample |
482 |
||
With regard to the distribution of the number of words in the reports, the analysis by percentiles shows that just over 60% of the reports are less than 400 words (see Table 2). It should be noted that approximately 10% of the total, or 5174 reports, have a report length of less than or equal to 20 words (which does not correspond to a normal length of a reference report). At the same time, 182 reports exceed 3,000 words (up to a maximum of 13,671 words), which represents approximately 0.3% of the total. We excluded from the regressions these two extremes of the distribution adding noise to the data. A preliminary econometric analysis shows the total lack of significance of the coefficient associated with the length of reports on any citation measure used, when the number of words is less than 200. Consequently, in the following, the econometric analysis will relate to the slices between 200 and 3000 words, which represents a little more than 64% of the reports. Table 3 shows the variables used for the regressions.
|
Type |
Variable |
Designation |
|---|---|---|
|
Dependent variable |
Citation Score |
Depending on the model, we used the citation score with or without transformation as follows: - the number of citations (without window), expressed in raw numbers, - using a logarithmic transformation, - using a square root, - Finally the logarithm of the Normalized Citation Score (NCS: see 28 for calculation method). |
|
Explanatory variable |
Review length words |
This is the word count per review report per publication. We tested two intervals: 200 to 3000 words, and 500 to 3000 words. |
|
Control variables |
Journal impact factor |
The classic two-year impact factor. |
|
Open Access |
The status of free access Yes / No (with "No" as a reference value). |
|
|
Funding |
The number of funding received by the publication (We used for this the WoS acknowledgment field). It is expressed in numbers with five binary modalities: - Without funding (Yes / No), - One funding (Yes / No), (reference value) - Two funding (Yes / No), - Three funding (Yes / No), - Four or more funding (Yes / No). |
|
|
Scientific collaboration |
The number of countries involved in the publication (using author addresses). It is expressed in numbers with four binary modalities: - A country (Yes / No), - Two countries (Yes / No), (reference value) - Three countries (Yes / No), - Four or more countries (Yes / No). |
|
|
Discipline |
27 + 2 disciplinary levels, according to the nomenclature used by the OST based on the WoS subjects categories and ERC panels. For more detail, see: https://figshare.com/articles/dataset/OST_-_Classification_of_WoS_subject_categories_into_27_2_ERC_panels_/21707543 |
|
|
Publication year |
The publication years, between 2010 and 2020 |
Table 4 shows the results of the estimates by model and by range of the word count of the evaluation reports. As can be seen on the value and significance of the coefficients, we find the same trends as those shown in the literature, concerning the control variables. Thus, open access has a positive impact on the citations received, as well as funding and international cooperation.
What interests us in this paper is the value and significance of our explanatory variable; the number of words per reviewer's report. Of the nine estimation models, eight indicate, ceteris paribus, the existence of a positive and statistically significant impact of the word count of reviewers' reports on the citations received by publications. This result tends to confirm our hypothesis that peer review contributes to improving the quality of publications. In other words, the number of words in the reports, which we used as a proxy for the modifications requested by the referees, improves the quality of the publications. The citations that the publications receive reflect this quality improvement, as here we consider the citations as a proxy for the interest aroused by the publication within the scientific community.
However, we should take this result with some caution for several reasons. If the correction of the structural bias allows a comparison of the Publons database with the control sample taken from WoS, it does not eliminate the selection bias. This selection bias appears if the publication of the reviewers’ reports in Publons is not done randomly but meets specific criteria (personal -age, gender-, and professionals -seniority, affiliation, scientific profile-, journal constraints concerning obligation/choice to publish reports, etc.). A few leads suggest the existence of a selection bias:
-
The Open Access rate in Publons is 43.3% (and 47.2% in the case of a sample limited to reports with a word count in the 200–3000 range) and in the control sample drawn of WoS is 28.7%. This over-representation in Publons may indicate non-random behavior in reviewer reporting. Therefore, it is not impossible that reviewers, who dump their entire reports in Publons, can do so strategically. In other words, we only put on Publons well-produced reports. A recent paper 29, based on evolutionary game theory, indicates that the type of peer review of the journal directly affects the behavior of researchers and reviewers. Specifically, the paper indicates that if the journal practices open peer review, it encourages researchers and reviewers to put in more effort (because it directly affects their reputation). Our results seem to indicate this phenomenon (especially on the side of the reviewers) and merit further investigation. Another paper 30 shows that the publication of reports can change the tone used by reviewers (especially younger ones) due in some cases to fears of reprisals.
-
The funding rate in Publons is 73.7% (and 76.3% in the case of a sample limited to reports with word counts in the 200–3000 range) and in the WOS control sample is of 53.9%. We know that the impact of funding on the citations of articles is positive and significant and, therefore, to hypothesize better quality. This could prompt some reviewers to declare their reports to Publons.
In sum, our regression results are both interesting and intriguing with respect to the Publons data, as they leave many questions unanswered. To be able to generalize our conclusions, it would be interesting to do the same work on a random sample of peer review reports. One idea would be to carry out the analysis using a panel of journals that practice open peer review, while respecting the problem of representativeness of the data and their structure in relation to the population (WoS, Scopus or any other large database - e.g. OpenAlex).
At this stage, we can simply say that the preliminary results suggest the existence of a positive link between the length of the peer evaluation reports and the quality of the publications (approximated by the citations).
We ran the regression in Table 4 several times, gradually increasing the word count interval by fifty with each regression. Figure 2 shows the evolution of the coefficient of the variable "length of evaluation reports" according to the number of words. We have circled the part of the curve when the coefficient is statistically significant (at an alpha risk of 5%). We carried out this exercise using the three models: Poisson (quote count), Negative Binomial (quote count) and GLM (Normalized Citation Score – log transformed).
Table 4: Results of estimates of the impact of the length of reviewers' reports on citations
As can be seen in Fig. 2, the results vary depending on the regression used. The less demanding regression in terms of significance is Negative Binomial, where the threshold is 1250 words (which corresponds to one page written), followed by Poisson regression (1400 words) and GLM regression (2400 words). These results suggest that a report that brings the most value to the publication is a deep report, which naturally requires more time to produce. This is an important result in a context where the deadlines granted to reviewers tend to decrease for some publishers.
Through this paper, we have sought to analyze empirically to what extent peer review improves the quality of publications. To do this, we used data from Publons (#85,093) with a structure adjusted to that of the WoS database (control group of 12,326,712 publications). The proxy used for peer review is the length of review reports expressed in number of words (provided by Publons). On the publications side, the proxy used to measure the quality (interest aroused within the scientific community) of the publications are the citations received. We carried out several regression models, each time putting the citation scores as the explained variable, and the length of the reviewers' reports as the explanatory variable (while adding a certain number of control variables).
In addition to the "classic" results on the importance of Open Access, funding or international collaboration on citations, we have shown the existence of a strong and statistically significant relationship between the length of reviewers' reports and citations received. The longer the referees' reports, the more the citations tend to increase. This result is consistent with our hypothesis formulated in the introduction, which consists in saying that peer review fulfills its function of improving scientific publications. In other words, long referees' reports go hand in hand with the number of modifications / improvements requested, and is a source of quality improvement, which is seen through the citations received.
This result is important from both the point of view of reviewers and that of journal's chiefs-editors. Thus, by affecting the intrinsic quality of publications, peer review work is an integral part of the creation of new knowledge and the development of science. Even if it is not a remunerated activity, it is important that it be more valued at least within the framework of research evaluation exercises, given its positive impact on science.
From the editors' point of view, this result underlines that a good peer review report is generally long (even if we can be short and concise), and therefore can take time. On this point, we analyzed (1) from what length of the reports that the impact on citations becomes significant, and (2) if the impact increases (regression coefficient) as the average number of words increases. The results differ between the types of regression used. The threshold is 1250 words (about one page) for Negative Binomial regression, 1400 words for Poisson regression and 2400 words for GLM regression. These results suggest that a report that brings the most value to the publication is a deep report, which naturally requires more time to produce. This is an important result in a context where the deadlines granted to reviewers tend to decrease for some publishers
Despite the effort to correct the data structure of the Publons database, there are still some limitations that cannot be addressed at this time. This is a base built from the volunteering of researchers to disseminate (promote) their reviewing activities. As a result, the data is not necessarily representative of all the evaluation reports due to this selection bias. Therefore, it would be interesting to carry out this same work with other data. For example, those of journals that practice open peer review. In other words, it would be interesting if we reached the same conclusions (or not) if we used more systematic and random data. This amounts to extracting on the one hand the reports of the publications and on the other hand, the citations received.
Another limitation, which is difficult to overcome, is that this work is based on reports from accepted publications. It would be interesting to include in the analysis the length of reviewers' reports of rejected publications.
Finally, it would be interesting to check whether there is a problem of endogeneity with regard to the size of the reports of reviewers. In other words, long reports are long because the papers are already of good quality. One way to address this issue is to perform author disambiguation to control the length of reports by authors’ notoriety.
Acknowledgement
The present paper is an extended version of the contribution sent to the nineteenth conference of the International Society of Scientometrics and Informetrics (ISSI2023) to be held in Bloomington, Indiana, USA, July 2-5. https://cns-iu.github.io/workshops/2023-07-02_issi/submission-information.html.
Compliance with Ethical Standards
The authors have no relevant financial or non-financial interests to disclose.
- Larivière, V. Croissance des revues savantes: de la connaissance et… du bruit. Acfas https://www.acfas.ca/publications/magazine/2017/12/croissance-revues-savantes-entre-bruit-connaissances (2017).
- Shen, C. & Björk, B.-C. ‘Predatory’ open access: a longitudinal study of article volumes and market characteristics. BMC Med 13, 230 (2015).
- Boukacem-Zeghmouri, C., Rakotoary, S. & Bador, P. La prédation dans le champ de la publication scientifique: un objet de recherche révélateur des mutations de la communication scientifique ouverte. (2020).
- Cobey, K. D. et al. Knowledge and motivations of researchers publishing in presumed predatory journals: a survey. BMJ Open 9, e026516 (2019).
- Siler, K. Demarcating spectrums of predatory publishing: Economic and institutional sources of academic legitimacy. Journal of the Association for Information Science and Technology 71, 1386–1401 (2020).
- Kolata, G. Many Academics Are Eager to Publish in Worthless Journals (Published 2017). The New York Times (2017).
- Frandsen, T. F. Why do researchers decide to publish in questionable journals? A review of the literature. Learned Publishing 32, 57–62 (2019).
- Djuric, D. Penetrating the Omerta of Predatory Publishing: The Romanian Connection. Sci Eng Ethics 21, 183–202 (2015).
- Björk, B.-C. & Solomon, D. Open access versus subscription journals: a comparison of scientific impact. BMC Med 10, 73 (2012).
- Publons. It’s not the size that matters. Publons https://publons.com/blog/its-not-the-size-that-matters/ (2018).
- Kareiva, P., Marvier, M., West, S. & Hornisher, J. Slow-moving journals hinder conservation efforts. Nature 420, 15–15 (2002).
- Cornelius, J. L. Reviewing the review process: Identifying sources of delay. Australas Med J 5, 26–29 (2012).
- Björk, B.-C. & Solomon, D. The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics 7, 914–923 (2013).
- Huisman, J. & Smits, J. Duration and quality of the peer review process: the author’s perspective. Scientometrics 113, 633–650 (2017).
- Bilalli, B., Munir, R. F. & Abelló, A. A framework for assessing the peer review duration of journals: case study in computer science. Scientometrics (2020) doi:10.1007/s11192-020-03742-9.
- Björk, B.-C. Publishing speed and acceptance rates of open access megajournals. Online Information Review ahead-of-print, (2018).
- Azar, O. H. The Slowdown in First-Response Times of Economics Journals: Can It Be Beneficial? Economic Inquiry 45, 179–187 (2007).
- Moizer, P. Publishing in accounting journals: A fair game? Accounting, Organizations and Society 34, 285–304 (2009).
- Toroser D et al. Factors impacting time to acceptance and publication for peer-reviewed publications. Curr Med Res Opin 33, 1183–1189 (2016).
- Chataway, J. & Severin, A. Purposes of peer review: A qualitative study of stakeholder expectations and perceptions. Learned Publishing (2020).
- Drvenica, I., Bravo, G., Vejmelka, L., Dekanski, A. & Nedić, O. Peer Review of Reviewers: The Author’s Perspective. Publications 7, 1 (2019).
- Pranić, S. M., Malički, M., Marušić, S. L., Mehmani, B. & Marušić, A. Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? A cross-sectional study in 12 journals across four research fields. Learned Publishing n/a, 1–11 (2020).
- Pautasso, M. & Schäfer, H. Peer review delay and selectivity in ecology journals. Scientometrics 2, 307–315 (2010).
- Shen, S. et al. Editorial delay and its relation to subsequent citations: the journals Nature, Science and Cell. Scientometrics 105, 1867–1873 (2015).
- Deming, W. E. & Stephan, F. F. On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known. The Annals of Mathematical Statistics 11, 427–444 (1940).
- Deville, J.-C., Sarndal, C.-E. & Sautory, O. Generalized raking procedures in survey sampling. Journal of the American Statistical Association 88, 1013–1021 (1993).
- Maddi, A. & Sapinho, D. Article processing charges, altmetrics and citation impact: Is there an economic rationale? Scientometrics (2022) doi:10.1007/s11192-022-04284-y.
- Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S. & van Raan, A. F. J. Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics 5, 37–47 (2011).
- Radzvilas, M., De Pretis, F., Peden, W., Tortoli, D. & Osimani, B. Incentives for Research Effort: An Evolutionary Model of Publication Markets with Double-Blind and Open Review. Comput Econ (2022) doi:10.1007/s10614-022-10250-w.
- Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B. & Squazzoni, F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat Commun 10, 322 (2019).
There is NO Competing Interest.