Capacity Building in Peer Review: a Student-led Journal Experience

doi:10.21203/rs.3.rs-637876/v1

Download PDF

Research Article

Capacity Building in Peer Review: a Student-led Journal Experience

https://doi.org/10.21203/rs.3.rs-637876/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Introduction: Peer review plays a pivotal role in optimizing articles’ quality. However, in a context of poor methodological publications and unreliable data, it is questionable which strategies to invest in to improve peer reviewing. An excellent start is by enhancing diversity through inclusion of undergraduate students to reviewers. We aim to report the peer reviewing policies, procedures, and practices of a medical student-led journal editorial board in 2020, detailing the challenges and the role of students in building capacity in peer review.

Methods: Through validated online training courses and peer education methodology, the students built capacity regarding the relevance of peer review, its models, structure, and publication process. The journal peer-reviewing was blinded for authors and reviewers, pursuing impartiality and minimization of identification bias. To add standardization to the submission and review processes, guidelines for authors and reviewers were developed, based on journals’ recommendations, and reporting guidelines.

Results: The journal had 254 submitted manuscripts from all five Brazilian geographic regions during the second semester of 2020, a considerable increase compared to the 72 submissions in the previous edition. After reviewing, 50 articles were accepted to the Brazilian Medical Students’ 7th edition, demanding minor or major corrections.

Discussion: Peer review contributes to the construction of content, standing for evidence-based medicine. Besides, it improves ethical, communication, and critical appraisal abilities, also desirable in the academic and professional spheres. Among the benefits and limitations of this medical students' peer review process, there is an interesting strategy to be studied and further promoted.

Scientific Communication

Medical Education

Research Peer Review

Self-Directed Learning

Teaching

Results' communication to researchers is a process that resembles long before journal publications and transitioned through distinct methods. The appraisal by field pairs (i.e., peer-reviewing) is one of the common ways to enable it, if not the dominant. The Committee on Publication Ethics (COPE) asserts that peer review in all its forms plays a pivotal role in ensuring the integrity and liability of scholarly records (1), which led to progress in articles' quality. However, several criticisms have been made for the process of peer-reviewing by itself or related models (2).

On one side, authors expect a rapid, detailed, judicious, and fair assessment. Well-intentioned reviewers go towards articles improvement, even if rejected, aiming to acknowledge the amount of work the authors have done (3). Besides, editors should ensure the liability of the peer-reviewing process and maximization of its quality, enabling proper scientific communication and dissemination of research findings. In turn, those who offer themselves to review a manuscript await evidence that may be relevant to the journal’s scope and scientific field of concern (4).

Nonetheless, the standard peer review process is not exempt from flaws. The current knowledge dissemination model is characterized by the presentation of research findings which go through the inspection of journals that define priority themes, formats, and quality criteria. In this regard, journals condition what is published, by whom, when, and in which vehicles (2, 5, 6). Due to this structure, such a model does not privilege controversy, public debate, neither is equitable nor inclusive, as peer review is carried out by only a proportion of researchers. Frequently, two to three reviewers work on voluntarily, in a short period, to evaluate written pieces with considerable information gaps and of low reproducibility (5–7). Apart from those assets, scientists should be rewarded and acknowledged by peer-reviewing activities, which have not been occurring over time (8).

With this rationale, it is questionable which strategies can be adopted in peer reviewing to improve a system, enhancing methods’ completeness, adequate reporting, and data reliability. An excellent manner to start this process is by enhancing diversity through the inclusion of graduate students to reviewers or promoting mentored-based undergraduate peer-reviewing activities in non-student-led and student-led journals. Giving freedom to appraisal, whoever is under this specific duty, is of high priority. In student-led journals, undergraduate students should receive training and mentoring by experienced researchers to realize the publishing process at the beginning of their careers (9).

The Brazilian Medical Students Journal (BMS) is a free, biannual, scientific journal published by the International Federation of Medical Students’ Associations of Brazil (IFMSA Brazil), a nonprofit and nongovernmental organization (10, 11). When created in 2016, BMS aimed to share projects, activities, and experiences written by IFMSA Brazil members. Nevertheless, the journal increased in proportions, and currently, BMS aims to propagate scientific evidence through its editions and advocates for access to research and open science. Its thematic scope includes public health and inequities, improvements in patient care, medical education development, knowledge mobility, and best research practices (11).

Intending to manage the journal, BMS has its editorial board composed of the Scientific Team members, who nationally run and administer IFMSA Brazil Publication, Research, and Community Services unit (10). The team is annually renewed, from December of one year until November of the next. In 2020, the new BMS editorial board restructured its peer review system to improve its reliability and transparency. Changing the process underlined the indispensability of promoting rapid dissemination of research findings to advance the medical field (12). Therefore, this article aims to report the peer reviewing policies, procedures, and practices of a medical student-led journal editorial board in 2020, detailing the challenges and the role of students in building capacity in peer reviewing.

Due to the design of our study and non-inclusion of sensitive data, ethics approval was not required. Two assistants of the Scientific Team attended freely available online training courses provided by Elsevier and Publons on peer reviewing to raise capacity on the matter (13, 14). Through a peer education methodology, these two members were responsible for capacitating the others in a webinar format. The learning objectives to be achieved were: understand the definition of peer review and its relevance to the scientific community; differentiate peer review models; comprehend traditional publication stages (editorial screening, reviewing, editor assessment of reviewers comments, the final decision on papers); be acknowledged with the role of editors and reviewers; discern how to write a helpful review report (1, 15).

Based on these aims, the BMS editorial board made some decisions. On the one hand, the team agreed on adopting a peer review format blinded both to reviewers and authors for the BMS journal, minimizing identification bias of authors, reducing the incidence of nepotism and both institutional and geographic biases, and reinforcing the need to provide impartial review and recommend a decision on the manuscript (16, 17). On the other hand, it was of utmost importance to release a pilot version of a guide for authors and for reviewers, adding clarity to the submission and review processes for prospective authors. The team divided itself into two small working groups of six members: one responsible for drafting the guidelines for authors and another document for reviewers. Each group had five participants and one coordinator to supervise the process. Afterwards, both groups shared their first drafts with the entire team to receive critical inputs and promote a collective decision-making process.

Concerning the instructions for authors, assistants searched for guides from different journals (18–20). Since BMS publishes articles in Portuguese and English, guidelines have editions in both languages. The document incorporates directions on the formatting of submission files and what each manuscript part should mention. In addition, the guide elucidates which ethical standards authors should adhere to and how to use the Contributes Roles Taxonomy (CRediT) instrument to give credit and recognition to the contribution roles of each author (21).

As for the instructions for reviewers, criteria for analyzing articles were dependent on study design. For this edition, the team used consolidated report guidelines, such as the PRISMA checklist for systematic reviews, CARE checklist for case reports, MacDermid et al. (2009) for original articles with specific items for qualitative and quantitative studies (22–24). For experience reports, the editorial board created its own checklist. The reporting guidelines were handled as guidance instruments to help reviewers consolidate their assessment, aligning it with each article section and originality. The instructions for reviewers also provide instructions for making recommendations for the manuscript and appropriate review tone. Additionally, the guide has a protocol when plagiarism is suspected (1).

After closing submissions for BMS 7th edition, the editor-in-chief screened 254 received manuscripts. The editor checked alignment to the BMS scope and compliance with the guide for authors. At this stage, there were 31 rejected submissions due to disagreement in fulfilling these criteria. In sequence, the editor-in-chief has randomized 223 eligible manuscripts by removing names and affiliations, then randomly assigned each paper to two reviewers from the Scientific Team, who independently appraised submissions. An overview of the peer reviewing process is illustrated in Fig. 1.

Reviewers were unaware of who their partners were, avoiding conflicts of interest. Provided that reviewers had any disclosures or conflicts of interest, they had to contact the editor-in-chief through email to invite another editorial board member. Reviewers had an average of three weeks to provide their comments about articles to both authors and editors. Subsequently, the editor-in-chief assessed the reviews, deciding which papers to accept with no revisions, accept with major or minor revisions, or reject. Independent of results, all comments to authors were delivered by personal communication through the journal’s official email address. In total, there were 50 accepted articles to the 7th edition, demanding minor or major corrections. After sending the revised manuscript to original reviewers, the Scientific Team edited the content of the manuscripts. Lastly, the editor sent the final editions to content designers. Overall, BMS had a considerable increase in the number of submitted articles. Whereas the sixth edition admitted 72 submissions, the seventh received 254 manuscripts from all five Brazilian geographic regions respectively in the second semester of 2019 and 2020.

The highlights of this experience were the possibility of developing an internal journal organization to enable greater standardization of the peer review process. Furthermore, being a reviewer made it possible to improve critical sense and allowed growth in the research field, considering it was necessary to analyze articles according to their quality. As learning points, reviewers have demonstrated an initial difficulty in differentiating what belongs to the comments for authors and the editor. Besides, the editorial board has used reporting guidelines as an instrument to assess the quality of the submitted articles, which is not their primary objective. This process was vital for the team to comprehend that a reporting guideline should be a structured tool for researchers to use while writing manuscripts, being more helpful for authors than for reviewers (25). Also, the editorial board was not aware of other reporting guidelines that could have been, for example, for experience reports such as the Standards for reporting qualitative research (SRQR) and Consolidated criteria for reporting qualitative research (COREQ) (26, 27), and other study designs. Based on these inputs, the editorial board decided to direct authors to these reporting guidelines explicitly and indicate more checklists to enhance articles' submission.

Well done peer review plays a pivotal role in optimizing the quality of studies inserted in the scientific environment. Even though this process cannot address steps prior to the article submission, a crucial stage to improve reproducibility and quality of research outputs, peer review contributes to shaping how research findings will be disseminated and understood by readers (5). The system applied in BMS illustrates how peer review contributes to the construction of content. Specifically to the biomedical field, the process stands for evidence-based medicine by fighting against misinformation, bias, and inappropriate content (28).

Although not all the reviewers adequately contribute for content improvements, the greatest virtue of peer reviewing is combining both objective and subjective features. By making their comments on manuscripts, reviewers give researchers a different perspective on the structure and message of the analyzed text, and improve technical assets of the paper’s methods and data presentation (29).

It is worth emphasizing that the quality of peer reviewing influences the reputations of journals. Thus, being a reviewer requires several responsibilities towards authors, editors, and readers. According to the Council of Scientific Editors, reviewers must give constructive and unbiased comments to authors, honoring the process confidentiality; provide a critical evaluation of the article to editors, pointing out improvement recommendations and notifying ethical concerns or any conflicts of interest; and ensure the quality of publications for readers, especially concerning research clarity and reproducibility (30). In addition to scientific context, these skills are helpful and expected in academic and professional spheres.

Regarding our report, reviewers could develop communication skills, including formality and cordiality, when elaborating comments to authors (31). Moreover, improving students' feedback quality potentializes their involvement in the medical curriculum, enhancing self-directed learning, leadership ability, and intellectual content (32–34). Such student proactiveness drove BMS reviewing and editing, beginning with the initiative to build the peer review model by self-teaching in the topic and required technological resources. Furthermore, by incorporating students in the center of the publishing process, preconceptions about who can participate in knowledge sharing are challenged. All parties, including students, should be allowed to extend their capabilities to use, share and create knowledge, whereas journals need to incorporate mechanisms to promote diversity and equitable involvement among academics and citizen researchers.

Ethical responsibility is another high value characteristic developed in peer review since the privacy of information is constitutional on human-related research, and sometimes deviated by authors, editors, or reviewers either intentionally or unintentionally. Examples of this commitment are compliance with journal norms of confidentiality, notification of eventual conflicts of interest or ethical concerns, and management of assignments and deadlines (30).

Assuming a reviewer’s role enabled students to exercise critical appraisal of submitted articles and to value methodological rigor and clarity of research, aspects of much importance in the scientific field owing to its constant updates. As BMS reviewers detailed their recommendations, they built teamwork with the editor-in-chief, prioritizing group decision-making, unlike the typical hierarchical flowchart of peer reviews in which decision-making is vertical from the editor-in-chief and associated editors to reviewers and editorial board, rather than horizontally (35).

Creating a student-led peer review system in which information is freely available to the general public is challenging. It is unusual to find opportunities that stimulate medical students to peer-review during the first years in Academia. Some dilemmas to be encountered in these situations are student adhesion and permanence throughout Academia (36). Beyond these points, students face a competitive need to publish more to receive achievement in the labor market. This pressured environment comes as a result of early exposure to the perish-or-publish system. Additionally, students’ work is usually underappreciated compared to mainstream publishers and senior researchers before being independent and trained researchers (36, 37).

This study has some limitations. First, being a student-led journal simultaneously composes a fortress and limitation of this paper, considering that the students are still learning, and prone to make mistakes while reviewing due to lack of guidance. For example, unawareness of certain reporting guidelines, resources and collaborations, such as the Equator Network (38) demonstrate ongoing consolidation and understanding of how the publishing process works. Second, confusion on whether reporting guidelines should be conditioners of an article’s appraisal was also a recurrent concern among reviewers. Future research directions are to comprehend if specific processes, such as the pairing of senior reviewers or journal editors with young researchers, increase students' knowledge, skills, and confidence in peer review.

Developing scientific competencies and abilities during graduation encompasses several acquisitions that exceed technical and academic knowledge, with peer reviewing as a skill that interests scientific development. Understanding this process as a fundamental piece of scientific publication is the first step to develop a better awareness of journal demands. Undergraduate students can progress in acquiring these skills to benefit current and future researchers and reviewers. Hence, the experience of making an academic journal led by students presented a new initiative, leading to the development of technical-scientific and critical reading skills for reviewers.

Funding support:

There was no sponsor or funding for this research.

Declaration of Interests:

The authors have no conflicts or financial disclosures to declare.

Authors Contribution:

All authors contributed to the conceptualization and A.T.Z.Y and L.N.C for the study design. A.T.Z.Y., L.N.C., S.S.D.E.M, T.S.S.N, B.O.L., M.C.C., R.M.G.G.A., A.H.B., L.L.P.A.R. and L.H. contributed to data collection, analysis, interpretation and writing the original draft. S.S.D.E.M was responsible for data visualization. Reviewing and supervision was made of L.H.. All authors agreed with the final version of the manuscript, and are held accountable of all aspects of this study.

Acknowledgements:

We would like to thank all members of previous editorial boards of the BMS journal who believe in science and the needs to improve and change every day.

Ethical guidelines for peer reviewers (English) [Internet]. Committee on Publication Ethics; 2013 Mar [cited 2021 Apr 9]. Available from: https://publicationethics.org/node/19886
Feliciani T, Luo J, Ma L, Lucas P, Squazzoni F, Marušić A, et al. A scoping review of simulation models of peer review. Scientometrics. 2019;121(1):555–94.
Nicholas D, Watkinson A, Jamali HR, Herman E, Tenopir C, Volentine R, et al. Peer review: still king in the digital age. Learn Publ. 2015 Jan 1;28(1):15–21.
Barata RB, Barata RB. Challenges for publishing Brazilian scientific health journals. Ciênc Amp Saúde Coletiva. 2019 Mar;24(3):929–39.
Glonti K, Cauchi D, Cobo E, Boutron I, Moher D, Hren D. A scoping review protocol on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. BMJ Open [Internet]. 2017 Oct 22 [cited 2021 May 3];7(10). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5665269/
Kovanis M, Porcher R, Ravaud P, Trinquart L. Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data. Scientometrics. 2016;106:695–715.
Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial Peer Reviewers’ Recommendations at a General Medical Journal: Are They Reliable and Do Editors Care? PLoS ONE [Internet]. 2010 Apr 8 [cited 2021 May 3];5(4). Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2851650/
Moher D, Bouter L, Kleinert S, Glasziou P, Sham MH, Barbour V, et al. The Hong Kong Principles for assessing researchers: Fostering research integrity. PLOS Biol. 2020 Jul 16;18(7):e3000737.
Bispo MDS, Bispo MDS. SE PUBLICAR É PRECISO, AVALIAR TAMBÉM É! Rev Adm Empres. 2018 Aug;58(4):438–42.
Lacerda PNM, Souza LR de, Endo RM, Nazima MTST. Extensão universitária na atualidade: a IFMSA Brazil como foco. Rev Med. 2015 Oct 29;94(2):81–6.
About the Journal | Brazilian Medical Students [Internet]. [cited 2021 Apr 9]. Available from: https://bms.ifmsabrazil.org/index.php/bms/about
Manchikanti L, Kaye AD, Boswell MV, Hirsch JA. Medical journal peer review: process and bias. Pain Physician. 2015 Feb;18(1):E1–14.
Elsevier Researcher Academy - Certified Peer Reviewer Course [Internet]. [cited 2021 Apr 9]. Available from: https://researcheracademy.elsevier.com/navigating-peer-review/certified-peer-reviewer-course
Learn to peer review with confidence [Internet]. Publons. [cited 2021 Apr 9]. Available from: http://publons.com/community/academy
Bravo G, Grimaldo F, López-Iñesta E, Mehmani B, Squazzoni F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat Commun. 2019 Dec;10(1):322.
Cox AR, Montgomerie R. The cases for and against double-blind reviews. PeerJ [Internet]. 2019 Apr 2 [cited 2021 May 3];7. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6450368/
Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci. 2017 Nov 28;114(48):12708–13.
Submissions | International Journal of Medical Students [Internet]. [cited 2021 Apr 9]. Available from: https://ijms.info/IJMS/about/submissions
Instructions for authors | The Brazilian Journal of Infectious Diseases [Internet]. [cited 2021 Apr 9]. Available from: https://www.bjid.org.br/en-guia-autores
Guide for authors - European Journal of Pharmacology - ISSN 0014-2999 [Internet]. [cited 2021 Apr 9]. Available from: https://www.elsevier.com/journals/european-journal-of-pharmacology/0014-2999/guide-for-authors
Allen L, O’Connell A, Kiermer V. How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship. Learn Publ. 2019;32(1):71–4.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Med. 2009 Jul 21;6(7):e1000097.
Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D, et al. The CARE guidelines: consensus-based clinical case reporting guideline development. Case Rep. 2013 Oct 23;2013:bcr2013201554.
MacDermid JC, Walton DM, Law M. Critical appraisal of research evidence for its validity and usefulness. Hand Clin. 2009 Feb;25(1):29–42, v.
What is a reporting guideline? | The EQUATOR Network [Internet]. [cited 2021 May 5]. Available from: https://www.equator-network.org/about-us/what-is-a-reporting-guideline/
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med J Assoc Am Med Coll. 2014 Sep;89(9):1245–51.
Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care J Int Soc Qual Health Care. 2007 Dec;19(6):349–57.
Rodrigues D, Dias CV, Heleno B. Como responder a dúvidas clínicas. Rev Port Med Geral E Fam. 2019 May 14;35(2):155–66.
Cobo E, Selva-O’Callagham A, Ribera J-M, Cardellach F, Dominguez R, Vilardell M. Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial. Scherer R, editor. PLoS ONE. 2007 Mar 28;2(3):e332.
2.3 Reviewer Roles and Responsibilities [Internet]. Council of Science Editors. [cited 2021 Apr 9]. Available from: https://www.councilscienceeditors.org/resource-library/editorial-policies/white-paper-on-publication-ethics/2-3-reviewer-roles-and-responsibilities/
Rachwal CM, Langer T, Trainor BP, Bell MA, Browning DM, Meyer EC. Navigating Communication Challenges in Clinical Practice: A New Approach to Team Education. Crit Care Nurse. 2018 Dec;38(6):15–22.
Bozung BR, Houston K, III JFL, Jordan SG, Fordham LA, Dallaghan GB. Student-led peer review of an online teaching file: perspectives after 2 years. Med Educ Online. 2021 Jan 1;26(1):1843356.
Camarata T, Slieman TA. Improving Student Feedback Quality: A Simple Model Using Peer Review and Feedback Rubrics. J Med Educ Curric Dev. 2020 Jan;7:238212052093660.
Ridge BL, Lavigne AL. Improving instructional practice through peer observation and feedback: A review of the literature. Educ Policy Anal Arch. 2020 Apr 13;28(0):61.
Peer Review Process [Internet]. ASHA Journals Academy. [cited 2021 May 3]. Available from: https://academy.pubs.asha.org/prep-the-asha-journals-peer-review-excellence-program/peer-review-process/
Al-Busaidi IS, Alamri Y. Peer review policies in medical student journals. Postgrad Med J. 2018 Jun;94(1112):362–3.
Nazha B, Salloum RH, Fahed AC, Nabulsi M. Students’ Perceptions of Peer-Organized Extra-Curricular Research Course during Medical School: A Qualitative Study. Sapino A, editor. PLOS ONE. 2015 Mar 12;10(3):e0119375.
About us | The EQUATOR Network [Internet]. [cited 2021 May 5]. Available from: https://www.equator-network.org/about-us/

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Capacity Building in Peer Review: a Student-led Journal Experience

Status:

Version 1

Abstract

Figures

Introduction

Methods

Results

Discussion

Conclusion

Declarations

Funding support:

Declaration of Interests:

Authors Contribution:

Acknowledgements:

References

Status:

Version 1