The importance of reproducibility in science
The term “reproducibility” refers to the ability to generate similar results each time an experiment is replicated. When a finding cannot be reliably reproduced, we question its validity. The ability to reproduce a study is key to the acceptance of scientific ideas, but despite its importance, many findings are not replicable.
One major contributor to the reproducibility problem is a lack of transparency: incomplete reporting of data and methodological details (e.g. analytical tools, materials). Our Methods Badge involves assessing and providing recommendations on the reporting of materials, study design, and analysis to promote transparency and help overcome irreproducibility by ensuring the details necessary to reproduce the work are included in the manuscript.
Here, we present an example preprint that underwent our evaluation and earned a Methods Badge on the Research Square platform.
Earning a Methods Badge - behind the scenes
The manuscript submitted by the authors for our Methods Badge certification was assessed by our team. Review of the methods reporting indicated that some details regarding the study design (in-lab replication, blinding), materials used in the study (human samples, antibodies, reagents, commercial assays), analysis software and data availability were not included. After the manuscript review, the authors received a report (Figure 1) detailing the revisions needed to receive the Methods badge.
Once the revisions were applied, the revised version of the manuscript was posted on the platform along with the Methods Badge icon, certifying that the manuscript adheres to established reporting standards (Figure 2).
A closer look: antibodies
Our review identified several components of the experiment that required further details. We will focus on the “Antibodies” section of the evaluation to show how our Methods Badge improved the reporting of this item in the manuscript.
As antibodies are a key reagent for several assays used in research, it is important to specify the source to ensure that they can be precisely identified. For commercial antibodies, the methods should indicate the supplier name, catalog number, and lot number, if available. If more than one type of experiment involved antibodies, the methods should clearly state for which application each antibody was used and specify the dilutions. It is also useful to note for which species and application the antibodies have been validated. This can be done by providing a link to the validation statement on the manufacturer’s website or an antibody profile in an online database (e.g., RRID), , citing any relevant publications or data provided in the manuscript. The details of the antibodies used in this example manuscript were summarized in the table shown below (Figure 3). Although many of the relevant antibody details had been provided in the original Table 1, the species and validation information were missing.
Based on the feedback provided from our badge review, the authors revised the table (Figure 4) to include all the necessary details (revisions shown as highlighted text).
How a Methods Badge improves reproducibility
The use of well-characterized and defined reagents is central to reproducibility in research; however, scientific publications often fail to provide complete descriptions of the antibodies used. Failure to demonstrate that antibodies have been validated for use in the experiments and species used in the study makes it difficult for reviewers and readers to assess whether the data are reliable. Similarly, incomplete descriptions of the antibodies make it difficult for other researchers to repeat the experiments accurately. Our Methods Badge review strengthened the reproducibility of the study by ensuring that the antibodies were clearly identified and characterized. A complete and transparent record of the methods ensures that readers and reviewers have better information with which to evaluate the study’s findings.
To earn a Methods Badge and improve the reproducibility of your study, please visit https://www.researchsquare.com/researchers/badges.