Background: A low replication rate has been reported in some scientific areas motivating the creation of resource-intensive collaborations to estimate the replication rate by repeating individual studies. The substantial resources required by these projects limits the number of studies that can be repeated and consequently the generalizability of the findings. We extend the use of a method from Jager and Leek to estimate the false discovery rate (FDR) for 94 journals over a five-year period using p-values from 30,000 abstracts enabling the study of how the FDR varies by journal characteristics.
Results: We find that the empirical FDR is higher for cancer vs. general medicine journals (p = 5.14E-6), especially for those with lower journal impact factors. Conversely, we find no significant evidence of a higher FDR, on average, for Open Access vs. closed access journals (p = 0.256, 95% CI: -0.014, 0.051).
Conclusions: Our results identify areas of research that may need additional scrutiny and support to facilitate replicable science. Given our publicly available R code and data, others can complete a broad assessment of the empirical FDR across other subject areas and characteristics of published research.