We obtained 4,226 study results from 619 eligible Cochrane meta-analyses, presented in 148 reviews (Fig. 1). Of these meta-analyses, 114 (18.4%) were of “high” quality of evidence, and 505 (81.6%) were of “moderate” quality of evidence. They only included RCTs. Two hundred eighty-two (6.7%) results were reported in open-access journals.
The application of exclusion criteria resulted in a sample of 2,827 study results usable for the pairwise comparisons of the deviation by JIF (no vs. any JIF). Of those, 368 (13%) were published in journals without an assigned JIF in the respective publication year. The difference in the relative deviation was 1.04 for no JIF compared to 0.98 for any JIF (95% CI: −0.04 to 0.15) and not significant (P = 0.271).
For the group of study results from journals with an associated JIF (N = 2,459), the pairwise comparisons of the relative deviation by JIF quartiles are shown in Fig. 2. The mean relative deviation varied by the magnitude of the JIF, ranging between 1.11 in the lowest quartile to 0.80 in the top quartile (95% CI: 0.21 to 0.40, P < 0.001). The mean relative deviation was higher in studies published in open-access journals compared to studies published in non-open-access journals (1.09 compared to 0.97, 95% CI: −0.02 to 0.25, P = 0.082).
The results of the correlation analyses confirmed the hypothesised relation between the JIF and the outcome variable, the log-transformed relative deviation (r = − 0.21, 95% CI: −0.24 to − 0.17, P < 0.001, see Table 1).
Table 1
Correlation matrix including the outcome variable and all covariates
|
log(rel. dev.)
|
Adjusted JIF
|
Study weight
|
Risk of bias
|
Open access
|
log(rel. dev.)
|
1.00
|
|
|
|
|
Adjusted JIF
|
−0.21
[− 0.24, − 0.17]
|
1.00
|
|
|
|
Study weight
|
−0.17
[− 0.20, − 0.13]
|
0.17
[0.13, 0.21]
|
1.00
|
|
|
Risk of bias
|
0.09
[0.50, 0.13]
|
−0.18
[− 0.22, − 0.14]
|
−0.10
[− 0.14, − 0.06]
|
1.00
|
|
Open access
|
0.03
[− 0.01, 0.07]
|
−0.14
[− 0.17, − 0.10]
|
−0.02
[− 0.06, 0.02]
|
−0.08
[− 0.11, − 0.04]
|
1.00
|
Pearson product-moment correlation coefficients with corresponding 95% CI
|
The weight of a point estimate in its respective meta-analysis also correlated negatively and significantly with the outcome variable (r = − 0.17, 95% CI: −0.20 to − 0.13, P < 0.001) and correlated positively with the adjusted JIF (r = 0.17, 95% CI: 0.13 to 0.21, P < 0.001). There was also a significant correlation between the methodical quality of a study and the impact factor of the publishing journal: the higher the aggregated risk of bias, the less influential the journal (r = − 0.18, 95% CI: −0.22 to − 0.14, P < 0.001). The correlation between the aggregated risk of bias and the outcome variable was less pronounced, yet significant (r = 0.09, 95% CI: 0.05 to 0.13, P < 0.001). Open access was not associated with the outcome variable (r = 0.03, 95% CI: −0.01 to 0.07, P = 0.196).
The results of the regression analysis show that the adjusted JIF was a notable and significant predictor of the log relative deviation (beta = − 0.174, 95% CI: −0.214 to − 0.134, P < 0.001) when adjusting for the covariates (see Table 2).
Table 2
Results from linear regression using the log-transformed relative deviation as outcome
|
|
Simple linear regression
|
|
Multiple linear regression
|
|
|
Coefficient (b) [95% CI]
|
Std. coefficient (beta) [95% CI]
|
P-value
|
|
Coefficient (b) [95% CI]
|
Std. coefficient (beta) [95% CI]
|
P-value
|
(Intercept)
|
|
−0.339
[− 0.395, − 0.282]
|
|
< 0.001
|
|
−0.288
[− 0.392, − 0.184]
|
|
< 0.001
|
Adjusted JIF
|
|
−0.067
[− 0.080, − 0.054]
|
−0.206
[− 0.244, − 0.167]
|
< 0.001
|
|
−0.057
[− 0.070, − 0.044]
|
−0.174
[− 0.214, − 0.134]
|
< 0.001
|
Study weight
|
|
|
|
|
|
−0.014
[− 0.018, − 0.010]
|
−0.131
[− 0.170, − 0.091]
|
< 0.001
|
Risk of bias
|
|
|
|
|
|
0.333
[0.069, 0.597]
|
0.050
[0.010, 0.089]
|
= 0.013
|
Open access
|
|
|
|
|
|
−0.019
[− 0.160, 0.197]
|
−0.004
[− 0.035, 0.043]
|
= 0.838
|
|
|
P < 0.001, adjusted r² = 0.04
|
|
P < 0.001, adjusted r² = 0.061
|
The study weight also affects the outcome (beta = − 0.131, 95% CI: −0.170 to − 0.091, P < 0.001). The interaction between the JIF and the study weight in the multiple linear regression model was weak and not significant (b = 0.001, 95% CI: -0.000 to 0.002, P = 0.202). Methodical quality contributed weakly, yet significantly (beta = 0.050, 95% CI: 0.010 to 0.089, P = 0.013) to the overall regression model (adjusted r² = 0.061, P < 0.001).