The List of Questionable Journals: Is It Trustworthy?

doi:10.21203/rs.3.rs-3105953/v1

Download PDF

Research Article

The List of Questionable Journals: Is It Trustworthy?

https://doi.org/10.21203/rs.3.rs-3105953/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

The list of questionable journals serves to stratify academic journals, distinguishing between good and bad journals, thereby providing a basis for academic information filtering and research evaluation. The formulation of such lists has been a topic of controversy in the literature, with limited attention paid to whether the criteria used to stratify journals in these lists can be trusted. In this study, we take the example of the warning journal list published by the National Science Library of the Chinese Academy of Sciences and categorize the journals based on their warning levels and warning years. Then, we compare them to determine if there are differences in key academic indicators, thereby analyzing whether the journal lists effectively identify academic journals with quality risks. Our study found that the patterns of differences in key academic indicators among different journal stratums were not consistent, and the same inconsistency was observed during the mobility between stratums. Furthermore, citation metrics such as Journal Impact Factor and Journal Citation Indicator may not have been considered as criteria for stratifying journals in the warning journal list, but this lack of detailed explanation from the creators. This indicates that the list of questionable journals needs to be more scientifically evaluated, and there is a greater need for sharing detailed criteria and data. Our study also provides recommendations for institutions in the future when formulating lists of questionable journals.

Library Science

Questionable journals

journal list

research policy

research evaluation

accountability in research

scholarly publishing

The list of questionable journals is considered an effective method of research evaluation by universities and research institutions (Nature 2018). It serves as a filter for research evaluation agencies to quickly identify low-quality publications. Moreover, it provides guidelines for researchers to shape their publishing behavior and helps them identify low-quality scholarly information. However, the list of questionable journals is highly controversial. Different developers of such lists employ varying criteria, resulting in the inclusion of different scholarly journals whose accuracy has been questioned (Strinzel et al. 2019; Tang and Jia, 2022).

Previous studies have concluded that 28 criteria of the list of questionable journals introduced by Cabell need clarification and revision, and 39 should be removed (Teixeira da Silva et al. 2023). Additionally, Beall’s list of predatory journals has also faced criticism due to its vague criteria (Teixeira da Silva and Kimotho, 2021). Indeed, there appears to be a gray area between “good” and “bad” (Dunleavy, 2022), which casts doubt on the credibility of the list of questionable journals (Teixeira da Silva and Tsigaris, 2018). In contrast, some scholars have defended the list of questionable journals and proposed ways to improve the criteria of the lists (Frandsen, 2019; Nelhans and Bodin, 2020; You et al. 2022).

Despite the criticism, the list of questionable journals continues to be widely used in evaluation practices. In December 2020, the National Science Library of the Chinese Academy of Sciences (NSLC) published a list containing 65 international scientific journals, all of which are indexed by the Web of Science (WoS). These journals were considered potentially lacking academic rigor (Zhang et al. 2022). Since then, more than 40 Chinese universities and institutions have promptly adopted the list as a basis for their research evaluations (Tang and Jia, 2022). Researchers who publish in the list of questionable journals are not eligible for material or honorary awards. However, the release of this list, like other journals, has sparked considerable controversy, leading Multidisciplinary Digital Publishing Institute (MDPI) to issue a statement emphasizing their strict review criteria and the high recognition of their journals in the academic community (MDPI 2022). They also assure readers that they will continue to communicate with the NSLC and remove their journals from the list as soon as possible. After analyzing the two lists, we discovered that the 2021 edition had a significant decrease in the number of MDPI-affiliated journals compared to the previous year. Specifically, the count decreased from 22 to seven.

We have discovered that there is scarce previous research extensively discussing whether the stratification or mobility in the list of questionable journals was sound. In simpler terms, the fundamental question arises: can we truly place our trust in the criteria utilized to assess the inclusion or exclusion of scholarly journals from these lists? If a list of questionable journals cannot guarantee credibility and trustworthiness, it may give rise to issues of stigmatization and could corrupt the symbolic capital of academic journals (Teixeira da Silva and Kimotho 2021). The very trustworthiness of these lists becomes a subject of doubt and scrutiny. By addressing this matter, we can provide global guidance for academic review practices and, more significantly, establish a more trustworthy and accountable system for scholarly evaluation. Consequently, we have chosen to focus our study on the list of questionable journals (called the warning journal list) published by the NSLC (2020 and 2021 editions) due to its significant representation and values (Brainard 2023).

The warning journal list developed by the NSLC was first published in 2020, and it is updated annually thereafter (although the release of the 2022 edition was suspended for unknown reasons). Considering the continuity of time, the focus will be on the 2020 and 2021 editions. The NSLC classifies questionable journals into three warning levels: high-level, middle-level, and low-level, with their risk indices decreasing in that order. With each annual update, the NSLC introduces three additional categories for the warning journal list. The first category is the retained group, which includes journals listed in both the 2020 and 2021 editions. The second category is the added group, which includes journals not listed in the 2021 version but included in the 2021 editions. The third category is the excluded group, which includes journals not listed in the 2021 version but included in the 2020 editions. By distinguishing these groups, it will assist us in this research.

Theoretical Frameworks and Research Questions

Stratification of Academic Journals

Academic publishing serves as a means for researchers to establish priority and validate their findings, while also functioning as an incentive within the academic community (Merton 1957). The publication of articles in reputable academic journals not only enhances researchers’ symbolic capital but also provides them with valuable economic resources (Beckman 2019; Carpenter, Cone, and Sarli 2014; Elliott 2013). This system, in conjunction with research evaluation mechanisms, has established a hierarchical framework. Researchers who publish more articles in highly prestigious journals gain greater recognition and may even attain esteemed academic status, while others may be regarded as having “lower” or “no” reputation (Bornmann and Williams 2017). Conversely, when highly respected researchers publish in academic journals, it elevates the journals' standing, leading to internal differentiation (Blanford 2016; Onodera and Yoshikane 2014). One such measure of journal stratification is the impact factor. For instance, the Web of Science (WoS) categorizes academic journals into four tiers based on their impact factor, with those in the top 25% considered of high quality and repute (Clarivate 2022). However, due to significant criticism of the impact factor, there has been an emergence of alternative indicators for journal stratification, such as the JIF index introduced by WoS and the CiteScore index introduced by Scopus (Chapman et al. 2019). In order to further discern the quality and reputation of academic journals, some research institutions have implemented whitelists or ranking systems (Soteropulos and Poore 2021). Drawing on theories of social stratification (Lambert and Griffiths, 2018, p. 36), the stratification of academic journals relies on distinct criteria to establish clear hierarchies. However, in the absence of these evident disparities, making definitive distinctions becomes challenging. Based on this, we propose the following research questions (RQ):

RQ1: Are there significant differences in key academic indicators between the low-warning-level group and the middle-warning-level group?

RQ2: Are there significant differences in key academic indicators between the low-warning-level group and the high-warning-level group?

RQ3: Are there significant differences in key academic indicators between the middle-warning-level group and the high-warning-level group?

Stratum Mobility of Academic Journals

Once social strata are formed, mobility within these strata begins (Sorokin 1927). As Schumpeter (1955) once said, stratum may be conceived of as conveyances that constantly carry different ‘passengers’ without changing their shape. The creation of academic journal lists has led to the division of academic journals into various strata. However, the structure of these strata is not fixed. Just like the warning journal list established by the NSLC, some journals may transition from being high-risk to becoming normal journals with no risk, while others may directly shift from non-risk to high-risk categories. In society, at times, the boundaries between strata are quite clear (Giddens 1975 p.106). For instance, when an individual’s total income reaches a certain threshold, they can ascend to the middle stratum (Madrueño-Aguilar 2017). Most academic journal lists, whether inclusion or exclusion lists, follow similar principles. For example, despite the criticism of the JIF (Nair and Adetayo 2018), when the JIF of an academic journal reaches the top 25% within its field, it is regarded as entering the high-impact stratum (Miranda and Garcia-Carpintero 2019). If it fails to meet the criteria, it is regarded as slipping down to the non-high-impact stratum. The advantage of clear boundaries is that it visualizes stratum mobility (Giddens 1975 p.106). Academic journals within a particular stratum can explicitly know the criteria they need to meet to move up to a higher stratum or slide down to a lower stratum. Therefore, based on this, we propose the following research questions:

RQ4: Are there significant differences in key academic indicators between the added group before (2020) and after (2021) inclusion?

RQ5: Are there significant differences in key academic indicators within the retained group over two consecutive years (2020 and 2021)?

RQ6: Are there significant differences in key academic indicators within the excluded group before (2020) and after (2021) exclusion?

Sample Selection

We have chosen the 2020 and 2021 editions of the warning journal list published by the NSLC as our main sample. The 2021 edition includes 35 journals, while the 2020 edition covers 65 journals. After removing 3 samples with missing data, we grouped them based on two criteria. Firstly, we classified them according to their warning levels. There are 28 low-level warning journals (2020) and 15 low-level warning journals (2021); 28 middle-level warning journals (2020) and 15 middle-level warning journals (2021); and 6 high-level warning journals (2020) and 5 high-level warning journals (2021). Secondly, we categorized them based on their flow status. There are 26 journals in the inclusion group, 9 journals in the retention group, and 53 journals in the exclusion group.

Among the selected samples, there are 5 journals in the field of Materials Science, 2 journals in GeoSciences, 14 journals in Engineering, 5 journals in Chemistry, 2 journals in Environment Science and Ecology, 3 journals in Computer Science, 1 journal in Agricultural and Forestry Science, 13 journals in Biology, 7 journals in Mathematics, 44 journals in Medicine, 1 journal in Multidisciplinary Science (see Fig. 1).

The Key Academic Indicators

The NSLC exclusively provides limited information about the journal lists it publishes, including the journal name, discipline, and warning level. No further details are available regarding individual journals. However, in late 2020, the library released a list that outlined seven criteria utilized for journal selection. These criteria include the number of articles in the journal, degree of internationalization, rejection rate, article processing charges (APCs), journal citation success index, self-citation rate, and retraction information (We have chosen to quantify this criterion using the number of retractions, and the number of retractions by Chinese authors).

The criteria were initially published in Chinese, but an English version was also made available alongside the latest 2021 edition (NSLC, 2021). Although the criteria remained unchanged from 2020, there are slight disparities between the original Chinese version and its English translation. Notably, the Chinese version includes the criterion of “degree of internationalization,” while the English version omits it. Additionally, the Chinese version describes the “number of articles in the journal” as the “growth rate of productivity” in the English version. Moreover, the English version explicitly mentions “paper mill,” whereas the Chinese version does not.

Since the rejection rate of academic journals is usually not publicly available, we excluded this criterion as a key academic indicator for subsequent analysis. Furthermore, the NSLC has not disclosed the Journal Citation Indicator (JCI) for the warning journals, and this information is unavailable to us. As a result, we have no choice but to consider using the JCI as a substitute for this key academic indicator. The reason is that both indicators serve as supplements and enhancements to the JIF (Torres-Salinas, Valderrama-Baca, and Arroyo-Machado 2022). To embark upon a profound exploration of the credibility surrounding the stratification and the stratum mobility within the warning journal list, we have embarked upon an extensive and meticulous investigation. This endeavor encompasses an all-encompassing scrutiny, with due regard given to an antecedent scholarly work (Zhang et al, 2022), augmenting the rigor of our analysis. Furthermore, we have incorporated supplementary key academic indicators, including JIF, article influence, number of open access (OA) articles, and the number of retractions by Chinese authors. Detailed information regarding the data sources and all indicators can be found in Table 1.

Table 1

Selection of key academic indicators, data sources, and their explanation
The key academic indicators	Explanation	Data sources
Number of articles in the journal (A1)	The number of citable articles published each year	InCites
Degree of internationalization (A2)	Percentage of publications that have international co-authors each year	InCites
APCs (A3)	The number of OA articles published by an academic journal each year multiplied by the journal’s APCs (converted to USD based on live exchange rates)	DOAJ and journals’ website
JCI (A4)	The average category-normalized citation impact for papers published in the prior three-year period	InCites
Self-citation rate (A5)	Calculated by dividing an academic journal’s self-citations by its total citations	InCites
Number of retractions (A6)	The number of articles retracted by an academic journal each year	Retraction Watch
JIF (A7)	All citations to the journal in the current Journal Citation Reports year to items published in the previous two years, divided by the total number of scholarly items published in the journal in the previous two years.	InCites
Article influence (A8)	It is calculated by multiplying the Eigenfactor by 0.01 and dividing by the number of articles in the journal, normalized as a fraction of all articles in all publications.	Incites
Number of OA articles (A9)	The number of OA articles published by an academic journal each year	InCites
Number of retractions by Chinese authors (A10)	The number of articles retracted (including those with Chinese authors) each year by an academic journal	Retraction Watch

Data Analysis

Firstly, we utilized Prism 10.0 to conduct the D’Agostino-Pearson test in order to examine the adherence of key academic indicators to a normal distribution. The D’Agostino-Pearson test assumes, under the null hypothesis, that all values are drawn from a population that follows a Gaussian distribution. If the obtained p-value is less than 0.05, we reject this hypothesis, indicating a departure from normality in the data.

Secondly, to answer RQ1, RQ2, and RQ3, we stratified our samples into three distinct groups based on their warning levels and proceeded to analyze potential statistically significant differences among these groups. For key academic indicators that satisfied the assumptions of normality and exhibited homogeneity of variances, we performed a parameter test using the Brown-Forsythe test within the framework of one-way analysis of variance (ANOVA). Subsequently, we reported both the F-value and the significance level to provide a comprehensive understanding of the results. Conversely, for key academic indicators that deviated from a normal distribution, we employed the nonparametric Kruskal-Wallis test in one-way analysis of variance. The resulting Kruskal-Wallis statistic and significance level were reported to facilitate a comprehensive interpretation of the findings.

To answer RQ4, RQ5, and RQ6, we categorized our samples into three groups based on their mobility status and investigated potential statistically significant differences before and after mobility changes (specifically, in 2020 and 2021). For key academic indicators that adhered to the assumptions of normality and exhibited homogeneity of variances, we conducted a parameter test called the Ration paired t-test within the framework of paired sample t-test. Subsequently, we reported the calculated t-value and significance level, allowing for a thorough analysis of the observed trends. For key academic indicators that deviated from a normal distribution, we employed the nonparametric Wilcoxon test within paired sample t-test. We reported the Sum of signed ranks (W) and significance level to ensure a comprehensive interpretation of the observed changes.

Finally, based on the outcomes of the group comparisons, we discuss the reliability of the warning journal list formulated by NSLC and offer a valuable reference for countries or institutions considering the development of similar lists.

Descriptive Analyses

Descriptive statistical analysis results (as shown in Table 2 and Fig. 2) indicate that, compared to the warning journals in 2020, the warning journals in 2021 had lower mean values in terms of the number of articles in the journal, international collaboration degree, and article influence. However, they had higher mean values in terms of the APCs, self-citation rate, number of retractions, JIF, number of OA articles, and number of retractions by Chinese authors. The mean values of article influence remained unchanged between 2020 and 2021 for the warning journals. It is worth noting that the mean JIF of warning journals in 2020 was 3.43, with a maximum value of 8.89, while in 2021, the mean JIF was 3.66, with a maximum value of 10.18. Some academic journals with JIF greater than 5 are also considered warning journals, although such journals are regarded as high-impact or high-quality journals in certain fields (McGrath et al. 2020).

Regarding APCs, the maximum value in 2020 for warning journals reached $34.91 million, while in 2021, it reached $37.13 million. However, correspondingly, the maximum number of OA articles in 2021 decreased by 3,831 compared to 2020, while the average increased by 376. In terms of retraction information, warning journals showed polarization. In the 2020 warning journal list, 44 journals had zero retractions, whereas, in 2021, there were 35. In the 2020 warning journal list, three academic journals had retractions exceeding 50, while in 2021, there were five. The number of retractions by Chinese authors is a matter of concern. In 2020, there were 36 warning journals in which the number of retractions by Chinese authors accounted for over 50% of the total retractions. This number increased to 41 journals in 2021.

Table 2

Descriptive statistical analysis divided by year
Descriptive statistics of key academic indicators for the warning journals in 2020
	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10
N	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00
Minimum	69.00	0.71	0.00	0.09	0.00	0.00	0.22	0.01	0.00	0.00
Median	1071.00	21.68	1703025.00	0.71	0.08	0.50	3.31	0.59	839.50	0.00
Maximum	17928.00	50.58	34912800.00	2.29	0.69	164.00	8.89	2.09	17904.00	164.00
Mean	1824.00	20.28	4142623.00	0.80	0.10	5.83	3.43	0.66	1723.00	5.40
SD	2747.00	11.50	6500750.00	0.41	0.10	19.77	1.69	0.42	2784.00	19.70
SEM	292.90	1.23	692982.00	0.04	0.01	2.11	0.18	0.04	296.70	2.10
Descriptive statistics of key academic indicators for the warning journals in 2021
	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10
N	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00
Minimum	36.00	1.73	0.00	0.05	0.00	0.00	0.14	0.01	0.00	0.00
Median	1327.00	21.43	2244260.00	0.76	0.08	1.00	3.51	0.54	1140.00	0.00
Maximum	14073.00	60.50	37131185.00	2.39	0.86	158.00	10.18	2.09	14073.00	155.00
Mean	2208.00	19.79	5119584.00	0.80	0.11	9.81	3.66	0.62	2099.00	9.23
SD	2995.00	11.94	7654389.00	0.44	0.12	22.80	1.81	0.38	3044.00	22.45
SEM	319.30	1.27	815961.00	0.05	0.01	2.43	0.19	0.04	324.50	2.39
Note. A1: number of articles in the journal; A2: degree of internationalization; A3: APCs; A4: JCI; A5: self-citation rate; A6: number of retractions; A7: JIF; A8: article influence; A9: number of OA articles; A10: number of retractions by Chinese authors.

Normality Test Results

As shown in Table 3, the international collaboration level of the warning journals in 2021 (P = 0.2331) and the JIF of the warning journals in 2020 (P = 0.1766) passed the D’Agostino-Pearson test, indicating that both variables follow a normal distribution. Therefore, parametric tests should be used to compare the differences between groups. However, the remaining variables did not pass the D’Agostino-Pearson test (P < 0.05), suggesting that they do not follow a normal distribution. Therefore, non-parametric tests should be used to compare the differences between groups.

Table 3

Normality test results for key academic indicators
Key academic indicators	Sample size		K2-values
Key academic indicators	2020	2021	2020	2021
A1	88	88	91.44****	63.13****
A2	88	88	10.89**	2.91
A3	88	88	70.19****	61.73****
A4	88	88	23.06****	33.19****
A5	88	88	79.20****	89.58****
A6	88	88	154.40****	107.30****
A7	88	88	3.467	8.203*
A8	88	88	25.56****	32.79****
A9	88	88	89.88****	61.28****
A10	88	88	155.70****	107.40****
Note. A1: number of articles in the journal; A2: degree of internationalization; A3: APCs; A4: JCI; A5: self-citation rate; A6: number of retractions; A7: JIF; A8: article influence; A9: number of OA articles; A10: number of retractions by Chinese authors.
p < .05, p < .01, p < .001, **p < .0001

Comparative Analysis by Groups

Group Comparison Based on Warning Levels

Table 4 presents the results of the comparative analysis after grouping based on warning levels. (1) The differences between the low-level group and the high-level group are quite significant. There are statistically significant differences in the number of articles (P = 0.0022), degree of internationalization (P = 0.0013), APCs (P = 0.0012), self-citation rate (P = 0.0036), and number of retractions in 2020 (P = 0.0006). In 2021, there are statistically significant differences in the number of OA articles (P < 0.0001) and the number of retractions by Chinese authors (P = 0.0001), but no significant difference in the degree of internationalization (P = 0.3110) is observed. (2) The differences between the low-level group and the middle-level group are also quite noticeable. There are statistically significant differences in the degree of internationalization (P < 0.0001), self-citation rate (P < 0.0001), number of retractions (P < 0.0001), number of OA articles (P = 0.0244), and the number of retractions by Chinese authors (P < 0.0001) in 2020. In addition to the aforementioned key academic indicators, there are significant differences in the number of articles (P < 0.0001) and APCs (P = 0.0006) in 2021. (3) The differences between the high-level group and the middle-level group are not significant. There are statistically significant differences in the degree of internationalization (P < 0.0001) and the number of OA articles (P = 0.0244) in 2020, while no differences are observed in the other key academic indicators. All key academic indicators in 2021 show no significant differences between the two groups. Furthermore, there are no statistically significant differences in the impact factor and article influence of the three groups. This suggests that (1) To a certain extent, the warning journal list published by NSLC can help researchers or research evaluation institutions quickly categorize academic journals with potential quality risks and journals with significant quality risks. However, the criteria for this categorization are not consistent. Different groups have variations in different key academic indicators in different years. (2) The JIF and JCI, as key academic indicator for stratifying academic journals in Scientometrics (Ansari, rahaman, and Hashem Hussein Al-Attas 2020), does not seem to be adopted in the warning journal list.

Table 4

Group difference test results based on warning levels
Group difference test results in 2020
Key academic indicators	P-values	Kruskal-Wallis statistic/F- statistic	P-values			Mean rank diff.			Sample size
Key academic indicators	P-values	Kruskal-Wallis statistic/F- statistic	Low-level vs. Middle-level	Low-level vs. High-level	High-level vs. Middle-level	Low-level vs. Middle-level	Low-level vs. High-level	High-level vs. Middle-level	Sample size
A1	0.0020**	12.46	0.1196	0.0022**	0.1500	11.99	29.24	17.25	88
A2	< 0.0001****	53.08	< 0.0001****	0.0013**	< 0.0001****	28.90	-30.49	-59.39	88
A3	0.0014**	13.15	0.1956	0.0012**	0.0694	10.76	30.74	19.99	88
A4	0.1542	3.738	> 0.9999	0.1701	0.2750	-1.687	-16.53	-14.84	88
A5	< 0.0001****	25.99	< 0.0001****	0.0036**	> 0.9999	27.75	28.09	0.3384	88
A6	< 0.0001****	24.02	< 0.0001****	0.0006***	> 0.9999	-22.83	-30.26	-7.428	88
A7	0.2075	0.2075	0.9959	0.9159	0.9519	-0.0741	-0.4281	-0.3540	88
A8	0.8310	0.3702	> 0.9999	> 0.9999	> 0.9999	3.036	4.017	0.9811	88
A9	0.0034**	11.35	0.0244*	> 0.9999	0.0244*	-15.38	0.000	15.38	88
A10	< 0.0001****	30.52	< 0.0001****	< 0.0001	0.9508	-25.07	-33.07	-8.001	88
Group difference test results in 2021
Key academic indicators	P-value	Kruskal-Wallis statistic/F- value	P-value			Mean rank diff.			Sample size
Key academic indicators	P-value	Kruskal-Wallis statistic/F- value	Low-level vs. Middle-level	Low-level vs. High-level	High-level vs. Middle-level	Low-level vs. Middle-level	Low-level vs. High-level	High-level vs. Middle-level	Sample size
A1	< 0.0001****	30.83	< 0.0001****	< 0.0001****	0.2341	25.12	40.63	15.51	88
A2	< 0.0005***	10.82	< 0.0001****	0.3110	0.6602	13.08	7.971	-5.112	88
A3	< 0.0001****	25.01	0.0006***	< 0.0001****	0.2083	21.70	37.68	15.98	88
A4	0.1058	4.493	> 0.9999	0.1612	0.1205	1.326	-16.73	-18.06	88
A5	< 0.0001****	31.61	< 0.0001****	0.0006***	> 0.9999	30.18	32.24	2.058	88
A6	< 0.0001****	32.05	< 0.0001****	0.0003***	> 0.9999	-28.97	-32.40	-3.431	88
A7	0.6786	0.7755	> 0.9999	> 0.9999	> 0.9999	1.560	-6.180	-7.740	88
A8	0.7234	0.6475	> 0.9999	> 0.9999	> 0.9999	4.690	2.504	-2.186	88
A9	< 0.0001****	26.95	0.0003***	< 0.0001****	0.1759	22.50	39.14	16.64	88
A10	< 0.0001****	38.08	< 0.0001****	0.0001***	> 0.9999	-30.65	-33.34	-2.692	88
Note. A1: number of articles in the journal; A2: degree of internationalization; A3: APCs; A4: JCI; A5: self-citation rate; A6: number of retractions; A7: JIF; A8: article influence; A9: number of OA articles; A10: number of retractions by Chinese authors. Due to key academic indicators A2 and A7 following a normal distribution, we employed parameter tests and reported the F-statistic.
p < .05, p < .01, p < .001, **p < .0001

Group comparison based on mobility status

In the context of mobility status analysis, a comparative grouping was conducted, and the results are presented in Table 5. (1) The results reveal significant differences between the added group before and during the warning. Several key academic indicators, such as the number of articles (P < 0.0001), degree of internationalization (P = 0.0055), APCs (P < 0.0001), article influence (P = 0.0005), number of OA articles (P = 0.0001), and the number of retractions by Chinese authors (P = 0.0312), exhibit statistically significant variations between the pre-warning period (2020) and the warning period (2021). (2) In the two consecutive years of continuous warnings, the retained group did not demonstrate any significant differences in key academic indicators. Except for the number of retractions (P = 0.0469), which showed a statistically significant distinction, other key academic indicators displayed no substantial variations between the two consecutive years of continuous warnings. (3) The excluded group showcased relatively weaker disparities before and during the warning. While the self-citation rate (P < 0.0001) of the excluded group demonstrated statistically significant differences between the pre-warning and warning periods, the differences in the number of articles (P = 0.0112), APCs (P = 0.0083), and the number of OA articles (P = 0.0116) were comparatively weaker. Furthermore, there were no statistically significant differences in the key academic indicators between the two periods. The findings shed light on the following aspects. (1) The criteria for including or excluding academic journals in the warning journal list may lack uniformity. Notably, indicators such as the degree of internationalization, self-citation rate, article influence, and the number of retractions by Chinese authors displayed varying degrees of disparity between the included and excluded academic journals during their mobility. (2) The selection of the retained group appears reasonable. Academic journals that were retained in the warning journal list did not exhibit significant differences in key academic indicators of quality and misconduct over two consecutive years, reaffirming the importance of continued attention to these journals. (3) It is plausible that JCI and JIF were not considered as criteria for assessing mobility, as none of the three groups exhibited notable differences during mobility or retention, based on these two key academic indicators.

Table 5

Group difference test results based on mobility status
Key academic indicators	P-values			Sum of signed ranks (W)/ t- statistic			Sample size
Key academic indicators	Added group 2020 vs. 2021	Retained group 2020 vs. 2021	Excluded group 2020 vs. 2021	Added group	Retained group	Excluded group	Added group	Retained group	Excluded group	Full sample
A1	< 0.0001****	0.3594	0.0112*	289.0	-17.00	568.0	52	18	106	176
A2	0.0055**	0.4682	0.6107	3.040	0.7614	0.5121	52	18	106	176
A3	< 0.0001****	0.4258	0.0083**	271	-15.00	541.0	52	18	106	176
A4	0.7217	0.3750	0.2116	-29.0	-16.00	-260.0	52	18	106	176
A5	0.3276	0.3954	< 0.0001****	-79.0	17.00	1029	52	18	106	176
A6	0.0752	0.0469*	0.2378	46.00	29.00	145.0	52	18	106	176
A7	0.1179	0.4397	0.1009	1.620	0.8131	1.670	52	18	106	176
A8	0.0005***	0.2031	0.1987	-229.0	-23.00	-260.0	52	18	106	176
A9	0.0001***	0.4258	0.0116*	267.0	-15.00	518.0	52	18	106	176
A10	0.0312*	0.0664	0.3375	32.0	32.00	99.00	52	18	106	176
Note. A1: number of articles in the journal; A2: degree of internationalization; A3: APCs; A4: JCI; A5: self-citation rate; A6: number of retractions; A7: JIF; A8: article influence; A9: number of OA articles; A10: number of retractions by Chinese authors. Due to key academic indicators A2 and A7 following a normal distribution, we employed parameter tests and reported the t-statistic.
p < .05, p < .01, p < .001, **p < .0001

Taking the warning journal lists published by NSLC (2020 edition and 2021 edition) as the research object (N = 176), we analyze the reasonableness of using warning journal lists as a journal stratification tool and a research evaluation tool. Previous studies have focused on creating journal lists to identify mainstream journals (Safón and Docampo 2023) or analyzing the overlap, purpose, and researchers’ perceptions of different journal lists (Chen 2019; Adams and Johnson 2008). There has been intense debate regarding journal lists, especially the so-called “list of questionable journals,” but these debates lack empirical verification.

To fill these potential gaps in the previous literature, we provide a new theoretical perspective by drawing on social stratification theory and stratum mobility theory, considering “journal lists” as institutionalized stratification tools in academia. This stratification tool is based on considerations of fairness in research evaluation, differentiating the quality of academic journals at various levels or labeling journals with risks. Consistent with the views of social stratification theory, we believe that although this tool has the potential to improve information identification and research evaluation efficiency, it may be difficult to trust if the stratification or stratum mobility does not reflect differentiation.

Based on this, we examined whether the stratums defined by the warning journal list exhibit differences in key academic indicators. We addressed RQ1, 2, and 3, which investigated the significant differences in certain key academic indicators between the low-warning-level vs. middle-warning-level and low-warning-level vs. high-warning-level, while finding only a few differences between the middle-level and high-level. Controversial citation-based metrics in Scientometrics did not show variations across the three groups in different years. Although the warning journal list attempts to move away from citation-based evaluation frameworks by integrating different indicators to assess the quality risk of academic journals and assigning stratums to them, it appears to lack stronger persuasiveness and scientific validity. After all, the inconsistencies in differences across key academic indicators among different warning levels raise doubts. Furthermore, the lack of disclosure of specific data details by the warning journal list further diminishes its transparency and trustworthiness. As some previous studies (Tsigaris and Teixeira da Silva, 2021) have suggested, despite the “ideal” nature of the list of questionable journals, biases in its development process, especially regarding criteria, have been overlooked. If each category in the list cannot achieve consistent differences in key academic indicators, it becomes challenging to determine whether academic journals have been treated unfairly and to maintain credibility and trustworthiness in the list.

We also examined whether there are differences in key academic indicators among the stratum mobility in the warning journal list published by NSLC. We addressed RQ4, 5, and 6, which found differences in six key academic indicators for added groups (2020 vs. 2021), one key academic indicator for retained groups (2020 vs. 2021), and four key academic indicators for excluded groups (2020 vs. 2021). Similarly, controversial citation-based metrics in Scientometrics did not exhibit variations among the three groups. Conversely, previous research emphasized the necessity of citation content analysis in identifying questionable and non-questionable journals (Emanuel et al, 2021). If a list of questionable journals fails to achieve consensus on exclusion and inclusion criteria, it becomes difficult to generate trust in it. The reason is that if a journal does not show differences in assessing academic quality or misconduct indicators before or after inclusion or exclusion, it lacks a rationale for being included or excluded (it does not display an inclination to change its academic quality or misconduct status). Additionally, if the lists of questionable journals decide not to adopt traditional Scientometric indicators, it should elucidate and justify its reasons rather than solely communicate stratification or mobility results to the academic community.

In some countries, a list of questionable journals is not merely a tool but also a policy guiding researchers’ publishing behavior to ensure the quality of academic research and higher education (Cyranoski, 2018; Bagues, Sylos-Labini, and Zinovyeva, 2019). Therefore, its formulation and promotion need to consider additional factors such as transparency, scientific validity, and accountability. The warning journal list published by NSLC carries significant influence. Despite claiming not to be used for research evaluation, we discovered through news searches that over forty universities or hospitals in China have adopted this list as a basis for research evaluation. This implies that the list of questionable journals not only stratifies academic journals but also potentially stratifies researchers through the research evaluation system. Researchers who publish papers in questionable journals may be labeled negatively. Thus, careful consideration is necessary regarding the consequences of the lists of questionable journals. Additionally, the stratification and mobility of the lists of questionable journals should be evaluated for differences when excluding or including academic journals, considering whether they exhibit variations in key academic indicators to determine whether there is an inclination to improve academic quality or address academic misconduct. Our recommendation is for the creators of the lists of questionable journals to openly share their criteria, detailed data, and evaluation methods, thereby establishing credibility and trustworthiness in the academic community regarding such lists. It would be beneficial to involve different expert panels in the formulation process, including representatives from various disciplines and research institutions. This would help mitigate biases and ensure a comprehensive evaluation process. Ultimately, the goal should be to create a robust and fair system that supports the improvement of academic quality and integrity while minimizing the potential negative consequences for researchers and journals associated with being included in or excluded from the lists.

Limitations

In the pursuit of evaluating the warning journal list issued by NSLC, this study considered several key academic indicators. However, it is important to acknowledge that it lacked an in-depth exploration of certain subjective evaluation criteria that could have enriched the analysis. NSLC claimed to have sought guidance from academic experts during the formulation of the warning journal list, but regrettably, they omitted the provision of comprehensive information regarding these professionals. Additionally, the study overlooked the incorporation of an expanded repertoire of the lists of questionable journals for examination. For instance, in China, certain institutions curate a distinct list of questionable journals tailored exclusively to their organizational needs. One cannot help but wonder about the potential overlap and disparities between these institution-specific compilations and the widely acknowledged warning journal list. Addressing these inquiries would undoubtedly augment the overall breadth and applicability of our research findings. Alas, the unavailability of public access to these lists of questionable journals precluded their inclusion as subjects of our investigation.

This study treats the list of questionable journals as an institutionalized stratification mechanism and leverages the widely disseminated warning journal list published by NSLC as a litmus test to scrutinize its effectiveness in distinguishing between scholarly publications. The intent was to ascertain whether this list’s stratification of journals exhibits discernible disparities, thus determining its degree of credibility and trustworthiness. The findings revealed that while the different stratums identified by the warning journal list demonstrate variations in key academic indicators, these discrepancies do not follow a consistent pattern. Different stratums display varying degrees of disparity across different years and different key academic indicators. Furthermore, widely accepted citation metrics such as JIF and JCI appear to have been overlooked as evaluation criteria by the warning journal list, yet no clarification or justification has been provided by the institution. Considering these findings, it is imperative that the creators of such lists embrace a more proactive approach by openly sharing their evaluation standards and providing comprehensive data to bolster the accessibility and transparency of list-related information. By doing so, the credibility and trustworthiness of these lists can be fortified, instilling greater confidence in their efficacy. As the proliferation of questionable journals persists, it becomes increasingly apparent that we must embark on a transformative journey toward adopting a more responsible and accountable approach to research evaluation policies.

Competing interests: The authors declare no competing interests.

Adams, Dennis, and Norman Johnson. 2008. “The Journal List and Its Use: Motivation, Perceptions, and Reality.” European Journal of Information Systems 17 (2): 158–62. https://doi.org/10.1057/ejis.2008.8.
Ansari, Khadeeja M N, Safiqur rahaman, and Hashem Hussein Al-Attas. 2020. “Gauging the Quality of Behavioral Science Journals by Using Bibliometric Indicators.” COLLNET Journal of Scientometrics and Information Management 14 (1): 135–52. https://doi.org/10.1080/09737766.2020.1819173.
Bagues, Manuel, Mauro Sylos-Labini, and Natalia Zinovyeva. 2019. “A Walk on the Wild Side: ‘Predatory’ Journals and Information Asymmetries in Scientific Evaluations.” Research Policy 48 (2): 462–77. https://doi.org/10.1016/j.respol.2018.04.013.
Beckman, Jenny. 2019. “Prizes, Publications, and Transnational Communication at the Royal Swedish Academy of Sciences in the Nineteenth Century.” Nuncius: Journal of the Material and Visual History of Science 34 (2): 377–91. https://doi.org/10.1163/18253911-03402009.
Blanford, Christopher F. 2016. “Impact Factors, Citation Distributions and Journal Stratification.” Journal of Materials Science 51 (23): 10319–22. https://doi.org/10.1007/s10853-016-0285-x.
Bornmann, Lutz, and Richard Williams. 2017. “Can the Journal Impact Factor Be Used as a Criterion for the Selection of Junior Researchers? A Large-Scale Empirical Study Based on ResearcherID Data.” Journal of Informetrics 11 (3): 788–99. https://doi.org/10.1016/j.joi.2017.06.001.
Brainard, Jeffery. 2023. “Fake Scientific Papers Are Alarmingly Common.” Science 380 (6645): 568–69. https://doi.org/10.1126/science.adi6523.
Carpenter, Christopher R., David C. Cone, and Cathy C. Sarli. 2014. “Using Publication Metrics to Highlight Academic Productivity and Research Impact.” Edited by Gary M. Gaddis. Academic Emergency Medicine 21 (10): 1160–72. https://doi.org/10.1111/acem.12482.
Chapman, Colin A., Júlio César Bicca-Marques, Sébastien Calvignac-Spencer, Pengfei Fan, Peter J. Fashing, Jan Gogarten, Songtao Guo, et al. 2019. “Games Academics Play and Their Consequences: How Authorship,  H  -Index and Journal Impact Factors Are Shaping the Future of Academia.” Proceedings of the Royal Society B: Biological Sciences 286 (1916): 20192047. https://doi.org/10.1098/rspb.2019.2047.
Chen, Xiaotian. 2019. “Beall’s List and Cabell’s Blacklist: A Comparison of Two Lists of Predatory OA Journals.” Serials Review 45 (4): 219–26. https://doi.org/10.1080/00987913.2019.1694810.
Clarivate. 2022. “Journal Citation Reports: Quartile Rankings and Other Metrics.” Clarivate.com. May 24, 2022. https://support.clarivate.com/ScientificandAcademicResearch/s/article/Journal-Citation-Reports-Quartile-rankings-and-other-metrics?language=en_US.
Cyranoski, David. 2018. “China Awaits Controversial Blacklist of ‘Poor Quality’ Journals.” Nature 562 (7728): 471–72. https://doi.org/10.1038/d41586-018-07025-5.
Dunleavy, Daniel J. 2022. “Progressive and Degenerative Journals: On the Growth and Appraisal of Knowledge in Scholarly Publishing.” European Journal for Philosophy of Science 12 (4). https://doi.org/10.1007/s13194-022-00492-8.
Elliott, David B. 2013. “Salami Slicing and the SPU: Publish or Perish?” Ophthalmic and Physiological Optics 33 (6): 625–26. https://doi.org/10.1111/opo.12090.
Emanuel, Kulczycki, Holowiecki Marek, Taskin Zehra, and Krawczyk Franciszek. 2021. “Do Impact-Factor Journals Make Blacklisted Journals Visible? A Systematic Citation Study.” In 18t International Conference on Scientometrics & Informetrics (ISSI2021), edited by Glanzel W, Heeffer S, Chi PS, and Rousseau R, 615–20. Leuven: The International Society for Informetrics and Scientometrics.
Frandsen, Tove Faber. 2019. “How Can a Questionable Journal Be Identified: Frameworks and Checklists.” Learned Publishing 32 (3). https://doi.org/10.1002/leap.1230.
Geuna, Aldo. 2015. Global Mobility of Research Scientists. Academic Press.
Giddens, Anthony. 1975. The Class Structure of the Advanced Societies. New York: Harper & Row.
Lambert, Paul, and Dave Griffiths. 2018. Social Inequalities and Occupational Stratification. London Palgrave Macmillan Uk.
Madrueño-Aguilar, Rogelio. 2017. “Global Income Distribution and the Middle-Income Strata: Implications for the World Development Taxonomy Debate.” The European Journal of Development Research 29 (1): 1–18. https://doi.org/10.1057/ejdr.2015.61.
McGrath, Trevor A, Joshua C Bowdridge, Ross Prager, Robert A Frank, Lee Treanor, Ana Dehmoobad Sharifabadi, Jean-Paul Salameh, et al. 2020. “Overinterpretation of Research Findings: Evaluation of ‘Spin’ in Systematic Reviews of Diagnostic Accuracy Studies in High–Impact Factor Journals.” Clinical Chemistry 66 (7): 915–24. https://doi.org/10.1093/clinchem/hvaa093.
MDPI. 2022. “MDPI’s Statement on the ‘Early Warning Journal List (Trial)’ Released by the National Science Library of the Chinese Academy of Sciences (NSLC) on 31 December 2021.” Www.mdpi.com. March 15, 2022. https://www.mdpi.com/about/announcements/3624.
Merton, Robert K. 1957. “Priorities in Scientific Discovery: A Chapter in the Sociology of Science.” American Sociological Review 22 (6): 635–59. https://doi.org/10.2307/2089193.
Miranda, Ruben, and Esther Garcia-Carpintero. 2019. “Comparison of the Share of Documents and Citations from Different Quartile Journals in 25 Research Areas.” Scientometrics 121 (1): 479–501. https://doi.org/10.1007/s11192-019-03210-z.
Nair, Lakshmi, and Oluwaseun A. Adetayo. 2018. “A Critique of the Impact Factor and Ramifications of Its Misuse in Plastic and Reconstructive Surgery.” Plastic and Reconstructive Surgery 142 (4): 566–68. https://doi.org/10.1097/prs.0000000000004774.
Nature. 2018. “Journal Blacklists: Show Your Working.” Nature 562 (7727): 308–8. https://doi.org/10.1038/d41586-018-07033-5.
Nelhans, Gustaf, and Theo Bodin. 2020. “Methodological Considerations for Identifying Questionable Publishing in a National Context: The Case of Swedish Higher Education Institutions.” Quantitative Science Studies 1 (2): 1–20. https://doi.org/10.1162/qss_a_00033.
NSLC. 2021. “Early Warning Journal List 2021.” Earlywarning.fenqubiao.com. December 31, 2021. https://earlywarning.fenqubiao.com/#/en/early-warning-journal-list-2021.
Onodera, Natsuo, and Fuyuki Yoshikane. 2014. “Factors Affecting Citation Rates of Research Articles.” Journal of the Association for Information Science and Technology 66 (4): 739–64. https://doi.org/10.1002/asi.23209.
Safón, Vicente, and Domingo Docampo. 2023. “What Are You Reading? From Core Journals to Trendy Journals in the Library and Information Science (LIS) Field.” Scientometrics 128 (5): 2777–2801. https://doi.org/10.1007/s11192-023-04673-x.
Schumpeter, Joseph A. 1955. Imperialism and Social Classes. Ohio: The World Publishing Company.
Sorokin, Pitirim. 1927. Social Mobility. New York: Harper and Brothers.
Soteropulos, Carol E., and Samuel O. Poore. 2021. “Navigating the Gray of Academic Publication.” Annals of Plastic Surgery 87 (6): e171–79. https://doi.org/10.1097/sap.0000000000002902.
Strinzel, Michaela, Anna Severin, Katrin Milzow, and Matthias Egger. 2019. “Blacklists and Whitelists to Tackle Predatory Publishing: A Cross-Sectional Comparison and Thematic Analysis.” Edited by Julie M. Wolf. MBio 10 (3). https://doi.org/10.1128/mbio.00411-19.
Tang, Gengyan, and Jingwen Jia. 2022. “Characteristics of Blacklisted Journals: Evidence from Chinese-Language Academic Journals.” Accountability in Research, August. https://doi.org/10.1080/08989621.2022.2112953.
Teixeira da Silva, Jaime A., and Stephen Gichuhi Kimotho. 2021. “Signs of Divisiveness, Discrimination and Stigmatization Caused by Jeffrey Beall’s ‘Predatory’ Open Access Publishing Blacklists and Philosophy.” The Journal of Academic Librarianship 48 (3): 102418. https://doi.org/10.1016/j.acalib.2021.102418.
Teixeira da Silva, Jaime A., Mina Moradzadeh, Yuki Yamada, Daniel J. Dunleavy, and Panagiotis Tsigaris. 2023. “Cabells’ Predatory Reports Criteria: Assessment and Proposed Revisions.” The Journal of Academic Librarianship 49 (1): 102659. https://doi.org/10.1016/j.acalib.2022.102659.
Teixeira da Silva, Jaime A., and Panagiotis Tsigaris. 2018. “What Value Do Journal Whitelists and Blacklists Have in Academia?” The Journal of Academic Librarianship 44 (6): 781–92. https://doi.org/10.1016/j.acalib.2018.09.017.
Torres-Salinas, Daniel, Pilar Valderrama-Baca, and Wenceslao Arroyo-Machado. 2022. “Is There a Need for a New Journal Metric? Correlations between JCR Impact Factor Metrics and the Journal Citation Indicator—JCI.” Journal of Informetrics 16 (3): 101315. https://doi.org/10.1016/j.joi.2022.101315.
Tsigaris, Panagiotis, and Jaime A. Teixeira da Silva. 2021. “Why Blacklists Are Not Reliable: A Theoretical Framework.” The Journal of Academic Librarianship 47 (1): 102266. https://doi.org/10.1016/j.acalib.2020.102266.
You, Taekho, Jinseo Park, June Young Lee, Jinhyuk Yun, and Woo-Sung Jung. 2022. “Disturbance of Questionable Publishing to Academia.” Journal of Informetrics 16 (2): 101294. https://doi.org/10.1016/j.joi.2022.101294.
Zhang, Lin, Yahui Wei, Gunnar Sivertsen, and Ying Huang. 2022. “The Motivations and Criteria behind China’s List of Questionable Journals.” Learned Publishing 35 (4): 467–80. https://doi.org/10.1002/leap.1456.

Download PDF

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

The List of Questionable Journals: Is It Trustworthy?

Status:

Version 1

Abstract

Figures

Introduction

Theoretical Frameworks and Research Questions

Stratification of Academic Journals

Stratum Mobility of Academic Journals

Materials and Methods

Sample Selection

The Key Academic Indicators

Data Analysis

Results

Descriptive Analyses

Normality Test Results

Comparative Analysis by Groups

Group Comparison Based on Warning Levels

Group comparison based on mobility status

Discussion

Limitations

Conclusion

Declarations

References

Status:

Version 1