Are the Lists of Questionable Journals Reasonable: A Case Study of Early Warning Journal Lists

doi:10.21203/rs.3.rs-3105953/v2

Download PDF

Research Article

Are the Lists of Questionable Journals Reasonable: A Case Study of Early Warning Journal Lists

https://doi.org/10.21203/rs.3.rs-3105953/v2

This work is licensed under a CC BY 4.0 License

Version 2

posted

You are reading this latest preprint version

The lists of questionable journals are regarded a policy or tool to ensure research quality and integrity. However, due to their lack of clear criteria, they remain highly debated. Taking a typological perspective, we assess the reasonableness of the lists of questionable journals by examining how well it reflects the differences in bibliometric attributes among distinct groups when categorizing and labelling them, and whether these differences are consistent. Using the Early Warning Journal Lists released by the National Science Library of the Chinese Academy of Sciences as an example, we grouped listed journals based on warning levels and years. Subsequently, we compared them in groups to determine if there are differences in key academic indicators among different warning categories, thereby evaluating the reasonableness of the warning journal list. Our research findings suggest that Early Warning Journal Lists may have employed inconsistent criteria when assigning warning levels. Variations in the degrees of differences or the absence of differences were observed among groups across different key academic indicators. Additionally, citation metrics like journal impact factor and journal citation indicator might not have been treated as grouping criteria in the Early Warning Journal Lists, yet this lack of detailed explanation from the creators is evident. This highlights the need for a more scientific and meticulous assessment of the lists of questionable journals, along with a greater emphasis on sharing detailed standards and data. Furthermore, our study offers recommendations for future formulation of lists of questionable journals by various institutions.

Library Science

Publishing/Media

Information Retrieval and Management

Educational Philosophy and Theory

Questionable journals

Early Warning Journal Lists

academic journals

research integrity

quality assurance

research policy

research evaluation

scholarly publishing

The lists of questionable journals gather academic journals with potential issues related to research quality and integrity onto a single list, such as Beall’s List[1], Cabell’s Journal Blacklist[2], and Early Warning Journal Lists[3]. The lists of questionable journals are considered an effective method of research evaluation by universities and research institutions (Nature 2018). They serve as filters for research evaluation agencies to quickly identify low-quality publications. Moreover, they provide guidelines for researchers to shape their publishing behavior and attempt to ensure research integrity and quality. However, the lists of questionable journals are highly controversial. Different developers of such lists employ varying criteria, resulting in the inclusion of different academic journals whose accuracy and trustworthiness has been questioned (Strinzel et al. 2019).

Previous studies have concluded that 28 criteria of Cabell’s Journal Blacklist need clarification and revision, and 39 should be removed (Teixeira da Silva et al. 2023). Additionally, Beall’s list has also faced criticism due to its vague criteria (Teixeira da Silva and Kimotho, 2021). Indeed, there appears to be a gray area between “good” and “bad” (Dunleavy, 2022), which casts doubt on the reasonableness and trustworthiness of the lists of questionable journals (Teixeira da Silva and Tsigaris, 2018). In contrast, some scholars have defended the lists of questionable journals and proposed ways to improve the criteria of the lists (Frandsen, 2019; Nelhans and Bodin, 2020; You et al. 2022).

Despite the criticism, the lists of questionable journals continue to be widely used in quality assurance. In December 2020, the National Science Library of the Chinese Academy of Sciences (NSLC) [4] published the “Early Warning Journal List 2020” (EWJL 2020) containing 65 English academic journals[5], all of which are indexed by the JCR (Journal Citation Reports). These journals were considered potentially lacking academic rigor. One year later, NSLC released the “Early Warning Journal List 2021” (EWJL 2021), which includes 35 English academic journals that are also indexed by the JCR[6]. The release of the EWJLs is seen as a response to China’s research integrity policies. In May 2018, the State Council of the People’s Republic of China issued “Several Opinions on Further Strengthening the Construction of Research Integrity” (State Council of the People’s Republic of China 2018). This policy mentioned several possible measures to achieve this goal:

“The Ministry of Science and Technology should establish an early warning mechanism for academic journals to support relevant institutions. An early warning list of domestic and international academic journals should be published, and dynamic tracking and timely adjustment should be implemented. Academic journals that neglect research quality, disorderly management, or put commercial interests first will be blacklisted.

In February 2020, the Ministry of Science and Technology of the People’s Republic of China introduced the policy document titled “Measures to Eliminate the Bad Orientation of ‘Paper Only’ in Scientific and Technological Evaluation (Trial)”, emphasizing the use of warning journal lists to ensure research quality (Ministry of Science and Technology of the People’s Republic of China 2020). Within the context of “nurturing and enhancing high-quality scientific journals in China”, it reiterated a focus on:

“Enhancing the effectiveness of early warning mechanisms for academic journals, regularly releasing warning lists for both domestic and international academic journals, and implementing continuous monitoring and prompt adjustments.”

After the release of EWJL 2020, more than 40 Chinese universities and institutions have promptly adopted the EWJL 2020 and EWJL 2021 as a basis for their research quality assurance (Tang and Jia, 2022). Researchers who publish papers in journals listed in EWJLs will not receive any material rewards or honors (Jiamusi University 2023). However, the release of EWJL 2020 has sparked considerable controversy. Multidisciplinary Digital Publishing Institute (MDPI) [7] had 22 academic journals under its umbrella included in EWJL 2020. Subsequently, MDPI issued a statement expressing concern that the criteria for inclusion of journals in EWJL 2020 were not sufficiently clear (MDPI 2022).

Upon reviewing the literature and various viewpoints, the task of defending the reasonableness of lists of questionable journals remains challenging. As a tool or policy designed to ensure research quality and integrity, the lists of questionable journals not only have the potential to significantly impact researchers’ publishing practices (Petrou 2021) but also may influence institutions or universities in their practices of ensuring research quality and integrity. Only when a policy or tool is reasonable can it potentially assist in enhancing research quality and ensuring research integrity. Therefore, evaluating the reasonableness of such lists holds significant importance. The generation of lists of questionable journals stems from typological thinking, as it attempts to categorize academic journals based on certain attributes and assign them concepts or symbols. For instance, EWJLs assign warning levels (low, middle and high) to questionable journals, in order to distinguish academic journals based on varying degrees of research quality and integrity risk. However, when delineating specific entities, the reasonableness of the criteria used for categorization must be considered. This involves significant and consistent differences in attributes between these entities. This viewpoint can be illustrated by Aristotle’s view in his work on zoology:

“A species is defined through its genus and its differentia (what came to be known as definition per genus et differentiam). What a species is, is determined by the genus to which the species belongs and what differentiates that species from all other species of the same genus.” (Sandford 2019, 6)

If we view academic journals as an ecosystem, then questionable journals represent an “species” within this system. They should exhibit distinct bibliometric attributes from other “species” (e.g., regular journals) since they fall within the category of publications. Does a significant difference exist between questionable journals and other regular journals? This question has amassed some evidence (Kulczycki et al. 2021). Building upon this, we raise another under-discussed issue: the reasonableness of the criteria used for categorizing within questionable journals. Considering the extensive influence of EWJLs and its affiliation with a top-tier Chinese research institution, along with its internal division into various warning levels and newly created groups after annual updates (such as excluded groups, retained groups, and added groups), we use it as an example to explore the following two questions regarding the reasonableness of this list:

RQ1: Did EWJLs consider consistent bibliometric attribute differences when assigning warning levels?

RQ2: Did EWJLs consider consistent bibliometric attribute differences when excluding, adding, and retaining academic journals?

Our findings can offer more reliable evidence for ongoing scholarly debates and provide reference points for countries developing lists of questionable journals, ultimately fostering the creation of more dependable policies or tools to ensure research quality and integrity.

[1] The Beall’s List is a compilation of predatory journals. The original website of Beal’s list has been removed for certain reasons, and the list has been archived on the following website: https://beallslist.net/.

[2] Cabell’s journal blacklist can be accessed through the following website, but it requires payment: https://noaa.cabells.com/about-predatory.

[3] While some studies utilize the concept of “journal blacklists”, we have opted for the alternative concept of “the lists of questionable journals” due to its more inclusive nature.

[4] NSLC is an affiliated institution of the Chinese Academy of Sciences, the highest-level academic institution for natural sciences in China (https://www.cas.cn/zz/yk/201410/t20141016_4225142.shtml). NSLC is recognized as one of the most prolific and cited research organizations in the field of library and information sciences in China, possessing a strong representation in this domain (http://english.las.cas.cn/About/about/).

[5] Detailed information about the EWJL 2020 can be accessed through the following link: https://earlywarning.fenqubiao.com/#/en/early-warning-journal-list-2020.

[6] Detailed information about the EWJL 2021 can be accessed through the following link: https://earlywarning.fenqubiao.com/#/en/early-warning-journal-list-2021.

[7] MDPI is a scholarly publishing institution headquartered in Switzerland. This mega-publisher was initially incorporated on Beall’s list and was subsequently excluded on 28th October 2015 (https://www.mdpi.com/about/announcements/534).

Sample Selection

We chose EWJL 2020 and EWJL 2021 as the primary samples (EWJL 2022 suspended publication for unknown reasons). EWJL 2021 comprises 35 journals, while EWJL 2020 comprises 65 journals. There are 9 journals that overlap between the two lists. Then we identified three journals with missing data and excluded them. Specifically, EWJL 2021 includes 35 journals, and EWJL 2020 includes 62 journals.

General Information

We classified the research domains of 88 warning journals according to the JCR classification criteria. The results are presented in Figure 1. As illustrated in Figure 1, 50% of the warning journals are categorized under medicine, followed by engineering (15.9%) and biology (14.7%). Moreover, there is one warning journal in each of the domains of multidisciplinary science and agricultural and forestry science.

Group Division

To address RQ1, we classified the journals according to their warning levels. In EWJL 2020, there are 28 journals with low-warning-level, 28 journals with mid-warning-level warnings, and 6 journals with high-warning-level. In EWJL 2021, there are 15 journals with low-warning-level, 15 journals with mid-warning-level, and 5 journals with high-warning-level. Since the overlapping journals have different warning levels in 2020 and 2021, we separately categorized them for statistical purposes. Therefore, this process resulted in 97 samples.

To address RQ2, we categorized the journals based on their flow status. For academic journals that appear in EWJL 2020 but not in EWJL 2021, we categorized them into the “excluded group”. Academic journals that are present in EWJL 2021 but not in EWJL 2020 are classified as the “included group”. Academic journals that appear in both EWJL 2020 and EWJL 2021 are grouped under the “retained group” (overlapping journals). There are 26 journals in the included group, 9 journals in the retained group, and 53 journals in the excluded group. Due to the need for comparing their data before and after the flow, we gathered relevant data for both 2020 and 2021[8]. Therefore, this process resulted in a total of 176 samples (88 samples each year).

The Key Academic Indicators

To measure the bibliometric attributes of different groups, we operationalized these attributes as key academic indicators. The NSLC exclusively provides limited information about EWJLs, including the journal name, discipline, and warning level. No further details are available regarding individual journals. However, in late 2020, the NSLC released a list that outlined seven bibliometric criteria utilized for journal selection. These bibliometric criteria include the number of articles in the journal, degree of internationalization, rejection rate, article processing charges (APCs), journal citation success index, self-citation rate, and retraction information (We have chosen to quantify this criterion using the number of retractions, and the number of retractions by Chinese authors).

The bibliometric criteria (it can also be regarded as bibliometric attributes, as their purpose is to distinguish between different questionable journals). were initially published in Chinese, but an English version was also made available alongside the EWJL 2021 (NSLC, 2021). Although the criteria remained unchanged from 2020, there are slight disparities between the original Chinese version and its English translation. Notably, the Chinese version includes the criterion of “degree of internationalization,” while the English version omits it. Additionally, the Chinese version describes the “number of articles in the journal” as the “growth rate of productivity” in the English version. Moreover, the English version explicitly mentions “paper mill,” whereas the Chinese version does not.

Since the rejection rate of academic journals is usually not publicly available, we excluded this criterion as a key academic indicator for subsequent analysis. Furthermore, the NSLC has not disclosed journal citation success index for the warning journals, and this information is unavailable to us. As a result, we have no choice but to consider using the Journal Citation Indicator (JCI) as a substitute for this key academic indicator. The reason is that both indicators serve as supplements and enhancements to the JIF (Torres-Salinas, Valderrama-Baca, and Arroyo-Machado 2022). To investigate the reasonableness of EWJLs, we conducted a study where we regarded these bibliometric criteria (key academic indicators) as the bibliometric attributes of questionable journals. This effort involves thorough scrutiny, taking into consideration prior scholarly work (Zhang et al., 2022). Furthermore, we have incorporated supplementary key academic indicators, including JIF, article influence, number of open access (OA) articles, and the number of retractions by Chinese authors. Detailed information regarding the data sources and all indicators can be found in Table 1.

Table 1. Selection of key academic indicators, data sources, and their explanation

The key academic indicators	Explanation	Data sources
Number of articles in the journal (A1)	The number of citable articles published each year	InCites
Degree of internationalization (A2)	The number of international collaborations (papers that contain one or more international co-authors) for an entity divided by the total number of documents for the same entity represented as a percentage[9]	InCites
APCs (A3)	The number of OA articles published by an academic journal each year multiplied by the journal’s APCs (converted to USD based on live exchange rates)	DOAJ and journals’ website
JCI (A4)	The average category-normalized citation impact for papers published in the prior three-year period	InCites
Self-citation rate (A5)	Calculated by dividing an academic journal’s self-citations by its total citations	InCites
Number of retractions (A6)	The number of articles retracted by an academic journal each year	Retraction Watch
JIF (A7)	All citations to the journal in the current Journal Citation Reports year to items published in the previous two years, divided by the total number of scholarly items published in the journal in the previous two years.	InCites
Article influence (A8)	It is calculated by multiplying the Eigenfactor by 0.01 and dividing by the number of articles in the journal, normalized as a fraction of all articles in all publications.	Incites
Number of OA articles (A9)	The number of OA articles published by an academic journal each year	InCites
Number of retractions by Chinese authors (A10)	The number of articles retracted (including those with Chinese authors) each year by an academic journal	Retraction Watch

Data Analysis

Firstly, we utilized Prism 10.0 to conduct the D’Agostino-Pearson test in order to examine the adherence of key academic indicators to a normal distribution. The D’Agostino-Pearson test assumes, under the null hypothesis, that all values are drawn from a population that follows a Gaussian distribution. If the obtained p-value is less than 0.05, we reject this hypothesis, indicating a departure from normality in the data.

Secondly, to answer RQ1, we classified the journals into three distinct groups based on their warning levels and proceeded to analyze potential statistically significant differences among these groups. For key academic indicators that satisfied the assumptions of normality and exhibited homogeneity of variances, we performed a parameter test using the Brown-Forsythe test within the framework of one-way ANOVA (analysis of variance). Subsequently, we reported both the F-value and the significance level to provide a comprehensive understanding of the results. Conversely, for key academic indicators that deviated from a normal distribution, we employed the nonparametric Kruskal-Wallis test in one-way analysis of variance. The resulting Kruskal-Wallis statistic and significance level were reported to facilitate a comprehensive interpretation of the findings.

To address Research Question 2, we divided our samples into three distinct groups according to their flow status. We then proceeded to examine potential statistically significant variations between the years 2020 and 2021 (before and after the flow). For key academic indicators that adhered to the assumptions of normality and exhibited homogeneity of variances, we conducted a parameter test called the Ration paired t-test within the framework of paired sample t-test. Subsequently, we reported the calculated t-value and significance level, allowing for a thorough analysis of the observed trends. For key academic indicators that deviated from a normal distribution, we employed the nonparametric Wilcoxon test within paired sample t-test. We reported the Sum of signed ranks (W) and significance level to ensure a comprehensive interpretation of the observed changes.

Finally, based on the outcomes of the group comparisons, we discuss the reliability of the warning journal list formulated by NSLC and offer a valuable reference for countries or institutions considering the development of similar lists.

[8] Since EWJLs have not disclosed any details about the data, and considering its update frequency is once a year, we speculate that EWJLs focus on dynamic and real-time data. Therefore, we utilized data from the status of the journal in 2020 (before the flow) and in 2021(after the flow), without addressing its long-term data variations.

[9] As Zhang Lin et al. (2022) pointed out, “The geographic diversity of authors may reflect the widespread recognition of the journal in the world.” Therefore, this indicator is considered an important academic metric in China.

Descriptive Analyses

To provide a comprehensive description of the sample situation, we conducted a descriptive statistical analysis on the data from 88 journals (n=176) for the years 2020 and 2021. The results are presented in Table 2 and Figure 2. The findings of the descriptive statistical analysis indicate that compared to the year 2020, the mean values of key academic indicators such as the number of articles, degree of internationalization, and article influence have decreased for all journals. However, they have shown higher mean values in terms of APCs, self-citation rate, number of retractions, JIF, number of OA articles, and number of retractions by Chinese authors. From 2020 to 2021, the mean values of JCI remained consistent across all journals. Notably, the mean JIF for all journals in 2020 was 3.43 with a maximum value of 8.89, while in 2021, the mean JIF was 3.66 with a maximum value of 10.18. Some academic journals with JIF greater than 5 are also considered warning journals, despite being seen as high-impact or high-quality journals in certain fields (McGrath et al., 2020).

In the year 2020, the highest APCs value among all journals was 34.91 million USD, and in 2021, it increased to 37.13 million USD. However, correspondingly, the maximum number of OA articles in 2021 decreased by 3831, while the average number increased by 376 compared to 2020. Retraction information is also noteworthy. In 2020, there were 44 journals with zero retractions, whereas in 2021, there were 35. In 2020, there were 3 journals with retraction occurrences exceeding 50 times, and in 2021, this number increased to 5. The number of retractions by Chinese authors is concerning. In the year 2020, 36 journals had retraction occurrences by Chinese authors accounting for more than 50% of the total retractions. This number increased to 41 journals in 2021.

Table 2. Descriptive statistical analysis divided by year

Descriptive statistical analysis of the warning journals’ data for the year 2020
	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10
N	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00
Minimum	69.00	0.71	0.00	0.09	0.00	0.00	0.22	0.01	0.00	0.00
Median	1071.00	21.68	1703025.00	0.71	0.08	0.50	3.31	0.59	839.50	0.00
Maximum	17928.00	50.58	34912800.00	2.29	0.69	164.00	8.89	2.09	17904.00	164.00
Mean	1824.00	20.28	4142623.00	0.80	0.10	5.83	3.43	0.66	1723.00	5.40
SD	2747.00	11.50	6500750.00	0.41	0.10	19.77	1.69	0.42	2784.00	19.70
SEM	292.90	1.23	692982.00	0.04	0.01	2.11	0.18	0.04	296.70	2.10
Descriptive statistical analysis of the warning journals’ data for the year 2021
	A1	A2	A3	A4	A5	A6	A7	A8	A9	A10
N	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00	88.00
Minimum	36.00	1.73	0.00	0.05	0.00	0.00	0.14	0.01	0.00	0.00
Median	1327.00	21.43	2244260.00	0.76	0.08	1.00	3.51	0.54	1140.00	0.00
Maximum	14073.00	60.50	37131185.00	2.39	0.86	158.00	10.18	2.09	14073.00	155.00
Mean	2208.00	19.79	5119584.00	0.80	0.11	9.81	3.66	0.62	2099.00	9.23
SD	2995.00	11.94	7654389.00	0.44	0.12	22.80	1.81	0.38	3044.00	22.45
SEM	319.30	1.27	815961.00	0.05	0.01	2.43	0.19	0.04	324.50	2.39

Normality Test Results

When conducting ANOVA and paired t-tests, it is essential to evaluate the normal distribution of the data to determine the appropriate statistical tests to employ. As shown in Table 3, we performed a test for normality on the data of the 88 journals for the years 2020 and 2021 (n=176). The results indicate that the degree of internationalization of the journals in 2021 (P=0.2331) and JIF of the journals in 2020 (P=0.1766) passed the D’Agostino-Pearson test, suggesting that these two variables follow a normal distribution. Therefore, parametric tests should be used to compare inter-group differences for these variables. However, the remaining variables did not pass the D’Agostino-Pearson test (P<0.05), indicating that they do not follow a normal distribution. Therefore, non-parametric tests should be used to compare inter-group differences for these variables.

Table 3. Normality test results for key academic indicators

Key academic indicators	Sample size		*K2-values*
Key academic indicators	2020	2021	2020	2021
A1	88	88	91.44****	63.13****
A2	88	88	10.89**	2.91
A3	88	88	70.19****	61.73****
A4	88	88	23.06****	33.19****
A5	88	88	79.20****	89.58****
A6	88	88	154.40****	107.30****
A7	88	88	3.467	8.203*
A8	88	88	25.56****	32.79****
A9	88	88	89.88****	61.28****
A10	88	88	155.70****	107.40****

**P<.05, ** P<.01, *** P<.001, **** P<.0001

Comparative Analysis by Groups

Group Comparison Based on Warning Levels

Table 4 presents the comparative analysis results grouped according to warning levels. (1) In EWJL 2020, there were differences between the low-warning-level and mid-warning-level groups in seven key academic indicators: number of articles in the journal (P=0.0008), degree of internationalization (P<0.0001), APCs (P=0.0127), self-citation rate (P<0.0001), number of retractions (P<0.0001), number of OA articles (P=0.0072), and number of retractions by Chinese authors (P=0.0002). In EWJL 2021, the low-warning-level and mid-warning-level groups only differed in four key academic indicators: degree of internationalization (P=0.0001), self-citation rate (P=0.0486), number of retractions (P=0.0085), and number of retractions by Chinese authors (P=0.0043). (2) In EWJL 2020, differences were observed between the low-warning-level and high-warning-level groups in five key academic indicators: degree of internationalization (P=0.0475), self-citation rate (P=0.0163), number of retractions (P=0.0006), number of OA articles (P=0.0379), and number of retractions by Chinese authors (P<0.0001). In EWJL 2021, the low-warning-level and high-warning-level groups also differed in five key academic indicators: number of articles in the journal (P=0.0002), APCs (P=0.0007), self-citation rate (P=0.0181), number of OA articles (P=0.0001), and number of retractions by Chinese authors (P=0.0375). (3) There were only marginal differences between the high-warning-level and mid-warning-level groups. In EWJL 2020, no statistically significant differences were found between the high-warning-level and mid-warning-level groups in any key academic indicators. In EWJL 2021, the high-warning-level and mid-warning-level groups differed only in number of articles in the journal (P=0.0451) and APCs (P=0.0483), with no differences in other key academic indicators. This suggests: (1) EWJL 2020 and EWJL 2021 can aid researchers or research institutions in swiftly distinguishing journals with potential quality risks from those with substantial quality risks. However, distinguishing journals with higher quality risks from those with significant quality risks may be unreasonable (mid-warning-level and high-warning-level groups exhibit minimal differences across many key academic indicators). (2) NSLC might have employed inconsistent criteria when assigning warning levels to academic journals. For instance, in EWJL 2020, the low-warning-level group differed from the mid-waning-level group in number of articles in the journal, while it exhibited no differences with the high-warning-level group in the same indicator. (3) In scientometrics, the JIF and JCI, which serve as crucial metrics for assessing the quality and impact of academic journals (Ansari, Rahman, and Hashem Hussein Al-Attas, 2020), do not appear to have been adopted by the NSLC.

Group difference test results in EWJL 2020
Key academic indicators	*P-values*	Kruskal-Wallis statistic/F- statistic	*P-values*			Mean rank diff.			Sample size
Key academic indicators	*P-values*	Kruskal-Wallis statistic/F- statistic	Low-level vs. Mid-level	Low-level vs. High-level	High-level vs. Mid-level	Low-level vs. Mid-level	Low-level vs. High-level	High-level vs. Mid-level	Sample size
A1	0.0005***	15.10	0.0008***	0.0583	>0.9999	17.64	18.97	1.327	62
A2	<0.0001****	21.73	<0.0001****	0.0475*	>0.9999	21.82	19.58	-2.238	62
A3	0.0049**	10.65	0.0127*	0.0563	>0.9999	13.79	19.07	5.286	62
A4	0.0807	5.034	0.6647	0.0924	0.4553	-5.893	-17.52	-11.63	62
A5	<0.0001****	24.79	<0.0001****	0.0163*	>0.9999	23.00	22.57	-0.4286	62
A6	<0.0001****	24.02	<0.0001****	0.0006***	>0.9999	-22.83	-30.26	-7.428	62
A7	0.6901	0.3771	0.7733	0.9884	0.9926	-0.3428	-0.1825	0.1603	62
A8	0.5306	1.268	0.7807	>0.9999	>0.9999	-5.429	-2.714	2.714	62
A9	0.0025**	12.01	0.0072*	0.0379*	>0.9999	-14.64	20.24	5.595	62
A10	<0.0001****	27.07	0.0002***	<0.0001****	0.1551	-18.48	-33.51	-15.02	62
Group difference test results in EWJL 2021
Key academic indicators	*P-value*	Kruskal-Wallis statistic/F- value	*P-value*			Mean rank diff.			Sample size
Key academic indicators	*P-value*	Kruskal-Wallis statistic/F- value	Low-level vs. Mid-level	Low-level vs. High-level	High-level vs. Mid-level	Low-level vs. Mid-level	Low-level vs. High-level	High-level vs. Mid-level	Sample size
A1	0.0002****	16.80	0.0778	0.0002***	0.0451*	8.333	21.20	12.87	35
A2	0.0003***	13.72	0.0001***	0.1387	0.4765	15.92	10.34	-5.579	35
A3	0.0010***	13.82	0.2158	0.0007***	0.0483*	6.733	19.47	12.73	35
A4	0.1419	3.905	0.2866	>0.9999	0.3409	6.233	-2.133	-8.367	35
A5	0.0071**	9.895	0.0486*	0.0181*	0.8871	9.000	14.53	5.533	35
A6	0.0060**	10.22	0.0085**	0.1008	>0.9999	-10.37	-10.43	-0.06667	35
A7	0.3437	2.136	0.9813	>0.9999	0.5209	3.667	-3.533	-7.200	35
A8	0.1221	4.206	0.1267	0.9773	>0.9999	7.600	5.200	-2.400	35
A9	0.0001***	17.78	0.0646	0.0001***	0.0378	8.600	21.80	13.20	35
A10	0.0021**	13.32	0.0043**	0.0375*	>0.9999	-10.57	-11.70	-1.133	35

Note. A1: number of articles in the journal; A2: degree of internationalization; A3: APCs; A4: JCI; A5: self-citation rate; A6: number of retractions; A7: JIF; A8: article influence; A9: number of OA articles; A10: number of retractions by Chinese authors. Due to key academic indicators A2 in 2020 and A7 in 2021 following a normal distribution, we employed parameter tests and reported the F-statistic.

*P<.05, ** P<.01, *** P<.001, **** P<.0001

Group comparison based on flow status

In the context of flow status analysis, a comparative grouping was conducted, and the results are presented in Table 5. (1) The results reveal significant differences between the added group before (2020) and during (2021) the warning period. Several key academic indicators, such as the number of articles in the journal (P<0.0001), degree of internationalization (P=0.0055), APCs (P<0.0001), article influence (P=0.0005), number of OA articles (P=0.0001), and the number of retractions by Chinese authors (P=0.0312), exhibit statistically significant variations between the pre-warning and warning periods. (2) In the two consecutive years of continuous warnings, the retained group did not demonstrate any significant differences in key academic indicators. Except for the number of retractions (P=0.0469), which showed a statistically significant distinction, other key academic indicators displayed no substantial variations between the two consecutive years of continuous warnings. (3) The excluded group showcased relatively weaker disparities before and during the warning period. While the self-citation rate (P<0.0001) of the excluded group demonstrated statistically significant differences between the pre-warning and warning periods, the differences in the number of articles in the journal (P=0.0112), APCs (P=0.0083), and the number of OA articles (P=0.0116) were comparatively weaker. Furthermore, there were no statistically significant differences in the key academic indicators between the two periods. The findings shed light on the following aspects. (1) The criteria for including or excluding academic journals in the EWJLs may lack uniformity. Notably, indicators such as the degree of internationalization, self-citation rate, article influence, and the number of retractions by Chinese authors displayed varying degrees of disparity between the included and excluded academic journals during their flow. (2) The selection of the retained group appears reasonable. Academic journals that were retained in the warning journal list did not exhibit significant differences in key academic indicators of quality and misconduct over two consecutive years, reaffirming the importance of continued attention to these journals. (3) It is plausible that JIF and JCI were not considered as criteria for assessing flow, as none of the three groups exhibited notable differences during flow or retention, based on these two key academic indicators.

Table 5. Group difference test results based on mobility status

Key academic indicators	*P-values*			Sum of signed ranks (W)/ t- *statistic*			Sample size
Key academic indicators	Added group 2020 vs. 2021	Retained group 2020 vs. 2021	Excluded group 2020 vs. 2021	Added group	Retained group	Excluded group	Added group	Retained group	Excluded group	Full sample
A1	<0.0001****	0.3594	0.0112*	289.0	-17.00	568.0	52	18	106	176
A2	0.0055**	0.4682	0.6107	3.040	0.7614	0.5121	52	18	106	176
A3	<0.0001****	0.4258	0.0083**	271	-15.00	541.0	52	18	106	176
A4	0.7217	0.3750	0.2116	-29.0	-16.00	-260.0	52	18	106	176
A5	0.3276	0.3954	<0.0001****	-79.0	17.00	1029	52	18	106	176
A6	0.0752	0.0469*	0.2378	46.00	29.00	145.0	52	18	106	176
A7	0.1179	0.4397	0.1009	1.620	0.8131	1.670	52	18	106	176
A8	0.0005***	0.2031	0.1987	-229.0	-23.00	-260.0	52	18	106	176
A9	0.0001***	0.4258	0.0116*	267.0	-15.00	518.0	52	18	106	176
A10	0.0312*	0.0664	0.3375	32.0	32.00	99.00	52	18	106	176

*P<.05, ** P<.01, *** P<.001, **** P<.0001

We analyzed the appropriateness of using EWJL 2020 and EWJL 2021, published by NSLC, as a means to ensure research quality and integrity. Previous studies have primarily focused on creating academic journal lists to identify mainstream academic journals (Safón and Docampo, 2023) or analyzing the overlap, purposes, and researchers’ perceptions of different academic journal lists (Chen, 2019; Adams and Johnson, 2008). There has been ongoing debate, especially concerning the so-called “the lists of questionable journals”, but these debates lack empirical validation.

To address the limitations of existing studies, we examined whether groups with different warning levels within EWJLs exhibit differences in key academic indicators. To address RQ1, we studied the significant differences in specific key academic indicators among the following groups: low-warning-level vs. mid-warning-level, low-warning-level vs. high-warning-level, and high-warning-level vs. mid-warning-level. We found only a few differences between the mid-warning-level and high-warning-level groups. Citation indicators in scientometrics (JIF and JCI) did not show differences across the three groups with EWJL 2020 and EWJL 2021. Indicators related to research integrity, such as the number of retractions by Chinese authors, exhibited significant differences between the low-warning-level and high-warning-level groups, and the low-warning-level and the mid-warning-level groups. This suggests that NSLC’s grouping of warning journals may have considered the academic journals' performance in terms of research integrity. However, even so, the assignment of different warning levels to academic journals by NSLC lacks stronger persuasion and scientific rigor. After all, the inconsistent differences among key academic indicators across different warning levels raise suspicions. As highlighted in a previous study (Tsigaris and Teixeira da Silva, 2021), despite the “ideal” nature of the lists of questionable journals, the biases in its formulation process, especially regarding standards, have been overlooked. Hence, if academic journals on the EWJLs fail to exhibit consistent differences in key academic indicators, maintaining the reasonableness for EWJLs becomes significantly challenging.

We also examined whether different academic journals in EWJLs exhibited differences in key academic indicators during their flow. To address RQ2, we categorized academic journals in EWJL 2020 and EWJL 2021 into “added group”, “excluded group”, and “retained group” based on their flow status. We analyzed differences in key academic indicators for these groups in 2020 and 2021. We found differences in six key academic indicators for the added group, one key academic indicator for the retained group, and four key academic indicators for the excluded group. Similarly, citation indicators in scientometrics (JIF and JIC) did not exhibit differences across the three groups. Conversely, a previous study has emphasized the necessity of citation content analysis in identifying questionable journals and non-questionable journals (Emmanuel et al., 2021). If the lists of questionable journals fail to reach consensus on inclusion and exclusion criteria, it becomes difficult to have confidence in their reasonableness. This is because if academic journals do not exhibit differences in research quality or integrity indicators before and after inclusion or exclusion, then there is a lack of reason to include or exclude them (they do not exhibit a tendency to improve their research quality or alter their misconduct behavior status). Furthermore, if the lists of questionable journals decide not to adopt traditional scientometric indicators, the reasons should be elucidated and explained, rather than merely presenting a final result to the academic community.

In some countries, the lists of questionable journals serve not only as a tool but also as a policy to ensure research quality and integrity (Cyranoski, 2018; Bagues, Sylos-Labini, and Zinovyeva, 2019). However, while using the lists of questionable journals to address issues like low quality and research misconduct, the processes of creating these lists and establishing their standards should also undergo reflection and critical examination. As policies that can have a broad impact on the academic community, the formulation and promotion of the lists of questionable journals require consideration of various factors, including reasonableness, transparency, accountability, and potential ethical issues that the lists might introduce, such as stigmatization.

One key step in addressing these issues is to ensure that the standards for compiling the lists of questionable journals are reasonable. Using NSLC’s published EWJLs as an example, our study found that EWJLs may adopt inconsistent criteria for including, excluding, or retaining academic journals, as well as for assigning warning levels to them. Although the issues with EWJLs might be isolated cases, they could also potentially reflect broader problems with the lists of questionable journals. Therefore, we should approach the lists of questionable journals with greater skepticism and exercise caution regarding its reasonableness, as we cannot ensure consistent treatment for all academic journals warned or labeled. Our recommendations are as follows. Firstly, those compiling the lists of questionable journals should assess whether academic journals exhibit differences in key academic indicators before and after inclusion, exclusion, or retention, in order to determine whether there is a tendency to improve reserch quality or address academic misconduct behavior. When the lists of questionable journals updating the status of the questionable journals, it might be worth considering the use of more extended-term data or even reducing the frequency of update, to yield more robust outcomes. Secondly, the creators of the lists of questionable journals should openly share their criteria, detailed data, and assessment methods, thereby increasing the confidence of the academic community in such lists. Thirdly, involving diverse expert groups in the creation process, including representatives from various disciplines and research institutions, will bring forth a broader range of critical perspectives, thereby enhancing the reasonableness of the formulation of the lists of questionable journals. This will help reduce biases and ensure comprehensive evaluation.

Ultimately, our goal should be to establish a robust and reasonable system that supports the enhancement of research quality and integrity while minimizing potential negative impacts on researchers and academic journals due to inclusion or exclusion from the lists of questionable journals.

Limitations and Future Directions

Taking EWJLs as an example, we examined the reasonableness of its internal grouping. However, attention to its applicational reasonableness remains lacking. In other words, while universities or research institutions hope that lists of questionable journals can play a constructive role in ensuring research quality and integrity, there is still a dearth of critical thinking and evidence regarding potential ethical risks and the possible negative impact on researchers and academic publishers when implementing them. Therefore, future research could focus on this aspect, exploring the potential ethical concerns of the lists of questionable journals through qualitative studies.

Based on the principles of typology, we believe that the compilers of the lists of questionable journals, when categorizing and labeling academic journals, which can also be seen as endowing academic journals with new concepts or symbols, should carefully consider whether the bibliometric attributes (key academic indicators) of the designated categories exhibit inter-group differences, and whether these differences are consistent. This is a crucial factor in determining the reasonableness of such a list. This is because entities can only be classified into different categories when there are fundamental differences between them (Podsakoff, MacKenzie, and Podsakoff 2016). Building upon this perspective, we utilized the EWJLs published by NSLC as a case study to scrutinize its reasonableness in categorizing warning levels and making decisions about excluding, including, and retaining academic journals. This effort aimed to provide more comprehensive evidence for research in this domain. Our findings suggest that EWJLs may have employed inconsistent criteria when assigning warning levels. Varied degrees of differences or the absence of differences were observed among groups across different key academic indicators. Consequently, the formulation of EWJLs might lack reasonable and requires careful consideration of category criteria. Furthermore, widely-adopted citation metrics such as JIF and JCI seem to have been disregarded by EWJLs as evaluation criteria, though the organization has not furnished any clarification or justification on this matter. In light of these revelations, creators of such lists must adopt a more proactive stance by openly sharing their evaluation standards and furnishing comprehensive data to enhance the accessibility and transparency of list-related information. This endeavor can elevate the rationale behind these lists and foster greater confidence in their reasonableness. As questionable journals continue to emerge, we are becoming increasingly aware of the imperative to embark on a path of transformation, adopting more responsible approaches in devising quality assurance and research integrity policies.

Acknowledgments

We would like to express our gratitude for the constructive suggestions provided by the associate editor and the two anonymous reviewers. Their valuable input has significantly enhanced the quality of this study.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability

The data that support the findings of this study are available at: https://doi.org/10.5281/zenodo.8245329.

Funding

The authors of this study did not receive any funding neither from industry nor from academic institutions.

Ethical Considerations

This article does not contain any studies with human participants or animals performed by any of the authors.

Authors’ Contributions

GYT developed the protocol for the study, conducted the search, extracted, updated, and analyzed the data, interpreted the results, and wrote, revised and edited the manuscript.

JYP conducted the search and extracted the data for some samples, and contributed to the revision of the manuscript.

Adams, Dennis, and Norman Johnson. 2008. “The Journal List and Its Use: Motivation, Perceptions, and Reality.” European Journal of Information Systems 17 (2): 158–62. https://doi.org/10.1057/ejis.2008.8.

Ansari, Khadeeja M N, Safiqur rahaman, and Hashem Hussein Al-Attas. 2020. “Gauging the Quality of Behavioral Science Journals by Using Bibliometric Indicators.” COLLNET Journal of Scientometrics and Information Management 14 (1): 135–52. https://doi.org/10.1080/09737766.2020.1819173.

Bagues, Manuel, Mauro Sylos-Labini, and Natalia Zinovyeva. 2019. “A Walk on the Wild Side: ‘Predatory’ Journals and Information Asymmetries in Scientific Evaluations.” Research Policy 48 (2): 462–77. https://doi.org/10.1016/j.respol.2018.04.013.

Chen, Xiaotian. 2019. “Beall’s List and Cabell’s Blacklist: A Comparison of Two Lists of Predatory OA Journals.” Serials Review 45 (4): 219–26. https://doi.org/10.1080/00987913.2019.1694810.

Cyranoski, David. 2018. “China Awaits Controversial Blacklist of ‘Poor Quality’ Journals.” Nature 562 (7728): 471–72. https://doi.org/10.1038/d41586-018-07025-5.

Dunleavy, Daniel J. 2022. “Progressive and Degenerative Journals: On the Growth and Appraisal of Knowledge in Scholarly Publishing.” European Journal for Philosophy of Science 12 (4). https://doi.org/10.1007/s13194-022-00492-8.

Frandsen, Tove Faber. 2019. “How Can a Questionable Journal Be Identified: Frameworks and Checklists.” Learned Publishing 32 (3). https://doi.org/10.1002/leap.1230.

Jiamusi University. 2023. “Notice Regarding the Publication of Jiamusi University’s Early Warning Journal List.” Jiamusi University. February 1, 2023. https://kjc.jmsu.edu.cn/info/1118/2276.htm.

Kulczycki, Emanuel, Marek Hołowiecki, Zehra Taşkın, and Franciszek Krawczyk. 2021. “Citation Patterns between Impact-Factor and Questionable Journals.” Scientometrics 126 (10): 8541–60. https://doi.org/10.1007/s11192-021-04121-8.

McGrath, Trevor A, Joshua C Bowdridge, Ross Prager, Robert A Frank, Lee Treanor, Ana Dehmoobad Sharifabadi, Jean-Paul Salameh, et al. 2020. “Overinterpretation of Research Findings: Evaluation of ‘Spin’ in Systematic Reviews of Diagnostic Accuracy Studies in High–Impact Factor Journals.” Clinical Chemistry 66 (7): 915–24. https://doi.org/10.1093/clinchem/hvaa093.

MDPI. 2022. “MDPI’s Statement on the ‘Early Warning Journal List (Trial)’ Released by the National Science Library of the Chinese Academy of Sciences (NSLC) on 31 December 2021.” Www.mdpi.com. March 15, 2022. https://www.mdpi.com/about/announcements/3624.

Ministry of Science and Technology of the People’s Republic of China. 2020. “Measures to Eliminate the Bad Orientation of ‘Paper Only’ in Scientific and Technological Evaluation (Trial).” Ministry of Science and Technology of the People’s Republic of China. February 23, 2020. https://www.most.gov.cn/xxgk/xinxifenlei/fdzdgknr/fgzc/gfxwj/gfxwj2020/202002/t20200223_151781.html.

Nature. 2018. “Journal Blacklists: Show Your Working.” Nature 562 (7727): 308–8. https://doi.org/10.1038/d41586-018-07033-5.

Nelhans, Gustaf, and Theo Bodin. 2020. “Methodological Considerations for Identifying Questionable Publishing in a National Context: The Case of Swedish Higher Education Institutions.” Quantitative Science Studies 1 (2): 1–20. https://doi.org/10.1162/qss_a_00033.

Petrou, Christos. 2021. “An Early Look at the Impact of the Chinese Academy of Sciences Journals Warning List.” The Scholarly Kitchen. April 14, 2021. https://scholarlykitchen.sspnet.org/2021/04/14/guest-post-an-early-look-at-the-impact-of-the-chinese-academy-of-sciences-journals-warning-list/.

Podsakoff, Philip M., Scott B. MacKenzie, and Nathan P. Podsakoff. 2016. “Recommendations for Creating Better Concept Definitions in the Organizational, Behavioral, and Social Sciences.” Organizational Research Methods 19 (2): 159–203. https://doi.org/10.1177/1094428115624965.

Safón, Vicente, and Domingo Docampo. 2023. “What Are You Reading? From Core Journals to Trendy Journals in the Library and Information Science (LIS) Field.” Scientometrics 128 (5): 2777–2801. https://doi.org/10.1007/s11192-023-04673-x.

Sandford, Stella. 2019. “From Aristotle to Contemporary Biological Classification: What Kind of Category Is ‘Sex’?” Redescriptions: Political Thought, Conceptual History and Feminist Theory 22 (1): 6. https://doi.org/10.33134/rds.314.

State Council of the People’s Republic of China. 2018. “Several Opinions on Further Strengthening the Construction of Research Integrity.” State Council of the People’s Republic of China. May 30, 2018. https://www.gov.cn/zhengce/2018-05/30/content_5294886.htm.

Strinzel, Michaela, Anna Severin, Katrin Milzow, and Matthias Egger. 2019. “Blacklists and Whitelists to Tackle Predatory Publishing: A Cross-Sectional Comparison and Thematic Analysis.” Edited by Julie M. Wolf. MBio 10 (3). https://doi.org/10.1128/mbio.00411-19.

Tang, Gengyan, and Jingwen Jia. 2022. “Characteristics of Blacklisted Journals: Evidence from Chinese-Language Academic Journals.” Accountability in Research, August. https://doi.org/10.1080/08989621.2022.2112953.

Teixeira da Silva, Jaime A., Mina Moradzadeh, Yuki Yamada, Daniel J. Dunleavy, and Panagiotis Tsigaris. 2023. “Cabells’ Predatory Reports Criteria: Assessment and Proposed Revisions.” The Journal of Academic Librarianship 49 (1): 102659. https://doi.org/10.1016/j.acalib.2022.102659.

Teixeira da Silva, Jaime A., and Stephen Gichuhi Kimotho. 2021. “Signs of Divisiveness, Discrimination and Stigmatization Caused by Jeffrey Beall’s ‘Predatory’ Open Access Publishing Blacklists and Philosophy.” The Journal of Academic Librarianship 48 (3): 102418. https://doi.org/10.1016/j.acalib.2021.102418.

Teixeira da Silva, Jaime A., and Panagiotis Tsigaris. 2018. “What Value Do Journal Whitelists and Blacklists Have in Academia?” The Journal of Academic Librarianship 44 (6): 781–92. https://doi.org/10.1016/j.acalib.2018.09.017.

Torres-Salinas, Daniel, Pilar Valderrama-Baca, and Wenceslao Arroyo-Machado. 2022. “Is There a Need for a New Journal Metric? Correlations between JCR Impact Factor Metrics and the Journal Citation Indicator—JCI.” Journal of Informetrics 16 (3): 101315. https://doi.org/10.1016/j.joi.2022.101315.

Tsigaris, Panagiotis, and Jaime A. Teixeira da Silva. 2021. “Why Blacklists Are Not Reliable: A Theoretical Framework.” The Journal of Academic Librarianship 47 (1): 102266. https://doi.org/10.1016/j.acalib.2020.102266.

You, Taekho, Jinseo Park, June Young Lee, Jinhyuk Yun, and Woo-Sung Jung. 2022. “Disturbance of Questionable Publishing to Academia.” Journal of Informetrics 16 (2): 101294. https://doi.org/10.1016/j.joi.2022.101294.

Zhang, Lin, Yahui Wei, Gunnar Sivertsen, and Ying Huang. 2022. “The Motivations and Criteria behind China’s List of Questionable Journals.” Learned Publishing 35 (4): 467–80. https://doi.org/10.1002/leap.1456.

Appendix.docx

Download PDF

Version 2

posted

You are reading this latest preprint version

Are the Lists of Questionable Journals Reasonable: A Case Study of Early Warning Journal Lists

Status:

Version 2

Abstract

Figures

Introduction

Materials and Methods

Results

Discussion

Conclusion

Declarations

References

Supplementary Files

Status:

Version 2