Weighted P-Rank Algorithm Based on aHeterogeneous Scholarly Network

The evaluation of scientiﬁc article has always been a very challenging task because of the dynamic change of citation networks. Over the past decades, plenty of studies have been conducted on this topic. However, most of the current methods do not consider the link weightings between different networks, which might lead to biased article ranking results. To tackle this issue, we develop a weighted P-Rank algorithm based on a heterogeneous scholarly network for article ranking evaluation. In this study, the corresponding link weightings in heterogeneous scholarly network can be updated by calculating citation relevance, authors’ contribution, and journals’ impact. To further boost the performance, we also employ the time information of each article as a personalized PageRank vector to balance the bias to earlier publications in the dynamic citation network. The experiments are conducted on three public datasets (arXiv, Cora, and MAG). The experimental results demonstrated that weighted P-Rank algorithm signiﬁcantly outperforms other ranking algorithms on arXiv and MAG datasets, while it achieves competitive performance on Cora dataset. Under different network conﬁguration conditions, it can be found that the best ranking result can be obtained by jointly utilizing all kinds of weighted information.


Introduction
Scholarly impact assessment and ranking have always been a hot issue, which plays an important role in the process of the dissemination and development of academic research [1] [2] [3]. However, it is diffcult to assess the real quality of academic articles due to the dynamic change of citation networks [4]. Furthermore, the evaluation result will be heavily influenced by utilizing different bibliometrics indicators or ranking methods [5]. As early as in 1972, Garfield proposed journal impact factor (JIF) to rank various academic journals [6]. In 1983, Garfield applied the JIF mechanism to rank individual authors [7]. According to the calculation of the H-index [8], Braun et al. [9] introduced journal H-index to indicate the quality of a journal and the number of published articles comprehensively.
• We evaluate the performance of weighted P-Rank method under different conditions by manipulating the corresponding parameters that can be used to structure graph configurations and time settings.
• By introducing the corresponding link weightings in each heterogeneous graph, the performance of the weighted P-Rank algorithm significantly outperforms the original P-Rank algorithm on three public datasets.
The rest of this paper is organized as follows. In Section 2, the details of the heterogeneous scholarly network and the weighted P-Rank algorithm are introduced, respectively. In Section 3, experiments are conducted to analyze how the configurations of the graphs and time settings influence the performance of the proposed algorithm and to further validate link weighting in heterogeneous scholarly network is beneficial to the article ranking. Finally, Section 4 presents the concluding remarks.

Article Ranking Model
In this section, we introduce the proposed article ranking algorithm in detail. Specificially, we first define and describe a heterogeneous scholarly network that is composed of author layer, paper layer and journal layer, and how the different elements in the three layers are linked and interacted. Furthermore, a link weighting method based on P-Rank algorithm is developed to compute the article score in the heterogeneous scholarly network.

Heterogeneous Scholarly Network
A complete heterogeneous scholarly network consists of three subnetworks (i.e., author network, paper citation network, and journal network). There exist three types of edges in the network i.e., undirected edge between the authors and the papers, directed citation edge between the original paper and its citing papers, and undirected edge between the papers and the published journals. As stated in [14], the heterogeneous scholarly graph of papers, authors, and journals can be expressed as the following form: where V P , V A , and V J are the paper nodes, author nodes, and journal nodes in the three layers respectively. E P denotes the citation link in the paper layer, E PA denotes the link between paper and author, and E PJ denotes the link between paper and journal. As shown in Fig. 1a, the paper-author network and paper-journal network are two undirected graphs which can be represented as G PA = (V P ∪V A , E PA ) and G PJ = (V P ∪V J , E PJ ), respectively. In Fig. 1b, by contrast, the paper citation network is a directed graph G P = (V P , E P ), the arrows point in the direction of paper citation: P 4 → P 5 means P 4 cites P 5 . In this work, we assign link weights to the corresponding subnetworks such that the three unweighted graphs can be updated as , in which W P , W PA , and W PJ refer to the link weight in the three graphs, respectively. With link weightings (W P , W PA , and W PJ ) defined in the corresponding G P , G PA , and G PJ , the unweighted heterogeneous scholarly graph G(V, E) becomes which can be verified by substituting W P , W PA , and W PJ into Eq. (1) to get back Eq. (2).
To evaluate articles, authors, and journals more objectively, the proposed algorithm should be predicated on the following assumptions: • Articles are more important if they are cited by many other important publications, and therefore they are more likely to be published on certain top journals [14] [18] [19] [20].
• Articles with high-quality are more likely to be written by the authors with higher-reputation [4] [21].
• Journals would hold a higher impact if the published articles are cited by other important articles (or cited by some scholars with higher reputation) [2] [22].
• The citation relevance between two articles is mainly influenced by text similarity and citation network structure (see Sect. 2.2 for more details) [17].
In the following, we introduce the link weightings in paper citation graph G P (Sect. 2.2), paper-author graph G PA (Sect. 2.3), and paper-journal graph G PJ (Sect. 2.4), respectively.

Link Weighting in Paper Citation Graph (G P )
As discussed in Sect. 1, almost all the ranking algorithms such as PageRank [10], FutureRank [4], and P-Rank [14] only handle article citation in a binary way, rather than paying attention to the combination of citation relevance and link weighting systems. In reality, however, the citation relevance plays a very important role in the process of assessing article quality, and also should be considered in citation network. In this study, we develop a link weighting to assign weight in the paper citation graph (G P ) based on the citation relevance between two papers, which can be utilized to improve the reasonability of the article ranking. In comparison with initial citations, the weighted citations are more advanced at evaluating the real impact of scientific article because it takes full account of certain latent but significant factors in the process of citations. To be specific, the citation relevance (link weighting) between two different papers is mainly influenced by two factors, namely, text similarity (semantic-based) and citation network structure (structure-based). Supposing that the citation relevance between two papers is higher if the two papers are more likely to be similar in semantic and share mutual links and common nodes in the citation network. The comparison between unweighted citation network and weighted citation network is shown in Fig. 2.     [29] to measure the semantic similarity between the titles and abstracts of two papers. Technically, ADW calculates the weighted ranking of the importance of senses in WordNet 3.0 [30] for each lexical item. The relevant experimental results in [17] demonstrated that ADW is not only able to handle the texts with different sizes, but also achieves superior performance on different datasets. In our work, the "slide" weighted overlap approach improved by ADW is employed, which can be used to compute the semantic similarity between the abstracts T i and T j from papers i and j. Let S be the intersection of overlapping senses with non-zero probability in both signatures and r j i be the rank of sense s i ∈ S in signature j, where rank 1 represents the highest rank. The slide overlap Similarity 1 (P i , P j ) can be computed using: where tanh(·) is hyperbolic tangent function, and ∑ |S| i=1 (2i) −1 is the maximum value to bound the similarity distributed over the interval [0,1]. Note that the maxium value would occur when each sense has the same rank in both signatures. Moreover, we normalize parameters α and β such that α+β = 1. It should be explained that the slide ADW will be very helpful to reduce the interdependence of parameters. This is nicely verified by the the performance on the training speed and precision in comparsion to the original ADW.
Cosine similarity and Euclidean disatance are widely used to measure the neighbourhoods nodes in citation network [31] [32] [33]. In this work, we employ cosine similarity to measure the citation relevance of two papers in terms of network structure. Assume that if one paper cites another paper, there exist three types of connections between these two papers, as illustrated in Fig. 3. The cosine similarity between two paper nodes in the citation network can be calculated by: where N P i denotes the neighborhood of node P i , and N P i ∩ N P j denotes the number of nodes that link to both P i and P j .
Based on the Similarity 1 (semantic-based) and Similarity 2 (structure-based), the link weight between two paper nodes in the paper citation graph (G P ) can be represented as follows: where W i, j is the weight from paper i to paper j in G P , Similarity 1 and Similarity 2 are the semantic-based and structurebased similarities between two papers respectively. Parameters λ 1 and λ 2 are two corresponding coefficients, which can be defined as the following form: with µ being a parameter shaping the exponential function, and ε 1 and ε 2 being the media values of Similarity 1 and Similarity 2 respectively. Here let µ = 6 so that those similarity values that exceed the threshold can be constrained by the exponential curve. Parameters λ 1 and λ 2 are normalized as λ 1 + λ 2 = 1.
For a G P with n papers, the adjacency matrix of the citation network can be denoted as an n × n matrix (see Fig. 2), where the link weight between two paper nodes can be calculated by: Let e be the n-dimensional vector whose elements are all 1 and v be an n-dimensional vector which can be viewed as a personalized vector [34]. Next let x(v) paper denote the PageRank vector corresponding to the vectror x(v) paper , and x(v) can be calculated from x = Mx where M = dM + (1 − d)ve T . Thus, PageRank vector x can be computed using: For any given v, PageRank vector x(v) can be obtained from Qv.

Link Weighting in Paper-Author Graph (G PA )
An academic paper with higher quality implicates the innovativeness and the contribution of the authors behind it [2] [3]. In the paper-author graph (G PA ), let P= {p 1 , p 2 , ..., p n } denote the set of n papers and A= {a 1 , a 2 , ..., a m } denote the set of m authors, then G PA can be represented as an n × m adjacency matrix, where the link weight A author i, j from author j to paper i is: However, the vast majority of scientific articles are multi-authored, and author's contributions to each article is also different in most cases [35]. Actually, the concept of authorship has become a hot topic in recent years, which can be attributed to the proliferation of authors over the years [36] [37]. In this study, the link weights in G PA can be deemed as the level of authors' contributions to their articles. Modified Raw Weight (W R, j ) [23] is adopted to assess the authors' contributions according to the relative rankings of authors in co-authored publications. The great advantage of W R, j is that the individual contribution to each paper can be evaluated in a relatively accurate manner, regardless of the total number of authors, and how many publications they have written jointly. The experimental results demonstrated that it achieved many practical successes in different scientific fields such as economic management and bioscience [38] [39]. For the author of rank j the Modified Raw Weight is: where W R, j is the Modified Raw Weight of author j, j is the position of author j in the author list, n is the total number of authors in the paper, and ∑ n j=1 n j is the sum of author positions. Hence, the unweighted G PA can be updated by:

Link Weighting in Paper-Journal Graph (G PJ )
In the initial P-Rank algorithm, the paper-journal graph (G PJ ) can be represented as an n × q adjacency matrix, where n and q are the number of papers and journals, respectively: So far as our knowledge is concrned, the vast majority of article ranking algorithms only consider the link weighting in paper citation graph (G P ) and paper-author graph (G PA ), while the link weighting in the paper-journal graph (G PJ ) is not yet taken into account. From an analysis of experimental results in literature [17], it can be found that there exists a low correlation between paper layer and journal layer since the link weighting is not adopted in G PJ . Clearly, scientific publication and citation relevance need to be studied comprehensively, particularly as a complex network system [40] [41]. To address this issue, we develop a weighted G PJ in which the corresponding link weight can be updated by the journal impact factors [6] [42]. Similar to G PA , the link weights in G PJ can be regarded as the level of journals' impact to the published articles. As one of the most significant scientific evaluation indicators in SCI, journal impact factor (JIF) is computed by the scientific division of Clarivate Analytics, and can be usually utilized for ranking and assessing the grades of various scientific journals in the Journal Citation Report (JCR) database [7]. Notwithstanding JIF first aims at assessing scientific journals, it is now increasingly used to evaluate research and orient publishing strategies of researchers [2] [43]. In [24], Larivière et al. suggested that JIF not only reflects the "quality" of that paper but also that of the journal in which it is published because the possibility that citations to a paper are also influenced by the impact factor of the journal. Here, the "mapminmax" function defined in MATLAB R2018b version is used to normalize the JIF list, the range distributed over the interval [0.1,1]. The formula 13 can thus be rewritten as below:

Weighted P-Rank Algorithm
The weighted P-Rank score of papers can be expressed as x(v) paper in Eq. 9, where the personalized vector is where n p − author represents a vector with the number of publications for each author, and n p − journal represents a vector with the number of publications for each journal. The mutual dependence (intra-class and inter-class walks) of papers, authors, and journals is coupled by the parameters ϕ 1 and ϕ 2 , which are set at 0.5 as default. The weighted P-Rank scores of author and journal can be expressed as: As discussed in Sect. 1, in the real citation network, articles are published and cited in time sequence. Such approaches do not consider the dynamic nature of the network and are always biased to old publications. Therefore, the recent articles tend to be underestimated due to the lack of enough citations. To tackle this problem, Sayyadi and Getoor [4] proposed a time-aware method to achieve a better ranking result in the citation network with time information. In [16] [17], Wang et al. and Zhang et al. employed the time information of article to boost the prestige of recent published articles and thus improve the ranking accuracy. In this study, we also adopt a time weight T i to eliminate the bias to earlier publications, which can be regarded as a personalized PageRank vector. Here according to the time-aware method proposed in FutureRank [4], the function T i is defined as: where T publish denotes the publication time of paper i, and T current − T publish denotes the number of years since the paper i was published. ρ is a constant value set to be 0.62 based on FutureRank [4]. The sum of T i for all the articles is normalized to 1.
Taken together, the weighted P-Rank score of a paper can be calculated by: with parameters γ and δ being constants of the algorithm. (1 − γ − δ ) · 1 n p represents the probability of random jump, where n p is the number of paper samples.
In the proposed algorithm, the initial score of each paper is set to be 1 n p . For articles which do not cite any other papers, we suppose that they hold links to all the other papers. Hence, the sum of x(v) paper for all the papers will keep to be 1 in each iteration. The steps above are recursively conducted until convergence (threshold is set at 0.0001). The pseudocode of the weighted P-Rank algorithm is given in Algorithm 1.

Algorithm 1: Weighted P-Rank Algorithm Based on Heterogeneous Network
Input : G P , G PA , G PJ , JIF list of all journals, and time list of all papers Output : Weighted P-Rank score of paper x(v) paper Parameters : α, β , γ, δ , ρ, λ 1 , λ 2 , µ, ε 1 , ε 2 , d, ϕ 1 , ϕ 2 Steps : 1 Initialize all the scores of papers: x(v) paper = ones(n p ,1) n p , where n p is the number of paper samples 2 Normalize JIF of each journal in dataset: mapminmax[J] ← JIF list (J) 3 Compute and normalize time score of each paper based on Eq. 18: T i = Normalize[e −ρ×(T current −T publish ) ] 4 Update G P by Eqs. 5 and 8: C w ← C 5 Update G PA by Eqs. 11 and 12: A w ← A 6 Update G PJ with Eq. 14: J w ← J 7 while not converging do 8 Eq. 17: Calculate Pagerank(M, v) 12 Update the score of each paper based on time information (Eq. 19):

Experiments
In this section, the weighted P-Rank algorithm is comprehensively evaluated under different conditions and graph configurations. Moreover, extensive experiments are also conducted to analyze the robustness and ROC performance of the proposed method on three different datasets.

Datasets and Experimental Settings
Three public datasets are used in this study, i.e. arXiv (hep-th), Cora, and MAG. The three datasets were chosen since they should be better suited to showing general results. Moreover, the convergence rate and the robustness of the proposed algorithm could be tested on three different datasets. In this study, each article can be featured by seven items, namely article ID, article title, author list, published journal, JIF, publication time, and article score. An example of article list information is illustrated in Fig. 4   All experiments are conducted on a computer with 3.30GHz Intel i9-7900X processor and 64GB RAM under Linux 4.15.0 operating system. The program codes of data preprocessing and graphs modeling are written by Python 3.6.5, which is available on https://github.com/pjzj/Weighted-P-Rank.

Spearman's Rank Correlation
The evaluation of article ranking has always been a very challenging task because the real "quality" of article is difficult to be quantified in practice. Moreover, the ranking results will be strongly influenced by using different ranking indicators [44] [45]. In 2009, Sayyadi and Getoor suggested that the future PageRank score can be regarded as ground truth [4]. However, certain old articles achieved higher PageRank scores since the PageRank algorithm itself is biased to old publications. Wang et al. [16] and Zhang et al. [17] adopted future citation number as the ground truth, and then they evaluated the similarity between the estimated rank and ground truth rank by calculating the Spearman's rank correlation coefficient. In this paper, Spearman's rank correlation coefficient is also used to assess the performance of proposed algorithm under different conditions. For a dataset X = [x 1 , x 2 , ..., x N ] ∈ R D×N with N samples, N original data are converted into grade data, and the correlation coefficient ρ can be calculated by: where R 1 (P i ) denotes the position of paper P i in the first rank list, R 2 (P i ) denotes the position of paper P i in the second rank list, and R 1 and R 2 denote the average rank positions of all papers in the two rank lists respectively.

Robustness
Robustness is an evaluation indicator that can be used to reflect the anti-interference (consistency) ability of system or algorithm. Here according to the corresponding historical time point on three datasets, the whole time on each dataset can be divided into two periods. The time period before the historical time point can be denoted as T 1 , while the whole period can be denoted as T 2 . The robustness of algorithm can thus be measured by calculating the correlation of ranking scores in T 1 and T 2 . This shows that the higher the correlation between two rank lists, the higher the robustness of the algorithm would tend to be.

Graph configurations:
Two parameters can be set in graph configurations: ϕ 1 and ϕ 2 . If ϕ 1 = 0 and ϕ 2 = 0, there exists no coupling, which can be regarded as the situation of ranking the papers using PageRank algorithm only. Of course, if G PA or G PJ is introduced into the network, the parameters can be updated as ϕ 1 = 1, ϕ 2 = 0 or ϕ 1 = 0, ϕ 2 = 1. The final heterogeneous graph is composed of one intra-walk (G P ) and two inter-walks (G PA and G PJ ). By using various combinations of graphs, we compare and assess four different cases of P-Rank algorithm with previous works. The cases and the associated parameters are listed below: • G P (ϕ 1 = 0, ϕ 2 = 0): which is the traditional PageRank algorithm for rank calculation.
• G P + G PA (ϕ 1 = 1, ϕ 2 = 0): A new graph (G PA ) is introduced into the heterogeneous network which only utilizes citation and authorship. • G P + G PJ (ϕ 1 = 0, ϕ 2 = 1): A new graph (G PJ ) is introduced into the heterogeneous network which only utilizes citation and journal information.
• G P + G PA + G PJ (ϕ 1 = 0.5, ϕ 2 = 0.5): Two new graphs (G PA and G PJ ) are introduced into the heterogeneous network which uses citation, authorship, and journal information simultaneously.

Time settings:
Based on whether to use time information, there exist two kinds of settings: • No-Time (δ = 0): which does not utilize article time information to enhance the effect of the recent published articles.
• Time-Weighted (see Eq. 19): which can be used to balance the bias to earlier published articles in the citation network.
With these assumpations, we are now ready to verify Spearman's ranking correlation of different cases on three datasets, as shown in Tabs. [2][3][4][5]. From an analysis of Tab. 2, it can be found that the best performance (arXiv: 0.5449; Cora: 0.3352; MAG: 0.4994) of proposed algorithm is all achieved from the weighted graph configurations as follows: G P + G PA + G PJ . In addition, we note that under the four graph configuration conditions (G P ; G P + G PA ; G P + G PJ ; G P + G PA + G PJ ), an important observation from the experimental results is that weighted graphs significantly outperform unweighted graphs. This result seems to show that article ranking might benefit from link weighting in heterogeneous scholarly network.    The best performance is highlighed in bold.
By comparing and analyzing the data from Tabs results demonstrate that Time-Weighting is beneficial to improve the reasonability of ranking algorithm, which is completely consistent with the conclusion in Ref. [16].
For better comparison, we also measure the performance of the weighted P-Rank and five famous algorithms (PageRank, FutureRank, HITS, CiteRank, and P-Rank) on three datasets by using Spearman's rank correlation and robustness. Note that all the algorithms above are implement with the optimal parameters. In addition, for each dataset, a suitable historical time point is set based on the distribution of papers over a period of time. We see from Fig. 5   To further verify the performance of each ranking algorithm, we also conduct experiments on three datasets by using ROC curve and AUC value. Specifically, we first rank all the articles in the dataset based on PageRank algorithm. We then set a median threshold to hold different numbers of articles. According to the threshold, all articles can be divided into two classes i.e., higher ranked articles (positive objects) and lower ranked articles (negative objects). To make the results more reliable, 10 times of independent experiments are conducted on each test dataset and the average results are obtained (see Fig. 6). It can be seen from Fig. 6 that weighted P-Rank algorithm (as plotted by red curve) significantly outperforms other ranking algorithms on arXiv and MAG datasets, while it achieves competitive performance on Cora dataset. It shows better performance and generalization ability of weighted P-Rank algorithm on three datasets. The AUC vales obtained by weighted P-Rank on arXiv, Cora, and MAG datasets are 0.6733, 0.5586, and 0.6593 respectively. By a sharp contrast, the AUC values achieved by initial P-Rank algorithm are unsatisfactory, especially on arXiv dataset (only 0.3461). This result indicates that link weighting plays an important role in heterogeneous graphs, which will be very helpful to improve the performance of the article ranking algorithm.

Conclusion
Assessing scientific papers is an important but challenging task, mainly due to the dynamic nature and complexity of the heterogeneous scholarly network. This paper developed a weighted P-Rank algorithm based on a heterogeneous scholarly network for article ranking evaluation. The study is dedicated to assigning weight to the corresponding links in G P , G PA , and G PJ by calculating citation relevance (G P ), authors' contribution (G PA ), and journals' contribution (G PJ ). The experiments are conducted on three public datasets (arXiv, Cora, and MAG). The performance of the proposed weighted P-Rank method is measured by using three evaluation metrics (Spearman's ranking correlation, robustness, and ROC curves). Moreover, under conditions of two weighting combinations (Unweighted and Weighted) and four graph configurations (G P , G P + G PA , G P + G PJ , and G P + G PA + G PJ ), the performance of weighted P-Rank algorithm is further evaluated and analyzed. The experimental results showed that the weighted P-Rank method achieved promising performance on three different datasets, and the best ranking result can be achieved by jointly employing all kinds of weighting information. Additionally, we note that the article ranking result can be further improved by utilizing time-weighting information, which is nicely demonstrated by the comparsion results between "Time-Weighted" and the corresponding "No-Time" settings. In summary, it can be found that link weighting in heterogeneous graphs will be very helpful to improve the performance and reasonability of the article ranking method.
In the future, a series of meaningful studies can be conducted subsequently, combining network topology and link weighting. In addition to article ranking algorithms such as P-Rank and normal PageRank, the advantages of various approaches and certain factors that can be weighted should be taken into account. For instance, we would test the effect of link weighting on more ranking methods and verify how the parameters influence the performance of the algorithms. Furthermore, one may consider employing some machine learning methods such as principal component analysis (PCA), linear boundary discriminant analysis (LBDA) and Laplacian score (LS) for further analyzing the correlation between weighted P-Rank score and article citation counts.

Data Availability
The program code will be made available on https://github.com/pjzj/Weighted-P-Rank.