A Multi-feature Fusion Approach Based on Domain Adaptive Pretraining for Aspect-based Sentiment Analysis

doi:10.21203/rs.3.rs-2620712/v1

Download PDF

Research Article

A Multi-feature Fusion Approach Based on Domain Adaptive Pretraining for Aspect-based Sentiment Analysis

https://doi.org/10.21203/rs.3.rs-2620712/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 28 Nov, 2023

Read the published version in Soft Computing →

You are reading this latest preprint version

Aspect-based sentiment classification (ASC) is a popular task that aims to identify the corresponding emotion of a specific aspect for aspect-based sentiment analysis. Dependency parsing is currently considered as an efficient tool for recognizing the opinion words in the sentiment text. However, many dependency-based methods might be susceptible to the dependency tree and inevitably introduce noisy information and neglect the rich relation information between words. In this paper, we propose a multi-feature fusion approach based on domain adaptive pretraining for ASC and reducing dependency noisy information. We use the Multi-task Learning (MTL) technique for domain adaptive pretraining, which combines Biaffine Attention Model (BAM) and Mask Language Model (MLM) by jointly considering the structure, relations of edges, and linguistic features in the sentiment text. The pretrained dependency graph will be input into a double graph fusion-based message passing neural network (MPNN) that is initialized with the optimal parameters of the pretrained BAM for MPNN training, which fully considers these different features that are affected with each other for ASC. Extensive experiments were made on four benchmark datasets for comparing our approach with the state-of-the-art ASC approaches, and the results show that our model is very competitive in the ASC task compared with the state-of-the-art alternatives.

aspect-based sentiment analysis

dependency parsing

multi-task learning

double graph structure

domain adaptive pretraining

message-passing natural network

Aspect-based sentiment analysis (ABSA) is a fine-grained sentiment analysis task (Dai et al. 2021; Fu et al. 2019;‍ Sun et al. 2019), which aims to extract the emotions of the given one or more aspects in the text. ABSA mainly deals with two tasks including aspect extraction (AE) and aspect-based sentiment classification (ASC). Aspect extraction is to extract the aspect words in the text, while aspect-based sentiment classification is with the purpose of identifying the sentiment polarity of the given aspect in the text (Liu et al. 2022; Zhang et al. 2019). The polarity of a sentence is identified not only by context or opinion words, but also by considering aspect words.

The earlier researches mainly explore the relationship between aspect words and the sentence polarity instead of the structure and relation features in the sentiment text due to the inherent difficulty of syntax parsing tasks (Dai et al. 2021; Barnes et al. 2016). For example, the sentence "the interior of the car is very nice, but the engine is poor". For the aspect word "interior", the polarity of the sentence is positive (nice), but a negative polarity (poor) occurs when considering the aspect word "engine". In some work for ABSA such as Wang et al. (2016) and Jiang et al. (2019), they mainly consider the influence of aspect words on the sentence by using some deep learning models such as the long short term memory model (LSTM), Bidirectional Encoder Representation from Transformers (BERT), and the attention mechanism. But it is difficult for these earlier methods to ascertain which emotion words actually influence the aspect-based polarity of the entire phrase. For example, the sentence polarity of the aspect word in the phrase "interior of the car" may correspond to the emotion word "poor" due to an imprecise classification model, while it actually corresponds to the aspect word "engine".

Syntactic dependency trees have been paid more attention in recent years for addressing the problem mentioned above and further improving the performance of polarity identification in ABSA (Vo and Zhang 2015). By the spatial structure of the dependency tree (DT), dependency-based techniques have demonstrated a very promising potential in significantly enhancing ABSA task performance. Dependency-based approaches such as Brun et al. (2016) and Kampffmeyer et al. (2019) are intended to use graph neural networks (GCN) to build the DT as the input graph data (Kipf and Welling 2016; Sun et al. 2019; Luo et al. 2019) However, dependency tree-based approaches might be susceptible to the noise of DT because some unimportant noisy information may be introduced. A dependency tree with more noise will inevitably offer a risk for the downstream tasks such as ASC. Additionally, most of existing approaches only consider the structure of dependency tree and neglect the relations between words. We give an example from the sentiment dataset as shown in Fig. 1. For the utterance "We also got the added bonus of a monitor, which really helps to extend my screen and keep my eyes fresh!", we can find that "monitor" and "screen" are two aspect words. Their emotional orientations are positive and neutral, respectively. If we only consider the structure of the dependency tree without considering the relationship, the distance between the word "screen" and the phrase "keep my eyes fresh" is closer than that between "monitor" and "keep my eyes fresh". Thus, this will make "screen" tend to be positive instead of neutral. Alternatively, if we observe that the edge relation "conj" exists between "extend" and "screen", then "keep my eyes fresh" cannot be used for judging the emotional orientation of "screen" even if they are nearest. As such, "screen" is kept neutral. It is not difficult to find that relations between words are helpful for improving the performance of ASC.

In this paper, we consider how to effectively utilize the different features such as structure, relations of edges and linguistic features for improving the performance of ASC, which can be reduced to the following two challenging questions: 1) How to fuse the multi-feature information of structure and relations of edges between words for ASC? Most of existing approaches only focus on the structure information and neglect the impact of different relation types of edges between words on ASC. With notable exceptions, a few methods (Tian et al. 2021a; Tian et al. 2021b; Chen et al. 2022; Lin et al. 2022; Zhuang et al. 2022) consider to use multiple features including structure, edge relations and linguistic features for sentiment analysis. However, these few methods often take each of the multiple features as an independent input source, and do not explore how to these various features are actually affected with each other. 2) How to effectively handle the multi-feature noisy information from DT for ASC? Too much noisy information from the dependency tree generated by a tool like spaCy¹ may seriously hamper the performance of dependency-based approaches. It has been reported that direct application of DT is susceptible to noisy information in the dependency tree, which may achieve lower performance compared to using only the flat structure (Tang et al. 2020). We have known that dependency parsing currently deals with 45 kinds of relations of edges between words such as nsubj, amod and conj, and so on. However, few works have been made to pretrain the various relation types for reducing noisy information. We believe that dependency parsing requires an effective way to handle the various relations for reducing the noisy information.

To handle the problems mentioned above, we propose a multi-feature fusion approach based on domain adaptive pretraining for ASC. First, we use the Multi-task Learning (MTL) technique for domain adaptive pretraining, which combines Biaffine Attention Model (BAM) and Mask Language Model (MLM) by incorporating features such as structure, relations and linguistics in the sentiment text for pretraining the dependency graph and reducing the noisy input of the classification model. In the MTL task, BAM is used to iteratively train dependency relations for reducing the noise of dependency graph compared with the static DT tree-based methods because the generated dependency graph can be dynamically updated through back propagation. Second, we further transform the pretrained dependency graph generated by MTL into a matrix representation-based double graph structure for multi-feature fusion. Because syntactic dependency is directional, we combine the double graph matrices with their transpose matrices to generate the directed double graph data. Third, the directed double graph data and the MLM-based words representation is taken as the input of the MPNN model that is initialized with the optimal parameters of the pretrained BAM. During the MPNN training, the dependency in the double graph structure can be continuously enhanced and optimized by deeply fusing the structure, relations and linguistic features for improving the performance of ASC. Extensive experiments were made on four benchmark datasets for comparing our approach, and the results show that our model is very competitive in the ASC task compared with the state-of-the-art alternatives. To the best of our knowledge, this is the first work to jointly use the domain adaptive pretraining-based dependency parsing and the double graph structure-based multi-feature fusion for aspect-based sentiment classification.

The main contributions of this paper are as follows.

• We propose a learnable dependency-enhanced MPNN framework to learn and fuse dependency-based features for ASC.
• A dependency-based double graph fusion model is presented to fully consider the mutual affect between structure and relation features.
• We present an MTL-based domain adaptive pretraining by combining Biaffine Attention and MLM model for efficient dependency parsing.
• Extensive experiments were made on four benchmark datasets for comparing our approach with the state-of-the-art ASC approaches, and the results show that our model is superior to the state-of-the-art alternatives.

[1] http://github.com/explosion/spaCy

Sentiment analysis typically seeks to predict the emotion of a phrase, sentence or document. A fine-grained ABSA task first recognizes the aspect words and then predicts the emotion of the aspect. Early approaches for sentiment analysis generally make use of machine learning techniques to predict the polarity (Jiang et al. 2011; Ding et al. 2015). Later, deep neural network-based techniques are employed to conduct sentiment analysis and have illustrated promising performance. A common treatment is to model the sequence of aspect words and context by using some deep learning models such as long short-term memory model (LSTM) and the attention mechanism. Wang et al. (2016) adopt a combination of aspect embedding and word embedding to get the hidden state through LSTM, and aspect embedding representation is simultaneously input into the attention layer. Li et al. (2022) use graph convolutional networks with hierarchical multi-head attention for aspect-level sentiment classification. Huang et al. (Huang et al. 2018) build an attention-over-attention (AOA) module to calculate the attention weight of the text because the earlier models neglect the interaction between aspect words and the whole sentence. These approaches do not consider to fuse the structure and relations feature in the text, as well as the relationship between aspect words and emotion words. In addition, because the use of word embedding such as Glove and Word2Vec is not very effective in extracting the feature of words, the representation based on the pre-trained model has recently received more attention (Devlin et al. 2021), which can be used for word feature representation and greatly improve the performance of sentiment analysis (Liao et al. 2022; Zhang et al. 2022; Rietzler et al. 2020).

Dependency-based approaches utilize a unique dependency tree structure that considers the relationship between two words for ABSA (Kiritchenko et al. 2014; Brun et al. 2016; Zhou and Law 2022). The syntactic information between the words can be represented via the dependency tree. Dependency tree-enabled sentiment analysis approaches use deep learning-based models to extract the syntactic characteristics, which significantly improves the performance of ABSA tasks (Sun et al. 2019; Ke et al. 2021; Liang et al. 2021). Graph-based dependency parsing and transition-based dependency parsing are the two main categories of dependency parsing. For graph-based techniques, the problem comes into being because the flat representation cannot efficiently capture the structure feature in dependency tree. In contrast, Graph Neural Networks (GNNs) has powerful capacity to directly model non-Euclidean data like dependency tree. In order to extract features from both linear and spatial word representations, Graph Neural Networks (GCN) (Kipf and Welling 2017), are used in modeling dependency tree, where the dependency tree is perceived as an undirected dependency tree to build a symmetry adjacency matrix for GCN (Zhang et al. 2019; Sun et al. 2019). Dai et al. (2021) introduced an induced tree structure based on the pre-trained model instead of dependency tree for sentiment analysis, and the experimental result also shows that pre-trained models can optimize the induced tree in the downstream ABSA tasks. A new aspect-centric tree is built to reduce the distance between aspect words and emotional terms (Zhang et al. 2019). To learn a representation based on GNN, 18 merge a dependency graph with an attention-based graph. Zhou and Law (2022) present a semantic relatedness enhanced graph network for aspect category sentiment analysis. Reinforcement learning-based methods have been employed for aspect-based sentiment classification (Wang et al. 2021). What is worth noting is that most of existing dependency-based approaches make use of the structure features and neglect the relations between words. A dependency tree actually contains a wide variety of relationships like ‘nsubj’, ‘amod’, and ‘dobj’. There are also some methods to further examine the application of a certain relation on dependency tree (Tian et al. 2021a; Tian et al. 2021b; Chen et al. 2022). Chen et al. (2022) use a multi-channel graph structure to represent structure, relation, distance and linguistic features in dependency tree, but they failed to analyze how to fuse these different kinds of features and how these features are affected with each other. There is also some work to consider the structure and relation features in a unified framework (Tian et al. 2021a; Tian et al. 2021b). However, static dependency tree-based approaches often suffer from the noise residing in the dependency tree and thereby consequently hinder the further improvement of ASC performance.

In this paper, different from the existing dependency-based aspect-based sentiment classification, jointly use the domain adaptive pretraining-based dependency parsing and the double graph structure-based multi-feature fusion for aspect-based sentiment classification. We extend the framework and model architecture and supplement extensive experimental evaluations and analysis based on the previous work (Ma and Pang 2022). An MTL method for domain adaptive pretraining is adopted to generate a pretrained dependency graph as the input of MPNN training, which can reduce more noisy information than the static dependency graph without using MTL. During MPNN training, our multi-feature fusion with double graph structure will be continuously updated and optimized by learning BAM for reducing the noisy information and contributing to efficient ASC.

3.1 Framework of method

In this section, we propose a learnable dependency-enhanced deep learning framework to learn and fuse dependency-based features for ASC. Instead of directly using a dependency tree generated by syntax parsing tool like spaCy, our framework can learn dependency-based features for improving the performance of ASC. Because a DT generation tool may not articulate precise relationships in dependency tree, much noisy information might be produced and therefore consequently hinders the improvement of ASC performance.

We illustrate the framework of our method, which is shown in Fig. 2. The framework can be roughly divided into three stages: the multi-task learning stage, the graph features fusion stage, and the representation learning and classification. In the first stage, we use the multi-task learning technique based on the encoder and decoder models for domain adaptive pretraining, where dependency parsing is made to learn multi-feature information including structure, relations of edges, and linguistic features. The Encoder is the word embedding model, which takes the tokens as input and outputs the token embedding Z. Then, Mask Language Model is applied to the encoder for iterative model training and reducing the noisy information of word embedding representation. The token embedding Z is further fed into the decoder. Decoder is a dependency parsing model, which takes the hidden state Z as input, and outputs the predicted dependency graph ${G}({A},{X},{R})$, where A,${X}$ and ${R}$ respectively denote the structure feature of dependency tree, the feature of word nodes, and the features of dependency relations. After the new dependency graph is trained, the relevant parameters of the encoder and decoder are also saved for the subsequent processing. The details about how to perform the MTL-based domain adaptive training will be discussed in section 3.3.

In the graph feature fusion stage, these different graph features including structure and relations of edges between words will be effectively fused for message passing neural network-based model training. We propose a learnable dependency-based double graph structure to deeply fuse these different graph features, through which the pretrained dependency graph ${G}({A},{X},{R})$ generated in the previous stage will be transformed into a multi-feature fusion-based double graph structure ${G}(\tilde{{A}}, \tilde{{X}}, \tilde{{R}})$, where $\tilde{{A}}$, $\tilde{{X}}$ and $\tilde{{R}}$ are respectively the structure feature, the node feature, and the relation features after multi-feature fusion. These fused graph features and the token embedding Z will be fed into the message passing neural network to train an ASC classification model. The detail about double graph structure-based feature fusion and MPNN-based model training will be discussed in the sections 3.4.

3.2 Architecture of model

In this section, we specifically discuss the architecture of model in our method, which is shown in Fig. 3. The architecture of model mainly includes three parts: the domain adaptive pretraining, the fusion of structure and relations, and the multi-feature fusion-based MPNN for ASC.

As mentioned in the previous section, the multi-task learning (MTL) technique is applied for domain adaptive pretraining because it can effectively extract word semantic information and therefore reduce the noisy information in the dependency graph to be produced. The BERT model as the encoder for word representation. We further use the Mask Language Model (MLM) to pretrain the BERT model over the ABSA data sets, and the Biaffine Attention Model (BAM) as the decoder for dependency parsing. MLM and BAM are together trained in an MTL task to learn the structure and relations features for word representation and dependency parsing. Through the MTL for domain adaptive pretraining, we can get a pretrained dependency graph with less noisy information that can be transformed into the sparse score matrices of the structure graph and the relation graph. Both of them are fused and form the double graph structure. The double graph-based message-passing neural network (MPNN) is applied to incorporate the structure feature, the relation features of edges, and linguistic feature for improving the performance of ASC. During the MPNN training for ASC, our model can continuously learn and enhance the structure, relation and linguistic features by the double graph structure, and therefore improve the feature representation for ASC. The related experiments were also made for evaluating our method, which will be discussed in section 4.

3.3 MTL-based domain adaptive pretraining

The application of BERT model has shown its excellent efficiency in embedding representation by using a pretrained model. In order to further enhance feature representation of the sentiment text and reduce the noisy information of dependency parsing, we use the Mask Language Model (MLM) (Devlin et al. 2021) to further train the BERT model on the sentiment text. The enhanced pretrained feature is input to the biaffine attention parser to learn the structure feature and relations features for generating a dependency graph with less noise compared with a static dependency tree produced by a tool. The dependency parsing task and the MLM task will be combined as an MTL task. Unlike some previous studies such as Chen et al. (2022) that use the multi-channel technique to train different graph features independently, our method considers the interactions between these various features including structure, relations and linguistic features. Multi-task learning contributes to deeply fusing these different kinds of features affected with each other in a unified framework of domain adaptive pretraining.

Biaffine Attention (Dozat and Manning 2017; Bekoulis et al. 2018) is a popular graph-based dependency parsing model. method. It can handle the multi head selection by promptly splicing the head and tail tokens of the entities without allowing for any information interaction. Biaffine attention has illustrated very promising potential in graph syntax analysis (Dozat and Manning 2017) compared with some other models like a single affine model plus MLP.

Suppose that the sentence structure is denoted as ${H}=\{{{h}}_{0},{{h}}_{1},\dots ,{{h}}_{n}\}$, where n is the number of words in the sentence. The dependency structure-based biaffine attention ${Biaffine}_{arch}\left({H}\right)$ can be defined by following equations.

$${{h}}_{i}^{\left(arch-dep\right)}={MLP}^{\left(arch-dep\right)}\left({{h}}_{i}\right)$$

$${{h}}_{j}^{(arch-head)}={MLP}^{\left(arch-head\right)}\left({{h}}_{j}\right)$$

$${{s}}_{i}^{\left(arch\right)}={{W}}^{(arch-head)}{U}{{h}}_{i}^{(arch-dep)}+{{W}}^{(arch-head)}{b}$$

where ${{h}}_{i}\in {\mathbb{R}}^{m\times 1}$, m is the dimension of word embeddings, ${{W}}^{arch-head}\in {\mathbb{R}}^{n\times m}$ is a learnable transformation matrix. The notions ${U}\in {\mathbb{R}}^{m\times m}and {b}\in {\mathbb{R}}^{m\times 1}$ are respectively the trainable weights and the biases. The resulting score matrix ${{s}}_{i}\in {\mathbb{R}}^{n\times 1}$ is used as the input to the graph neural network.

As shown in the example of Fig. 1, the relations between words are important for high-quality ASC. The Biaffine model in the earlier research was mainly used for parsing dependency structure rather than relationship parsing (Kipf and Welling 2017; Sun et al. 2019; Luo et al. 2019; Tang et al. 2020), which may generate a dependency tree with more noisy information and further impede the downstream task such as sentiment classification. We argue that the dependency parsing that fuses the structure feature and the relations feature between words will contribute to lower noise in comparison to the static dependency tree produced by other tools. In this paper, we extend the dependency structure parsing for relationship parsing. The biaffine attention ${Biaffine}_{rel}\left({H}\right)$ for dependency relations parsing is defined in Eq. (4).

$${Biaffine}_{rel}\left({H}\right)={\bigcup }_{rRS}{\{Biaffine}_{arch}^{r}\left({H}\right)\}$$

where RS is the set of dependency relation types of edges between words, such as conj, prep, nsubj, probj, and so on. There are currently 45 kinds of dependency relation types in the set RS. Each ${Biaffine}_{arch}^{r}\left({H}\right)$ is a dependency-based biaffine attention for a given dependency relation type r.

Given a sentence in the form of $\{{w}_{0},{w}_{1},...,{w}_{n}\}$ as input, MLM is further applied to the BERT model for domain adaptation. Then, we can obtain the word embedding representation ${H}=\{{{h}}_{0},{{h}}_{1},\dots ,{{h}}_{n}\}$. Taking the result ${H}$ of the MLM-based pretrained model as input, we use the architecture biaffine and relation biaffine to obtain the score matrices as follows.

$${{S}}_{arch}={Biaffine}_{arch}\left({H}\right)$$

$${{S}}_{rel}={Biaffine}_{rel}\left({H}\right)$$

where ${{S}}_{arch}\in {\mathbb{R}}^{n\times n}$ is the structure feature, and ${{S}}_{rel}\in {\mathbb{R}}^{n\times n\times \left|RS\right|}$ is the relation features for all the edge relation types in ${{S}}_{arch}$.

During domain adaptive pretraining, we do not use a spanning tree algorithm to generate a tree structure. Instead, in order to simplify the training process of dependency parsing, once we obtain the score matrices, we directly feed the score matrices into a softmax normalization layer to yield the most likely head node for each node, which is shown in Eq. (7).

$${{P}}_{arch}=softmax\left({{S}}_{arch}\right)$$

We use the cross-entropy loss to straightly obtain the tree structure on ${S}^{arch}$, where for all ${{p}}_{i}^{arch}$ in ${{P}}_{arch}$, the structure loss ${Loss}_{arch}$ is defined in Eq. (8).

$${Loss}_{arch}=-{\varSigma }_{i\in V}log{ {p}}_{i}^{arch}$$

where V denotes the set of tokens in a sentence (except for the ‘[ROOT]’ node).

Similarly, we use the cross-entropy loss to the softmax normalization layer to produce the relations feature, which is defined in equations (9) and (10).

$${{P}}_{rel}=softmax\left({{S}}_{rel}\right)$$

$${Loss}_{rel}=-{\varSigma }_{i\in RS}log{ {p}}_{i}^{rel}$$

where ${{P}}_{rel}$ is a probability matrix of relations feature consisting of the feature${ {p}}_{i}^{rel}$ for each relation $i\in RS$.

During the dependency parsing based on domain adaptive pretraining, the overall loss is a combination of structure loss, relations loss and the MTL loss, which is shown in Eq. (11).

$$Loss= {Loss}_{arch}+{Loss}_{rel}+{Loss}_{MLM}$$

where ${Loss}^{MLM}$ is the cross-entropy loss for the MLM task.

3.4 Double graph fusion of structure and relations

After the MTL-based train process is finished, we obtain both structure feature ${{S}}_{arch}$ and relations feature ${{S}}_{rel}$. They will be deeply fused by a double graph feature fusion method. The fused double graph data is input into the message passing neural network (MPNN) for ASC.

For dependency feature fusion, the structure graph ${{S}}_{arch}$ can be regarded as an adjacency matrix ${{A}}_{arch}$, and the relation graph ${{S}}_{rel}$ can be converted as a relation graph matrix ${{A}}_{rel}$ by the MLP operation as follows.

$${{A}}_{rel}=MLP\left({{S}}_{rel}\right)$$

where ${{A}}_{rel}\in {\mathbb{R}}^{n\times n}$, and ${{S}}_{rel}\in {\mathbb{R}}^{n\times n\times \left|RS\right|}$.

In dependency parsing, the dependency between two words is often directional. For example, an aspect word has a specific emotion represented by an emotion word; an emotion word can be related to an aspect word. In this paper, we use an adjacency matrix and its transpose to distinguish the two directions between head-dependent and dependent-head pairs in dependency graph. The structure and relations features are respectively represented by adjacency matrices ${{A}}_{arch}$ and ${{A}}_{rel}$, and their transposes ${{A}}_{arch}^{T}$ and ${{A}}_{rel}^{T}$. They can be fused into a dependency enhanced graph ${{A}}_{head}$ with transpose ${{A}}_{dep}$, which is shown in equations (13)-(14).

$${{A}}_{head}={{A}}_{arch}+{{A}}_{rel}$$

$${{A}}_{dep}={{A}}_{{a}{r}{c}{h}}^{T}+{{A}}_{rel}^{T}$$

where all the matrices are ${\mathbb{R}}^{n\times n}$ matrices.

The double graph data ${{A}}_{head}$ and ${{A}}_{dep}$ is subjected to row normalization operation, and then is input into the MPNN together with the pretrained tokens embedding for ASC. During MPNN training, the pretrained dependency graph will be further be trained and updated by learning BAM for continuously producing an optimal dependency graph with less noisy information.

Specifically, the token embeddings from the MLM pretraining model is denoted as ${{H}}^{0}$, which is directly fed into the MPNN graph neural network as node representations. We use bidirectional message transmission (Kampffmeyer et al. 2019) as a layer to gather the final hidden states in order to learn the node representation over both ${{A}}_{head}$ and ${A}_{dep}$, which is shown in equations (14)-(15).

$${{H}}^{l+\frac{1}{2}}=LN(Relu\left({{A}}_{head}{{H}}^{l}{\theta }_{head}^{l}\right)+{{H}}^{0})$$

$${{H}}^{l+1}=LN(Relu\left({{A}}_{dep}{{H}}^{l+\frac{1}{2}}{\theta }_{dep}^{l}\right)+{{H}}^{0})$$

where $LN$ represents layer normalization, ${A}\in {\mathbb{R}}^{n\times n}$ and ${H}\in {\mathbb{R}}^{n\times m}$. The notions ${{H}}^{l+\frac{1}{2}} \text{a}\text{n}\text{d} {{H}}^{l+1}$represent the MPNN forward propagation and backward propagation, respectively.

Because message passing in the graph convolution network (GCN) allows the multi-layer architecture to propagate knowledge to distant nodes in the graph, and every subsequent layer of the GCN to perform extensive Laplacian smoothing, which may dilute the knowledge and thereby consequently decrease performance (Kampffmeyer et al. 2019). The use of our double graph structure in the MPNN network can effectively alleviate the problem of knowledge dilution from distant nodes, which will be validated in the experiment of section 4.

3.5 Aspect-based sentiment classification

To produce the aspect representation ${\gamma }$, the pooling function is applied to the hidden state of aspect tokens. We utilize the final representation of the pooled aspect embedding as the input of classification function. The probability distribution ${p}_{c}$ is then obtained by using the softmax function.

$${{p}}_{{c}}=softmax({{W}}_{p}{\gamma }+{{b}}_{p})$$

where ${{W}}_{p}$ and ${{b}}_{p}$ are both trainable parameters. The loss $\mathcal{l}$ for ASC is defined by the cross entropy as follows.

$$\mathcal{l}=-{\varSigma }_{d\in D}{\varSigma }_{c\in P}log{ {p}}_{{c}}$$

Where D denotes the training dataset, and P denotes the set of all the polarities in ASC.

4.1 Datasets

We employ English Penn Treebank (PTB 3.3.0-SD) dependency datasets to train the encoder (BERT) and decoder (BAM). In order to validate our method, we evaluate the experiments over the following 4 datasets: Laptop14, Rest14, Rest15, and Rest16. Datasets Laptop14 and Rest14 are respectively from SemEval 2014 task 4 (Pontiki et al. 2014); Dataset Rest15 is from SemEval 2015 task12 (Pontiki et al. 2015) and Rest16 is from SemEval 2016 task5 (Pontiki et al. 2016). The datasets are from the two domains about laptops and restaurants. Each of the datasets includes three polarity categories: positive, negative and neutral. The detailed description about dataset statistics is shown in Table 1.

Table 1

Dataset statistics
Dataset		Positive	Neutral	Negative
Laptop14	Number of training set	994	464	870
Laptop14	Number of testing set	341	169	128
Rest14	Number of training set	2164	637	807
Rest14	Number of testing set	728	196	196
Rest15	Number of training set	912	36	256
Rest15	Number of testing set	326	34	182
Rest16	Number of training set	1240	69	439
Rest16	Number of testing set	469	30	117

4.2 Baselines

We listed some state-of-the-art baselines that will be compared with our method. Some brief descriptions about these state-of-the-art ASC methods are shown in Table 2.

Table 2

Description about baselines.
Baseline	Description of method	Reference
ASGCN + BERT	GCN is used to learn the aspect-specific representations in ABSA task. We will compare the result of ASGCN with both Glove and BERT embeddings.	Zhang et al. 2019
SK-GCN + BERT	It implements aspect sentiment categorization by using the syntax and knowledge based on GCN.	Zhou et al. 2020
BERT-BASE + MLP	It uses only BERT base model plus MLP.	Dai et al. 2021
R-GAT + BERT	A reconstruct dependency tree is built by using the GAT network.	Zhou and Law 2022
ACLT	An aspect-centric tree structure is optimized to shorten the distance between aspects and their corresponding aspect words.	Zhou et al. 2021
DGEDT-BERT	It uses an upgraded dual transformer network with a dependency graph to learn from the transformer and graph-based representations.	Tang et al. 2020
TGCN-BERT	A type aware graph is built according to the dependency tree and the edge relationship, and GCN is used for sentence embedding-based representation learning.	Tian et al. 2021a
KVMN-BERT	It uses KVMN to embed the dependency tree structure and relationship category representation of the aspect sub-neighborhood.	Tian et al. 2021b

4.3 Evaluation metrics

We use the accuracy and macro-averaged F1-score as evaluation metrics. Both of them are respectively defined by a ternary classification confusion matrix C, where C is a |P|⋅|P| matrix, and P={positive, neutral and negative} is the set of polarities in the ASC task. Suppose that every element ${m}_{ij}$ in the matrix C denotes the number of instances that originally belong to class ${c}_{i}$ but are classified into class ${c}_{j}$ by the classifier, where for any two natural numbers $i,j\in \left[\text{0,2}\right]$, we have ${c}_{i},{c}_{j}\in P$. Particularly, ${m}_{ii}$ is the number of instances that are correctly classified.

1) Accuracy (ACC): $ACC=\frac{{\varSigma }_{i=0}^{2}{m}_{ii}}{{\varSigma }_{i=0}^{2}{\varSigma }_{j=0}^{2}{m}_{ij}}$ (18)

2) Macro-averaged F1-score (F1): $F1={\frac{1}{3}(\varSigma }_{i=0}^{2}\frac{2*\left({Precision}_{i}*{Recall}_{i}\right)}{{Precision}_{i}+{Recall}_{i}})$ (19)

where for every $i\in \left[\text{0,2}\right]$, ${Prescision}_{i}$ and ${Recall}_{i}$ are defined as follows:

$${Prescision}_{i}=\frac{{m}_{ii}}{{\varSigma }_{j\in \left[\text{0,2}\right],j\ne i}{m}_{ij}}$$

$${Recall}_{i}=\frac{{m}_{ii}}{{\varSigma }_{j\in \left[\text{0,2}\right],j\ne i}{\varSigma }_{k\in \left[\text{0,2}\right],k\ne j}{m}_{jk}}$$

4.4 Hyperparameters settings

In our method, the word embedding based on the MLM pre-training is represented by a 768-dimensional vector. We used the similar settings in dependency parsing and MPNN-based ASC. All trainable parameters are optimized by using the common AdamW (Loshchilov and Hutter 2019) optimizer with an initial learning rate of 0.001. The warm-up ratio is 0.1, the dropout rate is 0.1, and the learning rate is 2e-5. The batch size was set to 16. For the domain adaptive pretraining for dependency parsing, the MTL-based model was trained for 6 epochs, while the MPNN model training for the ASC task was made for 10 epochs in which a two-layered MPNN model was used.

We implemented our method proposed in this paper, and the source codes are publicly available on the GitHub website². During the experiments and evaluation, each of experiments was run five times and we chose the best one from them as the result for the experiment.

4.5 Experimental results and analysis

In this section, we will evaluate our method from different perspectives. First, we will evaluate and analyze the overall results, which is made by comparing our method with the state-of-the-art baseline over the four datasets against the typical performance metrics Accuracy and macro-averaged F1-score. Second, we will make ablation study to validate whether every component in our method is indispensable to the whole method for improving the performance of ASC. Third, we will explore how our double graph MPNN layers make influence on the ASC classification. Four, we observe how the MTL-based domain adaptive pretraining make influence on the ASC classification. At last, we made two case studies by illustrating the effectiveness of our method.

4.5.1 Overall comparison results with baseline methods

We compared our method with the state-of-the-art baseline methods against the performance metrics ACC and F1. The experimental results are shown in Table 3. In Table 3, we observe that our model outperforms all the baseline methods over almost all datasets when we use simple message passing layers with residual connection. This shows that the utilization of double graph can enhance dependency parsing by deeply fusing structure and relation features based on domain adaptive pretraining, which indeed contributes to ASC performance improvement.

Table 3

The overall comparison about accuracy and F1 measures with baselines.
Baseline	Laptop14		Rest14		Rest15		Rest16
Baseline	ACC	F1	ACC	F1	ACC	F1	ACC	F1
ASGCN-BERT	77.90	73.01	83.78	75.02	80.69	62.02	89.99	74.46
SK-GCN-BERT	79.00	75.57	83.48	75.19	83.20	66.78	87.19	72.02
BERT + MLP	78.36	74.16	85.35	78.38	82.16	64.96	89.43	74.20
R-GAT + BERT	78.53	74.63	85.63	78.82	81.61	65.30	90.96	75.26
ACLT	79.68	75.83	85.71	78.44	84.44	72.08	92.15	78.64
DGEDT-BERT	79.80	75.60	86.30	80.00	84.00	71.00	91.90	79.00
TGCN-BERT	80.25	76.92	85.54	78.86	85.07	72.50	91.83	76.86
KVMN-BERT	79.78	76.14	85.98	77.94	84.14	68.49	90.52	73.68
Our method	80.56	76.61	85.98	79.45	85.79	72.11	92.69	80.48

Specifically speaking, the BERT pretraining is a common pretraining model and not task-specific although it has illustrated excellent performance in recent years. Most of these baselines only use the BERT pretrained model, which might be not enough to extract some domain-specific features in the sentiment text for a fine-grained sentiment analysis task. In contrast, our method uses the MTL-based domain adaptive pretraining for high-quality feature extraction and reducing noisy information. On one hand, we use MLM-based BERT pretraining to make the BERT pretrained model be further trained over the sentiment text for extracting fine-grained features. On the other hand, we use the biaffine attention to learn the dependency features of structure and relations, which can further generate more accurate features for dependency graph compared with those methods that do not consider relations features. Some noisy information residing in the text can be filtered out by our MTL-based domain adaptive pretraining.

In additions, we observe that our method has the slightly lower ACC and F1 values than DGEDT-BERT over the dataset Rest14. DGEDT-BERT uses a dependency enhanced dual-transformer network to fuse the transformer-based flat representation and the enhanced dependency graph information by mutual biaffine transformation, which makes DGEDT-BERT effectively fuse the structure feature and linguistic feature for ASC. But our method outperforms DGEDT-BERT over the other datasets possibly because our method considers the relations features and domain adaptive pretraining. We also observe that our method has the slightly lower F1 values than TGCN-BERT over the datasets Laptop14 and Rest15 because TGCN-BERT builds a type-ware graph by considering different relation types between words, but the accuracy of our method is superior to DGEDT-BERT over all the datasets.

4.5.2 Ablation study

In order to conduct the ablation study, our method with full components is denoted as LD2G. We designed five ablation baselines as follows.

1) BERT w/o MLM + MLP: we use the BERT model with MLM-based pretraining on the ABSA datasets, and the MLP for classification.

2) LD2G with static biaffine: We lock the parameters of biaffine layer to observe the effect of a learnable double graph structure.

3) LD2G w/o MTL pretrain: The MTL-based domain adaptive pretraining is removed from LD2G to observe the effect.

4) LD2G w/o rel. graph: The relation graphs for all the edges in dependency parsing are removed to observe the effects.

5) LD2G w/o arch. graph: Structure/Architecture graph in dependency parsing is removed to observe the classification effects.

6) LD2G: it includes all the removed components including MLM-based BERT pretraining on ABSA dataset, learnable biaffine attention, MTL-based domain adaptive pretraining, double graph fusion with architecture graph and relation graphs.

The experimental results are shown in Table 4. From Table 4, we observe that LD2G with full modules overall outperforms the others with incomplete functions over all the datasets against the performance metrics Accuracy and F1 score. First, we observe that LD2G with MTL-based domain adaptive pretraining performs better than BERT w/ MLM & MLP. This shows that the MLM-based BERT pretraining over the ABSA datasets is a kind of flat representation that is not enough to accurately represent the relevant features and reduce the DT’s noisy information, but the MTL with BAM and MLM can effectively capture the various features in dependency parsing and thereby significantly reduce the noise. Second, LD2G with static biaffine has the obviously lower performance than LD2G because the double graph-based MPNN in LD2G can continuously update and optimize BAM for generating the optimal dependency features with less noisy information. In contrast, LD2G with static biaffine parameters does not consider the mutual evolution between the structure/architecture graph and relation graphs to optimize dependency parsing. Third, LD2G with MTL domain adaptive pretraining has obviously much higher performance than LD2G without MTL pretraining, which shows that MTL is crucial to effectively capture dependency features and reduce the noisy information. Four, LD2G outperforms each of the two models: LD2G w/o rel. graph and LD2G w/o arch. This shows that the fusion of double graph data is effective and indeed contributes to performance improvement. All of these show that using learnable dependency-based double graph MPNN indeed contributes to ASC performance.

Table 4

Experiment results about ablation study.
Models	Laptop14		Rest14		Rest15		Rest16
Models	ACC	F1	ACC	F1	ACC	F1	ACC	F1
BERT w/ MLM & MLP	79.15	75.15	85.63	78.56	85.24	71.29	92.21	78.02
LD2G w/ static biaffine	77.36	73.01	84.46	75.79	84.50	68.34	90.09	72.09
LD2G w/o MTL pretrain	78.53	74.03	84.73	78.22	83.39	70.39	91.40	78.98
LD2G w/o rel. graphs	79.93	75.94	85.53	78.24	84.87	67.91	91.23	75.13
LD2G w/o arch. graph	79.62	76.12	85.36	78.24	84.50	65.84	92.21	77.67
Our method (LD2G)	80.56	76.61	85.98	79.45	85.79	72.11	92.69	80.48

4.5.3 Influence of MPNN layers on ASC classification

Because message passing GCN networks are to aggregate the information of adjacent nodes and then propagate the information to distant nodes in the graph by the multi-layer architecture. With the increasing number of MPNN layers, the differentiation of nodes will become very poor, resulting in a series of problems such as gradient disappearance, over-smoothing, and over-fitting. Every layer of the GCN has to perform extensive Laplacian smoothing, which may dilute the knowledge and thereby consequently decrease performance.

In this section, we will discuss how the number of MPNN layers in our LD2G method can make influence on the ASC results, and how many layers are appropriate to achieve superior classification performance. We use different number of MPNN layers from 1 to 12, and conduct ASC evaluation on the datasets Laptop14, Rest15 and Rest16. The experiment results are shown in Fig. 4, where Figs. 4.(a) and 4.(b) are respectively the ASC results for Accuracy and F1-score.

From the Fig. 4.(a), we observe that the accuracy of ASC begins to decline slowly after a very small increase when the number of layers ranges from 1 to 12. On the three datasets, the accuracy of ASC approximately reaches the maximal value when the number of layers is 2. We say that a two-layered MPNN is appropriate to achieve the highest ASC accuracy for our LD2G method, where we not only achieve high accuracy, but also consequently reduce the computational cost due to the decreased MPNN layers. Looking at the Fig. 4.(b), we observe that the F1 score reaches the maximal value when the number of layers is 2 on all the datasets. Considering these results, it is appropriate for our LD2G method to use a two-layered MPNN for ASC.

4.5.4 Influence of MTL epochs on ASC performance

Domain adaptive pretraining is an important component in our LD2G method, which can generate dependency enhanced graph representation with the structure, relations and linguistic features to reduce the noisy information for ASC. In this section, we discuss how the MTL epochs can make an influence on the downstream ASC task. We perform the domain adaptive pretraining at different times number of MPNN layers ranging from 1 to 10. We conduct ASC evaluation on the datasets Laptop14, Rest15 and Rest16 to observe the influence of the number of MTL epochs on ASC performance. The results are shown in Fig. 5, where Figs. 5.(a) and 5.(b) are respectively the ASC results for Accuracy and F1-score.

From the Fig. 5.(a), we observe that the accuracy of ASC on the three datasets fluctuates slightly but basically levels off when the number of MTL epochs ranges from 1 to 10. On the three datasets, the accuracy of ASC approximately reaches the maximal value when the number of MTL epochs is 5 or 6. Taking look at the Fig. 5.(b), we observe that the F1 score of ASC on the datasets Laptop14 and Rest16 basically levels off when the number of MTL epochs ranges from 1 to 10. For the dataset Rest15, the F1 value rapidly grows with the increasing number of epochs until the number reaches 6, and then shows significant fluctuation. From these results, we can find that our MTL-based domain adaptive pretraining is necessary and efficient to achieve high quality dependency parsing contributing to improving the performance of ASC. Meanwhile, it can be made with fewer epochs and therefore maintains lower computational cost.

4.5.5 Case study based on graph visualization

To illustrate the task-oriented learnable dependency graph optimization of our LD2G model, we conduct some case studies by giving two sentence examples and visualizing their adjacency matrices in different phases.

The first sentence example is: "I was very disappointed with this restaurant." As shown in Fig. 6, the subfigures from left to right respectively show the original output of trained dependency parser, the middle state of training process, and the adjacency matrix when the model converges. Lighter color indicates that the words in the location have higher weights. The generated dependency graph structure is continuously optimized with the iterative training of the model. The color of column of word "disappointed" is growing lighter from the left to the right in these subgraphs, which shows that word "disappointed" is important to predict the polarity.

As another example, we consider the sentence "We also got the added bonus of a monitor, which really helps to extend my screen and keep my eyes fresh!", which is analyzed in the section Introduction. The experimental result is shown in Fig. 7. At the beginning (the left subgraph in Fig. 7), we can find that the model will pay more attention to the positive words such as "helps" and "fresh" because their colors of column is relevantly lighter, but these positive words are actually not used to predict the correct polarity of the aspect word "screen". With the continuous training of the model, it gradually learns the relationship between the word "extend" and the aspect word "screen". We can find that the color of column of word "extend" has become more and more lighter, which can tend to correctly predict the neutral polarity for the aspect word "screen". This example illustrates that the relationship features generated by the dependency graph are useful for ASC classification based on our LD2G method.

In brief, from the experiments and result analysis above, we can find that our LD2G method is very competitive to the state-of-the-art ASC baselines and it outperforms all the baseline methods over almost all datasets, which shows that the utilization of double graph can enhance dependency parsing by deeply fusing structure and relation features based on domain adaptive pretraining, and indeed contributes to ASC performance improvement, in particular maintaining lower computational cost.

[2] https://github.com/LLLpc123/LDG

In this paper, we propose a multi-feature fusion-based double graph MPNN framework to learn dependency-based feature for ASC. We presented a dependency enhanced double graph fusion model to fully consider the mutual affect between structure and relation features. An MTL-based domain adaptive pretraining was proposed to combine the Biaffine Attention and MLM model for efficient dependency parsing. We made experiments on four benchmark datasets for comparing our approach with the state-of-the-art ASC approaches, and the results show that our model is superior to the state-of-the-art alternatives.

In the future work, we will combine dependency parsing-based multi-feature fusion with priori knowledge mining for ABSA. In addition, we will explore the fine-grained aspect sentiment triplet extraction (ASTE).

Compliance with Ethical Standards:

Funding: This study was funded by the National Natural Science Foundation of China under Grants (62072450), and the State Grid Corporation of China, Science and Technology Project under Grants (5700-202158177A-0-0-00).

Conflict of Interest: All the authors declare that they have no conflicts of interests to disclose.

Ethical approval: This article does not contain any studies with human participants or animals performed by any of the authors.

Bekoulis G et al. (2018) Joint entity recognition and relation extraction as a multi-head selection problem. Expert Systems with Applications 114:34-45
Barnes J et al. (2016) Exploring distributional representations and machine translation for aspect-based cross-lingual sentiment classification. In: Proceedings of 26th International Conference on Computational Linguistics (COLING 2016): 1613-1623
Brun C et al. (2016) XRCE at SemEval-2016 task 5: Feedbacked ensemble modeling on syntactico-semantic knowledge for aspect-based sentiment analysis. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval 2016): 277-281
Chen H et al. (2022) Enhanced Multi-Channel Graph Convolutional Network for Aspect Sentiment Triplet Extraction. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022): 2974-2985
Dai J et al. (2021) Does syntax matter? a strong baseline for aspect-based sentiment analysis with RoBERTa. In: Proceedings of the Conference, of the North American Chapter of the Association for Computational Linguistics (NAACL 2021): 1816-1829
Devlin J et al. (2021) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019): 4171-4186
Dozat T, Manning CD (2017) Deep biaffine attention for neural dependency parsing. In: Proceedings of 5th International Conference on Learning Representations (ICLR 2017)
Ding X et al. (2015) Deep learning for event-driven stock prediction. In: Proceedings of 24th International Joint Conference on Artificial Intelligence (IJCAI 2015): 2327-2333
Fu X et al. (2019) Semi-supervised aspect-level sentiment classification model based on variational autoencoder. Knowledge-Based Systems 171:81-92
Huang B et al. (2018) Aspect level sentiment classification with attention-over-attention neural networks. In: Proceedings of International Conference on Social Computing, Behavioral-cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation: 197-206
Jiang T et al. (2019) A Position-aware transformation network for aspect-level sentiment classification. In: Proceedings of 2019 International Joint Conference on Neural Networks (IJCNN 2019): 1-8
Jiang L et al. (2011) Target-dependent twitter sentiment classification. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011): 151-160
Ke W et al. (2021) Incorporating explicit syntactic dependency for aspect level sentiment classification. Neurocomputing 456:394-406
Kampffmeyer M et al. (2019) Rethinking knowledge graph propagation for zero-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR2019): 11487-11496
Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: Proceedings of 5th International Conference on Learning Representations (ICLR 2017)
Kiritchenko S et al. (2014) NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In: Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014): 437-442
Lin JW et al. (2022) Multi-channel word embeddings for sentiment analysis. Soft Computing 26(22):12703-12715
Liu M et al. (2022) Self-Attention Networks and Adaptive Support Vector Machine for aspect-level sentiment classification. Soft Computing 26(18):9621-9634
Liao W et al. (2022) Fine-grained attention-based phrase-aware network for aspect-level sentiment analysis. Artificial Intelligence Review 55(5):3727-3746
Li X et al. (2022) Graph convolutional networks with hierarchical multi-head attention for aspect-level sentiment classification. Journal of Supercomputing 78(13):14846-14865
Liang Y et al. (2021) A dependency syntactic knowledge augmented interactive architecture for end-to-end aspect-based sentiment analysis. Neurocomputing 454:291-302
Luo H et al. (2019) Improving aspect term extraction with bidirectional dependency tree representation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27(7):1201-1212
Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. In: Proceedings of 7th International Conference on Learning Representations (ICLR 2019)
Ma Y, Pang Y (2022) Learnable Dependency-based Double Graph Structure for Aspect-based Sentiment Analysis. In: Proceedings of the 29th International Conference on Computational Linguistics (COLING 2022): 7086-7092
Pontiki M et al. (2016) Semeval-2016 task 5: Aspect based sentiment analysis. In: Proceedings of International Workshop on Semantic Evaluation (SemEval 2016): 19-30
Pontiki M et al. (2015) Semeval-2015 task 12: Aspect based sentiment analysis. In: Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015): 486-495
Pontiki M et al. (2014) SemEval-2014 task4: Aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on SemanticEvaluation (SemEval 2014): 27–35
Rietzler A et al. (2020) Adapt or get left behind: Domain adaptation through BERT language model finetuning for aspect-target sentiment classification. In: Proceedings of the 12th Language Resources and Evaluation Conference: 4933-4941
Sun C et al. (2021) Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In: Proceedings of the Conference, of the North American Chapter of the Association for Computational Linguistics (NAACL 2021): 380-385
Sun K et al. (2019) Aspect-level sentiment analysis via convolution over dependency tree. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019): 5679-5688
Tian Y et al. (2021a) Aspect-based sentiment analysis with type-aware graph convolutional networks and layer ensemble. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2021): 2910-2922
Tian Y et al. (2021b) Enhancing aspect-level sentiment analysis with word dependencies. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: 3726-3739
Tang H et al. (2020) Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020): 6578-6588
Vo DT, Zhang Y (2015) Target-dependent twitter sentiment classification with rich automatic features. In: Proceedings of 24th International Joint Conference on Artificial Intelligence (IJCAI 2015): 1347-1353
Wang L et al. (2021) Aspect-based Sentiment Classification via Reinforcement Learning. In: Proceedings of 2021 IEEE International Conference on Data Mining (ICDM 2021): 1391-1396
Wang Y et al. (2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016): 606-615
Zhuang Yin et al. (2022) Implicit sentiment analysis based on multi-feature neural network model. Soft Computing 26(2):635-644
Zhang Y et al. (2022) Towards Unifying the Label Space for Aspect-and Sentence-based Sentiment Analysis. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2022): 20-30
Zhou T, Law KM (2022) Semantic Relatedness Enhanced Graph Network for aspect category sentiment analysis. Expert Systems with Applications 195:116560
Zhou Y et al. (2021) To be closer: Learning to link up aspects with opinions. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021): 3899-3909
Zhou J et al. (2020) SK-GCN: Modeling Syntax and Knowledge via Graph Convolutional Network for aspect-level sentiment classification. Knowledge-Based Systems 205:106292
Zhang C et al. (2019) Aspect-based sentiment classification with aspect-specific graph convolutional networks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019): 4568-4578

Download PDF

Journal Publication

published 28 Nov, 2023

Read the published version in Soft Computing →

Reviewers invited by journal
05 Jun, 2023
Reviewers agreed at journal
28 Mar, 2023
Editor invited by journal
03 Mar, 2023
Editor assigned by journal
01 Mar, 2023
First submitted to journal
24 Feb, 2023

You are reading this latest preprint version

A Multi-feature Fusion Approach Based on Domain Adaptive Pretraining for Aspect-based Sentiment Analysis

Status:

Journal Publication

Version 1

Abstract

Figures

1 Introduction

2 Related Work

3 The Proposed Method

3.1 Framework of method

3.2 Architecture of model

3.3 MTL-based domain adaptive pretraining

3.4 Double graph fusion of structure and relations

3.5 Aspect-based sentiment classification

4 Experimental Evaluations And Analysis

4.1 Datasets

4.2 Baselines

4.3 Evaluation metrics

4.4 Hyperparameters settings

4.5 Experimental results and analysis

4.5.1 Overall comparison results with baseline methods

4.5.2 Ablation study

4.5.3 Influence of MPNN layers on ASC classification

4.5.4 Influence of MTL epochs on ASC performance

4.5.5 Case study based on graph visualization

5 Conclusion

Declarations

References

Status:

Journal Publication

Version 1