Sentiment Analysis of Arabic Reviews Using a Hybrid Clustering Technique

doi:10.21203/rs.3.rs-1970100/v1

Download PDF

Research Article

Sentiment Analysis of Arabic Reviews Using a Hybrid Clustering Technique

https://doi.org/10.21203/rs.3.rs-1970100/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Sentiment analysis (SA) is the process of assessing the sentiment and attitude of digital audiences toward a range of topics and subjects. The aim of this research is to propose an effective approach for finding good-quality solutions for dialectal Arabic SA problems by addressing inherent challenges in an optimal way. This is achieved by determining the polarities of review texts by using the k-means clustering algorithm in a lexicon-based model and also applying a ML model where necessary in a hybrid approach. In this research, a sentiment lexicon (senti-lexicon) corpus of 3,824 positive and negative words/terms is used in a deep feature extraction process to convert the text into feature vectors. The experimental results showed that the k-means clustering model worked better after separating the observations with relative score values and moving them to be classified using the lexicon-based model. The k-means clustering model part of the hybrid model yielded high-performance results in terms of accuracy, recall, and F1 score metrics, especially in the positive and negative score value features and total score. Each technique has shortcomings, the hybrid model; as the results that are shared will represent; prove that it is an ideal and more flexible solution and approach to conducting SA in an effective and self-improving manner.

Sentiment analysis

lexicon-based

machine learning

k-means clustering

Arabic language

Sentiment analysis (SA), also called opinion mining, is a process that uses natural language processing (NLP) and machine learning (ML) to recognize and classify a piece of text into positive, negative, or neutral (Boudad, Faizi, Thami, & Chiheb, 2017). Sentiment analysis is considered one of the most important research topics in the computer science field (Ahmad, Bakar, Yaakub, & Darwich, 2020). Several online services for SA are now provided by many commercial companies, and these services encompass hotel bookings, fashion, financial services, and politics (Feldman, 2013).

Although a huge number of research articles have been published in the last three years on SA and opinion mining of various texts, including those in the Arabic language, the SA of the Arabic language is still in its early stages and there is scope for further studies (S. Alhumoud, 2019).One of the latest and most comprehensive surveys of SA of the Arabic language was conducted by Al-Ayyoub et al. (Al-Ayyoub, Khamaiseh, Jararweh, & Al-Kabi, 2019), who showed that the studies that have been performed since 2006 with low researches, however, the field witnessed rapid increase lately, nevertheless, it still has many areas to cover (S. R. Ahmad, A. A. Bakar, & M. R. Yaakub, 2019).

Internet content is increasing at an incredible rate, so there have been many attempts to make it easier to search this content by applying tagging as it known by the general public or labeling as it is termed by data scientists. Although this approach is helpful for finding useful information it is only a partial solution. Therefore, researchers have proposed several techniques in order to reduce the size of the problem and to facilitate the process of exploiting the data on the World Wide Web (WWW). However, many challenges are waiting for these techniques. For instance, the data are submitted in an unstructured way, for example, as freestyle text; the authors of the data have different educational, social, and behavioral backgrounds, so many incorrect or misspelled words may be present in the data; and there are many languages to deal with, and each language has many dialects (I. S. Ahmad, A. A. Bakar, & M. R. Yaakub, 2019). These factors make the process of mining WWW data in general and online text data specifically not straightforward but not very hard. Hence different SA techniques and procedures have been developed to make the data more unified and extractable in an automated and professional way, because a manual approach is extremely infeasible (Alrefai, Faris, & Aljarah, 2018).

To date, most SA studies have focused on texts in the English language, so there are many resources such as lexicons and corpora available to assist in this area of research. In contrast, researchers in the field of Arabic language SA are somewhat hampered by a relative shortage of resources, even though there are about 219 million Arabic internet users. Moreover, the application of SA to Arabic language text is relatively more complicated for a number of reasons, including the diversity of dialects that are used in each Arabic-speaking country or even region or city, as well as the limitations of existing resources and tools such as stemmers that are more suitable for use with Modern Standard Arabic (MSA), whereas most Arabic internet users prefer to give their opinions in microblogging and social media websites in their own dialects (Aliane, Aliane, Ziane, & Bensaou, 2016).

This research aims to suggest new methodologies help in solving text mining and SA problems especially on the Arabic language, therefore, new approaches are presented in order to get more accurate results within a reasonable time and to make these approaches applicable to any dataset regardless to its domain. The objectives of this study can be defined as: (1) To develop a new methodology help in finding the text polarity without knowing its class label previously (clustering), as all the previous Arabic Sentiment Analysis (ASA) researches use tools depend basically on the text class label (classification). (2) To develop a hybrid model combines between lexicon-based and machine learning approaches and measure its efficiency in solving and enhancing the solutions of ASA problem. (3) To carry out deep research on the Arabic problems and challenges, and then finding out the best practices in dealing with it in text mining field in particular. (4) To enhance the used dataset and Senti-lexicon Corpus by adding, fixing, and changing some words and phrases

In this study, about 1,200 Arabic reviews and tweets (both positive and negative) from different domains of the Multi-domain Arabic Sentiment Corpus (MASC) are used with an Arabic sentiment lexicon (senti-lexicon) that contains a set of 3,820 positive and negative synsets labeled with many attributes such as parts of speech (POS), polarity scores, dialect synsets and inflected forms (Al-Moslmi, Omar, Abdullah, & Albared, 2017).The MASC dataset and the Arabic senti-lexicon are used together to produce feature vectors. All the published literature on SA for the Arabic language has used the supervised learning approach in order to evaluate the research results. In contrast, in this research, an unsupervised learning model for SA of the Arabic language is presented, namely, the k-means clustering algorithm.

The remainder of this paper is organized as follows: First, an overview of the challenges of the Arabic language in the context of SA is provided in section 2. Next, a review of some related works on Arabic SA is presented in section 3. Then, details of the proposed method are given in section 4. The experiments and results are then discussed in section 5. Finally, a conclusion is drawn and some suggestions for potential directions for future enhancements are offered in section 6.

This section provides a brief explanation of the challenges of the Arabic language and the consequent difficulties encountered in the SA of the language. In 1973, the Arabic language became one of the six official languages and working languages in the United Nations and the General Assembly. Like any other language, there is an official form of the language, in this case Modern Standard Arabic or MSA, which is used primarily in books, radio newscasts, television newscasts, and written media. However, MSA is not used as a communication language in informal conversations between people and when writing posts etc. on social media. The latter element renders it a language variant, of which every nation or region has its expressions. The main challenges encountered in applying SA to the Arabic language can be summarized as follows:

Each Arabic country has its own dialects, which are widely used in social media instead of MSA. For example, all the following phrases have the same meaning of “nice car” [سيارة جميلة]: [خوش سيارة], [سيارة حلوة], [سيارة بتجنن], and some words, such as [خوش] in the above example, will never be listed in any standard linguistic resources such as dictionaries.
Sometimes the same word is written with different spellings such as [سيارة] in taa marbotah [ة] and [سياره] in haa [ه].
The Arabic language has many idioms that have an implicit meaning that is different from the explicit meaning such as حسبي الله ونعم الوكيل (Alrefai et al., 2018). This idiom is considered positive, but it is used in a negative context.
In text mining, the stop words must be removed from the original text because they do not have a direct impact on the result. However, in the Arabic language, there are some negation words such as {غير} that invert the text polarity, but which are considered to be stop words. Therefore, these negation words should be excluded from the stop words list and they should not be removed in the preprocessing stage.
The same word may have the same spelling and the same POS but give a different meaning, such as {قال}: its POS is a verb; however it has the meaning of “he said” and “he took a snooze.”
Diacritics in the Arabic language change the meaning of a word completely. For example {شعر} might refer to “hair” if a small diacritic mark is put on its first letter and “poem” if the diacritic is put below that letter (Almuqren & Cristea, 2016).

In this the section, most of the SA research on the Arabic language will be reviewed and summarized based on the used approach and the proposed solution that they introduced.

3.1 The Lexicon-Based Approach

The lexicon-based approach mainly depends on a list of sentiment terms, and each term has its own score value. Usually, positive terms have a score value greater than zero, negative terms have a score value less than zero, and neutral terms have a score value of zero. An example of a study that employed the lexicon-based approach is that by (Al-Twairesh, Al-Khalifa, Al-Salman, & Al-Ohali, 2017) who built a wonderful Arabic corpus composed of 17,573 Saudi tweets, which were extracted from a huge dataset of 2.2 million Arabic tweets for use in SA research. Then the corpus was annotated manually to add one of four labels; positive, negative, neutral, or mixed. The latter is a relatively new label to indicate that the text contains both positive and negative opinions such as {قارئ جرير رائع لكن الاسعار غالية} (The Jarir Reader [app] is fabulous but the prices are expensive). A group of benchmark experimental results has occurred in order to verify that the dataset suitable for the Arabic sentiment analysis (ASA) community. In addition, (Abdulla et al., 2014) focused on lexicon-based SA for the Arabic language because it has lower importance than the other approaches. The authors proposed a new tool that handles some of the most important lexicon features such as negation and intensification. They considered that lexicon construction and SA tool design are the main challenges in this approach. They therefore studied three different lexicon construction techniques, one manual and two automatic.

Adopting a different perspective, (Al-Smadi, Al-Ayyoub, Jararweh, & Qawasmeh, 2019) introduced a lexicon-based SA approach by carrying out in-depth research into a relatively less-used SA, namely, aspect-based sentiment analysis (ABSA). The authors investigated this type of SA from two points of view: aspect category determination and aspect category polarity determination. They applied their technique on a dataset called the Human Annotated Arabic Dataset and reported that their proposed methodology outperformed many models when evaluated on the same dataset. In another approach, (Tartir & Abdul-Nabi, 2017) proposed a semantic method to extract user opinions from social media in both standard and dialectal Arabic language. The authors also introduced a new Arabic lexicon containing positive and negative keywords, which they named Arabic Sentiment Ontology. One of the main objectives of their study was to understand the effect of sentiment terms on text polarity. Their proposed methodology was applied to different Twitter feeds and different domains in order to measure the model’s efficiency.

Al-Rubaiee, Qiu, and Li (2016) proposed a model to classify opinions about Mubasher products that were given in tweets in the Saudi Arabian dialect. The model involved the use of preprocessing and tokenization techniques before inserting the data into support vector machine (SVM) and naïve Bayes (NB) classifiers. The accuracy of the experimental results was found to be promising. In another approach, (Al-Smadi et al., 2019) proposed a novel SA approach to detect the effect of social media news on Arabic readers. The study used news items collected from two of the most popular news agencies, namely, Al-Jazeera and Al-Arabia, during the Gaza–Israel war in 2014. The research put extra effort into the ABSA and text context. Two ML classifiers were used in the study, conditional random fields and J48, and they both gave good results with ABSA.

Furthermore, Farha and Magdy (2021) provided a thorough comparative overview of the most successful methods for Arabic SA. They re-implement the majority of current approaches for Arabic SA and assess their efficacy on three of the most common Arabic SA benchmark datasets. Furthermore, they investigate the use of transformer-based language models for Arabic SA and prove that they outperform current methods, with the best implementation achieving F-scores of 0.69, 0.76, and 0.92 on the SemEval, ASTD, and ArSAS benchmark datasets. In another search ,Abdul-Mageed (2019) reported on efficient lexical input models in Arabic, a language with a rather complex morphology. Specifically, we assess the effects of both gold and automated segmentation on the challenge and develop successful models that outperform our baselines. Using forecast segments improves subjectivity classification by 6.02 percent F1-measure and sentiment classification by 4.50 percent F1-measure compared to the plurality class baseline surface term types in these models. Also I, these models perform in-depth (error) analysis of the models' actions, as well as comprehensive descriptions of subjectivity and emotion expression in Arabic against the morphological richness context in which the work is situated.

3.2 The Machine Learning Approach

Machine learning (ML) is a technique that aims to train a machine to act automatically like a human. This goal can be achieved by training the machine by using actual data patterns that can then be recognized when needed. In SA, ML is executed in the same way using a standard list of algorithms. The ML approach is widely used in SA rather than the lexicon-based approach for many reasons, but primarily because the former has better accuracy. In another search, Al Shboul, Al-Ayyoub, and Jararweh (2015) proposed a new SA approach called multi-way sentiment analysis in which they tried to classify text by using a star ranking system, where 1-star means very negative and 5-stars means very positive. They applied their approach to a large Arabic Book Reviews dataset, extracted from an online Arabic book reviews website. The J48, decision tree, k-nearest neighbor (KNN), SVM, NB, and multinomial NB were used as ML classifiers. The results were not accurate because the proposed method is more complex than the traditional SA. In contrast, Abdul-Mageed (Abdul-Mageed, 2017) focused on the segmentation of rich morphological languages such as Arabic and their effect in yielding accurate subjective sentiment analysis. He created a new Arabic TreeBank annotated dataset and introduced detailed error analyses of the response of some lexical models. He used an SVM classifier to measure model accuracy.

In another study, (Alayba, Palade, England, & Iqbal, 2018) explored the advantages of convolutional neural network (CNN) and long short-term memory (LSTM) approaches on ASA. As a consequence, the obtained accuracy for ASA on multiple datasets was improved. Furthermore, the DL approach for ASA was introduced in (Al-Azani & El-Alfy, 2017). On two freely accessible databases, the authors studied different variations of skip-gram and CBOW, namely CNN and LSTM. Using the introduced combined LSTMs, the best results in terms of accuracy and other efficiency measurements were obtained.To detect emotion polarity in Arabic microblogs, (Al-Azani & El-Alfy, 2018) used LSTM and its simpler version gated recurrent unit (GRU). They contrasted the efficiency of deep learning to that of conventional machine learning approaches as a benchmark. Versions focused on LSTM and GRU outperformed other classifiers, according to the findings. In fact, integrating deep learning models for Arabic SA is a successful alternative to conventional machine learning approaches that helps to improve accuracy.

Furthermore, (Mubarak, Rashed, Darwish, Samih, & Abdelali, 2020) built the largest Arabic dataset to date, complete with special tags for vulgarity and hate speech They closely examined the dataset to decide which subjects, dialects, and gender are most often correlated with offensive tweets, as well as how Arabic speakers use offensive language. Finally, they performed several trials in order to obtain good findings. According to the findings, using an Arabic-specific BERT model (AraBERT) and static embeddings trained on tweets provided competitive results on the dataset.

In addition, (Elnagar, Al-Debsi, & Einea, 2020) adopted and impartial datasets for single-label (SANAD) and multi-label (NADiA) Arabic text categorization tasks. All these corpora are publicly accessible to the Arabic computational linguistics research community. In addition, they investigated the effect of using word2vec embedding models to increase classification task efficiency. Their test results demonstrated that both models performed well on the SANAD corpus, with a minimal accuracy of 91.18 percent obtained by convolutional-GRU and a maximum accuracy of 96.94 percent reached by attention-GRU.Al-Smadi et al. (2019) used supervised machine learning to offer an improved method for Aspect-Based Sentiment Analysis (ABSA) of Hotel Arabic ratings. They utilizeed cutting-edge research to train a series of classifiers with morphological, syntactic, and semantic features to tackle the research tasks: T1: aspect category identification, T2: opinion target expression (OTE) Extraction, T3: sentiment polarity identification. Nave Bayes, Bayes Networks, Decision Tree, K-Nearest Neighbor (K-NN), and Support-Vector Machine (SVM) are among the classifiers used. The method was tested using a sample dataset from the Semantic Evaluation 2016 workshop (SemEval-2016: Task-5). The results demonstrate that the supervised learning method outperforms comparable studies using the same dataset.

In addition ,Abdi, Shamsuddin, Hasan, and Piran (2019)introduced a deep-learning-based approach for classifying a user's analysis opinion (called RNSA). The RNSA employs the Recurrent Neural Network (RNN), which is made up of Long Short-Term Memory (LSTM), to take advantage of sequential computation and solve many shortcomings in conventional approaches, such as the loss of order and knowledge about the expression. The experimental results indicate that mathematical, linguistic, and sentiment awareness function vectors, sentiment shifter rules, and word-embedding can increase the classification accuracy of sentence-level sentiment analysis.

3.3 The Hybrid Approach

The hybrid approach exploits the strong points of the lexicon-based and ML approaches by combining them in one approach, where the lexicon-based approach is executed first. If the accuracy level does not match, then the lexicon algorithm and its output is used as input for an ML algorithm. Another way of combining these two approaches is to use the lexicons and any other linguistic resources to convert the text into vectors, which makes the subsequent ML process more accessible and more practical because most ML classifiers tend to deal with vectors and numbers directly instead of raw text.

Many studies have employed a hybrid approach, such as Pandey, Rajpoot, and Saraswat (2017), who proposed a new metaheuristic model based on the hybridization of cuckoo search and k-means clustering, named CSK for short. The primary purpose of this hybridization is to find the best cluster-heads in a dataset’s opinionated content. The authors prepared the data and converted it into feature sets before starting the evaluation process. They applied the proposed model to some benchmark datasets, namely, Twitter-sanders-apple2, Twitter-sanders-apple3, Twitter dataset, and Testdata.manual.2009.06.14. The experimental results were compared with the results produced other methods, namely, differential evolution, particle swarm optimization, cuckoo search, improved cuckoo search, Gauss-based cuckoo search, and two n-grams. The comparison showed that the proposed hybrid model outperformed these traditional methods.

In another work with a focus on tweet text, (Al-Twairesh, Al-Khalifa, Alsalman, & Al-Ohali, 2018) presented a hybrid dialect-independent approach by engineering a suitable feature-backward set. They then applied this methodology to a dataset of several Arabic dialects and especially Saudi tweets to test the model and the feature selection efficiency. They used two-way (positive and negative), three-way (positive, negative, and neutral), and four-way (positive, negative, neutral, mixed) SA. The F1 scores showed that the two-way option was the most accurate, followed by three-way and four-way.

In addition,(Al-Moslmi et al., 2017) introduced a new Arabic senti-lexicon which contains a list of 3,824 polar words. These words are categorized based on their polarity (positive and negative), polarity score, POS, inflected forms, and dialect synsets. The authors also presented a publicly available MASC that contains 8,860 positive and negative texts. Several domains such as politics and software were collected from several resources including the Jeeran website, Qaym website, Google Play, Twitter, and Facebook. After applying the necessary preprocessing steps, they converted the data into vectors by choosing some feature sets such as the number of positive words in the review or the tweet. After that, they inputted the resultant feature sets into SVM, NB, KNN, NN, and logistic linear regression classifiers. The experimental results showed that some features worked better with some classifiers. The present research will use the aforementioned senti-lexicon corpus and MASC software dataset in verifying the results of the proposed model.

In a recent work, (Alrefai et al., 2018) analyzed the research studies on ASA and categorized them by methodology and technique into ML, lexicon-based, and hybrid approaches. The authors presented the advantages and disadvantages of each approach and how it could be refined. They also counted the research studies based on the ML process and then proposed a new hybrid technique that combined the most-used classifier, namely, the SVM, with the evolutionary algorithm (EA), which is one of the fastest optimization algorithms used in solving complex cases. The aim of this hybridization was to adjust the SVM parameters by EA instead of using manual adjustment.

In this research, a hybrid model is proposed and developed that combines a ML model and a lexicon-based model. The following subsections describe the development of each part and their combination.

4.1 The Machine Learning Model

The first step in developing a ML model involves preparing the data and converting it into a feature vector, so that it is ready to input into the ML model. In this study, the feature vector is input into the k-means unsupervised learning algorithm. From a review of the literature, previous ASA research studies have only used supervised learning algorithms. Therefore, the efficiency of unsupervised learning is yet to be verified. For the purpose of this study, the data are partitioned into two classes, positive and negative. Figure 1 shows the k-means-based ML model process.

4.2 The Lexicon-based Model

A lexicon-based model depends mainly on the available linguistic resources, so there may be a limitation in terms of coverage of all the words and phrases in the various Arabic dialects. So, for this study, some words have been added to the lexicon to solve this problem.

In this work, the lexicon-based model categorizes the reviews based on the reviews total score value (summation of the positive and negative scores). As the dataset contains either positive or negative reviews, the best threshold for detecting the sentiment class is 0. Consequently, if the review score is greater than 0, it is classified as positive, and if the review score is less than 0, it is classified as negative. If the score value equals 0, then there are three options: positive, negative, or neutral. Such cases are skipped because this research dataset targets only positive and negative reviews. Figure 2 describes the lexicon-based model process.

4.3 The Hybrid Model

In a given state, each model has some limitations and drawbacks in certain respects. For example, the k-means algorithm works better if the observations have distinct values. K-means would encounter some difficulties in the partitioning process if these values were very close to each other, and this would affect model accuracy. On the other hand, the lexicon-based model is very strict and will not cover many cases or feature sets than the total score value. In these circumstances, one of the ways to enhance model accuracy is to hybridize the two approaches in order to exploit the advantages of each model.

In this research, the proposed hybrid model works as follows:

Several threshold values were tested, and the total score value that was greater than 7, or less than − 7, resulted in the best accuracy, so the absolute value of 7 was used as the threshold.
If the score value is greater than or equal to 7, or less than or equal to 7, it is classified using the k-means ML model.
Otherwise, the review is classified using the lexicon-based model.

Figure 3 shows the hybrid model process diagram.

4.4 Data Collection and Preparation

This research evaluates the accuracy and efficiency of the proposed model by using 1,200 Arabic reviews, 600 positive and 600 negative, from a software domain dataset. This dataset was collected and annotated by Al-Moslmi et al. (Al-Moslmi, Albared, Al-Shabi, Omar, & Abdullah, 2018) and was made publicly available for use in ASA research. The dataset focuses on software applications. Its reviews are written in several Arabic dialects. Hence it presents a realistic example of the types of reviews that appear every day on any digital platform because the vast majority of Arabic user-generated content is written in informal dialects, not in MSA. This study also uses the Arabic senti-lexicon corpus created by Al-Moslmi et al. (Al-Moslmi et al., 2018). The corpus is composed of a set of 3,824 positive and negative synsets labeled with many attributes such as POS, polarity scores, dialects synsets, and inflected forms.

However, during the preparation process, many malformed rows of data were found mainly in the senti-lexicon list. Therefore, one of the contributions of the present research involved fixing these rows of data and sending them to the original authors to merge with the source dataset. The aforementioned MASC dataset and Arabic senti-lexicon have been used in intensive research to produce feature vectors.

The contribution of this research to the MASC and the senti-lexicon is summarized as follows:

Removing duplicated entries, especially from the senti-lexicon resource, because there were many repeated polar words with different score values listed as synonyms for different terms. For example, as shown in Fig. 4, the term {جميل} (nice) has three different score values of -3, 4, and 5 in row numbers 1820, 1821, and 1822 respectively. So this would affect accuracy of feature extraction.
Fixing the annotation of some entries, such as like {محبب} (favorite) in row 1820 in Fig. 4, which was annotated as a negative term and was thus given a negative score value of -3, but it is actually a positive term and should be given a positive score. These errors were detected during the feature extraction process. After the extraction process, many texts were categorized with the opposite of their class label, and after tracing the root cause of these errors, many terms were found to have invalid score values.
Some terms were given invalid synonyms. For example, for the negative term {هول}, as shown in row number 1811 Fig. 4, many terms are listed as synonyms and among of them {جود;سماحة;كرم;سهولة;سلس;انقياد;لين} are positive terms. All of these cases were fixed.
There were some cases with huge random and incorrect synonymous terms. All these cases were dealt with and fixed manually by two native Arabic speakers.
There was some duplication in some reviews. All duplicated items were removed.

4.5 Data Preprocessing and Tokenization

One of the most important challenges in the text mining process is data cleaning and preprocessing. This challenge is particularly when dealing with unstructured content because the author of the text might express his/her ideas in unexpected ways (Al-Osaimi & Badruddin, 2014; He, Zha, & Li, 2013; Rushdi-Saleh, Martín-Valdivia, Lopez, & Perea-Ortega, 2011). Therefore, to make the text mining process unified and increase its comprehension level, the preprocessing stage could be considered the backbone of the process because if the input is invalid, then an unexpected result will definitely be returned (Aliane et al., 2016; R. Duwairi & El-Orfali, 2014; Hahnen et al., 2017; Kang & Park, 2014; Pandey et al., 2017).

In this research, several steps were applied in the preprocessing phase. These steps are described below in order of implementation. The order of these steps is critical and plays a significant role in the process outcome:

4.5.1 Emoticon conversion

The conversion of emoticons is a very important step in the SA process because many online users prefer to express their opinions using emoticons instead of using naked words. Moreover, during the literature review conducted for this research, it became obvious that many researchers mentioned that emoticon mining is a significant task. However, treating emoticons in the early stages of preprocessing has yet to be done.

In this research, it was found that most of the emoticons contained special characters such as :-) and: @. If these characters are removed by the cleaning process, the emoticons get broken and become incomplete in meaning. Therefore, the following approach was applied in order to extract the emoticons from the dataset:

A web scraping script using Python programming language was developed to scrap the complete emoji list from the official Unicode Consortium website.

Each emoticon, along with its unique Unicode (displayed in the second ‘Code’ column in Fig. 5) was extracted and stored.

As shown in Fig. 5, each emoticon has many style variations per social media platform. However, all these style variations have one unique Unicode. Therefore, only the first style (displayed in the third ‘Browser’ column in Fig. 5) and its unique Unicode was stored in a Pythons’ script list.

Each emoticon was assigned a polarity score, which ranged from 5 for the most positive to -5 for the most negative. This emoticon polarity score is added to the final review score. The annotation of emoticons was done by two persons with experience in the SA field. The emoticons with sample codes and a guide on how to use them will be made publicly available for use in similar research.

A similar approach was applied in (Al-Osaimi & Badruddin, 2014) to deal with emoticons. However, the authors replaced or mapped the emoticon with its suitable sentiment word. In contrast, in the present study, a new list of Unicode emoticons was prepared, and this emoticons list was reviewed and annotated manually by two SA reviewers. In the present study, the emoticons were also given weights ranging between 5 for an extremely positive emoticon and − 5 for an extremely negative one. A similar methodology was applied in (Hutto & Gilbert, 2014), but the authors used LIWC software for the text analysis and this software is not available for free and needs a commercial license. Also, many studies have used plain text emoticons such as :-) and :-( which is very limited compared to the huge amount of emoticons employed by users. Hence the list compiled by the present study is more comprehensive and will also be freely available.

4.5.2 Contextual intensification

As in any language there are some keywords in Arabic such as {جدا ، بشدة، بقوة} (very, too, extremely) that act to strengthen and increase the opinion intensity of the speaker to make it stronger. These keywords are called intensifiers, booster words, or degree adverbs (Abdulla, Ahmed, Shehab, & Al-Ayyoub, 2013; Hutto & Gilbert, 2014; Nabil, Aly, & Atiya, 2015). For example, {أعجبني بشدة} “I liked it very much” has more sentiment polarity than {أعجبني} “I liked it”. Therefore, the existence of such an intensifier would give more determination to the sentence.

Intensifiers have a significant impact on sentence polarity, therefore deep research and implementation was applied in this research. Furthermore, the initial tests showed that dealing with the booster words as they are, without stemming mainly, retained their effect better and more clearly. For example, the stem of the booster word {جدا} (very) is {جد} (grandfather). The latter has another meaning in the Arabic language, so the stemming makes it loses its meaning and its main role as a booster word. Therefore, in order to handle such cases, a simple three-step approach was developed:

Generate a list of random unique tokens, and map each booster word to a token.

Replace the booster word with its corresponding token (generated in the previous step) in the dataset by iterating over all the text items one by one.

Use the tokens in the feature extraction process instead of the booster words.

By applying the above approach, the stemming process does not change the effect of the booster words.

Table 1 shows the list of boosters used in the dataset. It is evident that there are several informal boosters from different Arabic dialects. These words are frequently used by users in expressing their opinions on social media platforms.

Table 1

List of Arabic intensification words
Arabic intensification words
جدا ، فعلا، بشده، بقوه، بإفراط ، كثير، كتير، واجد، مره، وايد، حيل، قوي، بقوه، للغايه، كلش

Algorithm 1

shows the pseudocode of three of the main feature extraction steps: the handling of negation words, contextual intensification, and conversion of emoticons into scores.

Algorithm 1:Feature extraction steps for negation words, intensifiers, and emoticons

Begin

set list of booster words b, senti-lexicon terms s, posts p, negation words nw, emoticons e

for i = 1 to s do //iterate over all senti-lexicon words s

for j = 1 to p do //iterate over all posts

if s[i] in p[j] then //check if senti-lexicon word i is in post p

//handle negation words

if p[j][i.index–] in nw //check if the senti-lexicon word is preceded by a negation word

p[j][score] + = s[i][score] * -1 //then multiply the senti-lexicon word score by -1

else

p[j][score] + = s[i][score] //add the senti-lexicon word score to the post score

end if

//handle the contextual intensifiers

if p[j][i.index++] in b //check if the next word is a booster word

p[j][score] + = s[i][score] * 2 // then multiply the senti-lexicon word score by 2

else if p[j][i.index–] in b //check if the previous word is a Senti-lexicon word

p[j][score] + = s[i][score] * 2 // then multiply the senti-lexicon word score by 2

end if

end for //end the posts list iteration

end for //end the senti-lexicon words list iteration

//handle the emoticons score

for i = 1 to e do

for j = 1 to p do

if ic[i] in p[j] then //check if emoticon i is in post j

p[j][score] + = ic[i][score] // then add the emoticon score to the post score

end if

end for //end the posts list iteration

end for //end the emoticons list iteration

return p //return posts array after updating the scores

End

4.5.3 Miscellaneous text cleaning and preparation

After the emoticons and the booster words have been replaced with tokens, this creates text that is more suitable for cleaning. The cleaning of noisy characters increases the text’s readiness for the feature extraction process, which is described in detail in section 4.6.

In text cleaning, the following steps are prerequisites for text mining, and any NLP research study on the Arabic language will use at least some of these steps. However, the techniques and the way and the order of in which they are applied may differ from one study to another:

Removing the extra spaces and the new lines, as well as the non-Arabic letters because Arabic is the targeted language of this study.
Removing the repeated letters that are repeated more than three times because the presence of these letters confuses the stemmer and the feature extraction process. An example of this case is {روووووووعة}, which resembles (Coooooool). This way of writing reviews and posts has become very popular and it is used to intensify the opinion given (Abdulla et al., 2014). Removing repeated letters can be done automatically or manually. In this research, the repeated letters are removed automatically. Sometimes a letter is repeated two times such as {لفظ الجلالة الله} {the name of Allah}, but in this case the repetition should be kept because removing the repeated letters will change the meaning. This is why the threshold of three letters or more was selected.
Removing excessive repetitions of the same word. Some reviews are written in an overstated way, where a positive or negative word is repeated more than 10 times, e.g., { روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه ……} (cool cool cool ….). So, keeping this format would make this review have a big score and it would be considered as an outlier, which might confuse the model. The solution that was applied in this research to such cases was to create a regular expression to remove any repeated word automatically and keep it for two times only.
• Replacing {أ إ آ} with the plain letter { ا } and { ه } the haa with { ة } taa marboota. This technique is used in many research studies, such as (Al-Twairesh et al., 2017; Alrefai et al., 2018).
• Removing the stop words {من ، إلى ، عن ، على}the special characters {@ ! ? } and the diacritics{{فتحة ، ضمة ، كسرة because they do not have a major impact on the sentiment result. The Python Natural Language Toolkit (NLTK) [1] stop words list was used in this research, but in that list some stop words are also considered as negation words, such as {ما ، لا ، غير ، ليس} Therefore, these words were excluded from the stop words list. Tables 2 and 3 show the stop words list and special characters list, respectively.
Applying word stemming, a mechanism that aims to return the word to its original linguistic stem. This technique has a significant effect in reducing the redundancy of words (Al-Zyoud & Al-Rabayah, 2015) because many words have various inflected forms and share the same word stem. For example, all the words { أحب ، يحب ، تحب ، يحبون ، محبة} have a different form and POS but have the same stem { أحب }. Therefore, stemming helps significantly in the feature extraction process because the lexicons do not cover all the inflected word forms. In this research, the Python NLTK [1] and the Open Source ISRIStemmer were used for the stemming process (Aliane et al., 2016; Altowayan & Tao, 2016).
Dealing with negation words. In any language, there are some words called negation words. If they precede a positive term, they invert that term into a negative term and vice versa. For example, { أنا أحب هذا التطبيق } (I love this application) is a positive sentence. However, if the negation word { لا } is inserted before the positive term { أحب }, this converts its polarity from positive into negative and vice versa. Hence it is crucial to list these words and handle their effect to get better accuracy. This approach is used in most SA studies such as (S. O. Alhumoud, Altuwaijri, Albuhairi, & Alohaideb, 2015; R. M. Duwairi & Alshboul, 2015; Shukri, Yaghi, Aljarah, & Alsawalqah, 2015). Table 4 shows the list of Arabic negation words used in this research.

Table 2

NLTK stop words list after removing some negation words
NLTK stop words list
أنا ، انا ، لولا ، عسى ، لعل ، أكثر ، إنه ، تلكم ، به ، هي ، بل ، ماذا ، ذينك ، تلك ، عدا ، لها ، هاته ، بس ، هذا ، كل ، التي ، آه ، اللذين ، والذين ، بكما ، ذين ، هذان ، أولاء ، بخ ، مما ، بكن ، ذا ، منه ، كأين ، بهن ، ذو ، ذانك ، أقل ، تلكما ، شتان ، اللذان ، لكن ، الذي ، مع ، وإن ، عليك ، هنا ، هو ، هيت ، ذلكم ، ولو ، هكذا ، هيهات ، هاتين ، أن ، أوه ، إذما ، إليك ، هن ، منها ، كلتا ، لستن ، منذ ، ومن ، فإذا ، إن ، كذلك ، ثمة ، بين ، آي ، اللاتي ، هذين ، اللتين ، هلا ، تين ، إليكم ، لكم ، ذواتا ، كأن ، ته ، دون ، هل ، مذ ، بي ، هيا ، الذين ، ذات ، نحو ، ذلكما ، إنا ، لهن ، ثم ، مه ، أنتم ، أنتما ، كيفما ، فيها ، حين ، هؤلاء ، لو ، وهو ، فمن ، والذي ، اللواتي ، تي ، فيم ، ذلك ، عل ، أنى ، لكما ، كأي ، حبذا ، إيه ، لكي ، فيه ، بكم ، لاسيما ، لما ، أف ، على ، فإن ، بها ، أيها ، كما ، سوى ، كيت ، متى ، لهما ، هاتي ، سوف ، بعد ، هذي ، هم ، أما ، ذوا ، حيث ، أو ، خلا ، لي ، كأنما ، اللتان ، عليه ، هذه ، إذ ، ها ، إنما ، كيف ، وإذ ، هنالك ، لكنما ، كلاهما ، فيما ، بنا ، ذلكن ، حتى ، ذان ، إليكما ، هما ، بك ، أم ، مهما ، كلما ، ممن ، أولئك ، ذي ، أنا ، لوما ، في ، لنا ، إلا ، أين ، بلى ، بما ، كذا ، إذا ، عن ، يا ، من ، لهم ، ألا ، بعض ، بماذا ، حيثما ، ريث ، ذه ، كليكما ، هاهنا ، إلى ، كليهما ، بيد ، آها ، بهم ، ليت ، نحن ، ذاك ، لك ، لن ، أي ، عند ، اللتيا ، أنت ، هناك ، بمن ، إليكن ، هاتان ، ولكن ، حاشا ، اللائي ، لئن ، إما ، لدى ، وإذا ، عما ، ذواتي ، قد ، كم ، تينك ، أنتن ، كي ، نعم ، هاك ، إي ، بهما ، له ، أينما ، إذن

Table 3

List of special characters, marks, and diacritics
Special characters
'؛', '؟', '!', '.', ':', '-', '(', ')', '"', '.', '/', '\\', '\|', '\'', '*', '^', '@', '#', '$', '%', '+', '&', 'َ', 'ً', 'ُ', 'ٌ', 'ّ', 'ِ', 'ٍ', '~', 'ْ', ',', '،', '`', '[', ']', '{', '}', '>', '<', '?'

Table 4

List of negation words
Negation words
، ما‬ ، لا ، لم ، لن ، مش ، ليست ، لسنا ، لستما‬ ، ليسا‬ ، ليستا‬ ، لسنا‬ ، لستم‬ ، لستن‬ ، ليسوا‬ ، لسن‬ غير ، مو ، كلا ، بلا ولا ، فلا ، فلم ، ولم ، لست ، فليس ، فليست ، فلن ، ولن ، لكن‬ ، ولكن‬ ، ومش‬ ، بس بل ، ليس ، ‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬

Table 5 shows an example of applying the aforementioned preprocessing techniques step by step to some review text.

Table 5

applying the aforementioned preprocessing techniques
Remove special characters and diacritics
سئ جدا من افشل التحديثات اذا تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي قبل يومين وبعده قبل ساعاتين والي بعده يجيك قبل يوم حووووسه ماينفع فاشل جدا جدا جدا هالتحديث غير إنهم غييروا الايقونه وصار خايس	البرناااااامج روعة روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه يستحق5 نجوووووم بجدارة😍😍💕💏
Replace hamzat ‘أ إ اَ’ with plain ‘ا’ and taa marbotaa ‘ة’ with haa ‘ه’
سئ جدا من افشل التحديثات اذا تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي قبل يومين وبعده قبل ساعاتين والي بعده يجيك قبل يوم حووووسه ماينفع فاشل جدا جدا جدا جدا جدا هالتحديث غير انهم غييروا الايقونه وصار خايس	البرناااااامج روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه يستحق5 نجوووووم بجداره😍😍💕💏
Remove stop words
سئ جدا افشل التحديثات تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي يومين وبعده ساعاتين والي بعده يجيك يوم حووووسه ماينفع فاشل جدا جدا جدا جدا جدا هالتحديث غير انهم غييروا الايقونه وصار خايس	البرناااااامج روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه يستحق5 نجوووووم بجداره😍😍💕💏
Remove repeated letters
سئ جدا افشل التحديثات تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي يومين وبعده ساعاتين والي بعده يجيك يوم حوسه ماينفع فاشل جدا جدا جدا هالتحديث غير انهم غييروا الايقونه وصار خايس	البرنامج روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه روعه يستحق5 نجوم بجداره😍😍💕💏
Remove repeated words
سئ جدا افشل التحديثات تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي يومين وبعده ساعاتين والي بعده يجيك يوم حوسه ماينفع فاشل جدا جدا جدا هالتحديث غير انهم غييروا الايقونه وصار خايس	البرنامج روعه روعه روعه يستحق 5 نجوم بجداره 😍😍💕💏
Apply contextual intensification
#ADFGTRERTG# سئ افشل التحديثات تصفحت الانتسغرام مايعطين الصور بالترتيب يجيب لي اول شي يومين وبعده ساعاتين والي بعده يجيك يوم حوسه ماينفع فاشل #ADFGTRERTG# #ADFGTRERTG# #ADFGTRERTG# هالتحديث غير انهم غييروا الايقونه وصار خايس	البرنامج روعه يستحق5 نجوم بجدار 😍😍💕💏
Apply stemming
سئ #ADFGTRERTG# من فشل حدث اذا صفح انتسغرام ايعط صور ريب يجب اول شي يوم بعد سعا ولي بعد يجك يوم حوس ماينفع فشل #ADFGTRERTG# #ADFGTRERTG# #ADFGTRERTG# هالتحديث غير نهم غير يقو وصر خيس	رنمج روع حق 5 نجم جدر😍😍💕💏

4.6 Feature Extraction

The preprocessing and tokenization stage described in section 4.5 only prepared the dataset as the input for the feature extraction (also called feature engineering) stage, i.e., the emoticons were replaced with unique codes each with a score value, and xxxx. The effects of the emoticons and the total score value of the text (review, comment, or post) are yet to be extracted. Feature engineering is an essential task in the SA field, and basically concerns converting text into feature vectors so that it is easier for the ML algorithm to deal with them. This task can be corpus-based (as applied in this research), dictionary-based, or manual-based (Ismail, Alsammak, & Elshishtawy, 2016). In this section, the proposed methodology for mining the polar features in the text is described, which involves the calculation of the feature vector from the dataset, and the process of selecting the features.

4.6.1 Feature extraction calculation method

In this research, the Arabic senti-lexicon corpus used in (Al-Moslmi et al., 2017) is applied to calculate the dataset’s feature vector. The list of emoticons and booster words described in sections 4.5 1 and 4.5.2, respectively, were used to determine their impact on text subjectivity.

The steps used to calculate the feature vector of the dataset were as follows:

Using Python, an iterative scripting code was developed to go over all the dataset rows and split each row’s content by space.
Then iteration over each row of words was performed to check whether the word was in the senti-lexicon corpus.
If yes, the polarity and score value of the word was stored. Than a checking process identified whether a negation word preceded the word. If yes, the score value of the word was multiplied by -1 and its polarity was inverted.
A check was conducted to determine whether the sentiment word was preceded {السيارة جدا جميلة} or followed {السيارة جميلة جدا} by a booster word. If yes, the score value of the sentiment word was multiplied by 2.
A further check ascertained whether any emoticons in the text increased or decreased the total score value according to the score value assigned to the emoticon.

Figure 6 shows a sample of the features extracted from the dataset and how they existed and were stored in a CSV comma-delimited file. The proposed SA model is given these features as input instead of the dataset itself. The selection of the feature sets is discussed in section 4.6.2.

Algorithm 2

shows the pseudocode of the preprocessing methodology that was applied in this research.

Algorithm 2: Preprocessing, cleaning and tokenizing

Begin

set list of booster words b, list of emoticons e, list of unique codes u, list of posts p

//map booster words to a unique code

for i = 1 to b do //iterate over all booster words b

b[i][value] = u[i]

end for

//map emoticons to a unique code to prevent their removal in the cleaning process

for i = b.length to e do //iterate over all booster words b

e[i][value] = u[i]

end for

for i = 1 to p

replace booster word with Unicode

replace emoticon with Unicode

removing the extra spaces

remove the letters that are repeated more than three times

replace {أ إ آ} with the plain letter {ا}

replace {ة} with {ه}

remove the stop words, special characters, and the diacritics

stemming

end for

return p //return clean posts list to be input for feature vector conversion

End

4.6.2 Feature engineering

Feature engineering is an essential task in SA. It involves reading the text (e.g. a review) and converting it into numbers or vectors. Then, it selects the suitable feature vectors and combines them in the right way to construct the feature sets for input into the model. Thus it plays a crucial role in the SA classification process. Therefore, deep research was undertaken in selecting the features and feature sets and converting them into vectors and numbers.

For this research, 10 feature vectors were selected, and these mainly depend on the total score, word count, and presence of polar words/phrases in the reviews:

Total of Positive Scores F1: The summation of all the positive values that the review contains, such as positive words polarity score, positive emoticons score, effect of the booster words on the positive words.

Total of Negative Scores F2: The summation of all the negative values that the review contains, such as negative words polarity score, negative emoticons score, effect of the booster words on the negative words.

Total of Positive Words F3: The total number of positive words in the review, such as {جيد ، رائع ، راقي} (good, cool, elegant).

Total Negative Words F4: The total number of negative words in the review, such as {سيء ، بخيل ، قبيح} (bad, stingy, ugly).

Frequency of Positive Terms F5: The frequency of the positive words to the total number of words in the review text. For example, if the text contains a total of 10 words and two of them are positive, then the frequency of positive terms is 0.2.

Frequency of Negative Terms F6: The frequency of the negative terms to the total number of words in the review text. For example, if the text contains a total of 10 words and two of them are negative, then the frequency of negative terms is 0.2.

Presence of Positive Words F7: If the review has at least one positive word or phrase the value is 1, otherwise 0.

Presence of Negative Words F8: If the review has at least one negative word or phrase the value is 1, otherwise 0.

Total Score F9: The total score (summation of positive and negative scores) of the review.

Total number of words F10: The total number of words in the review.

Table 6 shows how the aforementioned feature vectors are related to each feature set. Each review in the corpus is represented by using the following feature vector: ((F1: value1, F2: value2, .., Fm: valuem), ‘polarity’).

Table 6

Features extracted for each review.
Feature set name	Feature name
FS1. Sentiment word score	F1. Total score of all positive scores in the review
FS1. Sentiment word score	F2. Total score of all negative scores in the review
FS2. Sentiment word count	F3. Total count of the positive words/phrases in the review
FS2. Sentiment word count	F4. Total count of the negative words/phrases in the review
FS3. Sentiment word frequency	F5. Ratio of positive terms count to all words count in the review
	F6. Ratio of negative terms count to all words count in the review
FS4. Sentiment word presence	F7. Presence of positive words
	F8. Presence of negative words
FS5. Total score & total word count	F9. Total score of the review
	F10. Total number of words in the review
FS6. All features	All the features together.

All of the proposed models (lexicon-based, ML, hybrid) were implemented and tested using Python 3.7 by Anaconda Spyder 3.3.1 and Jupiter Notebook 5.6.0, which is a scientific Python development environment. All experiments were conducted on the aforementioned Notebook computer which had the following specifications: operating system: Linux Ubuntu 18.10 64-bit; Processor: Intel(R) Core (TM) i7-6500U CPU @ 2.5 GHz; installed memory (RAM): 8.00 GB.

5.1 Dataset Description

As mentioned previously, the Arabic language has few corpora and datasets, so that the publicly published MASC2 software by (Al-Moslmi et al., 2017). MASC software contains 1,200 Arabic reviews in the software domain, they extracted from Google Play; the official mobile applications store for the Android operating system. These reviews classified to 600 positive and 600 negative, they were written as feedback by Arabic users for several application and products namely; Facebook, Instagram, WhatsApp Messenger, Snapchat, Twitter, Tango, Hangout, Chrome, Team Viewer, Skype, TED, WeChat, IMO, Viber, Line, Speed Night, Need for Speed, CAMERINGO +, and Asphalt.

5.2 Results of the Machine Learning Model

In this research, the k-means clustering algorithm was used as the ML method in order to verify its performance in solving ASA problems. K-means is an unsupervised method, which means it does not use class labels during the learning (training) process. Consequently, it tends to be very effective in cases where the data are unstructured or unorganized. As the k-means algorithm is built using mathematical equations, it is more robust if the input is numbers instead of text, and this is where feature engineering comes into play.

The k-means algorithm built-in function has several parameters, and most of them have a default value. In this research, the number of clusters was set to 2; positive and negative (n_clusters = 2), k-means + + was used as the default initialization method (init = ‘k-means++’), and the maximum number of iterations per single run was 300 (max_iter = 300), which is the default value.

The actual feature vectors extracted from the review texts were represented in diagrams in a color denoting their real class. If the actual review class was positive, it was displayed as a green point regardless of its coordinates; otherwise, its color was red. The predicted classes generated by the k-means algorithm were also represented in diagrams in the same way. This approach was intended to make the comparison of the actual and predicted classes easier.

For each feature set, Figs. 7 to 12 (left) show the actual distribution of the reviews based on their positive and negative scores, and Figs. 7 to 12 (right) shows the predicted clusters based on these scores. It can be seen that in the case of the actual clusters there are many vector values from various classes that are very close to each other, hence the error rate increases. The algorithm faces difficulty in resolving this overlapping because it is caused by the unstructured text of the reviews. On the other hand, the reviews with high positive or negative scores are not affected by cluster overlapping.

Table 7 summarizes the performance metrics per feature set of the ML model built on top of the k-means algorithm. It can be seen from the table that FS5 returned the best accuracy, recall, and F1 score and that FS3 had the best precision. On the other hand, FS2 gave the worst accuracy and precision, and FS4 had the worst recall and F1 score. Consequently, it can be inferred that the most important features that play a vital role in the ASA classification process are the polarity score and the review word counts.

Table 7

Performance metrics of the k-means cluster model
Feature Set	Accuracy	Precision	Recall	F1 Score
FS1. Sentiment word score	0.68	0.86	0.43	0.57
FS2. Sentiment word count	0.52	0.51	0.69	0.59
FS3. Sentiment word frequency	0.73	0.91	0.52	0.66
FS4. Sentiment word presence	0.60	0.77	0.28	0.41
FS5. Total score & total word count	0.82	0.85	0.77	0.81
FS6. All features	0.77	0.88	0.62	0.73

From the results of the ML model in Table 6, the following points can be made:

FS5, which combines the total score and the total word count of the review, yielded the best performance in terms of overall accuracy, recall, and F1 score with 82%, 77%, and 81%, respectively.
FS3, which combines the frequency of positive terms and the frequency of negative terms in the review, yielded the best performance in terms of precision with 91%.
The review score polar terms frequency and total number of words affect model performance more than the other features.
The k-means clustering algorithm can give good performance results to solve the ASA problem.

Table 8 shows the confusion matrix (usually called the matching matrix in unsupervised learning) for FS1, FS2, FS3, FS4, FS5, and FS6.

Table 8

FS1–FS6 confusion matrix
Method	True Negative (TN)	False Negative (FN)	True Positive (TP)	False Positive (FP)
FS1	557	341	259	43
FS2	208	188	412	392
FS3	569	288	312	31
FS4	549	433	167	51
FS5	520	141	459	80
FS6	548	227	373	52

5.3 Results of the Lexicon-based Model

The experimental results showed that 62 reviews (25 positive and 37 negative) had a score value equal to 0. As shown in Table 9, three classes of cases were determined: case 1 (C1) is a review with a score value equal to 0 that was predicted as positive;. case 2 (C2) is a review with a score value equal to 0 that was predicted as negative; and case 3 (C3) was a review that was predicted as neutral. However, the neutral cases were ignored because the dataset in this research only covered positive and negative labels.

Table 9

C1 ,C2 and C3 confusion matrix
Method	True Negative (TN)	False Negative (FN)	True Positive (TP)	False Positive (FP)
C1	404	60	540	196
C2	401	85	515	159
C3	404	60	515	159

Table X shows the performance metrics of the lexicon-based model. It is evident from the table that ignoring the reviews that have a score value equal to 0 (i.e., those in class C3) enhanced the performance slightly because these reviews are closer to the neutral class, but there is no neutral class in this research. Table 10 shows the confusion matrix of the lexicon-based model.

Table 10

The performance metrics of the lexicon-based model
Case ID	Accuracy	Precision	Recall	F1 Score
C1	0.79	0.73	0.90	0.81
C2	0.80	0.76	0.86	0.81
C3	0.81	0.76	0.90	0.82

The results of the lexicon-based model shown in Table X are summarized as follows:

C3, which excluded the reviews with a score value equal to 0, yielded the best performance results in all metrics, meaning that neutral reviews might deteriorate model performance.
C1 yielded high precision.
C2 yielded the best recall.
The lexicon-based model, in general, yielded good performance results. However, its results are based entirely on the total score values of the reviews.

5.4 Results of the Hybrid Model

The experimental results showed that the k-means clustering model worked better after separating the observations with relative score values and moving them to be classified using the lexicon-based model. As mentioned in section 4.3, the best threshold value was the absolute value of 7. For the hybrid model, the dataset was divided into two parts: one part was passed to the ML model, and consisted of 672 observations; 386 positive and 286 negative. The other part was the input for the lexicon-based model, and consisted of 528 observations.

As mentioned in section 5.2, there were 62 reviews with a score value equal to 0. These reviews were predicted as positive in C1, negative in C2, and neutral (ignored) in C3. Consequently, the final number of reviews passed to the lexicon-based model in the case of C1 and C2 was 528; 214 positive and 314 negative, and in the case of C3 was 466; 189 positive and 277 negative. Table 11 shows the performance metrics of the ML part, of the lexicon-based part, and of the two parts combined in the hybrid model.

Table 11

Performance metrics of the hybrid model

Performance metrics of the machine learning part of the hybrid model

Feature set	Accuracy	Precision	Recall	F1 Score
FSH1. Sentiment word score	0.90	0.87	0.98	0.92
FSH2. Sentiment word count	0.87	0.87	0.91	0.89
FSH3. Sentiment word frequency	0.81	0.51	0.69	0.59
FSH4. Sentiment word presence	0.58	0.93	0.29	0.45
FSH5. Total score & total word count	0.90	0.87	0.98	0.92
FSH6. All features	0.90	0.87	0.98	0.92

Performance metrics of the lexicon-based part of the hybrid model

Case ID	Accuracy	Precision	Recall	F1 Score
CH1	0.64	0.54	0.76	0.63
CH2	0.66	0.58	0.64	0.61
CH3	0.67	0.58	0.72	0.64

Performance metrics of the hybrid model (FSH1 with C1, C2, and C3)

Case ID	Accuracy	Precision	Recall	F1 Score
FSH1CH1	0.79	0.73	0.90	0.81
FSH1CH2	0.80	0.76	0.86	0.81
FSH1CH3	0.81	0.76	0.90	0.82

The results of the hybrid model in Table 11 are summarized as follows:

The k-means clustering model part of the hybrid model yielded high-performance results in terms of accuracy, recall, and F1 score metrics, especially in the positive and negative score value features, total score, and features together in FS6, and high precision in FS4.
The lexicon-based part of the hybrid model yielded good performance results because the reviews that are passed to the hybrid model confused the k-means clustering model.
Isolation of the reviews that have near feature values helps in improving the accuracy of the clustering process by exploiting the advantage of the lexicon-based model, which depends mainly on the score value feature.

Figures 13 to 18 (left) show the actual distribution of the data points of FSH1 to FSH6, respectively, before the clustering process, and Figs. 13 to 18 (right) show how k-means predicted and partitioned them.

Table 12 shows the confusion matrix of the feature sets FSH1–FSH6 of the ML part of the hybrid model.

Table 12

FSH1–FSH6 confusion matrix
Method	True Negative (TN)	False Negative (FN)	True Positive (TP)	False Positive (FP)
FSH1	228	8	378	58
FSH2	233	33	353	53
FSH3	270	110	276	16
FSH4	278	8	113	8
FSH5	228	8	378	58
FSH6	228	8	378	58

Table 13 shows the confusion matrix of the lexicon-based part of the hybrid model based on the case IDs CH1 to CH3.

Table 13

CH1 ,CH2 and CH3 confusion matrix
Method	True Negative (TN)	False Negative (FN)	True Positive (TP)	False Positive (FP)
CH1	176	52	162	138
CH2	213	77	137	101
CH3	176	52	137	101

Table 14 shows the combined confusion matrix of the ML and lexicon-based parts of the hybrid model. As FSH1 returned the best performance in the ML part of the hybrid model, it was used along with the three cases of the lexicon-based model.

Table 14

FSH1CH1, FSH1CH2 and FSH1CH3 confusion matrix
Method	True Negative (TN)	False Negative (FN)	True Positive (TP)	False Positive (FP)
FSH1CH1	404	60	540	196
FSH1CH2	441	85	515	159
FSH1CH3	404	60	515	159

5.5 Discussion

Unlike previous studies conducted in the field of SA, this research applies an unsupervised learning method (k-means), where the model learns by itself, rather than supervised learning methods where the model learns from predefined data labels that are given during the training phase. This research investigated the outcomes of applying the k-means clustering as a ML model, a lexicon-based model, and a hybrid model combining the aforementioned two models on a dataset of Arabic reviews about different software platforms. The performance metrics that were assessed in this research were accuracy, precision, recall and F1 score. In addition, the confusion matrix of each of the models was presented to provide more details on the results. Furthermore, all the k-means clustering algorithm results were plotted to provide a clear illustration of the distribution of the observations before the clustering process and after k-means partitioning.

The main findings are summarized as follows:

The feature set selection criteria and the senti-lexicon corpus played key roles in constructing a good SA model.
The k-means clustering model yielded good results. However, more enhancement is still needed.
The ML model in general gave more flexibility than the lexicon-based model.
The lexicon-based model depended mainly on the senti-lexicon corpus, so it was somewhat restricted in terms of capability.
The ML part of the hybrid model yielded high-performance results because the hybrid model isolated the data points that confused the clustering model.
The flexibility of the feature set selection methodology increased the robustness of the ML model, therefore more feature sets could be examined and suggested.

Also every technique has flaws, but the hybrid model, as shown by the findings, is an optimal and more flexible alternative and approach to performing SA in a successful and self-improving manner.

This ultimate goal of this research was to produce an effective model to solve and enhance the solution quality of ASA problems. The k-means clustering algorithm was employed due to its robustness and ability to group data points into clusters based on the similarity of data characteristics. K-means is easy to implement and computationally fast when the data have many features or independent attributes. Three SA approaches, namely, ML, lexicon-based, and hybrid, were proposed. The lexicon-based approach was built on the overall review score extracted from a public senti-lexicon corpus. The hybrid approach combined the ML and lexicon-based approaches. All three approaches depended on the feature engineering process that selects specific features as input for the model. The three approaches were tested on the publicly available MASC software dataset, which contains 1,200 Arabic reviews (600 positive and 600 negative).

Several previous research studies have used text emoticons in SA, but this is a limitation because they do not reflect most of the latest social media platform icons currently in use. Therefore, as part of this research, a new list of Unicode emoticons was collected and annotated in terms of polarity, and it will be made publicly available for use in the feature extraction process. This is just one example of the deep research that was conducted prior to the implementation of the proposed SA approach in order to resolve the challenges that are typically encountered in text mining.

The experiment results indicated that the proposed approaches were able to find high-quality solutions to the ASA problem. However, each approach had some limitations and drawbacks. The main disadvantage of the ML approach was that it became confused when a review contained both positive and negative terms, which led to unexpected computations and results. In contrast, the lexicon-based approach depended entirely on linguistic resources such as corpora and dictionaries, but these resources lack words in informal Arabic dialects.

The proposed hybrid approach overcame these drawbacks and the results indicate that there are some exciting directions for future research, including hybridizing the k-means clustering algorithm with another algorithm to better partition the reviews that have both positive and negative words because the number of clusters is an important parameter in clustering problems.

• Ethics approval and consent to participate

We confirm that any aspect of the work covered in this manuscript that has involved human patients has been conducted with the ethical approval of all relevant bodies and that such approvals are acknowledged within the manuscript.

• Consent for publication

We give my consent for the publication of identifiable details, which can include photograph(s) and/or videos and/or case history and/or details within the text (“Material”) to be published in the above Journal and Article. I confirm that I have seen and been given the opportunity to read both the Material and the Article to be published by Journal of big data.

• Availability of data and materials

The data that support the findings of this study are available on request.

• Competing interests

The author(s) declare(s) that they have no competing interests

• Funding

No funding was received for this work.

• Authors' contributions

We attest that all authors contributed significantly to the creation of this manuscript, each having fulfilled criteria. We confirm that the manuscript has been read and approved by all named authors. We confirm that the order of authors listed in the manuscript has been approved by all

• Acknowledgements

This work is supported by the Gulf University for Science and Technology, Kuwait.

Conflict of Interest

We wish to confirm that there are no known conflicts of interest associated with this publication

Abdi, A., Shamsuddin, S. M., Hasan, S., & Piran, J. (2019). Deep learning-based sentiment classification of evaluative text based on Multi-feature fusion. Information Processing & Management, 56(4), 1245-1259.
Abdul-Mageed, M. (2017). Not all segments are created equal: Syntactically motivated sentiment analysis in lexical space. Paper presented at the Proceedings of the third Arabic natural language processing workshop.
Abdul-Mageed, M. (2019). Modeling arabic subjectivity and sentiment in lexical space. Information Processing & Management, 56(2), 291-307.
Abdulla, N. A., Ahmed, N. A., Shehab, M. A., & Al-Ayyoub, M. (2013). Arabic sentiment analysis: Lexicon-based and corpus-based. Paper presented at the 2013 IEEE Jordan conference on applied electrical engineering and computing technologies (AEECT).
Abdulla, N. A., Ahmed, N. A., Shehab, M. A., Al-Ayyoub, M., Al-Kabi, M. N., & Al-rifai, S. (2014). Towards improving the lexicon-based approach for arabic sentiment analysis. International Journal of Information Technology and Web Engineering (IJITWE), 9(3), 55-71.
Ahmad, I. S., Bakar, A. A., & Yaakub, M. R. (2019). A review of feature selection in sentiment analysis using information gain and domain specific ontology. International Journal of Advanced Computer Research, 9(44), 283-292.
Ahmad, I. S., Bakar, A. A., Yaakub, M. R., & Darwich, M. Beyond Sentiment Classification: A Novel Approach for Utilizing Social Media Data for Business Intelligence.
Ahmad, I. S., Bakar, A. A., Yaakub, M. R., & Darwich, M. (2020). Sequel movie revenue prediction model based on sentiment analysis. Data Technologies and Applications.
Ahmad, S. R., Bakar, A. A., & Yaakub, M. R. (2019). A review of feature selection techniques in sentiment analysis. Intelligent Data Analysis, 23(1), 159-189.
Al-Ayyoub, M., Khamaiseh, A. A., Jararweh, Y., & Al-Kabi, M. N. (2019). A comprehensive survey of arabic sentiment analysis. Information Processing & Management, 56(2), 320-342.
Al-Azani, S., & El-Alfy, E.-S. (2018). Emojis-based sentiment classification of Arabic microblogs using deep recurrent neural networks. Paper presented at the 2018 International Conference on Computing Sciences and Engineering (ICCSE).
Al-Azani, S., & El-Alfy, E.-S. M. (2017). Hybrid deep learning for sentiment polarity determination of arabic microblogs. Paper presented at the International Conference on Neural Information Processing.
Al-Moslmi, T., Albared, M., Al-Shabi, A., Omar, N., & Abdullah, S. (2018). Arabic senti-lexicon: Constructing publicly available language resources for Arabic sentiment analysis. Journal of information science, 44(3), 345-362.
Al-Moslmi, T., Omar, N., Abdullah, S., & Albared, M. (2017). Approaches to cross-domain sentiment analysis: A systematic literature review. Ieee access, 5, 16173-16192.
Al-Osaimi, S., & Badruddin, K. M. (2014). Role of Emotion icons in Sentiment classification of Arabic Tweets. Paper presented at the Proceedings of the 6th international conference on management of emergent digital ecosystems.
Al-Rubaiee, H., Qiu, R., & Li, D. (2016). Identifying Mubasher software products through sentiment analysis of Arabic tweets. Paper presented at the 2016 International Conference on Industrial Informatics and Computer Systems (CIICS).
Al-Smadi, M., Al-Ayyoub, M., Jararweh, Y., & Qawasmeh, O. (2019). Enhancing aspect-based sentiment analysis of Arabic hotels’ reviews using morphological, syntactic and semantic features. Information Processing & Management, 56(2), 308-319.
Al-Twairesh, N., Al-Khalifa, H., Al-Salman, A., & Al-Ohali, Y. (2017). Arasenti-tweet: A corpus for arabic sentiment analysis of saudi tweets. Procedia Computer Science, 117, 63-72.
Al-Twairesh, N., Al-Khalifa, H., Alsalman, A., & Al-Ohali, Y. (2018). Sentiment analysis of arabic tweets: Feature engineering and a hybrid approach. arXiv preprint arXiv:1805.08533.
Al-Zyoud, A., & Al-Rabayah, W. A. (2015). Arabic stemming techniques: comparisons and new vision. Paper presented at the 2015 IEEE 8th GCC Conference & Exhibition.
Al Shboul, B., Al-Ayyoub, M., & Jararweh, Y. (2015). Multi-way sentiment classification of arabic reviews. Paper presented at the 2015 6th International Conference on Information and Communication Systems (ICICS).
Alayba, A. M., Palade, V., England, M., & Iqbal, R. (2018). A combined CNN and LSTM model for arabic sentiment analysis. Paper presented at the International cross-domain conference for machine learning and knowledge extraction.
Alhumoud, S. (2019). Twitter Analysis for Intelligent Transportation. The Computer Journal, 62(11), 1547-1556.
Alhumoud, S. O., Altuwaijri, M. I., Albuhairi, T. M., & Alohaideb, W. M. (2015). Survey on arabic sentiment analysis in twitter. International Science Index, 9(1), 364-368.
Aliane, A., Aliane, H., Ziane, M., & Bensaou, N. (2016). A genetic algorithm feature selection based approach for Arabic Sentiment Classification. Paper presented at the 2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA).
Almuqren, L., & Cristea, A. I. (2016). Framework for sentiment analysis of Arabic text. Paper presented at the Proceedings of the 27th ACM conference on hypertext and social media.
Alrefai, M. a., Faris, H., & Aljarah, I. (2018). Sentiment analysis for arabic language: a brief survey of approaches and techniques. arXiv preprint arXiv:1809.02782.
Altowayan, A. A., & Tao, L. (2016). Word embeddings for Arabic sentiment analysis. Paper presented at the 2016 IEEE International Conference on Big Data (Big Data).
Boudad, N., Faizi, R., Thami, R. O. H., & Chiheb, R. (2017). Sentiment classification of Arabic tweets: A supervised approach. Journal of Mobile Multimedia, 233-243.
Duwairi, R., & El-Orfali, M. (2014). A study of the effects of preprocessing strategies on sentiment analysis for Arabic text. Journal of information science, 40(4), 501-513.
Duwairi, R. M., & Alshboul, M. A. (2015). Negation-aware framework for sentiment analysis in Arabic reviews. Paper presented at the 2015 3rd International Conference on Future Internet of Things and Cloud.
Elnagar, A., Al-Debsi, R., & Einea, O. (2020). Arabic text classification using deep learning models. Information Processing & Management, 57(1), 102121.
Farha, I. A., & Magdy, W. (2021). A comparative study of effective approaches for Arabic sentiment analysis. Information Processing & Management, 58(2), 102438.
Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82-89.
Hahnen, E., Lederer, B., Hauke, J., Loibl, S., Kröber, S., Schneeweiss, A., . . . Jackisch, C. (2017). Germline mutation status, pathological complete response, and disease-free survival in triple-negative breast cancer: secondary analysis of the GeparSixto randomized clinical trial. JAMA oncology, 3(10), 1378-1385.
He, W., Zha, S., & Li, L. (2013). Social media competitive analysis and text mining: A case study in the pizza industry. International journal of information management, 33(3), 464-472.
Hutto, C., & Gilbert, E. (2014). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Paper presented at the Proceedings of the International AAAI Conference on Web and Social Media.
Ismail, S., Alsammak, A., & Elshishtawy, T. (2016). A generic approach for extracting aspects and opinions of Arabic reviews. Paper presented at the Proceedings of the 10th international conference on informatics and systems.
Kang, D., & Park, Y. (2014). based measurement of customer satisfaction in mobile service: Sentiment analysis and VIKOR approach. Expert Systems with Applications, 41(4), 1041-1050.
Mubarak, H., Rashed, A., Darwish, K., Samih, Y., & Abdelali, A. (2020). Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192.
Nabil, M., Aly, M., & Atiya, A. (2015). Astd: Arabic sentiment tweets dataset. Paper presented at the Proceedings of the 2015 conference on empirical methods in natural language processing.
Pandey, A. C., Rajpoot, D. S., & Saraswat, M. (2017). Twitter sentiment analysis using hybrid cuckoo search method. Information Processing & Management, 53(4), 764-779.
Rushdi-Saleh, M., Martín-Valdivia, M. T., Lopez, L. A. U., & Perea-Ortega, J. M. (2011). Bilingual experiments with an arabic-english corpus for opinion mining. Paper presented at the Proceedings of the International Conference Recent Advances in Natural Language Processing 2011.
Shukri, S. E., Yaghi, R. I., Aljarah, I., & Alsawalqah, H. (2015). Twitter sentiment analysis: A case study in the automotive industry. Paper presented at the 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT).
Tartir, S., & Abdul-Nabi, I. (2017). Semantic sentiment analysis in Arabic social media. Journal of King Saud University-Computer and Information Sciences, 29(2), 229-233.

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Sentiment Analysis of Arabic Reviews Using a Hybrid Clustering Technique

Status:

Version 1

Abstract

Figures

1. Introduction

2. Challenges Of The Arabic Language

3. Related Work

3.1 The Lexicon-Based Approach

3.2 The Machine Learning Approach

3.3 The Hybrid Approach

4. Proposed Method

4.1 The Machine Learning Model

4.2 The Lexicon-based Model

4.3 The Hybrid Model

4.4 Data Collection and Preparation

4.5 Data Preprocessing and Tokenization

4.5.1 Emoticon conversion

4.5.2 Contextual intensification

4.5.3 Miscellaneous text cleaning and preparation

4.6 Feature Extraction

4.6.1 Feature extraction calculation method

4.6.2 Feature engineering

5. Experiments And Results

5.1 Dataset Description

5.2 Results of the Machine Learning Model

5.3 Results of the Lexicon-based Model

5.4 Results of the Hybrid Model

5.5 Discussion

6. Conclusion

Declarations

References

Additional Declarations

Status:

Version 1