Data Retrieval
The primary data that is utilized for this project is written text data from Twitter users called ‘Tweets’ and is retrieved from two sources. The first set of tweets will be used for model selection and evaluation while the second set of data will be used for implementation of the dashboard.
Modelling Data
The first set of data are pre-labeled tweets retrieved from the Huggingface TweetEval sentiment testing dataset made up of 12,284 tweets labelled as positive, negative and neutral. (Barbieri, et al., 2020)
Table 1 displays the 1st five rows of the test sample. The text column contains user tweets that have been stripped of user details while the sentiment column denotes the sentiment label of the corresponding text made up of ‘neutral’, ‘positive’ and ‘negative’.
Implementation Data (Live Data)
The second set of data used in this project is the live data that was used for the implementation of the dynamic opinion dashboard, and it is extracted directly from Twitter using specific queries.
Query Building
Twitter allows the building of specific queries to ensure that the tweets retrieved match the requirements as much as possible. For this project, the dashboard will allow the user to build the query as detailed in Table 2 for retrieval of matching tweets from the Twitter website.
The retrieval of live tweets using the search query can be achieved either using third-party online scrapers or the authorized Twitter API.
Third-Party Twitter Search
Python has a range of third-party Twitter search libraries which provide a way to scrape Twitter data without using the official Twitter API. These libraries are particularly useful for researchers and data analysts who need to collect large amounts of Twitter data quickly and efficiently. Some examples of third-party Twitter search libraries include Tweepy, Twint, and Snscrape.
The Snscrape library was used for the implementation of a dynamic opinion dashboard in this project, and it is a scraper for social networking services (SNS) and supports social media sites such as Facebook, Instagram, Mastodon, Reddit, Telegram, and Twitter with varying degrees of access. For Twitter, which is the focus of this research, Snscrape can retrieve users, user profiles, hashtags, searches (live tweets, top tweets, and users), tweets (single or surrounding thread), list posts, communities, and trends. (JustAnotherArchivist, 2023).
Elon Musk's purchase of Twitter resulted in several changes, one of which had a negative impact on the use of third-party web scrapers on Twitter after the coding for this project had been concluded. In April 2023, Twitter stopped allowing unregistered users to utilize the search feature. Anyone attempting to access the site's search results is required to log in to their existing account or have accredited credentials for searching because of the change, making third-party web scrapers such as Snscrape unable to scrape tweets from the Twitter website unless Twitter changes course.
Twitter Developer API
The Twitter Developer API (Application Programming Interface) is a powerful tool that grants programmatic access to developers enabling them to create applications that interact with the Twitter platform in unique and advanced ways. With Twitter API, developers can access various endpoints and resources that provide real-time access to the Twitter stream, including tweets, timelines, user information, and more. The Twitter Developer API offers several different tiers with varying levels of access and capabilities, making it accessible to developers of all skill levels and requirements Table 3 details the different Access Levels available as of April 2023 and based on the requirement of the dashboard, only the paid tiers of Basic and Enterprise will suffice as they provide access to the required Twitter search functionality.
Pre-Processing and Cleaning
Data pre-processing and cleaning are essential steps in preparing text data for analysis. These steps involve transforming raw text data into a format that computers can easily analyze. Text data can be very messy and may contain irrelevant information, such as stop words, punctuation, and special characters, that can interfere with the analysis. As a result, text data must be cleaned and pre-processed to ensure that it is consistent, useful, and ready for analysis (DeVille & Bawa, 2021).
Pre-Processing
The below pre-processing steps were taken to prepare the data for analysis.
Apply date format to date column
This step helps to ensure consistency in the date format and makes it easier to analyze the data over time. It involves changing the date column formatted as a string as retrieved from Twitter a date format using Python Pandas.
Cleaning
The below text cleaning steps were further taken to reduce the complexity of the text data and make it more manageable for analysis. To compare the raw and cleaned text, a new column was created as a copy of the text column.
Removal of newlines and lowercasing all letters
In this step, the newlines were removed, and all letters were converted to lowercase to standardize the text and simplify the analysis.
Removal of user handles from tweets
In this step, the user handles from tweets were removed to eliminate noise and irrelevant information that could interfere with the analysis.
Expansion of Contractions
This step involves expanding contractions such as "can't" to "cannot", "won't" to "will not", and so on to ensure that the text is consistent.
Removal of Punctuations
This step involves removing all such punctuations such as periods, commas, exclamation marks, and question marks from the text because punctuations can interfere with text analysis.
Removal of Consecutive Characters (Simple spell check)
This step involves replacing letters that occur three or more times in a row, such as “loooove” or “wooooow” with just two occurrences of that character, such as “love” or “wow”. Users tend to extend words to show emphasis, however, this tends to cause a misinterpretation of the word by computers. The English language consecutive repetition of characters is a maximum of twice in words such as “Committee”, “Cheerfulness” or “Bookkeeper”.
Removal of Extra Spaces
This step involves removing all unnecessary spaces, such as multiple spaces between words or at the beginning/end of a sentence that a tweet may contain.
Removal of Stop Words
This step involves removing all stop words from the text. Stop words are words that are commonly used in a language but do not carry any meaning in the context of the text. Examples of stop words include "the", "and", "of", "in", etc.
Table 11 shows the differences in the text data before and after cleaning the pre-processed text for the 1st five rows of data from the test dataset.
Computational Environment
The computing technologies and tools used to develop and deploy the dynamic opinion dashboard in this project were Python programming language, Flask web framework and HTML.
Model Review
This project takes advantage of the available pre-trained models for sentiment analysis as they have already been trained on large datasets and have a high level of accuracy. Two popular pre-trained models for sentiment analysis are Vader and TweetNLP.
VADER
Valence Aware Dictionary and sEntiment Reasoner (Vader) is a lexicon and rule-based sentiment analysis tool built particularly for analysing the sentiment of social media communications, which frequently contain informal language, slang, and sarcasm. It was developed by C.J. Hutto and Eric Gilbert and presented in the paper “VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text”.
VADER was created using a human-centered methodology that combined qualitative analysis with experimental research, empirical validation, and evaluation. The model was built with a human-curated, industry-leading sentiment vocabulary that considers the polarity and strength of feelings expressed in social media microblogs. It also includes several lexical features that are typical of sentiment expression in microblogs, such as emoticons, acronyms, and widely used slang. A text's sentiment score is calculated by aggregating the sentiment scores of its lexical aspects, taking polarity, intensity, and context into consideration. VADER also applies a set of rules and heuristics to deal with negation, amplification, and other linguistic expressions that might have an impact on sentiment. For example, the sentence "I love this movie, but the ending was terrible." using a lexicon-based approach would yield a positive sentiment score for "love" and a negative sentiment score for "terrible." However, VADER's rule-based approach takes into account the word "but," which indicates a shift in sentiment, and yields a final sentiment score of 0.56, which is slightly positive. The results of the evaluation of VADER by the creators found that it performed exceptionally well in the social media domain and generalized favourably (C.J. Hutto & Gilbert, 2014).
TweetNLP
TweetNLP is a natural language processing tool that is specifically designed to process text data from social media platforms, such as Twitter. It was developed by a team of researchers in the paper “TweetNLP: Cutting-Edge Natural Language Processing for Social Media”.
TweetNLP was created to overcome the unique challenges posed by social media text data. Social media text is often characterized by its brevity, informality, and use of non-standard language such as abbreviations, slang, and emoticons. Additionally, social media text data is often unstructured, noisy, and contains a high degree of ambiguity. Below are some of the features of TweetNLP:
- TweetNLP utilizes a combination of machine learning techniques, including statistical models, neural networks, and rule-based systems.
- TweetNLP is trained using a considerable quantity of annotated social media text data to learn how to accurately parse and analyze tweets.
- TweetNLP can handle a variety of NLP tasks, for instance part-of-speech marking, entity identification, sentiment analysis, and more.
- TweetNLP can handle the unique features of social media text, such as hashtags, mentions, and URLs.
Evaluation Metrics
Evaluation metrics are used to evaluate how accurately models classify data after training. In this project, the sentiment analysis models are evaluated based on how accurate they are in classifying the sentiment of tweets when compared to the human classification. There are various evaluation metrics used in sentiment analysis, each with its strengths and weaknesses. Some of the most used metrics include accuracy, precision, recall and, F1 score. In this 3-class classification problem of this project that is sentiment analysis the Precision, Recall and F1 Score were the selected metrics that will be used for evaluation.
Modelling
The sentiment analysis predictions using the Vader and TweetNLP sentiment analysis models on a dataset of 12,284 tweets from the Huggingface TweetEval sentiment testing were evaluated and compared. The comparison was based on various metrics and visualizations, such as bar charts of sentiments in each class, confusion matrix, precision, recall, and F1 Score. The comparison was also performed on both raw and cleaned text, to investigate the effect of text preprocessing on the performance of the models and then compared to the human-classified sentiment labelled as positive, negative and neutral that was available in the test dataset. The below shows the name and description of each of the classifications.
Human Sentiment: This represents the pre-labelled human classified sentiment.
Vader Sentiment (Raw): This represents the sentiment classification using the Vader sentiment analysis model on the original text.
TweetNLP Sentiment (Raw): This represents the sentiment classification using the TweetNLP sentiment analysis model on the original text.
Vader Sentiment (Clean): This represents the sentiment classification using the Vader sentiment analysis model on the cleaned text.
TweetNLP Sentiment (Clean): This represents the sentiment classification using the TweetNLP sentiment analysis model on the cleaned text.
Figure 1 shows a bar chart comparing the results of the models on the clean and raw text against the actual human-classified sentiments. Looking at the individual model performance when applied to the raw and cleaned text, the graph shows that the Vader sentiment analysis model only performed well using the cleaned text in the negative sentiment class while in the positive and neutral classes, the Vader sentiment analysis model performed better using the raw text. The TweetNLP sentiment analysis model also only performed best on cleaned text in the neutral sentiment class and had a performance on the raw text in both the negative and positive sentiment classes. Figure 3 shows that the TweetNLP sentiment analysis model positive classified tweet count of 2372 almost exactly matched the human classified positive tweet count of 2375.
A comparison of both models to the human classified sentiment shows that for negative tweets, the Vader sentiment analysis model had the best performance when the cleaned text was analysed however for both the neutral and positive sentiment classes, the TweetNLP sentiment analysis model performed best on the raw text.
Figure 2 shows the confusion matrix of both the Vader and TweetNLP sentiment analysis models process on the cleaned and raw text evaluated against the human classified sentiments. The Vader sentiment analysis model when categorized on the raw text had 2176, 2739 and 1591 tweets that were classified as True Negative, True Neutral and True Positive respectively giving it a total of 6506 correctly classified tweets out of 12284 however, when it was categorized using the cleaned text, it had 2281, 2569 and 1641 classified as True Negative, True Neutral and True Positive respectively giving it a total of 6491 correctly classified tweets. A review of the TweetNLP sentiment analysis model confusion matrix showed that when categorized on the raw text the model had 3202, 3926 and 1692 tweets that were classified as True Negative, True Neutral and True Positive respectively giving it a total of 8820 correctly classified tweets and when it was categorized using the cleaned text, it had 3213, 4096 and 1442 classified as True Negative, True Neutral and True Positive respectively giving it a total of 8751 correctly classified tweets out of 12284.
Using the results from the confusion matrix in Figure 4 the precision, recall and f1 score metrics were extracted as displayed in Table 13. The Vader sentiment analysis model had a precision of 53.85, recall of 55.97 and f1 score of 52.61 when classified on the raw text. When the same model was classified using the cleaned text, it had a precision of 53.91, recall of 56.03 and f1 score of 52.67 showing that the models’ overall performance was better on the cleaned text. On the other hand, with the raw text, the TweetNLP sentiment analysis model had a precision of 71.60, recall of 72.66 and f1 score of 71.88 compared to the results of the analysis on the cleaned text which had had a precision of 71.49, recall of 71.59 and f1 score of 71.24. In the case of the TweetNLP sentiment analysis model, the models’ overall performance was better on the raw text.
Selection of Best Fit Model
Based on the insights gained from the results of the result and metric comparisons, it is evident that classifying the raw original tweet as written by the user with the TweetNLP Sentiment Analysis model is a better fit for this project. In all the metrics measured, the model performed far better than the Vader Sentiment Analysis model and achieved a precision of 71.60, recall of 72.66 and f1 score of 71.88 and a total of 8820 correctly classified tweets out of 12284.
The only notable shortcoming of the TweetNLP Sentiment Analysis model was the required runtime. On average, the model took 157.7s to classify 12284 tweets using a GPU hardware accelerator. This run time is much longer when run on an ordinary CPU however due to the immense improvement in metrics, the runtime can be deemed acceptable.
Analysis of Selected Model
The Human vs TweetNLP sentiment classification comparison shown in the bar chart in Figure 3 shows that the sentiment classification from the TweetNLP Sentiment Analysis model closely matches the human sentiment classification making it suitable for use in real-time sentiment analysis required by the dynamic dashboard.
The TweetNLP Sentiment Analysis model attaches a probability between 0 and 1 to each sentiment label and selects the sentiment label with the highest probability as the final sentiment of the text analysed and an extract of the TweetNLP classified dataset is shown in Table 14.
An investigation into the probability spread as highlighted in Figure 4 shows that most of the classification probabilities are 0.5 and higher with only 4% of the classifications having a probability less than 0.5. It can be inferred that sentiments that have probabilities less than 0.5 were ambiguous making it hard for the model to classify as seen in the samples in Table 15.
Using the findings from the analysis of the results available from the TweetNLP Sentiment Analysis model, this project will be filtering out all tweets with a probability less than 0.5 to maintain a high confidence level from the results that would be generated. Calculating the metrics using the condition of probabilities higher than 0.5 we have been able to increase the precision, recall and f1 score by 2% thereby achieving final metrics for the TweetNLP Sentiment Analysis model in the dynamic dashboard. Precision: 72.71, Recall: 73.93 and F1 Score: 73.03
Dynamic Opinion Dashboard Architecture and Design
The Dynamic Opinion Dashboard developed is a web application that allows users to collect and analyze public opinion on a variety of topics. The dashboard is built on a Python Flask app that interacts with a Twitter API to collect tweets, a sentiment analysis library to classify the tweets, and a variety of visualization libraries to create charts and graphs.
The Dashboard architecture diagram in Figure 5 illustrates how the user interacts with the HTML interface, which triggers a series of events that result in the sentiment analysis output. The dashboard can be divided into 3 main segments, the user interface built on HTML, the backend application running in Python and Twitter, and the data source.
Backend Functions
The backend of the dynamic dashboard is the Flask app and acts as the engine of the entire dashboard. It allows a user to search for tweets based on a topic, location, and date range, scrapes the tweets, applies sentiment analysis to each tweet, and produces two plots: a count plot of the sentiment classification of the tweets and a line plot of the number of tweets per day for each sentiment classification. The code in the application can be broken down into code blocks detailed below.
The home function sets up a Flask web application and defines a route for the root URL path ("/"). When a user visits the root URL of the web application, the home function is called, which returns the home.html template using the render template function. The results function defines a route for the /result endpoint with the POST method. When a user submits the form filled in the home.html the functions associated with the result endpoint decorator will be executed. The functions associated function will be used to extract and process the data and return a response to the user. The user input retrieved from the HTML form is then parsed to retrieve values for the topic, location, start date, end date, and max tweets fields from the form and then converts the max tweets value to an integer using int(). If any of the form fields are missing or if the max tweets value cannot be converted to an integer, it returns an error message by rendering an error template. Additionally, the topic input is modified by replacing any commas with " OR " based on the requirements of the Twitter query builder and replacing any double spaces with single spaces.
To enable the efficient scraping of tweets the start date and end date are converted from strings to datetime objects, and then the period requested is divided into 10 balanced periods. For each period, the code creates a query to search for tweets containing the specified topic and location within a 10 km radius. It then scrapes 10% of the total number of tweets requested, for each period and appends the scraped tweets to a list. Once all the tweets have been scraped, the code creates a Pandas data frame from the list of tweets. If there are no tweets, it renders an error template with a message saying, "No tweets found". If any exceptions occur during the scraping process, it catches them and renders an error template with an appropriate message.
After the initial pre-processing, the TweetNLP sentiment analysis model is applied to each of the pre-processed tweets. The model takes in a text input and returns the tweet's sentiment label and its associated probability.
The visualizations included in the results page of the dashboard are created from three functions for creating visualizations and generating summaries from the data. The first visualization creates a bar chart of the tweet sentiment classes. The second visualization groups the tweets by date, and sentiment class then counts the number of tweets in each group to create a line graph of each sentiment over time. The last visualization generates a word cloud of the most frequently occurring words in tweets from both the positive and negative tweets sentiment classes. The word clouds are generated from the cleaned tweets after further cleaning such as, the removal of the topic and removal of words with 2 characters or less has been done. The plots generated from all the visualizations are saved as static files for rendering in the results HTML.
Once all the code in the results function is run the results template is returned with several variables defined as parameters. These variables will be used in the HTML code of the 'result.html' template to display the various results and visualizations related to sentiment analysis of tweets.
User Interface
The User Interface (UI) of the dashboard is the front-end design and is implemented using HTML and CSS. The UI provides a visual representation of the data and functionality that was implemented in the back end. The UI consists of four templates - home.html, results.html, tweets_table.html, and error.html. These templates provide the foundation for the different pages of the dashboard and determine the layout, styling, and behaviour of the various UI elements. A well-designed UI is critical to the success of any dashboard as it helps to communicate the insights and functionality to the user intuitively and efficiently.
Home Page
The home page of the dashboard as seen in Figure 6 is the first page the user sees when accessing the dashboard. The home HTML template is a form that allows the users to search for tweets based on specified search criteria, location, start date, end date, and maximum number of tweets to retrieve. The form has been styled with CSS. Upon submission of the form, the user will be directed to the 'result' page to view the tweets.
Results Page
The HTML template displayed in Figure 6 is designed to illustrate the sentiment analysis of tweets regarding a certain topic. The template takes the variables and files rendered from the results function and uses them to populate the text summary. The template also includes sentiment plots detailing the sentiment count by class, sentiment count over time, word clouds that illustrate the frequency of words used in positive and negative tweets, and a button to view the tweets table. The user can initiate a new search by clicking the "New Search" button.