An Integrated Clustering and BERT Framework for Improved Topic Modeling

doi:10.21203/rs.3.rs-1986180/v1

Download PDF

Research Article

An Integrated Clustering and BERT Framework for Improved Topic Modeling

https://doi.org/10.21203/rs.3.rs-1986180/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Topic modelling is a machine learning technique that is extensively used in Natural Language Processing (NLP) applications to infer topics within unstructured textual data. Latent Dirichlet Allocation (LDA) is one of the most used topic modeling techniques that can automatically detect topics from a huge collection of text documents. However, the LDA-based topic models alone do not always provide promising results. Clustering is one of the effective unsupervised machine learning algorithms that are extensively used in applications including extracting information from unstructured textual data and topic modeling. A hybrid model of Bidirectional Encoder Representations from Transformers (BERT) and Latent Dirichlet Allocation (LDA) in topic modeling with clustering have been studied in detail. As the clustering algorithms are computationally complex, the complexity increases with the higher number of features, the PCA, t-SNE & UMAP-based dimensionality reduction methods are also performed. Finally, a unified clustering-based framework using BERT and LDA is proposed as part of this study for mining a set of meaningful topics from the massive text corpora. The experiments are conducted to demonstrate the effectiveness of the cluster-informed topic modeling framework using BERT and LDA by simulating user input on benchmark datasets. The experimental results show that clustering would help infer more coherent topics and hence this unified clustering and BERT-LDA based approach can be effectively utilized for building topic modeling applications.

Latent Dirichlet Allocation (LDA)

Topic modeling

k-means clustering

Dimensionality Reduction

Bidirectional Encoder Representations from Trans-formers (BERT)

As the online data volume is getting bigger exponentially, it is very much essential to understand the topics from the available data speedily and accurately. There is a demand for an automatic topic modeling system and text summarization, due to the enormous volume of text data available in present times and the limitation of human reading abilities. There are multiple techniques used to discover the topics from text, images, and video [1] Topic models have been extensively utilized in Topic Recognition and Tracking tasks, which helps to track, detect, and designate topics from a stream of documents or texts [4]. This machine learning technique is widely used in NLP applications to analyze the unstructured textual data and automatically discover the abstract topics within it.

This study intended to model different text mining techniques to infer hidden semantic structures and hence the topics in a text corpus. Topic modeling experiments are conducted using the CORD-19 dataset. The topic modeling investigations are performed based on sentence embedding after pre-processing. The experimental results are found very promising, and it also unveils the opportunity for further research, precisely by applying hybrid methods to improve the efficiency of the proposed based models. As an extended work, the applications of k-means clustering and Latent Dirichlet Allocation (LDA) in topic modeling have been studied. As the clustering algorithms are computationally complex, the complexity increases with the bigger number of features, and multiple dimensionality reduction processes are also performed. Finally, an efficient BERT based clustering framework to perform automatic topic modeling system is proposed. The proposed system is implemented, and the performance evaluation is also performed by computing Silhouette Score. The rest of the paper is structured as follows. Section 2 deals with the review of the related works. Section 3 describes the Dataset used, and the preprocessing techniques applied to it. Section 4 elucidates the comprehensive experimental results and finally, section 5 concludes the work and presents the future research directions.

Multiple machine learning based implementation of various topic modelling techniques have been extensively studied by many researchers [ 2,3,4,5,6]. Organizing, searching, and summarizing large volume of textual data are the major tasks involved in NLP and the topic modelling can be used to solve these problems to certain extent. Some of the important techniques that are broadly used for topic modelling are detailed below.

A. Latent Semantic Analysis (LSA): The Vector Space Model (VSM) was widely used in the initial stage of text processing, in which a document is denoted as a vector [16]. LSA came into picture to overcome the drawback of LSA, as it could not capture inter and intra- document statistical structure. It helps in examining relations between the set of documents and the terms it comprises by creating a set of conceptions related to the documents and terms.

B. Latent Dirichlet Allocation (LDA): It is the most widely used algorithm for topic modelling based on statistical analysis. It aims to discover the topic belongs to a document, based on the word that contains it [17]. Each latent topic of documents is distinguished by a probabilistic distribution around words and the word allocations of topics assign a common Dirichlet.

C. Correlated Topic Models (CTM): The correlated topic model is developed to overcome the issues of LDA [18]. CTM is more capable to model the correlation between topics than LDA. CTM is used when the topics are likely to be correlated.

D. Top2Vec Model: Top2Vec algorithm helps automatically identify the number of topics from the document and creates jointly embedded topic, document, and word vectors. Moreover, this algorithm does not require pre-processing like stop word removal and stemming/lemmatization [19].

This section describes the features of the dataset used for the study and the data preprocessing techniques applied prior to the feature extraction.

3.1 Dataset

To conduct of experiments to cluster and predict the topic, the Benchmark dataset, COVID-19 Open Research Dataset (CORD-19), which contains 900,000 + academic articles about COVID-19, SARS-CoV-2, and related coronaviruses for use by the worldwide research community and we considered 801857 documents which are published after Jan 2020[8]. The Covid-19 Open Research Dataset (CORD-19) is an increasing source of scientific papers on Covid-19 and related past coronavirus research. CORD-19 is designed to enable the development of text mining and information retrieval techniques over its rich collection of metadata and structured full-text papers.

3.2 Data Preprocessing

The data preprocessing techniques are used for information mining which includes the transformation of raw data and making it suitable for machine learning models. Real-world information is often inaccurate, incompatible, and/or missing in certain behaviors or trends and is probable to contain many mistakes. Preprocessing data is a proven technique for solving such problems as it prepares raw data for further processing. Initially, the input data is pipelined through Data Cleaning, Data Integration, Data Transformation, Data reduction, and Data discretization preprocessing techniques. The following preprocessing techniques and algorithms are used in this study.

BERT (Bidirectional Encoder Representations from Transformers): It is a Natural Language Processing Model proposed by Google Researchers [11]. It is a topic modeling technique that uses transformers (BERT embeddings) and class-based TF-IDF to generate dense clusters. The generated resultant topics can be easily interpreted and visualized using this BERT algorithm We have used Sentence Transformers in this paper.

Latent Dirichlet Allocation (LDA): It is a widely used text classification algorithm based on statical topic models. We can easily discover and visualize topics from the document using this model.

Clustering: It is an unsupervised technique used to group the data points into a similar group. We can use clustering to group the topics [12] so that we can find the topics within these clusters. We used the k-means algorithm for clustering.

K-means Clustering: It is an Unsupervised Learning algorithm, which groups the unlabeled dataset into different groups [13]. Here k describes the number of pre-defined clusters, as if k = 8, there will be 8 clusters. To identify the correct clusters, applied the Elbow Method.

Dimensionality Reduction: Dimensionality reduction is a significant step in text mining. Dimension reduction enhances the performance of clustering methods by reducing dimensions so that text mining techniques process data with a reduced number of terms. There are multiple methods available for dimensionality reduction. Here we have used PCA, UMAP, and T-SNE.

PCA (Principal component analysis): Principal Component Analysis is a dimensionality reduction method that is applied to reduce the dimensionality of massive data sets, by altering a large collection of variables into a smaller one which retains the information in the massive data set.

UMAP (Uniform Manifold Approximation and Projection): It is one of the novel techniques for dimension reduction. It is comparatively performing better, and it keeps a considerable portion of the high-dimensional structure into lower dimensional structure.

t-Distributed Stochastic Neighbor Embedding (t-SNE): It is a technique for dimensionality reduction which is predominantly suited for high-dimensional datasets visualization.

The following section describe the experimental setup and analyze the results in detail.

As previously stated, unsupervised clustering and transformer models are used in the work for topic modelling. The experiments are conducted to identify the topics from a set of large text corpora. The experiments have been performed on the CORD-19 dataset. The preprocessing techniques are applied on the collected subset of data. As a next step, the knowledge retrieval method that weighs a term’s frequency (TF) and its inverse document frequency (IDF) are computed from the preprocessed document. The TF-IDF vectors are passed to the LDA and BERT Model to identify the topics. Then the dimensionality reduction of the embeddings is performed as the clustering algorithms poorly handle high dimensionality. The PCA, UMAP & t-SNE algorithms are used for the dimensionality reduction.

The reduced dimensionality results are clustered using the k-means clustering algorithm and evaluated the results. The Elbow Method is used to identify the optimum k that can be used for clustering purpose. The combined outputs of LDA and BERT models are then passed to the Autoencoder to discover a lower-dimensional latent space representation.

The integrated development environment (IDE) named Spyder in open-source cross-platform is used for experimentation purposes. To perform the machine learning experiments, Scikit-learn and NLTK tools are used.

The clustering results are evaluated based on Silhouette Score/Silhouette Coefficient. The Silhouette Coefficient is a metric applied to determine the goodness of a clustering method. The value ranges from − 1 to 1. This indicated that the clusters are perfectly separated from each other and distinguished. The Silhouette Scores obtained for different topic modelling methods with PCA, t-SNE and UMP algorithms used for dimensionality reduction are tabulated in Table 1. From the experimental results it is evident that the combined BERT-LDA model with UMAP outperforms other methods. Figure 1 (a-h) shows the various clustering results generated as part of this study.

Table 1: Silhouette Scores obtained for various models using different dimensionality reduction methods

Topic Models	Silhouette Scores
Topic Models	PCA	UMAP	t-SNE
LDA	0.248675	0.346303	0.016543
BERT	0.394638	0.446197	0.36938
LDA-BERT	0.50843	0.584835	0.471261

A hybrid model of Bidirectional Encoder Representations (BERT) from Transformers and Latent Dirichlet Allocation (LDA) in topic modeling with unsupervised clustering have been applied. The PCA, t-SNE & UMAP-based dimensionality reduction methods are used as the clustering algorithms are computationally ineffectual with the higher number of features. The UMAP dimensionality reduction showed comparatively better results. The reduced dimensionality results are clustered using the k-means clustering algorithm and evaluated the results. The Elbow Method is used to identify the precise number of clusters. The combined outputs of BERT and LDA models are then passed to the Autoencoder to discover a lower-dimensional latent space representation.

Finally, a unified clustering-based framework using BERT-LDA is designed and implemented for mining a set of meaningful topics from the input text corpora. The experiments are conducted to demonstrate the effectiveness of the cluster-informed topic modeling framework using BERT-LDA. The experimental results show that clustering would help infer more coherent topics and hence this unified clustering and BERT-based approach can be effectively utilized for building topic modeling applications.

As a future work, the BERT variants and the Deep Neural Networks (DNNs) models can be used to infer topics. The work can be extended to verify the effectiveness of the proposed topic modeling framework with the advanced transformers and deep learning-based algorithms reported in the literature.

Competing Interest

The authors declare that they have no competing interests

Ethics approval and consent to participate

Not applicable.

Consent for publication

The authors give the Publisher the permission to publish the work.

Funding

This research was not supported by any funded Project.

Author Contribution

The authors together wrote the entire manuscript text with details . Both authors have read and reviewed the final version of manuscript.

S. Qian, T. Zhang, C. Xu, and J. Shao, “Multi-modal event topic model for social event analysis,” IEEE Transactions on Multimedia,vol. 18, Feb. 2016, pp. 233–246, doi: 10.1109/TMM.2015.2510329.
F. Zhang, W. Gao, Y. Fang and B. Zhang, "Enhancing Short Text Topic Modeling with FastText Embeddings," 2020 International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE), 2020, pp. 255–259, doi: 10.1109/ICBAIE49996.2020.00060.
S. Limwattana and S. Prom-on, "Topic Modeling Enhancement using Word Embeddings," 2021 18th International Joint Conference on Computer Science and Software Engineering (JCSSE), 2021, pp. 1–5, doi: 10.1109/JCSSE53117.2021.9493816.
X. Wu and C. Li, "Short Text Topic Modeling with Flexible Word Patterns," 2019 International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–7, doi: 10.1109/IJCNN.2019.8852366.
H. Chen, X. Wang, S. Pan and F. Xiong, "Identify Topic Relations in Scientific Literature Using Topic Modeling," in IEEE Transactions on Engineering Management, vol. 68, no. 5, pp. 1232–1244, Oct. 2021, doi: 10.1109/TEM.2019.2903115.
T. Ma, Q. Pan, H. Rong, Y. Qian, Y. Tian and N. Al-Nabhan, "T-BERTSum: Topic-Aware Text Summarization Based on BERT," in IEEE Transactions on Computational Social Systems, vol. 9, no. 3, pp. 879–890, June 2022, doi: 10.1109/TCSS.2021.3088506.
G. Xu, Y. Meng, Z. Chen, X. Qiu, C. Wang and H. Yao, "Research on Topic Detection and Tracking for Online News Texts," in IEEE Access, vol. 7, pp. 58407–58418, 2019, doi: 10.1109/ACCESS.2019.2914097.
Lu Wang L, Lo K, Chandrasekhar Y,et.al. CORD-19: The Covid-19 Open Research Dataset. ArXiv [Preprint]. 2020 Apr 22: arXiv:2004.10706v2. PMID: 32510522; PMCID: PMC7251955.
J. C. Campbell, A. Hindle and E. Stroulia, "Latent Dirichlet Allocation: Extracting Topics from Software Engineering Data", Art Sci. Anal. Softw. Data, vol. 3, pp. 139–159, 2015
S. Jia, and B. Wu, “Incorporating LDA based text mining method to explore new energy vehicles in China,” IEEE Access, vol. 6, Nov.2018, pp. 64596–64602, doi: 10.1109/ACCESS.2018.2877716.
N. Peinelt, D. Nguyen and M. Liakata, tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection, pp. 7047–7055, 2020
J. A. Lossio-Ventura, J. Morzan, H. Alatrista-Salas, T. Hernandez-Boussard and J. Bian, "Clustering and topic modeling over tweets: A comparison over a health dataset", 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1544–1547, 2019
M. Alhawarat and M. Hegazi, "Revisiting K-means and topic modeling a comparison study to cluster arabic documents", IEEE Access, vol. 6, pp. 42740–42749, 2018.
Q. Xie, X. Zhang, Y. Ding and M. Song, "Monolingual and multilingual topic analysis using LDA and BERT embeddings", Journal of Informetrics, vol. 14, no. 3, 2020.
E. S. Negara, D. Triadi and R. Andryani, "Topic Modelling Twitter Data with Latent Dirichlet Allocation Method", 2019 International Conference on Electrical Engineering and Computer Science (ICECOS), pp. 386–390, 2019.
G. Salton, and M. J. McGill, Introduction to modern information retrieval, New York: McGraw-Hill. 1983..
D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet Allocation,” Journal of Machine Learning Research, vol. 3, May 2003, pp. 993–1022, doi: 10.1162/jmlr.2003.3.4-5.993.
D.M. Blei and J. D. Lafferty, “Correlated Topic Models,” Advances in Neural Information Processing Systems 18 (NIPS 2005), NIPS Press, Dec. 2005, pp. 147–154, doi: 10.1145/1143844.1143859.
Piyush Ghasiya,Koji Okamura, Investigating COVID-19 News Across Four Nations: A Topic Modeling and Sentiment Analysis Approach,IEEE Access, March 2021, DOI: 10.1109/ACCESS.2021.3062875
Cheng X et al (2014) Btm: topic modeling over short texts. IEEE Transactions on Knowledge and Data Engineering 26(1):2928–2941
Xie W et al (2016) Topicsketch: real-time bursty topic detection from twitter. IEEE Trans Knowl Data Eng 28(8):2216–2229

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

An Integrated Clustering and BERT Framework for Improved Topic Modeling

Status:

Version 1

Abstract

Figures

1 Introduction

2 Review Of Related Works

3 Dataset Used And Preprocessing

3.1 Dataset

3.2 Data Preprocessing

4 Experimental Results And Discussion

5 Conclusion And Future Work

Declarations

References

Additional Declarations

Status:

Version 1