The literature identifies two distinct methods for creating the text: the traditional method of deep learning approach and the transformer approach.
Traditional method: Deep neural networks, such as RNN, LSTM, and CNN, have demonstrated an impressive capacity to produce a synthetic text in recent years [5].
Every language has its own rules and conventions. It is very improbable that the CNN model developed for the English model will work effectively in other languages. Therefore, advanced deep learning models such as transformer: BERT, GPT [6] were introduced to overcome these constraints.
The works on text generation models and sentiment analysis that have been done recently are summarized in this section.
One of the most recent tasks in NLP is text generation, which has been used in numerous applications, including text summarization, question answering, and machine translation. In the paper proposed by [7], they major focus on current studies on text generation, which means the input to produce the desired result. They worked for Vietnamese Text Generation using RNN. They used a deep neural network model that creates summary texts by combining the input text and subject information. In this study, they propose that subject information from an input document plays a significant role in producing the final content.
In recent years, researchers have become increasingly interested in text generation utilizing LSTM networks. For instance, the authors created new phrases using an LSTM transformer model that was trained on a real-world dataset. They made up their dataset in 917 days on Bangle language newspaper articles. They used LSTM networks as they operate well for processing text data sequences and word prediction, as their target is to generate sequence-to-sequence text generation [8].
Data imbalance, or having more samples in one category than another, is a common issue in classification tasks. To bring balance to severely unbalanced text datasets, models based on GPT-2 and LSTM have recently been presented. The tests in this study were done on three very asymmetrical datasets from various areas demonstrating the performance of identical deep neural network models [9].
Here in this paper [10], they proposed a switch-GPT model. They make use of GPT-2's powerful language modeling capability to create fluent and well-formulated sentences. They used the Amazon dataset with the benchmark of bounded text generation within a few settings across many domains.
The aim is to look at the classification of emotions and sentiments for automatic text generation using deep learning algorithms. In this study, [11] the authors used two million comments from the online learning community to train the GPT-2 and RNN deep learning language models. The autonomous creation's output was then assessed by humans and placed on a readability test. The results show that the GPT-2 language model can function as prompts similar to human-written text. Whether a machine can think and be creative like a human is one element. In this study, [12] they use two different corpora to train the Open-AI GPT-2 model, which can output long phrases and articles before doing a comparison analysis. Parallel to that, they utilized the BERT model for the prediction of intermediate words from the context.
Generating text using adversarial networks is another field of interest for researchers. Recently many works have been proposed using Generative adversarial networks: C-Senti GAN and Senti-GAN. The authors concentrated on one of GAN's drawbacks, the inability to generate discrete token sequences. Numerous sets of tests using fictitious data as well as evaluations using actual data are carried out [13]. They generated texts using Cat-GAN and Senti-GAN and primarily concentrated on data balance because they were using severely unbalanced data in an educational domain. They also investigated how synthetic text production affected the sentiment classification job for the severely unbalanced dataset using deep learning and machine learning models [14]
As transformers were built to fulfill the requirement of text creation in different languages as well as for generating large paragraphs. Here, [15] they used it for the creation of a narrow-profile text in Russian is the study's goal. They used the transformers architecture, i.e., pre-trained deep neural network models GPT-2 and BERT. A model and software program application were trained as part of the study's framework to produce cohesive articles in the Russian language.
The authors used a dataset of Amazon product reviews for the sentiment analysis job and gathered human comments on model-generated summaries of the reviews. They discovered that by fine-tuning GPT-2 with the use of human feedback; they were able to enhance the model's performance on the sentiment analysis job, obtaining an accuracy of 81.9%, outperforming numerous cutting-edge sentiment analysis models. Overall, the article emphasizes the potential of modifying pre-trained language models like GPT-2 for applications, like sentiment analysis, using human feedback [16].
Recent research has demonstrated that GPT-2 produces excellent text in a range of fields, including news. In one of these studies, it showed that GPT-2 can produce news articles with a high level of fluency and coherence. Based on this work, we generate news-like articles for our analysis using GPT-2 [17].
GPT-2 makes it possible to generate text intuitively by leveraging the previous context to estimate a probability distribution over the vocabulary of the model, which can then be simply decoded by selecting the following word from it. Despite the new architecture, these language models profit by training over bigger datasets. As can perform well across multiple domains and datasets when it is trained on a sufficiently large and diversified dataset [18].
BART is a denoising autoencoder, which means it is built as a sequence-to-sequence model with a bidirectional encoder following BERT [19] and a left-to-right autoregressive decoder similar to GPT [20].
This study, which focuses on MOOCs, suggests a methodology for automatically examining the perspectives stated by students in reviews. The methodology uses aspect-level sentiment analysis to determine the polarity of attitude towards several MOOC-related characteristics. They used a substantial real-world education dataset from Coursera as well as a dataset from a conventional classroom is used to assess the system. The experimental findings show how well the framework performs in classifying aspect sentiment and identifying aspect categories. It performs better than sentiment analysis methods that use manually labeled data [21].
This study investigates the performance of sentiment analysis classifiers concerning educational opinions in an Intelligent Learning Environment (ILE) called ILE-Java. The study focuses on three techniques: machine learning, deep learning, and an evolutionary tactic called EvoMSA. eduSERE, a corpus containing emotion labels specific to learning (engaged, excited, bored, and frustrated), and Senti-TEXT, a corpus with polarity labels (positive and negative). The EvoMSA algorithm had an accuracy rate of 93% for the Senti-TEXT corpus and 84% for the eduSERE corpus, according to trial results. This study demonstrates the efficacy of the EvoMSA technique for sentiment analysis in educational contexts in an Intelligent Learning Environment [22].
This study investigates the use of fine-tuned language models to apply reward learning to problems involving natural language. On the CNN/Daily Mail datasets, the authors use generative pretraining of language models and reward learning to tackle four distinct objectives. With only a few human evaluations, the results demonstrate encouraging performance in stylistic continuation. For summarization, the models earn positive feedback from human labelers and reasonable ROUGE scores. Overall, the study emphasizes the potential of reward learning and tailored language models for language-based reinforcement learning tasks that are used in the real world [23].
The general pretraining framework presented in this study sees all-natural language processing (NLP) task as generation task. The authors contend that a uniform pretraining strategy can be created to handle multiple NLP tasks by defining various NLP jobs as text generation problems. Generation-Pretraining-Finetuning (GPT), a framework they suggest, was motivated by the popularity of language models like GPT-3. A language model is pre-trained on a sizable corpus using the GPT framework, and it is subsequently adjusted for downstream tasks. The study offers in-depth analyses of the proposed framework, its application, and experimental findings on a few NLP tasks, highlighting the efficiency and adaptability of the strategy [24].
With a focus on its formulation, techniques, and evaluation, this research paper offers a thorough review of neural language production. The authors talk about different neural language generation models, such as recurrent neural networks (RNNs), transformer models, and variational autoencoders (VAEs). They investigate several strategies, including autoregressive models, sequence-to-sequence models, and methods based on reinforcement learning, for producing natural language text. Techniques for evaluation are also explored, including human evaluation, artificial measures like BLEU and ROUGE, and intrinsic metrics [25].
In the framework of deep learning, this research paper offers an extensive assessment of text generation algorithms. The authors discuss numerous deep-learning methodologies and techniques that are used for text-generating problems. They talk about how text generation models have changed over time, encompassing both conventional techniques and more recent deep-learning innovations. This study investigates many models, including language models, generative adversarial networks (GANs), and variational auto-encoders (VAEs). The use of text generation in narrative creation, dialogue systems, and machine translation are also covered. The paper discusses difficulties, restrictions, and potential future approaches in the area of deep learning-based text production. Overall, it provides a useful overview of text generation models and their uses for them within the deep learning framework [26].
The authors of this study investigate how to comprehend the knowledge that language models have learned. They suggest an approach for learning about the knowledge representation and reasoning capacities of language models that involve probing tasks, rule-based analyses, and human evaluations. To test the models' understanding of various linguistic phenomena, tests were done on a variety of pre-trained models, including BERT and GPT-2. The results give us important new insights into the knowledge of language models and help us understand their strengths and weaknesses. In the paper, the difficulties of evaluating language model knowledge are also covered. In general, this research provides a framework for investigating the knowledge and internal representations of language models [27].
The author of this paper discusses the necessity for transparency and clarity in providing BLEU scores, which are frequently employed for assessing machine translation systems. The study covers variables that can affect the scores and emphasizes the drawbacks of depending entirely on BLEU results. The author offers recommendations for BLEU score reporting, highlighting the significance of including details on tokenization, reference translations, and segment-level and corpus-level scores. To ensure accurate and insightful comparisons of machine translation systems, the article promotes transparent and standardized reporting practices. In conclusion, it highlights the need for better BLEU reporting and interpretation in the context of machine translation evaluation [28].
This study presents a thorough analysis of natural language processing (NLP) models that have already been trained. The writers talk about how pre-training methods have changed, advanced, and taken many forms in NLP. They investigate several pre-trained models, such as transformer-based models, contextualized word representations, and word embeddings. The paper discusses the advantages and difficulties of pre-trained models, including computing needs and transfer learning. It also discusses the structures and applications of well-known models like BERT, GPT, and ELMo. Overall, this survey provides insightful information about the landscape of NLP's pre-trained models and how they affect various NLP activities [29].
The Generative Pre-Trained Transformer (GPT) technique is recommended by the authors of this study as a way to enhance language understanding. They train the GPT model on a big corpus of unlabeled text data using word prediction. The model learns the basic linguistic structures and patterns during pre-training. For usage in further tasks like text categorization and question answering, the model is then improved. Experimental results show that GPT works better in terms of performance and generalization than other pre-training techniques. The work highlights GPT's ability to enhance language understanding in a variety of NLP tasks and details its design and training procedure [30].