Innovative Deep Learning-Based CEA-MMSA Framework for Cultural Emotion Analysis of Tamil and Sanskrit Siddha Palm Leaf Manuscripts

doi:10.21203/rs.3.rs-4847877/v1

Download PDF

Research Article

Innovative Deep Learning-Based CEA-MMSA Framework for Cultural Emotion Analysis of Tamil and Sanskrit Siddha Palm Leaf Manuscripts

https://doi.org/10.21203/rs.3.rs-4847877/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Tamil palm leaf manuscripts serve as invaluable cultural heritage repositories, housing a wealth of ancient wisdom spanning medical prescriptions and spiritual hymns. However, their profound significance is matched by the complexity of deciphering the sentiments they convey, attributed to their multimodal (text and visual content) and multilingual (Tamil and Sanskrit) nature. This study presents a Deep Learning-Based Cultural Emotion Analyzer (CEA-MMSA) designed for the multimodal and multilingual sentiment analysis of Tamil and Sanskrit Siddha palm leaf manuscripts. These manuscripts are invaluable cultural artifacts, containing ancient wisdom in complex textual and visual formats. Our innovative approach leverages Vision Transformers (ViTs) for visual sentiment analysis and Gated Recurrent Units (GRUs) with attention mechanisms for textual sentiment analysis, facilitating a nuanced understanding of emotional content. The proposed multimodal fusion model enhances data interpretation by integrating textual and visual sentiments, addressing the intricacies of the manuscripts' linguistic aspects. Empirical results demonstrate the efficacy of our methodology, achieving an accuracy of 97.38%, with precision at 96.87%, recall at 95.34%, and an F1 score of 95.37%. This advancement not only enriches the study and preservation of these manuscripts but also illuminates the emotional and cultural narratives encapsulated within them.

Siddha palm leaf manuscripts

multimodal sentiment analysis

multilingual sentiment analysis

deep learning

Vision Transformers

GRU

attention mechanisms

transfer learning

cultural heritage

sentiment classification

Tamil has a rich literary tradition and is considered one of the world's oldest languages. Ancient poets, especially those from the Indian state of Tamil Nadu, would encase their poems in palm leaves to keep their meaning secret. The invention of paper did not preclude using palm leaves as a fundamental and early written material. Many researchers have taken an interest in restoring the ancient Tamil medical literature throughout the last decade. The historical literature incorporates several palm leaf works, such as Sangam literature, medical texts, astrology, jewels, music, dance and drama, Siddha, Saiva, Vaishnava, culinary, and Vaastu. But learning the old Tamil and Sanskrit script is no easy feat. The palm leaf writings include Tamil devotional melodies, information on astrology, and herbal remedies. Many scholars have tried various techniques, smearing the Tamil and Sanskrit palm leaf texts with turmeric, to decipher them, but none of these approaches have succeeded. Since these texts contain information about traditional medicine, it is believed that deciphering them by digital document reading would benefit society. It is challenging for modern Tamil and Sanskrit speakers who focus solely on learning new vocabulary and grammar to recognize the ancient characters used in the language. It is laborious and challenging to decipher the old Tamil and Sanskrit words from palm leaves and comprehend their contexts. Still, an improved recognition system is required to increase readability and safeguard written medical practices and traditions. In the medical field, in particular, it is essential that practitioners fully grasp the ideas of a text to implement them consistently. Thus, a recognition system is highly beneficial for comprehending, interpreting, and implementing the methods outlined in the manuscript. Written forms, most commonly palm leaf manuscripts, have allowed this knowledge to be transmitted from generation to generation. The multimodal (text and visual) goal of learning from different kinds of data has recently attracted much interest. The wide-ranging potential uses of multimodal sentiment analysis have piqued its interest. Current approaches only consider one modality, limiting their ability to gather content across all available channels.

Furthermore, most research on multimodal learning has ignored the intricate relationships between the two modalities instead of focusing on their simple combination. More clarification is required, however, regarding the analysis of multimodal activities in circumstances involving many languages. This research lays forth a simple method for improving current datasets available to the public to perform Multimodal Multilingual Sentiment Analysis (MMSA) on Siddha Palm Leaf Manuscripts written in Tamil and Sanskrit. Therefore, more people will be able to understand both texts better thanks to digitization. There is a lack of materials for Tamil and Sanskrit-recognized characters due to the high linguistic understanding, technological brilliance, and time required to manually identify and map each palm leaf text with the current Tamil character. Researchers have utilized data augmentation techniques and self-synthesized datasets to address this issue. For texts written on parchment, bark, metal plates, or palm leaves, unique pre-processing methods are required to restore the scripts that have deteriorated over the years and, in these languages, compiled existing datasets where every annotated post has a picture and some text with a favourable, harmful, or neutral categorization. A multimodal model integrating information from picture and text embedding was constructed to identify the target labels further. This study assesses the training dataset by separating palm-leaf manuscripts written in Tamil and Sanskrit.

The framework thoroughly comprehends multimodal content by integrating Vision Transformers (ViTs) for visual sentiment analysis and Gated Recurrent Units (GRUs) with cognitive mechanisms for text sentiment evaluation in palm-leaf manuscripts. When presented with an image, visual sentiment evaluation attempts to identify the underlying emotional tone, whether good, harmful, or neutral. Applications of Transformer-based models to image data are still in their early stages but show promise. Vision Transformer (ViT) achieves outstanding results on large-scale vision benchmarks using Tamil and Sanskrit palm leaf manuscripts. Beyond exploring ViT's suitability for visual sentiment evaluation, it takes a step further by integrating idea orientation into the Transformer's central self-attention mechanism.

Additionally, a novel multimodal combination model combines textual and visual sentiment data to enhance the depth of analysis. In RNN training, Gated Recurrent Units (GRU) are frequently employed to address issues like gradient vanishing and explosion. The principle of attention is among the fundamental approaches extensively used in text categorization and sentiment analysis in the last several years, alongside the already indicated models. Adding the attention mechanism makes the model more versatile in focusing on information at different points in the input sequence, which improves its ability to process significant texts. Text sentiment analysis and categorization are areas where deep learning algorithms excel, and this research incorporates a novel approach using the GRUs model to improve upon previous work in this area. Additional creative ways are expected to emerge due to technology's continual development and advancement. It will open up further options for growth and space for this subject.

The main contributions of the article include

This paper introduces the Cultural Emotion Analyser for Multimodal and Multilingual Sentiment Analysis (CEA-MMSA) in Tamil and Sanskrit Siddha Palm Leaf Manuscripts, which uses deep learning algorithms for textual and visual sentiment analysis.
The approach uses Vision Transformers (ViTs) for visual sentiment analysis and Gated Recurrent Units (GRUs) with attention mechanisms for textual sentiment analysis to interpret emotional content.
A novel multimodal fusion model enhances analysis by combining textual and visual sentiment data. The suggested method illuminates these manuscripts' cultural legacy by revealing subtle emotional and linguistic patterns.
The accuracy, precision, recall F1 score and confusion matrix were used to evaluate the suggested system.

The study's overview is given in this section. Section 2 examines present research approaches and extensively reviews the literature. The study technique, strategy, and processing procedures for Siddha palm leaf manuscripts are described in Section 3. The results and analysis are presented in Section 4. Section 5 discusses the key takeaways and plans for moving forward.

Cheng et al. [16] present the AT-MC-BiGRU (Multichannel convolution and bidirectional GRU)-Capsule. This multiple-head attention capsule models text sentiments using vector neurons instead of scalar neurons and employs capsules to characterize those emotions. Conventional methods also struggle to extract multi-level text sequence features. Combining multi-head attention with CNNs and Bi-GRUs, which may express word dependence and sentiment words in the text, and adding a global average pooling layer gives us a more complete multi-level feature representation of the text sequence. This work overcomes basic algorithms utilizing experimental information from the large corpus of sentiment classification, three English and one Chinese dataset.

Subramani et al. [17] aim to create a massive database of Tamil characters by extracting medically related old palm leaf manuscripts. In this research, the developed characters teach expert systems to decipher the selected medical papers based on their meaning and context. There needs to be a better recognition method to make the old Tamil letters legible on palm leaves and understand their contexts; otherwise, understanding these recorded medical customs and procedures will be daunting and time-consuming. The contents of a book must be understood by practitioners so that they can apply them daily, especially in the medical field. Giant characters have been manually detected, and datasets are constructed using Gaussian distortion.

In the research [18], the characters' traits from palm leaves are trained using a novel Convolutional Neural Network (CNN) method. This training allows CNN to perform exceptionally well when classifying palm leaf characters during instruction. It all starts with geometrical operations, a pre-processing approach that removes distortion in the input image. This research uses feature interpreting, which entails three primary phases: textual spacing among lines, barrier-free spacing, and obstacle-filled spacing. The last step in the categorization process is to feed the retrieved cursive characters into the CNN technique. The suggested CNN performs, compared to current deep learning methods, conducts tests using a collection of handwritten Tamil palm leaf manuscripts. While the current ResNet only managed 88% accuracy, the suggested network managed 94%.

Sabeenian et al. [19] present a method for character identification using convolutional neural networks, emphasizing Tamil palm-leaf characters. The CNN presented in this work has five layers: pooling, activation, convolution, fully connected, and soft maximum classifier. The character set database was created using scanned palm-leaf manuscripts. Every one of the database's fifteen categories has about a thousand distinct instances. Results show that the CNN Classifier has a recognition rate of 96.1–100%. Large quanta of characteristics extracted for each CNN layer lead to a greater prediction rate.

Haritha et al. [20] focus on an optical character recognition system using Convolutional Neural Networks (OCR-CNNs) to correctly digitize and identify the characters seen in texts written on palm leaves in Tamil. The numerous layers of a convolutional neural network—the pool, activation, fully connected, and classification layers—are utilized in this article. The character set database is based on scanned palm-leaf manuscripts. About 100 examples in each of the 67 categories make up the database. There was an improvement in the recognition rate of the CNN model. Due to the high amount of features recovered for each CNN layer, the forecasting rate and reliability are exceptional.

Given the necessity of digitalization and transcription, character recognition in palm-leaf manuscripts remains an unfinished business. The characteristics of the characters retrieved from the palm leaf are trained using a recurrent neural network (RNN) proposed by Athisayamani et al. [21]. First, the method's pre-processing step gets rid of noise. Then, the character is extracted from the picture and taught through a BLSTM network. The characters are trained and tested using a feature vector with nine character stroke zones. The characteristics of the characters are trained using a rich range of examples. Outperforming competing artificial neural network-based character detection methods, this one delivers superior recognition accuracy.

Wang et al. [22] propose EERCA-ViT, a visual sentiment classification framework that improves upon previous work using Context-Aware Vision Processors and Effective Regions. This model incorporates both a main branch and an auxiliary branch. Extensive experiments performed on six benchmark datasets show that the proposed approach outperforms the latest methods for image sentiment analysis. Furthermore, comparative trials have proven that the various framework components (P-SE and CAM) perform as advertised.

Alzamzami et al. [23] suggested transformer architecture for massive data from online social networks (TAOSNs); the technique can gather emotion and emotion characteristics from three different image types: those with text, those without text, and photos featuring faces. Three well-known datasets—the Domain Free Multimedia Sentiment Dataset (DFMSD), the newly suggested dataset, and another—are used for training. According to the data, learning with the threshold-moving approach improves sentiment learning ability by 5–8% compared to training without it. The learning performance improved by including the threshold-moving method in the two-stage approach during training. Along with improved accuracy and F-score, the suggested method has a beneficial effect on learning when it comes to merging the three models.

The proposed system, Cultural Emotion Analyzer for Multimodal and Multilingual Sentiment Analysis (CEA-MMSA), leverages deep learning algorithms to interpret and analyze the sentiments encoded in Tamil and Sanskrit Siddha palm leaf manuscripts. The CEA-MMSA framework integrates Vision Transformers (ViTs) for visual sentiment analysis and Gated Recurrent Units (GRUs) with attention mechanisms for textual sentiment analysis. Initially, images of palm leaf manuscripts are collected and pre-processed to remove noise, enhance text clarity, and prepare them for sentiment extraction. The textual data undergoes sentiment analysis using GRUs, where the attention mechanism improves the model’s ability to focus on significant textual elements. Simultaneously, visual data is processed through ViTs, which capture the emotional tone embedded in the images. The multimodal fusion model then combines the textual and visual sentiment data to enhance the analysis's depth and accuracy. This approach allows for a nuanced understanding of the emotional and cultural narratives within the manuscripts, achieving high performance metrics and contributing to the preservation and interpretation of this invaluable cultural heritage.

3.1 Structure of CEA-MMSA

The objective characteristics are documented by collecting images of palm leaf manuscripts from Tamil and Sanskrit Siddha and storing them in a database. Multiple pictures of palm leaf manuscripts written in script are compiled. The collected palm leaf manuscripts of the Tamil and Sanskrit Siddha sentiment analysis are shown in Fig. 1. The following section presents the various ways to enrich and experiment with the data collection. Data preparation, datasets, and all the methodologies used to develop the final model are detailed in this section of Cultural Emotion Analyser for Multimodal and Multilingual Sentiment Analysis in Tamil and Sanskrit Siddha Palm Leaf Manuscripts.

The multimodal data is in the foreground against a dark, leaf-coloured backdrop in palm-leaf manuscripts. Blurring or other morphology techniques can lessen noise detection in digital camera scans or photos of palm-leaf manuscripts. In the proposed model, the multimodal data includes text, emotions, and visual sentiments of Tamil and Sanskrit Siddha medicine palm scripts. Text and Visual sentiment data merge raw data from two languages to move on to the next step, and text with emotions is a fantastic tool for supplying information and expressing ideas and feelings. Efficient noise removal is essential in extracting valuable data from textual images. So that the characters stand out and are straightforward to manipulate, the pre-processing involves turning the backdrop to black and the foreground text to white, narrowing the colour space from 0 to 255 to just 0 and 1. Images can be prepared for text line and visual extraction by performing morphological procedures and removing the background. The structural processes take the input images of the leaves and apply a structure component to them, such that the final images are the same size.

Consider a palm leaf manuscript with a black backdrop and white text, where 0 represents the background colour, and 1 represents the text in the forefront. In this case, T displays the 52-point global threshold. After eliminating the background for the input image, the processed instance output image is considered the pre-processed image. Here is the representation of Eq. 1 for the pre-processing image:

$$\:p\left(x,y\right)=\begin{array}{c}1\:\:\:\:\:\:if\:f\left(x,y\right)\ge\:t\\\:0\end{array}$$

In this case, x and y represent the two image inputs from the multiple modalities, while p represents the matching image output. After pre-processing, the data is given to the GRU for text sentimental analysis, which thoroughly comprehends emotional information and ViTs for vision sentimental analysis. Two popular models utilized for text sentiment analysis and classification tasks are Recurrent Neural Networks (RNNs) and GRU, which have many uses in palm leaf manuscripts. RNN is better at capturing context knowledge and is more suited for analyzing data in series. Still, CNN is better at capturing local features when processing text and extracting multiple levels of semantic data through multilayer convolutional processes. Architectures like GRU are commonly utilized to address issues like vanishing gradients and acceleration of gradients during RNN training.

Recently, text sentiment analysis in palm leaf manuscripts has extensively used the attention mechanism, among other models, to learn about traditional Siddha procedures and classification problems. An attention mechanism allows the model better to comprehend blurred texts from multimodal data in many languages. It makes it more adaptable to focus on information at different points in the input sequence. The work introduces a novel approach to text sentiment analysis and classification using a bilateral GRUs model with an attention mechanism, demonstrating the superior performance of deep learning algorithms on this problem. Simultaneously, the image data used in the ViT transformer often convey emotions better than words alone. Results from task-based image classification utilizing ViTs have been demonstrated to be more robust than those obtained using CNNs. Obtaining such outstanding performance stability is critically dependent on the attention process. Evidence suggests that ViT outperforms CNN in picture categorization learning thanks to its multi-head self-attention layer, which allows it to incorporate data about global features into the complete picture. This study used the transformer architecture to train three models. One model was used for each image type to obtain task-specific, high-quality data. After that, the multimodal visual sentiment classifier will be modelled utilizing these features.

Additionally, the information is given to the multimodal fusion model for data fusion. It can occur at various points in the model learning process using intermediary fusion since it combines information at varying depths. When data from many sources, such as Tamil and Sanskrit Siddha medicine manuscripts, are combined, a process known as multimodal data fusion is employed. This results in information that is easier to interpret and use. With deep learning, multimodal data fusion has significantly enhanced learning performance. The intermediary fusion takes the higher-level representations learned by deep learning layers as input. Machines can learn a picture from each separate model through multimodal deep learning's intermediate fusion, simultaneously merging many model presentations into a hidden layer. Next, the multilingual and multimodal data will be sorted using a deep learning-based classification layer according to the text's emotional tone and the visual image's clarity in identifying each letter in the Siddha palm manuscript.

3.2 GRU with attention mechanism for text Sentimental Analysis

GRU can handle time series, text, and speech, representing sequential data. By including an attention mechanism, the transformer transformation model can enhance its ability to express features and acquire more layers of information in phrases from various places. This study takes a deep learning approach, drawing on the emotional model of literature, and suggests an attention model that integrates GRU to address the issue of textual sentiment evaluation. This model, shown in Fig. 2, builds sentiment evaluation for each emotion category by combining the attention mechanism and using text sentiment information from vector neurons. The representation of features improves both the generalizability and robustness of models. This approach outperforms alternatives that need linguistic and emotional data regarding classification accuracy and brevity.

The core principle of GRU is that at each time step, the network's hidden state can be selectively updated via gating methods. Information can be controlled inside and outside of the network using gating techniques. One popular model in neural network research for processing sequence and time series data is the gated recurrent unit model, which is incorporated into the suggested model. Traditional machine learning methods rely on restricted prefix lexicon knowledge as the dependent elements of the semantic model; however, RNN can include the entire preamble vocabulary into its linguistic knowledge set, resulting in superior performance. However, vanishing or inflating gradients is a problem with traditional RNNs. Attention and GRU circumvent these problems by allowing input to selectively alter the model's state at any given point through the structure of specific controls. GRU uses update gates instead of an input gate and a forget gate, making it a version of LSTM. Equations (2) through (5) display the GRU structure description and the applicable calculation formulas.

$$\:{G}_{t}=\sigma\:({w}^{G}{x}_{t}+{u}^{G}{h}_{t-1})$$

$$\:{R}_{t}=\sigma\:({w}^{R}{x}_{t}+{u}^{R}{h}_{t-1})$$

$$\:{h}_{t}=\left(1-{G}_{t}\right)\times\:\widehat{{h}_{t}}+{G}_{t}\times\:{h}_{t-1}$$

$$\:\widehat{{h}_{t}}=\text{t}\text{a}\text{n}\text{h}({w}^{h}{x}_{t}+{u}^{h}{h}_{t-1}\times\:{h}_{t-1})$$

In this case, $\:w$ and $\:u$ denote the GRU weighted matrices, $\:\sigma\:$ stands for the analytical logistic operation, and $\:\times\:$ indicates the element amplification. Concealed by the present input condition and the prior layer is $\:{G}_{t}$, the gate that controls the amount of updates for the activation value of the GRU unit. The hidden layer $\:{h}_{t}$ and the candidate concealed a layer $\:\widehat{{h}_{t}}$ are both involved in determining the layers' states; the reset gate $\:{R}_{t}$ blends the updated information given alongside the prior understanding.

By simplifying the model and reducing the number of variables, the GRU helps to keep trial expenditures to a minimum. State propagation in a traditional recurrent neural network only goes in one direction, from the front to the back. There are issues, though, when the present output depends on the prior and subsequent states. The bilateral recurrent neural network emerged to address the challenge of missed word prediction, which involves both the previous assessment and the message of the current one. Two RNNs are fed contrary input directions simultaneously, and their output is jointly computed, resulting in a more precise outcome. Making a bilateral GRU is the same as making an RNN in a bilateral recurrent neural network. Y is the multi-head attention output matrix the model in this research learns from using a Bilateral GRU network. During training, the network produces the $\:{h}_{t}$ with the use of two GRUs that stand in for feelings along the text series' forward and backward paths at the same time. GRU unit vector dimension Eq. 6 illustrates the precise method of calculation.

$$\:\widehat{{h}_{t}}=GRU({x}_{t},{h}_{t-1})$$

With the help of the attention mechanism, GRU can spotlight the text's most crucial details. The significant data of the text sequence is captured across several subspaces using multi-head attention in this article. Eq. 7 gives the dimensional vector d for the supplied text line L=(w₁,w₂…w_n) of length n, where w_i is the i-th word in sentence L.

$$\:L\in\:{R}^{n\times\:d}$$

The first step is to transfer various subspaces to the vector words matrix L, which has been linearly processed and divided into three identical matrices such as query($\:{q}_{i}$), key($\:{k}^{t}$) and value of matrices with the subspaces. Finally, in simultaneously determining the attention value of every subspace is given in Eq. 8

$$\:head=softmax\left(\frac{{q}_{i}{k}^{t}}{\sqrt{a}}\right){V}_{i}$$

The head represents the value of attention in the i-space, $\:\sqrt{a}$ ensures that the colour gradient remains visible during the backpropagation process by transforming its attention matrix into a conventional distribution of values. Next, considering the line spacing, there are two ways to classify an obstruction in the white space among the lines of text: contacting and overlaying. An impediment has a significant role in shaping a character's personality in Tamil literature. "Touching text lines" refers to the extent of an obstruction that stretches and approaches the following text line. The characters in the first and second lines are striking. Tamil and Sanskrit text line splitting is intricate because while ignoring an obstacle causes the character "யு" to transform into "య," cutting an obstruction at a fixed length causes the second line symbol "ఞ/tha/" to become "ఞి/thi/." As with contacting textual lines, the same inaccurate forecast problem occurs when utilizing pre-existing methods for text line separation with overlapping text lines. The proposed GRU ensures accurate character projections by repositioning the hurdle's separating line, eliminating touching and overlaying lines of content. Segmenting the text lines is an essential first step before character segmentation. Line separation in text gets more complicated and ineffective as lines of writing touch and overlap. Text line separation becomes more complex when lines of text intersect. An impediment invades the text zone of following lines, getting in the way of the character's strokes, which could lead to incorrect character interpretation or an unexpected character.

3.3 Vision Transformers (ViTs) for visual sentiment analysis

The pre-processing phase is critical to the learning procedure. It entailed cleaning, denoising, and preparing the data before feeding it into the VIT for sentimental analysis to train on multilingual palm leaf manuscripts written in Tamil and Sanskrit. The work employs the 2020-introduced visual Transformer (ViT) as its deep learning framework for sentiment analysis in multimodal visual analysis. A visual categorization approach called the Vision Transformer applies a Transformer-like architecture to some areas of an image. This incorporates various architectural elements in the Transformer design commonly employed in natural language processing, such as Multi-Head Attention and Expanded Dot-Product Attention. An alternative to CNNs for image identification tasks is ViT. Both computationally and in terms of accuracy, it quadruples the performance of state-of-the-art CNNs, particularly on massive data regimes. Because ViT can learn its own biases, CNNs' inductive biases are superfluous in enormous data regimes. The ViT model can learn when and how to attend without prejudice due to the shorter layers' ability to process local and global information in images. ViTs are valuable for interpreting the intricate patterns found in palm leaf manuscripts because they can successfully record the links among faraway regions of an image. It is easier for ViTs to deal with images of varying sizes than with CNNs, which typically need fixed-size data. Scaling up ViTs with more significant records and greater computing power could improve their efficiency on challenging tasks.

Figure 3 provides a high-level picture of the ViT design. The palm leaves manuscripts for the Tamil and Sanskrit languages are divided into sections, subsequently turned over and smoothed out. Images with only two dimensions are fed into the Transformer model. The input image is divided into shorter multifaceted patches according to its length (h), width (w), and number of channels (N). It is done to format the input information in a way that is similar to how it is organized in the natural language processing domain, where each word is represented by its sequencing. The results based on the number of patches and pixels (p) are given in Eq. 9

$$\:n=\frac{hw}{{p}^{2}}$$

The compressed patches undergo linear projection and directional encoding before feeding the data into the multilayer transformer encoders. The building blocks of an encoder are a multiple-layer perceptron module and a multihead attention module. An MLP with one hidden layer during initial training and a single linear layer during fine-tuning implements the classification head. As its central component, a classification head is an MLP that encodes positions. A series of integrated pictures enhanced with positional data is fed into the encoder as input. A state-dependent categorization output is produced by a classification head connected to the encoder's output and takes the value of the accessible class embedding as input. During the refinement process, an individual feedforward layer is used within the framework of the MLP, where size is the number of classes or responses relevant to the job. Throughout the training, the input images are consistently chopped into a single patch size; however, higher-resolution images are utilized for modification. The outcome is a more extended input sequence for fine-tuning compared to pre-training. If the input sequence is extended, fine-tuning will necessitate more positional embedding than pre-training. A multiple-layer perceptron head, functioning as the categorizing module, produces class predictions by receiving the encoder's information.

3.3 Multimodal Fusion model

Architecture for intermediary fusion that combines output from GRU with attention and ViT models based on deep learning to construct a multimodal and multilingual Siddha palm leaf text for in-depth analysis. Creating a dynamic text and picture classifier that works across several modalities is depicted in Fig. 4. Because diverse perceptions complement each other, multimodal fusion improves emotion recognition skills. The benefits of deep neural networks are better utilized in model-level fusion than in determination-level and feature-level fusion. Here, the Transformer model combines modalities at the model level specific to different languages. After encoding the text and visual modalities of the Siddha palm leaf manuscript, the multi-head attention uses a common semantic feature space to generate multimodal emotional intermediate representations. Data fusion can occur at various points in the model learning process using intermediate fusion since it combines information at varying depths. With deep learning, multimodal data fusion has significantly enhanced learning performance. Middle fusion takes the features, or higher-level representations, learned by various deep learning layers as input. Intermediary fusion in multimodal deep learning is simultaneously merging multiple model presentations into a hidden layer. This allows the algorithm to learn a demonstration from each of the many models. It is the fusion layer's job actually to fuse the layers. This research recommends a ViT-based fusion for multimodal, aesthetic, and affective analysis of Siddha meds found in palm manuscripts. We train three models—one for single-modality sentiment, one for facial emotion, and one for textual sentiment—using Transformer backbones, specifically ViT, for visual content and GRU with an attention mechanism for textual information. After that, each model is placed into an MLP classification head using a single combined feature extracted by each model. The improved analysis of Siddha medication from palm scripts, which might be helpful for letter identification, is the outcome of the models.

Vision Transformers offers a practical and versatile method for visual sentiment analysis of multilingual palm leaf manuscripts. ViTs can decipher these significant materials' emotional and cultural significance using the capabilities of transformer designs despite their inherent complexity and peculiarities. Both sentiment analysis and the study and protection of historical artefacts benefit from this application's use.

3.4 CEA-MMSA Multimodal Fusion Algorithm

// Algorithm 1: CEA-MMSA

// Step 1: Data Collection and Storage

1.1 Collect palm leaf manuscript images in Tamil and Sanskrit.

1.2 Store collected images in a database.

// Step 2: Preprocessing

FOR each image I IN database:

2.1 Convert I to grayscale.

2.2 Apply morphological techniques to reduce noise.

2.3 Invert colors:

FOR each pixel (x, y) IN I:

IF pixel value f(x, y) > = threshold T:

SET pixel value p(x, y) = 1

ELSE:

SET pixel value p(x, y) = 0

2.4 Ensure final images are the same size.

// Step 3: Text Sentiment Analysis Using GRU with Attention Mechanism

INITIALIZE GRU model parameters.

FOR each textual input sequence S:

3.1 Encode S using embedding layers.

3.2 Pass encoded text through GRU layers.

3.3 Apply attention mechanism to focus on important parts of S.

// GRU Equations

COMPUTE update gate G_t:

G_t = sigmoid(W^G * x_t + U^G * h_(t-1))

COMPUTE reset gate R_t:

R_t = sigmoid(W^R * x_t + U^R * h_(t-1))

COMPUTE candidate hidden state h_hat_t:

h_hat_t = tanh(W^h * x_t + U^h * (R_t ⊙ h_(t-1)))

UPDATE hidden state h_t:

h_t = (1 - G_t) ⊙ h_(t-1) + G_t ⊙ h_hat_t

// Step 4: Visual Sentiment Analysis Using Vision Transformers (ViTs)

INITIALIZE ViT model parameters.

FOR each image I:

4.1 Divide I into patches.

4.2 Flatten and linearly project patches.

4.3 Add positional embeddings.

4.4 Pass patch sequence through transformer encoder layers.

4.5 Use multi-head attention to capture global features.

// ViT Patch Processing

COMPUTE number of patches n:

n = (h * w) / (p^2)

APPLY linear projection and positional encoding to each patch.

// Step 5: Multimodal Fusion and Classification

5.1 Extract features from both GRU and ViT models.

5.2 Fuse features at intermediate layers using a common semantic space.

5.3 Combine textual and visual features using multi-head attention.

5.4 Pass combined features through MLP classification head.

// Step 6: Classification

FOR each combined feature vector:

6.1 Apply MLP classification layers.

6.2 Output sentiment classification.

// Step 7: Evaluation

7.1 Calculate accuracy, precision, recall, and F1 score on the validation set.

7.2 Compare against baseline models.

END

CEA-MMSA involves collecting and storing images of Tamil and Sanskrit palm leaf manuscripts, preprocessing them to reduce noise and enhance readability, and then using GRU with an attention mechanism for text sentiment analysis and Vision Transformers (ViTs) for visual sentiment analysis. The model fuses textual and visual features through intermediate layers, combining them with multi-head attention, and classifies the sentiment using a multi-layer perceptron (MLP). The system is evaluated for accuracy, precision, recall, and F1 score, demonstrating its effectiveness in analyzing and classifying sentiments from multimodal and multilingual data.

The proposed model utilizes the data set taken from the Tamil and Sanskrit palm leaf manuscripts. Tamil manuscript is taken from https://eap.bl.uk/archive-file/EAP810-1-1-1. Akattiyar, also known as Kumpamuṇi, penned three manuscripts that make up the book. It is a first-rate manuscript. Author Kumpamuṇi (Akattiyar) wrote the text Mati Veṇpa. The book has one hundred poems spread over 18 palm leaves, numbered 1–18. Aside from discussing spirituality and philosophy, the text briefly references a few medical conditions with medical solutions. 2-The title of the text is Akattiyar Taṇṭakam-100. It is unfinished and made with palm leaves numbered 1–8. The book delves into alchemy and magic, revealing the eight types of magical deeds (aṭṭakarmam) and the philosophical paths of yōka and ñāṉam. 3: A Glossary of Terms from the Siddha Tradition (Kummpamuṇi Paripācai Tiraṭṭu). The palm leaves are arranged in five sections, with numbers ranging from one to eleven. 3 − 1.

The Sanskrit is taken from https://factumfoundation.org/our-projects/digitisation/archiox-analysing-and-recording-cultural-heritage-in-oxford/sanskrit-palm-leaf-manuscripts/.The photometric stereo technology digitized some rare palm leaf Sanskrit writings in the early months of ARCHiOx. Accessing and studying fragile things is difficult due to their delicate, nearly illegible marks. ARCHiOx has created a high-res digital edition for virtual analysis without damaging the manuscripts. The proposed model is compared with the OCR-CNNs, TAOSNs, EERCA-ViT and AT-MC-BiGRU using the evaluation metrics Accuracy, Precision, Recall and F1-score.

Evaluation Metrics

As a last step in the process, review the sentiment analysis of the proposed results to ensure the strategy was accurate and working. Examining the ground truth sentiment results will demonstrate how accurately the multimodal analysis of sentiment is performed. The performance of the confusion matrix was evaluated in terms of True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). Typically, these methods are used to determine if the text and visual classification of the Siddha palm leaf manuscript are viable. The amount of correctly identified positive occurrences, correctly recognized negative instances, erroneously categorized positive cases, and wrongly classed negative instances are measured by the variables TP, TN, FP, and FN, respectively. After that, it checks the palm leaf script for positive, negative, or neutral thoughts. True positives (TPs) occur when the original textual sentiment is favourable and the visual sentiment is also positive. A false negative (FN) happens when the visual sentiment is negative even though the initial sentiment was positive. An FP occurs when the visual sentiment is neutral, but the actual sentiment is positive. A genuine negative (TN) occurs when there is a negative connotation in both the text and the visuals. A false positive (FP) occurs when the visual sentiment is favourable while the written sentiment is negative. A false negative (FN) occurs when the text has a negative tone but no discernible tone in the screenshot. A TP occurs when there is no appreciable difference between the initial and picture emotions. In the case of a false positive (FP), the initial emotion was neutral, but the visual sentiment was positive. True negatives (FNs) occur when the initial sentiment is neutral, but the image's sentiment is negative.Therefore, the parameters are evaluated independently for each given class. Recall(R) and Precision (P) measure how well a sentimental analysis is performed relative to all positive examples in the dataset. Recall counts how many instances were adequately identified, while precision counts how many were correctly classified relative to all positive instances. The F1 score is a harmonic average of the recall and precision scores. Accuracy (A) is the percentage of occurrences in a dataset classified correctly relative to all instances. Contrarily, specificity evaluates the fraction of negative cases correctly identified out of all the negative instances in the collection. The evaluation metrics are calculated using these equations:

$$\:Accuracy=\frac{TP+TN}{TP+FN+FP+TN}$$

$$\:Precision=\frac{TP}{TP+FP}$$

$$\:Recall=\frac{TP}{TP+FN}$$

$$\:F1-Score=2\times\:\frac{P\times\:R}{P+R}$$

With three categories of sentiment—positive, negative, and neutral—here is the sentiment analysis confusion matrix is shown in Fig. 5. In an image-based representation of the matrix, the degree of intensity of each color indicates the number of occurrences in that cell.

Figure 6 shows the graphical representation of accuracy was measured using element combinations of Tamil and Sanskrit datasets to evaluate the role of the three technical components—MMSA of palm leaf manuscript structure, emotion, and interaction—in word embedding and visual analysis based on the cultural emotional analyzer. The model, CEA-MMSA with bidirectional GRUs (biGRUs), does not outperform OCR-CNNs, TAOSNs, EERCA-ViT, and AT-MC-BiGRU structure in terms of accuracy. Beyond that, the data clearly shows that position information is becoming more crucial for model accuracy since the performance of an MMSA structure with position is significantly better than the standalone GRU model.

Figure 7 shows the graph representation of the proposed precision model for the deep analysis of Siddha medicine. CEA-MMSA has the highest performance, around 97.43%, compared to the traditional methods.

The precision, recall, and F1-score of all the baseline techniques have grown steadily as the amount of data grows, as seen in Figs. 8–9. At the beginning of the data set, CEA-MMSA had a low precision and F1 score. According to the research, at 61% of the data, the recommended model's precision and F1 score have nearly peaked. The model may reap the benefits of positional knowledge and a multilayer sentimental mechanism, or it may be that the data is rich. Rich data can help the model learn more comprehensive semantic features compared to using a small amount of data. Hence, compared to baseline models, sentiment analysis performs better. As soon as the data amount surpasses 65%, the model's performance improves and eventually peaks. Therefore, according to a holistic viewpoint, MMSA outperforms baseline algorithms in sentiment analysis tasks involving palm-leaf manuscripts.

The results show that using higher-dimensional vectors to capture text's emotional elements improves the model's emotional recognition precision. Training with a vector enhances its ability to capture the text's emotive aspects and represent specific text properties. With the help of graphic representations of the palm scripts' word weighting arrangements in phrases, this article helps readers better grasp how multi-headed attention can encode word dependency and catch emotional expressions in text.

Table 1

Performance Comparison of Different state of the art models with proposed Model
Method	Accuracy	Precision	Recall	F1 score
OCR-CNN	85.31	77.12	72.18	74.93
TAOSN	85.89	84.58	79.04	82.36
EERCA-ViT	89.64	92.78	86.41	89.19
AT-MC-BiGRU	92.34	93.43	93.754	94.148
CEA-MMSA	97.38	96.87	95.34	95.37

Table 1 show the comparative analysis of the proposed model. Compared to OCR-CNNs, TAOSNs, EERCA-ViT, and AT-MC-BiGRU, the suggested model stands out in recall, accuracy, precision, and F1-score. The confusion matrix analysis shows that the suggested model performs well on the test set with a recall rate of 95.34%, accuracy rate of 97.38%, precision rate of 96.78%, and F1 score of 95.37%. The ability to handle noise, identify sarcasm, and account for dialect differences are all areas that may use some work. The study suggests that to overcome these challenges and greatly enhance the accuracy of sentiment analysis, more sophisticated computational methods and a larger pool of language resources should be used in future.

Future research directions for the Cultural Emotion Analyzer for Multimodal and Multilingual Sentiment Analysis (CEA-MMSA) could include expanding its application to other historical manuscripts from various ancient languages and cultures, enhancing its capabilities for real-time sentiment analysis, and developing advanced preprocessing techniques to improve the readability of degraded manuscripts. Additionally, integrating transfer learning and domain adaptation can enhance the model's robustness for multilingual and cross-domain data. Further research might also focus on improving emotion recognition and contextual understanding, developing user-friendly interfaces and tools for easier interaction, and implementing collaborative annotation platforms to enrich training data and enhance cultural sensitivity.

This research introduces a groundbreaking method for analyzing the sentiment of Siddha palm leaf manuscripts using deep learning algorithms. The procedure takes into account both textual and visual sentiment. The system blends visual sentiment analysis with Vision Transformers (ViTs) and textual sentiment analysis with Gated Recurrent Units (GRUs) with attention mechanisms to provide a holistic view of emotional content. Learn how to utilize deep learning algorithms to analyze textual and visual sentiment in Tamil and Sanskrit Siddha Palm Leaf Manuscripts with the Cultural Emotion Analyzer for Multimodal and Multilingual Sentiment Analysis (CEA-MMSA). This method deciphers emotional material using visual sentiment analysis using Vision Transformers (ViTs) and textual sentiment analysis using Gated Recurrent Units (GRUs) with attention mechanisms. A new multimodal fusion model combines visual and textual sentiment data to improve analysis. By illuminating subtle emotional and linguistic patterns, the proposed method sheds light on the cultural legacy of these manuscripts. Using the assessment criteria Measured against OCR-CNNs, TAOSNs, EERCA-ViT, and AT-MC-BiGRU in terms of precision, recall, accuracy, and F1-score, the proposed model stands on its own. According to the confusion matrix analysis, the proposed model also does well on the test set, with a 97.38% accuracy rate, 96.78% precision rate, 95.34% recall rate, and 95.37% F1 score. It is clear from the visual depiction that the suggested model aids in the multimodal and multilingual thorough analysis of Siddha drugs from palm inscriptions, thanks to its greater analysis rate.

Author Contribution

Author Contributions:• R.G. conceived and designed the study, and wrote the main manuscript text.• A.V. and R.G. performed the data collection and pre-processing.• A.V. developed and implemented the Vision Transformers (ViTs) model for visual sentiment analysis.• R.G. developed and implemented the Gated Recurrent Units (GRUs) with attention mechanisms for textual sentiment analysis.• A.V. and R.G. performed the multimodal fusion and empirical analysis.• R.G. prepared figures 1-5 and A.V. prepared figures 6-9.• R.G. prepared Table 1.• Both R.G. and A.V. reviewed and edited the manuscript.We confirm that this manuscript has not been published elsewhere and is not under consideration by another journal. All authors have approved the manuscript and agree with its submission to Machine Learning Journal. Thank you for considering our submission. We look forward to your positive response.Sincerely,R.Geethanjali,Full-Time Research Scholar,Faculty of Information and Communication Engineering,UCE-BIT Campus,Tiruchirappalli,Anna University, Chennai

Das, R., Thoudam Doren Singh: Multimodal sentiment analysis: a survey of methods, trends, and challenges. ACM Comput. Surveys. 55(13s), 1–38 (2023)
Khan, A.: Improved multilingual sentiment analysis and recognition using deep learning. J. Inform. Sci. : 01655515221137270. (2023)
Yadav, A., Dinesh Kumar Vishwakarma: A deep multi-level attentive network for multimodal sentiment analysis. ACM Trans. Multimedia Comput. Commun. Appl. 19(1), 1–19 (2023)
Wang, L., Peng, J., Zheng, C., Zhao, T.: A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning. Inf. Process. Manag. 61(3), 103675 (2024)
Liu, Xia, Z., Xu, Huang, K.: Multimodal emotion recognition based on cascaded multichannel and hierarchical fusion. Computational Intelligence and Neuroscience no. 1 (2023): 9645611. (2023)
Gong, P., Liu, J., Zhang, X., Li, X.: A multi-stage hierarchical relational graph neural network for multimodal sentiment analysis. In ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, (2023)
Huang, C., Zhang, J., Wu, X., Wang, Y., Li, M.: TeFNA: Text-centered fusion network with crossmodal attention for multimodal sentiment analysis. Knowl. Based Syst. 269, 110502 (2023)
Gan, C., Fu, X., Feng, Q., Zhu, Q., Cao, Y., Zhu, Y.: A multimodal fusion network with attention mechanisms for visual–textual sentiment analysis. Expert Syst. Appl. 242, 122731 (2024)
Li, H., Lu, Y., Zhu, H.: Multimodal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism. Electronics 13, no. 11 : 2069. (2024)
Braytee, A., Yang, A.S.-C., Anaissi, A., Chaturvedi, K., Prasad, M.: A Novel Dual-Pipeline based Attention Mechanism for Multimodal Social Sentiment Analysis. In Companion Proceedings of the ACM on Web Conference 2024, pp. 1816–1822. (2024)
Subbaiah, B., Murugesan, K., Saravanan, P., Krishnamurthy Marudhamuthu: An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network. Artif. Intell. Rev. 57(2), 34 (2024)
Rani, N., Shobha: A modified deep semantic binarization network for degradation removal in palm leaf manuscripts. Multimedia Tools Appl. : 1–33. (2024)
Jailingeswari, I., Gopinathan, S.: Tamil handwritten palm leaf manuscript dataset (THPLMD). Data Brief. 53, 110100 (2024)
Wang, Y., Wen, M., Zhou, X., Gao, F., Tian, S., Jue, D.: Hongmei Lu, and Zhimin Zhang. Automatic damage identification of Sanskrit palm leaf manuscripts with SegFormer. Herit. Sci. 12(1), 8 (2024)
Kumar, A., Bhatia, S., Khosravi, M.R., Mashat, A., Parul Agarwal: Semantic and context understanding for sentiment analysis in Hindi handwritten character recognition using a multiresolution technique. ACM Trans. Asian Low-Resource Lang. Inform. Process. 23(1), 1–22 (2024)
Cheng, Y., Sun, H., Chen, H., Li, M., Cai, Y.: Zhuang Cai, and Jing Huang. Sentiment analysis using multi-head attention capsules with multichannel CNN and bidirectional GRU. IEEE Access. 9, 60383–60395 (2021)
Subramani, K., Subramaniam, M.: Creation of original Tamil character dataset through segregation of ancient palm leaf manuscripts in medicine. Expert Syst. 38(1), e12538 (2021)
Gayathri, D.S., Vairavasundaram, S., Teekaraman, Y., Kuppusamy, R., Radhakrishnan, A.: A Deep Learning Approach for Recognizing the Cursive Tamil Characters in Palm Leaf Manuscripts. Computational Intelligence and Neuroscience: CIN (2022). (2022)
Sabeenian, R.S., Paramasivam, M.E., Anand, R., Dinesh, P.M.: Palm-leaf manuscript character recognition and classification using convolutional neural networks. In Computing and Network Sustainability: Proceedings of IRSCNS 2018, pp. 397–404. Springer Singapore, (2019)
Haritha, J., Balamurugan, V.T., Vairavel, K.S., Ikram, N., Janani, M., Indrajith, K.: CNN based character recognition and classification in tamil palm leaf manuscripts. In 2022 International Conference on Communication, Computing and Internet of Things (IC3IoT), pp. 1–6. IEEE, (2022)
Athisayamani, Suganya, A.R., Singh, Sivanesh Kumar, A.: Recurrent neural network-based character recognition system for Tamil palm leaf manuscript using stroke zoning. In Inventive Communication and Computational Technologies: Proceedings of ICICCT 2020, pp. 165–176. Springer Singapore, (2021)
Wang, X., Yang, J., Hu, M., Ren, F.: EERCA-ViT: Enhanced effective region and context-aware vision transformers for image sentiment analysis. J. Vis. Commun. Image Represent. 97, 103968 (2023)
Alzamzami, F.: and Abdulmotaleb El Saddik. Transformer-based feature fusion approach for multimodal visual sentiment recognition using tweets in the wild. IEEE Access. (2023)

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Innovative Deep Learning-Based CEA-MMSA Framework for Cultural Emotion Analysis of Tamil and Sanskrit Siddha Palm Leaf Manuscripts

Status:

Version 1

Abstract

Figures

1. Introduction

2. Literature Survey

3. System Methodology

3.1 Structure of CEA-MMSA

3.2 GRU with attention mechanism for text Sentimental Analysis

3.3 Vision Transformers (ViTs) for visual sentiment analysis

3.3 Multimodal Fusion model

3.4 CEA-MMSA Multimodal Fusion Algorithm

1.1 Collect palm leaf manuscript images in Tamil and Sanskrit.

1.2 Store collected images in a database.

2.4 Ensure final images are the same size.

6.1 Apply MLP classification layers.

6.2 Output sentiment classification.

7.1 Calculate accuracy, precision, recall, and F1 score on the validation set.

7.2 Compare against baseline models.

4. Experimental Results

5. Future Research Directions

6. Conclusion

Declarations

Author Contribution

References

Additional Declarations

Status:

Version 1