A study on different deep learning algorithms used in deep neural nets:MLP SOM and DBN

doi:10.21203/rs.3.rs-875603/v1

Download PDF

Research Article

A study on different deep learning algorithms used in deep neural nets:MLP SOM and DBN

https://doi.org/10.21203/rs.3.rs-875603/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Deep learning is a wildly popular topic in machine learning and is structured as a series of nonlinear layers that learn various levels of data representations. To implement various computer models, deep learning employs numerous layers to represent data abstractions. Deep learning approches like generative, discriminative models and model transfers approaches have transformed information processing. This article proposes a comprehensive review of various deep learning algorithms Multi layer perception (MLP), Self-organizing map (SOM) and deep belief networks (DBN) algorithms. It first briefly introduces historical and recent state-of-the-art reviews with suitable architectures and implementation steps. Then the various applications of those algorithms in various fields such as speech recognition engineering, medical applications, natural language processing, material science and remote sensing applications, etc are classified.

Machine learning algorithms

Multi layer perception

self organising maps

deep belief networks

deep neural nets

application of algorithms

Nowadays, deep learning models significantly impact extracting information or hidden patterns from enormous amounts of data with greater accuracy. Compared to conventional machine learning approaches, deep learning can solve complex problems and correlate the interdependent variables. However, traditional neural networks SOM, MLP and DBN have limited high-level data abstraction, which can be alleviated, when combined with deep neural nets. This paper analyzes different hybrid neural networks MLP,SOM and DBN with deep learning models and the techniques used to improve the performance of the model. The hybrid architecture of the algorithms with CNN produces better results compared to standalone CNN models.

It is the commonly used neural network based on supervised learning in which information flows in one direction, and it has no loops. The main objective is to find the optimized function f() that maps input to the desired output and learning the optimized value of bias(θ) for it. Learning occurs in the MLP Using a back propagation algorithm by adjusting the connection weights when there is a deviation from the expected output and the actual output. Their main applications are to solve optimization problems in the area of finance, transportation, fitness and energy.

2.1Architecture, Algorithm and Characteristics of MLP:

It has three layers, namely the input, output and one or more hidden layers. The input layer collects the input features to be processed. An arbitrary number of hidden layers lies between both the input and output layers. They work as the computational unit of the MLP. The output unit performs tasks such as prediction and classification.

Property 1

Universality

MLP is capable of learning both linear as well as non-linear functions. MLPs are designed to approximate any continuous function and can solve problems that are not linearly separable.

Property 2

Adaptive learning and Optimal

MLP can learn how to do tasks from the data given for training and initial experience. MLP minimizes the loss function. Hence it is optimal. Learning the function that maps the inputs to the outputs reduces the loss to an acceptable level.

Property 3

Stochastic

MLP is a stochastic program. In a stochastic program, some or all problem parameters are uncertain and use probability distributions to solve highly complex optimization problems.

Property 4

The power of depth

Compared to shallow ones, deep nets can represent some functions more compactly, such as parity function and a deep network, whose size is linear in the number of inputs computes it.

2.2 Application of MLP:

There are various convolution neural network-based models for remote sensing image classification and better performance. VHR remote sensing image scene classification plays a vital role in remote sensing research; hence they help manage land resources, urban planning, tracking of disasters, and traffic monitoring. Osama A. Shawky et al. (2020) proposed a VHR(Very High Resolution) image scene classification model comprising three phases: Data augmentation to learn robust features, a pre-trained CNN model to extract features from the original image, and an adaptive gradient algorithm multi-layer perceptron to improve the accuracy of the classifier.

With the advent of modern remote sensing technologies, various very fine spatial resolutions (VFSR) dataset is now commercially available. These VFSR images have opened up many opportunities such as urban land use rescue, agriculture, and tree crown description. Zhang, C. et al.(2018) proposed a hybrid classification system that combines the contextually based classifier CNN and pixel-based classifier MLP with a rule-based decision fusion strategy. The decision fusion rules formed based on the confidence distribution of the contextual-based CNN classifier. If the input image patch is at the homogeneous region, the confidence is high.

On the other hand, if the image pixels contains other land cover classes as related information, the confidence is low. As a result, the MLP can rectify the classified pixels with low confidence at the pixel level. This paper also compares the proposed method's performance with benchmark standards such as pixel-based MLP, spectral texture-based MLP, and contextual-based CNN classifiers.

Md Manjurul Ahsan et al. (2020) proposed a hybrid model with a combination of a Convolutional Neural Network (CNN) and Multilayer Perceptron (MLP) in which MLP handles the numerical/categorical data, and CNN extracts features from the X-ray images. Parameter tuning using the grid search method to decide the number of hidden layers, number of neurons, epochs, and batch size. Meha Desai et al.(2020), in their study, compare and analyze the function and designing of MLP and CNN for the application of breast cancer detection and conclude CNN give slightly higher accuracy than MLP.

Vinod Kumar et al.(2020) suggested a hybrid CNN-MLP model that analyzes novel and diversified attacks. The problem of intrusion detection is a classification task using machine learning and deep learning techniques. The model used feature selection and reduction techniques, random forest regressor, along the correlation parameter. The CICIDS2017 dataset used the performance of the proposed model outperforms that of the performance of the individual CNN and MLP models. Hanwen Feng et al.(2020) suggested a CNN model for Classification of Points of Interest in Side-channel attacks and compared it with MLP and concludes MLP is more suitable for PCA traces and CNN is for POI traces; shorter traces improves the classification results. Bikku(2020) proposed a model using MLP to predict future health risk with a certain probability, compare it with LSTM and RNN, and suggest MLP outperforms the other two. Salah, L. B(2019) presented a model to control a bioreactor using deep learning feed-forward neural networks with different MLP structures. The trained model emulates the inverse dynamics of the bioreactor and then uses neural controllers for neural control strategies of the chosen bioreactor. [8]

S..Bairavel et al. (2020) suggested a model for multimodal sentiment analysis using feature-level fusion technique and novel oppositional grass bee optimization (OGBEE) algorithm for fusing the extracted features from different modalities and MLP for classification.[9] Foody. G et al. compared three different neural network approaches, MLP, RBF and PNN, for Thematic mapping from remotely sensed data. For the proposed model, PNN outperforms. [10] Singh, N.H. et al.(2018) designed a model to find the optimal collision-free path and control the robot's speed in a dynamic environment for the mobile robots to reach the destination using MLP. The ultrasonic sensors in the robot sense the obstacle in its path and calculate the distance between them. [11] Meng Wang et al. (2020) devised a model to detect the Distributed Denial of Service (DDoS) attack using MLP with feature selection for optimal feature selection and the Back Propagation algorithm to reconstruct the detector when errors are perceived. The model comprises three modules knowledge base, detection model, and feedback mechanism and MLP act as binary classifier during attack detection.[12]

The model proposed by Morteza Taki et al.(2018) predicts the irrigated and rainfed wheat output energy using artificial network models MLP, RBF and Gaussian Process Regression (GPR). The RBF model performs better than the other two models in predicting wheat output energy under various irrigated and rainfed farms. [13]

Author & Year	The objective of the articles	Experimental design Techniques	Datasets used	Response and Performance measure
A. Shawky et al. (2020)	Remote sensing image scene classification using CNN-MLP with data augmentation	Augmentation to expand the image dataset., pre-trained CNN(The xception pre-trained on the ImageNet dataset) model for feature extraction, Adaptive gradient with MLP for classification Activation function used RELU and softmax, learning rate 0.01	VHR image datasets used UC-Merced, Aerial Image (AID), and NWPU-RESISC45	Accuracy UC-Merced dataset:99.86 AID Dataset: 98.10 NWPU-RESISC45 dataset: 97.40
Zhang, C.et al. (2018)	A hybrid MLP-CNN classifier for very fine resolution remotely sensed image classification	contextual-based classifier CNN, pixel-based classifier MLP with shallow structures	Custom dataset with aerial imagery of Southampton with 50 cm spatial resolution and four multispectral bands (Red, Green, Blue and Near Infrared). Two study sites S1 (3087 X 2750 pixels) and S2 (2022 X 1672 pixels)	Accuracy:89.64
Md Manjurul Ahsan et al.,(2020)	Deep MLP-CNN Model Using Mixed-Data to Distinguish between COVID-19 and Non-COVID-19 Patients	MLP to analyze and classify numerical or categorical data and cnn to analyze and classify the X ray image Optimization algorithms used:adaptive learning rate optimization algorithm (Adam), stochastic gradient descent (Sgd), and root mean square propagation (Rmsprop).	COVID-19 data set collected from the open-source GitHub repository	For balanced dataset, Adam algorithm achieved highest accuracy of 96.3 For imbalanced dataset Rmsprop algorithm achieved highest accuracy of 95.38
Meha Desai et al.,(2020)	An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (MLP) and Convolutional neural network (CNN)	CNN CNN MLP MLP MLP	BreakHis dataset Mammographic Image Ananlysis Society database Digital database for Screening Mammograms (DDSM) are used Wisconsin breast cancer dataset WDBC dataset	98.86 82.71 65.21 95.74 97.51
Vinod Kumar et al. (2020)	Evaluating Hybrid Cnn-Mlp Architecture For Analyzing Novel Network Traffic Attacks	feature selection and reduction technique - random forest regressor with the correlation parameter, hybrid CNN-MLP architecture for classification	CICIDS2017 dataset	Analyzing the novel and diversified attacks. Accuracy 0.972 Precision 0.9827 Recall 0.9813
Hanwen Feng et al.(2020)	MLP and CNN-based Classification of Points of Interest in Side-channel Attacks	CNN and MLP	ANSSI SCA Database (ASCAD) ,SM4 traces	MLP is more suitable for PCA traces and CNN is more suitable for POI traces
Tulasi Bikku,(2020)	Multi-layered deep learning perceptron approach for health risk prediction	MLP	real historical medical data from the University of California at Irvine (UCI) ML Repository	Better performance than LSTM and RNN
Salah, L. B(2019)	An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (MLP) and Convolutional neural network (CNN)	MLP		neural network structure with two hidden layers gives better performances
S.Bairavel et al. (2020)	Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis	Feature-level fusion technique- to fuse the extracted features from different modalities. Novel oppositional grass bee optimization (OGBEE) algorithm -feature selection MLP- for classification	web recordings that utilize audio, video, and textual modalities	Accuracy in sentimental analysis classification 95.2%
M.Foody et al., (2001)	Thematic mapping from remotely sensed data with neural networks: MLP, RBF and PNN based approaches	MLP, RBF and PNN for classification	Remotely sensed data from ATM dataset	The accuracy of the MLP, RBF and PNN models are 86.56,82.5 and 87.18, respectively.
Singh, N.H. et al.(2018)	Mobile Robot Navigation Using MLP-BP Approaches in Dynamic Environments	MLP	Real-time system	Identifies the Optimal collision-free path for mobile robots to reach the destination
Meng Wang et al. (2020)	A dynamic MLP-based DDoS attack detection method using feature selection and feedback, Computers & Security	MLP Feedback mechanism used to detect the error and reconstruct the detector to improve the performance.	ISOT dataset comprises of attack and regular packets and ISCX dataset comprises of normal packets	Classifies the samples as normal or attack
Morteza Taki et al. (2018)	Assessment of energy consumption and modelling of output energy for wheat production by neural network (MLP and RBF) and Gaussian process regression (GPR) models	MLP, RBF and GPR		Predicts output energy consumptions in wheat farms, and the RBF model outperforms the other two.

A Self-organizing map (SOM) is an unsupervised-based neural network algorithm. It is also referred to as the dimensionality reduction algorithm or Kohonen Network with input and output layers without a hidden layer. Since this algorithm reduces the input dimension of the data, the final output is represented as a feature map. Similar samples of maps are merged as a map. Generally, this algorithm is used to convert the high-dimensional dataset into 2D discretized pattern.

3.1 Architecture, Algorithm and Characteristics of SOM:

The SOM architecture has two layers: the input and output layers with the feature map. It does not contain any hidden layer like a neural network. So, it just passes weight values to the output layer without performing any activation function in neurons. At the same time, each neuron is assigned with some weight value based on the input space. The SOM architecture has a feed-forward structure with a 2D computational layer of nodes arranged in rows and columns and connected fully with all other sources of the input layer. Figure 3 depicts an architectural overview of Kohonen’s SOM.

The SOM uses competitive learning to update its weights. It consists of three methods as Competition, Cooperation and Adaptation. In the competition process, compute the distance between each neuron of the Kohonen layer and the input layer and identify the minimum or maximum distance of neuron based upon the applications. That will be considered as the winner of the process. Following the cooperation process, select the neighborhood neurons depends on the time and distance of the winner neurons.

At last, the adaptation process updates the weight values of the winner and cooperative neurons. Finally, it produces a feature map from input variables. The main properties of the SOM are described as, Property 1: Approximation of the Input Space: The feature map in the output space, which is expressed by a collection of weight vectors, is a fair estimation of the input space. Property 2: Topological Ordering: The SOM algorithm produces a topologically ordered feature map, meaning that the spatial position of a neuron in the output lattice or grid correlates to a specific domain or feature of the input data. Property 3: Density Matching: The feature map represents differences in the input distribution's statistics regions in the input space. For example, the high probability sample training data are mapped into more significant domains of the output space, and thus with higher resolution, than regions of input space where training vectors are produced with low probability values. Property 4: Feature Selection: The self-organizing map will pick a set of optimal features for quantifying the underlying distribution of given data from input data with a non-linear distribution. Figure 4represents the algorithmic steps of SOM.

3.2 Application of SOM:

Kohonen et al., 1996 initially used them for speech recognition. But nowadays it is used in various applications such as, Pattern recognition, speech processing, industrial and medical diagnostics, and data mining etc using some hybrid architecture merged with RNN, CNN and back tracking approaches. The phonetic typewriter of Kohonen is one of the earliest and most well-known applications of the SOM. The challenge is to identify phonemes in real time so that they can be used to drive a typewriter from dictation in the field of speech recognition. The speech signals (Cesar et al., 2010) are pre-processed before applied to the SOM. The Fourier transforming and filtering process are used to sample the data using 24 dimensional spectral vectors. The proposed network was effectively trained using speech waveforms, and the output nodes are naturally clustered with the ideal phonemes. Finally, the model output generated logical phoneme strings from real-world speech applications.

Frias-Martinex et al. (2012) used a SOM to create a map that partitions the urban segments into geographic segments with various proportions of parts in the timeframe. Behnisch and Ultsch (2009) used an extension of SOM as an emergent self-organizing map (ESOM) for clustering and classification of the data. It keeps the high-dimensional data's neighborhood relationships. On the other hand, the finite grid has a drawback because neurons on the map's edges have somewhat different mapping qualities than neurons in the middle versus those on the boundary. Growing hierarchical self-organizing maps (GHSOM) used by Chifu and Letia (2008) consist of a set of SOMs (particularly bidimensional grids) organized as nodes in a hierarchy. GHSOM is initialized with a hierarchy mirroring the one in the taxonomy, and concepts are mapped to some nodes in the corresponding SOM by initializing the node’s weights with the vector description of the concept; all other unmapped nodes are initialized randomly.

D.Chen et al 2000 designed the architecture for Breast cancer diagnosis using neural network based self-organizing maps (SOM).It classified tumour breast or benign and non tumour or malignant lesions with the samples of 243 breast tumours. The SOM based autocorrelation texture features used to classify the tumours. Osman et al. 2021 suggested a corona virus detection technique based on the Locality weighted learning and self-organization map (LWL-SOM) technique for capturing the images and identify the diseases of COVID-19 cases. They grouped the chest X-ray data patterns based on SOM strategy to categorize between the positive and negative cases of COVID-19. Then, they built locality weighted learning model for diagnosing the cases. Materials Informatics application

Author & Year	Application & Objective	Experimental design Techniques	Datasets used	Response and Performance measure
Kohonen; et al.,1996.	Speech Recognition: Engineering applications of the self-organizing map	Pre processing and st processing, Filtering ,Clustering and fourier transform	real speech signals	logical phoneme strings
D.Chen et al 2000	Medical application : Breast cancer diagnosis using self-organizing map for sonography	RNN-SOM based autocorrelation texture classification	Cancer tumour diagnosis	Classified cancer and non cancer tumours with the accuracy 85. 6%
Osman et al 2021	Medical application : OM-LWL method for identification of COVID-19 on chest X-rays	locality weighted learning and self-organization map (LWL-SOM) based classification	COVID-19 diagnosis	Classified COVID 19 with the accuracy 0.885 probability
Mehrbakhsh et al., 2020	Medical application : Remote tracking of Parkinson's Disease progression using ensembles of Deep Belief Network and Self-Organizing Map	Deep Belief Network based Self-Organizing Map technique is used	Remote tracking of Parkinson's Disease progression.	The accuracy improvement in UPDRS prediction was significant.
Khanzadeh et al., 2019	Medical application : identify abnormal pool and predict porosity	self-organizing map (SOM)	identify abnormal melt pool and predict porosity with the help of ex-situ CT scan data
Jafari et al ., 2019	Material Science: cost-driven decision-making framework for laser-based additive manufacturing	Neural network based self-organizing map (SOMNN)	Thermal profile and location of the melt pool	Detect the location and size of pores
Weihua et al., 2013	Material Science: Semisupervised Distance-Preserving Self-Organizing Map for Machine-Defect Detection and Classification	semisupervised diagnosis method based on a distance-preserving SOM technique	machine-fault detection and classification	effective in detecting incipient gear-pitting failure and classifying different bearing defects and levels of ball-bearing defects.
Jimin et al., 2019	Materials Informatics application: Visualize and validate relationships between high-dimensional materials dataset	Gruneisen and U matrix map based SOM technique.	Visualize and validate the materials form large data set.	.
Nicolas et al., 2019	Medical applications: Diagnosis spinal column problem	Diagnosis and compare spinal column problem using SOM and K-Clustering	Classified the images from UCI machine learning repository using the parameters such as specificity, negative predictive value, precision, and Cohen's Kappa index.	SOM performed better than the K-means approach, for detection of vertebral problems.
Felix et al 2019	Remote sensing: classification of land cover or for the regression of physical parameters using SuSi Framework	Evaluation of hyper spectral datasets using regression and classification techniques.	Applied the semi-supervised regression and SOM classification using two different hyper spectral datasets.	Unsupervised and supervised of the SuSi framework solved the problem semi-supervised concepts.
Srivatsa et al., 2021	Remote sensing: Classification of high dimensional hyper spectral data.	SOM and Cellular Self-Organizing Map used to analysis the data sets.	t-Distributed Stochastic Neighbor Embedding technique used for exploration and visualizing of data.	CSOM performs better than SOM for the overlapping classes while SOM is efficient for classes which are well-separated or separable.
(Jimin et al., 2019) used the machine learning technique of SOM to visualize and validate relationships between high-dimensional materials dataset. Compared to conventional methods, the SOM categorized and validated the materials using various mapping techniques like U-matrix map, heat maps, cluster-based map, and Gruneisen parameter. Nicolas et al., 2019, proposed the classification of vertebral problems using K-means and SOM algorithm. In his approach, SOM outperformed than the K-Means clustering analysis. Felix et al 2019 and Srivatsa et al., 2021 used the hybrid version of self organising map algorithm with SuSi framework and Cellular Self-Organizing Map for classifying the hyper spectral datasets.

Nowadays, Machine Learning dominates research interests due to its vast application in various fields. “Deep Learning,” a type of machine learning algorithm also knowns as Representation Learning (Li Deng, 2014) has its application in multimedia concept retrieval, text mining, social network analysis, video recommendation. Deep Learning represents ANN with layered network topologies of neuron models.

4.1 Architecture, Algorithm and Characteristics of DBN:

Hinton et al.’s Deep Belief Network (DBN) (Hinton et al., 2006)is a popular deep learning algorithm representing advanced learning methodology, more profound architecture, and high-level abstraction of biological modelling, giving rise to simplified mathematical models. Its network architecture inspired by artificial intelligence (AI) research study replicate human-level intelligence.

DBN, an alternate class of Deep Neural Network, is a graphical model with multiple layers of ‘hidden units’ with a connection within layers and not within each layer (Hinton G,2009). Trained Unsupervision DBN reconstructs its inputs probabilistically acting as feature detectors, whereas trained Supervision DBN is utilized for classification. Restricted Boltzmann Machine (RBM) / autoencoders is an unsupervised DBN representing undirected, generative energy-based model where hidden layer acts as a visible layer for the successor. DBN, when trained greedily layer-wise, leads to an effective deep learning algorithm (Bengio 2009). Overall deep belief network real-time application scenarios include computer vision, electroencephalography (Movahedi et al 2018 ), drug discovery (Movahedi et al., 2018), natural language processing, speech recognition, material inspection, board game programs, and so forth, with fantastic outcomes surpassing human expert achievement. Figure 6 represents the algorithmic steps of Deep Belief Network.

4.2 Application of DBN:

G.Liu et al., (2017) suggested a model for image classification using Deep belief network. In this, the stacked restricted Boltzmann machine make use of contrastive divergence algorithm for feature extraction and softmax layer make use of Evolutionary Gradient Descent(EGD) strategy to classify the extracted features. The acceleration rate of EGD is remarkable compared to Gradient Descent algorithm. Jianjian Yang et al.,(2020) proposed a model for deep fault recognition using Deep Belief Network. To refine the result of the DBN, the proposed method makes use of stochastic adaptive particle swarm optimization (RSAPSO) algorithm. To address the limitations such as local optimization and low search accuracy of conventional PSO algorithms, the proposed system used RSAPSO algorithm that allows particles to reset the position from the original with an assured probability, and continue its searching again. Hence, the proposed method minimizes the probability of the trapping in a local minimum of particle swarm. Dan Wang et al., (2014) presented a system that applied raw physiological data to Deep Belief Networks (DBNs) with three classifiers to envisage the levels of emotions such as arousal, valence, and liking based on the known features.

The classification accuracies obtained are better than the results acquired by Gaussian Naïve Bayes classifier. Xiaoai Dai et al., (2020) devised a DBF model to extract artificial target features in cities, as a hyper-spectral image. DBF performs dimensionality reduction and extracts the depth features of pixels. DBF provides better robustness and separability compared with Principal component analysis. O’Connor., P. et al., (2013) This paper proposes a method based on the Siegert approximation for Integrate-and-Fire neurons to map an offline-trained DBN onto an efficient event-driven spiking neural network suitable for hardware implementation. The method is demonstrated in simulation and by a real-time implementation of a 3-layer network with 2694 neurons used for visual classification of MNIST handwritten digits with input from a 128 × 128 Dynamic Vision Sensor (DVS) silicon retina, and sensoryfusion using additional input from a 64-channel AER-EAR silicon cochlea. The system is implemented through the open-source software in the jAER project and runs in realtime on a laptop computer. It is demonstrated that the system can recognize digits in the presence of distractions, noise, scaling, translation and rotation, and that the degradation of recognition performance by using an event-based approach is less than 1%. Movahedi, faezeh et al.,(2017) discussed the state of- the-art algorithms for deep belief networks and their performances in electroencephalographic applications in medical fields such as emotion recognition, sleep stage classification, and seizure detection. It also includes the challenges and future research direction of DBF in electroencephalographic applications. Abdellaoui, M., Douik, A. (2020) suggested an optimal HAR system with a two-phase DBN model that offers a better quality of classification prediction.

Author & Year	Objective of the articles	Experimental design Techniques	Datasets used	Response and Performance measure
G.Liu et al., (2017)	Image Classification with Deep Belief Networks and Improved Gradient Descent	Deep belief networks for image feature extraction and improved gradient descent method to classify the feature vectors	MNIST dataset, UCI dataset	MNIST dataset Training set error: 4.05% Test set error:6.11% UCI dataset Training set error: 0.42% Test set error:0.73%
Jianjian Yang et al.,(2020)	Design and Application of Deep Belief Network Based on Stochastic Adaptive Particle swarm Optimization	DBF for fault detection and RSAPSO algorithm for improve the recognition rate	data from Case Western Reserve University in the United States	recognition rates 87.75% and 93.75%
Dan Wang et al., (2014)	Modelling Physiological Data with Deep Belief Networks	DBF with back propagation algorithm	DEAP dataset	The accuracies were 0.570 for arousal, 0.627 for valence, and 0.591 for liking, respectively.
Xiaoai Dai et al., (2020)	Deep Belief Network for Feature Extraction of Urban Artificial Targets	DBF for feature extraction and dimensionality reduction	Hyper spectral datasets	DBF outperforms compared with other feature dimensionality reduction techniques
O’Connor., P. et al., (2013)	Real-time classification and sensor fusion with a spiking deep belief network	offline-trained DBN based on the Siegert approximation for Integrate-and-Fire neurons	MNIST dataset	Above 94%
Movahedi, Faezeh et al.,(2017)	Deep Belief Networks for Electroencephalography	DBN: A Review of Recent Contributions and Future Outlooks.	DEAP dataset	-
Abdellaoui, M., Douik, A. (2020)	Human Action Recognition in Video Sequences	Human Action Recognition in Video Sequences using Deep Belief Network	KTH and UIUC human action datasets	94.83% for KTH and 96% for UIU

Limitations of MLP, SOM and DBN algorithms:

MLP with back propagation has feeble generalization ability for statistically neutral problems. Hence the model does not know the expected output, and the relation between different input variables determines the output.MLP has too many parameters as it is fully connected and results in redundancy and inefficiency.MLP disregards Spatial information on solving problems. SOM minimizes the volume of the dataset, making it easier to visualize and form clusters. However, it has several flaws, including poor handling of categorical variables. Solutions for enormous data sets those are computationally demanding and potentially inaccurate. As per DBN, it needs High energy and large space requirement for its execution. Huge requirement of Random Number Generators (RNGs) drops deep belief networks energy efficiency (Yidong et al., 2018).Moreover in DBN Learning time is prolonged in a back-propagation neural network with multiple hidden layers. Greedy learning is inefficient in the directed module as posterior is not factorizable in each training case. Integrating the overall organization of higher variables prior to the first hidden layer makes learning layer-wise difficult in a sigmoid belief network.

Conclusion:

The field of machine learning simulating the deep learning methods slowly becomes the dominated domain in this era. This chapter discusses the issues and challenges and aggregates numerous existing solutions in medical, material and NLP applications. Though, there are still several issues and challenges that need to be tackled in future. MLP is one of the deep learning approaches with universality and stochastic properties. Hence its application is versatile, such as medical image processing for disease prediction, remote sensing for image scene classification and networks to identify the attacks, energy engineering to predict the output energy in various applications, and robot navigation to determine the optimal collision-free path the destination and so on. The self-organizing maps and modified versions of SOM are discussed based on various application platforms. The unsupervised-based learning algorithm has some standard steps and properties to create a map from the large or complicated input data set. Due to its high adaptiveness, many applications like medical diagnosis, data compression techniques, Bibliographic classification, Image browsing has used this algorithm for classification. Deep learning is the fast-growing approach that provides solutions for hectic challenges in a variety of applications. Moreover, machine learning is transforming into its new phase of intelligent Artificial Intelligence applications.

We author of the above titled paper hereby declare that the work included in the above paper is original and is an outcome of the research carried out by the authors indicated in it. Further, We author declare that the work submitted for Wireless Personal Communications An International Journal has not been published already or under consideration for publication in any Journals/Conferences/Symposia/Seminars.

Funding (Not Applicable)

Conflicts of interest/Competing interests (Not Applicable)

Availability of data and material (data transparency) (Not Applicable)

Code availability (software application or custom code) (Not Applicable)

Shawky, O. A., Hagag, A., El-Dahshan, E.-S. A., & Ismail, M. A. (2020). Remote Sensing Image Scene Classification Using CNN-MLP with Data Augmentation. Optik, 165356. doi:10.1016/j.ijleo.2020.165356
Zhang, C., Pan, X., Li, H., Gardiner, A., Sargent, I., Hare, J., & Atkinson, P. M. (2018). A hybrid MLP-CNN classifier for very fine resolution remotely sensed image classification. ISPRS Journal of Photogrammetry and Remote Sensing, 140, 133–144. doi:10.1016/j.isprsjprs.2017.07.014
Ahsan, M.M.; E. Alam, T.; Trafalis, T.; Huebner, P. Deep MLP-CNN Model Using Mixed-Data to Distinguish between COVID-19 and Non-COVID-19 Patients. Symmetry2020, 12, 1526. https://doi.org/10.3390/sym12091526
Meha Desai, Manan Shah,An anatomization on breast cancer detection and diagnosis employing multi-layer perceptron neural network (MLP) and Convolutional neural network (CNN), Clinical eHealth, Volume 4, 2021, Pages 1-11, ISSN 2588-9141, https://doi.org/10.1016/j.ceh.2020.11.002.
Vinod Kumar, Kanika Rana, Jyoti Malik, Ayushi Tomar, Evaluating Hybrid Cnn-Mlp Architecture For Analyzing Novel Network Traffic Attacks, INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH VOLUME 9, ISSUE 03, MARCH 2020, ISSN 2277-8616
Hanwen Feng, Weiguo Lin^*, Wenqian Shang, Jianxiang Cao, Wei Huang, MLP and CNN-based Classification of Points of Interest in Side-channel Attacks, International Journal of Networked and Distributed Computing, Volume 8, Issue 2, 2020, Pages 108 - 117
Bikku, T. Multi-layered deep learning perceptron approach for health risk prediction. J Big Data 7, 50 (2020). https://doi.org/10.1186/s40537-020-00316-7
Salah, L. B., & Fourati, F. (2019). Deep MLP neural network control of bioreactor. 2019 10th International Renewable Energy Congress (IREC). doi:10.1109/irec.2019.8754572
Bairavel, S., Krishnamurthy, M. Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis. Soft Comput 24, 18431–18445 (2020). https://doi.org/10.1007/s00500-020-05049-6
Foody, G. Thematic mapping from remotely sensed data with neural networks: MLP, RBF and PNN based approaches. J Geograph Syst 3, 217–232 (2001). https://doi.org/10.1007/PL00011477
Singh, N.H., Thongam, K. Mobile Robot Navigation Using MLP-BP Approaches in Dynamic Environments. Arab J Sci Eng 43, 8013–8028 (2018). https://doi.org/10.1007/s13369-018-3267-2
Meng Wang, Yiqin Lu, Jiancheng Qin,A dynamic MLP-based DDoS attack detection method using feature selection and feedback, Computers & Security, Volume 88, 2020, 101645, ISSN 0167-4048, https://doi.org/10.1016/j.cose.2019.101645.
Morteza Taki, Abbas Rohani, Farshad Soheili-Fard, Abbas Abdeshahi, Assessment of energy consumption and modeling of output energy for wheat production by neural network (MLP and RBF) and Gaussian process regression (GPR) models, Journal of Cleaner Production, Volume 172, 2018, Pages 3028-3041, ISSN 0959-6526, https://doi.org/10.1016/j.jclepro.2017.11.107.
Teuvo Kohonen. Self-Organizing Maps. Springer, Berlin, Heidelberg, 1995.
Oyana TJ, Boppidi D, Yan J, Lwebuga-Mukasa . Exploration of geographic information systems (GIS)-based medical databases with self-organizing maps (SOM): a case study of adult asthma. In: Bernard L, Friis-Christensen A, Pundt H, editor. The European Information Society. Springer Berlin Heidelberg; 2008.
D Chen , R F Chang, Y L Huang , Breast cancer diagnosis using self-organizing map for sonography , Ultrasound Med Biol . 2000 Mar;26(3):405-11. doi: 10.1016/s0301-5629(99)00156-8.

Liao G., Chen P., Du L., Su L., Liu Z., Tang Z., et al., “Using SOM neural network for X-ray inspection of missing-bump defects in three-dimensional integration,” Microelectronics Reliability, vol. 55, pp. 2826–2832, 2015.
Mehrbakhsh Nilashi, Hossein Ahmadi, Abbas Sheikhtaheri, Roya Naemi, Reem Alotaibi, Ala Abdulsalam Alarood, Asmaa Munshi, Tarik A. Rashid, Jing Zhao,Remote tracking of Parkinson's Disease progression using ensembles of Deep Belief Network and Self-Organizing Map,Expert Systems with Applications, Volume 159,2020.
Li, S. Zhang and G. He, “Semisupervised distance-preserving self-organizing map for machine-defect detection and classification,”IEEETrans. Instrum. Mea., vol. 62, no. 5, pp. 869–879, May 2013.
Jafari-Marandi, M. Khanzadeh, W. Tian, B. Smith, L. Bian, From in-situ monitoring toward high-throughput process control: cost-driven decision-making framework for laser-based additive manufacturing, Int. J. Ind. Manuf. Syst. Eng. 51,(2019) 29–41.
Khanzadeh, S. Chowdhury, M.A. Tschopp, H.R. Doude, M. Marufuzzaman,L. Bian, In-situ monitoring of melt pool images for porosity prediction in directed energy deposition processes, Iise Trans. 51 (5) (2019) 437–455.
Ramaswamy S, Tamayo P, Rifkin R, Mukherjee S, Yeang CH, Angelo M, Ladd C, Reich M, Latulippe E, Mesirov JP, Poggio T, Gerald W, Loda M, Lander ES, Golub TR, “Multiclass cancer diagnosis using tumor gene expression signatures”, Proc Natl Acad Sci U S A. 2001 Dec 18; 98(26):15149-54.
Koua EL, Kraak MJ, “Geovisualization to support the exploration of large health and demographic survey data” ,Int J Health Geogr. 2004 Jun 4; 3(1):12.
Jimin Qiana,1, Nam Phuong Ngu, “Introducing self-organized maps (SOM) as a visualization tool for materials research and education”, Results in Materials,vol4, 2019.
Osman AH, Aljahdali HM, Altarrazi SM, Ahmed A , “SOM-LWL method for identification of COVID-19 on chest X-rays”, PLoS ONE 16(2): e0247176. https://doi.org/10.1371/journal.pone.0247176, 2021.
Weihua Li; Shaohui Zhang; Guolin He, “Semisupervised Distance-Preserving Self-Organizing Map for Machine-Defect Detection and Classification”, IEEE Transactions on Instrumentation and Measurement, Volume: 62, Issue: 5, May 2013.
Nicolas Andres Melo Riveros, Bayron AlexisCardenas Espitia, Lilia EdithAparicio Pico, “Comparison between K-means and Self-Organizing Maps algorithms used for diagnosis spinal column patients”, Informatics in Medicine Unlocked, Volume 16, 2019.
Felix M. Riese, Sina Keller, Stefan Hinz, “Supervised and Semi-Supervised Self-Organizing Maps for Regression and Classification Focusing on Hyperspectral Data”, Remote Sensing ,Volume 12 ,Issue 1 ,3390/rs12010007,2019.
Srivatsa Mallapragada; Chih-Cheng Hung, “Statistical Perspective of SOM and CSOM for Hyper-Spectral Image Classification”, IEEE explorer, doi: 1109/IGARSS39084.2020.9324200,2021.
Cesar Estrebou,Laura Lanzarini,Waldo Hasperuée, “ Voice recognition based on probabilistic SOM”, Conferencia Latinoamericana en Informática ,2010. Li Deng. 2014. ‘A tutorial survey of architectures, algorithms and applications for deep learning’ APSIPA Transactions on Signal and Information processing 3(2014), 1-29.
Hinton, S. Osinder, Y. W. Teh ‘A fast learning algorithm for deep belief nets’, Neural Comput. 18(7), 2006, 1527-1554.
Hinton G(2009), ‘Deep belief networks’, Scholarpedia.4(5): 5947. Bibcode : 2009 Schpj. 4.5947H. doi : 10.4249/Scholarpedia.5947.
Bengio Y(2009). ‘Learning deep architecture for AI’ (PDF). Foundations and trends in machine learning. 2 : 1-127. Citeseerx 10.1.1.701.9550, doi : 10.1561/ 2200000006.
Movahedi F, Coyle JL, Sejdic E (May 2018). ‘Deep belief networks for Electroencephalography : A Review of recent contributions and future outlooks’. IEEE journal of biomedical and health informatics 2(3) : 642-652.Doi: 10.1109/jbhi.2017.2727218
Ghasemi, Perez – Sanchez, Mehri, Perez – Garrido(2018). ‘Neural network and deep-learning algorithms used in QSAR studies merits and drawbacks’, Drug Discovery Today 23(10): 1784 – 1790, doi : 10.1016/j.drudis.2018.06.016 PMID 29936244
Yidong Liu, Yanzhi Wang, Fabrizio Lombardi, Jie Han 2018, ‘An energy-efficient stochastic computational deep delief network’, Design, Automation and Test in Europe conference and exhibition(DATE), 1175-1178, 2018.
Xiaoyu Zhang, Rui Wang, Tao Zhang, Yabin Zha 2016, ‘Short-term load forecasting based on a improved deep belief network’, Interntional conference on smart grid and clean energy technologies (ICSGCE), 339-342, 2016.
Liu, L. Xiao and C. Xiong, "Image Classification with Deep Belief Networks and Improved Gradient Descent," 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC), 2017, pp. 375-380, doi: 10.1109/CSE-EUC.2017.74.
Jianjian Yang, Boshen Chang, Xiaolin Wang, Qiang Zhang, Chao Wang, Fan Wang, Miao Wu, "Design and Application of Deep Belief Network Based on Stochastic Adaptive Particle Swarm Optimization", Mathematical Problems in Engineering, vol. 2020, Article ID 6590765, 10 pages, 2020. https://doi.org/10.1155/2020/6590765
Wang, Dan, and Yi Shang. “Modeling Physiological Data with Deep Belief Networks.” International journal of information and education technology (IJIET) vol. 3,5 (2013): 505-511. doi:10.7763/IJIET.2013.V3.326
Xiaoai Dai, Junying Cheng, Yu Gao, Shouheng Guo, Xingping Yang, Xiaoqian Xu, Yi Cen, "Deep Belief Network for Feature Extraction of Urban Artificial Targets", Mathematical Problems in Engineering, vol. 2020, Article ID 2387823, 13 pages, 2020. https://doi.org/10.1155/2020/2387823
O’Connor, P., Neil, D., Liu, S.-C., Delbruck, T., & Pfeiffer, M. (2013). Real-time classification and sensor fusion with a spiking deep belief network. Frontiers in Neuroscience, 7. doi:10.3389/fnins.2013.00178
Movahedi, Faezeh & Coyle, James & Sejdic, Ervin. (2017). Deep Belief Networks for Electroencephalography: A Review of Recent Contributions and Future Outlooks. IEEE Journal of Biomedical and Health Informatics. PP. 1-1. 10.1109/JBHI.2017.2727218.
Abdellaoui, M., Douik, A. (2020). Human action recognition in video sequences using deep belief networks. Traitement du Signal, Vol. 37, No. 1, pp. 37-44. https://doi.org/10.18280/ts.370105
https://medium.datadriveninvestor.com/deep-learning-deep-belief-network-dbn-ab715b5b8afc
https://www.artificial-intelligence.blog/education/dl-algorithms-deep-belief-networks-dbn.

Download PDF

Editorial decision: Minor revisions
28 Jun, 2022
Reviewers agreed at journal
22 May, 2022
Reviewers invited by journal
15 May, 2022
Editor assigned by journal
03 Sep, 2021
First submitted to journal
03 Sep, 2021

You are reading this latest preprint version

A study on different deep learning algorithms used in deep neural nets:MLP SOM and DBN

Status:

Version 1

Abstract

Figures

1. Introduction:

2. Multi-layer Perceptron(Mlp) :

2.2 Application of MLP:

3. Self-organizing Map (Som):

3.1 Architecture, Algorithm and Characteristics of SOM:

3.2 Application of SOM:

4. Deep Belief Network:

4.1 Architecture, Algorithm and Characteristics of DBN:

4.2 Application of DBN:

Declarations

References

Status:

Version 1