Deep Learning Based Classication of Time Series of Chaotic Systems over Graphic Images

In this study, for the rst time in the literature, identication of different chaotic systems by classifying graphic images of their time series with deep learning methods is aimed. For this purpose, a data set is generated that consists of the graphic images of time series of the most known three chaotic systems: Lorenz, Chen, and Rossler systems. The time series are obtained for different parameter values, initial conditions, step size and time lengths. After generating the data set, a high-accuracy classication is performed by using transfer learning method. In the study, the most accepted deep learning models of the transfer learning methods are employed. These models are SqueezeNet, VGG-19, AlexNet, ResNet50, ResNet101, DenseNet201, ShueNet and GoogLeNet. As a result of the study, classication accuracy is found between 96% and 97% depending on the problem. Thus, this study makes association of real time random signals with a mathematical system possible.


Introduction
Chaotic systems are nonlinear mathematical models that are used to de ne chaotic behaviour. In other words, chaotic systems are nonlinear systems that exhibit chaotic behaviour. In the recent years, chaotic systems are used in many different areas of engineering such as cryptography, image and audio encryption, secure communication, data security, random number generation, digital signature applications, weak signal detection and DC-DC converters. Another hot topic in the recent years is deep learning. There are a great number of studies on deep learning, however most of these studies focus on classi cation processes in different areas. In this study, two of the most topical subjects of the literature, namely chaos and deep learning, are focused on and time series of the chaotic systems are classi ed with deep learning.
Deep learning is a subbranch of machine learning [1] that employs multi-layer arti cial neural network on different areas such as image processing [2], [3], voice recognition [4], [5], natural language processing [6]- [8], and object recognition [9]. The difference of deep learning from the traditional machine learning algorithms is that deep learning makes automatic learning from a database (image, audio, video etc.) possible instead of using predetermined rules [10]- [12]. To the best of the author knowledge, there is no deep learning-based classi cation study on the images of chaotic systems. However, in the literature, there are studies in which deep learning based classi cation of the chaotic systems from signals is carried out.
In their study, Boulle et al. [13], have used ShallowNet, Multilayer perceptrons -MLP, the Fully Convolutional Neural Network -FCN, Residual Network -ResNet, Large Kernel Convolutional Neural Network -LKCNN methods to classify time series of the discrete and continuous time dynamic systems. They stated that the highest classi cation performance is achieved with LKCNN method. Yeo [14], used the Long Short Term Memory Network -LSTM to predict a chaotic dynamical system from observed noise. He concluded that LSTM predicts the nonlinear dynamics with high accuracy by ltering out the noise effectively. Kuremoto et al. [15], proposed generation of Deep Belief Nets -DBNs using Restricted Boltzmann Machine -RBM and Multi-Layer Perceptron -MLP to predict time series of Lorenz and Henon map chaotic systems. They showed that the prediction sensitivity of their proposed DBNs is higher than traditional DBNs. Sangiorgio et al. [16], compared the prediction accuracy and delity of their four proposed models (namely, FF-recursive, FF-multi-output, LSTM-TF and LSTM-no-TF). These models are based on Feed-Forward -FF and Teacher Forcing -TF education model and Recurrent Neural Networks of the arti cial, noise free time series of the chaotic oscillator. They show that LSTM predictor gives much better results than FF-recursive and FF-multi-output methods.
As it can be seen in the mentioned studies, while there is very little study on deep learning-based classi cation of the chaotic signals, there is no classi cation of chaotic systems over graphic images. Yet, classi cation of chaotic behaviours or random signals is seen as a necessity. In this article, a method to classify time series of chaotic systems based on SqueezeNet, VGG-19, AlexNet, ResNet50, ResNet101, DenseNet201, Shu eNet and GoogLeNet of transfer learning methods is presented to address the mentioned necessity. An original study is presented by classifying time series of two different chaotic system with high accuracy, for the rst time in the literature.
In the second section of the study, the used chaotic systems and data set obtains from these chaotic systems are presented. In the third section of the study, the deep learning methods used in the classi cation is brie y presented. In the fourth section of the study, simulation of the classi cation processes and their performance results are presented. Finally, in the last section the conclusion is presented.

Used Caotic Systems And Obtaining Of Data Set
In this section, the three different chaotic systems (namely, Lorenz, Chen and Rossler) whose time series to be classi ed and data set constructed from time series of the three chaotic systems are mentioned. In the literature, there are a very large number of chaotic systems. Hence, some criteria are considered for the selection of the chaotic systems that are used in the study. These Lorenz, Chen and Rossler systems are selected because of they are the most known systems in the literature, 3 dimensional and their mathematical models contains similar nonlinear terms. In addition to this, they can be used to model real life physical systems such as atmospheric, electrical, or chemical systems. Moreover, the time series and phase portraits of the Lorenz and Chen systems are similar and the time series and phase portraits of Rossler systems is different from that of the other two systems makes selection of these systems is logical in terms of evaluation of classi cation performance.

Lorenz System
Lorenz system was developed by Edward Lorenz in 1963 as a simpli ed mathematical model of atmospheric convection [17], [18]. Lorenz system consists of three ordinary differential equations as seen in Equation 1. Here α, β and γ are the system parameters and x, y and z are the state variables of the system.
Lorenz systems is also used as a simpli ed mathematical model of thermosiphons [19], lasers [20], electrical circuits [21], brushless DC motors [22], dynamos [23], and chemical reactions [21]. In Figure 1, the time series and phase portraits of the Lorenz system is given for the system parameters α = 10, β = 8/3, γ = 28 and the initial values x 0 = 5, y 0 = -5 and z 0 = 7.5. As it is seen in Figure 1, the time series has random variations and the phase portraits which show the relationship between two state variables have their own order and orbit. By utilizing this feature of the chaotic systems, the chaotic systems can be used in areas like encryption and data security.

Chen System
Guanrong Chen and Ueta were proposed a double scroll chaotic system in 1999. This system is called Chen System or Chen Chaotic Attractor which resembles the Lorenz System. However, the Chen System is not equivalent to the Lorenz System [24], [25]. Chen System also consists of three ordinary differential equations as shown in Equation 2. Here α, β and γ are the system parameters and x, y and z are the state variables of the system.

Rossler System
The Rossler system, developed by Otto Rossler in 1976, is used in modelling of chemical reactions [26], [27]. Rossler system also consists of three ordinary differential equations as the previous two systems.
The Rossler system is given in Equation 3. Here α, β and γ are the system parameters and x, y and z are the state variables of the system.
Rossler system resembles Lorenz systems equation-wise, but Rossler system is easier to analyse, and it is a single scroll system. In Figure 3, the time series and phase portraits of the Rossler system is given for the system parameters α = 0.1, β = 0.1, γ = 14 and the initial values x 0 = 11, y 0 = 11 and z 0 = 0. Since the time series and phase portraits of the Rossler system is different than those of the other two systems, the time series of the Rossler system will be easily classi ed. Hence, the performance of the deep learningbased classi cation method on an easy problem will be evaluated.

Creating the Data Set
In this section, the information regarding constructed data set is mentioned. To calculate the state variables of all systems and to obtain the time series which constitute the data set, Runge-Kutta 4 (RK4) algorithm is used. Moreover, the values of the calculation and system parameters are varied to diversify time series data. For every time series, 750 different results are obtained by changing the length of time series (the number of total calculated points), the step size of RK4 algorithm, the values of the system parameters, or the values of the initial conditions. Since all the systems are a 3-dimensional system, 2250 different time series are generated for every system. Hence, the data set is generated by calculating 6750 different time series. In Table 1, the values of calculation and system parameters are given. Since the transfer learning method is very e cient at the classi cation of images, all the graph of the obtained time series are saved as a 128x128 pixel image. The data set contains 6750 different graphic images of the time series.

The Used Deep Learnİng Models
In this study, a deep learning-based algorithm to classify the x, y, and z time series of the three chaotic systems is presented. The data set contains images of the time series of the three chaotic systems. The time series are obtained with RK4 algorithm, then the graph of the time series is saved as an image.

Deep Learning and Convolutional Neural Networks -CNN
Deep learning is a subbranch of machine learning algorithms and tries to learn differences on data by employing different architectures [28]. Deep learning methods based on arti cial neural network perform learning process by employing multilayer neural networks [29], [30].
Machine learning algorithms are traditionally designed for analysing data sets with very few features. On the other hand, convolutional neural network (CNN) is developed to analyse data sets with a great number of features where machine learning algorithms are not suitable to analyse such data sets. Since the size of image data is high and every image contains hundreds or thousands of pixels, the CNNs perform much better than traditional machine learning algorithms for processing such data sets [31].
Lecun et al. proposed LeNet networks to analyse large size images in 1988 and they had developed this network until 1998. LeNet networks accepted as the rst of the CNN models [32], [33]. LeNet networks consists of sublayers of successive convolutional and maximum pooling layers. The layers corresponding to Multi-Layer Perceptron-MLP make the top layers. The weights of the network are trained to minimize the average error between obtained and predicted results [31].
In the literature, there are many proposed CNN network models. Some of these are SqueezeNet, VGG-19, AlexNet, ResNet50, ResNet-101, DenseNet-201, Shu eNet, and GoogLeNet [34]. With these pre-trained networks, the operations of CNN models can be carried out faster, the prediction performance can be improved, and training processes can be performed faster. The performance of the networks depends on the problem they applied. Hence, these networks applied to the problem then the pre-trained network with best performance is selected. In this study, many different networks have been tested and the ones with highest performance are selected.

GoogLeNet
Szegedy et al. proposed GoogLeNet which contains 22 layers and is a pre-trained CNN network architecture. In ILSVRC2014 contest, GoogLeNet was selected as the most successful network. [35]- [37]. As shown in Figure 5, the architecture of the network employs parallel connected layers to minimize the possibility of memorization.
The inception modules that enable the realization of multi-core convolution and maximum pooling at one layer simultaneously provide training of the network with optimum weights and extraction of more useful features [39]. To achieve this, every starting layer has 1×1, 3×3 and 5×5 variable sized convolutional core and every layer uses an extra 3×3 maximum pooling layer to extract more distinctive features with respect to the ones obtained from the previous layer [38].

SqueezeNet
SqueezeNet CNN network was proposed by Iandola et al. in 2016. This network was developed by improving AlexNet architecture. The difference between these two networks is that AlexNet CNN network has 240MB of parameters while SqueezeNet has 5MB of parameters. Moreover, the performance of SqueezeNet is almost good as that of AlexNet. SqueezeNet network contains Fire layers. In this layer, the number of features to be calculated is decreased by reducing the lter size to 1×1 [43]- [45]. Hence, SqueezeNet performs faster than AlexNet by decreasing the workload of neural network [46].

ResNet50 and ResNet101
ResNet50 network architecture consists of 152 layers and won the ImageNet contest in 2015 [47]. The architecture contains convolution layer, activation layer, pooling layer, and fully connected layer [48].
There are 5 convolutional blocks that contains 1×1, 3×3 and 1×1 convolutional layers in the structure of the architecture [49]. The global averaging pooling layer and the two-step sampling process in the architecture shrink the size of images [49]. In the fully connected layer, softmax is used as an activation function.

DenseNet-201
DenseNet is a CNN model that is used for classi cation problem and is a dense convolutional network with its dense connection model. The input of the architecture of DenseNet is 224×224×3 sized images as shown in Figure 10.
The dense blocks of the architecture consist of normalization layer, ReLU layer and 3×3 convolutional layer [51]. DenseNet architecture connects the layers instead of adding up as in the previous architectures like ResNet. The connection layers combine all the features at the previous layers and transfer these combined features to the next layers. While, in the other architectures, the extracted feature maps are transferred to the next layers, in DenseNet architecture, the excess feature maps are removed. Hence, DenseNet architecture prevents the retraining of these excess features in the next layers [52].

Shu eNet
Shu eNet was proposed by Zhang et al. for mobile devices with low processing power and it is a very well performed CNN model [53]. While this network architecture decreases computation cost, it preserves accuracy. Moreover, it achieves a superior performance than other architectures on ImageNet classi cation and MS COCO object recognition studies [53]. The architecture of Shu eNet is given in Table 3. Table 3 Shu eNet architecture.

Layer
Output size K size Stride Repeat Output Channels (g groups)   g=1  g=2  g=3  g=4  g=8   Image  224x224  3  3  3  3  3   Conv1  112x112  3x3  2  1  24  24  24  24  24 MaxPool 56x56 3x3 2 Stage2 28x28 As shown in Table 3, the network model contains a stack of Shu eNet units that is grouped in three stages. In every stage, the structure of every rst block starts with 2 steps. While some parameters in one stage remain the same, the output channels are double for the next stage [54], [55]. The group number g given in Table 3 [59]. This architecture is usually preferred for image classi cation [37]. In Figure 11, the architecture of AlexNet is shown.

Simulation Results And Evaluation Of Performance
In the experimental study, two different study groups were generated, and three different experimental works were performed for every group. Moreover, two different data sets were generated for every study group. In the rst study group, the images obtained from Lorenz and Rossler chaotic systems were used. In the second group, the images obtained from Lorenz and Chen chaotic systems were used. In every group, the time series images of the x, y and z state variables of the chaotic systems were classi ed separately, and their classi cation performance was evaluated.
In the data set which contains the images of the time series of Lorenz and Rossler chaotic system, 600 images of the x state variable of Lorenz chaotic system are created for the rst class and 600 images of the x state variable of Rossler chaotic system are created for the second class. By using various pretrained networks, the classi cation tests were performed on these total 1200 images. In the data set which contains the images of the time series of Lorenz and Chen chaotic system, 720 images of the x state variable of Chen chaotic system are created for the rst class and 600 images of the x state variable of Lorenz chaotic system are created for the second class. By using various pre-trained networks, the classi cation tests were performed on these total 1300 images. In the study, tests were conducted with numerous pre-trained networks and the 8 pre-trained networks with highest performance were used. These networks are Squeezenet, VGG-19, AlexNet, ResNet50, ResNet101, DenseNet201, Shu eNet, and GoogLeNet.
In the study, the classi cation of the images of the time series of totally different systems like Lorenz-Rossler and the classi cation of the images of the time series of similar systems like Lorenz -Chen is performed by utilizing deep neural networks. The purpose here is evaluation of classi cation performance of different time series images and similar time series images and compare these performance results.

Evaluation of results and performance of Lorenz -Rossler systems
In the rst experimental study of this group, the images obtained from the time series of x state variable of Lorenz and Rossler chaotic systems are classi ed. In Figure 12, the sample images of the time series of x state variable of Lorenz chaotic system are given.
In Figure 13, the sample images of the time series of x state variable of Rossler chaotic system are given.
As it is seen in Figures 12 and 13, the series of the two systems are very different from each other and this makes the performance of the classi cation to be very high. Thanks to the preferred pre-trained networks and the optimizations performed on these networks, higher classi cation performance results are observed in the study. In Table 4, the opposition matrices of the images shown in Figures 12 and 13 are given. From Table 4, the classi cation performance of the pre-trained networks is seen clearly. In Table 5, the performance of the pre-trained networks on the classi cation of images shown in Figure   12 and 13. As it is seen in Table 5, the highest classi cation performance is obtained with SqueezeNet and DenseNet201. In the second experimental study of this group, the images obtained from the time series of y state variable of Lorenz and Rossler chaotic systems are classi ed. In Figure 14, the sample images of the time series of y state variable of Lorenz chaotic system are given.
In Figure 15, the sample images of the time series of y state variable of Rossler chaotic system are given.
As it is seen in Figures 14 and 15, the series of the two systems are very different from each other and this makes the performance of the classi cation to be very high. Thanks to the preferred pre-trained networks and the optimizations performed on these networks, higher classi cation performance results are observed in the study. In Table 6, the opposition matrices of the images shown in Figures 14 and 15 are given. From Table 6, the classi cation performance of the pre-trained networks is seen clearly. In Table 7, the performance of the pre-trained networks on the classi cation of images shown in Figure  14 and 15. As it is seen in Table 7, the highest classi cation performance is obtained with SqueezeNet and DenseNet201. In the last experimental study of this group, the images obtained from the time series of z state variable of Lorenz and Rossler chaotic systems are classi ed. In Figure 16, the sample images of the time series of z state variable of Lorenz chaotic system are given.
In Figure 17, the sample images of the time series of z state variable of Rossler chaotic system are given.
As it is seen in Figures 16 and 17, the series of the two systems are very different from each other and this makes the performance of the classi cation to be very high. Thanks to the preferred pre-trained networks and the optimizations performed on these networks, higher classi cation performance results are observed in the study. In Table 8, the opposition matrices of the images shown in Figures 16 and 17 are given. From Table 8, the classi cation performance of the pre-trained networks is seen clearly. In Table 9, the performance of the pre-trained networks on the classi cation of images shown in Figure   16 and 17. As it is seen in Table 9, the highest classi cation performance is obtained with SqueezeNet and DenseNet201.

Evaluation of results and performance of Lorenz -Chen systems
In the rst experimental study of this group, the images obtained from the time series of x state variable of Lorenz and Chen chaotic systems are classi ed. In Figure 12, the sample images of the time series of x state variable of Lorenz chaotic system are given. In Figure 18, the sample images of the time series of x state variable of Chen chaotic system are given.
The time series given in Figure 12 and 18 of Lorenz and Chen systems, respectively, are close to each other since Lorenz and Chen systems are similar systems. This makes the classi cation of these signals di cult. However, high classi cation performance results are observed in the study with the help of the preferred pre-trained networks and the optimizations performed on these networks. In Table 10, the opposition matrices of the images shown in Figures 12 and 18 are given. From Table 10, the classi cation performance of the pre-trained networks is seen clearly. In Table 11, the performance of the pre-trained networks on the classi cation of images shown in Figure   12 and 18. As it is seen in Table 11, the highest classi cation performance is obtained with SqueezeNet and DenseNet201. In the second experimental study of this group, the images obtained from the time series of y state variable of Lorenz and Chen chaotic systems are classi ed. In Figure 14, the sample images of the time series of y state variable of Lorenz chaotic system are given. In Figure 19, the sample images of the time series of y state variable of Chen chaotic system are given.
The time series given in Figure 14 and 19 of Lorenz and Chen systems, respectively, are close to each other since Lorenz and Chen systems are similar systems. This makes the classi cation of these signals di cult. However, high classi cation performance results are observed in the study with the help of the preferred pre-trained networks and the optimizations performed on these networks. In Table 12, the opposition matrices of the images shown in Figures 14 and 19 are given. From Table 12, the classi cation performance of the pre-trained networks is seen clearly. In Table 13, the performance of the pre-trained networks on the classi cation of images shown in Figure   14 and 19. As it is seen in Table 13, the highest classi cation performance is obtained with SqueezeNet and DenseNet201. In the last experimental study of this group, the images obtained from the time series of z state variable of Lorenz and Chen chaotic systems are classi ed. In Figure 16, the sample images of the time series of z state variable of Lorenz chaotic system are given In Figure 20, the sample images of the time series of z state variable of Chen chaotic system are given.
The time series given in Figure 16 and 20 of Lorenz and Chen systems, respectively, are close to each other since Lorenz and Chen systems are similar systems. This makes the classi cation of these signals di cult. However, high classi cation performance results are observed in the study with the help of the preferred pre-trained networks and the optimizations performed on these networks. In Table 14, the opposition matrices of the images shown in Figures 16 and 20 are given. From Table 14, the classi cation performance of the pre-trained networks is seen clearly. GoogLeNet 160 213 20 3 In Table 15, the performance of the pre-trained networks on the classi cation of images shown in Figure   16 and 20. As it is seen in Table 15, the highest classi cation performance is obtained with SqueezeNet and DenseNet201.

Declarations
The Potential Con ict of Interest Disclosure

Figure 4
Page 26/32 The relationship between deep learning, machine learning and arti cial intelligence.

Figure 9
Training sample of the ResNet architecture with residual layers [47].

Figure 12
Sample images obtained from x coordinates of time series via Lorenz Chaotic System.

Figure 13
Sample images obtained from the x coordinates of the time series over the Rossler chaotic system.

Figure 14
Sample images obtained from y coordinates of time series over Lorenz chaotic system Figure 15 Sample images obtained from the y coordinates of the time series over the Rossler chaotic system.

Figure 16
Sample images obtained from z coordinates of time series over Lorenz chaotic system.

Figure 17
Sample images obtained from the z coordinates of the time series over the Rossler chaotic system.

Figure 18
Sample images obtained from x coordinates of time series via Chen Chaotic System.

Figure 19
Sample images obtained from y coordinates of time series over Chen chaotic system.

Figure 20
Sample images obtained from z coordinates of time series over Chen chaotic system