Data simulation of optimal model for numerical solution of differential equations based on deep learning and genetic algorithm

doi:10.21203/rs.3.rs-2578168/v1

Download PDF

Research Article

Data simulation of optimal model for numerical solution of differential equations based on deep learning and genetic algorithm

https://doi.org/10.21203/rs.3.rs-2578168/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 06 May, 2023

Read the published version in Soft Computing →

You are reading this latest preprint version

Calculus equation is an important tool for mathematical research and plays an important role in most natural science research. Since the beginning of the 18th century, people have gradually used differential and integral equations to solve physical problems. In general, several different aspects of differential equations in the field of mathematics are concerned and studied by most scholars. However, this paper studies and establishes the optimal model for numerical solution of differential equations through deep learning and genetic algorithm. In this paper, the solution of ordinary differential equations is solved through the use of polynomial function space, while the linear combination of simple function x and its product nx can obtain multinomial function space. The space function form of polynomial is very simple, and the operation ability is very strong. Almost all functions can be approximated, and the function space can be transformed by a simple function. Through data simulation test results, it can be found that the oscillation of neural network output is stronger and stronger with the increase of depth, that is to say, the deeper depth endows the neural network with stronger oscillation properties, so for the oscillation function, the depth neural network fitting effect is better than the shallow neural network. Therefore, by combining deep learning and genetic algorithm, this paper studies and establishes the optimal model for numerical solution of differential equations, and finds that the deep neural network can largely complete data simulation.

deep learning

Genetic algorithm

Differential equation numerical solution optimal model

Data simulation

With the development of the real world, scientific theory and technology, there are more and more applications that use differential equations to build models to discover the laws of the real world [1]. By solving differential equations, we can explore the phenomena that people are interested in many scientific and industrial applications. The computer industry has developed rapidly in recent years. The development of scientific computing ability and computer's ability to store data has greatly improved. And then aroused the wave of deep learning research [2]. So far, there are many deep learning methods for solving and constructing differential equations, but there are still some deficiencies and defects. Some equations also need the analytical solution form of differential equations as a priori information, which limits the scope of its application [3]. Therefore, the problem of solving and constructing differential equations still needs to be studied in depth. Deep neural network is a new kind of scientific research tool, which involves many disciplines and has a very wide application background [4]. Since Newton studied calculus, most problems in science and engineering have been solved slowly in this mathematical form, but for some complex problems, differential equations are usually difficult to analyze and solve.

The paper discusses the influence of ReLU, Sin and Sigmaid functions as activation functions on the function expression ability of neural networks, and proves the approximation principle of neural networks based on different activation functions [8]. In the literature, a general solution framework, called genetic algorithm (GA), which is used to optimize practical application problems in complex systems and does not depend on specific problem domains, is introduced, and has strong robustness. Therefore, this algorithm has been widely used in many disciplines. With the help of biological incentive mechanism, the algorithm improves the characteristics of chromosomes participating in natural competition with the continuous evolution of the population [9]. Moreover, the algorithm also uses the population search technology. When individuals continuously breed enough individuals to form a population, each population individual is a potential solution to the problem. The fitness function can reflect the advantages and disadvantages of the population individual [10]. The larger (smaller) the function, the closer to the optimal solution. The optimal approximate solution of the problem is generated after the population has experienced several generations of evolution. Only by comprehensively analyzing the information of the objective function in the evolution process, that is, the information of the number of individuals in the population, can the optimal solution of the population function be found [11]. Differential equations can be used to solve problems in natural science and engineering technology, but it is difficult to solve other problems. The literature uses the principle of least square method to transform the problem of solving differential equations into the problem of finding the minimum value of functions. The feasibility of the algorithm is verified by simulation experiments for solving ordinary differential equations and partial differential equations [12]. In the literature, the solution and construction of differential equations and the characteristics of its related neural networks are introduced in detail, and the research methods and construction of solving differential equations using neural networks are also introduced [13]. Because it is sometimes relatively difficult to solve the forward and reverse problems of some equations, for example, the adaptability of the region shape will be greatly reduced through the orthogonal network of the finite difference method, and the workload of the finite element method restricts the application in practical problems [14]. The rapid development of computing and storage capabilities provides new opportunities for the study of differential equations. The literature proposed two improved algorithms based on the Physics Informed deep neural network: differential equation solving and differential equation construction algorithms, and constructed the accurate prediction differential equation dynamic model and revealed the hidden partial differential equation model [15]. Compared with the existing methods, the improved differential equation solving algorithm in this paper is more flexible. The model does not require the solution of the differential equation as a priori information, and the error is small. The differential equation construction algorithm can solve the inverse problem of partial differential equations with variable parameters [16]. In the literature, algorithms for solving and constructing differential equations are proposed The architecture of differential equation solving algorithm is based on Tensor fl ow and Pytorch to solve differential equations [17]. Differential equation construction algorithm is used to solve the inverse problem of partial differential equations The two algorithms mainly study one-dimensional Burgers equation, one-dimensional advection equation and hyperbolic conservation partial differential equation, and obtain specific experimental results.

3.1 Deep learning method

Deep learning (DL) processes multi-level information, extracts and combines low-level features and forms high-level features.

The main function of artificial neural network is to realize the interaction between biological neural system and real objects. In biological neural network, when other n neurons input signals to the neuron, the signals will be transmitted through the neuron. Compare the total input value received by the neuron with the excitability threshold of the neuron, and use the "activation function" to process the output value generated by the neuron after processing the input signal.

Taking the feedforward neural network with a single hidden layer as an example, for a given set of input vectors X={xa, xb,..., xn}, the jth output of the neural network is:

$${y}^{j}=\sigma \left(\sum _{k=a}^{n} {v}^{kj}{b}_{k}-{\theta }_{j}\right)$$

Including:

$${b}_{k}=\sigma \left(\sum _{i=a}^{n} {\omega }^{ik}{x}_{i}-{u}_{k}\right)$$

In neural networks, sum and difference of activation functions Ϭ The essence of (·) is the rule that neurons receive input signals and generate output signals. Common activation functions are:

Sigmaid activation function:

$$g\left(x\right)=\frac{1}{1+{e}^{-x}}$$

Hyperbolic tangent activation function:

$$\text{t}\text{a}\text{n}\text{h}\left(x\right)=\frac{\text{s}\text{i}\text{n}\text{h}\left(x\right)}{\text{c}\text{o}\text{s}\text{h}\left(x\right)}=\frac{{e}^{x}-{e}^{-x}}{{e}^{x}+{e}^{-x}}$$

SoftPlus activation function:

$$f\left(x\right)=\text{l}\text{n}\left(1+{e}^{x}\right)$$

This paper proposes a method to solve high-dimensional PDEs. Its basic idea and algorithm steps are: (i) PDEs are constructed by BSDEs through nonlinear Feynman Kac formula (ii) The gradient operator of the function u to be solved is regarded as a policy function, and the solution of BSDE problem is regarded as a stochastic control (SC) problem, which can be regarded as a DL problem (iii) A DNN is used to approximate the high-dimensional strategy function, and DL method is used for training.

In order to strengthen the connection between PDEs and BSDEs, the method needs to set the terminal conditions in advance. The proposed method is called depth backward stochastic differential equation (BSDE) method. The studied partial differential equations are limited to the nonlinear parabolic PDEs defined by the following equation.

$$\begin{array}{c}{Y}_{t}=g\left(\xi +{W}_{T}\right)+{\int }_{t}^{T} f\left({Y}_{s},\left({\nabla }_{x}u\right)\left(s,\xi +{W}_{s}\right)\right)ds\\ -{\int }_{t}^{T} {⟨\left({\nabla }_{x}u\right)\left(s,\xi +{W}_{s}\right),d{W}_{s}⟩}_{{{\Omega }}^{d}}.\end{array}$$

For any t1, t2 ∈ [0, T] and t1 ≤ t2, the following equation holds almost everywhere:

$$\begin{array}{c}{Y}_{{t}_{2}}={Y}_{{t}_{1}}-{\int }_{{t}_{1}}^{{t}_{2}} f\left({Y}_{s},\left({\nabla }_{x}u\right)\left(s,\xi +{W}_{s}\right)\right)ds\\ +{\int }_{{t}_{1}}^{{t}_{2}} {⟨\left({\nabla }_{x}u\right)\left(s,\xi +{W}_{s}\right),d{W}_{s}⟩}_{{\mathfrak{R}}^{d}}\end{array}$$

When N ∈ N is large enough, it can be obtained from the above equation:

$$\begin{array}{c}{Y}_{{t}_{n+1}}\approx {Y}_{{t}_{n}}-f\left({Y}_{{t}_{n}},\left({\nabla }_{x}u\right)\left({t}_{n},\xi +{W}_{{t}_{n}}\right)\right)\left({t}_{n+1}-{t}_{n}\right)\\ +{⟨\left({\nabla }_{x}u\right)\left({t}_{n},\xi +{W}_{{t}_{n}}\right),\left({W}_{{t}_{n+1}}-{W}_{{t}_{n}}\right)⟩}_{{\mathfrak{R}}^{d}}.\end{array}$$

The above formula is the forward discretization result of BSDE

3.2 Genetic algorithm model

Genetic algorithm is an adaptive random search method, which can complete the evolutionary computation of optimization problems. The search method is determined by the information provided by the population; At the same time, by setting individuals in the population as potential possible solutions, the global optimal solution can be obtained with greater probability by using the algorithm and convergence. This algorithm can be used to solve many complex and difficult problems. Since De Jong, function optimization has become a method to deal with common examples in classical application fields and genetic algorithms.

In the subsequent breeding, genetic algorithm provides both crossover operator and mutation operator. Two randomly selected codes can be exchanged by the crossover operator, while the mutation operator can be directly converted into a randomly selected code. Through this population breeding and natural selection law, a new population code is formed. Repeat the above natural selection and breeding until the conditions are met. The practical results show that the algorithm has a high computational efficiency and a wide range of applications. Holland proposed a schema theorem for this purpose. In addition, Holland also found that implicitness and computation are the characteristics of the algorithm. The data after continuous experimental tests show that if the above algorithm is properly improved, a global optimal solution with a probability of convergence of 1 will be generated for any optimization problem.

The nature of the target function, such as the function is deceptive and does not meet the module assumption, leads to other algorithms can not solve well. Therefore, the genetic algorithm is constantly improved. For example, dynamic coding scheme and real coding scheme are proposed for the original long binary coding scheme. In view of the proportional selection mechanism, propose improvement plans such as competition and sequential selection. Through a lot of practice, two-point, multipoint and uniform crossover operators are proposed by scientists on the basis of the original one point crossover operator. At the same time, degenerate genetic algorithm and adaptive genetic algorithm are introduced for the first time to solve the problem of parameter invariance in the process of algorithm control evolution. He also developed distributed and parallel genetic algorithms to solve various problems.

In the fitness function, it is difficult to achieve a non negative probability for individual roulette selection, and the distribution difference of several function values to be solved is large. Therefore, the average fitness obtained does not reflect the average performance of the individual population well, thus affecting the performance of the algorithm. The minimum values can be converted as follows:

$$Fit\left(f\right(x\left)\right)=\left\{\begin{array}{c}{c}_{max}-f\left(x\right)\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}f\left(x\right)<{c}_{max}\\ 0\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\text{ }\text{o}\text{t}\text{h}\text{e}\text{r}\text{ }\end{array}\right.$$

For the problem of finding the maximum, make the following conversion:

$$Fit\left(f\right(x\left)\right)=\left\{\begin{array}{c}f\left(x\right)+{c}_{min}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}f\left(x\right)>{c}_{\text{min }}\\ 0\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\text{ }\text{o}\text{t}\text{h}\text{e}\text{r}\text{ }\end{array}\right.$$

4.1 Properties of differential equation

The random increment following the normal distribution 𝑁 (0, ℎ) has the following characteristics:

$$E\left[{\int }_{{t}_{0}}^{T} g\left(X\right(t\left)\right)dW\left(t\right)\right]=0$$

$$E\left[{\left|{\int }_{{t}_{0}}^{T} g\left(X\right(t\left)\right)dW\left(t\right)\right|}^{2}\right]={\int }_{{t}_{0}}^{T} E\left[\left|g\right(X\left(t\right)){|}^{2}\right]dt$$

4.2 Construction of optimal model for numerical solution

Hamilton Jacobi Bellman (HJB) equation in dynamic programming is a high-dimensional nonlinear partial differential equation. This section uses the proposed method to solve the 100 dimensional HJB equation, which is as follows:

$$u(T,x)=g\left(x\right).$$

And:

$$\frac{\partial u}{\partial t}(t,x)+\left({{\Delta }}_{x}u\right)(t,x)={∥\left({\nabla }_{x}u\right)(t,x)∥}_{{\mathfrak{R}}^{d}}^{2}$$

As shown in Table 1, it represents the numerical experiment results using Adam optimizer. When iteration step m reaches 2000, the loss function meets the requirements and the numerical solution is output.

Table 1

Results of numerical experiment of HJB equation with Adam optimizer
Iteration steps m	0	700	1400	2100	2800
Objective function UѲm average	0.386123	2.399754	4.169131	4.782336	4.828425
Loss function J(Ѳ m) Average	18.000523	2.227252	0.43632	0.023735	0.023028
Operation time (s)	-	37	64	91	118

It can be seen from the data in Table 1 that as the number of iteration steps increases from 0 to 2800, the average value of the objective function increases from 0.386123 to 4.828425, and the average value of the loss function decreases from 18.000523 to 0.023028, but the corresponding operation time also increases.

As shown in Table 2, it is the value of HJB equation when the new parameter optimization method is adopted.

Table 2

Table of numerical experiment results of HJB equation with new parameter optimization method
Iteration steps m	0	700	1400	2100	2800
Objective function UѲm average	0.792951	2.9019	4.542506	4.785456	4.8279
Loss function J(Ѳ m) Average	14.605812	1.405213	0.082719	0.022119	0.021614
Operation time (s)	-	46	78	111	143

The mathematical models in scientific research and engineering application are mostly ordinary differential equations and differential equations, but few special differential equations can be solved correctly, and it is difficult to obtain analytic expressions in most cases. Therefore, approximate solutions can be obtained by numerical approximation. Euler method, modified Euler method and Runge Kutta method are the numerical calculation methods with high frequency. Due to the existence of various complex factors, although people have studied for many years, there is still no one to solve complex and treatable functions, and there are also good results of numerical methods. Especially when large-scale problems occur, the search space for computational optimization expands rapidly. Therefore, the difficulty and the minimum possibility of finding the optimal solution strictly have also been gradually discovered, so the general algorithm for obtaining the optimal solution of numerical function approximation with high accuracy in a short time is urgently needed to be studied.

In this chapter, the solution of ordinary differential equations will be realized by genetic algorithm. The problem of solving differential equations will be transformed into the optimization problem of solving the minimum value of functions by using the principle of least square method to solve the optimization problem. Therefore, genetic algorithms can analyze and solve many nonlinear, multi model and multi-objective problems, and the most important is non differentiable or multimodal optimization problems. In order to transform the ordinary differential equation problem into an optimization problem, the analytical solution of the constructed equation must be solved first. This is why the Taylor extension of a function of one variable is the key to converting the solution of the equation constructed. Therefore, we use the space of polynomial functions to construct the solutions of ordinary differential equations. Since the function form of polynomial function space is very simple, and its essence is just a feature of the function space generated by a simple function (this feature is very important because it has the advantage of being convenient for computer implementation), all the more important functions are infinitely approximated by this function space, so we can also think that the function representation ability of this function space is very strong, so people often use this function space.

The boundary conditions are allowed to be treated in the penalty function. The following is to optimize the problem by solving ordinary differential equations, and transform it into a specific realization process: finding the least squares solution in the polynomial function space is the simplest method to approximate the solution of ordinary differential equations with polynomial functions. The specific implementation method is as follows: select N equidistant nodes xa, xb,.... In [a, b], Xn, the following functions are constructed:

$$E=\sum _{j=0}^{N-1} f{\left({x}_{j},M\left({x}_{j},{\omega }_{0},{\omega }_{1},\dots ,{\omega }_{k}\right),\dots ,{M}^{\left(n\right)}\left({x}_{j},{\omega }_{0},{\omega }_{1},\dots ,{\omega }_{k}\right)\right)}^{2}$$

The function of penalty function construction is to transform unconstrained optimization problems into each other by dealing with boundary conditions. The following formula is penalty function construction:

$$P\left(M\right)=\sum _{k=0}^{n} {g}_{k}{ }^{2}{\left(x,M,{M}^{{\prime }},\dots ,{M}^{(n-1)}\right)}_{\text{t}\text{x}={t}_{k}}$$

First of all, use the toolbox of genetic algorithm to compile M file, which receives the line vector and then returns it with scalar quantity. The optimal process of screening experimental parameters for this problem is shown in the following process:

Set the number of variables in the problem to 3, 5, 7, 9 in the MATLAB genetic algorithm toolbox, that is, construct the solutions of polynomial approximation equations of degree 2, 3, 4, and 8 respectively. The experimental results are shown in Fig. 1:

The experimental results are shown in Fig. 2:

4.3 Analysis of data simulation results

Figure 3 shows that in MATLAB software, the improved Euler method and trapezoidal Euler Maruyama method are used to compare the numerical solution with the accurate solution:

As shown in Fig. 3, the improved Euler method is used to compare the exact solution and numerical solution of the equation. The numerical fluctuation of the refined solution value is large, and the numerical solution is relatively stable, but the trend between the two is roughly the same. When the t value reaches 0.125, the X value of both has dropped to the lowest value. After data analysis, the average absolute error after comparison using this method is 1.159 × 10^− 4。

The average absolute errors of trapezoidal Euler Maruyama method and improved Euler method are 1.159 respectively × 10 − 4 and 5.637 × 10 − 5。 Therefore, according to the average absolute error obtained by two different algorithms, the numerical solution obtained by the improved Euler method is close to the exact solution.

As can be seen in Fig. 4, the oscillation of the neural network output becomes stronger and stronger with the increase of depth, that is, the deeper the depth gives the neural network a stronger oscillation property.

Figure 5 shows the result of the change in the value of the loss function.

Before the 18th century, scientists put forward differential equations and calculus together. Differential equations are one of the main bridges connecting natural, social and mathematical sciences. They are also a highly practical basic tool for expressing the basic laws in natural and social sciences and solving various practical problems in nature and society. Therefore, the basic rules of various practical problems can be summarized and elaborated by differential equations, In other words, consistent with the background of the actual problem, the best solution (numerical or qualitative) of the problem obtained by listing the differential equation can solve the problems such as prediction devices and conditions that people design to anticipate certain "activities".

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interests

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Data Availability

Data will be made available on request.

Berezansky L, Braverman E, Idels L (2010) Nicholson’s blowflies differential equations revisited: main results and open problems. Appl Math Model 34(6):1405–1417
Shen C (2018) A transdisciplinary review of deep learning research and its relevance for water resources scientists. Water Resour Res 54(11):8558–8593
Liu H, Lang B (2019) Machine learning and deep learning methods for intrusion detection systems: A survey. Appl Sci 9(20):4396
Samek W, Binder A, Montavon G et al (2016) Evaluating the visualization of what a deep neural network has learned. IEEE Trans neural networks Learn Syst 28(11):2660–2673
Bouwmans T, Javed S, Sultana M, Jung SK (2019) Deep neural network concepts for background subtraction: A systematic review and comparative evaluation. Neural Netw 117:8–66
Sun H, Chang A, Zhang Y, Chen W (2019) “A review on variable-order fractional differential equations: mathematical foundations, physical models, numerical methods and applications. 22:27–59” Fractional Calculus and Applied Analysis1
Mo Y, Ling L, Zeng D (2022) Data-driven vector soliton solutions of coupled nonlinear Schrödinger equation using a deep learning algorithm. Phys Lett A 421:127739
Sibi P, Jones SA, Siddarth P (2013) Analysis of different activation functions using back propagation neural networks. J theoretical Appl Inform Technol 47(3):1264–1268
Katoch S, Chauhan SS, Kumar V (2021) A review on genetic algorithm: past, present, and future. Multimedia Tools and Applications 80(5):8091–8126
Gatenby RA, Brown J (2017) “Mutations, evolution and the central role of a self-defined fitness function in the initiation and progression of cancer,” Biochimica et Biophysica Acta (BBA)-Reviews on Cancer, vol. 1867, no. 2, pp. 162–166,
Jensen MC (2010) Value maximization, stakeholder theory, and the corporate objective function. J Appl Corp finance 22(1):32–42
Zadeh KS (2011) An integro-partial differential equation for modeling biofluids flow in fractured biomaterials. J Theor Biol 273(1):72–79
Dhage BC, Lakshmikantham V (2010) Basic results on hybrid differential equations. Nonlinear Anal Hybrid Syst 4(3):414–424
Wang H, Basu TS (2012) “A fast finite difference method for two-dimensional space-fractional diffusion equations,”SIAM Journal on Scientific Computing, vol. 34, no. 5, pp. A2444-A2458,
Bau D, Zhu JY, Strobelt H et al (2020) “Understanding the role of individual units in a deep neural network,” Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30071–30078,
Odibat Z, Momani S, Xu H (2010) A reliable algorithm of homotopy analysis method for solving nonlinear fractional differential equations. Appl Math Model 34(3):593–600
Lu L, Meng X, Mao Z, Karniadakis GE (2021) DeepXDE: A deep learning library for solving differential equations. SIAM Rev 63(1):208–228

Download PDF

Journal Publication

published 06 May, 2023

Read the published version in Soft Computing →

Reviewers agreed at journal
05 Mar, 2023
Editor assigned by journal
13 Feb, 2023
First submitted to journal
11 Feb, 2023

You are reading this latest preprint version

Data simulation of optimal model for numerical solution of differential equations based on deep learning and genetic algorithm

Status:

Journal Publication

Version 1

Abstract

Figures

1 Introduction

2 Related Work

3 Deep Learning And Genetic Algorithm Research

3.1 Deep learning method

3.2 Genetic algorithm model

4 Simulation Analysis Of Differential Equation Numerical Solution Optimal Model Data

4.1 Properties of differential equation

4.2 Construction of optimal model for numerical solution

4.3 Analysis of data simulation results

5 Conclusion

Declarations

References

Status:

Journal Publication

Version 1