Numerical solution for initial and boundary value problems of high-order ordinary differential equations using Hermite neural network algorithm with improved extreme learning machine

In this paper we put forth Hermite neural network (HNN) algorithm with improved extreme learning machine (IELM) to solve initial/boundary value problems of high-order ordinary differential equation(ODEs) and high-order system of ordinary differential equations (SODEs). The model function was expressed as a sum of two terms, where the ﬁrst term contains no adjustable parameters but satisﬁes the initial/boundary conditions, the second term involved a single-layered neural network structure with IELM and Hermite basis functions to be trained. The approximate solution is presented in closed form by means of HNN algorithm, whose parameters are obtained by solving a system of linear equations utilizing IELM, which reduces the complexity of the problem. Numerical results demonstrate that the method is effec-tive and reliable for solving high-order ODEs and high-order SODEs with initial and boundary conditions. linear ODEs and higher order linear SODEs.

least human intervention [28,27,29]. Depending upon these advantages, ELM algorithm has been successfully applied to solve ordinary differential equations, partial differential equations and integro-differential equations [34,30,35,[31][32][33]. Among others, Yunlei et al. [34,35] proposed Legendre Neural Network algorithm to solve ODEs and elliptic PDEs. Hongli et al. [36] introduced Bernstein Neural Network algorithm based on ELM and Bernstein polynomial to solve directly first order ODEs, second order ODEs and partial differential equations(PDEs). Yangjin and Yanfei respectively proposed trigonometric neural network algorithm and Legendre Neural Network algorithm for solving the ruin probabilities in the classical risk model and the Erlang(2) risk model [37], [38]. Yinghao [39] proposed a numerical method of the generalized Black-Scholes differential equation using Laguerre neural network.
In this article, we propose a neural network algorithm based on Hermite improved extreme learning machine to directly solve IVPs and BVPs of high-order ODEs and high-order SODEs. In contract with the traditional methods, the approximate solution obtained from HNN algorithm is is available in the form of a continuous and differentiable function, and the algorithm does not need to reduce high-order differential equations to a system of first-order differential equations. The product of two Hermite polynomials is chosen as basis function of hidden neurons, the approximate solution and its derivatives are expressed by using Hermite network. The model function was expressed as a sum of two terms. The first term, which contains no adjustable parameters, was related to the initial/boundary conditions and the second term involved a single-layered neural network structure with extreme learning machine and Hermite basis functions to be trained. Furthermore, HNN algorithm adopts the Moore-Penrose generalized inverse operation to obtain the output weights of network, which can effectively overcomes the drawbacks of the traditional BP neural network, and have higher simulation accuracy.
The remainder of this paper is organized as follows: Section 2 discusses the Hermite neural network model. We describe the form and construction of the solution of IVPs and BVPs of high-order ODEs and high-order SODEs in Section 3. In Section 4, we present numerical experiments to exhibit the effectiveness and superiority of the HNN algorithm for solving IVPs and BVPs of high-order SODEs. Finally, concluding remarks are presented in Section 5.

The Hermite neural network model
Hermite neural network model consists of an input node, a functional expansion block based on Hermite polynomials and an output node. Figure 1 shows the architecture of Hermite neural network algorithm.
The polynomials orthogonal with respect to the weighting function e −x 2 are the Hermite polynomials [40]. They can be defined by the formula It is easy to check that H j (x) is a polynomial of degree j.

Input layer
Output layer x Hermite expansion

Hermite expansion
Hermite expansion The Hermite polynomials have orthogonality property, given by These polynomials satisfy a recursive relationship given by The Hermite basis functions are differentiable. Their derivatives are also differentiable, and satisfy Differentiate this equation again to get Parameters: (1) L: Number of the collocation points; (2) N: Number of the hidden neurons; (3) Θ : The hidden layer output matrix; (4) β k j : The output weights. Steps: Step 1: The interval [a, b] is discretized into a series of collocation points Step 2: Assume the approximate solution by using Hermite polynomials as an activation function, that is, is related to the initial or boundary conditions and ψ k (x, ω j , b j ) involved a singlelayered neural network structure.
Step 3: Calculate the partial derivative of y k (x) and substitute it into the high-order ODEs and high-order SODEs to obtain a system of linear equations.
Step 4: Solving the system of linear equations obtained in step 3 by using ELM algorithm. The samples x l , l = 0, 1, 2, ..., L will be used as the inputs of the neural networks, and training the neural network is simply equivalent to finding a least squares solution β k j of the linear system Θ k β k j = G k : Step 5: The weights β k j from hidden layer to output layer can be obtained. From the theory of Moore-Penrose generalized inverse of matrix Θ k , we have β k j = Θ † k G k , where the minimum norm least-square solution is unique and has the smallest norm among all the least-square solutions.
Step 6: We can find the output of the HNN model of high-order ODEs and high-order SODEs.
. Using these basis functions, the approximate solution to high-order ordinary differential equation can be written as where x is the input value of the Hermite neural network algorithm, ψ k (x, ω j , b j ) is the output value. ω j is the weight of the input node to the jth hidden node, b j is the threshold for the jth hidden node, β k j is the weight vector of the jth hidden node to the output node. The steps of Hermite neural network model for solving high-order ODEs and high-order SODEs are in Table 1.
For simplicity, we take ω j = 1, b j = 0 in the following discussion. So the model becomes

Formulation of the method
This section we introduce the form and construction of the solution of IVPs and BVPs of the high-order ODEs and high-order SODEs using HNN algorithm. The model function was expressed as a sum of two terms. The first term, which contains no adjustable parameters, is related to the initial or boundary conditions and the second term involved a single-layered neural network structure with extreme learning machine and Hermite basis functions to be calculated.

Formulation of the method for IVP in high-order ODEs
The general form of high-order ODEs is subject to initial conditions y (m) (a) = A m , m = 0, 1, ..., M − 1.
Corresponding HNN trial solution may be written as [19] y

Formulation of the method for BVP in high-order ODEs
Let us assume that the high-order ODEs with BVP has the following form subject to boundary conditions y (s) (a) = p s , HNN trial solution for the high-order ODEs with these boundary conditions is formulated as where Z(x) and M(x) are general form of functions used to construct y(x). The trial solution satisfies following relations The where P s n and P r n are number of permutations.

Formulation of the method for IVP in high-order SODEs
Consider the initial value problems in high-order SODEs of the following form subject to initial conditions y Corresponding HNN trial solution may be written as [19] where

Formulation of the method for BVP in high-order SODEs
Boundary value problem of high-order SODEs to be solved can be stated as follows HNN trial solution for the high-order SODEs is formulated as where Z k (x) and M(x) are general form of functions used to construct y k (x). The trial solution satisfies following relations where P s n and P r n are number of permutations.

Numerical experiments
In this section, to demonstrate the feasibility and superiority of the proposed Hermite neural network algorithm, we carry out seven numerical experiments for IVPs and BVPs of higher order linear ODEs and higher order linear SODEs. We may take the maximum absolute error (MAE) and mean square error (MSE) to measure the accuracy and efficiency of HNN algorithm. They can be defined as follows: where ∆ m l =| y(x l ) −ŷ(x l ) |, l = 0, 1, ..., L. Moreover, let y(x l ) denote the close solution of the higher order linear ODEs and higher order linear SODEs, andŷ(x l ) represent the approximate solution obtained by the proposed algorithm. In particular, our method is compared with LS-SVM [25], non-polynomial spline [41] and Runge-Kutta (R-K) method and a detailed analysis is made.

Example 1
Consider a fourth-order linear ODE with boundary conditions [25] The analytical solution is y = cos πx/2 .
Twenty-one equidistant points are used for the training points and the first eight order Hermite polynomials are used as basis functions. The approximate solution obtained by the proposed HNN algorithm is compared with the exact solution in Fig. 2 (a) and the errors are reported in Fig. 2 (b). From the obtained results, we can calculate that the MSE is approximately 9.77 × 10 −23 . Although training was performed applying just a small number of samples in the domain [−1, 1], the errors remain high accuracy, which confirms the reliability of this method. Besides, we have solved this problem by the proposed HNN algorithm with L = 20, N = 5, L = 20, N = 10 and L = 100, N = 10, the errors are reported in Tab. 9.   In Tab. 2, ten test points at unequal intervals in the domain [−1, 1] are adopted, clearly, the proposed method is compared with LS-SVM algorithm, which shows very good accuracy for solving BVPs of higher order linear ODEs. Note that the maximum absolute error is approximately 2.56 × 10 −9 in [25], the maximum absolute error is approximately 2.00 × 10 −11 by the proposed algorithm.

Example 2
Consider a eighth-order boundary value problem [41] The analytical solution is y = When 21 equidistant points are used for the training points and the first eight Hermite polynomials are used as basis functions, the approximate solution is described in Fig. 3 (a), and the error is shown in Fig. 3 Table 9 shows the MSE and MAE of example 2 with different numbers of training points and hidden neurons. We have compared HNN solution with non-polynomial spline solution in [41], and it is apparent that our proposed method outperforms the method in [41] in terms of accuracy from Tab. 3. Indeed, the accuracy of the error in [41] is O(10 −9 ), but the accuracy of the error obtained by our proposed method is O(10 −15 ) (Note that [41], the maximum absolute error is approximately 2.71 × 10 −8 ).
The HNN algorithm for IVP of third-order linear ODE has been trained with 21 equidistant points in the domain [0, 1] and eight neurons. Comparison between the true solution and the approximate solution via our proposed HNN algorithm is given in Fig. 4 (a). Plot of the error function is cited in Fig. 4 (b), where the MSE and MAE are approximately 2.79 × 10 −25 and 1.11 × 10 −12 . Nonuniform partitions of [0, 1] using ten points are used for test process. Comparison among the exact solution, HNN solution and R-K solution is shown for example 3 in Tab. 4, from which we can see that the maximum absolute error by HNN method is approximately 1.09 × 10 −12 and the maximum absolute error by R-K method is approximately 0.15 × 10 −8 . It is clear that the approximate solution obtained by HNN method is of higher accuracy compared to the solution obtained by R-K method.

Example 4
Consider a third-order linear ordinary differential equation with initial conditions [42] The analytical solution is y = e −x . The approximate solution obtained by HNN algorithm is compared with the exact solution and results are shown in Fig. 5 (a). The absolute errors for twentyone equidistant points in the domain [0, 1] are depicted in Fig. 5 (b). Furthermore, the results of the approximate solution via our proposed HNN algorithm and R-K method for ten non-equidistant test points are shown in Table 5, in which we can see that the maximum absolute error obtained by HNN method is approximately 2.57 × 10 −13 and the maximum absolute error obtained by R-K method is approximately 5.37×10 −10 . It is obvious that the proposed method is better than R-K method in terms of accuracy. Table 9 lists the MSE and MAE of example 4 with different numbers of training points and hidden neurons. It is apparent that our proposed method still has a better performance in accuracy even with fewer training points.
Finally, comparison among exact, HNN and R-K results for some testing points is shown in Table 6. The accuracy of the error obtained by R-K method is O(10 −6 ), while the accuracy of the error obtained by our proposed method is O(10 −15 ), the advantage of the HNN method can be seen particularly.   Consider system of third-order linear ordinary differential equations [13] subject to initial conditions The analytical solutions are given by The HNN models for IVP of system of third-order linear ordinary differential equations have been trained with 21 equidistant points in the domain [0, 1] and the first eight Hermite polynomials. From Fig. 7, we can easily observe that the results from the HNN models coincide with those from the exact results, moreover, the mean square error of y 1 (x) is 3.04 × 10 −15 , the mean square error of y 2 (x) is only 1.79 × 10 −14 , and the mean square error of y 3 (x) is 4.63 × 10 −15 .  Table 7 lists the absolute errors of the approximate solutions obtained by our proposed HNN models for some random test points, the accuracy of the errors obtained by our proposed method is O(10 −7 ), which shows that our method has good generalization ability. Moreover, we list the MSE and MAE of the numerical solutions obtained by HNN method when more training points are added and different numbers of hidden layer neurons are configured in Table 9. x Absolute error(y 1 (x)) Absolute error (y 2 (x)) Absolute error (y 3 (x)) 0.0915 0.11e-08 0.  Consider system of second-order linear ordinary differential equations [44] where subject to boundary conditions The analytical solutions are given by The exact solutions and the obtained solutions via our proposed HNN models are plotted in Fig. 8 (a) and (c), the error plots between analytical and HNN solutions are cited in Fig. 8 (b) and (d), where twenty-one equidistant points in the interval [0, 1] are utilized for the training process.
Moreover, Table 8 lists results of the exact solutions and the approximate solutions by our proposed HNN models for ten random testing points, the accuracy of the error of y 1 (x) is O(10 −11 ) and the accuracy of the error of y 2 (x) is O(10 −10 ). All of this said that the better generalization ability of HNN even under the conditions of limited training samples. Table 9 compares the errors of the numerical solutions obtained by HNN models when more training points are added and different numbers of hidden layer neurons are configured. From these results, it can be seen that the largest absolute errors are not beyond 10 −4 magnitude order, which show that the proposed HNN algorithm could reach a quite agreeable accuracy. Therefore, we can conclude that the proposed x Absolute error(y 1 (x)) Absolute error (y 2 (x)) 0.0915 0.08e-11 2.25e-10 0.1518 0.69e-11 1.21e-10 0.2410 1.75e-11 6.69e-10 0.3604 1.52e-11 5.94e-10 0.5267 2.27e-11 8.56e-10 0.6395 2.29e-11 5.97e-10 0.7590 1.31e-11 6.64e-10 0.8482 1.48e-11 1.16e-10 0.9134 1.41e-11 1.88e-10 0.9784 0.19e-11 3.51e-10 3.61e-10 4.62e-24 2.25e-24 3.31e-05 3.59e-12 2.77e-12 Table 9: Comparison of the errors of different example with different numbers of training points and hidden neurons.

Conclusion
In this paper, the Hermite neural network algorithm with improved extreme learning machine has been developed to solve IVPs and BVPs of higher order linear ODEs and higher order linear SODEs. Our proposed HNN model can directly solve the initial and boundary value problems of higher order linear ODEs and higher order linear SODEs with high accuracy. At the same time, this paper compares HNN model with least squares support vector machines method (LS-SVM method), non-polynomial spline method and Runge-Kutta method (R-K method). Experiments on the common and random data sets indicate that the HNN model achieves much higher accuracy, lower complexity but stronger generalization ability than existed methods. Evidently, the proposed HNN algorithm could be a good tool to solve higher order linear ODEs and higher order linear SODEs.