Multimodal Optimization with the Local Optimum Ranking 2 Algorithm

: In optimization tasks, it is interesting to achieve a set of efficient solutions instead of one single output, in the case the best solution is not suitable. Many niching methods offer a diversified response, yet some important problems are common: (1) The most interesting solutions of each local optimum are not identified. Thus, the output is the overall population of solutions, which increases the work of the designer in verifying which solution is the most interesting. (2) Existing niching algorithms tend to distribute the solutions on the most promising regions, over-populating some local optima and sub-populating others, which leads to poor optimization. To solve these challenges, a novel niching method is presented, named local optimum ranking 2 (LOR2). This sorting methodology favors the exploration of a defined number of local optima and ranks each local population by objective value within each local optimum. Thus, is performed a multi-focus exploration, with an equalized number of solutions on each local optimum, while identifying which solutions are the local apices. To exemplify its application, the LOR2 algorithm is applied in the design optimization of a metallic cantilever beam. It achieves a set of efficient and diverse design configurations, offering both performance and diversity for structural design challenges. In addition, a second experiment describes how the algorithm can be applied to segment the domain of any function, into a mesh of similar sized or custom-sized elements. Thus, it can significantly simplify metamodels and reduce their computation time.


Introduction
The design of many engineering systems is an example of the application of some of the most known global optimization algorithms (GOAs). In single-objective optimization (SOO), at each iteration, the population of solutions is ranked by the single objective value, and the best solutions are selected to be part of the next generation of the optimization process, until the algorithm converges, most often to a single point in the domain. Several niching algorithms offer the alternative of providing a diversified set of distinct and competitive responses, in what is also called multimodal optimization (MMO).
The local optimum ranking 2 (LOR2) provides a novel approach among niching algorithms, in ranking a population of solutions inside each local optimum and downgrading duplicates, allowing to identify and properly process the most competitive local optima of the design space in a multifocus approach. By downgrading replicate solutions, it allows also to have less local conglomeration, facilitating a broader exploration inside each local optimum. The LOR2 provides several metrics for the population of solutions, such as the local optimum each solution belongs to, the rank of each solution according to its objective value inside its local optimum, the level of redundancy of each solution, and which are the apices of each local optimum region. Thus, the LOR2 can be combined with GOAs in multiple ways. Besides substituting regular objective value sorting by the local rank sorting, cross-over operations can be applied only to solutions of the same local optimum, the local apices can substitute the global best vector in swarm methods, creating multiple local swarms, and many other combinations can be generated.
In optimization processes, expensive functions often have their output evaluated more frequently at the most promising regions of the domain, rather than in an overall Cartesian grid mapping. Thus, function evaluations occur most often inside a promising local optimum, or a set of local optima if the optimization is a multimodal optimization (MMO). Metamodels become an interesting option in many cases to reduce computation costs, in providing a fast approximation of the response to dismiss unpromising candidate solutions and provide promising local optimum guesses to speed up optimization. Yet, achieving response surfaces that can provide greater precision with low computation cost remains an important challenge. Several response surface models are available in public literature, such as polynomial regression (PR) [1], Kriging (KR) [2], radial basis functions (RBF) [2] among others, which are popular response surfaces applied to speed up optimization processes. Yet their reduced training times and increased accuracy often are the object of study.
The local optimum LOR2 also can be applied also to segment the domain of a function, or a response surface, with any number of coordinates into many elements of similar or custom size.

2.
Related work Mathematical or numerical optimization has started at the same time as the advent of Newton's classic mechanics and calculus, with the use of numerical optimization algorithms based on gradient. These methods have been for centuries the core of numerical optimization with application in several fields of engineering. Since the last few decades, several heuristic optimization algorithms have been created. A subgroup of these heuristics is called GOA, which works with a population of solutions. Global optimization is presently applied in many areas of engineering, especially those where a small percentage of efficiency increase can represent the economy of large financial amounts. This is the case of logistics, telecommunication, and systems in space, aerospace, automotive, energy sectors, artificial intelligence (AI), among others, with many publications available in the public domain.
The genetic algorithm (GA) is a very popular global optimization algorithm (GOA), which is inspired by the theory of evolution of living organisms that reproduce by mating. It states that beneficial mutations tend to accumulate over time, thus generating its evolution. Goldberg [3] describes the functionality of the GA. Kennedy [4] described the method called particle swarm optimization (PSO), which is based on the flocking movement of birds and the schooling of fishes. Other GOAs which are also popular, are differential evolution (DE) [5], simulated annealing (SA) [6], and cuckoo search (CS) algorithm [7], among many others.
To find multiple feasible solutions of a function, multimodal optimization (MMO) algorithms or niching methods were created, which work in conjunction with GOAs. Some well-known niching algorithms are presented:


Fitness sharing (FS) -Created by Holland [8] and improved by Goldberg and Richardson [9], this algorithm mimics how living species compete for resources on nature, which favors its broader distribution on a given geographic region. In this algorithm, the fitness value is divided by a function that integrates the presence of other solutions within a niching radius.


Clearing method (C)-In this MMO algorithm, a number of the best solutions within a niching radius have their fitness preserved. Exceeding solutions have their fitness cleared.  Crowding with redistricted tournament selection (RTS) -Created by Harik in 1995 [10], in this method, a GOA has a population POP_P which generates POP_Q. A set of N solutions randomly chosen from POP_P is selected. A single solution of POP_Q competes with the closer solution of the set, and the best solution is kept in the next generation. This process is repeated to all solutions of POP_Q.  Deterministic Crowding (DC) -In a GOA which performs crossover operations, each offspring of the generated pair competes with the closest parent. The solution with the best fitness is kept for the next generation. This method was created in 1992 by Mahfoud [11].  Species Conservation (SC) -This algorithm, created by Li et al. [11] identifies solutions that have the best objective value within a niche radius, denominated seeds. At each generation the algorithm equals the position of all non-dominant solutions to its respective seeds, while generating mutations of them with the GA. Thus, multiple local optima are identified and kept in the optimization process.
The Institute of Electrical and Electronics Engineers (IEEE) presents the world congress of computational intelligence (WCCI) [16], providing conferences on neural networks, fuzzy systems, and evolutionary computation. The congress of evolutionary computation (CEC) promotes a niching competition every year [17], where participants evaluate the efficiency of their niching algorithms by benchmarking it with a test work function suite [18]. Li et al. published in 2017 a survey on the latest progress of MMO algorithms [19], listing several of the most popular methods and some recent developments. They describe the operation of several niching methods like FS, crowding variations, DC, derating, among others. Also, they mention that other non-evolutionary algorithms, like PSO and DE, have been applied with a niching approach, and expound the challenges commonly faced in using niching algorithms while testing it on benchmark functions.
Sareni and Krahenbuhl [20] describe the operation of several niching algorithms, including FS, four types of crowding and clearing, and exemplify their application in test problems. Passaro and Starite propose a niching PSO [21], exploring several variations. Wong provides a short survey on MMO algorithms [22]. Qu Truss structures are interesting numerical models to test the performance of optimization algorithms. While it has a simple formulation, it illustrates well the challenges faced in the optimization of structural designs and other domains, like the presence of several local optima. Deb and Surendra applied GA for the weight reduction of 2D and 3D trusses [30], by seeking optimal topology and cross-section rib sizes. Deb also describes the optimization of design parameters of a class of welded structures, using GA and proving it has better performance than other tested methods [31]. Gomes and Beck [32] describe the use of the PSO algorithm in the optimization of a steel-frame transmission line tower, with the aid of a neural network as a metamodel. Wang et al. [33] describe the optimization of truss structures with nodes position displacement and cross-section variation, considering frequency constraints, and describe the challenges faced in not getting trapped in local optimum response. Kaveh [34]- [37] also applies several heuristic algorithms like ant colony, PSO, among others, in the optimization of truss structures. Azad and Hasançebi [38] propose a selfadaptive algorithm in the optimization of trusses, and Lingyun et al. [39] apply GA to optimize the shape and size of trusses with frequency constraints. Gomes [40], Perez and Behdinan [41] also optimize a truss, applying a particle swarm algorithm. Cheng [42] applied GA in the design of steel truss arch bridges.
In the field of MMO applied to the design of structures, interestingly there are fewer examples. Lin and Chen propose a multistage algorithm for multimodal optimization of structures [43]. Islam proposes a niching evolutionary algorithm to find multiple solutions of a truss structure [44]. Huang et al. propose a niching PSO [45] for the design of composite structures. Other publications also present the MMO applied to structures [46]- [48], in dealing with the challenge of providing a diverse set of efficient designs.
There are a large number of publications in SOO of structures, mainly related to industrial research and development in the field of automotive, civil engineering, space, and aerospace sectors. In order to evaluate the efficiency of more complex structural designs, finite element models (FEM) are one of the most popular type of mathematical models. This method segments the structure in several elements, and their interaction is evaluated generating a field of deformation and stress. Cook et al. offer a reference with the fundaments of FEM [49]. Ugural [50] also offers an introduction of analytical stress analysis of structures, and FEM formulation.
Liu and Tovar [51] generated a topology optimization study in MATLAB, to reduce the weight of structures. Gauchia et al. [52] performed the torsional stiffness and weight optimization of a bus structure using GA and FEM. Park and Dang [53] applied a CAD-CAE interface integration to automate the design optimization of metallic structural components, using GA and supported with metamodel to reduce computation costs. Zheng et al. [54] designed a lunar lander using FEM, using commercial software called Abaqus and LS-Dyna. These are only a few of many examples of structural design studies applying FEM and optimization algorithms.
In optimization, a metamodel, which is also called a response surface or surrogate model, can be applied to reduce computation costs. This is achieved by dismissing unpromising candidate solutions and suggesting promising alternatives. Among well-known response surface Kriging (KR) also called Gaussian process [55], polynomial regression [1], radial basis functions [2], and ANNs [56]. To evaluate the efficiency of a response surface in forecasting the output of a function, many benchmark functions were created [2].
In the design of very efficient structures, GOAs, gradient-based, and other optimization algorithms are applied in conjunction with mathematical models of the structures. Since the computation costs of the structural models are often expensive, metamodels are often applied. If the structural model is an aeronautical or spatial structure, aerodynamics models, which also are computationally expensive, are added to the structural model in a so-called multidisciplinary design optimization (MDO).
In the field of AI, Rumelhart et al. described the backpropagation method [56], which is applied to train multi-level ANNs. These ANNs are a type of metamodel, which consists of a mathematical model which mimics the flow of information observed in organic neuronal tissues of several living species, which allows many living organisms to have intelligence. This methodology has been very efficient in text, image, and speech recognition, among other applications. Among some of its generalized applications, ANNs can be applied to forecast the response of a mathematical function, what is called regression, and for category classification.
Deep learning (DL) [57], is a technique created by Lecun, Bengio, and Hinton, where many layers of ANNs are trained over data set to recognize several levels of patterns. It has unsupervised training, different than ANNs, and has many important applications for the automation of many complex and challenging tasks, substituting human work in many fields of activity with similar or greater levels of intelligence. DL and other areas of AI have as their basic building blocks response surfaces such as ANNs, KR among others metamodels. Since usual AI tasks involve large data sets, large metamodels are required, which require elevated training times. This demands large parallel computing infrastructure for their often computationally very expensive metamodel training procedures.

3.
Methodology Several niching algorithms provide options to explore both efficiency and diversity of a function. However, as mentioned, these algorithms have some important hindrances: (1) they provide as output the overall population, not identifying which solutions are the local apices, giving a large workload to designers to verify, often manually, which solution is the most suitable for the design task. Also, (2) most niching algorithms distribute the population of solutions on the locations of the domain which presented the best fitness, which can overpopulate some local optima and reduce the number of solutions exploring other promising local optima. Since a local optimum can present a poor outcome in the initial phase of the exploration, yet a competitive output if properly processed, these niching algorithms can present low performance for several functions.
The algorithm LOR2 presents some similarities to the SC algorithm, yet it differs in some important points: (1) the number of apices, called seeds in the SC, is unlimited in the SC, which can hinder localized exploration, especially for large design spaces. (2) On each generation of the SC, the non-dominant solutions have their position equalized to their respective seed, which can reduce diversity and reduce convergence speed. (3) The LOR2 presents several parameters for each solution, which allows it to be combined with many GOAs in multiple ways.
The LOR2 keeps in the optimization process all best-evaluated regions of interest of the design space, with an equalized number of solutions on each local optimum to properly process it. Also, it downgrades solutions considered redundant, thus a broader exploration is achieved inside each local region. In addition, as mentioned, it provides several metrics for the population of solutions, which allows it to be combined with GOAs in multiple ways: the rank of each solution inside each local optimum region, its redundancy level, and which solution is the local apex.
The LOR2 evaluates the solutions of a population-based GOA, in order to explore the most feasible regions of the design space. For this, the design space is normalized, where each design variable is defined between zero and one. Once each solution is evaluated by the single objective value, the population of solutions is sorted with the solution with the best objective value at first. With a normalized optimization space, is possible to calculate the distance matrix between each solution of the sorted population. Assuming a normalized optimization space with dimensions can be represented by a hypercube, the size of the greatest diagonal inside this hyper cube is given as the square root of . To have all distances in the design space between 0 and 1, the normalized distance matrix is defined as the distance matrix divided by the size of the greatest diagonal of the design space hyper-cube .
In the LOR2 the local optimum apex is a solution with the best objective value within the local optimum, and a sub solution is a point belonging to a local optimum that is not an apex. Each local optimum apex receives a local optimum rank equal to zero, and each sub solution receives the local optimum rank equal to the number of solutions non-penalized for redundancy in the same local optimum, which have better objective value. Four operational variables are defined: 1 which is the local optimum radius, 2 the redundancy radius, the number of allowed replicates , and the number of allowed local apices . If a solution does not have another solution with a better objective value within a distance smaller than 1 , it can potentially be an apex. If the number of local apices already identified by the algorithm is lower than the maximum number of allowed apices , then this solution is also considered as an apex. Since there is a limited number of allowed apices, exceeding apex candidates are defined as a sub solution of the closest apex recorded by the algorithm.
Following this, it is necessary to define the redundancy penalty of each solution. If two solutions have their distance smaller than the redundancy radius 2 , then the solution with a worse objective value has its redundancy penalty increased by one. In order to allow intensive localized exploration, a number of allowed replicates, , are allowed to each solution and are not penalized for redundancy. Once each point has its redundancy penalty and local optimum rank evaluated, a rank vector with three variables is generated, with the following sequence: redundancy penalty, local optimum rank, and objective value. This will prioritize non-redundant solutions in competitive regions of the design space, with the best objective values, promoting an equal number of solutions on each local optimum. With this, in the LOR2 sorting, the candidate solutions are sorted by the first rank variable, and then for identical values in the first variable, the second variable is the sorting parameter. The same rule is applied for the third rank variable, and thus the sorting process is accomplished. Figure 1 displays a curve with five local optima in a 2D domain.  The LOR2 is presented for a minimization task, at algorithms 1 and 2. As a variation, line 11 of Algorithm 1 can be substituted by: This will generate the local optimum ranking 2 with dominance propagation (LOR2b), where the local dominance is propagated to neighboring solutions with worse objective value, throughout the design space, thus identifying all local optima and the basins of attraction, while not considering noisy fluctuations within 1 distance. This variation has proven to be more efficient for algorithms that perform permutations such as GA, and worse for combination with swarm methods such as PSO.
With this, it is necessary to evaluate the redundancy penalty and the local optimum rank of each point, as described in algorithm 2, of LOR2 phase 2:

and select the best solution
It is important to note that the algorithm is configurated for minimization tasks. For a maximization task, the sorting at line 2 of algorithm 1 should have the greater objective value at first, and the objective value at line 25 of algorithm 2 must be multiplied by -1.
The normalized distance calculation formula is based on the Euclidean distance between two points, divided by the length of the greater diagonal of the normalized design with dimensions. Thus the matrix formulation is defined: An option to reduce computation costs is to only calculate distances as required by the algorithm and store the calculated values. In order to customize the operational parameters of the LOR2, it is possible to evaluate the maximum number of possible local optima in the design space and calculate which fraction of the domain is to be considered in the optimization.
If we call any positive integer close to , were: We define the floor function as: In a given space with dimensions, the number of possible local optima is given by the number of possible hyper-cubical subspaces which can fit in the design space, which is an approximation for the hyper-spherical sub-regions. The number of intersecting local optima that can fit in one dimension, , is given by the normalized length divided by the radius of the hyper-sphere of the local optimum. Since in the algorithm all distances are divided by the length of the greater diagonal, the radius of the hyper-sphere is multiplied by the square root of . Thus, we have: If we call the approximate quantity of possible local optima in the overall design space, we have: It is possible to see that for a design space with an increased number of coordinates, the space volume significantly increases. It is interesting that during the optimization, the exploration effort would concentrate mostly in the regions of greater interest, than in the overall domain. If we consider the volume occupied by the local optimum regions delimited by 1 , having a hyper spherical region approximated by a hyper cubical region, we have: Although the LOR2 operational parameters can vary, we define as default, or initial guess, the values following. It is considered that the local exploration can be concentrated in 10% of the volume of the design space. Thus, we have: While keeping 0.05 ≤ 1 ≤ 0.2. In addition, it is defined, 2 = 0.05 1 (eq. 8) With this, it is possible to properly define 1 and 2 according to the number of optimum regions and the number of coordinates of the design space. The number of solutions per local optimum must be enough to allow an appropriate local exploration. It is defined by the size of the population of solutions in the overall design space divided by the number of local optima, which is given by the number of allowed apices : = / (eq. 9) To allow an appropriate localized exploration, we maintain ≥ 10. The number of allowed replicates can be defined as equal to 4 for an initial guess, providing an average balance of localized and diffused exploration. Thus, the only parameter which should be defined by the user is the number of apices . Yet can also be automatically defined by the number of local optima of the function, given by the number of basins of attraction identified by one generation of the algorithm LOR2b, over a sufficiently populated domain.
The LOR2 can be applied to sort a population of solutions favoring an equal distribution on each considered local optimum of the design space. For this, the GOA working with the LOR2 sorting must be operating with aggregated population like occurs for example in the GA. In the GA, the operators of mutation and crossover are applied to an existing population POP_P, and thus another population, POP_Q, is generated. After POP_Q has its objective values defined, the two populations are added (aggregated), generating POP_R. This population POP_R, in the original configuration of the GA, is then sorted according to the single objective value, and the best solutions are kept, generating POP_P of the following iteration.
The LOR2 can be applied by substituting the regular objective value sorting with the LOR2 sorting, favoring a local optima distribution, as illustrated in figure 3. This sorting with the LOR2 is called local optimum ranking 2 population sorting process (L2PS). Some algorithms do not work with aggregated populations, but rather work with ameliorated populations, which is the case of simulated annealing and PSO. In these algorithms, the population POP_P of the following generation is made of some solutions of the existing POP_P population and some solutions of the POP_Q population, by following the particular selection process of each method. This does not allow the L2PS process to be applied, therefore, to apply a local rank sorting it is necessary to modify these algorithms to work with aggregated populations. Thus, it is generated a larger POP_R population which can then be sorted by the L2PS and generate the population POP_P of the following generation with a local optimum ranking distribution. By allowing a large or an unlimited number of local optima, the LOR2 algorithm also can be applied to generate a mesh for a populated domain with any number of coordinates, with a Cartesian, random or clustered distribution. It can generate a mesh for a 2D or 3D finite element model (FEM) [58], or to generate a response surface mesh (RSM) with any number of coordinates, among other applications. The advantages of an RSM is that segmentation allows parallel computing which can reduce training times, precision can be increased with mesh refinement, and individual mesh elements can be trained or updated individually. Also, some models like KR are significantly simplified by having its dataset partitioned, providing its great accuracy with much faster implementation.
In order to provide an efficient curve approximation inside each RSM element, each element must be sufficiently populated to provide gradient in all the dimensions of the function. For a linear approximation, the number of points inside each element needed to generate gradient must be equal to the number of dimensions of the function plus one. If the approximation is of a quadratic polynomial, as in the PR, one more point is needed. Thus, for a polynomial approximation of grade N, applied over a domain of M dimensions, the quantity needed of inputs inside each element is given by the sum of M and N. In order to provide greater precision in regions of interest by mesh refinement, the local optimum radius 1 can be a function of the position of the local optimum apex in the design space.
Once the population of local optima is defined by the LOR2, it is possible to calculate the position of the polarization vertex (PV) of each element, which is given by the average position of all inputs of the same RSM element. With all PVs defined, each polarized mesh element (PME) is generated by the points of the data set closer to each PV, as in a Dirichlet partition, which is also called Voronoi partition [59].
One challenge arising from generating a mesh over a large data set is the computing cost related to the calculation of distances between each input. A possible alternative is segmenting the data set in regions of similitude, having the care to include adjacent apices of other regions, which is also a viable alternative to generate a 2D or 3D mesh for a FEM. Another alternative is to define the PVs over a sparser data set and include all the inputs only for the evaluation of each PME function.
At figure 4 at the left side is noted the related local optimum subspaces of the function displayed at figure 1, and its local optimum apices, identified by the LOR2. At the right side at figure 4, its respective PMEs are displayed. These subspaces are defined for a limited number of local apices. The domain is segmented in five main subspaces, related to each local optimum. The local apices at left, and PVs at right, are marked with different colors.

Experiment I: Structural optimization
In order to demonstrate an application of the LOR2 algorithm, a structural design case is presented. Spacecrafts that are required to land without impact on a moon or a planet with gravity, need to have a set of stabilizing legs, called landers, to maintain the vehicle vertically oriented and facilitate lift-off. These stabilizing legs are structures that consist of a curved cantilever beam, which are required to have low weight, in order to avoid fuel consumption, and are also required to have limited deformation under loading. Achieving a near optimum design configuration for this structure can reduce the elevated costs associated with moon exploration or interplanetary missions and is an interesting optimization problem. This optimization problem is here used to demonstrate the capacity of the LOR2 algorithm in achieving as an optimization outcome a multiple set of efficient design configurations, providing both efficiency and diversity for SOO.
In order to derive the FEM formulation for the structure, we calculate the energy balance of the system, expressed in terms of displacement functions, according to Ugural [50]. This is given by the variation of elastic energy subtracted by the work performed by field and surface forces.
If the entire structure is divided in elements, the elastic potential energy increased on each element is given by the integration on the volume on each element of the generated strain deformation ∆ multiplied by its corresponding tension . The variation of elastic potential energy for the entire system is given, Similarly, the amount of work performed by field forces and surface forces on the structure can be defined as: = 0 (eq.14) Using a simplified notation for each element we define:  In order to calculate the deformations and tensions of the beam, a 2D FEM with triangular element mesh was generated in MATLAB ( figure  7). The optimization algorithm, the mesh generation code, and the FEM processing core were independently coded in MATLAB, thus no MATLAB builtin applications we applied to any of these design steps. The material coefficients used for the model are from the aluminum alloy 2024-T4 [60]. As boundary conditions, the beam is fixed on left side, and a vertical load is placed on the right side of the structure. Under these conditions, the structure must have all nodal displacements limited to a maximum allowed value of 0.05 m. Along all the structure it is necessary to maintain maximum Von Misses principal tension 1 lower than the maximum tension allowed, 300 MPa.
For this optimization task, the GA with segregated population identified by the LOR2b and LOR2b population sorting (L2bGA-L2bPS) algorithm is selected, where cross-over operations occur only inside each local optima. The goal of the optimization is to generate a diverse set of very efficient design configurations which minimizes the overall structural weight, while maintaining deformations and maximum tensions under defined values. At figure 1 and 2, previously presented, it is seen how the LOR2 algorithm can explore in parallel multiple local optima of a given function. Thus, similarly, the LOR2b algorithm is in this experiment applied for structural optimization, to generate a set of multiple local optima design configurations for the beam, providing both efficiency and diversity for a SOO.
The number of generations of the optimization is set to 800. Part of the operational parameters of the algorithms are presented in table 1, and additional details are provided in Appendix A. Using a computer with quad-core processor AMD A8-7410 APU with 2.2GHz, this optimization task took approximately two days to be completed. Since the algorithm is multi-focus, the intensive localized exploration one each local region is reduced when compared to a regular single modal optimization (SMO) task, thus it requires more computation time.  It is possible to see that the primary apex in light blue has a good convergion without fluctuation, while apices 2, 3, and 4 present some fluctuation along the generations. This is because when an apex optimizes, it dislocates its position inside the design space, and when it comes closer to an existing apex with lower objective value, this lower leader becomes a sub solution of the leader with better objective value. Thus, some apices can be suppressed along the optimization, and other further solutions with lower objective value are considered other local apices. Since some apices can be suppressed and not have other solutions to replace them, it is interesting to record the last population of solutions along the generations which presented all the allowed apices, to assure the desired diversity and efficiency output for the optimization task.  These solutions displayed in blue are the best local optima configurations, the apices of each local optima, which are the 4 most competitive solutions in terms of weight efficiency while having significant distance between each other in the design space. This distance between each leader is of at least 1 in the normalized design space, which translates into different design configurations for the structure. This illustrates that a single optimization study using a GOA combined with LOR2 algorithm, with a combination that has segregated populations, can generate distinct local apices designs with competitive performance, and several sub solutions. This multi-focus approach provides diversified solutions and also reduces the probability of the optimization process getting trapped in a poor local optimum, as can happen in the single modal optimization. With these mentioned advantages, the LOR2 algorithm is expected to give important contributions to several industrial process of advanced design. Figure 11 displays the field distribution of principal stress 1 measured in Pascals (Pa). It is possible to see that all maximum stresses are less than 300 MPa, as a design requirement. As expected, the greater concentration of stress occurs near thin regions of the structure.  The optimization of a cantilever beam under loading is performed with the L2GA-L2PS algorithm, a combination of the GA and LOR2, having a GA with segregated population -which means the cross-over process occurs only between solutions of the same local optimum region. As expected, the optimization task provided 4 different efficient designs as output, due to its multi-focus approach. With this multi-focus approach, it is generated a set of efficient configurations, which provides options for the design process, and a lower probability of having the optimization process getting trapped in poor local optima. The graphical analysis of structure profiles demonstrates the designs indeed differ, having several sub solutions, and the plots of 1 stresses and nodal displacements confirming the designs are under the specified requirements.
This optimization experiment showcases the advantages of the multi-focus approach of the LOR2 algorithm when compared to single-focus methods. By finding multiple near-optimum solutions, the optimization has a lower probability of getting trapped in a poor local optimum, and the most interesting design variations are presented. As the results demonstrate, it is expected that the LOR2 algorithm can provide important contributions to several design challenges of SOO.

Experiment II: Metamodel segmentation
Another application of the LOR2 algorithm is to segment response surfaces. In order to compare the efficiency of segmented and non-segmented response surfaces, several benchmark functions used for this purpose are selected [2]. The metamodel applied in this experiment is the model called Kriging (KR), also called Gauss Process, which is a popular metamodel in AI and optimization used to forecast the output of a function, well known for being very precise. Yet it is also very expensive in terms of computation costs for large datasets. Each function has its 2D domain defined in a Cartesian grid of 71 x 71 points, and the error of the output and other performance parameters are presented. Tables 7 to 10 display the comparison in terms of performance of the metamodel and time delayed on training. The metrics for metamodel performance measurement are the average error and variance of the error. Other metrics to measure the performance of metamodels are also applied, which are R square, RMAE, and RAAE, as described by Mahdi et al. [20]. It is important to note that for the response surface mesh (RSM), each element of the metamodel was training in Table 5 -Performance comparison for the Custom Probability Density Function From table 2 to 5 is possible to see that the training time of the KR model, which ranged from 27 to 7 minutes, was significantly reduced with the generation of the RSM with the LOR2. The RSM-KR presented a total computation time of about 11 to 13 seconds, a time reduction up to more than 120 times. These results were achieved with local optima generation with 1 equal to 0.1. Experiments have demonstrated that for a domain with an increased number of inputs, the training time for the non-segmented model can achieve many hours, while the segmented mesh remains less than 20 seconds for the tested cases. Also, it is noted that there is not much difference in terms of the precision of the RSM, which demonstrates that the LOR2 can be applied to generate a RSM, and is an important alternative to reduce computation costs of metamodels.

6.
Concluding Remarks GOAs presented a great advantage when compared with gradient-based optimization since these methods allow a better exploration of the overall design space. However, most GOAs converge to a single point in the design space, what can bring large variance values for the output, since many functions present several local optima. Thus, many niching methods were created to achieve a diversified set of solutions and identify the diverse local optima. The LOR2 presents important advantages before existing niching methods, like the identification of the most suitable and interesting solutions among all the population, and an equalized exploration effort on each local optimum region. Also, it has a great versatility in giving many ways to be combined with GOAs, since it provides several metrics of the population of solutions, which generates very efficient alternative methods.
It is also important to note that the experiments here described considered only the application of the LOR2 with the parallel exploration of selected local optima. Yet for many functions is possible to use the LOR2 to discretize the domain by listing and locating each local optimum that presented the most competitive responses. This can facilitate the methodical thorough exploration of the function and provide results with increased efficiency than those presented in this study.
The LOR2 demonstrated its capacity of achieving distinct and efficient design configuration in the multimodal design optimization of a metallic cantilever beam. Since the four main solutions are identified, it signifies an important advancement in MMO, by avoiding the need to verify manually a large number of possible solutions as the optimization output.
Metamodels such as KR, which is very popular for its precision yet very expensive, are significantly simplified and have their computation cost greatly reduced when partitioned. As the experiment demonstrate, the LOR2 is a good alternative to segment a dataset into regions of similitude. This LOR2 metamodel partitioning can be applied to any metamodel, and response surface elements can be trained or updated independently, and datasets size and precision can be significantly increased. Thus, the LOR2 provides increased operational speed and efficiency to all AI and metamodel aided optimization tasks in general. It is interesting to note that the LOR2 RSM generation was coded in MATLAB for these experiments, and the mesh generation was responsible for 95% of the computation time of the RSM. Yet it can perform significantly faster if coded in a faster computation language such C++ or JAVA.
Given the versatility and efficiency of the LOR2, it is expected that the LOR2 will benefit several advanced design tasks and optimization challenges in general, especially for functions with large domains that present multiple global optima. Also, it can help in reducing computation costs of AI tasks in general, significantly improving the performance of many metamodel assisted optimization and AI tasks.  The GA operates with the random pair selection. In the L2GA-L2PS, the pair selection is random selection, yet it occurs only inside the same local optimum. Experiments have demonstrated that generating individuals either mutated or recombined provides increased efficiency than generating individual solutions both mutated and recombined. In this study, the selected ratio of the mutated population size from the overall generated population is 0.5, as described in table A1.