Deep Reinforcement Learning and Performance Evaluation of Multi-layer Clustering Network with Enhanced Threshold Protocol

In this research, pure deterministic system has been established by a new Distributed Energy Efﬁcient Clustering Protocol with Enhanced Threshold (DEECET) by clustering sensor nodes to originate the Wireless Sensor Network. The DEECET is very dynamic, highly distributive, self-confessed and much energy efﬁcient as compared to most of the other existing protocols. The MATLAB simulation provides aim proved result by means of energy dissipation being emulated in the networks lifespan for homogeneous as well as heterogeneous sensor network, which when contrasted for other traditional protocols. An enhanced result has been obtained for equitable energy dissipation for systematized networks using DEECET.


Introduction
WSN is mostly recognized as an area, having small shape and simple sensing hardware devices called as wireless sensing nodes. These sensor nodes can senses the area and collects information data from these fields and transferred via wireless links. Data gathered is delivered through different hops towards sink (the base station and also called as controller) being used had locally as well as in the different networks [1]. Hence, creating the model for these type networks required full focus on the power consumption and network lifespan. Low cost and low power consumption are the basic requirement of sensor nodes . Therefore these hardware have so many constraints specially data gathering. Hence it is a necessary to search effective protocol for data aggregation [2]. From all these issues, capacity of batteries is very important parameter in the construction of wireless sensor network models. The substitution of set of batteries is difficult, when the nodes are connected. Every sensor has limited power of battery and accordingly, the battery reinstatement cells are not possible. The constraint of energy is tough to be solved in less time because of the less progress in the battery capacity development. Thus, the only method to follow is the searching of an algorithm that may provides more power for a longer life span of network [3]. Normal wireless sensor network contains sensor nodes in the range of five hundred to one thousand. Efficient energy sensor may improve the life span of sensor network up to large extent. Clustering can be executed on the basis of two major aspects, sensor nodes co-ordinates and energy levels [4]. Number of clustering methods are accessible that can lessen the dissipation of energy with the well known LEACH (low energy adaptive clustering hierarchy). Varied studies of clustering are there with respect to WSN. The formation of leaders among the sensor nodes is known as clustering. Cluster heads (CHs) are selected, than they collect the data from the clustering members. The collected information can be refined by utilizing techniques of data compression and the integrated data is transferred towards the "Base Station" (BS). The main energy consuming task is cluster-head formation [5]. The fixed cluster heads consume more energy as compared to rotating cluster-heads.To conclude the most thing to keep in mind during distributive wireless sensor networks protocol design is energy dissipation. In previous work the cluster-heads are rotated randomly therefore the optimum election is not guaranteed. But in this case we used an DEEC-ET technique that based on a residual energy for the nodes for the optimum selection of CHs.

Clustering Protocols
There are various clustering protocols proposed by different researchers.

LEACH Protocol
Low Energy Adaptive Cluster Hierarchy (LEACH) is one among the most widely used hierarchical cluster-based routing protocols for WSN's [6]. In this protocol, as mentioned above the network is split into the different clusters that are based on Time Division Multiple Access (TDMA) schedule for member nodes. Protocol consists of number of rounds 'r' and each round has two phases: set up phase and a steady-state phase. In set up phase CH selection is based on the two factors. Firstly, percentage p of the nodes in network, and second history of nodes that has served as CH. A threshold value T (n) is set and also decision is taken by each node n based on the random number in between the value 0 and 1. Therefore, value of T(n) will be given by expression The generated random number is if less then threshold value then node becomes CH for that round. Thereafter, each node has probability 1/p of becoming CH in each round. In steady-state phase, nodes send their collected data by using their allocated TDMA slot to CH. The CH aggregates the data when received and send it to the BS. There are some problems associated with LEACH Protocol: The CH is selected randomly so each node in cluster has same probability to become CH. After numerous rounds, the node with high remaining energy and node with low remaining energy have same probability of becoming CH. If the node with low remaining energy is chosen as CH, it will run out of energy and die quickly which affects network lifetime. The division of clusters is occurring in random fashion which results in uneven distribution of clusters. The divided clusters have non uniformities like: one cluster has many nodes than any other cluster as well as some CH's are at middle and someone are at the edge of cluster that is far from others. These non uniformities affect the performance of network.

PEGASIS Protocol
Power Efficient Gathering Sensor Informative System (PEGASIS) is an improved version of LEACH protocol [7]. This protocol is guaranteed by two characteristics [8], only one of the nodes connects with BS at the same time and remaining nodes connects only with their neighbours. Each node communicates with the nearest neighbour by adjusting its power signal strength. In this way, each node measure the distance to the neighbourhood nodes in order to find the nearest node. In this process a chain is formed as shown in Figure1 and a leader is elected from chain on the basis of Residual Energy (RE) in every round. The leader collects data from neighbours and transmits the collected data to BS and thus the average energy consume by each node per round is decreased. This approach also reduces the bandwidth requirement by lowering the overhead required. The PEGASIS also has limitations for example: it is complicated work for all the nodes is that to maintain the complete database about all the location of the other nodes within that network and also the communication in PEGASIS protocol suffering from extreme delays that are caused by one chain for the distant nodes. Also, a high probability that for any of nodes to become a barrier. The Hop PEGASIS approach which is more efficient than the LEACH and PEGASIS. If there is a direct transmission between CH's and BS, the CH that is situated by far from the BS that uses strong signals while transmission to the BS thus leads to more energy consumption, so in this protocol the lifetime of network gets improved. The experimental results shows that, in Hop PEGASIS the nodes survived around 3000 rounds better than LEACH (1100 rounds) as well as PEGASIS (2000 rounds).

HEED Protocol
Hybrid Energy Efficient Distributed Clustering (HEED) protocol is an extension of LEACH protocol, in this protocol the RE is used as primary parameter whereas other topology characteristics like node degree, spans to neighbours that are used as second parameter to obtain power balancing in sensor network. The clustering process is splitting into the number of repetitions, in every repetition nodes which are not enclosed by any CH and therefore doubles their possibility of being a CH. Therefore, these energy-efficient cluster protocols that in addition enable each node to possibly and independently determine its role in this clustered network. Moreover they cannot guarantee optimal elected set of CH's [1].
These protocols introduced the uniform CH distribution that is across the network through enabling low power levels of cluster endorse an increase in the spatial reuses although the high power levels of the clusters are needed for inter-cluster communication. The long range communication from CH to sink is possible as communication between CH and BS provides more energy conservation. The main limitation of HEED is that it appoint compelling overhead in network which causes remarkable energy dissipation that resulting in the reduced network lifetime. The CH's near the sink might die earlier because of high workload on them and also the uncertain CH's which don't becoming the final CH's leave some uncovered nodes, these nodes are forced to become CH and these enforced CH may be not having any of the members related with them. As a conclusion, unpredictable numbers of CH's that are develop which generate the instability in energy dispersion of network. The heterogeneous HEED (H-HEED) has been proposed, this protocol basically used in heterogeneous WSNs. The recent developments in H-HEED also serve same purpose which is main requirement of WSNs, to prolong network lifetime and energy efficiency [9]. The simulation results shows that the throughput and packet delivery ratio are improved by using H-HEED and the delay and energy consumption is also less than HEED.

DEEC Protocol
The Distributed Energy-Efficient Clustering Protocol (DEEC) was proposed to cope with energy-heterogeneity. The DEEC protocol election of CH's based on the ratio between the RE of each node and the average network energy. The epos CH's of being CH's for the nodes are different according to the initial and also their residual energies. The nodes with more initial and remaining energy have greater chances of the becoming CH's compared to nodes with lower energy. They proposed a set of equations to ensure high energy nodes have more chances of being elected as CH's. They choose the probability P to become CH as: where E i (r) is the RE of node i at round r, p opt is the initial probability of a node to become CH in a homogeneous setup as used in LEACH and E(r) is the estimated average energy of the network at round r given as: Where RT is the total round of the network lifetime, n is the number of nodes in the network, E total and E round are the total energy at the beginning of the network and the energy consumed in the network in each round respectively. By using this method of estimation, DEEC is able to further extending the network's lifetime compared with the LEACH and Stable Election Protocol (SEP) protocols [10].

SEP Protocol
Stable Election Protocol (SEP) is the first approach employed to extending the network's lifetime by investigating the possibility and the impact of energy-heterogeneity, which is a result of sensor nodes consuming different amount of energy in the network over time. This could be due to: -Their distance or position to the BS.
-The amount of data aggregation performed as a CH, which depends on the size of each cluster. -Some nodes could fail during transmission as a result of the deployment terrain.
-The network could be recharged by introducing new set of additional nodes.
-The nodes could be powered by energy harvesting sources like solar, wind etc.
All the above reasons could potentially be sources of energy-heterogeneity. Based on this investigation, SEP protocol is based on weighted election probabilities of each node to become CH according to the remaining energy in each node. In this two node types of energy nodes are used to characterize the energy-heterogeneity problem in the network. They further adapted the weighted probabilities according to the types of nodes in the network. The nodes are labelled as 'normal nodes' and 'advanced nodes'. Each type of node elects itself according to new sets of threshold indication function given below: In the above equations the probability P in LEACH is replaced with P nrm and P adv signifies the weighted probabilities, which translates into thresholds T (n nrm ) and T (n adv ) of election for both the normal and advanced nodes respectively. The values for P nrm and P adv are calculated as: Where m is the proportion of the advanced nodes with α times more energy than the normal nodes. With the above governing equations the SEP protocol improved significantly the network lifetime of WSNs as compared with the LEACH protocol [11].

ESEP Protocol
Enhanced Stable Election Protocol (ESEP) is a modified clustering algorithm with a three-tier energy set-up to characterize energy-heterogeneity, where energy consumption among sensor nodes is adaptive to their energy levels. This protocol is a direct enhancement to the original SEP protocol and the nodes are called 'intermediate nodes' to further characterize energy-heterogeneity. In this method each node chooses a different probability for election as CH depending on the type. The probabilities for each type of nodes are given below: Where P nrm , P int and P adv are the probabilities of becoming normal, intermediate and advanced nodes respectively. Here m is the proportion of the advanced nodes with α times more energy than the normal nodes, b is the proportion of the intermediate nodes with times more energy than the normal nodes. P opt is the initial probability of a node to become CH in a homogeneous setup as used in LEACH. However, to guarantee that the protocol adapts to the type of nodes (heterogeneous setup), the threshold indication function used in LEACH and SEP must be changed to T (n nrm ), T (n int ), T(n adv ) for normal, intermediate and advanced nodes respectively, which reflect the CH election probabilities. These probabilities must equally satisfy the following conditions: -The advanced nodes must be CH's exactly (1 + α) times every 1+mα+bµ P opt -The intermediate nodes must be CH's exactly (1+µ) times every 1+mα+bµ P opt -Every normal nodes must become CH's once every 1+mα+bµ P opt -The average number of cluster in the network should be n P opt Following this analysis the threshold indication functions for normal, intermediate and advanced nodes can be computed as: From the above equations, we have n (1 − m − b) normal nodes, where G is the set of all normal nodes that have not become CH in the past 1 P nrm round r. The same analogy follows for the intermediate and advanced nodes.
There are nb intermediate nodes, with G the set of intermediate nodes that have not become CH in the past 1 P int round r.
Similarly, there are nm advanced nodes, with G as the set of advanced nodes that have not become CH in the past 1 P avd round r. Hence, the average total number of CH's per round will be: To sum-up, ESEP protocol is designed to exploit different levels of heterogeneity in the network, which proved to further significant extend in network lifetime compared with the LEACH and SEP protocols [12].

DEC Protocol
Deterministic Energy-efficient Clustering (DEC) protocol is dynamic, distributed, self-organizing and more energy efficient than existing protocols. This approach of the DEC protocol is very simple that is to reduce the computational overhead costs to the self-organized networks. This protocol has better performance with respect to energy consumption in both heterogeneous and homogeneous network. This protocol uses RE of each node that in cluster that for the election process of CH's. The uncertainties in the CH elections have been minimized in DEC protocol. The setup phase that used in LEACH protocol is modified, but the steady-state phase is kept same as that of in LEACH protocol. Because energy of node can be defined a priori, that CH's election process is reconstructed by using the RE of every node [15]. In DEC protocol, the BS elects no. of CH's at round m for the network. The BS take only be part in the election of the CH's in the case that m=1. Also, elected CH's will advertise their role by using Carrier Sense Multiple Access (CSMA MAC) like in LEACH protocol. However, in DEC protocol similarly in LEACH protocol, the join-request message will contain Cluster Member identity (CM-ID), Cluster Head identity (CH-ID) and Cluster member-residual energy (CM-RE) and the header that indicates it as a request. This way the RE information of Cluster Members (CM's) is familiar to their corresponding CH's thus localized and it can be utilized for CH rotation in the subsequent rounds. With above mentioned systematic approach, establishment of a Heterogeneous WSN will be done by using DEC routing protocol in order to increase the network lifetime and to reduce the energy consumption.

Problem Formulation
Number of work is presented by researchers on probabilistic approach for decreasing energy dissipation for the network. The primary goal of the protocols was the enhancement of the network lifespan with wide information without focusing on local data gathering from the nodes residual energy. Major demerits of the protocol is that no confirmation of required amount of cluster heads will elected with sufficient amount of energy to perform its task. According to latest search new deterministic cluster's head selection approach give better results than old probabilistic model in the scenario of energy dissipation. The model having similar issues that of unguaranteed CH selection for one round with another model. LEACH protocol gives satisfactory results even by using their stochastic model, but results are not optimum because of the doubts in CH election technique. Equation of probabilistic system utilized in LEACH protocols is shown below: Px is the CH probability to be elected, here x will be defined as normal nodes , advance nodes or super nodes rand 'Q' is a constant value or function of ratio of an additional residual energy value added to every node. In many papers the value of 'Q' is one. The deterministic quotient for 'Q' is calculated for every round in each node for the enhancement of LEACH performance. According to T(Px) function of threshold indicator each and every node take part in electing the CH intended for every 'r' round. Then wireless node selects the number among zero and one. In case this value is less as compared to nodes 'n' threshold value, then the wireless node is considered as CH. Group of non-elected cluster member is represented by symbol 'G'. DEEC (Distributed Energy-Efficient Clustering Protocol) is an improved version of LEACH with the consideration of heterogeneous environment. Also, "Balanced and Centralized Distributed Energy-Efficient Clustering Protocol" (BCDEEC) enhance the DEEC with LEACH for heterogeneous environment by three-nodes. BCDEEC and DEEC protocols are used for the indicator function of threshold for their judgment.
In wireless sensing network different probabilities known as (P x ) for nodes are defined by equation 15. The process of clustering starts with a initial phase while each Fig. 2 Flowchart of DEC algorithm nodes in the network uses the "indicator function" to be elected like a CH. Advertisement message is broadcasted over carrier sense multiple access protocol by elected (CHs). Broadcasting message has CH ID's with a header known as "Announcement". A cluster member (non-elected nodes) discovers the cluster and chooses the cluster head having least transmission cost .Cost has been formulate on the strength of received signal containing advertising message. Using carrier sense multiple access protocol request of joining transmit towards selected CH by the cluster member. The message has cluster member-ID (CMID), cluster head-ID (CHID) with the header that shows the message being a request. Intra-cluster communication between cluster heads is held by time division multiple access at the end of setup phase. In the "Steady-state" phase of the protocol, the data will be transmits from the cluster members to the CHs and from cluster heads towards BS. Direct Sequencing Spread Spec-trum (DSSS) technique is used for inter-cluster communication Therefore, proposed deterministic model capable to give a maximum residual energy for the nodes and improves life span of network. This technique gives a better idea for energy dissipation in WSN. Therefore, adapts a deterministic approach to get permanent number CHs elections for each round.
The flowchart for DEC algorithm is presented in figure 2 Based on our simulation researches, the followings are realized, which makes this DEC protocol favorable: -The CH election is nearby decided on every node's RE and each round is independent of the subsequent round dissimilar to LEACH and ESEP -DEC guarantees every node a chance of election as long as it's RE is higher than its neighbours -DEC ensures a fixed N opt CH is chosen -DEC significantly reduces the overhead cost of computation associated with CH search in the existing protocols, by refining this search to cluster N i at round m -DEC guarantees that every CH has sufficient energy to take up the duty, till the end of the network life-period, dissimilar to LEACH protocol

Network Model
Different radio models with their energy dissipations applied in previous research are given below in Figure 1. For transferring the data frames by a distance (d) having satisfactory "Signal to Noise Ratio" (SNR), amplification energy (ε) to conquer the (free space loss) ( f s ) or (multi-path) (mp) loss, on the basis of transmission distances. So, for transmitting the k bits, the dissipation of energy is given below.
Eel denotes the energy consumption for each bit for "Transceiver circuit" and d 0 is the distance threshold for changing the Amplification models that could be computed as: To receive a k-bit message, the radio will expend: Consider a symmetry radio channel. Symmetric means it required equal value of energy to send or receive (k) number of bits from nodes 1 to node 2 and nodes 2 to node 1. In existing protocols for example in LEACH we consider an ideal "data aggregation" that collects the packets from varied cluster members integrated by their particular CHs and only one single packet is transferred towards BS. Now, total dissipated of energy in during a round in the network is given by: Where,(dtoCH) denotes average distance among cluster members with the CH.
(dtoBS) denotes average distance among CH with the BS K denotes the amount of clusters EDA denotes the data aggregation cost in CH we can optimize number of clusters by finding the derivative of Eround with respect to k to zero to BS E f s E amp (23)

Simulation and Result
In DEEC-ET the multilevel clustering is deployed at three different levels of nodes is defined which is better suited for defining heterogeneous environment as compared to two nodes levels being defined in DEEC protocol. The novel type of nodes being used in the DEEC-ET termed as "Super -nodes" for energy greater as compared to each two nodes (advanced nodes with the general nodes). It should be noted that network of total energy is remained similar like a DEEC protocol. Varied nodes has Eo as Energy, the nodes with fraction with the time having more energy as compared to each normal nodes equivalent to Eo (1+a) while the super nodes having fraction mo with b value time extra energy as compared to normal nodes as compared to Nmo as the total number for super nodes with the Nm(1-mo) as total number for the advanced nodes. Initial energy of three heterogeneous WSN is provided by: DEEC-ET uses almost similar CH mechanism selection and the energy estimation approximately than previous DEEC. At each round, nodes chooses while to be a CH or not by selecting an arbitrary number among zero or one. When the number of less as compared to T(px) being the threshold value as depicted in equation 1 as compared to nodes will select towards CH for the some provided round. In the DEEC-ET, the threshold value being accustomed which is dependent on that the value having the node which chooses the CH value by establishing the Eresi as a residual energy with the Ea as the average energy of that round respectively towards the optimal CHs. Modified Threshold value is presented by DEEC-ET protocol is shown below: Q opt = optimum no. of CHs E resi =residue energy of nodes E a = network average energy New clustering technique enhance the lifetime and stability of nodes in heterogeneous environment named DEECET protocol. Performance comparisons given below The levels taken for nodes are: normal nodes, intermediate nodes, advanced nodes, super nodes. The normal nodes are deployed randomly in the field as shown in Figure  4. The normal nodes are represented as circles with cyan colour and sink in the middle with blue colour. The percentage of nodes exist in this area is 60% of the total nodes. The energy associated with normal node is 0.5 J per normal node. The intermediate nodes are deployed randomly in the field as shown in Figure 4. These nodes have energy level greater than normal nodes but lesser than advanced and super nodes. The intermediate nodes are represented as squares with pink colour. The percentage of intermediate nodes exist in this area is 20% of the total nodes. The energy associated with intermediate node is 1.25 J per node. The advance nodes are deployed randomly in the field as shown in Figure 4. The advance nodes are represented as rhombus with green colour. The percentage of advance nodes exist in this area is 10% of the total nodes. The energy associated with advanced nodes is 2.0 J per node which is greater than normal as well as intermediate nodes.
In step 4 bunch of nodes elect its leader that act as cluster-head where, Non cluster-head nodes become member nodes of that cluster, this process is known as clustering. The clustering technique is shown in figure 5. The nodes which act as CH's are represented by pointer symbol on those nodes. At the end of each round the CH is changed.
After exact amount rounds, each network nodes become dead. The Figure 6 shows all dead nodes. At the end of lifetime of the network after particular number of rounds,  The results obtained from DEECET protocol shows improvement in number of rounds and stability of WSN. It is found that The first node dead (FND) also known as stability period is at 1839th round for DEEC means that the network is stable up to 1839 rounds. The last node dead (LND) is at 2350th round by using original DEEC protocol. Whereas DEECET performed better and the FND is at around 2350th round means that the network is stable up to 2350 rounds. The LND is at around 2400th round by using DEECET protocol. Figure 7 shows Performance difference plot between DEECET and previous DEEC protocol on the basis of alive nodes and rounds. This can be concluded from above  discussion that the EDEC is around 500 rounds more stable than original DEC and the network lifetime of EDEC is also increased by 50 rounds. Figure 8 shows comparison between original DEC and EDEC in terms of total energy versus number of rounds. The energy management of EDEC is better than Original DEC as shown in Figure 4.17 proving that EDEC is more energy efficient than original DEC protocol.

Conclusions
The performance evaluated from the results clearly shows that DEECET proposed deterministic protocol gives better network lifetime as compared to previous probabilistic protocols. DEECET purely deterministic protocol optimize the utilization of energy resources in wireless sensor network. The new proposed DEECET protocol uses heterogeneous environment to improve the performance of wireless network. Heterogeneous environment contains new nodes known as "super nodes" in the sensor network. This protocol can be used in those application where power consumption is the main constraint.