QoTa-MPR: QoS-oriented and traffic-aware multi-path routing protocol for internet of remote things

Internet of Remote Things (IoRT) is widely used in both military and civilian applications. However, due to the unique characteristics, which is characterized as long-distance and high-latency, the design of the routing protocol is a great challenge of IoRT. In this paper, a QoS-oriented and traffic-aware multi-path routing protocol (QoTa-MRP) is proposed for IoRT. QoTa-MRP is composed of two parts. The first part is the link traffic-aware based muti-paths source route discovery mechanism, which is used to establish multi-paths with lower link disjoint degree. The second part is path similarity and traffic priority based multi-path selection mechanism, which is used to determine the transmission mode of the traffic flows to enhance the reliability or effectiveness of the transmission. Simultaneously, the protocol is theoretically analyzed in terms of the successful transmission rate of routing transmission data packet. Finally, the dynamic source routing, which is a very representative protocol and is usually used as the baseline comparison protocol, is revisited for the performance verification of the QoTa-MRP in IoRT. It is shown in the simulation results that there are significant superiorities of QoTa-MRP than that of DSR in terms of network throughput, packet loss rate and routing packet header overhead in IoRT .


Background and motivation
Significant achievements in traditional ground Internet of Things (IoT) services have been witnessed in recent years. However, the deployment and maintenance costs of the original communication framework in IoT are very high in geographies such as deserts, oceans, and forests [1]. Therefore, communications in the above-mentioned geographical environments have become the main focus of the academic, in which case IoT is also renamed by Internet of Remote Things (IoRT) [2]. In particular, satellite-assisted terrestrial radio frequency channels and underwater acoustics are typical examples of physical communication media used in IoRT, which is characterized as long-distance and high-latency. During the past few years, IoRT has emerged as a potential technique in both military and civilian applications, including 5 Gansu Provincial Research Center for Conservation of Dunhang Cultural Heritage, Lanzhou, China disaster prevention, military target scouting, goods freight cities, battlefield distributed surveillance, road traffic management, oil-spill monitoring, aerial photography, coastline protection, wildlife monitoring, early warning systems for earthquakes and tsunamis etc. [3]. It is noted that the famous GoogleBalloon project is aslo a typical application of IoRT, which aims at providing subscribers of remote areas with connectivity.
There are not only many similarities but also the following outstanding differences between the IoRT and traditional ground IoT [4]. Firstly, due to the complex and changeable working environment, transmission media and the vulnerability to the attack, the link transmission quality between the network nodes is changed frequently and unstable in IoRT. In other words, the link between any two neighbor nodes is intermittently connected due to the complexity and variability of transmission environment. Therefore, how to use existing channel information to accurately evaluate even to predict the link quality is a key point that must be considered in the design of data transmission protocols for IoRT. Secondly, some nodes in the network are overloaded while some other nodes are lighter-load because of multi-hopping for data transmission and the particularity of node location, which results an unbalanced load of the whole network. To that end, how to overally balance the load of the nodes for the efficient data transmission as can as possible is another key point that must be considered in the design of the protocols for IoRT. Thirdly, the communication distance, which is usually from several kilometers to tens of kilometers, between nodes in IoRT is much longer than that of traditional ground IoT, and the propagation latency is usually far larger than sending latency. Therefore, the total end-to-end delay is much higher than that of traditional ground IoT. Specifically, in IoRT such as underwater acoustic network, satellite networks and outer space networks, the end-to-end delay of the information transmission is in order of tens of seconds. In other words, the IoRT network is a typical larger delay tolerant network and is not suitable for traffic that are sensitive to the latency. Therefore, the latency is usually not a key performance indicator for the network.
It should be noted that network of IoRT is essentially different from the delay tolerant network (DTN), which is also characterized by long distance and large delay. Firstly, the data stored in the node of DTN is exchanged when there is an effective link, i.e., the data is transmitted through "storingcarrying-forwarding" [5]. However, data transmission in the IoRT is realized by "storing-forwarding". Secondly, the data transmission operation of DTN is mainly depended on the "bundle layer", which is located between the application layer and transport layer, and the transmission performance is determined by the upper application [6]. The data transmission in IoRT is based on TCP/IP protocol architecture, and the communication performance is not determined by the upper application. Thirdly, the forwarding strategy of the node in DTN is the key of the routing, where replication based forwarding, utility based forwarding, social relationship based forwarding and coding based forwarding are included [7].
Because of the significant differences between IoRT and traditional ground IoT, most of the mechanisms and protocols belonged to the MAC layer or IP layer in the original communication framework are not applicable to IoRT directly [8]. For example, in terms of the access protocol, the RTS/CTS/DATA/ACK multiple handshake mechanism of CSMA/CA protocol in the short distance communication of traditional ground IoT will cause a large overhead and waste of resources in data transmission, which is unsuitable to IoRT due to the long distance and large delay transmission characteristics. On the contrary, the statistical priority based multiple access protocol (SPMA) [9] is widely used in the bandwidth-limited IoRT. Of course routing, as one of the most difficult problems in the area of the traditional IoT and there is some certain degree of mutual coupling with the MAC protocol, is also the key point of the research with respect to the emerging IoRT. The design of the routing protocol will be also seriously affected by the corresponding MAC protocol and the characteristics of IoRT such as longer transmission distance, huge propagation latency, extreme channel condition [10]. As a result, a significant challenge on the design of routing protocol for the applications of IoRT are posed due to the inherent characteristics of the physical medium.

Contribution and organization
Therefore, a QoS-oriented and traffic-aware multi-path routing protocol (QoTa-MRP) is proposed to realize the effective and reliable transmission in IoRT. The QoTa-MRP is composed of two parts, i.e., the traffic-aware based source route discovery mechanism, path similarity and traffic priority based multi-path selection mechanism. Our contributions are mainly four aspects and are highlighted as: • The traffic load based multi-path source routing strategy is designed. The estimation model for the link traffic load is used to the multi-path source route discovery. When the path similarity between the path carried in route discovery request (RDR) packet and the one stored is too high, the packet will be discarded by the forwarding node to reduce the number of the route signaling packets and the routing overhead in some extent. • The multi-path transmission mode for the traffic flows is proposed. The transmission mode of the traffic flow is determined with the comprehensive consideration of path similarity, QoS priority and link traffic load. Whether the copies or the different traffic flows are transmitted in the selected multi-paths is decided through the comparison with threshold. Therefore, either the effectiveness or the reliability of the data transmission is ensured. • The performance of the proposed protocol is theoretically analyzed. The successful transmission rate of routing transmission data (RDT) packet is analysed emphatically when there is no path stored in the source node to evaluate the performance of multi-path discovery mechanism in QoTa-MRP. Our analysis is also verified by the simulation results. • The proposed QoTa-MPR is simulated with abundant traffic scenarios and parameter setting. The QoTa-MRP is compared with dynamic source routing (DSR) through NS2 in terms of network end-to-end throughput, packet loss rate and routing packet header overhead. It is shown in the simulation results that there are more significant superiorities when QoTa-MRP is applied into IoRT.
The rest of the paper is organized as follows. The state-ofthe-arts in routing mechanisms and protocols designed for IoRT are summarized and analyzed from two perspective of the categories in Sect. 2. The system overview is described in Sect. 3. The details procedures are presented, and the performance evaluation of QoTa-MPR is given in terms of the transmission rate of RTD in Sect. 4. Simulation and results analysis are presented in Sect. 5. Finally, the conclusion and the future work of our paper are pointed out in Sect. 6.

Related work
Strictly speaking, IoRT is extension application of the traditional ground IoT, then the ideas of the routing (e.g., LEACH-C, LEACH-V, TEEN-V and V-DEEC-VD etc. [11][12][13]) may be learned and revised in the progress of the routing design for IoRT. For the effective and reliable data transmission in the IoRT, route protocols or mechanisms need to be well adapted into the unique characteristics of IoRT, such as unbalanced network load, longer transmission distance and more intermittent changes of link [14]. In general, the related work of the existing multi-hop routing in IoRT can be divided approximately into location-centered and nodecentered according to the selection criterion of next hop.
There are a few location-centered routing protocols proposed specifically for IoRT. In Ref. [15], the joint optimization of near-optimal routes, schedules and transmission power is formulated for the high-latency IoRT, and a decentralized algorithm is designed to iteratively solve the relaxed mixed-integer programming optimization problem. An opportunity route graphic (ORG) is constructed for each ship with the channel information in Ref. [16], then the multi-hop data routing opportunity transmitted to other ships is predicted. Based on ORG, an optimal routing strategy is planned for each ship through dynamic planning. In Ref. [17], a location-based delay-tolerant routing protocol is proposed for UAV networks in post-disaster operations, where the location-aided forwarding is combined with the store-carry-forward method. In Ref. [18], an energy-aware and void-avoidance based routing protocol is proposed to solve the problem of loop transmission, flooding and energy holes. The state-of-the-arts related geographic location routing protocols within three dimensional network scenario are comprehensively analyzed and comparatively evaluated in Ref. [19]. Aimed at the problem of routing hole, an energyaware and dual-path geographic routing protocol is proposed in Ref. [20] for better route recovery from routing holes. The machine learning based routing mechanism is a research hotspot in the design of wireless network, and is attracted the attention and research. In Ref. [21], a Q-learning-based Geographic routing scheme is proposed with the use of reward function learning, where the transmission efficiency of the unmanned robotic networks is enhanced. A novel algorithm for the deployment of the SDN switches is proposed in Ref. [22], where the network observability is improved, and then a K-nearest neighbor regression algorithm is designed for the prediction of the real time reliability of legacy links at the SDN controller with historic data. A Fuzzy logic-based Q-Learning Routing Protocol is proposed in Ref. [23], the next optimal forwarder of a message is predicted on the basis of a reward mechanism where the node's energy, movement, and buffer space are considered as parameters. Two energyefficient routing protocols are designed for the underwater sensor networks in Ref. [24], where the packet delivery ratio is maximized while the occurrence ratio of void hole is minimized. In summary, local location information is exploited in the location-centered routing to route data instead of global topology information.
Many routing protocols are designed from the aspect of node-centric routing. A source routing based protocol is proposed for underwater acoustic networks in Ref. [25], it is demonstrated in the results that source routing can be conveniently and feasibly used into the underwater networks when the characteristics of underwater acoustic communications is properly implemented and tailored. In Ref. [26], a protocol named FRUDP is investigated for the reliable and efficient data transmission of aeronautical ad hoc networks, where the reliable UDP and fountain code is combined. A network coding based multi-path cooperative routing protocol is proposed for LEO satellite networks in Ref. [27], where different portions of the data flow are transmitted along multiple link-disjoint paths dynamically and cooperatively. In Ref. [28], a novel multi-criterion decision based load balancing algorithm is designed for multi-UAVs relayed tactical adhoc network. Subsequently, a stochastic packet forwarding mechanism is designed in Ref. [29], where the selection decision of the forwarding drone is made with the consideration of multiple real-time network metrics. In Ref. [30], a SDN-based network architecture is proposed firstly to manage and schedule the wireless resources in satellite-ground integrated networks, then a dynamical cooperation based efficient routing transmission algorithm is designed. In Ref. [31], a multi-levels QoS-oriented and fuzzy-logic-based routing strategy is proposed for satellite IP network, where the fuzzy rules set is built for the map of the congestion degrees and the linguistic variables. A topology-aware and reinforcement learning based routing protocol is proposed for underwater sensor networks in Ref. [32], where the resource waste caused by the forwarding in a wrong direction is eliminated. In Ref. [33], an adaptive cross-layer routing protocol is investigated and simulated, where the best coded modulation and relay node in the next transmission slot are selected under the comprehensive consideration of link quality, energy consumption and topological data. In short, the quality of service(QoS) for the traffic flows is pay more attention in the node-centered routing protocols.
Through the above-mentioned analysis and comparison between the state-of-the-arts, it is found that each protocol is adapted to its specific environment due to its unique advantages and disadvantages. The shortest path based routing, the latest information based routing, the least cost based routing, the optimal link quality based routing and the mobility prediction based routing are focused by most existing methods. However, most of the existing protocols are based on the frequent RTS/CTS/DATA/ACK handshakes mechanism of CSMA/CA protocol, which may be not applicable for IoRT due to the larger transmission latency, links interruption and topology changes etc. Therefore, extra efforts should be made in the cross-layer routing protocol design for the robustness and end-to-end connections [34,35]. In other words, routing protocol with QoS-oriented and traffic-perception should be concerned synthetically for IoRT. In view of this urgent problem, a QoS-oriented and traffic-aware based multi-path routing (QoTa-MRP) is proposed for the IoRT.

System overview
In this section, the network model of the IoRT and the link traffic-load based routing mechanism are briefly explained according to the given toy example in the Sect. 3.1. The concept and the model of link traffic load, which is core criterion for the route operations in QoTa-MRP, are described in the Sect. 3.2.

Network model
It is assumed that there are N nodes deployed randomly in the IoRT. The effective communication distance between adjacent nodes is denoted by d. Each node can be both acted as the relay node for packets forwarding and also can be acted as the source/destination node for data transmission/reception. The main information in n-th node, which is denoted as node n , is composed of identification number (node n .id), traffic load value (node n .load), source routing table (node n .T s ), forwarding routing table (node n .T r ) and so on. It is assumed that the maximum number of routing terms in T s and T r are c 1 and c 2 , then T s can be expressed as The network topology model of IoRT is shown in Fig. 1. For the convenience of description, node that want to send traffic flow is defined as source node, and the operation executed in the source node for the protocol is called source node operation. Correspondingly, the destination of the traffic flow is called destination node, and the operation processed in the destination node for the protocol is called destination node operation. All other nodes are collectively called forwarding nodes, and the operation performed in the forwarding node is called forwarding node operation correspondingly.
A toy example shown in Fig. 1 is used to illustrate the node types, corresponding operations and the basic principle of the protocol. As can be seen from Fig. 1, node 1 is the source node and node 8 is the destination node, all other nodes are forwarding nodes. If there is traffic need to be transmitted in the link between the two adjacent nodes, then there is a traffic flow between them. It is assumed that there are multiple transmission priorities existed in the traffic flows, the transmission priorities of the traffic flows in the network are divided into M priorities. When there is any traffic flow with some certain priority sent to node 8 from node 1 , and there is also no available path destinated to node 8 in node 1 , then the route discovery request (RDR) packets are first broadcasted to the neighbor nodes by node 1 . When a RDR has been received by neighbor node 2 , the packet would be processed with the agreed rules and would be broadcasted for continuous forwarding. Until the RDRs are received by the destination node node 8 or some other forwarding nodes, for example node 7 , in which there is at least one path that sat-isfying the conditions stored in the forwarding table, then node 1 would be replied through a generated route discovery answer (RDA) packet, which will be sent reversely in line with the source route stored in RDR. When multi-paths destinated to node 8 are established, traffic flow of node 1 can be delivered on the multi-paths according to the multi-path selection mechanism. The specific operation steps are given in Sect. 4.

Link traffic load model
It is assumed that the set of neighbor nodes in the effective communication range of node n is set nbr n . Then the degree of node n is denoted as degree(n) = set nbr n , where |·| is the operator of solving cardinality for the set. The link between node n and any neighbor node node j ( j ∈ set nbr n ) is denoted as (n, j). Because that larger overhead and waste of resources will be caused in the IoRT by the RTS/CTS/DATA/ACK multiple handshake mechanism of CSMA/CA used in the MANET of traditional ground IoT, statistical priority based multiple access (SPMA) [9] is usually adopted in the MAC layer of IoRT under the consideration of the unique network characteristics [26,27]. The total number of packets received from neighbor nodes in the past C periods are counted by node n , where the length of each period is expressed as T . The number of the packets received from neighbor node j by ndoe n in each period is denoted as a c j (1 ≤ c ≤ C). If the size of each packet transmitted in the network is denoted by b, then the traffic load on the link (n, j), which is denoted by , in period c can be expressed as x c j = a c j × b/T . It is noted that x c j can be viewed as a discrete random variable. It is assumed that x c j in each statistical period is independent of each other. Then the expectation and the variance of x c j , which is denoted by E[X c j ] and V ar[X c j ] separately, can be obtained as formula (1) and (2).
Therefore, the network traffic load, which is denoted by node n .load, in node n is defined as formula (3).
According to the probability and stochastic theory, proximity between the load of link (n, j) in period c and the averaged link load in all C periods can be indicated by V ar[X c j ]. It should be clearly pointed out that the smaller V ar[X c j ] is, the stronger the stability of (n, j) is. This also shows that the link in the actual network is insensitive to malicious attacks or state changes caused by the reason such as node mobility, channel change and link failure etc. Therefore, in terms of link stability, V ar[X c j ] can be used to characterize the stability of traffic load on (n, j) between node n and node j . This attribute is also can be defined as the link quality (LQ) between node n and node j , which is denoted by Q (n, j) , that expressed by formula (4).
Therefore, the total link quality (TLQ) value of node n , which is denoted by node n .lq, can be correspondingly expressed as formula (5).
Given the random variables, which is denoted as X n and is used to indicate the network load of node n , the Chebyshev inequality in probability and stochastic theory is used as the function of the comprehensive consideration between link load and link quality, i.e.
where ε is any small positive value. As a matter of fact, the comprehensive consideration of link load (LL) and link quality (LQ) can be used as the routing criterion. When the above-mentioned concepts and models are described, the detailed technological process and mechanism of QoTa-MRP are presented in next section.
Maximum number of nodes stored in the domain of RDR.
Number of different nodes between the ones carried in RDR and the ones stored in T r .
p Priorities of flows.  Table  1 for their clear-cut presentation.

Source node operation
The main routing packets in QoTa-MRP are route discovery request packet(RDR, used for routing discovery), route discovery answer packet (RDA, used for routing response), route transmission data packet (RTD, used for pure data transmission) and route error packet (RER, used for routing failure report). All the packets format and the corresponding physical meaning of each field in various packets are shown in Fig. 2.
There are three types of packets that may be processed by the source node, i.e., RDR, RDA, and RTD. The flow diagram of the source node operations is showed as Fig. 3.
When there is any data flow in the source node, which is denoted as S, transmitted to destination node, which is denoted by D, the routing table T s of S is censored firstly. When there is at least one route from S to D stored in node S .T s , then the corresponding entry, which is denoted by t s i , 1 ≤ i ≤ c 1 , is selected from node S .T s according to the multi-path route selection mechanism described in Sect. 4.2. Then the assembled RTDs are transmitted through the selected routes. If there is no route from S to D in node S .T s , the route discovery process would be launched by the source node S.
When the route discovery process is launched by node S , the RDR is generated first, then the following key fields of RDR are padded, i.e., identification number of source node (rdr.souid = node S .id), identification number of destination node (rdr.desid = node D .id), list of forwarding nodes (rdr. path_node_list = null), traffic load-aware list of the forwarding node (rdr. path_ load_list = null), QoS level (rdr.qos = tra f f ic.qos), sequence number (rdr.seq = rdr.seq+1, rdr.seq is initialized from 0), transmitting time (rdr.ttime = current time). Then, the RDR is broadcasted to the neighbor nodes. At the same time, the timer for waiting RDA, which is denoted by timer rda , is set to T rda , i.e., timer rda = T rda . If the RDA is received by the node S , it is first confirmed that timer rda is not expired and then the timer rda is canceled. It is determined that the sequence number in the RDA is larger than the sequence number stored recorded in routing . It is noted that the minimum value of the load for all nodes in the source route S is defined as the path load of the source route, which means that the transmission capability of path is mainly depended on the worst one of the many links in the whole path. Finally, the traffic flow destinated to D is fetched and encapsulated as RTDs, then the RTDs are transmitted on the multi-path, which are selected out through the multi-path selection mechanism described in Sect. 4.2.

Forwarding node operation
There are three types of packets that may be handled by the forwarding node, i.e., RTD, RDR and RDA. The flow diagram of the forwarding node operation is shown in Fig. 4. If RDR is received by the forwarding node, which is denoted by node n , it is first confirmed that the sequence number in RDR is larger than that of stored in the forwarding routing table node n .T r , i.e., rdr.seq > t r i .seq. Secondly, it is confirmed that its identification number is not covered in the field of forwarding node list, i.e., node n .id / ∈ rdr. path_node_list. The reason for the processing is that the routing loops are eliminated. It is ensured that the number of nodes stored in the domain of the RDR is not exceeded by the predetermined threshold, which is denoted by N I N T E R max , i.e., |rdr. path_node_list| ≤ N I N T E R max . For example, N I N T E R max is set as 6 in our paper. Thirdly, it is determined that the number of forwarding nodes, which are carried in RDR but are different from the ones that stored in node n .T r , is not larger than the preset value, which is denoted by is set as 2 in this paper. The reason for this processing is that the number of the broadcasted routing packets are controlled in a large extent. As a result, the network overhead can be reduced substantially, and disjunction between multi-paths is enhanced correspondingly. Fourthly, node n .id is added to the domain of the forwarding node list in the RDR, i.e., rdr. path_node_list = {rdr. path_node_ list, node n .id}. Finally, the load-aware value is calculated in the light of formula (5), and is added into the field of forwarding node traffic load-aware list in RDR, i.e., rdr. path _load_list = {rdr. path_ load_list, node n .load}, then the RDR is broadcasted out to the neighbor node.
If a RDA is received by the forwarding node node n , it is determined that the sequence number of the RDA is larger than the one recorded in the forwarding routing If RTD is received by the forwarding node n , then RTD is directly forwarded to the next hop according to the source route stored in the header of RTD.

Destination node operation
There are two types of packets, i.e., RTD and RDR, that can be mainly handled by the destination node D. The flowprocess diagram of the destination node operation is shown in Fig. 5.
When the RDR is received by the destination node, which is denoted by node D , it is first confirmed that the sequence number carried in the RDR is larger than the one recorded in the routing table node D .T s , i.e., rdr.seq > t s i .seq, then the sequence number of the routing table is updated, i.e., t s i .seq = rdr.seq. Secondly, the timer that is used to wait the RDR, which is denoted by timer rdr , is activated and is set as timer rdr = T rdr . What needs to be specially stated is that timer rdr is cancelled if the total number of the arrived RDRs is larger than the corresponding reception threshold, which is denoted by N rdr , within timer validity period. On the contrary, the received RDRs are also discarded even if the total number of the arrived RDRs is lower than N rdr while the timer rdr is expired. As a result, the waiting time of the corresponding RDA in source node node S is reduced, thus the end-to-end delay is also depressed in a large extent. Thirdly, a RDA used to reply the required route to the source node node S is generated, and the corresponding label, which is used for the label-conversion-based routing and is denoted by label, is also generated randomly for the source route carried in the RDR by the destination node node D . Then, the value of the key field in the generated RDA packet is updated with the corresponding domain of the received RDR packet as follows: the identification number of source node (rda.sourid = rdr.destid), identification number of destination node (rda.destid = rdr.sourid), the list of the covered forwarding nodes (rda. path_node_list = −rdr. path_node_list), the traffic load list of the covered forwarding nodes (rda.load _node_list = −rdr.load_node_list), the QoS priority of the flows (rda.qos = rdr.qos), the sequence number of the RDR (rda.seq = rdr.seq), the random label (rda.label_id = label) and the transmission delay (rda. delay = rda.tans_time). Fourthly, the entry t r i of the forwarding route table T r is updated as follows: t r i .label _id = rda. label_id, t r i .sourid = rda.sourid, t r i .destid = rda.destid, t r i .seq = rda.seq, t r i .next_id = rda. path_ node_list [i] , t r i .qos = rda.qos. Finally, the RDA packet is sent out.
When the RTD is received by the destination node node D , the RDT is reordered and turned over the upper layer in the protocol stack for further processing.

Source route maintenance
During the process of multi-path source route maintenance, there is a lifetime timer, which is denoted by T S,D , for each source route in the T s from source node S to destination node D . If the timer T S,D is expired, all of the routing entries in the routing table T s are deleted. If there is any new traffic flow transmitted to some destination node D , then the route discovery process is restarted by the node S . The load value of the network traffic carried in RDA is used as the statistics result of the traffic load for the maintenance of network traffic-aware, and the value is updated until the next route discovery process is started. If there is a link fault occurred in the path, a route error (RER) packet is transmitted by the sending end of the link for the failure report to node S . When the RER is received by node S , then the route item whose label is equal to the one carried in the RER, i.e., t s i .label_id = rer.label_id, is deleted.

Multi-path selection
The multi-paths selection mechanism is considered comprehensively with the traffic load and the path similarity. As a result, the traffic flow with some priority is transmitted according to the selected multi-paths. It is assumed that there are M types of traffic flow priority in IoRT, which is the QoS evaluation parameter and is denoted by The QoS level of the traffic flow is decremented with the increase of the traffic priority value. In other words, the QoS level of the traffic flow with priority 1 is the highest, and the QoS level of the traffic flow with priority M is the lowest. However, it is should be noted that the traffic flow with low-priority can not be 'starved' under the consideration of the transmission fairness. It is assumed that there are K paths destinated to node D maintained in the routing table of node S . The stored paths are denoted by {P 1 , P 2 , ...,

The corresponding traffic load vector of the paths is denoted by
is the value of traffic load in path P k . The similarity between any two paths P k 1 (1 ≤ k 1 ≤ h k 1 ) and P k 2 (1 ≤ k 2 ≤ h k 2 ) is defined as the same number of forwarding nodes on the two paths and is denoted by ω k 1 ,k 2 . The smaller ω k 1 ,k 2 is, the stronger disjoint degree between P k 1 and P k 2 is. In other words, the same traffic flow transmitted on path P k 1 and path P k 2 are more reliable, and the different traffic flow transmitted on path P k 1 and path P k 2 are more effective. Therefore, the similarity of the two most independent paths in node S .T s is given as formula (7).
If the priority value of the traffic flow transmitted in node S is p m , 1 ≤ m ≤ M, the value of the mode selection for the flow transmission is denoted by δ and is expressed by formula (8).
where α, β and μ are the weighting factors, L min = min 1≤k≤K (L k ) and α + β + μ = 1. The threshold value for the mode selection is denoted by v. If v > δ, the same data flow is transmitted on the two selected paths P k 1 and P k 2 .
As a result, the reliability of network communication and the invulnerability caused by the network topology, channel state change and link failure are improved and enhanced. Therefore, some malicious attacks, such as low-power attack, selective forwarding attack, routing attack and so on, in IoRT is prevented in a certain extent. On the contrary, different traffic flows are transmitted on the two selected paths P k 1 and P k 2 when v ≤ δ. As a result, the transmission effectiveness of traffic flows is enhanced in large extent.

Performance analysis
For the performance evaluation of the QoTa-MRP, the transmission rate of RTD is analysed emphatically when there is no source route in the source node S . As above-mentioned, the RDR destinated to node D must be sent for the discovery of the multi-paths before the transmission of the RTDs. The RDA is produced in node D and is replied to node S . The RTDs can be transmitted when the RDA is received. It is assumed that N nodes are randomly distributed in the region, and the area of the region is X × Y m 2 . The packet loss rate, which is caused by the change of geographical environment and channel state, is denoted by ρ. The probability that the RDR is received by node D is denoted by R rdr D . The probability that the RDA is received by node S is denoted by R rda S . The probability that the RTDs are received by node D is denoted by R rtd D . Then the successful transmission rate of RTDs is denoted by R RT D and can be expressed as formula (9).
It is assumed that the average number of hops between node S and node D is denoted by η. In the light of Ref. [27] and [36], η can be obtained as formula (10).
where φ is the average number of hops in every route discovery, and ϕ is the average distance between node S and node D . d is the effective communication range between the nodes. γ is the average number of nodes in the effective coverage, and is expressed as formula (11).
Firstly, the probability that the RDR is received by node D with η hops is expressed as formula (12).
Secondly, the probability that the corresponding RDA is received by node S with η hops is expressed as formula (13). Finally, the probability that the RTD is received by destination node with η hops is expressed as formula (14).
Therefore, when there is no source route in source node S, the successful transmission rate of the RTD can be expressed as formula (15).
It is assumed that there are 50 to 100 nodes uniformly distributed in a 250 × 250 (m 2 ) simulation area, the effect communication distance of each node is 120 (m). The relationship between the number of hops from the source node to destination node and the number of the nodes is shown in Fig. 6. It can be seen from Fig. 6 that the number of hops, which keeps at 2.4 hops, from source node to destination node is mainly not sensitive to the number of nodes. This is because that the average number of hops is not changed significantly with the increased node density.
The relationship between RDT reception rate of destination node and the number of nodes is show in Fig. 7. As shown in Fig. 7, the RDT reception rate is smooth and steady with the enhanced number of nodes. The probability of receiving one RTD is higher than that of receiving two RDTs, this is because that each RDT may be lost in the transmission due to bad channel quality or disconnected link.

Simulation and analysis
In this section, the parameters used in protocol simulation and the parameters for the protocol performance analysis are introduced firstly, which is corresponding to the Sect.

Simulation parameters
For performance comparison, QoTa-MRP and DSR are simulated with NS2 under the scenarios with unbalanced loads, then the results are shown and analysed in terms of packet loss rate, throughput and route header overhead. In the simulation, the maximum number of back-off times for sending packets is set as 50. Time period for looking up route table is 2s, i.e. T S,D = 2s. Packet size b is set as 1024 bit, d is set as 120m. It is noted that the nodes in the simulation are divided into three categories for the scenario of unbalanced network load. Firstly, some sending and receiving nodes are selected to transmit traffic flow for a period of time, so that the load unbalanced network can be generated. These kind of nodes are called load node (LN). Secondly, some common communication nodes are set for the test of the protocol performance. These kind of nodes are called communication node (CN). The number of the CNs or the traffic volume of the CNs can be changed for the verification of QoTa-MRP and DSR. The other nodes are background node pair and are referred to as background node (BN), all of whose traffic is very low. The performance comparison parameters are as follows: • Network throughput (NT) The amount of the effective data transmitted per unit time of IoRT in the case of the imbalanced load, with which the effectiveness of QoTa-MRP can be demonstrated. • Packet loss rate (PLR) The percentage of packets lost in the process of multi-hop forwarding to the destination node from the source node. The robustness of the QoTa- Two situations are considered in the simulation for the the full results: • Scenario 1 The number of the nodes is fixed, and the traffic rate of the CN nodes is changed. In this scenario, two situations are considered and simulated. The one situation is that traffic rate of LN is fixed while traffic rate of CN is changed. The other situation is that traffic rate of LN and CN are both changed. The main parameter settings under scenario 1 are shown in Table 2. • Scenario 2 Traffic rate of the node is fixed, and the number of the CN pairs is changed. Similarly, two situations are considered and simulated in this scenario. The one situation is that traffic rate of the node is small. The other situation is that traffic rate of the node is big. The main parameter value under scenario 2 are shown in Table 3.
Finally, the function of MAC is realized by SPMA for the support of QoTa-MRP. The SPMA protocol is composed of multiple modules, which are multiple priority queues, priority contention back-off window, priority threshold, concurrent scheduling, channel load statistics, transmission and reception antennas and corresponding distributed control.

Result analysis
The relationship between the traffic rate of the CN and the network throughput in scenario 1 is shown in Fig. 8. As can be seen from the figure, the network throughput of QoTa-MRP and DSR is tended to rise with the increase of the traffic rate. However, the network throughput of QoTa-MRP is significantly higher than that of DSR. When the throughput reaches a steady state, the throughput of QoTa-MRP is about 33% higher than that of DSR. In the situation that the traffic rate of LN is varied with horizontal axis, the traffic rate of all sending nodes (i.e. CN and LN) is lower, then the load in the network is smaller. The imbalance phenomenon is not obvious in this progress, where the traffic rate is increased from 0 Mbps to 0.0512 Mbps. So that there is no obvious advantages in the network throughput of QoTa-MRP than that of DSR. When the traffic rate is bigger than 0.0512 Mbps, the load imbalance is more and more obvious in the network with the gradually increasing of the traffic rate for LN. At this time, the traffic flow in QoTa-MRP is transmitted on the path with lower load, so the throughput of QoTa-MRP is obviously higher than that of DSR. In the process where the traffic rate is increased from 0.17Mbps to 0.256Mbps, the throughput of DSR reaches saturation, while the throughput of QoTa-MRP is still increasing. However, the increase of throughput becomes smaller due to the nonnegligible conflicts between the nodes. When the traffic rate increases to 0.34 Mbps, the throughput of QoTa-MRP arrives at the saturation state and remains unchanged. In the situation that the traffic rate of LN is fixed, the throughput of QoTa-MRP is always higher than that of traditional DSR with the increase of traffic rate. This is because the network is always in a state of unbalanced load from the beginning, so even if the traffic rate of CN is very small, some advantages are also shown in QoTa-MRP. The relationship between the traffic rate of CN and the PLR in scenario 1 is shown in Fig. 9. As can be seen, the PLR of QoTa-MRP and DSR are both tended to rise with the increase of traffic rate. When the PLR is at the maximum value, the PLR of QoTa-MRP is about 16.7% lower than that of DSR. In the situation that the traffic rate of LN is varied with horizontal axis, the traffic rate of all sending nodes (i.e. CN and LN) is lower. The imbalance phenomenon is not obvious in this progress, where the traffic rate is increased from 0 Mbps to 0.0512 Mbps. Therefore, the PLR of both route protocols is relatively low. When the traffic rate of LN is bigger than 0.0512 Mbps, the load imbalance is more and more obvious in the network. The network congestion is emerging and conflicts are also increasing. Due to the multi-path and traffic load-aware characteristics of QoTa-MRP, the network congestion is alleviated at some extent, so the PLR of QoTa-MRP is lower than that of DSR. The transmission rate is saturated and unchanged with the increase of traffic rate, so the PLR tends to be unchanged. In the situation that the traffic rate of LN is fixed, the PLR of QoTa-MRP is always lower than that of DSR with the increase of traffic rate. This is because the network is in a state of unbalanced load at the beginning, the advantage is shown in QoTa-MRP from then.
It is shown in Fig. 10 that the relationship between the traffic rate of CN and the overhead of packet header in scenario 1. It can be seen that the overhead of both both the two schemes decreases with the increase of traffic rate of CN. The speed of overhead reduction is faster at the beginning in both mechanisms. This is because that the throughput and data forwarding are increased significantly with the increase of traffic rate, while the overhead of RDR and RDA is fixed, so the total packet header overhead for source routing is greatly reduced. As the traffic rate is increased continuously, the increase of throughput is slower while the decrease of overhead is also slower. When the traffic rate is increased to 0.17Mbps, the throughput is tended to be saturated and the overhead is remained unchanged. With the comparison of the overhead results between the two situations in scenario 1, where the traffic rate of LN is fixed is called situation 1, and the the traffic of LN is varied is called situation 2. It can be seen that the overhead of situation 2 is higher than that of situation 1 in the beginning. This is because that data forwarding volume is small at the beginning and throughput is also low in situation 1, while header overhead of RDR and RDA is fixed, so the total overhead is relatively small.
The relationship between throughput and the number of CN pairs in scenario 2 is shown in Fig. 11. As can be seen that the throughput of QoTa-MRP is about 20% higher than that of DSR on average. The throughput performance of the two routing schemes is increased first and then is decreased with the increase of the number of CNs pairs. In the situation where the traffic rate of CN is 0.512 Mbps, the network capacity is not reached saturation for QoTa-MRP until the number of the CN pairs is at 12, so the throughput is continued to rise. When the number of CNs exceeds 12, keep increasing the number of CNs will increase the sending conflict, and the throughput begin to decline. When the number of CNs is at 16, the network load is entered into the saturated steady state and the throughput is remained unchanged. For the same reason, the throughput trend of traditional DSR routing is basically the same as that of QoTa-MRP. However, because there is no function of load-aware for DSR, it is easier to be saturated in the network, so the throughput begins to decline when the number of CNs pairs is 10. In the case that the traffic rate of CN is 0.17 Mbps, the network get into the saturation state more slowly than that in the situation where the traffic rate of CN is 0.512 Mbps with the increase of the number of CNs.  Figure 12 shows the relationship between PLR and the number of CN pairs in scenario 2. As can be seen from the figure, the PLR of the two mechanisms are increased with the increase of the number of CNs pairs, the PLR of QoTa-MRP is significantly lower than that of DSR on the whole. In the case that traffic rate is 0.512 Mbps, the PLR of QoTa-MRP is about 14-22% lower than that of DSR. In the case that traffic rate is 0.17Mbps, the PLR of QoTa-MRP is about 15.7-20% lower than that of DSR. This is because the path with lower load is chosen by QoTa-MRP for data transmission, and the transmission conflict is smaller, so the PLR is much smaller. However, the path with high load is easily chosen by DSR for data transmission. Thus the transmission conflicts and the PLR are both enhanced with the increase of the number of CNs.
The relationship between the PHO and the number of CNs pairs in scenario 2 is shown in Fig. 13. As can be seen from The reasons are as follows. For QoTa-MRP, in the process that the number of CNs is increased from 4 to 12, the packet header overhead for the transmission of source routing information is increased slightly due to the enhanced number of RDRs and RDAs. When the number of CNs is increased to 12, the network capacity is saturated and the throughput is decreased, so the overhead is increased significantly. When the network throughput is at maximum, the transmission of RDR and RDA is increased with the raise of the number of CNs, so the overhead is increased continuously. There is no any capability of load perception in DSR, in which the network throughput is more easily saturated. Compared with the situation that the traffic rate is 0.512Mbps, the network is saturated more slowly than that in the situation where the traffic rate is 0.17Mbps with the increase of CNs.

Conclusion and further work
For IoRT with sparse deployment of nodes and unbalanced network load, a QoS-oriented and traffic-aware based multiple route protocol is proposed. Firstly, the main components of the protocol are introduced, i.e., link load based source route discovery mechanism and the path similarity and link load based multi-path selection mechanism. Then the performance of QoTa-MRP is analyzed from the aspect of the transmission rate of RTD. Finally, the performance advantages and gains of QoTa-MRP over traditional DSR are proved by simulation in terms of NT, PLR, PHO.
However, the high-speed mobility characteristics of node should not be ignored, which should be considered compatibility in our future work, when the routing protocol is designed for some IoRT application scenarios such as aeronautical ad hoc network. Different advantages are existed in the node-centered routing protocol and the location-centered routing protocol, how to combine node-centered routing with the location-centered routing to improve the reliability of routing and reduce the complexity of the signaling control process is also the future work of the QoTa-MPR. In addition, how to estimate the QoS performance parameters of physical links from the perspective of statistics and coding is the key content for link level simulation and verification, and also need to be improved in our future work. Finally, the routing security requirements in IoRT is not taken into account in QoTa-MRP. This is also where the protocols should be improved and perfected in the future. To that end, the better network robustness and security will be presented in the improved QoTa-MRP.

Declarations
Conflict of interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work in this paper.