Optimization on the multi-period empty container repositioning problem in regional port cluster based upon inventory control strategies

Within the area of regional port clusters, this paper establishes a multi-period mixed integer programming model to optimize the empty container repositioning between public hinterlands and ports, comprehensively considering the quantitative and periodic inventory control strategy. At the same time, this paper dynamically and heuristically determines the empty container inventory threshold (D, U) under quantitative strategy and S under periodical strategy at each port within the regional port clusters. On this basis, the empty container repositioning schemes between public hinterlands and ports are optimized by using Markov decision process combined with dynamic programming method. In addition, Liaoning coastal regional port cluster and its northeast hinterland are selected as the objects to solve this model and the results show that the total cost of shipping company can be saved by 14.16% and 11.92%, respectively, by the quantitative and periodical inventory control strategy. Selecting the quantity of public hinterland terminals, the empty container demand of public hinterland terminals and ports, the inventory threshold of empty containers and other factors, this paper carries on the sensitivity analysis. This paper validates inventory control strategy can weaken the shipping company in the influence of the external environment changes. And the quantitative inventory control strategy can reduce the total cost value to a greater extent and more effective in cost control than periodical strategy.


Introduction
Every 1 in 3 containers globally is moved empty, and according to Boston Consulting Group (BCG) it costs the shipping industry up to $20 billion per year. While the increasingly fierce competition among ports, resources and environmental pressure also bring serious test for ports and shipping companies. Giving play to the overall advantage of the regional port cluster and carrying out resource integration has become an important way to improve the comprehensive competitiveness of regional ports. As the future direction of port development, regional port clusters are unstoppable. China has formed five regional port clusters, which are significantly enhancing the radiation and driving effect of coordinated development of regional economy. Therefore, under the development mode of regional port clusters, the hinterland that was in a competitive situation has been transformed into the shared public hinterland within the regional port clusters, and the container terminals within the public hinterland have also changed from the competitive mode to the cooperative sharing mode. Empty containers can be circulated in the broader hinterland through the transportation between the public hinterlands within the areas of regional port cluster. It is of great importance for the cost reduction of shipping companies' daily operation to decide how to effectively manage the empty containers using the advantage of the regional port cluster, and how to form a well empty container repositioning mode between ports and hinterland terminals which is of great significance for shipping companies to reduce the total cost in the operation process.
From the perspective of empty container repositioning, the traditional research is mainly studied by maritime transportation, combining with joint optimization of full containers transportation, route selection, shipping transportation network designing and other aspects (Xu et al. 2012;Song and Dong 2012;Song 2008Song , 2009Song and Dong 2011;Quang-Vinh et al. 2012). Secondly, the empty container repositioning under the inland transportation is mainly about the research on the inland container terminals. Wadhwa et al. (2019) proposed to establish an empty container terminal in the hinterland area, so as to minimize the total system cost and avoid excessive empty mileage. Hjortnaes et al. (2017) established an offshore empty container repositioning system consisting of inland terminals, port yards and inland dry ports and the results can save 15% of the direct transportation cost between inland terminals. Shan et al. (2020) studied the location of terminals in the inland transportation system and studied the empty container repositioning between terminals considering the uncertainty of demand. Kuźmicz and Pesch Kuźmicz et al. (2020) studied the optimal site selection for inland container terminals, so that the repositioning of empty containers can be seamless. Sarmadi et al. (2020) proposed a dry port network of empty container repositioning considering the randomness of demand and jointly optimized the flow of full and empty containers. Peng et al. (2019) comprehensively considered the two modes of inland transportation and put forward the optimization method of empty container intermodal transport so as to maximize the profit of logistics enterprises.
In recent years, intermodal transport has become increasingly prosperous, and there are more and more studies on empty container repositioning between sea-land transport, which mainly include the cooperation between terminals and ports, sea-land coordinated transport and so on. Xie et al. (2017) studied the transport coordination of empty containers based on the cooperation between single port and single terminal. Kolar et al. (2018) proposed that empty container repositioning requires the cooperation of all parties involved in container transport in the hinterland. Zhao et al. (2018) studied the influence of random demand and supply changes on empty container repositioning under sea-land intermodal transport. Kuzmicz and Pesch Kuzmicz (2019) studied the joint railway and maritime transport between China and Europe and believed that sea-rail intermodal transport would become the main mode of empty container repositioning.
From the perspective of empty container inventory, it is necessary to control the empty container inventory at the port within a certain range in order to avoid the loss of customers caused by the shortage of empty containers, as it takes certain time to transport or supply empty containers. When the empty container demands cannot be fully met, they can be made up by the empty containers stored at ports. Inventory theory is often used in warehouse management, while it is used to solve the optimization problem of empty container inventory in recent years. Luo and Chang (2019) studied empty container inventory management when customer demand changes in intermodal transport systems, and discussed the influence of empty container repositioning on the optimal inventory level. Li et al. ( 2004) combined inventory theory and penalty function to solve the problem of unbalance of empty container demands. Legros et al. (2019) manage empty containers through threshold policy to save the cost of empty containers. Zhang et al. (2014) proposed a pair of critical points for the empty container inventory at ports and designed a polynomial time algorithm to determine the two thresholds for each period. Hariga et al. (2016) established a mixed integer nonlinear programming and solved the optimal replenishment period and quantity of empty containers. Sahoo et al. (2016) proposed a single-period inventory fuzzy probability model with fuzzy demand and fuzzy storage space under the opportunity constraint, so as to maximize the total profit.  proposed that container reposition is inevitable phenomena, but it can be minimized. The container inventory management mix is a set of controllable variables that attempt to this task. A carrier can optimize the use of container inventory mixing three ingredients, namely, freight, forecasting, and flexibility that suits under different conditions.  proposed virtual container yard to manage the empty container inventory. It was evidenced that container exchange between carriers that leads to virtual container yard (VCY) is an effective solution and there are opportunities for carriers to exchange containers.
In the joint optimization study of empty container repositioning and empty container inventory, Song and Dong (2010) studied the shipping process of empty container loading and unloading under the uncertain destination port, and set the threshold values of empty container inventory at the port. Jiang and Grossmann (2015) proposed two mixed integer linear programming models, continuous and discrete, to optimize the inventory and repositioning volume. Hemmati et al. (2015) designed a heuristic algorithm for the inventory and route selection and optimized the inventory and route selection. De et al. (2017) studied the inventory route problem by taking container ships with time window constraints as the research object. Wang (2017) integrated the inventory cost of containers into the existing shipping route network design and studied the inventory cost and empty container transportation network. Eide et al. (2020) incorporated ship speed and load factors into the model when studying the offshore inventory path problem, and the results showed that considering these two factors could save 38% of the cost. Dong et al. (2020) determined the navigation route of each voyage on the trade route, and the empty container inventory of the visited ports can be controlled within a reasonable range. Rodrigues et al. (2019) studied a kind of maritime inventory route problem under the circumstance of uncertain sailing time considering the possible delay during the voyage. At the same time, the methods to deal with uncertainty intensively, such as deterministic model with stock buffer, robust optimization and stochastic programming are also considered.
Based on the above researches on empty container repositioning and empty container inventory, it can be concluded that three following problems need to be further optimized. One is the static problem of a single period; the second is that the empty container repositioning coordinated by sea-land of regional port cluster is not mature yet. The third is disconnection between regional port cluster and hinterlands. According to Braekers et al. (2011) and Song and Dong (2015), this paper adopts Tables 1 and 2 to show the differences between the current paper and existing papers.
To solve above problems, compared with the previous literature, the difference of this paper is that the empty container inventory threshold in each period is dynamically processed. Considering the quantity conversion between full containers and empty containers, the empty container inventory in each period will change with the import and export of full containers. At the same time, in the Markov dynamic process design, the transition probability is used to make the model transit from the current state to the next state, so as to solve the randomness in the process of empty container transportation. This paper comprehensively considers multi-period, inventory control, setting up empty container repositioning system between the public hinterland and ports within the regional port cluster, through the combination of dynamic programming and the Markov decision process. The two different types of inventory control strategies, the quantitative and the periodic inventory control strategy are used to optimize empty containers repositioning quantity and inventory holding threshold at each different decision period, which is to minimize the shipping company's total cost. The paper is organized as follows. The problem formulation is given in Sect. 2. Empty container repositioning model and the corresponding mathematical formulations are discussed in Sect. 3. In Sect. 4 we describe the calculation of empty container inventory threshold and the design of Markov dynamic process. In Sect. 5 we report the computa-tional results and conduct sensitivity analysis, and in Sect. 6 we draw the main conclusions.

Problem formulation
In the regional port cluster transportation network, the empty and full container transportation is carried out reciprocating not only among the ports, but also between the ports and the hinterland terminals. Due to the existence of regional port clusters, the landward hinterland between two or more ports is crossed, which is also known as the cross hinterland or the public hinterland. Within the confines of the public hinterland, the range and influence of the ports are overlapped. When ports are short of containers, inland terminals from the shared hinterland can provide empty containers for multiple ports, which adds a new source of empty containers for shipping companies in this way. Figure 1 shows the empty container repositioning mode between the regional port cluster and the landward shared hinterland.
In this paper, the conversion between full and empty containers is also considered. In each cycle, the shipping company transports the full containers from one port to another. After a cycle of transportation, the full containers arrive at the port, and part of the full containers will be placed in the port waiting for the consignees to pick up the goods. The remaining full containers will be transported to the inland container terminals by inland transportation, waiting for the inland consignees to pick up the goods. When the consignees take out the goods from the port or inland container terminal, it means that the container changes from full state to empty one. Due to the uncertainty of the consignee's pickup time, this paper assumes that it takes one cycle time, that is, the time from a full container to an empty container is a cycle. After being converted into empty containers, they can be used as empty container supply for ports or inland container terminals. In other words, the quantity of full containers to be shipped out at ports or inland container terminals in the current period can be regarded as the empty container demand of the ports or inland container terminals in the current period. In this mode, as shown in Fig.  2, for each node, empty container inflows can come from the remaining of the empty container inventory of last period, repositioning quantity from other ports at last period arriving current period, the repositioning quantity from public hinterlands, the quantity of full containers conversion from empty containers in the previous period and the quantity of lease containers. Outflows of empty containers include repositioning to other ports or to public hinterland terminals and the quantity of full containers shipped out of the port. For the inland container terminals, the empty container inflows include the remaining empty container inventory of the terminals in the previous period, the conversion quantity of full    containers transported from the port to the terminal in the previous period, the empty container transported from the port in the previous period, and the empty container outflows include the empty container repositioned to the port and the full container transported to the port. So on the whole, for each port node, before empty container repositioning, it is necessary to calculate the net empty container outflow of the port, so as to judge the empty containers situation of remaining or shortage and then judge the direction of empty container repositioning; then make a decision on the empty containers to be transported in each cycle; after the weekly empty and full container transportation between ports and terminals, the remaining empty container inventory shall be calculated. Thus, the remaining inventory of empty containers that can be used in the next cycle can also be obtained. In addition, in order to make up the time cost of empty container repositioning, this paper considers to set reason-able empty container inventory threshold at every port within the area of regional port cluster, and applies quantitative and periodic inventory control strategy to empty container repositioning problem. Based on the inventory theory, the quantitative inventory control strategy adopts the (T , s, S) method to set a minimum inventory point s and a maximum inventory S. The remaining inventory is ST . When ST > s, empty container repositioning is not required. When ST ≤ s, empty container repositioning or leasing containers is required to supplement the empty container inventory to S. Therefore, under this strategy, whether empty container repositioning is required depends on whether the remaining inventory is lower than the minimum inventory s. The empty container inventory is checked every fixed cycle t to see whether the remaining inventory ST is lower than the minimum inventory s, then determine whether empty container repositioning or leasing is required, and determine empty container repositioning and leasing. The periodic inventory control strategy adopts the (T , S) method, and the shipping company carries out repositioning and replenishment of empty containers according to the fixed repositioning cycle t (1 week or 1 month, etc.). The empty container repositioning cycle of this method is fixed, but the quantity of every repositioning is variable. Generally, the quantity of repositioning shall be judged according to the historical demand data, and then a more appropriate maximum inventory s shall be set. When each period is about to end, the empty container inventory shall be checked, and the repositioning shall be started after recording the remaining inventory. The inventory after empty container repositioning shall reach the maximum inventory S. Due to the regular inspection, it is necessary to ensure that the maximum empty container demand is met every cycle, that is, to maintain the maximum inventory.
Under this theory, the interval (D, U ) is set at the port under quantitative inventory control strategy , the parameter D is the minimum value of port inventory, and the parameter U is the maximum value of port empty container inventory. When the empty container inventory of the port is lower than D, the quantity of empty container is replenished until it reaches D by means of empty container repositioning among ports or inland terminals and even by means of leasing containers. When the empty container inventory of the port exceeds U , the empty container inventory of the port should be reduced by repositioning the empty containers outward to other ports and the shared public hinterlands. Under the periodical inventory control strategy, the maximum inventory S is set at every port for each decision period. At the end of each decision-making period, check the empty container inventory of ports, and record the remaining empty container inventory, so that the port inventory can reach the maximum value after repositioning and leasing containers. The inventory interval of every period is changing with the transportation of full containers, the repositioning and leasing quantity of empty containers. The dynamic programming method is adopted to make a decision on the inventory threshold at the beginning of every decision period. In this way, the empty container inventory of the port at the end of the decision period can be kept within the optimized range.
Aiming at minimizing the total cost of all periods, this paper comprehensively considers the constraints such as the balance of empty container inflow and outflow at each port, the balance of inventory at ports and inland terminals, the capacity limitation of inventory at ports and inland terminals, and the capacity limitation of transportation. Three kinds of situations are set up. Method 1 optimizes the empty container repositioning scheme without considering the empty container inventory control strategy. Method 2 and Method 3 consider the empty container inventory holding interval under the quantitative inventory control strategy and periodical inventory control strategy, respectively, at the same time calculate various costs and optimize the empty container repositioning scheme. Comparing the costs under different methods, this paper verifies that the strategies of inventory control can effectively reduce the total operating cost of shipping companies. At the same time, the advantages and disadvantages of adopting two different empty container inventory control strategies can also be verified by comparing the costs of Method 2 and Method 3.

Hypothesis
(1) The full containers that arrived at the port in the last period are converted into empty containers as empty container supply for the current period. For the full containers arrived at the port, part of them will remain at the current port waiting for the consignees to pick up the goods. After that, the full containers will be converted into empty containers. The time in general practice is about 5-7 days, because there is a free storage period of 1 week in the ports, and exceeding the free time will lead to detention charges. If the port-related terms and charges change, the time will also change. In fact, the conversion time needs to be truly described through a large number data of empty and full containers. This paper only makes a general description; (2) The empty container repositioning between hinterland terminals is not considered. In fact, there are two modes of transportation between hinterland terminals. The other mode is repositioning between terminals, that is, "streetturn," involving a more complex transportation network, which will not be discussed in this paper; (3) Shipping companies may lease empty containers from other companies to fulfill its gap resulting from the demand of containers. The lease containers will arrive within the current period without any consideration of returning the containers. In this paper, the flow of empty containers is studied, and the flow of full containers is not considered. Because it is difficult to determine the container leasing time in the practical operation of containers, and it is also difficult to calculate the leasing time of each container with a specific formula, in this paper, in order to simplify the calculation, the time of a decision period is used to represent the container leasing time. In the calculation of empty container leasing cost, the unit leasing cost in one cycle is given, and the total empty container leasing cost in all cycles is obtained through the accumulation of multiple cycles (Peng et al. 2019); (4) The handling charges of empty containers are not considered. They are uniformly charged into the transportation costs of inter port transportation. In the calculation process, we do not make a separate cost parameter in the model; (5) Empty containers repositioned by sea between ports in the previous period will arrive in current period and be empty container supplies at ports in this period. This paper considers the empty container repositioning of liner transportation in regional port cluster. The departure frequency of the liner is basically 1 week. Therefore, it is assumed that the empty containers transported by sea need to go through a decision cycle, that is, 1 week to become empty container supply. If the empty container transportation is carried out in a larger area, the empty container transportation takes longer to complete; (6) Only 20-foot containers are considered in this paper. In other words, this is single commodity mode in the case (Peng et al. 2019).

Notations
(1) Sets and Index T : Set of periods, t ∈ T P: Set of ports of a regional port cluster, i, j ∈ P H : Set of ports of hinterland terminals, m ∈ H (2) Input parameters C P i j : Unit transportation cost of empty containers between port i and port j C S m 1 : Unit storage cost of empty container in terminal m C S i 2 : Unit storage cost of empty container in port i C E im : Unit transportation cost of empty containers between port i and hinterland terminal m C L i : Unit lease cost of container in port i for one decision period I q t ji : Quantity of full containers transported from port j to port i at period t, which can be used as empty container supply after conversion and an independent identically distributed random quantity Oq t i j : Quantity of full containers transported from port i to port j at period t, the equivalent of empty container demand for the current period, which is an independent identically distributed random quantity λ im : 1 if the hinterland terminal m is within the coverage of port i ; 0 otherwise Q H t im : Quantity of full containers transported from port i to the hinterland terminal m at period t D H t m : Empty container demand of the hinterland terminal m at period t, which is a random quantity W 1 max m : Maximum storage of empty containers in hinterland terminal m W 2 max i : Maximum storage of empty containers in port i F t i j : Maximum transport capacity between port i and port j at period t (3) Decision variables Q P t i j : The empty container volume received by port j from other ports. Correspondingly, Q P t ji represents the empty container volume received by port i from other ports QL t i : Quantity of lease containers in port i at period t Q E t im : Quantity of empty containers transported between port i and the terminal m at period t . A positive value represents the quantity of empty containers transported from port to terminal, and a negative value represents the opposite direction (4) Derived variables ST 1 t m : Empty container inventory of terminal m at the beginning of period t. It refers to the quantity of empty containers at terminal m after receiving the inflows from other ports ST 2 t i : Empty container inventory of port i at the beginning of the period t. It refers to the quantity of empty containers at port i after receiving the inflows from other ports/terminals ω t i : Difference of empty container inflow and outflow of port i at period t. Inflow represents the empty container volume transported into the port, that is, the empty container volume received from other ports and terminals within its coverage. Outflow represents the empty container volume transported out of the port, that is, the empty container volume transported from the current port to other ports and terminals within its coverage. The difference between the inflow and outflow of empty containers represents the net inflow of empty containers in the port, which can judge the surplus and shortage of empty containers in a port.

Mathematical model in Method 1
In Method 1, we study empty container repositioning problem without considering the inventory strategy.
Before modeling, the difference of inflow and outflow of containers in port i is calculated as Eq. (1). If ω t i > 0, port i is a surplus container port, which means that the surplus container quantity of the port can meet its empty container demand and can provide supplies for other ports and inland terminals within the coverage; otherwise, it means that the port is in the state of shortage of containers, so it is not able to reposition empty container, and it needs to rent containers or wait for other ports and inland terminals to reposition.
The objective function is to minimize the total cost of the shipping company in all periods, which consists of three parts, namely, the repositioning cost between ports , between ports and public hinterland terminals, the container leasing cost, and the empty container inventory cost of hinterland terminals and ports.
Equations (6) and (7), respectively, represent the empty container inventory of hinterland terminal and port. Equation (8) represents the quantity of empty container repositioned from port i to the hinterland terminal m within period t. Equation (9) represents the quantity of empty containers repositioned from port i to other port j within period t. Equations (10) and (11) represent that the quantity of empty container repositioned from port i should not exceed the empty container inflow within period t. Equation (12) represents that the quantity of empty container repositioned from the hinterland terminal m should not exceed its remaining empty containers within period t. Equation (13) is the quantity limit of lease container. Equations (14), (15) and (16) are, respectively, maximum inventory and maximum transportation capacity limits. Equation (17) is a non-negative constraint.

Mathematical model in Method 2
In Method 2, we consider the empty container inventory threshold under the quantitative inventory control strategy. Set D t i as the minimum value of the empty container repositioning at port i ∈ P during period t ∈ T . U t i is the maximum value of the empty container at t ∈ T period at port i ∈ P . That is, importing empty containers up to D t i when the quantity of empty containers in the port is less than D t i , or exporting the empty containers down to U t i when the quantity of empty containers is more than U t i . In Method 2, the objective function is the same as that in Method 1, and the constraints are the same with Eqs. (2)-(17). At the same time, the following constraints about empty container inventory at ports are included: Equations (18) and (19) indicate that the empty container inventory of the port is controlled within the holding interval.
Equations (20) and (21) indicate that the empty container inventory of ports can meet the requirements of the holding interval by repositioning and lease containers.

Mathematical model in Method 3
In Method 3, we consider the empty container inventory under the periodical inventory control strategy. Periodical inventory strategy replenishes empty containers according to fixed replenishment period . Let Q t i be the empty container supply quantity of port i ∈ P at period t ∈ T , which is equal to the sum of quantity of empty container repositioning between ports, the empty container repositioning from hinterland terminals to ports, and lease quantity at the port. It can be expressed as Q t i = j∈P Q P t i j + m∈H Q E t im + QL t i . S t i is the maximum empty container inventory of port i ∈ P at period t ∈ T . The objective function of Method 3 is the same as that of Method 1, and constraint conditions (2)-(17) are still adopted, while the following constraints about the empty container inventory of port are increased: Equation (22) represents the limit of empty container replenishment under the periodical inventory control strategy. Equations (23)-(24) indicate that the maximum empty container inventory of the port can be achieved by repositioning and leasing containers.

Markov dynamic process design and determination of inventory threshold
In view of the nature of multiperiodicity, randomness and dynamics of empty container repositioning, this paper establishes an empty container repositioning model with multiperiod, dynamics and randomness. Therefore this paper selects the method of combining dynamic programming with Markov decision process. It can not only solve the multiperiod problem by using the dynamic programming process, but also realize the characteristics of dynamics. At the same time, it can also use the Markov decision process to deal with the randomness in the empty container repositioning process by using the transfer probability. So this paper applies Markov decision with dynamic programming to calculate the optimal cost and scheme under three different scenarios.
And the threshold parameters of empty container inventory are also heuristically determined.

Markov dynamic process design
Set p t 1 be the probability of I q t ji , and p t 1 {I q t ji = z 1 } = p t 1 (z 1 ). Set p t 2 be the probability of Oq t i j , and p t 2 {Oq t i j = z 2 } = p t 2 (z 1 ). Set M be the smaller value of the port throughput and the ship's maximum transportation capacity. The MDP process of port empty container repositioning can be summarized as follows: For any period t , the empty container inventory of the port is ST 2 t i at the beginning of the period t, is set as the system state space. So for ∀ST 2 t i ∈ S, set a t as an available action in this state, which defines the actual operations of empty container including repositioning or leasing, then a t = (Q P t i j , QL t i , Q E t im ); set A (i) be the available action set of the whole system, then t here is a t ∈ A (i) . The quantity of empty containers available to the shipping company at port i at period t, including the influence of random quantity, is ϕ t = ST 2 t i + j∈P Q P t i j + QL t i + m∈H Q E t im + j∈P I q t−1 ji . In the above formula, the quantity of empty containers available for shipping company is ψ t = ST 2 t i + j∈P Q P t i j + QL t i + m∈H Q E t im without considering the influence of the random quantity of full containers transported in and out. According to the above formula, the system state is ST 2 t+1 i = ψ t + j∈P (I q t−1 ji − Oq t i j at the beginning of the next week. The probability of going from state Figure  3 shows the dynamic process of the state transition.
Then, in the deducing process we use the backwarddeduction method of dynamic programming and the optimal cost and state of each period are derived.
(1) In the stage T , the initial state of the system is ST 2 T i . In this state, there is a feasible practical action a T , namely, the operation of repositioning and leasing containers. After repositioning and leasing containers, the quantity of available empty containers in the port can reach ψ T , and the corresponding lease and repositioning fees will be charged, described as C T 1 + C T 2 . Under the state ψ T , C T 3 the inventory fees will be charged due to the influence of full container transported in and out. If V t is the sum of costs of all t − T periods, V T = C T 1 +C T 2 +C T 3 can be obtained, and a corresponding cost V T will be generated for every available action a T .
(2) In the T − 1 stage, the initial state of the system is ST 2 T −1 i , quantity of empty containers will be changed to ψ T −1 after action a T −1 , and the cost is C T −1 1 + C T −1 2 at this time. Affected by transporting the full containers in and out, the initial state of the system is ST 2 T i =

Fig. 3 Markov dynamic decision process of empty container inventory
at period T , and the probability of the state ST 2 T i for the p T −1 (a T −1 ), the total cost at this time is shown in formula (25). Thus, in the T − 2 stage, the total cost is shown by formula (26). In the t stage, the total cost for the period t − T is shown in formula (27).
To calculate the transition probability, let ST 2 t i = s t , ST 2 t+1 i = s t+1 ,ST 2 t+n i = s t+n ,s t , s t+1 , s t+n ∈ S, then the state transition probability of n step in the MDP decision system can be expressed as p (n) = s t+1 ∈S p s t s t+1 p (n−1) s t+1 s t+n . It can be proved as follows:

Determination of empty container inventory threshold
where μ t i j is the mean of empty containers shipped out of port i to port j at t period. μ t im is the mean of empty containers shipped out of port i to terminal m at t period. The sum of them can be regarded as the demand for empty containers. μ t ji is the mean of empty containers shipped into port i at every period (namely, empty container supply), and σ t i j is the standard deviation of empty containers shipped out of port i at every period. Since different quantities of full containers are transported in and out of each port at each period, the weekly empty container inventory threshold also changes with different periods and ports.
In the upper limit formula, the sum of the mean value and the standard deviation of the total demand of empty containers at every period of the port i is adopted. The upper limit is calculated by taking the sum of empty container demand at each period and controlling the range of fluctuation through standard deviation. In the formula for calculating the lower limit, the sum of the standard deviation and the mean value of the total difference (net outflow) of the empty container inflow and outflow quantity of port i at every period is adopted. From the formula we can see, we need to take the maximum value between the net outflow of empty containers and 0, which means that when the net outflow is negative, the first half of formula (29) needs to be taken as 0. At this time, the port is in the state of lack of containers and can no longer transport empty containers to other ports and terminals, so the lower threshold is equal to the standard deviation. If the net outflow is positive, it is the sum of remaining empty container and standard deviation. This is to enable empty containers to flow from the surplus port to the deficient port and reduce the empty container repositioning in the opposite direction. Generally speaking, when the port is in the state of shortage of containers, the lower limit value should be set relatively low, and when it is in the state of surplus containers, the lower limit value should be set relatively high.
In addition, under the strategy of periodical inventory control strategy, combined with the dynamic decision-making process of MDP mentioned above, the maximum empty container inventory of each port within the regional port cluster is calculated according to following formula (30). Let L be the lead time of empty container repositioning. Let α be the service-level coefficient; Z be the interval period of empty container repositioning; σ t i j is the standard deviation of the empty container demand in the advance period of empty container repositioning at port i of period t. The interval period of empty container repositioning is 1 week, that is, a decision-making period. The lead time means how long the shipping company needs to send requests for empty container repositioning to other ports and hinterland terminals in advance. Then, the calculation formula of the maximum empty container inventory of port i at every period is shown as formula (31). The first half of formula (31) is the product of the sum of empty container repositioning decision period and lead time, and empty container demands, which requires that the empty container inventory threshold must first meet the empty container demands during the period of empty container transportation, so as to prevent empty container shortage. At the same time, the corresponding service-level coefficient is determined according to the satisfaction of the demands. When the empty container demand follows the normal distribution with standard deviationσ t i j , α can be obtained by looking up the table (normal distribution table). The second half of formula (31) adjusts the upper and lower floating range of the threshold through the service-level coefficient and standard deviation.
Model 1 is the basic empty container transportation model. Constraints (1)-(17) show the conditions such as empty container inflow and outflow balance and capacity constraints required in the process of empty container repositioning. On this basis, the concept of lead time is introduced to calculate the changes of costs by applying the periodic inventory control strategy and (D,U) is introduced to calculate the changes of costs by applying the quantitative inventory strategy. The purpose is to make the final result meet the empty container repositioning demand of each cycle. Therefore, the constraints of model 2 include (1) (31), which together constitute the empty container repositioning model under the periodic inventory control strategy.

Numerical settings and experimental results
In this paper, Liaoning coastal regional port clusterhinterland of northeast China is selected as an example to conduct model calculation. From the perspective of economic situation and geographical location, as shown in Fig. 4, the hinterland of Dalian Port is the Northeast Economic Zone, including the three provinces of Liaoning, Jilin and Heilongjiang as well as the Eastern Four Leagues of Inner Mongolia. The indirect hinterland of Yingkou Port and The number of decision periods is 10. According to the frequency characteristics of liner, every period is set as 7 days. Dalian Port, Yingkou Port and Dandong Port are selected as the research objects, and terminals of Shenyang, Anshan, Changchun, Tonghua and Harbin city are selected as the public hinterlands of the regional port cluster, as shown in Fig. 4. The algorithm program is written by MATLAB 2018a, and the i73770 CPU processor of windows 10 operating system is used for operation. Standard empty containers are 4.5 yuan/container/km for road transportation and 1.8 yuan/container/km for railway transportation. The distance between the port and the public hinterland is more than 200 km by rail, otherwise by road. The average storage cost of public hinterland terminals is 10 yuan/container/day. In addition, due to the strong randomness of the empty container supply and demand, this paper describes and assumes the mean value and standard deviation of the demand based on the historical data and the actual operation situation of the port. It sets that the empty container demand of all ports and terminals is subject to normal distribution. All data of the port are shown in Table 3.
Combined with Markov decision process and dynamic programming, under the quantitative inventory control strategy, the empty container inventory holding intervals of the three ports within 10 decision-making periods can be calculated, as shown in Table 4 Under the periodical inventory control strategy, the advance decision-making period for empty container replenishment is L = 2/7 ≈ 0.29 ; the empty container repositioning periodic decision is Z = 1 ; under the condition that the demand satisfaction rate reaches 99.99%, α = 3.5, it can calculate the maximum empty container inventory threshold of every port within the regional port cluster of every period, as shown in Table 5.   After the above-optimized empty container holding interval and the maximum threshold of empty container inventory are known, the models under the three Methods are calculated, and the results are shown in Table 6.
According to the above calculation results, through quantitative and periodic inventory control strategy, the optimal empty container inventory holding thresholds are established for the shipping company during different periods. Given that the supply and demand of empty containers are relatively stable in a certain time, for the shipping company at every decision period goes on an empty inventory threshold calculation is extremely tedious operation. Thus after obtaining the optimal inventory range or threshold value of each cycle, the operation of taking average value is carried out, and the optimal value of ten periods is regarded as the inventory control standard of shipping company over a period of time. It can be seen from Table 4 that both quantitative and periodic inventory control strategy can effectively reduce the total cost of shipping companies compared with Method 1. In particular, the quantitative inventory control strategy can bring a greater degree of total cost reduction for shipping companies. The port keeps a certain quantity of empty container inventory, which can timely meet the shipping company's customer demand, reduce the shipping company's dependence on the container leasing, and avoid the shipping company being greatly affected when the external environment changes. At the same time, in the early stage of empty container repositioning, full container transportation needs a cycle time, and full container transformation also needs a certain time. Therefore, empty container repositioning is difficult to be fully realized in the early stage. In this case, shipping companies need to rely on a large number of leasing containers to meet the demand of empty container. With the continuous transportation of full containers and the conversion of full containers into empty containers, the volume of empty container repositioning between ports and between ports and terminals gradually increases, so in the later stage of empty container repositioning, the volume of leasing containers will decrease compared with the previous stage.

Sensitivity analysis
In order to verify the universality and effectiveness of the models and methods, we selected relevant factors for sensitivity analysis. The cost data of the numerical experiments come from the actual research of inland city terminals such as Dalian and Shenyang. The inflow and outflow of containers are characterized by normal distribution according to the actual volume of ports.

Test of the number of public hinterland terminals
For exploring the influence of changes in the number of public hinterland terminals on all costs in three methods, we conduct experiments with different number of terminals to compare the different costs, as shown in Fig. 5. First, on the whole, with the reduction of the number of public hinterland terminals, the cost values in the three Methods are in the state of growth, which is mainly due to the increase of the number of lease containers and repositioned containers between ports. It can be found that when the number of public hinterland decreases to 4, the advantages of sea-land container repositioning between the regional port cluster and the public hinterland are no longer obvious. In Method 1, the lease cost has increased by 22.82% and the total cost by 20.74% compared with the initial state. When public hinterland quantity is reduced from 5 to 1, in Method 1, the total cost increased by 33.9%, total cost under the Method 2 increased by 26.41%, total cost under Method 3 increased by 27.97%. At this time only one public hinterland terminal serves for ports of regional port cluster, which leads to sharp gap of empty containers at ports. Under this circumstance only a large number of containers could be rented to make up for the empty container demand, and the cost of leasing containers increased by 46.1% compared with the initial state.
In addition, the total cost and other costs of the ports within the regional port cluster have been significantly reduced after the management of empty containers by adopting quantitative and periodical inventory control strategy. In the initial state, that is, when there are 5 public hinterland terminals serving the regional port cluster, the total cost of Method 1 is 16.36% more than that of Method 2, and the total cost of Method 1 is 13% more than that of Method 3. Decrease in the number of public hinterland terminals, the total cost of the gap is sharper, when the number of public hinterland is reduced to 3, the cost difference then has been expanded to 41.03% and 38.59%, respectively. While the number of public hinterlands finally becomes one, the total cost of Method 1 has exceeded the total cost of Method 2 by 68.59% and Method 3 by 61.74%. Taking a comprehensive view of the changes of various cost values, the empty container repositioning scheme after adopting inventory control strategy can bring the shipping company a great degree of cost savings. As the number of public hinterland terminals changes, it is obviously seen that the repositioning cost, lease cost and inventory cost grow in Method 1, while in Method 2 and in Method 3, these costs do rise but not as obvious as in Method 1, which further illustrates the inventory control strategies play an important role for shipping company in saving total cost of the empty container repositioning. In addition, taking a comprehensive view of the two empty container inventory control strategies, all costs under the quantitative inventory control strategy are lower than the periodical inventory control, which can control the shipping company's total cost to a greater extent. Therefore, when there are few hinterland terminals serving for the ports, it is more effective to adopt quantitative inventory control strategy for empty container management.

Test of demand for empty containers at ports and public hinterland terminals
Empty container demand is one of the important factors affecting the empty container repositioning scheme and the total operating cost of shipping company. Changes in the demand of ports within regional port cluster will have different degrees of influence on the repositioning cost, lease cost and storage cost, and then on the total empty container operating cost of shipping companies. Figure 6 shows the influence of changes in empty container demand of ports on various costs. Clearly, the total cost in all three methods is increasing as port demand continues to grow. When the cost increases to 20%, the total cost of Method 1 and Method 2 fluctuated significantly. However, when increasing from 20 to 100%, the total cost in Method 1 changed by 68.07%, while the total cost in Method 2 fluctuated by 23%, and the total cost in Method 3 changed by 28.2%. The larger the growth range of demand, the smaller the fluctuation range of total cost and the stable growth of other costs under the two inventory control strategies. This is because under the empty container control strategies, the empty container inventory has been controlled within a reasonable range. Even if there is a large number of random empty container demand, the empty container inventory at the port can be timely responded, and there is no need to rent a large number of containers to make up for the lack of empty containers. In addition to the influence of empty container demand of ports, the change of empty container demand at public hinterland terminals will also have a profound influence on shipping companies' empty container repositioning costs and plans. Figures 7 and 8 together show the changes of costs under the circumstance of the change in empty container demand in the public hinterland. Firstly, the growing demand for public hinterland has led to the increase of overall costs in all three methods. Secondly, with the increasing of empty container demand of public hinterland terminals, the cost of container leasing and repositioning will continue to rise. This is because when the demand for empty containers in the public hinterland increases, a large number of empty containers are used to meet the needs of inland shippers, which leads to less and less empty containers flowing from public hinterland terminals to ports and makes shipping companies increase the volume of repositioning and leasing. Thirdly, the storage cost in the three Methods showed a significant downward trend. When the inland empty container demand increases from 20 to 100%, the storage cost of Method 2 decreases by 38.38% and by 44.09% of Method 3. This shows that under the inventory control strategies, shipping companies keep a certain amount of empty containers in the port, so when empty containers in public hinterland cannot be replenished in time, a large number of port inventory can meet the demand gap of empty containers. With the increasing gap, the consumption of port inventory is also increasing, and the storage cost of Method 2 and Method 3 shows an obvious downward trend. In Method 1, due to the lack of sufficient port empty container inventory, the shipping company can only make up the empty container by leasing and repositioning. When the terminal demand increases to 100%, the leasing cost in Method 1 has increased by 14.55%, while in Method 2 and Method 3, it only increases by 9.16% and 9.78%, respectively.
Finally, with the increasing demand for inland terminals, the difference between the total cost of Method 2, 3 and Method 1 becomes larger and larger, as shown in Fig. 7. When the inland demand for empty containers increases to 100%, the difference between Method 1 and Method 2 has expanded from 23.47 to 40.44%, and the difference between Method 1 and Method 3 has expanded from 20.15 to 35.44%, which proves once again that adopting the corresponding inventory control strategy can effectively reduce the total cost of shipping companies. Compared with the periodical inventory control strategy, the quantitative inventory control strategy can enlarge the total cost difference to a greater extent. Therefore, when the inland terminal demand for empty containers changes, the quantitative inventory control strategy can help shipping companies to reduce the total cost more effectively. Figure 9 shows the influence of changes in unit lease costs on every cost under the three different methods. It can be seen from the figure that there is a positive correlation between the unit lease cost and the total cost. When the unit lease cost increases to 20%, the growth rate increases rapidly. When it changes to 40%, the lease cost increases by 15.86% compared with the initial condition in Method 1, Method 2 by 9.89%, and Method 3 by 6.85%. When the growth rate reaches 80%, the growth rate slows significantly and the total repositioning cost increases in Method 1. This also shows that when the cost of leasing containers continues to increase, shipping companies still lease containers to meet the demand for empty containers, but they also pay more attention to making up the gap of empty containers through repositioning. Under the development mode of regional port cluster, container repositioning between ports cannot meet the demand for empty containers, so shipping companies reduce their dependence on container leasing through empty container repositioning between ports and public hinterland, so as to control the influence of the increase of lease cost on the total cost of shipping companies.

Test of unit lease cost
In addition, under the quantitative and periodical inventory control strategy, the shipping company at the port yard will have a certain number of empty containers. With the growing of unit lease cost, empty containers inventory will play an important role. It is obvious that after the lease cost increases to 20%, the fluctuation of the total cost with inventory control is more stable, and the influence of the increase of lease cost is not obvious. With the continuous expansion of the lease cost, the total cost difference between Method 1 and Method 2, and Method 1 and Method 3 is also gradually expanding, from 24.81 and 22.04% to 41.54% and 41.69%, respectively, which makes the shipping company less affected by the change of external container leasing cost. In this case, the extent of cost control of quantitative inventory control is almost the same as that of periodical inventory control strategy. When the cost of lease containers changes from 40 to 80%, the quantitative inventory control strategy is more effective. When the growth rate expands to 100%, the difference of cost control of the two strategies is very small, so it has the same advantages.

Test of unit inventory cost
Figures 10 and 11 show the changes of costs under three methods when unit inventory cost changes. There is a positive correlation between unit inventory cost and total cost. In the case of increasing unit inventory cost, the storage cost in Method 1 shows an obvious trend of fluctuation and rising, and gradually approaches the inventory cost in Method 2. When the unit inventory cost increases to 20%, the storage cost has increased by 30.17% compared with the initial state, but only by 18.56% in Method 2 and 18.15% in Method 3. It can be found from Fig. 10, at the same time as the unit cost of inventory incremental expands unceasingly, the growth of inventory costs under three Methods is falling. This also indicates that when unit inventory cost increases to a certain extent, the quantity of empty containers stored at ports and inland terminals decreases, and a large number of empty containers are used for repositioning between ports and between ports and inland terminals.
Secondly, the total cost of Method 1 is always higher than the total cost of Method 2 and Method 3 with the increase of unit inventory cost, and the gaps among three Methods keep widening as the increase amplitude. When the growth rate is 20%, the total cost difference between Method 1 and Method 2, Method 1 and Method 3 is 21.73% and 18.56%. When the growth rate is 100%, the difference has been widened to 31.81% and 30.36%, respectively. Taking inventory control strategy to manage empty container inventory can help shipping companies to develop more superior empty container repositioning scheme. At the same time, the quantitative inventory control strategy can control the shipping company's total cost more effectively than the periodical inventory control strategy. Scaling from 20 to 100%, quantitative inventory control strategy can reduce total cost by about 2% more than periodical inventory control strategy and is more effective when unit inventory cost changes.

Test of empty container inventory threshold
In this paper, the optimal empty container inventory thresholds under quantitative and periodic inventory control strategy are calculated through Markov decision-making and dynamic programming process. In order to further compare and analyze the two inventory control strategies, the empty  container inventory thresholds under the two inventory strategies are changed, and the influence on all costs is shown in Fig. 12. The optimal empty container holding interval under the quantitative inventory control strategy and the maximum empty container inventory under the periodic inventory control strategy are increased by 20-100%.
First of all, it can be found that in the whole process of change, the change of the lease cost under the two inventory control strategies is not obvious, and the increase of the inventory range will not affect the quantity of lease containers. Secondly, inventory costs under the two strategies are increasing, especially the inventory cost under the Method 3 is much higher than Method 2, which is due to the increase of holding inventory. After meeting the empty container demands of current period, a large number of surplus empty containers will store at the port yards and inland terminals. At the end of all decision-making periods, the port yards and inland terminals have accumulated the remaining empty containers for ten periods, resulting in a large amount of overstock and waste of empty container resources. This also indicates that in the management of empty containers, the holding threshold of empty containers should be reasonably established. Too little inventory cannot timely meet the empty container demand of ports and public hinterland term too many inventories will lead to large inventory costs and waste of empty container resources.
Finally, it can be found from Fig. 12 that when the threshold is expanded to 60%, all costs under the periodical inventory control strategy are no longer changing. It means that with the increase of the maximum empty container holding quantity, the maximum empty container quantity of the port storage yard has been reached and can only be maintained at 60%. When the optimal empty container inventory threshold increases from 20 to 80%, compared with the original optimal total cost, the total cost of the quantitative inventory control strategy is lower than that of the periodical inventory control strategy. When the change is from 80 to 100%, the total cost of the periodic inventory control strategy begins to be lower than that of the quantitative inventory control strategy, while the total cost of the quantitative inventory control strategy is still rising. Therefore, it is more effective to choose the periodic inventory control strategy to manage Fig. 13 Comparison of total cost under different modes empty containers. All the above tests also prove once again that the empty container inventory holding threshold under the quantitative inventory control strategy calculated in this paper and the maximum empty container inventory under the periodic inventory control strategy are optimal.

Test of service mode of hinterland terminals
In order to highlight the value of bringing the port group and hinterland terminals into the model, we have the numerical experiment to compare the total cost under the single-port and single-terminal mode with the empty container repositioning under the sea-land coordination mode within the port cluster proposed in this paper. The so-called single port and single station mode means that the hinterland terminal only carries out the transportation of empty and full containers with only one corresponding port, which means that a hinterland terminal only provides empty containers for one port. However, under the port group mode proposed in this paper, an inland terminal can replenish empty containers for the ports within its coverage. In these two modes, the total cost is compared. The calculation results are shown in Fig. 13.
It can be clearly seen from the figure that the port group model proposed in this paper can save at least 35.48% of the total cost compared with the single-port and single-terminal model, and at least 42.87% of the cost can be reduced if the inventory control strategy is adopted. This is also because under the single-port and single-terminal mode, only a small amount of empty containers are replenished from inland areas to the port, shipping companies need to reposition containers between ports, and also need a large amount of empty container leasing, resulting in an increase in costs. On the contrary, if the port cluster and hinterland terminals are inte-grated into the model, whether the empty container inventory control strategy is adopted or not, it can save a lot of costs for the shipping company. The replenishment of inland empty containers can reduce the number of containers leased by the shipping company and then reduce the total cost.

Conclusions
This paper establishes a system of empty containers repositioning between regional port cluster and public hinterlands, using the mixed integer programming model, considering the quantitative and periodical inventory control strategy, and applies the Markov decision process combined with dynamic programming method, proving that empty containers from public hinterlands can be important supplies for shipping companies. At the same time, the results of sensitivity analysis prove that the quantitative and periodical inventory control strategy can effectively help reduce total cost of shipping companies. When the external factors change, it can reduce the influence on the shipping company's empty container allocation scheme and the total operating cost.
Compared with the two inventory control strategies, when external factors such as the number of public hinterland terminals, empty container demands and unit inventory cost change, the quantitative inventory control strategy can reduce the total cost of shipping company to a greater extent than the periodical strategy, which is more effective. When the unit lease cost continues to increase by two times, the two have the same control effect on the total operating cost of the shipping company. When the threshold of empty container inventory is increasing, quantitative inventory control strategy will be an advantage at first, but when the inven-tory intervals increase indefinitely, the total cost has been on the rise under quantitative inventory control strategy. Due to the limitation of terminal capacity, the maximum empty container inventory threshold S cannot be increased continuously under the periodic inventory control strategy, and the total cost of the shipping company can be controlled within a certain range. Therefore, the periodic inventory control strategy is more effective than the quantitative inventory control strategy in cost control in the late growth period.
As for the future development of shipping companies, they should be more involved in the development of regional port clusters, strive for more public hinterland resources and more available empty container replenishment resources from the regional port clusters, and transform the competitive hinterland into cooperative sharing mode, thus forming a development mode of terminal sharing. According to the dynamic quantity changes of empty container repositioning and leasing, factors such as the set of empty containers holding interval or threshold, shipping companies should reasonably determine the empty container inventory, to maximum extent, which can meet the demand of empty container, but also avoid the waste of empty container resources caused by excessive accumulation of empty containers. In this way, it can reduce the dependence on leasing containers, which is a long-term development plan of shipping companies.
The limitation of this paper is that it only roughly describes the conversion time and quantity between empty and full containers. Although it is close to reality as much as possible, the description is still not accurate enough. The transportation time of empty containers is only calculated according to the departure frequency of the liner, which is not applicable in a larger transportation area; empty container repositioning between hinterland terminals is not considered.
Further research direction is to study the empty container repositioning between public hinterland terminals and exclusive inland terminals, and between public inland terminals of regional port cluster considering the optimization of empty container's inventory holding threshold, build the linkage empty container repositioning system between regional port cluster and landward hinterland. At the same time, it is necessary to further characterize the conversion relationship of quantity and time between empty and full containers and container transportation time.
Author Contributions All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by JC, YL and YY. The first draft of the manuscript was written by JC, writing review and editing by YL, YY and XW and supervision by ZJ. Edirisinghe Lalith was responsible for the language revision of the full paper. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Data availability Enquiries about data availability should be directed to the authors.