Optimizing the agent decisions for a Cloud actuator using Deep reinforcement learning

doi:10.21203/rs.3.rs-4000624/v1

Download PDF

Research Article

Optimizing the agent decisions for a Cloud actuator using Deep reinforcement learning

https://doi.org/10.21203/rs.3.rs-4000624/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

With the increasing use of deep reinforcement learning (DRL) techniques to build intelligent systems, the application of it to real-world problems is rampant. Resource allocation in a cloud environment that need dynamic and auto-scaling features is evolving. The agent-based decisions that are offered by DRL are in use by software robotics. Auto-scaling of resources in cloud applications introduces intelligence to agents thus built by these DRL techniques. Markov decision process as a tool minimizes the target rewards to agents such that auto-scaling of applications is performed by agent decisions. Analysis of optimizing the convergence errors that are measured while the agent performs in an online environment is the challenge. Speedy Q-learning (SQL), Generalized SQL(GSQL) algorithm variants relax the parameter values of convergence with a model-free space. The authors applied heuristic values for one such relaxation parameter in our experiments. The study is an extension of works that introduced GSQL-w, where w is the convergence parameter. The authors designed a new GSQL-wh algorithm that heuristically fixes a value for w optimally in cases with over-utilization of resources. This is presented as a novel solution in this study for cloud resource workloads.

Agent Decisions

Cloud Autoscaling

Deep Reinforcement Learning

Optimization with MDP Generalized

Q-learning

Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned, and released with minimal management effort or service provider interaction. This cloud model is composed of five essential characteristics, three service models, and four deployment models. Rapid Elasticity as one of the characteristics’ is about how capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time. Infrastructure as a Service (as a service model) is the capability provided to the consumer to provision processing, storage, networks, and other fundamental computing resources. Here the consumer will be able to deploy and run arbitrary software, which can even include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but can oversee the operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls). The IaaS Cloud Service model allows computing users to start and stop virtual machines of any type and size in the pay-per-use scheme. This dynamic introduces an auto-scaling strategy that tries to solve scaling and scheduling optimization problems. Scaling predicts and plans resources in time whereas scheduling is to assign such planned resources [10]. The sub-problems like Scaling and Scheduling are NP-hard as they are not easily solved with non-deterministic polynomial solutions, but rather with the nearest optimizations [9]. This is also the reason that it can be approached with heuristics values, ie., the rule of thumb, on the parameters determined. The variability of these optimization problems in recent times is addressed by combining Reinforcement Learning and Deep Learning which is known as DRL. [13] [10]

Related work

A model-free (i.e., learning from an environment that has huge data) is when the agent learns and improves its behavior based on the history of its interaction with the environment, which is already infinite space. This is called Q-Networks. Hence the definitions are based on the current or online state of the environment there exist two different ways The authors could look at applying appropriate methods. 1. Dynamic programming (DP) (that depends on models) 2. Temporal Difference methods (TD) (that is model-free behavior). TD is based on Q-learning techniques viz., Successive Overrelaxation in Deep Q- networks (SOR-DQN), Speedy Q-learning (SQL), Generalized SQL(GSQL), GSQL2, etc. [12]. Markov Decision Process (MDP) mathematically defines a discrete-time stochastic control process. It provides a theoretical framework for modeling decision-making in situations where outcomes are partly random and partly under the control of a decision-maker. Hence for approximating, and obtaining appropriate policy a technique used in MDP is Q-learning, which itself is prone to slow convergence as it takes samples from the whole space. As it is purely for online learning which is not bound to any models there lies its advantage as a modern technique DRL. The simulation-based methodology for solving MDPs, which like DP is also rooted in the Bellman equations, goes by several names and one such naming is reinforcement learning. Policy Gradient Methods with Function Approximation use the popular methods of Q-learning strongly to elucidate the opportunities with online learning. MDP elements are defined with a set of states, a set of actions, P probabilities/the dynamics, R the reward, and an initial state to start with. The agent gets a reward at each time step (rather than just a final reward called R aka Q-that defines a probability distribution governing the rewards thus received.

Problem Statement:

In the area of Cloud auto-scaling, it is fundamental to study the specific implications of representational choices (states and actions) and how they impact the performance of auto-scalers. For example, interesting questions to answer are (1) What information from a real Cloud environment is relevant to properly learning a policy (b) What could be an adequate representation of this information to accelerate the learning process given a large data set? On the other side, the problem of Cloud autoscaling is closely related to Multi-objective Optimization (MOO) [7]. Wang et al. explore the use of RL techniques for horizontal scaling in the Cloud. The idea of this work is further to learn policies capable of adjusting the infrastructure, achieving a balance between performance and costs. [8]. However, not many of the surveyed works propose a solution that covers either scaling or schedule, but not both [1]. Applying the modified SQL algorithms demonstrates that the agents learn faster from the environment in the online mode thereby allowing the over-utilization parameters that could be optimized [3][4][5]. That is proof that auto scalars with reinforcement bring agents closer to intelligence systems.

Authors contributions

The population of heuristic to the w parameter in the algorithm GSQL2.
Analyzed the mean square error deviations of the GSQL-w algorithm with w in a range of values.
Introduction of the new GSQL-wh framework for optimally selected w parameters.
Evaluation of the optimal value of w to help analyze the cases of over-utilization in time series data set, representing a cloud workload.

In the GSQL-w algorithm, the w-parameter [3] is for convergence of errors, defining a few options to consider, depending on the possible choice of its value. In a system that needs to realize the optimization of resources, the selection of w determines two cases.

0 < w 6 1 (Under-relaxation)
1 < w 6 w (Over-relaxation)

w being the convergence factor of choice that belongs to probability functions. The mean square errors were observed to be less thus proving the value of w relaxes the convergence of the agent to take appropriate action. GSQL2 is theoretically evaluated already with the w parameter to have an extended scope with heuristics. Our motivation is hence to perform the experiments by extending the GSQL-w for varying w, parallel also for several space and action values. This is our novel attempt at heuristics on w, the convergence parameter that was a limitation in [3]. The steps involved in our new framework are presented in the subsection.

Steps in GSQL-wh

The heuristics-based rule-of-thumb strategies shorten decision-making time. As it was observed for w=1 the update rules derived from the Q-learning were similar. Gradient descent is for minimizing the mean squared error between the target and current estimates. By modifying target values, the algorithms learn faster, ie., collect rewards quickly. This kind of performance is expected in an application where automatic resource allocations happen. The effective initial phase of learning is crucial in the overall system performance. The modification in this algorithm hence is to first choose this convergence parameter with exiting probability of action state pairs. In the next section, the authors discuss one such application that delves into early convergence ie., the automatic allocation of resources to Cloud applications.

Auto-scaling of Cloud Applications

Systems hosted on the cloud demands are based on the amount of real-time workload they must operate upon. Auto-scaling means automatic allocation and de-allocation to such applications as and when they need resources. The resource requirements of an application hosted are agreed upon with the cloud service provider. This definition of SLAs is based on cost and performance parameters like throughput and response time. The authors defined this kind of problem of the cloud systems to be modeled with the Markov Decision Process as explained here under.

1) State(S): A state at time t is (W,v). W is the workload of an application and v is the number of units of resource currently that is allocated.

[→] New state (W⁰,v⁰) means on how W⁰as the workload characteristics drives a previous value ie., v⁰= v + a

2) Action (A): A decision at time t, is an action to allocate or de-allocate a resource for the next time period t+1

3) Reward (R): A cost function like the one below defines the reward. S ×A×S → R.

Reward increased basically when the cost of the system reduces, hence it is a negative of the cost function. The authors define here how the cost per unit of resource is arrived at based on the response time of the application per (1). There exists an SLA as a penalty until which performance is pre-defined. This Service Level Agreement (SLA) is documented between the client and a cloud service provider.

C((W,v),a) := c × v + SLA_penalty(r) (1)

where c is the cost per unit of resource, r is the response time of the application and SLApenalty is redefined performance. Deep reinforcement learning became a solution to such problems as the authors have usually large actions and spaces. [2]. SOR-DQN proposals attempted and helped reduce the training time through environments’ differing statuses. GSQL-w proposed improvements for further error reduction by arriving at the empirical operators equal to the generalized bellman operator. The authors combined the idea of training time reduction by applying the various values with heuristics on the parameter w itself suggested by GSQL. The authors used the finite-time performance of the algorithm as defined in an existing PAC (Probably Approximately Correct) framework [5]. Algorithm convergence was assured for the over-relaxation as defined in (2).

1 < w 6 w⁰(Over − relaxation) (2)

where w tends to change over iterations and depends on the underlying MDP, the defined discount factors, etc., Further it is shown that for values of w greater than 1, GSQL-w is superior to Speedy Q-learning. Thus, the authors enhanced the GSQL to a level thereby applying heuristics for the 5 values of w. Our numerical experiments on the value of w evaluate the theory of successive relaxation on the bellman operator, which in turn propagates information throughout the space. MDPs the authors analyzed are of discrete state space sizes of 3, 5, and 10. The authors define further the details of this new framework as follows. The modified architecture from [7] forms an auto-scaling system based on DRL with a simulation introduced for the new GSQL-wh based on [11]. Fig. 1 illustrates the main building blocks of the GSQL-wh framework. While the application runs on a cloud platform, the modified GSQL-w guides resource provisioning through error convergence. More precisely, it follows the number of resources allocated to reach a certain goal, i.e.., to monitor, analyze, plan, and execute (MAPE) control loop [6], where different characteristics of the application (e.g. workload and response time) are continuously monitored, the satisfaction of system goals are checked, and accordingly, the resource allocation is adapted in case of deviations causing any lapse in efficiency. The goals (i.e., SLA, cost, response time) are reflected in the reward function that is set automatically from the environment parameters. An auto-scaling agent reads in the states at time t for a workload of the application and several units of resources currently allocated. The output from that agent is the action, which is considered as cost and performance functions rewarded to the cloud platform, that constitutes also as the main component. The monitoring function collects metrics and feeds the agent. An actuator issues adaptation commands from the heuristic controller (GSQL-wh) at each control time stamp/ interval. Overall, it is the application under test that is what makes use of the allocated resources to it. Illustrated in Fig. 1 adapted from [11] and [7].

2.3. Experiments

The setup with Python evaluates the proposed algorithm GSQL-wh. Auto-scaling cloud resources are based on the data generated by passing random state, action pairs of simulated workloads. The no of virtual machines that are allocated for applications will hence be a calculation that is based on requests for every second as represented in a time series data set. After the initial setup, and after the estimation error (Bellman equations’) calculations the authors arrived at a few example relaxation techniques based on pure w value [5]. The case the authors observed based on over relaxation on where the value of a parameter convergence happened with w could normally fit as explained in GSQL-w. With the randomizing parameter varied for each run, our algorithm converges the errors early in the training runs in the over-relaxation. This is demonstrated in the Fig 2., by plotting the mean squares arrived for 5 values of w. The average error of different RL algorithms at the convergence parameter w=1.3 for the states at 10 and actions at 3.

Illustrated in Fig. 2(a) shows the average error for the different algorithms plotted against the iteration number. The average error is defined as the difference between the optimal value function and its estimate based on the current Q function. This is returned by the algorithm as a modified Q value, which is averaged across the MDPs over 4 runs for different space action pairs viz., S in 3, 5, & 10. The actions were the matrix of values computed for every state. More the states, there are more probabilities, and hence w relaxation parameter widens when the authors read in more workloads in this setup. It is seen that the average errors of GSQL-wh decrease with the number of iterations at a faster rate as compared to GSQL-w. This empirically shows that our framework works well for several different MDPs and their superiority over others. The authors also demonstrate the clear advantage of GSQL-wh over GSQL2-w when the relaxation parameter w is large. For this, the authors have generated a few random MDPs with a discount factor of 0.9 and with the probability defined as (3):

P(i|i,a) = 0.9 ∀(i,a) such that w^∗= 1.3 (3)

The result is illustrated in Fig. 2(b). The authors studied the rate of convergence of the GQL-wh experimentally. The results are that the heuristics values of w determine the coverages as linear or non-linear. Further to validate the relaxation parameter and to observe the training time performance, the authors collected the results from the simulated workload. The linearity of the data set determined the training time reduction refer to Table 1.

Table 1. The performance of algorithms

Algorithm	Error (convergence)	Training time
SQL	3.20	0.50
GSQL	3.00	0.25
GSQL-w	2.80	0.05
GSQL-wh	2.50	0.05

The authors adopted the value of w between 1 to 2, and the results obtained were recorded. And when the authors looked at the approximation factor it was not promising to represent after 1.3 as it all converged to the same value function. Hence, the authors must define a w in such a way the convergence is appropriate for the data under test. This type of model-free learning with Q-learning develops scope for increasing agents’ decision-making power early on with the heuristics.

Our approach here is to use Deep Reinforcement learning for Autoscaling with the agents learning from environments. However, a major finding is that none of the surveyed works proposes a solution that covers partially scaling or scheduling or both problems. Hence, the inception of full-fledged auto scalers purely based on RL techniques for any type of Cloud application remains to be seen in the area. The authors proposed the optimal value of w to be between 1 to 2 for SQR-DQN that is what GSQL-w was improvised with to enhance the performance during the training phase. Through our study on a randomized sample as datasets, it was inferred that lowering the training cost in auto-scaling cloud applications was purely based on the optimal w values. The developed heuristics’ help for optimal agent policies is not changing beyond w=1.3. As future work for scalability of GSQL-wh, one can experiment with the different state spaces if one has simulators setup with an appropriate time series data set. As a final comment, it is important to note that RL is key for Distributed techniques [7]. In future work, one can evaluate OpenStack for distributed infrastructure in academic cloud environments.

FUNDING: This work is not funded by any agency or a university.

Research Data Policy and Data Availability

Data sets generated during the current study are available from the corresponding author on reasonable request.

Author Contribution

AUTHORS’ CONTRIBUTIONS:Lakshmi Sankaran: Conceptualization, Research, Data Curation, Formal Analysis, Methodology, Experiments, Editing. Saleema JS: Supervision, Conceptualization, Editing, & ReviewingBasem Suleiman: Conceptualization, Editing, & ReviewingAll authors read and approved the final manuscript.

D. Edsinger, “Auto-scaling cloud infrastructure with Reinforcement Learning A comparison between multiple RL algorithms to auto-scale resources in cloud infrastructure”, Chalmers University of Technology, Sweden 2018.
C. Bitsakos, I. Konstantinou, and N. Koziris, “DERP: A Deep Reinforcement Learning Cloud System for Elastic Resource Provisioning,” in 2018 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), vol. 2018-Decem, Dec 2018, pp. 21–29,10.1109/CloudCom2018.2018.00020.
I. John and S. Bhatnagar, “Deep Reinforcement Learning with Successive Over-Relaxation and its Application in Autoscaling Cloud Resources,” International Joint Conference on Neural Networks (IJCNN), 2020, pp. 1–6, 10.1109/IJCNN48605.2020.9206598.
P. Singh, P. Gupta, K. Jyoti, and A. Nayyar, “Research on Auto-Scaling of Web Applications in Cloud: Survey, Trends and Future Directions,” Scalable Comput. Pract. Exp., vol. 20, no. 2, pp. 399–432, May 2019,10.12694/scope.v20i2.1537.
I. John, C. Kamanchi, and S. Bhatnagar, “Generalized Speedy Q-Learning,” IEEE Control Syst. Lett., vol. 4, no. 3, pp. 524–529, Jul. 2020,10.1109/LCSYS.2020.2970555.
M. Imdoukh, I. Ahmad, and M. G. Alfailakawi, “Machine learning-based auto-scaling for containerized applications,” Neural Comput. Appl., vol. 32, no. 13, pp. 9745–9760, Jul. 2020,https://doi.org/10.1007/s00521-019-04507-z.
Y. Gar´ı, D. A. Monge, E. Pacini, C. Mateos, and C. Garc´ıa Garino, “Reinforcement learning-based application Autoscaling in the Cloud: A survey,” Eng. Appl. Artif. Intell., vol. 102, p. 104288, Jun. 2021, 10.1016/j.engappai.2021.104288.
Z. Wang, C. Gwon, T. Oates, and A. Iezzi, “Automated Cloud Provisioning on AWS using Deep Reinforcement Learning,”, Sep. ,http://arxiv.org/abs/1709.04305.
A. Iranmanesh and H. R. Naji, “DCHG-TS: a deadline-constrained and cost-effective hybrid genetic algorithm for scientific workflow scheduling in cloud computing,” Cluster Comput., vol. 24, no. 2, pp. 667–681, Jun. 2021,https://doi.org/10.1007/s10586-020-031458.
A. Yenel, V. Podolskiy, and M. Gerndt, “Predictive Auto Scaling Scheduling Application,” no. October 2018.
P. Jamshidi, A. Sharifloo, C. Pahl, H. Arabnejad, A. Metzger, and G. Estrada, “Fuzzy Self-Learning Controllers for Elasticity Management in Dynamic Cloud Architectures,” 12th International ACM SIGSOFT Conf. on QoSA, Apr. 2016, pp. 70–79, 10.1109/QoSA.2016.13.
M. G. Azar, M. Ghavamzadeh, S. Project, R. Munos, S. Project, and H. J. Kappen, “Speedy Q-Learning,” in Advances in Neural Information Processing Systems, Spain, Dec. 2011, pp. 2411–2419, https://hal.inria.fr/hal-00830140/.
Y. Al-Dhuraibi, F. Paraiso, N. Djarallah, and P. Merle, “Elasticity in Cloud Computing: State of the Art and Research Challenges,” IEEE Trans. Serv. Comput., vol. 11, no. 2, pp. 430–447, Mar. 2018, doi: 10.1109/TSC.2017.2711009.

No competing interests reported.

Download PDF

Editor assigned by journal
27 Mar, 2024
Submission checks completed at journal
11 Mar, 2024
First submitted to journal
29 Feb, 2024

You are reading this latest preprint version

Optimizing the agent decisions for a Cloud actuator using Deep reinforcement learning

Status:

Version 1

Abstract

Figures

INTRODUCTION

Related work

Problem Statement:

Authors contributions

METHODS

Steps in GSQL-wh

Auto-scaling of Cloud Applications

2.3. Experiments

RESULTS AND DISCUSSION

CONCLUSION

Declarations

Author Contribution

References

Additional Declarations

Status:

Version 1