Real-time Multiple-Workflow Scheduling in Cloud Environment

—With the development of cloud computing, an increasing number of applications in different fields have been deployed to the cloud. In this process, the real-time scheduling of multiple workflows composed of tasks from these different applications must consider various influencing factors which strongly affect scheduling performance. This paper proposes a real-time multiple-workflow scheduling (RMWS) scheme to schedule workflows dynamically with minimum cost under different deadline constraints. Due to the uncertainty of workflow arrival time and specification, RMWS dynamically allocates tasks and divides the scheduling process into three stages. First, when a new workflow arrives, the latest start time and the latest finish time of each task are calculated according to the deadline, and the subdeadline of each task is obtained by probabilistic upward ranking. Then, each ready task is allocated according to its subdeadline and the increased cost of the virtual machine (VM). Meanwhile, only one waiting task can be assigned to each VM to reduce delay fluctuations. Finally, when the task is completed on the assigned VM, all the parameters of the relevant tasks are updated before allocating them to appropriate VMs. The experimental results based on four real-world workflow traces show that the proposed algorithm is superior to two state-of-the-art algorithms in terms of total rental cost, resource utilization, success rate and deadline deviation under different conditions.


INTRODUCTION
loud computing facilitates IaaS, PaaS and SaaS modes by adopting virtualization technology, which has the advantages of heterogeneity, convenience and scalability and provides external services through the network [1]. In the IaaS model, users can customize the use of different types and amounts of computing resources on their actual needs, usually in the form of VM and pay based on rental hours. This mode, which is different from traditional computing platforms, not only avoids investment and maintenance costs by users for hardware but also greatly improves the utilization rate of infrastructure in cloud platforms, thus benefiting both users and cloud providers. Due to the advantages of this new computing mode, with its pay-as-you-go basis, efficiency and scalability, increasingly more applications requiring large-scale computing, such as weather forecasting, earthquake prediction and gene sequencing, are being migrated to cloud environments [2].
These different applications, which require large-scale computing, are typically represented as workflows in the cloud. A workflow is mainly modeled as a Directed Acyclic Graph (DAG), in which the vertices represent various tasks, and the edges represent the sequential relationships between tasks [3]. The weights of the vertices and edges represent the calculation quantity of the tasks and the transfer of data between tasks. As an NP-hard problem, workflow scheduling has always been an important research field [4]. Furthermore, workflow scheduling in cloud computing refers to assigning and executing tasks of workflow to appropriate VMs to meet user-defined Quality of Service (QoS) requirements. Because it is difficult to solve this NP-hard problem directly, different methods are usually adopted to obtain an approximate optimal solution. When scheduling workflows in grid, cluster and other environments, the most popular optimization objective is to shorten the completion time of the workflow as much as possible. Round Robin, Max-Min, Min-Min and other traditional scheduling algorithms have been adopted to achieve better solutions. Further, both reducing task execution time and considering implementation costs are important for workflow scheduling in the cloud. Moreover, given the fluctuations of hardware performances in cloud environments, the task execution time of workflows can be delayed, thereby requiring dynamic scheduling.
However, many studies on workflow scheduling fail to fully consider the characteristics of cloud computing, such as its heterogeneity, unlimited resources, and fluctuating performance. Most studies on this topic have emphasized the scheduling of a single workflow in a cloud, whereas few studies have focused on multiple-workflow scheduling in real time [5] [6]. Due to the sharing method, tasks in multiple workflows will be implemented on the same resource simultaneously. Different deadlines for multiple workflows will make real-time scheduling in cloud computing more difficult than for single workflow. In addition, VM performance fluctuations caused by the hardware devices, network environment and other factors will affect the makespan and cost of workflow scheduling [7] [8]. Therefore, various factors, such as the task execution time, data transfer time, and VM rental cost, need to be considered for multiple-workflow scheduling in cloud environments.
Given the nature of cloud computing, this paper pro-C poses a Real-time Multiple-Workflow Scheduling (RMWS) scheme to minimize the rental cost of leased VMs under deadline constraints. The main contributions of this paper are as follows: 1. An online scheduling algorithm for multiple workflows conforming to different applications in cloud computing is proposed.
2. All the characteristics of cloud computing such as dynamic expansion, heterogeneity, VM performance deviation, and bandwidth variation are considered to reduce their impacts on the makespan and cost of scheduling. 3. A simulation cloud environment is constructed, and extensive experiments are implemented based on real workflow traces. Via comparisons with two other state-ofthe-art algorithms, it shows that the proposed algorithm in this paper achieves superior performance under various conditions. The remainder of the paper is organized as follows: Section 2 surveys and analyses relevant works. Section 3 presents the platform structure and the scheduling problem. Then, the real-time multiple-workflow scheduling algorithm is proposed in Section 4, followed by the related experiments carried out in Section 5. Finally, conclusions and future work are given in Section 6.

RELATED WORK
As an NP-hard problem, workflow scheduling on distributed resources has been studied extensively. In this section, the existing scheduling strategies are briefly reviewed and divided into three categories, i.e., scheduling in a distributed environment, single-workflow scheduling in a cloud and multiple-workflow scheduling in a cloud.

Scheduling in a Distributed Environment
Distributed system, which provides users with highperformance and high reliability resource sharing services, can be seen as an assemble of multiple individual compute nodes interconnected through the network [9] [10]. A wide spectrum of approaches, such as integerprogramming, Pareto optimality and graph theory, are used for scheduling in multiprocessor systems and distributed environments [11]. For instance, Topcuoglu et al. [12] proposed a Heterogeneous Earliest-Finish-Time (HEFT) algorithm to schedule tasks among a fixed number of heterogeneous processors. According to the average cost of computation and communication, the HEFT algorithm calculates the rank of each task and sorts them in descending order of priority. Then, tasks are successively selected and assigned to the appropriate processor, which can minimize the completion time of the selected task. In addition, some similar methods, such as RHEFT [13] and BHEFT [14], have been extended from HEFT for the scheduling optimization problem in different environments.
Moreover, the work in [15] introduced two schedulers based on integer linear programming to schedule VMs in grid resources and tasks on those VMs. Based on each task was exactly scheduled on a single host by the first scheduler, another scheduler applied relaxation techniques to shorten the execution time without significantly reducing the scheduling quality. Yu et al. [16] presented a cost-based workflow scheduling algorithm to minimize the execution cost under deadline constraints. They adopted task partitioning and total deadline assignment to generate the optimal execution scheme and rescheduling. To minimize the total execution cost of branch tasks, the Markov Decision Process (MDP) [17] approach is used to decide on which service to execute each task after the completion of its parent task. In another work, Yu et al. [18] adapted the Genetic Algorithm (GA) [19] to schedule workflows in utility grids. In their method, each individual is coded by a 2D string that includes the numbers of services and the order of tasks in each service. Then, the algorithm measures the quality of the individuals in the population based on the cost fitness and time fitness, which represent the optimization goal. Subsequently, the authors adopted random selection and exchange for the operations of crossover and mutation to generate new individuals.
Although many studies have used different methods to study the scheduling problem in distributed systems, most of them have focused on the single-user and singleworkflow mode, and the main optimization goal is to reduce the task execution time in a relatively stable environment.

Scheduling for Single Workflows in a Cloud
Unlike the traditional computing model, cloud computing, given its characteristic on-demand procurement, heterogeneity, elasticity and dynamics, provides a simple and efficient platform for different applications but creates additional challenges to workflow scheduling. Correspondingly, many novel methods have been proposed to solve this problem.
Using the concept of critical path in DAGs, an algorithm called IaaS Cloud Partial Critical Paths (IC-PCP) [20] has been applied to workflow scheduling in cloud computing. After calculating the latest finish time of each task, IC-PCP recursively assigns tasks on the same partial critical path to the least expensive instance that can finish the chosen task before its latest finish time, and this process is repeated until all tasks are assigned. Calheiros et al. [21] used the idle time of leased resources and the remaining budget to mitigate the effects of performance variations in public clouds. Their method, which is an Enhanced IC-PCP with Replication (EIPR) algorithm, attempts to increase the likelihood of completing the execution of a workflow under a user-defined deadline via three distinct processes. The first is to combine task scheduling and provisioned resources, which means determining the number and type of VMs and the order and placement of tasks on VMs. The next step is adjusting the data transfer process to dictate the start and stop times of the VMs. Finally, EIPR attempts to increase the performance of task execution by replicating tasks on different VMs.
With the increasing complexity, proliferation, and ambition of cloud computing, heuristic algorithms, such as Simulated Annealing (SA) [22], Ant Colony Optimization (ACO) [23], Particle Swarm Optimization (PSO) [24], and other nature-inspired methods, have been used for workflow scheduling [25]. Rodriguez et al. [26] adjusted the PSO to solve the scientific workflow scheduling problem with cost minimization and deadline constraints in a cloud environment. With the update of particle position and velocity, the task execution time and the data transfer time on different VMs were constantly calculated during the iteration process until the globally optimal solution was generated. In [27], a Temporal Task Scheduling Algorithm (TTSA) used hybrid simulated-annealing PSO to solve the cost minimization scheduling problem in hybrid clouds. The algorithm decomposes each arriving task into multiple parallelized subtasks to complete them within one time slot. The constraint cost minimization problem was converted into an unconstrained problem in each time slot. Based on the advantages and disadvantages of the SA and PSO algorithms, their method updates the position of each particle according to the metropolis acceptance criterion, which can avoid the algorithm falling into a local optimum.

Scheduling for Multiple Workflows in a Cloud
Compared with single-workflow scheduling, multipleworkflow scheduling in clouds is more complex. When applying various technologies to obtain feasible solutions, such scheduling not only must resolve the relationships among tasks in the same workflow but also should consider the structures of different workflows.
Malawski et al. [28] used exponential scoring to characterize the optimization problem to complete as many workflows as possible based on their priorities, budget and deadline constraints. In their first algorithm named Dynamic Provisioning Dynamic Scheduling (DPDS), VMs are created and terminated based on the complete billing cycle, budget and deadline constraints in the provisioning phase. And all tasks with different priorities are added to a queue and successively scheduled to idle VMs. Similarly, Wang et al. [29] proposed a Mixed-parallel Online Workflow Scheduling (MOWS), in which the scheduling phase is performed based on task prioritizing, waiting queue scheduling, task rearrangement, and task allocation. And four methods, named shortest-workflow first, priority-based backfilling, preemptive task execution and All-EFT task allocation, are used in these phases. Zhou et al. [30] proposed a Dyna scheduling system to minimize costs under user-specified probabilistic deadline constraints in an IaaS cloud. The Dyna system defines a hybrid instance configuration of a task as an n-dimension vector, which is used for the sequential execution of the task to meet the deadline. Then, their method applies a two-step approach to find an efficient solution while sat-isfying the probabilistic performance requirement.
Toward controlling the number of tasks directly waiting for each VM, Chen et al. [31] developed an Uncertainty-aware Online Scheduling Algorithm (ROSA) to schedule workflows in the cloud. ROSA obtains the latest start and finish times of all tasks in the new workflow according to the task execution time and data transfer time. Then, the algorithm allocates the ready tasks to service instances that minimize the expected cost under the deadline, while all the non-mapped tasks are moved to the task pool. If a service instance has more than one awaiting task, the first task will be executed after the service instance finishes a task. Liu et al. [32] proposed an Online Multiple-workflow Scheduling Framework named NOSF to schedule multiple workflows in real time. The algorithm can be divided into three parts: workflow preprocessing, VM allocation and feedback process. First, NOSF calculates the initial parameters of each task based on the structures and deadlines of the randomly arriving workflows. Then, the ready tasks are scheduled by the subdeadline and increasing cost, while other tasks are stored in a task pool. Finally, the feedback process recalculates the original parameters of certain relevant tasks after a task has completed.
Though there are other scheduling strategies have been widely studied in the workflow scheduling problem [33][34][35][36][37][38], most of the existing scheduling algorithms focus on fixed resource allocation optimization whereas few can realize online scheduling. Meanwhile, almost all existing scheduling algorithms fail to completely incorporate all the characteristics of cloud, such as dynamic expansion, heterogeneity, VM performance deviation and bandwidth variation, or ignore the data transfer time between tasks of a workflow. Based on the shortcomings of these studies and our previous researches [39][40][41][42], this paper proposes a cost-efficient method inspired by the above-mentioned literature for the scheduling of multiple workflows in IaaS clouds. Considered all the characteristics of workflow scheduling in cloud computing, the proposed algorithm divides the online scheduling process into three phases and adjusts the priority of relevant tasks according to the feedback process of the completed tasks. The main processes are described below.

PROBLEM DEFINITION
This section introduces the model of cloud platforms and workflows and then describes the architecture of multiple-workflow scheduling in the cloud. Based on the model and architecture, the multiple-workflow scheduling problem is formulated. Some important symbols used in this study are provided in Table 1.

System Model
After users submit their workflows and requirements, the cloud platform allocates the tasks to appropriate resources and charges the user according to the usage time after the workflows are completed. In this process, the IaaS cloud platform delivers IT infrastructure based on virtual resources as a commodity to customers. These virtual resources providing services to end users are commonly measured in terms of CPU type, memory, storage, and network bandwidth; this allows VMs to be classified based on rankings and costs. A VM with a higher ranking provides a higher task execution speed; however, it is more expensive to rent. We use a weight ( ) to represent the processing performance of VMs with different rankings. For a VM of type , we define its weight ( ) as the ratio of its task execution time to the task execution time of the VM with the highest level. Furthermore, a new VM must take time to boot before it can be used normally, and the shutdown time when a VM is released is ignored because of the negligible impact on scheduling.
In our proposed model, is used to represent the mth VM of type k, and ( ) represents the rental price per unit time for a VM of type k. This means that a VM with a higher CPU speed usually has a higher rental price, and the rental time is generally billed in multiples of a unit of time. For example, if the unit of time is an hour, a VM leased for 1.3 hours will be charged for two time units.

Workflow Model
In this paper, different applications submitted by users in real time are presented in the form of a workflow, which constitutes a dynamic workflow set = { 1 , 2 , … , }. Each workflow has a different arrival time and deadline . Furthermore, workflow can be regarded as a DAG = ( , ), in which represents the set of tasks and represents the set of edges between tasks with directed relationships. In addition, is used to represent the jth task in . If tasks and have a direct dependence, which is represented by an edge , , task is called the immediate predecessor or parent task of task , while task is called the immediate successor or child task of task .
( ) and ( ) represent the set of all the immediate predecessors and all the immediate successors of task , respectively. Task can only start executing after all the tasks in ( ) have been completed and after all the transferred data have been received. Fig. 1 shows two workflows with different structures. For example, task 25 has two immediate predecessors 22 and 23 , which means that task 25 cannot be executed unless tasks 22 and 23 are both finished and after all the output data have been received.

Scheduling Architecture
The scheduling architecture in the cloud environment is shown in Fig. 2. Different Users may randomly submit new workflows with various deadlines to the cloud at any time. Then, the scheduler module assigns tasks of each arrived workflow to different VMs provided by the IaaS cloud. After all tasks in a submitted workflow have been completed, the result is returned to the corresponding user. Furthermore, the scheduler can be divided into four units: workflow initialization, task pooling, task assignment and VM management. The main processes of these four units are as follows: •Workflow initialization: If a new workflow has arrived, the latest start time, the latest finish time and the subdeadline of each task are calculated based on the deadline and structure of the workflow. After tasks with no immediate predecessors are marked as ready tasks and after other tasks are marked as waiting tasks, all tasks are sent to the task pool.
•Task pooling: All the tasks whereby their immediate predecessors have not been completed will be stored in the task pool and are regarded as waiting tasks. If all the immediate predecessors of a task are completed, the task status will be changed to the ready state.
•Task assignment: If there are ready tasks in the task pool, this process will allocate them to the appropriate VMs by the proposed algorithm. The algorithm first tries to assign tasks to the active VM that can satisfy the subdeadline and has the minimum cost. If all the active VMs cannot meet these requirements, a new VM will be leased by a VM manager.
•VM management: The process of leasing and releasing VMs is based on the execution of tasks. When existing VMs cannot meet the requirements of a ready task, a new suitable VM is leased. When all tasks on a VM are completed and after all data have been transferred, idle VMs will be released.
To minimize the fluctuations caused by task execution time delays, only one waiting task can be allocated on the same VM in each phase. Each VM can only execute one task at a time with a non-preemptive model. Furthermore, after a task is completed on the assigned VM, all parameters, such as the latest start time, the latest finish time, and the subdeadline of the relevant tasks in the task pool, are updated.

Problem Formulation
The purpose of the proposed algorithm in this paper is to schedule workflows in real time while minimizing the total rental cost under different deadline constraints. Based on the above model, we make the following definitions.
1) Task execution time and data transfer time. When task is scheduled on VM , the execution time of depends on its workload and on the computing performance of . Meanwhile, the data transfer time between and its parent depends on the size of the data transmitted and the bandwidth between VMs if and are assigned to different VMs. This means that the data transfer time will be zero if and are assigned to the same VM.
Moreover, the CPU performance of VMs and the bandwidth of the cloud cannot always be assumed constant, and fluctuations will cause uncertainty in the task execution time and the data transfer time. The task execution time is always regarded as a stochastic variable with independent and normal distributions [43] [44]. For an expected value and variance 2 , ( , 2 ) can denote the normal distribution of the variable task execution time. We define three parameters, , and , to represent the base execution time, the actual execution time and the predicted execution time, respectively. Furthermore, is the expected task execution time when a task is executed under the fixed computing performance of the VM with the highest level, is the actual task execution time, which can be obtained when the task is ready assigned to a VM with the computing performance at the corresponding time, and is the predicted task execution time if a task will be executed on a VM with weight ( ). Similarly, we use , , , and , to represent the base data transfer time, the actual data transfer time and the predicted data transfer time, respectively.
2) Latest finish time and latest start time . The latest finish time of task is the time in which can complete its execution on the fastest VM such that the finish time of workflow is no greater than its deadline . The definition of the latest start time is similar. and are recursively obtained as 3) Predicted start time and predicted finish time . The predicted start time of task on VM depends on the arrive time of workflow , the actual finish time of all predecessors of , the predicted data transfer time , between and all its predecessors, and the available time of . The recurrence relations are where ( ) and represent the available time of VM and the actual start time of task , respectively. 4) VM available time ( ). The available time of VM is the time at which a new task can be executed on the VM. If task is the waiting task on , the ( ) will be updated to . The ( ) of a new leased VM is equal to the initial boot time (init_time). 5) VM lease start time ( ), VM lease finish time ( ) and VM rental cost ( ). The lease start time of VM is the time at which is ready to execute tasks and is equal to if task is the first executed task on . The VM lease finish time is the time at which is released by the VM manager and can be seen as the actual finish time of the last task executed

on
. Therefore, the rental cost of can be calculated as follows: 6) The finish time ( ) and total execution cost ( ). The finish time of workflow is equal to the latest finish time of all tasks in , and the total execution cost of all the workflows is the total rental cost for all of the leased VMs.
Finally, the problem of workflow scheduling can be defined as finding a feasible solution for the workflows such that ( ) is minimized, and ( ) does not exceed the deadline of workflow . The problem is defined as Based on Eqs. (6) and (7), the objective of multipleworkflow scheduling optimization in this paper is to minimize the overall rental cost of leased VMs while meeting the deadline constraints of each workflow. Furthermore, the decision variables in the optimization process can be seen as the unit rental price and the lease time of each VM. Therefore, when selecting the appropriate VM for each ready task, the start execution time of the task should be shortened as much as possible to minimize the rental cost by reducing the idle time of the allocated VM and improving the resource utilization.

THE PROPOSED ALGORITHM
In this section, the proposed algorithm, named RMWS, is presented in detail. We divide the algorithm into three stages: workflow initialization, task allocation and feedback processing. The detailed process of each stage is as follows.

Overview
According to the architecture of workflow scheduling in the cloud, the workflow scheduler can be divided into four main parts: workflow initialization, the task pool, task assignment and the VM manager. Based on this, the core of the proposed algorithm contains three processes: the initialization of the workflow, the allocation of tasks, and the feedback of task completion.
When a user submits a new workflow to the cloud platform, the workflow initialization process is triggered. The process preprocesses all tasks according to the arrival time, deadline and DAG structure, including the calculation of the latest start time, the latest finish time and the subdeadline of each task. Subsequently, the algorithm sends the ready tasks that have no immediate predecessors to the task allocation process, while other tasks are stored in the task pool. Then, the task allocation process first attempts to assign the selected task to an appropriate active VM. If all the active VMs fail to meet the condi-tions, a new VM will be leased to assign the selected task. When a task is finished, the latest start time, latest finish time and the subdeadline of its immediate successors for which their parent tasks have completed will be updated during feedback processing.

Workflow Initialization
The main purpose of workflow initialization is to determine the initial priority of tasks in the arriving workflow, including the main parameters such as the latest start time, the latest finish time, and the subdeadlines. The pseudocode is shown in Algorithm 1.

Algorithm 1. WorkflowInitialization( )
Input: An arriving workflow with arrival time and deadline Output: A queue saving the ready tasks in , the task pool contains the waiting tasks 1. ←Ø 2.
for each task from the exit tasks to the entry tasks 3.
calculate and by Eqs. (1) and (2)  Then, probabilistic upward rank is used to generate a subdeadline for each task (line 4) [45]. The upward rank of task can be defined as the longest path from it to the exit task , and is represented recursively as Eq. (8) shows that the upward rank of a task is related to its execution time and data transfer time between the task and its immediate successors. However, when a task and its immediate parents are assigned to the same VM, the corresponding data transfer time is zero and will not affect the upward rank of the task. Therefore, a probabilistic variable is introduced to eliminate possible interference. The probabilistic upward rank of a task is defined as where is a Boolean variable indicating whether to consider the data transfer time.
can be calculated according to Eqs. (10) and (11), in which is the computation-to-communication ratio, is a parameter greater than 1, and is a random number that generates in [0,1). The larger is, the more probability is 1, and vice versa.
According to the above definition, a larger means that the algorithm is more likely to consider the data transfer time when calculating . The data transfer time will never be considered if is set to 1. Based on this, the subdeadline of task can be calculated proportionally as where means the probabilistic upward rank of the entry task . Finally, all the entry tasks are added to the ready queue , while other tasks are stored in task pool (lines 6-7). After that, the ready tasks in queue will be scheduled through the task allocation process (line 8).

Task Allocation
The task allocation process is the core of multipleworkflow scheduling. It not only determines the makespan and rental cost of each workflow but also affects the resource utilization in the cloud platform. If new ready tasks are generated after a new workflow has arrived or after an assigned task is completed, the task allocation process shown in Algorithm 2 will be triggered. The main purpose of Algorithm 2 is to find suitable assigned VMs for the ready tasks to minimize the rental cost under the subdeadline constraints. First, all tasks in the ready queue are sorted in ascending order according to the latest finish time (line 1). Then, the first task in queue will be selected in turn, and the predicted start time , the predicted finish time , and the predicted increased cost ( ) of on each active VM with no awaiting tasks are calculated (lines 3-9). Moreover, the predicted start time is the maximum time between the available time ( ) and the data transfer time from ( ) to task (line 6). The predicted finish time is the sum of the predicted start time and the predicted execution time of task on the selected VM (line 7). The predicted increased cost ( ) is the difference in cost before and after assigning to (lines 8-9). The VM with the minimal increased cost under the subdeadline constraint of will be selected (lines 10-13). Next, if the appropriate VM cannot be found from the active VMs, , , and ( ) with different VM types are calculated (lines [17][18][19]. A new VM of type ℎ , on which has the minimum rental cost under the subdeadline, or a new VM with the highest rank K, will be leased as the selected VM (lines 20-28). Finally, if the selected VM is idle, the task will begin to execute after receiving all the transferred data from its immediate predecessors (line 31); otherwise, the task will be the waiting task on the selected VM (line 33).

Feedback Processing
Algorithm 3 shows the feedback process, which will be triggered after a task is completed on the assigned VM . First, the waiting task will be executed after receiving all the transferred data from its immediately parent (lines 2-4). Subsequently, for each immediate successor of , if all of ( ) have been completed, the task will be considered a ready task (lines 5-6). After the latest start time, the latest finish time, and the subdeadline of are calculated, will be moved from into (lines 7-8). Finally, the task allocation process is called for scheduling the tasks in (line 11).

Input:
A completed task and its assigned VM , the task pool Output: A queue saving the ready tasks from ( ) 1. ←Ø 2.
if there exists a waiting task on 3.
execute the waiting task after receiving the transferred data 4. endif 5.

Algorithm analysis
According to the formal definition of multiple-workflow scheduling optimization problem in Section 3, the data transmission time between a task and its predecessors should be considered while calculating the predicted start time of the task. Meanwhile, the tasks on the critical path plays a decisive role in the total execution time of the workflow. We use the concept of probabilistic upward rank to generate the subdeadline for each task on the critical path according to the deadline of workflow, and realize the constraints in Eq. (7). Based on these, the latest start time, latest finish time and subdeadline of ready tasks are calculated or updated respectively in Algorithms 1 and 3, and the appropriate VM is selected according to these indicators. For example, the VM with higher performance and higher rental cost will be selected when the task to be assigned has a tight subdeadline, and vice versa. Therefore, the rental cost of all leased VMs can be minimized under the deadline constraint of workflows by the real-time adjustment feedback of these three parameters.
In addition, we define the actual execution time of tasks as a random variable according to the performance fluctuation of VMs. It means that the exact execution time of tasks cannot be obtained by the processing performance of VMs before task allocation, which is also an important factor in the scheduling optimization problem. To reduce the fluctuation caused by the change of VM performance, only one waiting task is assigned to each active VM at the same time in Algorithm 2, which avoids the delay of subsequent tasks. Therefore, the proposed algorithm not only meets the subdeadline constraint of each task, but also optimizes the total rental cost by reducing the idle waiting time of active VMs.

Time Complexity
Theorem1. The time complexity of scheduling an arriving workflow using the proposed algorithm RMWS is ( 2 + • ( ) + • + • • ) , where represents the task number, is the maximum in-degree of a task among , is the maximum out-degree of a task among , and denotes the number of active VMs. Proof. The scheduling process for an arriving workflow can be divided into three parts: calculating parameters, searching ready tasks, and assigning VMs. In the DAG composed of tasks, the maximum number of dependencies is ( − 1)/2 , the time to calculate , and of each task is ( 2 ) (lines 2-5 in Algorithm 1 and line 7 in Algorithm 3). Hence, the time complexity of the first part is ( 2 ).
The search of ready tasks is performed when a new workflow arrives and when a task is completed. For the former, a task with no in-edges will be seen as a ready task, and it requires time ( ) to traverse all tasks to find the ready tasks (line 6 in Algorithm 1). When a task t is completed, the algorithm requires time ( ) to traverse all immediate successors. Moreover, the algorithm needs to determine if each immediate successor is ready by examining whether the immediate predecessors have been completed, which requires time ( ) (lines 5-10 in Algorithm 3). Thus, the time complexity of searching for ready tasks is ( )+ ( • ).
When assigning the ready tasks to VMs, the algorithm needs time ( • log( )) to reorder the tasks by merging sort [46] (line 1 in Algorithm 2). For each task in , the algorithm traverses all active VMs and each immediate predecessor of to calculate , which requires time ( ) (lines 5-14 in Algorithm 2). Thus, the corresponding time complexity is ( • • ) while the maximum number of tasks in is . In addition, the time required to traverse each VM type can be regarded as a constant because the number of VM types is far less than that of leased VMs (lines 15-29 in Algorithm 2).
In summary, the time complexity of RMWS for scheduling an arriving workflow is . This proves the theorem.

EXPERIMENTS
Based on the workflow scheduling model and algorithm proposed in this paper, a simulation platform is constructed to simulate the real-time multiple-workflow scheduling in the cloud. The performance of the proposed algorithm is analyzed based on the simulation results. The details of the experiments are discussed as follows.

Workflows
Four real workflow traces, named LIGO, GENOME, Cybershake and SIPHT, are used as the experimental input data. These workflows have different structures and characteristics such as tree structures, pipelines, computing intensity, and I/O intensity. According to the number of tasks contained, we divide each type of workflow into small (approximately 50 tasks), medium (approximately 200 tasks) and large (approximately 1000 tasks) workflows. There are 12 different workflows used in the experiments. A simplified structure of each workflow is shown in Fig. 3, and a detailed description of these workflows is presented in [47].
Synthetic workflows with similar structures are created by workflow generator 1 . The generated workflows are represented in the form of a DAG in the XML (DAX) format, which contains tasks, dependency edges, transfer data and other information. Then, the experimental workflows can be constructed as DAX files.

Baseline Algorithms
To evaluate the performance of the proposed algorithm RMWS, we evaluated it along with ROSA [31] and NOSF [32], which are two online algorithms for multipleworkflow scheduling in the cloud.
ROSA estimates the task execution time and data transfer time by the quantile of a normal distribution. For randomly arriving workflows, the algorithm calculates the predicted latest start time and the predicted latest finish time of all tasks of the workflows according to their deadlines. Then, after the non-descending order of the predicted latest start time of the ready tasks is determined, a VM that has the lowest cost and satisfies the set conditions is selected to allocate each task.
NOSF uses a variance factor to simulate the normal distribution of the task execution time. After a new workflow arrives at a random time following a Poisson distribution, the algorithm first calculates the relevant parameters of all tasks such as the earliest start time, the earliest completion time and the subdeadline. Then, after sorting the tasks in a non-descending order according to the priority, the ready tasks are allocated to the VMs with the minimum idle time according to the predicted completion time and the predicted cost. When the assigned tasks are completed, the priority parameters of their relevant successors are updated.

Evaluation Metrics
The core scheduling goal of multiple-workflow scheduling in the cloud is to minimize the rental cost of VMs under deadline constraints; thus, we use Eqs. (6) and (7) to obtain the total rental cost. In addition, to better evaluate the scheduling performance of the algorithms, other indicators are considered as follows: 1) Resource utilization We use the proportion of the VM usage time in its lease time to express the resource utilization, which can be defined as ) is equal to the sum of the execution times of all tasks assigned to , and ( ) is equal to the time from initial availability to the release of , which includes the execution time and the idle time.
2) Success rate The success rate can be used to indicate the success and failure of workflow scheduling and can be defined as where ̃ represents the number of workflows that satisfied the deadlines, and is the number of workflows.

3) Deadline deviation
The arrive time, execution time and deadline of each workflow are used to reflect the average duration percentage of workflow deadlines exceeded, defined as where represents the execution time of the workflow , which can be obtained by subtracting the minimum task start time from the maximum task finish time of . 4) Scheduling time Scheduling time, which is calculated as the ratio of the total scheduling time to the total number of tasks, can be used to show the allocation efficiency of each task. For each task, the scheduling time mainly includes the time to calculate the priority parameters and the time to select the appropriate VM.

Experimental Setup
We implemented a discrete simulator using Java and deployed it in a computer with a 3.3 GHz Intel Core i5 processor and 12 GB of memory. The experiments simulated a cloud environment with a single datacenter and eight different VM types. The configurations and prices of the VMs, based on the Amazon EC2 services 1 , are shown in Table 2. During the experiments, VMs of different types can be dynamically and indefinitely leased without any restrictions, and it is assumed that the VM can satisfy all workflow execution environments after initialization. Additionally, based on Amazon EC2, the boot time of a VM is set to 97 s, the bill unit time is 60 min, and the bandwidth between VMs is 1000 Mbps.
To better simulate multiple-workflow scheduling in real-time environments, the relevant parameters are defined as follows: •Workflow arrival time: Different workflows arrive at the cloud platform dynamically; thus, some of the 12 different workflows mentioned above are randomly selected and submitted to the platform at random intervals. Moreover, the arrival time interval is set to follow a Poisson distribution with = 0.2.
•Task execution time: Due to the performance deviations of the VM, the actual execution time of a task is assumed to follow a normal distribution. Furthermore, when task is assigned to VM , the actual execution time can be expressed as 11 ( , ) * ( ) where 1 corresponds to the expected value and 1 means the standard deviation. They can be denoted as  where the variance factor represents the variation of the task base execution time. In the experiments, the actual execution time of on is obtained from a normal random number generator with the expected value of 1 and the standard deviation of 1 .
•Data transfer time: Influenced by the network environment, the data transfer time between tasks is also considered to obey a normal distribution. Similar to the task execution time, the data transfer time in the experiments is obtained from a normal random number generator with the expected value of 2 and the standard deviation of 2 . These values are defined as where the variance factor represents the variation of the data transfer time.
•Workflow deadline: A benchmark deadline with the shortest makespan is used to assign deadlines to workflows.
is defined as the makespan of a workflow when each workflow task is executed on a distinct VM of the fastest type, and the data transfer times among tasks are ignored. Then, the deadline of workflow can be calculated as * where denotes the deadline factor. In addition, to make the result of the three algorithms more accurate and credible, the predicted execution time and the predicted data transfer time , in the task allocation processing are set as  (24) In the experiments, we evaluate the performance of algorithms using different values of the main parameters, which are summarized in Table 3. The value of , which is used in Eq. (10), is set to 1.5 after the tests on RMWS. For each group of settings, all three algorithms are run 10 times independently to calculate the average results.

Variance of Task Execution Time
To analyze the performance of the scheduling algorithm under varying task execution times due to fluctuations in VM performance and other factors, the variance factor is increased from 0.1 to 0.5 in steps of 0.05 while holding the other parameters fixed. The experimental results for each evaluation metric of the three algorithms are shown in Fig. 4. Fig. 4 (a) shows the change in the total rental cost under each algorithm under different values. With an average value of 10.2*10 4 , ROSA consistently has a high cost, which is nearly 3.5 times higher than the costs of the other two algorithms. Although the average costs of NOSF and RMWS are close to 2.91*10 4 and 2.95*10 4 , respectively, the cost fluctuation range of NOSF is larger than that of RMWS. In contrast, RMWS can maintain a fixed cost range; in particular, when the value of exceeds 0.3, the average cost is approximately 2.17% lower than that of NOSF, and its cost advantage is greater. This also shows that the proposed RMWS algorithm is less susceptible to the impact of task execution time fluctuations than ROSA and NOSF. Fig. 4 (b) shows the resource utilization performance of the algorithms under different values. RO-SA shows significantly poor efficiency, being closely related to its task allocation strategy. Both NOSF and RMWS have high resource utilization rates of over 77%, which are much higher than the value of 28.44% of ROSA. This shows the obvious performance advantages of NOSF and RMWS in terms of resource utilization. Fig. 4 (c) shows the success rate results of the three algorithms. With increasing , the success rate of each algorithm decreases gradually. This is because when the task execution time increases, it will be difficult to complete the workflow within the deadline. From the results, ROSA always maintains a high success rate, which is mainly due to its large use of higher performance VMs with high lease costs. When = 0.1, the success rate of RMWS is approximately 7.08% higher than that of NOSF. However, with increasing , the success rate of RMWS gradually increases, and the average value is nearly 20.1% higher than that of NOSF. In particular, when = 0.5, the average success rate of RMWS is 58.1%, which is significantly higher than the average success rate of 43.5% of NOSF. Fig. 4 (d) shows the deviation from the workflow dead-  lines. Although ROSA achieves the shortest workflow execution time under different values, it cannot reasonably allocate tasks to VMs according to the deadline, which is an important cause of the high execution cost and low resource utilization of the algorithm. In contrast, the deviation between the makespan and deadline of the workflows in the other two algorithms is significantly reduced. Among them, the average deviation value of NOSF when ∈ [0.3,0.5] is 1.86%, whereas RMWS shows a superior value of -0.06%. The overall deviation value range of RMWS is significantly smaller than other algorithms.
As shown in Fig. 4, when the variance factor of the task execution time changes, RMWS can obtain better overall results for various indicators such as the total rental cost, success rate, resource utilization and deadline deviation. The experimental analysis shows that the algorithm proposed in this paper has the best scheduling performance among the three algorithms under varying task execution times.   Fig. 5 (b) reflect the variation of the rental cost and resource utilization of the three algorithms under different , respectively. All the algorithms still demonstrate their performance characteristics, as shown in Fig. 4 (a) and Fig. 4 (b). Specifically, ROSA always maintains the highest total rental cost and the lowest resource utilization, while the average cost and resource utilization of NOSF and RMWS are similar. However, Fig.  5 (a) shows that the cost of these algorithms under different is relatively stable and has not shown an upward trend under variations of the variance factor . This means that compared with fluctuations in the task execution time, changes in data transmission times caused by bandwidth instabilities and other reasons have a lower impact on workflow scheduling. Fig. 5 (c) shows the success rate with different . When = 0.1, the average success rates of ROSA, NOSF and RMWS are 88.6%, 63% and 71.8%, respectively. These values become 79.8%, 54.4% and 74.1%, respectively, when = 0.5. Compared with the declining trend of the other two algorithms, RMWS maintains a relatively gentle change in the success rate. This is mainly because the proposed algorithm considers the data transmission time when calculating the subdeadlines of the tasks, while ROSA and NOSF do not consider this factor. Compared to Fig. 4 (d), the deadline deviation of ROSA does not change significantly in Fig. 5 (d), which is also slightly reflected in NOSF and RMWS. This further explains that the variation in the data transmission time will impact the cost, resource utilization, success rate and other indicators when scheduling workflow in the cloud; however, its impact is significantly weaker than the performance change provided by fluctuations in the task execution time. Especially in a high-bandwidth environment, this situation will be more obvious.

Variance of Deadline Factor
In addition to the rental cost, the deadline is another key optimization target in workflow scheduling. To reflect the performance of each algorithm under different deadline constraints, we increase the deadline factor from 1 to 8 in step 1, and the evaluation indicators are shown in Fig.  6. Fig. 6 (a) shows that when increases from 1 to 4, the total rental costs of ROSA, NOSF and RMWS decreased extensively, by approximately 36.29%, 75.83% and 74.75%, respectively. This shows that each algorithm will give priority to assign tasks to low-performance VMs to reduce the lease cost when the deadline is further away. However, the cost of ROSA does not change significantly when ∈ [5,8], and the average value is 10.7*10 4 . In comparison, both NOSF and RMWS achieve lower costs. When increases from 5 to 8, RMWS, with an average cost of 0.96*10 4 , outperforms NOSF, with an average cost of 1.67*10 4 . In Fig. 6 (b) and Fig. 6 (c), the resource utilization rate and success rate of these algorithms are shown. Although the success rate of ROSA under the loose deadline constraint ( ∈ [5,8]) is close to 100%, its resource utilization rate is low (24.88%). This is because ROSA fails  to fully utilize existing VMs when scheduling tasks; more tasks are allocated to the new VMs. However, the resource utilization rate and success rate of NOSF and RMWS follow an upward trend. Overall, RMWS achieves a better resource utilization rate and success rate than NOSF. Moreover, the difference is more pronounced when the deadline factor is greater than 4. Under the loose deadline constraint ( ∈ [5,8] ), for NOSF and RMWS, the average resource utilization rates were 83.66% and 87.44%, respectively, and the average success rates were 80.28% and 85.8%, respectively. Fig. 6 (d) shows the deadline deviation of the algorithms under different . The deadline deviations with abnormal values are not marked when = 1. With increasing deadline factor, the deadline deviation of ROSA increases sharply, as it uses many high-performance and new VMs to reduce the makespan of the workflow. NOSF and RMWS also show trends of increasing deadline deviation when ∈ [1,4]; however, this situation is alleviated under the condition of ∈ [5,8]. This shows that the two algorithms can effectively allocate VMs for tasks in workflow scheduling under various deadline constraints.

Variance of Workflow Count
In the cloud environment, the scale of the workflows submitted by users is uncertain; thus, we use different counts of workflows to simulate workflow scheduling on different scales. Fig. 7 shows the performance of each algorithm under different workflow counts. Fig. 7 (a) shows that as the number of workflows increases, the total cost of the three algorithms increases due to the need for more VMs to execute tasks. The cost of ROSA is highest among the algorithms due to the lease of greater numbers of high-performance VMs; NOSF and RMWS have similar lease costs, with 1.68*10 4 and 1.55*10 4 , respectively, on a small scale (workflow count is no more than 100). When the workflow count is greater than 100, the average cost of RMWS exceeds that of NOSF by approximately 58.42%, mainly because RMWS allocates additional tasks to high-performance VMs to meet the deadline. This is verified in Fig. 7 (b) and Fig. 7 (c). In the comparison of the resource utilization shown in Fig. 7 (b), the average value of RMWS is 80.63%, which is significantly better than the average value of 32.11% of ROSA and slightly lower than the 81.62% of NOSF. A more pronounced difference is evident in the comparison of success rate shown in Fig. 7 (c). Although ROSA has the highest average success rate, 86.06%, among the algorithms under the different numbers of workflows, it is accompanied by an overall high rental cost and low resource utilization. In contrast, the success rate of RMWS is approximately 4.12% higher than NOSF when the workflow count is 50 and reaches 58.22% when the workflow count is 1000. In the comparison of deadline deviations shown in Fig. 7 (d), ROSA obtains the maximum average value, -35.94%, with a lower workflow makespan than the other algorithms, and the average value of NOSF is -1.37%, which is slightly better than that of RMWS, -4.68%. In general, when the number of workflows is less than 500, RMWS can achieve performance better than or similar to that of the other two algorithms with respect to the four factors. Although RMWS is worse than NOSF in terms of rental cost, resource utilization and deadline deviation when the workflow count increases from 500 to 1000, it is more focused than NOSF on meeting the deadline constraints and thus has the highest success rate, which is particularly important in the scheduling process.

Discussion of algorithm performance
The comparisons of total rental cost, resource utilization, success rate and deadline deviation among the algorithms under various conditions reveal that ROSA achieves the best success rate but has far higher cost than NOSF and RMWS and consistently low resource utilization. ROSA allocates tasks according to the predicted completion time and the predicted cost of tasks. However, ROSA does not update the predicted completion time of subsequent tasks after each task is completed, which increases the uncertainty of this parameter. In addition, the start time of all waiting tasks will be delayed when the performance of the allocated VM is degraded. Therefore, ROSA has to lease VMs with higher performance to perform subsequent tasks, which will affect the related optimization objects of workflow scheduling.
NOSF effectively reduces the idle time and lease cost of the used VMs by continuously updating the subdeadline of the tasks and assigning only one waiting task to a VM at the same time. Compared with ROSA, its performance in most aspects, e.g. rental cost, resource utilization, and so on, has been significantly improved. However, NOSF ignores the change of the rental cost of each selected VM in the task allocation phase, and only relies on the estimated execution time of the assigned task to obtain the minimum cost, which cannot be determined as the optimal allocation. Moreover, when updating the subdeadline of the tasks, NOSF fails to sufficiently consider the impact of the data transmission time, which is also an important factor in workflow scheduling. Therefore, the scheduling performance of NOSF will be affected if the bandwidth performance fluctuates. By comparison, we consider all the characteristics of workflow scheduling in cloud environment, the proposed RMWS uses the strategy of probabilistic upward ranking to calculate the subdeadline of each task and takes the data transmission time as a variable. Afterwards, RMWS uses the change of the candidate VM rental cost to assign tasks to achieve the goal of global cost optimization. In general, RMWS achieves a better cost, resource utilization, and success rate and has other performance advantages compared to ROSA and NOSF under different conditions, so it is much more likely to generate superior workflow scheduling scheme under the defined deadline constraints.
Furthermore, as shown in Fig. 8, ROSA has the highest scheduling time among the algorithms, which reaches 3.31, and NOSF shows the best execution performance, with an average time of 0.39, which is lower than that of RMWS, 0.5. This is mainly because ROSA uses more new VMs to execute tasks. New VMs require a boot time that will result in an increase in the scheduling time. RMWS is more complex than NOSF in calculating the subdeadline of tasks, and it also takes more time to calculate the increased rental cost of VMs when assigning tasks. Therefore, the scheduling time of RMWS is slightly worse than that of NOSF, which will be optimized in the future.

CONCLUSIONS AND FUTURE WORK
To reduce the impact of VM performance fluctuations and heterogeneity in workflow scheduling in the cloud, this paper proposes a real-time multiple-workflow scheduling algorithm named RMWS to minimize the total rental cost under different deadline constraints. RMWS divides the scheduling process into three stages: workflow initialization, task assignment and feedback processing. The algorithm adjusts the parameters of relevant tasks in real time after the current task is completed, which reduces the delay fluctuations caused by variations in the VM performance and network environments. Compared with two state-of-the-art algorithms, RMWS achieves better performance in terms of total rental cost, resource utilization, success rate and deadline deviation under various conditions.
In the future, we will further optimize the performance of the algorithm and consider the scheduling strategy when task execution fails due to hardware faults. In addition, the load balancing of tasks on different VMs is also an important factor to be considered.