Workflow management system.

A workflow management system is defined as a process consisting of services of steps

that simplifies the execution and management of cloud application.

Architectural Overview of WFMS

 Figure presents a high-level architectural view of a Workflow Management

System (WfMS) utilizing cloud resources to drive the execution of a scientific
workflow application.

 The workflow system comprises the

 workflow engine,

 a resource broker ,

 and plug-ins for communicating with various technological platforms, such as

Aneka and Amazon EC2
These features are delivered through a web portal or through stand-alone applications that is
installed of the user’s end.
 Figure 12.1 depicts two scenarios
 one where the Aneka platform is used in its entirety to complete the workflow,
 and the other where Amazon EC2 is used to supplement a local cluster when there are insufficient resources to
meet the QoS requirements of the application.

 where the Aneka platform is used in its entirety to complete the workflow.

 the other where Amazon EC2 is used to supplement a local cluster when
there are insufficient resources to meet the QoS requirements of the
application.

 Aneka is capable of transparently provisioning additional resources by

acquiring new resources in third-party cloud services such as Amazon EC2
to meet application demands.

 This relieves the WfMS from the responsibility of managing and allocating
resources directly, to simply negotiating the required resources with Aneka.

 Aneka also provides a set of Web services for service negotiation, job
submission, and job monitoring.
 The WfMS would orchestrate the workflow execution by scheduling jobs
in the right sequence to the Aneka Web Services.

 The typical flow of events when executing an application workflow on

Aneka would begin with the WfMS staging in all required data for each job
onto a remote storage resource, such as Amazon S3 or an FTP server.

 In this case, the data would take the form of a set of files, including the
application binaries.

 These data can be uploaded by the user prior to execution, and they can be
stored in storage facilities offered by cloud services for future use. The
WfMS then forwards workflow tasks to Aneka’s scheduler via the Web
service interface.

 These tasks are subsequently examined for required files, and the storage
service is instructed to stage them in from the remote storage server, so that
they are accessible by the internal network of execution nodes. The
execution begins by scheduling tasks to available execution nodes (also
known as worker nodes).

 The workers download any required files for each task they execute from
the storage server, execute the application, and upload all output files as a
result of the execution back to the storage server.

 These files are then staged out to the remote storage server so that they are
accessible by other tasks in the workflow managed by the WfMS. This
process continues until the workflow application is complete.

 The second scenario describes a situation in which the WfMS has greater
control over the compute resources and provisioning policies for executing
workflow applications.

 Based on user-specified QoS requirements, the WfMS schedules workflow

tasks to resources that are located at the local cluster and in the cloud.
Typical parameters that drive the scheduling decisions in such a scenario
include deadline (time) and budget (cost)
 For instance, a policy for scheduling an application workflow at minimum
execution cost would utilize local resources and then augment them with cheaper
cloud resources, if needed, rather than using high-end but more expensivecloud
resources.

 On the contrary, a policy that scheduled workflows to achieve minimum execution

time would always use high-end cluster and cloud resources, irrespective of costs.
The resource provisioning policy determines the extent of additional resources to
be provisioned on the public clouds.

 In this second scenario, the WfMS interacts directly with the resources
provisioned.

 When using Aneka, however, all interaction takes place via the Web service
interface.

 The following sections focuses on the integration of workflow management

systems and clouds and describes in detail practical issues involved in using clouds
for scientific workflow applications.
Architecture of Workflow management system.
Scientific applications are modeled as work flows consisting of tasks, data elements,
control sequences and data dependencies.

Workflow management systems are responsible for managing and executing these workflows.

The Cloud bus workflow management system consists of components that are responsible for
Landlines tasks, data and resources.

The architecture consists of three major parts.

 The user Interface

 The Core System
 The Plug-ins

User Interface:

The user interface allows end users for work with

 Workflow Composition
 Workflow Execution Planning
 Submission
 Monitoring
The Core System:

The core components are responsible for managing the execution of workflows. They
facilitate the transaction of high-level workflow description into task and data objects.

These objects are used by the execution sub system. The scheduling component applies user
selected scheduling policies and plans to the work flow of various stages on their execution.

Plug-ins:

The Plug-ins support workflow execution on different environments and platforms. The plug-ins
are used for querying task transferring the execution status of tasks and applications and
measuring the energy consumption.
 The resources are at the bottom layer of the architecture which includes clusters
global grids and clouds. The resources managers may communicate with the

 Market Maker
 Sealable Application Manager
 Intercloud Services for global resource manager

UTILIZING CLOUDS FOR WORKFLOW EXECUTION

 First, Aneka serves as a useful tool for utilizing clouds, including platform abstraction and
dynamic provisioning.
 Second, we describe later in the chapter a case study detailing the use of Aneka to execute a
scientific workflow application on clouds.

1.Aneka
 Aneka is a distributed middleware for deploying platform-as-a-service (PaaS) offerings (Figure
12.3).
 Developed at CLOUDS Lab, University of Melbourne, Aneka is the result of years of research on
cluster, grid, and cloud computing for high-performance computing (HPC) applications.
 Aneka, which is both a development and runtime environment, is available for public use (for a
cost), can be installed on corporate networks, or dedicated clusters, or can be hosted on
infrastructure clouds like Amazon EC2.
 In comparison, similar PaaS services such as Google AppEngine [19] and Windows Azure [20]
are in-house platforms hosted on infrastructures owned by the respective companies
 . Aneka was developed on Microsoft’s.NET Framework 2.0 and is compatible with other
implementations of the ECMA 335 standard [21], such as Mono.
 Aneka can run on popular platforms such as Microsoft Windows, Linux, and Mac OS X,
harnessing the collective computing power of a heterogeneous network.
 The runtime environment consists of a collection of Aneka containers running on physical or
virtualized nodes.
 Each of these containers can be configured to play a specific role such as scheduling or
execution.
 The Aneka distribution also provides a set of tools for administrating the cloud, reconfiguring
nodes, managing users, and monitoring the execution of applications.
 The Aneka service stack provides services for infrastructure management, application
execution management, accounting, licensing, and security.
 Aneka’s Dynamic Resource Provisioning service enables horizontal scaling depending on the
overall load in the cloud.
 The platform is thus elastic in nature and can provision additional resources on-demand from
external physical or virtualized resource pools, in order to meet the QoS requirements of
applications. I
 in a typical scenario, Aneka would acquire new virtualized resources from external clouds such as
Amazon EC2, in order to meet the minimum waiting time of applications submitted to Aneka.
 Such a scenario would arise when the current load in the cloud is high, and there is a lack of
 available resources to timely process all jobs.
 The development environment provides a rich set of APIs for developing applications that can
utilize free resources of the underlying infrastructure.
 These APIs expose different programming abstractions, such as the task model, thread
model, and MapReduce
 The task programming model is of particular importance to the current discussion.
 It models “independent bag of tasks” (BoT) applications that are composed of a collection of
work units independent of each other, and it may be executed in any given order.
 One of the benefits of the task programming model is its simplicity, making it easy to run legacy
applications on the cloud.
 An application using the task model composes one or more task instances and forwards them as
work units to the scheduler.
 The scheduling service currently supports the First-In-First-Out, First-In-First-Out with
Backfilling, Clock-Rate Priority, and Preemption-Based Priority Queue scheduling
algorithms.
 The runtime environment also provides two specialized services to support this model:
1. the task scheduling service and
2. the task execution service.
 The storage service provides a temporary repository for application files— that is, input files that
are required for task execution, and output files that are he result of execution.
 Prior to dispatching work units, any files required are staged-in to the storage service from the
remote location.
 This remote location can be either the client machine, a remote FTP server, or a cloud storage
service such as Amazon S3.
 The work units are then dispatched to executors, which download the files before execution.
 Any output files produced as a result of the execution are uploaded back to the storage service.
From here they are staged-out to the remote storage location.
2.Aneka Web Services
Aneka exposes three SOAP Web services for
1. service negotiation,
2. reservation, and
3. task submission, as depicted in Figure 12.4.
 The negotiation and reservation services work in concert, and they provide interfaces for
negotiating resource use and reserving them in Aneka for predetermined timeslots
 As such, these services are only useful when Aneka has limited resources to work with
and no opportunities for provisioning additional resources.
 The task Web service provides a SOAP interface for executing jobs on Aneka.
 Based on the task programming model, this service allows remote clients to submit jobs,
monitor their status, and abort jobs.

3.General Approach
Traditional WfMSs were designed with a centralized architecture and were thus tied to a single
machine.
Moving workflow engines to clouds requires
(a) architectural changes and
(b) integration of cloud management tools
 Architectural Changes.
 Most components of a WfMS can be separated from the core engine so that they can be
 executed on different cloud services.
 Each separated component could communicate with a centralized or replicated workflow
engine using events.
 The manager is responsible for coordinating the distribution of load to its subcomponents,
such as the Web server, persistence, monitoring units, and so forth.

 In our WfMS, we have separated the components that form the architecture into the
following:

o user interface,
o core, and
o plug-ins.
 The user interface can now be coupled with a Web server running on a “large” instance of cloud
that can handle increasing number of users.
 The Web request from users accessing the WfMS via a portal is thus offloaded to a different set
of resources.
 Similarly, the core and plug-in components can be hosted on different types of instances
separately.
 Depending on the size of the workload from users, these components could be migrated or
replicated to other resources, or reinforced with additional resources to satisfy the increased load.
 Thus, employing distributed modules of the WfMS on the basis of application requirements helps
scale the architecture.
 Integration Of Cloud Management Tools.
 As the WfMS is broken down into components to be hosted across multiple cloud
resources, we need a mechanism to
 (a) access, transfer, and store data and
 (b) enable and monitor executions that can utilize this approach of scalable distribution
of components.
 The cloud service provider may provide APIs and tools for discovering the VM instances
that are associated to a user’s account.
 Because various types of instances can be dynamically created, their characteristics such
as CPU capacity and amount of available memory are a part of the cloud service
provider’s specifications.
  Similarly, for data storage and access, a cloud may provide data sharing, data movement,
and access rights management capabilities to user’s applications.
 Cloud measurement tools may be in place to account for the amount of data and
computing power used, so that users are charged on the pay-per-use basis.
 A WfMS now needs to access these tools to discover and characterize the resources
available in the cloud.
 It also needs to interpret the access rights (e.g., access control lists provided by
Amazon), use the data movement APIs, and share mechanisms between VMs to fully
utilize the benefits of moving to clouds.
 In other words, traditional catalog services such as the Globus Monitoring and
Discovery Service (MDS) , Replica Location Services, Storage Resource Brokers,
Network Weather Service , and so on could be easily replaced by more user-friendly
and scalable tools and APIs associated with a cloud service provider.
 We describe some of these tools in the following section.

4.TOOLS FOR UTILIZING CLOUDS IN WFMS

 The range of tools and services offered by cloud providers play an important role in integrating
WfMSs with clouds (Figure 12.5).
 Such services can facilitate in the deployment, scaling, execution, and monitoring of workflow
systems.
 This section discusses some of the tools and services offered by various service providers that can
complement and support WfMSs
 . A WfMS manages dynamic provisioning of compute and storage resources in the cloud with the
help of tools and APIs provided by service providers.
 The provisioning is required to dynamically scale up/down according to application
requirements.
 For instance, data-intensive workflow applications may require large amount of disk space for
storage.
 A WfMS could provision dynamic volumes of large capacity that could be shared across all
instances of VMs (similar to snapshots and volumes provided by Amazon).
 Similarly, for compute-intensive tasks in an workflow, a WfMS could provision specific instances
that would help accelerate the execution of these compute-intensive tasks.
 A WfMS implements scheduling policies to assign tasks to resources based on applications’
objectives.
 This task-resource mapping is dependent on several factors: compute resource capacity,
application requirements, user’s QoS, and so forth.
 Based on these objectives, a WfMS could also direct a VM provisioning system to consolidate
data center loads by migrating VMs so that it could make scheduling decisions based on locality
of data and compute resources.
 A persistence mechanism is often important in workflow management systems and for managing
metadata such as available resources, job queues, job status, and user data including large input
and output files.
 Technologies such as Amazon S3, Google’s BigTable, and the Windows Azure Storage
Services can support most storage requirements for workflow systems, while also being
scalable, reliable, and secure.
 If large quantities of user data are being dealt with, such as a large number of brain images used
in functional magnetic resonance imaging (fMRI) studies [12], transferring them online can be
both expensive and time-consuming.
 In such cases, traditional post can prove to be cheaper and faster. Amazon’s AWS
Import/Export5 is one such service that aims to speed up data movement by transferring large
amounts of data in portable storage devices.
 The data are shipped to/from Amazon and offloaded into/from S3 buckets using Amazon’s high-
speed internal network.
  The cost savings can be significant when transferring data on the order of terabytes. Most cloud
providers also offer services and APIs for tracking resource usage and the costs incurred.
 This can complement workflow systems that support budget-based scheduling by utilizing real-
time data on the resources used, the duration, and the expenditure.
 This information can be used both for making scheduling decisions on subsequent jobs and for
billing the user at the completion of the workflow application.
 Cloud services such as Google App Engine and Windows Azure provide platforms for building
scalable interactive Web applications.
 This makes it relatively easy to port the graphical components of a workflow management system
to such platforms while benefiting from their inherent scalability and reduced administration.
 For instance, such components deployed on Google App Engine can utilize the same scalable
systems that drive Google applications, including technologies such as BigTable and GFS .

CASE STUDY: EVOLUTIONARY MULTIOBJECTIVE OPTIMIZATIONS

 EMO is a technique based on genetic algorithms.
 Genetic algorithms are search algorithms used for finding optimal solutions in a large space
where deterministic or functional approaches are not viable.
 Genetic algorithms use heuristics to find an optimal solution that is acceptable within a
reasonable amount of time.
 In the presence of many variables and complex heuristic functions, the time consumed in finding
even an acceptable solution can be too large.
 However, when multiple instances are run in parallel in a distributed setting using different
variables, the required time for computation can be drastically reduced.
Objectives
 The following are the objectives for modeling and executing an EMO workflow on clouds:
1. Design an execution model for EMO, expressed in the form of a workflow, such that
multiple distributed resources can be utilize.
2. Parallelize the execution of EMO tasks for reducing the total completion time.
Dynamically provision compute resources needed for timely completion of the
application when the number of tasks increase.
3. Repeatedly carry out similar experiments as and when required.
4. Manage application execution, handle faults, and store the final results for analysis.
Workflow Solution
 In order to parallelize the execution of EMO, we construct a workflow model for systematically
executing the tasks.
 A typical workflow structure is depicted in Figure .In our case study, the EMO application
consists of five different topologies, upon which the iteration is done.
 These topologies are defined in five different binary files. Each file becomes the input files for the
top level tasks (A0emo1, A0emo,...).
 We create a separate branch for each topology file.
 In Figure there are two branches, which get merged on level 6
 . The tasks at the root level operate on the topologies to create new population, which is then
merged by the task named “emomerge.”
 we see two “emomerge” tasks in the 2nd level, one task in the 6th level that merges two branches
and then splits the population to two branches again, two tasks on the 8th and 10 th levels, and the
final task on the 12th level.
 In the example figure, each topology is iterated two times in a branch before getting merged.
 The merged population is then split. This split is done two times in the figure. The tasks labeled
 B0e and B1e (depicted as darker shade in Figure )is the start of second iteration.

 VISIONARY THOUGHTS FOR PRACTITIONERS

 The cloud computing paradigm is emerging and is being adopted at a rapid rate. Gartner ranks it
at the top of the hype cycle for the year 2010 [29].
 As the technology is being adopted by practitioners industry-wide, there are numerous
challenges to overcome. Moreover, these challenges could be addressed via a realistic vision of
the cloud computing models of the near future.
 This section discusses some of them. Software and service giants such as Google, Amazon, and
Microsoft own large data centers for providing a variety of cloud services to customers.
 These independent and disparate initiatives would eventually lead to an interconnection model
where users can choose a combination of services from different providers in their applications.
 Our vision provides an entity responsible for brokerage of resources across different cloud
providers, termed the market maker [16].
 These inter-cloud environments would then facilitate executions of workflow applications at
distributed data centers. Large scientific experiments would then be able to use inter-cloud
resources, brokered through the market maker.
 The essence of using cloud services is to be able to dynamically scale the applications running on
top of it.
 Automating resource provisioning and VM instance management in clouds based on
multiobjectives (cost, time, and other QoS parameters) can help achieve this goal.
 The automation process should be transparent to the end users who would just be interested in
running workflow applications under their time and budget constraints.
 Users would specify either flexible or tight deadline for the cost they pay for using cloud services.
It becomes the responsibility of the workflow engine running in the cloud to dynamically scale
the application to satisfy multiple users0 request.
 In order to facilitate fair but competitive use of cloud resources for workflow applications, a
service negotiation module must be in place.
 This entity would negotiate with multiple service providers to match users0 requirements to a
service provider’s capabilities.
 Once a match is found, required resources can then be allocated to the user application.
 A cloud market directory service is needed to maintain a catalog of services from various cloud
 service providers.
 Data and their communication play a vital role in any data-intensive workflow application
 . When running such applications on clouds, storage and transfer costs need to be taken into
account in addition to the execution cost.
 The right choice of compute location and storage service provider would result in minimizing the
total cost billed to a user.
 A cloud market maker could handle these task and communication issues at the time of
negotiation between various cloud service providers.