The Eucalyptus Open-source Cloud-computing System

Daniel Nurmi, Rich Wolski, Chris Grzegorczyk

Graziano Obertelli, Sunil Soman, Lamia Youseff, Dmitrii Zagorodnov

Computer Science Department

University of California, Santa Barbara
Santa Barbara, California 93106

Abstract ability, application performance profiling, software ser-

Cloud computing systems fundamentally provide ac- vice requirements, and administrative connections. While
cess to large pools of data and computational resources great strides have been made in the HPC and Grid Com-
through a variety of interfaces similar in spirit to exist- puting communities [15, 7] toward the creation of resource
ing grid and HPC resource management and program- provisioning standards [14, 18, 33, 38], this process re-
ming systems. These types of systems offer a new pro- mains somewhat cumbersome for a user with complex re-
gramming target for scalable application developers and source requirements.
have gained popularity over the past few years. However, For example, a user that requires a large number of
most cloud computing systems in operation today are pro- computational resources might have to contact several
prietary, rely upon infrastructure that is invisible to the different resource providers in order to satisfy her re-
research community, or are not explicitly designed to be quirements. When the pool of resources is finally deliv-
instrumented and modified by systems researchers. ered, it is often heterogeneous, making the task of per-
In this work, we present EUCALYPTUS – an open- formance profiling and efficient use of the resources diffi-
source software framework for cloud computing that im- cult. While some users have the expertise required to ex-
plements what is commonly referred to as Infrastructure ploit resource heterogeneity, many prefer an environment
as a Service (IaaS); systems that give users the ability to where resource hardware, software stacks, and program-
run and control entire virtual machine instances deployed ming environments are uniform. Such uniformity makes
across a variety physical resources. We outline the ba- the task of large-scale application development and de-
sic principles of the EUCALYPTUS design, detail impor- ployment more accessible.
tant operational aspects of the system, and discuss archi- Recently, a number of systems have arisen that at-
tectural trade-offs that we have made in order to allow tempt to convert what is essentially a manual large-scale
Eucalyptus to be portable, modular and simple to use on resource provisioning and programming problem into a
infrastructure commonly found within academic settings. more abstract notion commonly referred to as elastic, util-
Finally, we provide evidence that EUCALYPTUS enables ity, or cloud computing (we use the term “cloud com-
users familiar with existing Grid and HPC systems to ex- puting” to refer to these systems in the remainder of this
plore new cloud computing functionality while maintain- work). As the number and scale of cloud-computing sys-
ing access to existing, familiar application development tems continues to grow, significant study is required to de-
software and Grid middle-ware. termine directions we can pursue toward the goal of mak-
ing future cloud computing platforms successful. Cur-
rently, most existing cloud-computing offerings are either
1 Introduction proprietary or depend on software that is not amenable
There are many ways in which computational power to experimentation or instrumentation. Researchers in-
and data storage facilities are provided to users, rang- terested in pursuing cloud-computing infrastructure ques-
ing from a user accessing a single laptop to the alloca- tions have few tools with which to work.
tion of thousands of compute nodes distributed around the The lack of research tools is unfortunate given that
world. Users generally locate resources based on a va- even the most fundamental questions are still unanswered:
riety of characteristics, including the hardware architec- what is the right distributed architecture for a cloud-
ture, memory and storage capacity, network connectivity computing system? What resource characteristics must
and, occasionally, geographic location. Usually this re- VM instance schedulers consider to make most efficient
source location process involves a mix of resource avail- use of the resources? How do we construct VM instance

1
networks that are flexible, well-performing, and secure? system” that is, by design, hypervisor agnostic. However,
In addition, questions regarding the benefits of cloud com- the current implementation of the system uses Xen-based
puting remain difficult to address. Which application do- virtualization as its initial target hypervisor.
mains can benefit most from cloud computing systems and Grid computing must be acknowledged as an intellec-
what interfaces are appropriate? What types of service tual sibling of, if not ancestor to, cloud computing [7, 15,
level agreements should cloud computing provide? How 33, 38]. The original metaphor for a computational utility,
can cloud-computing systems be merged with more com- in fact, gives grid computing its name. While grid com-
mon resource provisioning systems already deployed? puting and cloud computing share a services oriented ap-
Cloud computing systems provide a wide variety proach [16, 17] and may appeal to some of the same users
of interfaces and abstractions ranging from the ability (e.g., researchers and analysts performing loosely-coupled
to dynamically provision entire virtual machines (i.e., parallel computations), they differ in two key ways. First,
Infrastructure-as-a-Service systems such as Amazon EC2 grid systems are architected so that individual user re-
and others [4, 12, 27, 9, 30]) to flexible access to hosted quests can (and should) consume large fractions of the
software services (i.e. Software-as-a-Service systems total resource pool [34]. Cloud systems often limit the
such as salesforce.com and others [37, 20, 21, 29]). All, size of an individual request to be tiny fraction of the to-
however, share the notion that delivered resources should tal available capacity [4] and, instead, focus on scaling to
be well defined, provide reasonably deterministic perfor- support large numbers of users.
mance, and can be allocated and de-allocated on demand.
A second key difference concerns federation. From its
We have focused our efforts on the “lowest” layer of cloud
inception, grid computing took a middleware-based ap-
computing systems (IaaS) because here we can provide
proach as a way of promoting resource federation among
a solid foundation on top of which language-, service-,
cooperating, but separate, administrative domains [14].
and application-level cloud-computing systems can be ex-
Cloud service venues, to date, are unfederated. That is, a
plored and developed.
cloud system is typically operated by a single (potentially
In this work, we present EUCALYPTUS: an open-source
large) entity with the administrative authority to mandate
cloud-computing framework that uses computational and
uniform configuration, scheduling policies, etc. While
storage infrastructure commonly available to academic re-
EUCALYPTUS is designed to manage and control large col-
search groups to provide a platform that is modular and
lections of distributed resources, it conforms to the design
open to experimental instrumentation and study. With EU -
constraints governing cloud systems with regards to fed-
CALYPTUS , we intend to address open questions in cloud
eration of administrative authority and resource allocation
computing while providing a common open-source frame-
policies.
work around which we hope a development community
will arise. E UCALYPTUS is composed of several compo- Thanks in part to the new facilities provided by virtu-
nents that interact with one another through well-defined alization platforms, a large number of systems have been
interfaces, inviting researchers to replace our implementa- built using these technologies for providing scalable In-
tions with their own or to modify existing modules. Here, ternet services [4, 1, 8, 10, 11, 19, 37], that share in com-
we address several crucial cloud computing questions, in- mon many system characteristics: they must be able to
cluding VM instance scheduling, VM and user data stor- rapidly scale up and down as workload fluctuates, support
age, cloud computing administrative interfaces, construc- a large number of users requiring resources “on-demand”,
tion of virtual networks, definition and execution of ser- and provide stable access to provided resources over the
vice level agreements (cloud/user and cloud/cloud), and public Internet. While the details of the underlying re-
cloud computing user interfaces. In this work, we will source architectures on which these systems operate are
discuss each of these topics in more detail and provide a not commonly published, EUCALYPTUS is almost cer-
full description of our own initial implementations of so- tainly shares some architectural features with these sys-
lutions within the EUCALYPTUS software framework. tems due to shared objectives and design goals.
In addition to the commercial cloud computing of-
2 Related Work ferings mentioned above (Amazon EC2/S3, Google Ap-
Machine virtualization projects producing Virtual Ma- pEngine, Salesforce.com, etc.), which maintain a propri-
chine (VM) hypervisor software [5, 6, 25, 40] have en- etary infrastructure with open interfaces, there are open-
abled new mechanisms for providing resources to users. source projects aimed at resource provisioning with the
In particular, these efforts have influenced hardware de- help of virtualization. Usher [30] is a modular open-
sign [3, 22, 26] to support transparent operating sys- source virtual machine management framework from
tem hosting. The “right” virtualization architecture re- academia. Enomalism [12] is an open-source cloud soft-
mains an open field of study [2]): analyzing, optimiz- ware infrastructure from a start-up company. Virtual
ing, and understanding the performance of virtualized sys- Workspaces [27] is a Globus-based [14] system for pro-
tems [23, 24, 31, 32, 41] is an active area of research. visioning workspaces (i.e., VMs), which leverages sev-
E UCALYPTUS implements a cloud computing “operating eral pre-existing solutions developed in the grid comput-

2
ing arena. The Cluster-on-demand [9] project focuses on
the provisioning of virtual machines for scientific com- CLC
and
Walrus


puting applications. oVirt [35] is a Web-based virtual ma-

Public

chine management console. network


While these projects produced software artifacts that

are similar to EUCALYPTUS, there are several differences. CC
 CC


First, EUCALYPTUS was designed from the ground up to

be as easy to install and as non-intrusive as possible, with- Private
 Private

out requiring sites to dedicate resources to it exclusively network
 network


(one can even install it on a laptop for experimentation.)

Second, the EUCALYPTUS software framework is highly

NC


NC


NC


NC


NC


NC


NC


NC

modular, with industry-standard, language-agnostic com-
munication mechanisms, which we hope will encourage
third-party extensions to the system and community de- Cluster
A
 Cluster
B

velopment. Third, the external interface to EUCALYPTUS
is based on an already popular API developed by Amazon.
Finally, EUCALYPTUS is unique among the open-source Figure 1. E UCALYPTUS employs a hierarchical de-
offerings in providing a virtual network overlay that both sign to reflect underlying resource topologies.
isolates network traffic of different users and allows two or
more clusters to appear to belong to the same Local Area • Node Controller controls the execution, inspection,
Network (LAN). and terminating of VM instances on the host where it
Overall, we find that there are a great number of cloud runs.
computing systems in design and operation today that ex-
pose interfaces to proprietary and closed software and re- • Cluster Controller gathers information about and
sources, a smaller number of open-source cloud comput- schedules VM execution on specific node controllers,
ing offerings that typically require substantial effort and/or as well as manages virtual instance network.
dedication of resources in order to use, and no system an- • Storage Controller (Walrus) is a put/get storage
tecedent to EUCALYPTUS that has been designed specif- service that implements Amazon’s S3 interface, pro-
ically with support academic exploration and community viding a mechanism for storing and accessing virtual
involvement as fundamental design goals. machine images and user data.
3 E UCALYPTUS Design • Cloud Controller is the entry-point into the cloud
The architecture of the EUCALYPTUS system is simple, for users and administrators. It queries node man-
flexible and modular with a hierarchical design reflecting agers for information about resources, makes high-
common resource environments found in many academic level scheduling decisions, and implements them by
settings. In essence, the system allows users to start, con- making requests to cluster controllers.
trol, access, and terminate entire virtual machines using
The relationships and deployment locations of each
an emulation of Amazon EC2’s SOAP and “Query” in-
component within a typical small cluster setting are shown
terfaces. That is, users of EUCALYPTUS interact with the
in Figure 1.
system using the exact same tools and interfaces that they
use to interact with Amazon EC2. Currently, we support Node Controller
VMs that run atop the Xen [5] hypervisor, but plan to add An Node Controller (NC) executes on every node that
support for KVM/QEMU [6], VMware [40], and others in is designated for hosting VM instances. An NC queries
the near future. and controls the system software on its node (i.e., the
We have chosen to implement each high-level system host operating system and the hypervisor) in response to
component as a stand-alone Web service. This has the queries and control requests from its Cluster Controller.
following benefits: first, each Web service exposes a well- An NC makes queries to discover the node’s physical
defined language-agnostic API in the form of a WSDL resources – the number of cores, the size of memory, the
document containing both operations that the service can available disk space – as well as to learn about the state of
perform and input/output data structures. Second, we VM instances on the node (although an NC keeps track of
can leverage existing Web-service features such as WS- the instances that it controls, instances may be started and
Security policies for secure communication between com- stopped through mechanisms beyond NC’s control). The
ponents. There are four high-level components, each with information thus collected is propagated up to the Cluster
its own Web-service interface, that comprise a EUCALYP - Controller in responses to describeResource and describe-
TUS installation: Instances requests.

3
 Cluster Controllers control VM instances on a node by access to the underlying network interfaces. This ability
making runInstance and terminateInstance requests to the can cause security concerns, in that, without care, a VM
node’s NC. Upon verifying the authorization – e.g., only instance user may have the ability to acquire system IP
the owner of an instance or an administrator is allowed to or MAC addresses and cause interference on the system
terminate it – and after confirming resource availability, network or with another VM that is co-allocated on the
the NC executes the request with the assistance of the hy- same physical resource. Thus, in a cloud shared by dif-
pervisor. To start an instance, the NC makes a node-local ferent users, VMs belonging to a single cloud allocation
copy of the instance image files (the kernel, the root file must be able to communicate, but VMs belonging to sep-
system, and the ramdisk image), either from a remote im- arate allocations must be isolated. Finally, one of the pri-
age repository or from the local cache, creates a new end- mary reasons that virtualization technologies are just now
point in the virtual network overlay, and instructs the hy- gaining such popularity is that the performance overhead
pervisor to boot the instance. To stop an instance, the NC of virtualization has diminished significantly over the past
instructs the hypervisor to terminate the VM, tears down few years, including the cost of virtualized network inter-
the virtual network endpoint, and cleans up the files as- faces. Our design attempts to maintain inter-VM network
sociated with the instance (the root file system is not pre- performance as close to native as possible.
served after the instance terminates).
Within EUCALYPTUS, the CC currently handles the set
Cluster Controller up and tear down of instance virtual network interfaces in
three distinct, administrator defined “modes”, correspond-
The Cluster Controller (CC) generally executes on a
ing to three common environments we currently support.
cluster front-end machine, or any machine that has net-
The first configuration instructs EUCALYPTUS to attach
work connectivity to both the nodes running NCs and to
the VM’s interface directly to a software Ethernet bridge
the machine running the Cloud Controller (CLC). Many
connected to the real physical machine’s network, allow-
of the CC’s operations are similar to the NC’s opera-
ing the administrator to handle VM network DHCP re-
tions but are generally plural instead of singular (e.g.
quests the same way they handle non-EUCALYPTUS com-
runInstances, describeInstances, terminateInstances, de-
ponent DHCP requests. The second configuration allows
scribeResources). CC has three primary functions: sched-
an administrator to define static Media Access Control
ule incoming instance run requests to specific NCs, con-
(MAC) and IP address tuples. In this mode, each new
trol the instance virtual network overlay, and gather/report
instance created by the system is assigned a free MAC/IP
information about a set of NCs. When a CC receives
tuple, which is released when the instance is terminated.
a set of instances to run, it contacts each NC compo-
In these modes, the performance of inter-VM communi-
nent through its describeResource operation and sends the
cation is near-native, when VMs are running on the same
runInstances request to the first NC that has enough free
cluster (any performance overhead is that imposed by the
resources to host the instance. When a CC receives a de-
underlying hypervisor implementation), but there is not
scribeResources request, it also receives a list of resource
inter-VM network isolation. Finally, we support a mode
characteristics (cores, memory, and disk) describing the
in which EUCALYPTUS fully manages and controls the
resource requirements needed by an instance (termed a
VM networks, providing VM traffic isolation, the defini-
VM “type”). With this information, the CC calculates
tion of ingress rules (configurable firewalls) between logi-
how many simultaneous instances of the specific “type”
cal sets of VMs, and the dynamic assignment of public IP
can execute on its collection of NCs and reports that num-
addresses to VMs at boot and run-time.
ber back to the CLC.
In this mode, the users are allowed to attach VMs, at
Virtual Network Overlay
boot time, to a “network” that is named by the user. Each
Perhaps one of the most interesting challenges in the de- such network is assigned a unique VLAN tag by EUCA -
sign of a cloud computing infrastructure is that of VM LYPTUS, depicted in Figure 2, as well as a unique IP sub-
instance interconnectivity. When designing EUCALYP - net from a range specified by the administrator of EUCA -
TUS , we recognized that the VM instance network so- LYPTUS in advance (typically a private IP range). In this
lution must address connectivity, isolation, and perfor- way, each set of VMs within a given named network is
mance. isolated from VMs on a different named network at us-
First and foremost, every virtual machine that EUCA - ing VLANs, and further using IP subnetting. The CC acts
LYPTUS controls must have network connectivity to each as a router between VM subnets, with the default policy
other, and at least partially to the public Internet (we use blocking all IP traffic between VM networks. If the user
the word “partially” to denote that at least one VM in- wishes, they may associate ingress rules with their named
stance in a “set” of instances must be exposed externally networks, allowing for example ICMP ping traffic to and
so that the instance set owner can log in and interact with from the public Internet, but only allowing SSH connec-
their instances). Because users are granted super-user ac- tions between VMs that they control. The CC uses the
cess to their provisioned VMs, they may have super-user Linux iptables packet filtering system to implement and

4
 Cluster
Controller


Physical
Resource
 Rou<ng
Rules

To
physical
 Deny
ALL
between
NETWORK
A
and
NETWORK
B

Ethernet
 Allow
ICMP
from
ALL
to
NETWORK
A

Physical
Interface
 Allow
TCP:SSH
from
NETWORK
B
to
NETWORK
A

VM
Instance
 Forward
PUBLICIP
X
to
PRIVATEIP
Y

To
physical

VLAN
Tagged
 Ethernet

Virtual
Interface

Virtual
Interface
 VLAN

A
Tagged
 VLAN

B
Tagged

Virtual
Interface
 Virtual
Interface

So<ware

Ethernet
Bridge


VM
Instance
 VM
Instance

(VLAN
A)
 (VLAN
B)


Figure 2. Each EUCALYPTUS VM instance is assigned Figure 3. The CC uses the Linux iptables packet filter-
a virtual interface that is connected to a software Eth- ing system to allow users to define inter-VM network
ernet bridge on the physical machine, to which a VLAN ingress rules, and to assign public IP addresses dy-
tagged interface is further connected. namically at boot or run-time.

control VM network ingress rules. Finally, note that all Storage Controller (Walrus)
VMs in this mode are typically assigned form a pool of EUCALYPTUS includes Walrus, a data storage service
private IP addresses, and as such cannot be contacted by that leverages standard web services technologies (Axis2,
external systems. To manage this situation, the CC al- Mule) and is interface compatible with Amazon’s Simple
lows the administrator to specify a list of public IPv4 ad- Storage Service (S3) [36]. Walrus implements the REST
dresses that are unused, and provides the ability for users (via HTTP), sometimes termed the “Query” interface, as
to dynamically request that an IP from this collection be well as the SOAP interfaces that are compatible with S3.
assigned to a VM at boot or run-time; again using the Walrus provides two types of functionality.
Linux iptables Network Address Translation (NAT) fa-
cilities to define dynamic Destination NAT (DNAT) and • Users that have access to EUCALYPTUS can use Wal-
Source NAT (SNAT) rules for public IP to private IP ad- rus to stream data into/out of the cloud as well as
dress translation (see Figure 3 for details). from instances that they have started on nodes.
From a performance perspective, the solution we em- • In addition, Walrus acts as a storage service for
ploy exhibits near native speed when two VMs within a VM images. Root filesystem as well as kernel and
given named network within a single cluster communi- ramdisk images used to instantiate VMs on nodes can
cate with one another. When VMs on different named be uploaded to Walrus and accessed from nodes.
networks need to communicate, our solution imposes one
extra hop through the CC machine which acts as an IP Users use standard S3 tools (either third party or those
router. Thus, we afford the user with the ability to choose, provided by Amazon) to stream data into and out of Wal-
based on their specific application requirements, between rus. The system shares user credentials with the Cloud
native performance without inter-VM communication re- Controller’s canonical user database.
strictions, or suffer an extra hop but gain the ability to re- Like S3, Walrus supports concurrent and serial data
strict inter-VM communication. transfers. To aid scalability, Walrus does not provide lock-
ing for object writes. However, as is the case with S3,
When VMs are distributed across clusters, we provide users are guaranteed that a consistent copy of the object
a manual mechanism for linking the cluster front-ends via will be saved if there are concurrent writes to the same
a tunnel (for example, VTUN [28]). Here, all VLAN object. If a write to an object is encountered while there
tagged Ethernet packets from one cluster are tunneled to is a previous write to the same object in progress, the pre-
another over a TCP or UDP connection. Performance of vious write is invalidated. Walrus responds with the MD5
cross-cluster VM communication is likely to be dictated, checksum of the object that was stored. Once a request
primarily, by the speed of the wide area link. However, has been verified, the user has been authenticated as a
the performance impact of this tunnel can be substantial if valid EUCALYPTUS user and checked against access con-
the link between clusters is sufficiently high performance. trol lists for the object that has been requested, writes and
More substantial performance evaluation study of this and reads are streamed over HTTP.
other cloud networking systems is being performed, but is Walrus also acts as an VM image storage and manage-
beyond the scope of this work. ment service. VM root filesystem, kernel and ramdisk im-

5
 WS
 VMs
&
CCs

CC1


CCN

WWW
 PRS

Cloud
Controller
Services


Figure 4. Overview of Cloud Controller services. Figure 5. E UCALYPTUS includes Walrus, a S3 com-
Dark lines indicate the flow of user requests while light patible storage management service for storing and ac-
lines correspond to inter-service system messages. cessing user data as well as images.

ages are packaged and uploaded using standard EC2 tools The Resource services process user virtual machine
provided by Amazon. These tools compress images, en- control requests and interact with the CCs to effect the
crypt them using user credentials, and split them into mul- allocation and deallocation of physical resources. A sim-
tiple parts that are described in a image description file ple representation of the system’s resource state (SRS) is
(called the manifest in EC2 parlance). Walrus is entrusted maintained through communication with the CCs (as in-
with the task of verifying and decrypting images that have termediates for interrogating the state of the NCs) and
been uploaded by users. When a node controller (NC) re- used in evaluating the realizability of user requests (vis
quests an image from Walrus before instantiating it on a a vis service-level agreements, or SLAs). The role of the
node, it sends an image download request that is authenti- SRS is executed in two stages: when user requests arrive,
cated using an internal set of credentials. Then, images are the information in the SRS is relied upon to make an ad-
verified and decrypted, and finally transferred. As a per- mission control decision with respect to a user-specified
formance optimization, and because VM images are often service level expectation. VM creation, then, consists of
quite large, Walrus maintains a cache of images that have reservation of the resources in the SRS, downstream re-
already been decrypted. Cache invalidation is done when quest for VM creation, followed by commitment of the
an image manifest is overwritten, or periodically using a resources in the SRS on success, or rollback in case of
simple least recently used scheme. errors.
Walrus is designed to be modular such that the authen- The SRS then tracks the state of resource allocations
tication, streaming and back-end storage subsystems can and is the source of authority of changes to the proper-
be customized by researchers to fit their needs. ties of running reservations. SRS information is leveraged
Cloud Controller by a production rule system allowing for the formulation
of an event-based SLA scheme. Application of an SLA is
The underlying virtualized resources that comprise a
triggered by a corresponding event (e.g., network property
EUCALYPTUS cloud are exposed and managed by, the
changes, expiry of a timer) and can evaluate and modify
Cloud Controller (CLC). The CLC is a collection of web-
the request (e.g., reject the request if it is unsatisfiable)
services which are best grouped by their roles into three
or enact changes to the system state (e.g., time-limited al-
categories:
locations). While the system’s representation in the SRS
• Resource Services perform system-wide arbitration may not always reflect the actual resources, notably, the
of resource allocations, let users manipulate proper- likelihood and nature of the inaccuracies can be quanti-
ties of the virtual machines and networks, and moni- fied and considered when formulating and applying SLAs.
tor both system components and virtual resources. Further, the admission control and the runtime SLA met-
rics work in conjunction to: ensure resources are not over-
• Data Services govern persistent user and system data committed and maintain a conservative view on resource
and provide for a configurable user environment for availability to mitigate possibility of (service- level) fail-
formulating resource allocation request properties. ures.
• Interface Services present user-visible interfaces, A concrete example from our implementation allows
handling authentication & protocol translation, and users to control the cluster to be used for the VM al-
expose system management tools providing. locations by specifying the ”zone” (as termed by Ama-

6
zon). Further, we have extended the notion of zone to for user-controlled virtual machine creation and control
meta-zones which advertise abstract allocation policies. atop existing resources. Its hierarchical design targets re-
For example, the “any” meta-zone will allocate the user- source commonly found within academic and laboratory
specified number of VMs to the emptiest cluster, but, in settings, including but not limited to small- and medium-
the face of resource shortages, overflow the allocation to sized Linux clusters, workstation pools, and server farms.
multiple clusters. We use a virtual networking solution that provides VM
The middle tier of Data Services handle the creation, isolation, high performance, and a view of the network
modification, interrogation, and storage of stateful system that is simple and flat. The system is highly modular,
and user data. Users can query these services to discover with each module represented by a well-defined API, en-
available resource information (images and clusters) and abling researchers to replace components for experimen-
manipulate abstract parameters (keypairs, security groups, tation with new cloud-computing solutions. Finally, the
and network definitions) applicable to virtual machine and system exposes its feature set through a common user in-
network allocations. The Resource Services interact with terface currently defined by Amazon EC2 and S3. This
the Data Services to resolve references to user provided allows users who are familiar with EC2 and S3 to tran-
parameters (e.g., keys associated with a VM instance to sition seamlessly to a EUCALYPTUS installation by, in
be created). However, these services are not static config- most cases, a simple addition of a command-line argument
uration parameters. For example, a user is able to change, or environment variable, instructing the client application
what amounts to, firewall rules which affect the ingress of where to send its messages.
traffic. The changes can be made offline and provided as In sum, this work aims to illustrate the fact that the
inputs to a resource allocation request, but, additionally, EUCALYPTUS system has filled an important niche in
they can be manipulated while the allocation is running. the cloud-computing design space by providing a system
As a result, the services which manage the networking and that is easy to deploy atop existing resources, that lends
security group data persistence must also act as agents of itself to experimentation by being modular and open-
change on behalf of a user request to modify the state of a source, and that provides powerful features out-of-the-box
running collection of virtual machines and their support- through an interface compatible with Amazon EC2.
ing virtual network. Presently, we and our users have successfully deployed
In addition to the programmatic interfaces (SOAP and the complete system on resources ranging from a single
“Query”), the Interface tier also offers a Web interface for laptop (EC2 on a laptop) to small Linux clusters (48 to 64
cloud users and administrators. Using a Web browser, nodes). The system is being used to experiment with HPC
users can sign up for cloud access, download the cryp- and cloud computing by trying to combine cloud comput-
tographic credentials needed for the programmatic inter- ing systems like EUCALYPTUS and EC2 with the Teragrid
face, and query the system, e.g., about available disk im- (presented as a demo at SuperComputing’08 as part of the
ages. The administrators can, additionally, manage user VGrADS [39] project), as a platform to compare cloud
accounts, inspect the availability of system components. computing systems’ performance, and by many users who
Lastly, the collection of interface web services adver- are interested in experimenting with a cloud computing
tises entry points for user requests using a variety of in- system on their own resources.
terface specifications (e.g., EC2’s SOAP & “Query”, S3’s In addition, we have made a EUCALYPTUS installation
SOAP & REST’) where single sign authentication is done available to all who wish to try out the system without
according to the best practices common among cloud ven- installing any software [13]. Our experience so far has
dors. Users can make requests using either the EC2 SOAP been extremely positive, leading us to the conclusion that
or EC2 “Query” protocols [4]. In particular this has al- EUCALYPTUS is helping to provide the research commu-
lowed the wide variety of tools which comply with the nity with a much needed, open-source software frame-
EC2 and S3 interfaces to work without modification. The work around which a user-base of cloud-computing re-
key design goal achieved by the interface services is a searchers can be developed.
insulation of the internal communication data types by
mapping requests from these disparate protocols to an in- References
dependent system-internal protocol. Consequently, inter-
[1] 3Tera home page. http://www.3tera.com/.
nal services are unconcerned with details of the outward-
[2] K. Adams and O. Agesen. A comparison of software and
facing interfaces utilized by users while being able to
hardware techniques for x86 virtualization. In ASPLOS-
mimic the functionality, syntax, and structure of the in- XII: Proceedings of the 12th international conference on
terface primitives preserving existing investment in tools Architectural support for programming languages and op-
and code. erating systems, pages 2–13, New York, NY, USA, 2006.
ACM.
4 Conclusions
[3] Advanced Micro Devices, AMD Inc. AMD Virtualization
The EUCALYPTUS system is built to allow administra- Codenamed “Pacifica” Technology, Secure Virtual Ma-
tors and researchers the ability to deploy an infrastructure chine Architecture Reference Manual. May 2005.

7
 [4] Amazon Web Services home page. http://aws. [23] W. Huang, M. Koop, Q. Gao, and D. Panda. Virtual ma-
amazon.com/. chine aware communication libraries for high performance
[5] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, computing. In Proceedings of Supercomputing 2007.
A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and [24] W. Huang, J. Liu, B. Abali, and D. K. Panda. A case for
the art of virtualization. In SOSP ’03: Proceedings of the high performance computing with virtual machines. In ICS
nineteenth ACM symposium on Operating systems princi- ’06: Proceedings of the 20th annual international confer-
ples, pages 164–177, New York, NY, USA, 2003. ACM. ence on Supercomputing, pages 125–134, New York, NY,
[6] F. Bellard. QEMU, a Fast and Portable Dynamic Transla- USA, 2006. ACM.
tor. Proceedings of the USENIX Annual Technical Confer- [25] Hyper-v home page – http://www.microsoft.
ence, FREENIX Track, pages 41–46, 2005. com/hyperv.
[7] F. Berman, G. Fox, and T. Hey. Grid Computing: Making [26] Intel. Enhanced Virtualization on Intel Architecture-based
the Global Infrastructure a Reality. Wiley and Sons, 2003. Servers. Intel Solutions White Paper, March 2005.
[8] F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wal- [27] K. Keahey, I. Foster, T. Freeman, and X. Zhang. Virtual
lach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. workspaces: Achieving quality of service and quality of
Bigtable: A Distributed Storage System for Structured life in the grid. Sci. Program., 13(4):265–275, 2005.
[28] M. Krasnyansky. VTun-Virtual Tunnels over TCP/IP net-
Data. Proceedings of 7th Symposium on Operating System
works, 2003.
Design and Implementation(OSDI), page 205218, 2006.
[29] P. Laplante, J. Zhang, and J. Voas. What’s in a name?
[9] J. Chase, D. Irwin, L. Grit, J. Moore, and S. Sprenkle. Dy-
distinguishing between saas and soa. IT Professional,
namic virtual clusters in a grid site manager. High Per-
10(3):46–50, May-June 2008.
formance Distributed Computing, 2003. Proceedings. 12th
[30] M. McNett, D. Gupta, A. Vahdat, and G. M. Voelker.
IEEE International Symposium on, pages 90–100, 2003.
Usher: An Extensible Framework for Managing Clusters
[10] J. Dean and S. Ghemawat. MapReduce: Simplified Data of Virtual Machines. In Proceedings of the 21st Large
Processing on Large Clusters. Proceedings of 6th Sym- Installation System Administration Conference (LISA),
posium on Operating System Design and Implementa- November 2007.
tion(OSDI), pages 137–150, 2004. [31] A. Menon, A. Cox, and W. Zwaenepoel. Optimizing Net-
[11] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, work Virtualization in Xen. Proc. USENIX Annual Tech-
A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, nical Conference (USENIX 2006), pages 15–28, 2006.
and W. Vogels. Dynamo: amazon’s highly available key- [32] M. F. Mergen, V. Uhlig, O. Krieger, and J. Xenidis. Virtu-
value store. Proceedings of twenty-first ACM SIGOPS alization for high-performance computing. SIGOPS Oper.
symposium on Operating systems principles, pages 205– Syst. Rev., 40(2):8–11, 2006.
220, 2007. [33] NSF TeraGrid Project. http://www.teragrid.
[12] Enomalism elastic computing infrastructure. http:// org/.
www.enomaly.com. [34] J. P. Ostriker and M. L. Norman. Cosmology of the early
[13] Eucalyptus Public Cloud (EPC). http: universe viewed through the new infrastructure. Commun.
//eucalyptus.cs.ucsb.edu/wiki/ ACM, 40(11):84–94, 1997.
EucalyptusPublicCloud/. [35] oVirt home page. http://ovirt.org/.
[14] I. Foster and C. Kesselman. Globus: A metacomputing [36] Amazon simple storage service api (2006-03-01) –
infrastructure toolkit. International Journal of Supercom- http://docs.amazonwebservices.com/
puter Applications, 1997. AmazonS3/2006-03-01/.
[15] I. Foster and C. Kesselman, editors. The Grid – Blueprint [37] Salesforce Customer Relationships Management (CRM)
for a New Computing Infrastructure. Morgan Kaufmann, system. http://www.salesforce.com/.
1998. [38] T. Tannenbaum and M. Litzkow. The condor distributed
[16] I. Foster, C. Kesselman, J. Nick, and S. Tuecke. The phys- processing system. Dr. Dobbs Journal, February 1995.
iology of the grid: An open grid services architecture for [39] Virtual Grid Application Development Software project.
distributed systems integration, 2002. http://vgrads.rice.edu/.
[17] I. Foster, C. Kesselman, and S. Tuecke. The anatomy of [40] Vmware home page – http://www.vmware.com.
[41] L. Youseff, K. Seymour, H. You, J. Dongarra, and R. Wol-
the grid: Enabling scalable virtual organizations. Int. J.
ski. The impact of paravirtualized memory hierarchy on
High Perform. Comput. Appl., 15(3):200–222, 2001.
linear algebra computational kernels and software. In
[18] D. Gannon. Programming the grid: Distributed software
HPDC, pages 141–152. ACM, 2008.
components, 2002.
[19] Google – http://www.google.com/.
[20] D. Greschler and T. Mangan. Networking lessons in deliv-
ering ‘software as a service’: part i. Int. J. Netw. Manag.,
12(5):317–321, 2002.
[21] D. Greschler and T. Mangan. Networking lessons in deliv-
ering ’software as a service’: part ii. Int. J. Netw. Manag.,
12(6):339–345, 2002.
[22] R. Hiremane. Intel Virtualization Technology for Directed
I/O (Intel VT-d). Technology@Intel Magazine, 4(10), May
2007.