CC - Unit III - Chapter-1 & 2
CC - Unit III - Chapter-1 & 2
scale_down_policy = ScalingPolicy(name='scale_down',
adjustment_type='ChangeInCapacity', as_name='My-Group',
scaling_adjustment=-1, cooldown=180)
conn.create_scaling_policy(scale_up_policy)
conn.create_scaling_policy(scale_down_policy)
Amazon AutoScaling – Python Example
• CloudWatch Alarms #Connecting to CloudWatch
cloudwatch = boto.ec2.cloudwatch.connect_to_region(REGION,
aws_access_key_id=ACCESS_KEY,
• With the scaling policies defined, the next step is to aws_secret_access_key=SECRET_KEY)
alarm_dimensions = {"AutoScalingGroupName": 'My-Group'}
create Amazon CloudWatch alarms that trigger these
policies. #Creating scale-up alarm
scale_up_alarm = MetricAlarm( name='scale_up_on_cpu',
namespace='AWS/EC2', metric='CPUUtilization',
statistic='Average', comparison='>', threshold='70',
• The scale up alarm is defined using the CPU Utilization period='60', evaluation_periods=2,
alarm_actions=[scale_up_policy.policy_arn],
metric with the Average statistic and threshold greater dimensions=alarm_dimensions)
cloudwatch.create_alarm(scale_up_alarm)
70% for a period of 60 sec. The scale up policy created
previously is associated with this alarm. This alarm is #Creating scale-down alarm
scale_down_alarm = MetricAlarm( name='scale_down_on_cpu',
triggered when the average CPU utilization of the namespace='AWS/EC2', metric='CPUUtilization',
statistic='Average', comparison='<', threshold='40',
instances in the group becomes greater than 70% for period='60', evaluation_periods=2,
more than 60 seconds. alarm_actions=[scale_down_policy.policy_arn],
dimensions=alarm_dimensions)
cloudwatch.create_alarm(scale_down_alarm)
conn = boto.connect_s3(aws_access_key_id='<enter>',
aws_secret_access_key='<enter>')
#Connecting to DynamoDB
• After connecting to DynamoDB service, a conn = boto.dynamodb.connect_to_region(REGION,
aws_access_key_id=ACCESS_KEY,
schema for the new table is created by calling aws_secret_access_key=SECRET_KEY)
• To upload a file the objects().insert method of the gs_service = build('storage', API_VERSION, http=auth_http)
Google Cloud Storage API is used. # Upload file
fp= open(FILENAME,'r')
• The request to this method contains the bucket fh = io.BytesIO(fp.read())
name, file name and media body containing the media = MediaIoBaseUpload(fh, FILE_TYPE)
MediaIoBaseUpload object created from the file request = gs_service.objects().insert(bucket=BUCKET,
name=FILENAME,
contents. media_body=media)
response = request.execute()
Google Cloud SQL – Python Example # Python program for launching a Google Cloud SQL instance (excerpt)
def main():
#OAuth 2.0 authorization.
• This example uses the OAuth 2.0 scope flow = flow_from_clientsecrets(CLIENT_SECRETS,
(https://www.googleapis.com/auth/compute) and scope=GS_SCOPE) storage = Storage(OAUTH2_STORAGE)
credentials = storage.get()
credentials in the credentials file to request a refresh
and access token, which is then stored in the if credentials is None or credentials.invalid:
credentials = run(flow,
oauth2.dat file. storage) http =
httplib2.Http()
• After completing the OAuth authorization, an auth_http = credentials.authorize(http)
instance of the Google Cloud SQL service is
gcs_service = build('sqladmin', API_VERSION, http=auth_http)
obtained.
# Define request body
• To launch a new instance the instances().insert instance={"instance":
method of the Google Cloud SQL API is used. "mydb", "project":
"bahgacloud", "settings":{
• The request body of this method contains "tier": "D0",
"pricingPlan":
properties such as instance, project, tier, "PER_USE",
pricingPlan and replicationType. "replicationType": "SYNCHRONOUS"}}
# Linux VM configuration
linux_config = LinuxConfigurationSet('bahga', 'arshdeepbahga', 'Arsh~2483',
True)
#Create instance
sms.create_virtual_machine_deployment(service_name=na
me, deployment_name=name,
deployment_slot='production', label=name,
role_name=name, system_config=linux_config,
os_virtual_hard_disk=os_hd, role_size='Small')
Azure Storage – Python Example
# Python example of using Azure Blob Service (excerpt)
#!/usr/bin/env python
import sys
for line in sys.stdin:
doc_id, content = line.split(’’)
words = content.split()
for word in words:
print ’%s%s’ % (word, doc_id)
Python for MapReduce
#Inverted Index Reducer in Python
• The reducer reads the key-value pairs grouped by for line in sys.stdin:
the same key from the standard input (stdin) and # remove leading and trailing
creates a list of document-IDs in which the word whitespace line = line.strip()
# parse the input we got from
occurs. mapper.py word, doc_id =
line.split(’’)
• The output of reducer contains key value pairs if current_word == word:
where key is a unique word and value is the list of current_docids.append(do
document-IDs in which the word occurs. c_id)
else:
if current_word:
print ’%s%s’ % (current_word,
current_docids) current_docids = []
current_docids.append(doc_id)
current_word = word
Python Packages of Interest
• JSON
• JavaScript Object Notation (JSON) is an easy to read and write data-interchange format. JSON is used as an alternative to XML and is
is easy for machines to parse and generate. JSON is built on two structures - a collection of name-value pairs (e.g. a Python
dictionary) and ordered lists of values (e.g.. a Python list).
• XML
• XML (Extensible Markup Language) is a data format for structured document interchange. The Python minidom library provides a
minimal implementation of the Document Object Model interface and has an API similar to that in other languages.
• Django is an open source web application framework for developing web applications in Python.
• A web application framework in general is a collection of solutions, packages and best practices
that allows development of web applications and dynamic websites.
• Django is based on the Model-Template-View architecture and provides a separation of the data
model from the business rules and the user interface.
• Django provides a unified API to a database backend.
• Thus web applications built with Django can work with different databases without requiring any
code changes.
• With this fiexibility in web application design combined with the powerful capabilities of the Python
language and the Python ecosystem, Django is best suited for cloud applications.
• Django consists of an object-relational mapper, a web templating system and a regular-expression-
based URL dispatcher.
Django Architecture
• Django is Model-Template-View (MTV) framework.
• Model
• The model acts as a definition of some stored data and handles the interactions with the database. In a
web application, the data can be stored in a relational database, non-relational database, an XML file,
etc. A Django model is a Python class that outlines the variables and methods for a particular type of
data.
• Template
• In a typical Django web application, the template is simply an HTML page with a few extra
placeholders. Django’s template language can be used to create various forms of text files (XML,
email, CSS, Javascript, CSV, etc.)
• View
• The view ties the model to the template. The view is where you write the code that actually generates
the web pages. View determines what data is to be displayed, retrieves the data from the database and
passes the data to the template.
Django Setup on Amazon EC2
Cloud Application
Development in
Python
Outline
• Design Approaches
• Design methodology for IaaS service model
• Design methodology for PaaS service model
• Cloud application case studies including:
• Image Processing App
• Document Storage App
• MapReduce App
• Social Media Analytics App
Design methodology for IaaS service model
Component Design
•Indentify the building blocks of the application and to be performed by each block
•Group the building blocks based on the functions performed and type of cloud resources required and
identify the application components based on the groupings
•Identify the inputs and outputs of each component
•List the interfaces that each component will expose
•Evaluate the implementation alternatives for each component (design patterns such as MVC, etc.)
Architecture Design
Deployment Design
•Map the application components to specific cloud resources (such as web servers,
application servers, database servers, etc.)
Design methodology for PaaS service model
• For applications that use the Platform-as-a-service (PaaS) cloud service model, the architecture and
deployment design steps are not required since the platform takes care of the architecture and
deployment.
• Component Design
• In the component design step, the developers have to take into consideration the platform specific features.
• Platform Specific Software
• Different PaaS offerings such as Google App Engine, Windows Azure Web Sites, etc., provide platform specific software
development kits (SDKs) for developing cloud applications.
• Sandbox Environments
• Applications designed for specific PaaS offerings run in sandbox environments and are allowed to perform only those
actions that do not interfere with the performance of other applications.
• Deployment & Scaling
• The deployment and scaling is handled by the platform while the developers focus on the application development
using the platform-specific SDKs.
• Portability
• Portability is a major constraint for PaaS based applications as it is difficult to move the
Image Processing App – Component Design
• Functionality:
• A cloud-based Image Processing application.
• This application provides online image filtering capability.
• Users can upload image files and choose the filters to apply.
• The selected filters are applied to the image and the
processed image can then be downloaded.
• Component Design
• Web Tier: The web tier for the image processing app has front
ends for image submission and displaying processed images.
• Application Tier: The application tier has components for
processing the image submission requests, processing the
submitted image and processing requests for displaying the
results. Component design for Image Processing App
• Storage Tier: The storage tier comprises of the storage for
processed images.
Image Processing App – Architecture Design
• Component Design
• Web Tier: The web tier for the Cloud Drive app has front ends for
uploading files, viewing/deleting files and user profile.
• Application Tier: The application tier has components for
processing requests for uploading files, processing requests for
viewing/deleting files and the component that handles the
registration, profile and login functions.
• Database Tier: The database tier comprises of a user Component design for Cloud Drive App
credentials database.
• Storage Tier: The storage tier comprises of the storage for fi les.
Cloud Drive App – Architecture Design
• Architecture design step which defines the
interactions between the application
components.
• For each component, the corresponding Architecture design for Cloud Drive App
• The user receives an email notification with the download link for
the results when the job is complete.
MapReduce App – Deployment Design