Serverless on OpenStack

with Mistral and StackStorm

Dmitri Zimine
@dzimine

@Stack_Storm
Agenda

Definition, stack, frameworks

What is Serverless overview

Toy example,
Show & Tell Real world use case

Serverless, OpenStack, and

A path forward StackStorm

2
 Serverless
DevOps Agile
Cloud
3
What is Serverless?

5
Let’s call it FaaS

6
What is Serverless?

Authoritative: Mike Roberts on martinfowler.com:

https://goo.gl/bTfgfU

7
What is Serverless?

• Event driven programming model

• Functions, not services – “down when done”

• Scale – “infinite”, transparent for dev, handled by Platform/Framework

8
 SCALE?

IT’S NOW YOUR PROBLEM !

9
What is Serverless?

• Event driven programming model

• Functions, not services

– “down when done”

• Scale – infinite, transparent,

- “not my problem”

• Pay-per use consumption model

10
What is Serverless?

Good for:
• Spiky inconsistent workloads
• Occasional requests
• Saving $$$ money

11
Value-driven development
• Link to that article

12
Serverless Use Cases

• Data Processing

• Services for web & mobile apps

• Other automation
– DevOps automation
– Email clients
– Load testing (Artillery)
– IoT, Alexa, ChatBots
– Exotic (serverless Wordpress)

13
Serverless Use Cases

14
Serverless Use Cases

• Thomson Reuters
– processes 4,000 requests per second with
Serverless.

• FINRA (Financial Industry Regulatory

Authority)
– processes half a trillion validations of stock
trades, every day.

• Expedia
– triggers 1.2 billion serverless requests each
month Source: Tim Wagner, AWS
#serverlessconf Austin
15
Example: Nordstrom

Hello-Retail by Nordstrom
https://github.com/Nordstrom/hello-retail
Serverless stack formed
Serverless > FaaS
AWS Azure
FaaS Lambda Azure Functions
API Gateway API Gateway API Management
Event streams Kinesis, SNS Event Hubs, Notification Hubs
Authentication IAM, Cognito Azure Apps auth
Database DynamoDB Tables, DocDB
File storage S3 Azure Storage
Workflows Step Function Logic Apps
17
 Serverless ecosystem:
platforms & frameworsk
 Standalone Usability shims
Serverless PaaS for AWS FaaS

Serverless
from public cloud
giants

FaaS extensions Open source

For k8s & Swarm DIY platforms
19
Serverless from public cloud giants

20
Standalone Serverless PaaS offerings

21
Usability shims for AWS Lambda/FaaS
FaaS extensions for Kubernetes / Swarm

Docker FaaS by Alex Ellis

github/alexellis/faas
Serverless-swarm
github.com/dzimine/serverless-swarm
Open source DIY platforms
SHOW & TELL, part 1
Serverless 101 with StackStorm

25
Ingredients

MISTRAL

Trigger Call
Rules Workflows

Sensors Actions

Monitoring Networking Storage Cloud Infra Containers Config mgmt Ops Support

IT Domains
26
Take aways

• Functions easy to create, on Python (pref), Bash, Ruby, Java, any …

• Functions (actions) get API (with access control)
• Rules define events and criteria to trigger actions

• Workflows? Coming up next…

27
Genomic Annotation Pipeline
A serverless solution with
StackStorm, Mistral, & Swarm

Image by Miki Yoshihito, Creative Commons license

29
Genomic Sequencing and Annotation
Sequenced Genome

DNA Sample ACGTGACCGGTACTGGTAACGTACA

CCTACGTGACCGGTACTGGTAACGT
ACGCCTACGTGACCGGTACTGGTAA
CGTATACACGTGACCGGTACTGGTA
ACGTACACCTACGTGACCGGTACTG
CTGGTAACGTATACCTCT...

Compute
in silko

Sequencer

Annotated Sequence
So that…

31
Typical genomic annotation pipeline
Prediction of Search for similar Compilation and
genes and proteins in presentation of
proteins databases results

1Mb-3Gb
KOALA(KEGG)

fgenesb GCView

Blast(NR)
Steps:
- batch jobs / functions
- diverse stacks, “variety”
Run time:
KEGG Highly
- hours/days NR
Pipeline orchestration Parallel-able
- call functions, pass data
50-100Gb
Why Serverless
Typical serverless requirements:
• Elastic scale: handle spiky workload pattern
• “Functions”, not “servers”, down when done
• BYOC: package programs into containers, run everywhere
• Launch on [high variety of] events

Additional requirements:
• Long running times: hours
• Pipeline orchestration: execution logic and data passing
• Local Dev environment, consistent and convenient
33
Why not <…>

• Not Lambda?
• 5 min limitation - jobs run for hours and days

• Not OpenWhisk?
• Need powerful workflow to orchestrate pipelines

• Not Azure?
• No native support for Jobs in docker containers

34
Serverless solution
The three sides to the story

End
User

DevOps Developer
35
 1. Admin: deploys the serverless solution

Controller
Worker Worker WorkerScale
f(x)

other infra…
f(x) f(x) f(x)

Registry f(x) f(x) f(x)

DevOps StackStorm f(x) f(x) f(x)

$ function

/share /data

share(:rw) data(:ro)
36
2. Developer:
creates functions, defines pipeline
Create functions (BYOC),
1 pack into Docker image,
push to local Registry
f(x)

Registry f(x)
f(x)
f(x)

Define pipelines
2 as StackStorm workflows
Developer

StackStorm

37
3. User submits data,
System runs pipeline & produces results
Swarm
controller

Swarm schedules
StackStorm runs workflow 3 services
User sends 2 schedules functions
1 sequence data as jobs on Swarm

StackStorm Swarm
Worker f(x)

End
User 6 StackStorm Docker pulls
function’s images 4 5
sends results
back to user Functions run in
containers, produce
Registry data

f(x) f(x)
SHOW & TELL, part 2
Serverless solution with StackStorm and Swarm
for Genomic annotation computations

39
fgenesb pipeline: some [prev] results

40
Serverless on OpenStack

the path forward

SHOW & TELL, part 3
Serverless, with Swarm, on OpenStack,
and something VERY SPECIAL
Credit:
Winson Chan, Mistral core contributor, old Stormer
Anirudth Rekhi, a new Stormer

42
Swarm auto-scale: what did it take?
• Heat template with scale policies
– https://github.com/dzimine/serverless-swarm/blob/master/terraform/
openstack/docker_cluster_autoscaling.yaml
• Scale-up action (auth + web call)
– https://github.com/dzimine/serverless-swarm/blob/master/pipeline/
actions/scaling.py
• Swarm “pending tasks queue” sensor to trigger scale-up event
– https://github.com/dzimine/serverless-swarm/blob/master/pipeline/
sensors/pending_queue.py
• Rule to call ‘scale-up action’ on “pending tasks over threshold”
trigger
– https://github.com/dzimine/serverless-swarm/tree/master/pipeline/
rules
43
Serverless on OpenStack?
Who cares?
• Cloud providers ?
• Private clouds ?
• OpenStack ?
• Developers !
– Use event centric programming models
– Workloads where serverless is best fit

44
Serverless on OpenStack: Options

Build
ourselves

Open Whisk
Stack Storm
45
Serverless on OpenStack: Options
Adopt
OpenWhisk

Open Whisk
Stack Storm
46
Serverless on OpenStack: Options

Open Whisk
Stack Storm +1
Adopt
StackStorm

47
Serverless stack on OpenStack with StackStorm

AWS OpenStack
FaaS Lambda StackStorm
API Gateway API Gateway StackStorm ?
Event streams Kinesis, SNS Zaquar
Authentication IAM, Cognito Keystone
Database DynamoDB Trove
File storage S3 Swift
Workflows Step Function Mistral

48
 Swift

StackStorm
StackStorm facts

• Apache 2.0
• Since April ‘14
• 100,000,0002 contributors (=256)
• ~3,000 installations/month
• ~100 integration packs
50
StackStorm facts

• Largely, OpenStack’s tech stack:

Python, RabbitMQ, Pecan,
Oslo config & utils, eventlets, etc.
• Team part of community -
• - Mistral contributors since inception
51
StackStorm Exchange

© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 52

Discuss, decide
• Get StackStorm, decide on merits
– st2vagrant – play – github.com/StackStorm/st2vagrant
– st2-docker – play & light prod - github.com/StackStorm/st2-docker
– Chef/Puppet/Ansible – for production @ scale
– RTFM: docs.stackstorm.com/install
• Today’s show materials:
– github.com/dzimine/serverless-swarm
– github.com/dzimine/openstack_boston
• Discuss:
– here in Boston – find us at D26 or @dzimine
– on openstack-dev, Subj: [FaaS]
53
Thank You!
Dmitri Zimine
@dzimine

@Stack_Storm
http://github.com/StackStorm/st2 Star 1,950

Image by Miki Yoshihito, Creative Commons license