A Path To Event Sourcing With Amazon MSK - James Ousby

A path to Event Sourcing with
Amazon MSK
James Ousby
Senior Solutions Architect
Amazon Web Services
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s build a bank
Learning objectives
• Understand the what and why of event sourcing and how Amazon
Managed Streaming for Apache Kafka (MSK) can help
• Think differently about how you capture, store, and process your data
• Explore the design and build process behind an event sourcing demo
built around MSK
What is event sourcing?
What it is not – persist entities by updating in place
invoice { id: 1, amt: 50, status: ’issued’ }

invoice { id: 1, amt: 50, status: ’overdue’ }

invoice { id: 1, amt: 50, status: ’paid’ }

Persist entities by storing a sequence of state-changing events
IssuedEvent { id: 1, amt: 50, status: ’issued’ }
invoice { id: 1, amt: 50, status: ’issued’ }


OverdueEvent { id: 1, status: ’overdue’ }
invoice { id: 1, amt: 50, status: ’overdue’ }


OverdueEvent { id: 1, status: ’overdue’ }
PaidEvent { id: 1, status: ’paid’ }
invoice { id: 1, amt: 50, status: ’paid’ }

Event sourcing’s close friends
• CQRS (Command Query Responsibility Segregation)
• Domain-driven design
• Distributed logs (often Apache Kafka)

Event sourcing challenges
It’s not a new concept. Why isn’t this the default way to think
about data?
• Platform and tooling maturity
• Lack of framework support
• More complex/more moving parts

Future-proofing your data architecture
• Maintain complete history (one-way and two-way doors)
• It’s increasingly difficult to anticipate long-term data usage patterns

at design/build time
• You would like your data architecture to support new patterns of

consumption as technologies evolve over time
Amazon Aurora
Events Amazon Redshift
Amazon MSK
Amazon DynamoDB
Amazon Elasticsearch
Service
Amazon Aurora AWS Lake Formation
Amazon MSK
Amazon DynamoDB
Amazon Elasticsearch Amazon Neptune

Service
Capturing your raw data is great, but…
Event sourcing + CQRS benefits
• Data architectures that can
scale elastically as demand
increases
AWS Fargate Amazon MSK
• Data architectures that can Write
prioritize read/write
performance Process
Client
application AWS Fargate
• Compute components can Read
be scaled independently
AWS Fargate Amazon Elasticsearch

Service
Apache Kafka
Why is Apache Kafka a good fit for event sourcing?
• Immutable append-only log
• Guaranteed ordering Producer
0 1 2 3 4 5 6 7 8 9
Slow Consumer Fast Consumer

(offset=4) (offset=8)
Multiple consumers all at different log offsets
Amazon Aurora AWS Lake Formation
Amazon MSK
Amazon DynamoDB
Amazon Elasticsearch Amazon Neptune

Service
Challenges operating Apache Kafka
Difficult to Hard to Hard to Hard to capture Labor-intensive

set up achieve scale metrics
Amazon Managed Streaming for Apache Kafka
Fully managed and highly available Apache Kafka service
Migrate and run your Manages provisioning, Multi-AZ replication Multiple levels of
existing Apache Kafka configuration, and within an security for your
applications on AWS maintenance of AWS Region Apache Kafka clusters
without code changes Apache Kafka clusters
Health monitoring
Amazon MSK deployment architecture
Customer VPC Amazon MSK VPC (AWS-managed)
Availability Zone 1 Availability Zone 2 Availability Zone 3
Kafka Broker Nodes

Producers
Consumers
Zookeeper Nodes
Management
(e.g., create topic)
Open monitoring with Prometheus
Getting started with Amazon MSK is easy
• Fully compatible with Apache Kafka 1.1.1, 2.2.1, and 2.3.1
• AWS Management Console and AWS API for provisioning

• Clusters are automatically set up
• Provision Kafka brokers and storage
• Create and tear down clusters on demand
• Deeply integrated with AWS services

Amazon MSK roadmap (what I can share, TBC)
• In-place cluster upgrades
• Cluster autoscaling
• Identity and Access Management (IAM) support
• Schema registry
• AWS Lambda integration
• Storage tiering
Our banking app
Design process
1. Model your domain as
Terminology commands, events, aggregates,
and views
• Command
2. Design the command handlers
• Event
• Aggregate 3. Design the event processor(s)
that create your projection
• Projection/
materialized view 4. Design the read side
Simple Machines
Simple Machines is a leading Australian technology
consultancy at the intersection of data architecture,
data engineering, and enterprise software delivery.
The company specializes in engineering real-time,
data-driven platforms and applications.
Simple Machines – experimenting
Simple Sourcing – event source data abstraction built to
use Apache Kafka as primary data store
Simple Saga – Saga (coordination) layer built using

Simple Sourcing to provide a robust execution engine
with compensation semantics
https://simplesource.io/
Demo architecture
Write
Client
Read

Service
Demo architecture Command handlers
Events
Commands
Client
application
Events
Commands
Events
Event handlers
Client
Application AWS Fargate
Projection
Amazon Elasticsearch
Service
Events
Commands
Events
Event handlers
Client
Read view Projection
Read
view
Service
Read handler
< time for some code />
Learning objectives
• Understand the what and why of event sourcing and how Amazon
Managed Streaming for Apache Kafka (Amazon MSK) can help
• Think differently about how you capture, store, and process your data
• Explore the design and build process behind an event sourcing demo
built around MSK
Thank you!
James Ousby
joousby@amazon.com

A Path To Event Sourcing With Amazon MSK - James Ousby

Uploaded by

Copyright:

Available Formats

A Path To Event Sourcing With Amazon MSK - James Ousby

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

A Path To Event Sourcing With Amazon MSK - James Ousby

Uploaded by

Copyright:

Available Formats

A path to Event Sourcing with

invoice { id: 1, amt: 50, status: ’issued’ }

invoice { id: 1, amt: 50, status: ’overdue’ }

invoice { id: 1, amt: 50, status: ’paid’ }

IssuedEvent { id: 1, amt: 50, status: ’issued’ }

invoice { id: 1, amt: 50, status: ’issued’ }

IssuedEvent { id: 1, amt: 50, status: ’issued’ }

invoice { id: 1, amt: 50, status: ’overdue’ }

IssuedEvent { id: 1, amt: 50, status: ’issued’ }

invoice { id: 1, amt: 50, status: ’paid’ }

• Distributed logs (often Apache Kafka)

• Platform and tooling maturity

• Lack of framework support

• More complex/more moving parts

• It’s increasingly difficult to anticipate long-term data usage patterns

• You would like your data architecture to support new patterns of

Events Amazon Redshift

Amazon Aurora AWS Lake Formation

Events Amazon Redshift

Amazon Elasticsearch Amazon Neptune

• Data architectures that can Write

AWS Fargate Amazon Elasticsearch

Slow Consumer Fast Consumer

Amazon Aurora AWS Lake Formation

Events Amazon Redshift

Amazon Elasticsearch Amazon Neptune

Difficult to Hard to Hard to Hard to capture Labor-intensive

Availability Zone 1 Availability Zone 2 Availability Zone 3

Kafka Broker Nodes

• Fully compatible with Apache Kafka 1.1.1, 2.2.1, and 2.3.1

• AWS Management Console and AWS API for provisioning

• Provision Kafka brokers and storage

• Create and tear down clusters on demand

• Deeply integrated with AWS services

• In-place cluster upgrades

• Identity and Access Management (IAM) support

• AWS Lambda integration

Simple Saga – Saga (coordination) layer built using

AWS Fargate Amazon MSK

AWS Fargate Amazon Elasticsearch

AWS Fargate Amazon MSK

AWS Fargate Amazon MSK

AWS Fargate Amazon MSK

Read view Projection

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.