What is DevOps?

INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
INTRODUCTION TO DEVOPS
Organizational structure
Online products are complex

Expensive to develop and maintain

Thousands of engineers

How to collaborate efficiently?

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DevOps

DevOps is a combination of

methods,

tools, and

cultural behavior

that improves how software is developed and maintained.

It helps organizations deliver higher-quality online products faster.

INTRODUCTION TO DEVOPS
Traditional Change Management DevOps

Independent teams Different teams work together

Different teams have different goals Software Development + IT Operations =

Slow development DevOps

Different teams have similar goals

High speed development

INTRODUCTION TO DEVOPS
Traditional release

INTRODUCTION TO DEVOPS
Minimum Viable Product
A minimum viable product (MVP) is an
early version of a product with limited
functionality.

Cheaper to build

High speed time-to-market

INTRODUCTION TO DEVOPS
Product improvements

INTRODUCTION TO DEVOPS
DevOps benefits

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Use Cases for
DevOps
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
DevOps
DevOps: Software Development + IT
Operations

Collaborative and Shared Responsibilities

MVP Releases

Adaptable to various use cases

INTRODUCTION TO DEVOPS
Infrastructure Engineering
Design, Develop, and Maintain the IT
infrastructure

This infrastructure requires power from the

cloud or from hardware the company owns
Infrastructure engineers take care of the
hardware, network, and cloud components

INTRODUCTION TO DEVOPS
Product Engineering
Design, develop, and maintain the Software
Products

Customer serving components

INTRODUCTION TO DEVOPS
 Data Engineering
Data engineering refers to the building of systems to enable the collection and usage of
data.

1 https://en.wikipedia.org/wiki/Data_engineering

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DataOps MLOps
Software is powered by data Predicting the future using the past data
Data is moved in data pipelines Data Preparation (Historical data)

Ensuring data moves smoothly Model Training

Data is not lost while moving Model Testing

Move the data for the use of data scientists Deployment

and ML Engineers
Maintenance

INTRODUCTION TO DEVOPS
No competition

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Project
Management
Methodologies for
DevOps
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
Why is project management important to What is project management?
DevOps?
Project management is the use of
Defines how the team will operate
specific knowledge,
Drives change
skills,
Timeline/Resource Management
tools, and
Collaboration within/across teams
techniques

to deliver something of value to people.

INTRODUCTION TO DEVOPS
Waterfall

An old methodology

No return back to earlier stage

Develop all of the product at once

INTRODUCTION TO DEVOPS
Agile
Series of cycles
Achieving one small goal at a time

First MVP, then improvements

Going through the cycle each time

INTRODUCTION TO DEVOPS
Choosing the right project management model
The best methodology depends on:

Team's skill set

Budget
Complexity of the project

Expectations

Agile is the de facto standard for most

software development teams.

INTRODUCTION TO DEVOPS
Scrum Kanban
A sprint in Scrum is a two-to-four week No sprints in Kanban, instead tracking
timeframe with a light development goal for improvements continuously.
the team.

INTRODUCTION TO DEVOPS
 Scrum Kanban
Regular, fixed-length sprints (i.e., two Continuous flow
weeks)
Use visuals to improve work-in-progress
Learn through experiences
Visualize the flow of work, limit work-in-
Sprint planning, sprint, daily standup, sprint progress, manage flow, incorporate
review, sprint retrospective feedback loops

Product owner, scrum master, development No defined roles

team

1 https://www.atlassian.com/agile/kanban/kanban-vs-scrum

INTRODUCTION TO DEVOPS
Both Scrum and Kanban are under Agile methodology
Both Scrum and Kanban improves collaboration

Both powerful methodologies when applied to correct use case

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 The DevOps Change
Management Model
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
Requirements
Developing a new product

Leadership: Sets business expectations

Product Experts: Define user experience

requirements

INTRODUCTION TO DEVOPS
Design

Product Engineers: Designing the customer

experience

Infra Engineers: Designing the parts that

product engineers will use

Data Engineers: Designing the data

pipelines for the new product

INTRODUCTION TO DEVOPS
Develop and test
Product Engineers
Develop for the customer experience

Customer interaction

Infra Engineers
Supporting the product

Providing testing environment

Data Engineers
Providing test data

INTRODUCTION TO DEVOPS
Testing

Testing and developing at the same time

Testing before customer interaction

Otherwise poor user experience and
security issues

INTRODUCTION TO DEVOPS
Experimentation

Limited set of users

Early user interactions

Observe if the users like the product

Real user feedback

INTRODUCTION TO DEVOPS
Cyclical development

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Main Software
Architecture
Systems
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
 Software architecture

Software architecture refers to the

fundamental structures of a software
system and the discipline of creating such
structures and systems. Each structure
comprises software elements, relations
among them, and properties of both
elements and relations.

1 https://en.wikipedia.org/wiki/Software_architecture

INTRODUCTION TO DEVOPS
Application Programming Interfaces (APIs)

Data flows through APIs

They are essential for software and architecture

INTRODUCTION TO DEVOPS
Various architectural systems
Factors to consider

Type of software

In machine or in the browser

Complexity of the software

Simpler software parts can be treated as a single unit

Separate the pieces for more complex software

INTRODUCTION TO DEVOPS
Microservices
Most important architectural system for
complex software

Common in modern organizations

Rather than a big single unit, have many

smaller independent units

Smaller units are called the microservices

They communicate to each other via APIs

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DevOps for microservices architecture

INTRODUCTION TO DEVOPS
Benefits of DevOps for microservices
Software is changed constantly in large organizations
Change them one at a time

Microservices are more secure and reliable

If anything goes bad with one microservice, the rest will still function well

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Modern IT
Infrastructure
Concepts
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
Main components of IT infrastructure

Combination of various components

Customized for organizations specific

needs

INTRODUCTION TO DEVOPS
Hardware
Can be hosted on organization's own
premises

Or it can be accessed through a cloud

service provider

INTRODUCTION TO DEVOPS
Network
Communication with other parts of the
software

Communication with the users through the

Internet

INTRODUCTION TO DEVOPS
Software

We need software to: The goal of infrastructure software:

Manage the hardware and network enable the product teams to develop and
serve customers.
Configure them for the organization's use

INTRODUCTION TO DEVOPS
Change management

Software needs constant maintenance and improvement

These changes could be very expensive and disruptive to the services of the organization
Infrastructure Engineering provides the tools that's necessary to build products

INTRODUCTION TO DEVOPS
Developer platform

INTRODUCTION TO DEVOPS
Codebase & version control

Version Control: Git

INTRODUCTION TO DEVOPS
DevOps CI/CD pipelines

DevOps in narrow meaning - CI/CD

Code needs to be built

Turn code file into machine executable file

INTRODUCTION TO DEVOPS
Deployment platform

INTRODUCTION TO DEVOPS
Change management in microservices

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Elements of DevOps
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
DevOps change management model

Continuous Integration & Continuous Delivery

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Principles of CI

1. Version Control Software

2. Frequent integrations

INTRODUCTION TO DEVOPS
Version control

Main branch as the source-of-truth for the codebase

Multiple teams working on the same software

Peer Review

Testing in every step

INTRODUCTION TO DEVOPS
Principles of CD

1. Automated builds

2. Automated tests
3. Predictable and short change times

INTRODUCTION TO DEVOPS
Experimentation

Limited set of users

Collect their feedback

Happens once a change is deployed

INTRODUCTION TO DEVOPS
Feedback loops
Continuously improving the product
Use the review stage for the feedback loop

Turn the feedback into action

Change the software if necessary

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Data Engineering
Implications of the
DevOps Architecture
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
Microservices architecture
Microservices are small-scale software
programs

Microservices deployed separately

Each microservice take care of a different

functionality

Each microservice has its own data and

logic

They store the data in private databases

INTRODUCTION TO DEVOPS
Monolithic architecture

Opposite of the microservices architecture

A big single unit

Much simpler compared to microservices

Maintenance and change is very hard and

risky

INTRODUCTION TO DEVOPS
Monolithic Architecture Microservices Architecture

Limited number of databases A database for each microservice

All application uses the same databases Microservices must do API calls to reach
another services database
Could be a viable option for small scale
applications Microservices are effective in large
organizations and complex products

INTRODUCTION TO DEVOPS
Microservices private databases
Some product functionalities require collaboration from multiple microservices

Microservices can not access each others databases freely, so they do API calls

Microservices always communicate with each other.

INTRODUCTION TO DEVOPS
Data Engineering applications in microservices

INTRODUCTION TO DEVOPS
Data Engineering applications in microservices

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Data Pipelines
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
Data pipelines

INTRODUCTION TO DEVOPS
 ETL

1 https://www.istockphoto.com/nl/vector/data-extract-transform-load-gm1361894912-434102842?phrase=ETL

INTRODUCTION TO DEVOPS
Batch processing

Move offline data in batches

Batch Processing: A large amount of data

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Recap

Data Engineering is an integral part of

Infrastructure Engineering

Data pipelines are robust tools important

for handling data

Batch processing works on regular

schedules, stream processing works
continuously

Batch processing is useful for handling the

accumulated data
Stream processing handles the real-time
d t
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Analytics &
Reporting with
DevOps
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
DevOps benefits for reporting

Handling Multiple Data Sources in harmony

Producing many logs during the change management process

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Reporting architecture For DevOps

DevOps produces a lot of data

Data sources are spread out

Data needs to be collected and moved to a database
Data Scientists can use the data for insights, reports, and dashboards

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Tools: Ecosystem
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
 Requirements

Business and engineering interaction

Used throughout the software change

management process

Project Management Tools

Communication Tools

1 https://atlassian.design/foundations/logos 2 https://slack.com/media-kit 3 https://discord.com/branding

INTRODUCTION TO DEVOPS
 Version Control Software
Main change management technology is
git

Git is a protocol used for version control

Git ensures multiple developers can work

on the same software

Git has many implementations, most

common ones are GitHub and GitLab

1 https://git-scm.com/downloads/logos 2 https://github.com/logos 3 https://about.gitlab.com/press/press-kit/

INTRODUCTION TO DEVOPS
 Build Tools

Software needs to be built to become

executable

Main build tools are: Maven and Gradle

1 https://maven.apache.org/ 2 https://gradle.com/brand/

INTRODUCTION TO DEVOPS
 CI/CD Tools

CI/CD pipelines are the main principles of

DevOps

CI/CD pipelines ensure automated building,

testing, and deployment of software

Main CI/CD tools are Jenkins and CircleCI

1 https://www.jenkins.io/press/ 2 https://circleci.com/legal/trademark-guidelines/

INTRODUCTION TO DEVOPS
 Deployment

Microservices are developed and deployed

independently from each other

Containers imitate separate machines

Microservices are deployed on separate

containers

Containers: Docker and Podman

Container orchestration: Kubernetes

1 https://www.docker.com/company/newsroom/media-resources/ 2 https://podman.io/ 3 https://kubernetes.io/

INTRODUCTION TO DEVOPS
 Monitoring Tools
Products need to be closely monitored and
observed for quality and reliability issues

Monitor the DevOps health and change

management metrics

Example tools used for monitoring are

SignalFX and AppDynamics

1 https://www.splunk.com/en_us/products/observability.html?301=/en_us/devops.html

INTRODUCTION TO DEVOPS
 Data management tools: Kafka

Kafka is a message publishing system

Kafka is heavily used in microservices

architecture

Microservices keep a journal of the work

they do on Kafka

1 https://kafka.apache.org/trademark

INTRODUCTION TO DEVOPS
 Data pipeline management tools

Most tools used for both batch and

streaming processing

Main tools used for data pipeline

management are: Apache Airflow, Hevo
Data, and Prefect

1https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow%20logos 2 https://design.hevodata.com/ 3
https://www.prefect.io/newsroom/logos/

INTRODUCTION TO DEVOPS
Recap

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Ensuring Data
Quality Standards
with DevOps
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
DevOps ensures good code

Most software handle data

Good software drives high-quality data

DevOps helps maintain high-quality in software

INTRODUCTION TO DEVOPS
Data quality

How trusted is the information?

Most recent and accurate information

Can be hard and costly to achieve

Not all data requires the same quality

There are many elements to Data Quality

INTRODUCTION TO DEVOPS
Elements of data quality

Elements breaks down and define Data

Quality

They should be traced and standardized

INTRODUCTION TO DEVOPS
Is the data correct in all details?
Should be closely monitored

INTRODUCTION TO DEVOPS
How comprehensive is the data?
Check if data is lost when it is stored in a
database

Check if data is lost when it is moved to

different databases

INTRODUCTION TO DEVOPS
Consistency is about the reliability of the
information

It is also referred to as Data Integrity

Should be closely monitored and tested

INTRODUCTION TO DEVOPS
Keeping and handling only the relevant
information

Irrelevant information wastes time and

resources

INTRODUCTION TO DEVOPS
How up to date is the information?

Old data could be replaced by more recent

information
Raw data within microservices is not used
in data analysis

Microservices talk to each other via API

calls to get the most recent information

The data within microservices should be

replicated at an acceptable frequency

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Observability and
Reliability
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
 Observability
Observing inner components of software

Components might consist of:

Microservices

APIs

Databases

Other components

Check they are working correctly.

1https://www.istockphoto.com/nl/vector/het-lijnpictogram-van-de-verrekijker-oceaanconcept-verrekijkerteken-
op-witte-gm1245624425-363092800?phrase=field%20glass

INTRODUCTION TO DEVOPS
Why is observability needed?

Many different parts tied to each other

Spot specific problems and their root cause

Monitoring systems effectively

INTRODUCTION TO DEVOPS
 Alerting

Alerts help detect problems early

Set alerts if software fails to run

Setting up alerts enable teams to react fast
when problems happen

1 https://www.istockphoto.com/nl/vector/notification-bells-icon-gm1364077374-435399123?phrase=alerting

INTRODUCTION TO DEVOPS
Why is reliability important?

Examples:

Government services

Defense systems

Banking software

Many other software also require high

reliability

INTRODUCTION TO DEVOPS
How to increase reliability?

Two main principles:

Testing

Observability

DevOps automates testing, and enforce

observability.

INTRODUCTION TO DEVOPS
Testing Observability

Testing is used to detect the problems before

Observability is used to detect the problems
they go to market.
as they happen in user interaction.
DevOps ensures automated testing.
Observability is needed even when the
software is tested enough.

INTRODUCTION TO DEVOPS
Reliability in microservices

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 DevOps Cultural
Concepts
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
INTRODUCTION TO DEVOPS
Concepts

Collaboration

Autonomous Teams
Shared Responsibilities

Automation

Post-Mortem

INTRODUCTION TO DEVOPS
Collaboration

Heart of DevOps

Collaboration between Development and Operations

INTRODUCTION TO DEVOPS
Autonomous teams

Teams are diverse

Self sufficient teams

Teams have members who specialized in:

Product Engineering

Operations

Data Engineering, etc.

INTRODUCTION TO DEVOPS
Shared responsibilities

Autonomous teams take care of specific

parts of the software

They are responsible for developing and

running

You build it, you run it

INTRODUCTION TO DEVOPS
Automation

CI/CD pipelines helps automate change

management, testing, and deployment

Automation is a time-saver

Developers focus on creativity, not the

simple tasks

Without CI/CD pipelines, testing and

deployment must be done manually

INTRODUCTION TO DEVOPS
Post-mortem
All software is prone to errors

Something can always go wrong

When a software problem is identified:

First solve it

Then review what went wrong

This review is called Post-Mortem

No one is blamed in the Post-Mortem

INTRODUCTION TO DEVOPS
How this issue could have been How this issue could have been
prevented? better handled?

Common Answers More observability helps with:

More testing Earlier detection of issues

More code review Earlier detection of the root cause

INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 Tying it all together
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
What is DevOps after all?
DevOps is a combination of: It helps ensure high productivity, high quality,
and managed costs.
tools

technologies
cultural elements

Software Development -> DevOps

Data Engineering -> DataOps

Machine Learning Engineering -> MLOps

INTRODUCTION TO DEVOPS
What is DevOps about?
Reorganizes into autonomous teams

Collaboration of teammates with different

backgrounds

Teams being responsible for different parts

of software

Automating change management

Testing

Experimentation

INTRODUCTION TO DEVOPS
Why is DevOps necessary?
It is very efficient compared to other systems.

Automation and collaborations boost productivity

Testing and observability is necessary for reliability

Experimentation helps with product quality

INTRODUCTION TO DEVOPS
When is DevOps applicable?

DevOps principles are applicable most of

the time.

It is necessary for large organizations.

INTRODUCTION TO DEVOPS
What are the components of DevOps?

INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
 CONGRATS!
INTRODUCTION TO DEVOPS

Cem Sakarya
DevOps Risk Advisor
What did we learn?

1. Introducing ourselves to DevOps and its

main components

2. Looking deeper into modern software

architecture and infrastructure

3. Exploring the data engineering

applications of DevOps and DevOps
reporting

4. Ensuring reliability and data quality

INTRODUCTION TO DEVOPS
You can use your skills to...

navigate through software development

for software design, development, and testing

apply data engineering implications of DevOps

INTRODUCTION TO DEVOPS
Time to celebrate!
INTRODUCTION TO DEVOPS