Intel® Distribution of OpenVINO™ Toolkit

Digital Courseware for Educators

Course: Deploying Deep Learning Applications

Powered by oneAPI
Intel® Distribution of OpenVINO™ toolkit

Deploying Deep Learning

Applications
Module 1: Introduction to AI and OpenVINO™
Notices and disclaimers

Performance varies by use, configuration, and other factors. Learn more at intel.com/PerformanceIndex .
Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available
updates. See backup for configuration details. No product or component can be absolutely secure.
Your costs and results may vary.
Intel® technologies may require enabled hardware, software, or service activation.
Intel® optimizations, for Intel® compilers or other products, may not optimize to the same degree for non-Intel products.
Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy.
Results have been estimated or simulated.
Intel is committed to respecting human rights and avoiding complicity in human rights abuses.
See Intel’s Global Human Rights Principles. Intel® products and software are intended only to be used in
applications that do not cause or contribute to a violation of an internationally recognized human right.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.
Other names and brands may be claimed as the property of others.

3
Module 1

Introduction to AI and OpenVINO™

4
Table of Contents

▪ Introduction to AI ▪ Intel® Distribution of OpenVINO™Toolkit

▪ What is AI ? ▪ System Requirements
▪ History of AI ▪ Installation Methods
▪ Machine Learning (ML) ▪ Open Model Zoo
▪ Deep Learning (DL) ▪ Model Downloader
▪ Breakthroughs in Deep Learning ▪ OpenVINOToolkit Workflow
▪ AI use cases across every industries ▪ Intel® DevCloudfor Edge Workloads
▪ Why Deep Learning ▪ Hands-on Labs
▪ Deep Learning Development cycle ▪ [Lab1.1] –Open Model Zoo
▪ Challenges in deep learning ▪ [Assignment 1.1] –Download a pretrained model
from the Open Model Zoo and obtain model
information

5
Module 1: Learning Objective
▪ Learn about the history and fundamentals of artificial intelligence
▪ Understand and compare Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning
(DL) techniques.
▪ Recognize the various applications of artificial intelligence in the industry, as well as their economic
impact.
▪ Discover the workflow for developing Deep Learning applications.
▪ Gain insight into the critical factors to consider when selecting an AI compute platform.
▪ Investigate the Intel® AI Ecosystem and learn about its features.
▪ Discover the OpenVINO™ toolkit and see how it works.
▪ Learn about Intel® DevCloud for Edge Workloads, including its motivations and capabilities.
▪ Become acquainted with the numerous sample applications, demos, and resources available on
Intel® DevCloud for Edge Workloads.
▪ Discover the processes required for designing and prototyping applications on Intel® DevCloud for
Edge Workloads.

6
Module 1: Learning Outcomes

After completing this module, students should be able to:

▪ Learn about the history and fundamentals of artificial intelligence

▪ Explain what Artificial Intelligence, Machine Learning, and Deep Learning are.
▪ Compare and choose the ideal AI techniques for their applications.
▪ Recognize the various parts of a Deep Learning application.
▪ Gain a thorough understanding of Intel® software and hardware offerings.
▪ Recognize terms associated with the OpenVINO™ toolkit workflow.
▪ Acquire a thorough understanding of the hardware and feature choices available when using
Intel® DevCloud for Edge Workloads.
▪ Get started with Intel® DevCloud for Edge Workloads.

7
Module 1: Key Questions Addressed

▪ How has AI research progressed?

▪ What is the distinction between artificial intelligence, machine learning, and deep learning?
▪ What are some industrial AI applications?
▪ What are the various steps involved in the Deep Learning development cycle?
▪ What are the most important factors to consider when selecting an AI compute platform?
▪ What are the offerings and benefits of the Intel® AI Compute Ecosystem?
▪ What is OpenVINO™ toolkit?
▪ Which platforms and operating systems are supported by the OpenVINO toolkit?
▪ What are the components of OpenVINO toolkit?
▪ What are the steps involved in the OpenVINO toolkit workflow?
▪ What is the Intel® DevCloud for Edge Workloads?
▪ What are the features of Intel® DevCloud for Edge Workloads and how to get started with it?

8
What is Artificial Intelligence

Artificial Intelligence (AI) is a field

of study, which aims to use
algorithms and systems to
simulate intelligent behavior in
computers. It is defined as the
study of ’intelligent agents’: any
system that perceives its
environment and takes actions to
maximize its chance of achieving
its goals.

9
History of Artificial Intelligence

Timeline of the Development of Artificial Intelligence

10
Machine Learning

▪ Machine Learning algorithms build

a model based on sample data –
Training Data - to make predictions
or decisions without being
explicitly programmed to do so.
▪ Machine Learning algorithms uses
statistical methods to learn
patterns and features directly from
data.
▪ Machine Learning models become
progressively better in their
function. Example of a Machine Learning Application: Spam Email Detector

11
Deep Learning
• Subset of Machine Learning techniques,
involving complex models called Deep Neural ARTIFICIAL INTELLIGENCE
Networks (DNN).
• The algorithm is able to discover features
from the data on its own using “neural
networks.”
• Deep Learning involves a deep neural network
MACHINE LEARNING
that has many layers of artificial neurons.
• Neurons are the basic unit of a neural network.
They are interconnected using “weights” that
represent the strength of connections
between neurons. DEEP
LEARNING

12
Breakthroughs in Deep Learning
Machines able to meet or exceed human image and speech recognition

Image Recognition Speech Recognition

30% 30%
97%
23% person 23% 99%
“play song”
Error

Error
15% 15%

8% 8%
Human Human
0% 0%
2010 Present 2000 Present

$13 Additional economic impact

in 2030
trillion
Source: ILSVRC ImageNet winning entry classification error rate each year 2010-2016 (Left), https://www.microsoft.com/en-us/research/blog/microsoft-researchers-achieve-new-conversational-speech-recognition-milestone/
(Right)
Source: https://www.mckinsey.com/featured-insights/artificial-intelligence/notes-from-the-ai-frontier-applications-and-value-of-deep-learning

13
Machine Learning vs. Deep Learning
Machine Learning Deep Learning

A class of statistical methods that uses parameters from known A type of Machine Learning that has received increasing focus
existing data and then predicts outcomes on similar new data. in the last several years.

Traditionally, relies on a prescribed set of “features” that are The algorithm doesn’t need to be told about the important
considered important within the dataset. features –able to discover features from data on its own using a
“Neural Network.”
Feature Engineering –the process of building features into a Involves a “Deep” Neural Network (DNN) that has many layers
Machine Learning algorithm (labor - intensive process). of artificial neurons. Artificial neurons in a DNN are
interconnected.
-Strength of connection between neurons is represented
by a number called “Weight”. The process of determining
weights is called “training”.
-Avoids Feature Engineering –shift
fromhowtoengineerthebestfeaturestohowtoguidemodel
stowardsdiscoveringthebestfeatures.
-Neural Network can learn features directly from the data,
given a sufficient datasize and optimal architecture.

14
Discussion Points

• How has Artificial Intelligence evolved over the years?

• How do Artificial Intelligence, Machine Learning, and Deep Learning relate
to one another?
• What are the main driving forces behind the present Deep Learning
growth?

15
AI is changing industries

Emergency Energy Education Cities Finance Health

response
Maximize Transform Enhance safety, Turn data Revolutionize
Real-time production the learning research, and into valuable patient outcomes
emergency and and uptime experience more intelligence
crime response

Industrial Media Retail Smart homes Telecom Smart cities

Empower Create Transform stores Enable homes Drive network Efficient and
truly intelligent thrilling and inventory that see, hear, and operational robust traffic
Industry 4.0 experiences and respond efficiency systems

16
AI use cases exist across every industry, and some of the
most prevalent today include:
Generative AI Image & video generation, Code generation, Text generation

Image Recognition Applications in the domain of Manufacturing for Defect

Detection, Medical Imaging in the field of Medicine, Crop
Pesticide Detection in the field of Agriculture, etc.
Video Classification A Security Camera Detecting a Break-Infor Security purposes,
Social Distance Monitoring, Detection of different objects for
multiple scenarios, etc. via live streaming of Videos
Speech-to-Text or Text-to-Speech Dictating to Your Smartphone, changing voice accent, Cloning
of Voices, etc.

Natural Language Processing Text Analytics, Sentiment Analysis, Chatbots, etc.

Recommendation Systems Personalized Advertising, Product Recommendation, Movie

Recommendation, etc.

Tabular & Time Series Data Applications Financial Analysis, Spam Filters, Smart Wireless Network
Routing, Monitoring, and Predictive Maintenance of Machines
in the Manufacturing domain, etc.

17
Why deep learning
There is tremendous opportunity in the global growth in
chipset revenue driven by AI inference
Inference TAM by Chipset
$40,000

$35,000

$30,000

$51.9B $25,000

$20,000
Driven by
by 2025 AI inference
$15,000

$10,000

$5,000

$0
2020 2025

CPU GPU FPGA ASIC

Source: Intel AI Market Model/Omdia Q2’20 AI Chipsets Report.
All figures are in US Dollars.

18
Factors that Led to the Rise of Deep Learning

Several factors have contributed to the widespread

adoption of Deep Learning based techniques for AI
applications resulting an explosion of AI products and
startups, and accuracy break-throughs in image and
speech recognition.

▪ Availability of Large Datasets

▪ Better Computational Capabilities
▪ Can Learn from Different Types of Data
▪ Open-Source Development

19
Deep Learning Development Cycle

Data acquisition Create models

and organization

Integrate trained models Adjust models to meet

with application code performance and
accuracy objectives

20
Discussion Points

▪ How Artificial Intelligence is helping in changing the world that we live in?
▪ How are organizations using AI to create dynamic solutions across industries?
▪ How can AI impact our society, and the society of the future?
▪ What are some of the challenges faced at each stage of Deep Learning development cycle?
▪ What are some of the key things that you should consider before pushing Deep Learning
models to production?

21
AI compute considerations
How do you determine the right computing for your AI needs?

Workloads Requirements Demand

ASIC/
1 FPGA/

Utilization
GPU

2 3 CPU

Scale

What is my What are my use case How prevalent is AI in

workload profile? requirements? my environment?

22
Factors to be Considered
While selecting AI Compute Platforms

▪ It starts by taking a cross-section of the environment to determine the breakdown of AI and other workloads.

▪ Figure out the use case requirements. These may include factors like: Throughput, accuracy, response time,
cost, size, power, data size, security, manageability, reliability, etc.

▪ Finally, determine how much demand there is for AI in the environment.

▪ What percentage of the overall computing is running AI?
▪ Is AI persistently in use?
▪ What kind of scale is required for AI workloads?

23
Challenges in Deep Learning
Development and deployment challenges in deep learning

Unique inference needs Integration challenges No one size fits all

Gap in performance and No way to streamline Diverse requirements for myriad
accuracy between trained and end-to-end development use cases require unique
deployed models workflow approaches

Low-performing, lower- Slow time to solution and Inability to meet use-case-

accuracy models deployed time to market specific requirements

24
Discussion Points

▪ What are some other challenges that you might face in the above
example?
▪ Why is it harder to maintain a multi-hardware platform?
▪ What are some ways to overcome the challenges of working with various
hardware platforms?
▪ Why is it essential to optimize your hardware and software stack in
conjunction with each other?
▪ Why it’s important to choose the right compute platform for your
application?

25
Object Detection + Intel® Xeon®: Compelling SW+HW
AI Inference Performance Increases Over Time
Improvements mean exponential performance increase

1.0x baseline 4x faster 5x faster 10x faster 13x faster

2018 R5 2019 R3 2021.2 2021.4 2022.1

4116 1st Gen Intel® Xeon® 4216 2nd Gen Intel® Xeon® 4216R 2nd Gen Intel® Xeon® 4316 3rd Gen Intel® Xeon® 4316 3rd Gen Intel® Xeon®
Scalable processor with FP32 Scalable processor with Scalable processor with Scalable processor with Scalable processor with
Intel® Deep Learning Boost Intel® Deep Learning Boost Intel® Deep Learning Boost Intel® Deep Learning Boost
(INT8 VNNI) (INT8 VNNI) (INT8 VNNI) (INT8 VNNI)

See here for workloads and configurations. Results may vary.

2018 R5 obtained on system configuration 1
2019 R3 obtained on system configuration 2
OV-2021.2 obtained on system configuration 3
Intel® Distribution of OpenVINO™ Toolkit OV-2021.4.1 and OV-2022.1 obtained on system configuration 4
26
Compounding effect of hardware and software
Use Intel® Iris® Xe graphics + CPU combined for maximum inferencing
Tiger Lake + Intel® Distribution of OpenVINO™ toolkit vs. Coffee Lake CPU

3.9x
2.9x 3.6x 3x
2.8x
2.2x 2.2x
2.1x 1.7x
2x 1.9x 1.8x 1.7x 1.6x
1.7x

deeplabv3-TF mobilenet-ssd-CF resnet-50-TF ssd300-CF squeezenet1.1-CF

Core i5-1145G7, CPU (FPS) Core i5-1145G7, GPU (FPS) Core i5-1145G7, GPU+CPU (FPS)
11th Gen Intel® Core™ i5-1145G7 (Tiger Lake) relative inference FPS compared to Intel® Core™ i5-8500 (Coffee Lake).

The above is preliminary performance data based on preproduction components. For more complete information about
performance and benchmark results, visit intel.com/benchmarks. See backup for configuration details.

27
OpenVINO™ Toolkit Overview
Fast, accurate results with high-performance, deep learning inference

1 MODEL

Convert and
optimize models,
and deploy across 2 OPTIMIZE Optimized Performance

a mix of hardware CPU GPU VPU FPGA

and environments,
on-premises and
on-device, in the
browser or in
the cloud 3 DEPLOY

Powered by oneAPI
The productive, smart path to freedom for accelerated computing from the economic and technical burdens of proprietary
alternatives.

28
Why Intel® Distribution of OpenVINO™ toolkit
Across key industries
▪ Generative AI trained to speed up text, image, and video creation
Creative
Help rapidly generate creative content for ▪ Develop software code to speed product development
story telling, marketing, and more. ▪ Automate personalized responses in more authentic tones

Industrial ▪ Machine vision with existing infrastructure

Help enable safe, smart, and efficient ▪ Designed for reduced need for on-site maintenance and improved uptime
industrial applications.
▪ Enhanced human-machine interfaces

Health and life sciences ▪ Deep learning with existing infrastructure

Support improved patient care and enhance ▪ Enhanced speed and accuracy of diagnostics in healthcare operations
medical diagnoses.
▪ Support enhanced overall patient health

Retail ▪ Compatibility and ease of management

Create more-personalized, more- ▪ Enable new use cases with faster time to market
convenient experiences.
▪ Advanced analytics for personalized experiences

Safety and security ▪ Customizability and scalability with portfolio of offerings

Help improve public safety with deep ▪ Rapid software development
learning.
▪ Maximized performance per dollar per watt

29
OpenVINO™ Toolkit Developer Journey
1 MODEL 2 OPTIMIZE 3 DEPLOY

Model Conversion API & Tools OpenVINO™ Model Server

Convert trained model from supported Serve models over gRPC, REST, or C API endpoints
frameworks

OpenVINO™ Runtime
Read, load, infer Common Python, C and C++ APIs that abstracts low-level
programming for each device below
IR Data OpenVINO format (intermediate
representation file) (.pb, .tflite, .onnx,)

Training optimization with NNCF

Quantization aware training as well as model Primary CPU
pruning and sparsity and GPU
Options
Intel® Developer Cloud for the Edge
Try and test your model on different Intel
Open Model Zoo hardware
280+ open source and optimized pretrained
models
Legacy Tools in release 2022 or older
Post-Training Optimization Tool Secondary
Reduces model size into low-
precision without retraining VPU and
Model compression FPGA Options
Post-training optimization with NNCF as well as Deep Learning Workbench*
model quantization and size reduction Visually analyze and fine-tune
*also available on Intel® DevCloud for the Edge

30
 1 MODEL 2 OPTIMIZE 3 DEPLOY

Model selection

Expedite the model training process

Leverage the expansive variety of open-source pre-trained models from the following model zoos

Supported frameworks and formats: https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Supported_Frameworks_Layers.html#doxid-openvino-docs-m-o-d-g-prepare-model-

supported-frameworks-layers
Convert models with Model Optimizer: https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html

31
 1 2 3
Developer journey BUILD OPTIMIZE DEPLOY

Gesture
recognition
Object Machine
recognition translation
Text
Object recognition

300+
detection

Question
Semantic answering

pretrained and
segmentation Compressed
models
Human pose
estimation optimized models
Action
recognition

Image Text
processing detection Image
Instance retrieval
segmentation
Text
spotting

32
 1 2 3
Developer journey BUILD OPTIMIZE DEPLOY

Model Optimizer
• A Python-based tool to import trained models
and convert them to intermediate
representation (IR)
• Optimizes for performance or space with Read, load, infer
conservative topology transformations
OpenVINO format
Trained model Model Optimizer
• Hardware-agnostic optimizations IR data
(intermediate representation)
(.xml, .bin)

Optimization techniques available are: .xml – Describes the network topology

.bin – Describes the weights and biases binary data
• Linear operation fusing
• Stride optimizations
• Group convolutions fusing

Note: Except for ONNX (.onnx model formats), all models have to be
converted to an IR format to use as input to the inference engine.

Development guide  https://docs.openvino.ai/latest/openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html

33
 1 2 3
Developer journey BUILD OPTIMIZE DEPLOY

Neural Network Compression Neural Network

Framework (NNCF) Compression Framework

NEW Bootstrap
Optimize network size before retraining NAS – Discover
higher-performance
Bootstrap NAS ‒ Load alternative
models models automatically
The NNCF reduces neural network sizes for
faster training cycles and more-compact Bootstrap NAS uses machine learning to
models for deep learning inference. Quantization-aware discover alternative, higher-performing
training
models optimized for Intel® hardware.

Mixed-precision
quantization

Works with ▪ Smaller

TensorFlow and Filter pruning ▪ No experts
PyTorch models needed

Use any of these algorithms to

quantize or prune networks
 1 2 3
Developer Journey BUILD OPTIMIZE DEPLOY

Post-Training Optimization Tool

Note: Legacy tool deprecated and to be
removed in release 2024.0. NNCF is the
newer tool for post training optimization Trained model
Model trained
using one of the Data set and annotation
supported
Conversion technique that reduces frameworks
model size into low precision
without retraining Statistics
and
Accuracy and
Reduces model size while also improving performance check
JSON
latency, with little degradation in model Model Optimizer
Post-Training
Converts and IR Accuracy
accuracy and without model retraining. optimizes trained
Optimization Tool
Checker
Full-precision Conversion technique to
model using a IR
Different optimization approaches are supported
quantize models to low Environment
precision for high (hardware)
supported: quantization algorithms, etc. framework
performance specifications JSON

Available as a command-line tool and API

and inside Deep Learning Workbench.
IR

Optimized IR Inference engine

35
 1 2 3
Developer Journey BUILD OPTIMIZE DEPLOY

Deep Learning Workbench

Note: Legacy tool in release 2022 or older. NNCF
is the newer tool for post training optimization.
Note: Available in the Intel® DevCloud for the Edge

Web-based UI extension tool for

model analysis and graphical
measurements
Import/Select Model
▪ Visualize performance data for topologies and layers to aid in
model analysis
▪ Automate analysis for optimal performance configuration Import/Select Target
(streams, batches, latency)
▪ Experiment with int8 or Winograd calibration for optimal Import/Select Data Set
tuning using the Post-Training Optimization Tool
▪ Provide accuracy information through Accuracy Checker Run
▪ Direct access to models from public set of Open Model Zoo
▪ Enable remote profiling, allowing the collection of
performance data from multiple machines without any
additional setup

36
 1 2 3
Developer journey BUILD OPTIMIZE DEPLOY

Runtime
• High-level C, C++, and Python inference runtime API
• Interface is implemented as dynamically loaded plugins for each
hardware type
• Delivers superior performance for each type without requiring users to
implement and maintain multiple code pathways

Development guide  https://docs.openvino.ai/latest/openvino_docs_OV_UG_OV_Runtime_User_Guide.html#

37
 1 2 3
Developer Journey BUILD OPTIMIZE DEPLOY

Write once, deploy anywhere

Common high-level inference runtime for
cross-platform flexibility
Write once, and deploy across different platforms
with the same API and framework-independent
execution.
Full environment utilization, or multidevice plugin,
across available hardware for superior performance
results.

EDGE TO CLOUD

For more details on supported platforms, see system requirements  https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/system-requirements.html

38
 Developer journey

Write once, deploy anywhere

OpenVINO® Runtime architecture

Applications
Inference engine
runtime
OpenVINO™ Runtime

Multidevice plugin (optional but recommended for full system utilization)

MKL-DNN and oneDNN Myriad and

clDNN GNA plugin
plugin HDDL plugins
Plugin
architecture

Intrinsics OpenCL™ GNA API Movidius API

Intel®
GNA
(IP)

Intel Hardware Plugin

39
Additional Tools and Add-ons
Streamlined development experience and ease of use

Model Downloader Deployment Manager

Provides an easy way of accessing public models as well as a set ▪ Generate an optimal, minimized runtime package
of pretrained Intel® models for deployment
▪ Deploy with smaller footprint compared to development
package

Benchmark App Accuracy Checker

▪ Measure performance (throughput, latency) of a model Provides an easy way of accessing public models as well as a set
▪ Get performance metrics per layer and on overall basis of pretrained Intel® models

Deep Learning Streamer OpenVINO™ Model Server

Streaming analytics framework to create and deploy complex media Scalable inference server for serving optimized models over gRPC or
analytics pipelines REST API endpoints

Dataset Management Framework Training Extensions

Use this add-on to build, transform, and analyze data sets Trainable deep learning models for action recognition, segmentation,
image classification, object detection, and text spotting

40
Case Studies
Use cases and successful implementations powered by the Intel® Distribution of OpenVINO™ Toolkit

Vistry, Inc’s Eigen PMY Venue AerVision NodeWeaver

DarwinAI Shelfie
Discrn Platform Innovations Operational Insights AerWatch Edge Cloud

Provides restaurateurs Unlocks the power of Develops the COVID- Empowers retailers Delivers smart Utilizes AI enabled Enables streamlined
the insights and machine vision and AI Net convolutional with an intelligent shelf operating platform with video monitoring and and scalable edge cloud
automation services to improve neural network (CNN) monitoring system for Intel technology for event detection. computing.
needed to meet manufacturing architecture to help real-time, actionable venues and events. Read the brief Read the brief
the challenges of processes with defect clinicians rapidly and insights. Read the brief
increased online detection. accurately detect Read the brief
ordering and labor Read the brief COVID-19
shortages. in patients.
Read the brief Read the case study

Read more success stories: https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/industry.html

41
GitHub Jupyter Notebooks
Valuable tutorials for computer vision and natural language processing

▪ Basic introduction to
▪ End-to-end training to deployment
OpenVINO's Python API Tensorflow
<Hello World/> Hello World ▪ Inference on a mobilenetv3 image Training
workflow starting with TensorFlow’s
Flowers classification demo
classification model

▪ Load Runtime and Show Info

▪ Loading a Model Model ▪ Download a model,
OpenVINO Model ▪ Convert to OpenVINO's IR format
{API} API
▪ Getting Information about a Model Optimizer & Tools ▪ Get model information,
▪ Doing Inference on a Model
Runtime ▪ Benchmark model
▪ Reshaping and Resizing

Tensorflow ▪ Demonstrates Monocular Depth

▪ Demonstrates how to convert Mono- Estimation with MidasNet model
to TensorFlow models to OpenVINO IR Depth ▪ Users can upload their own videos
OpenVINO and images, input data will be resized

▪ Convert PyTorch models to

OpenVINO IR ▪ Background removal in images
PyTorch to Background
▪ Uses Model Optimizer to convert the ▪ The open source U^2-Net model is
OpenVINO open source fastseg semantic Removal converted from PyTorch
segmentation model

42
Discussion Points

▪ What are the DL frameworks supported by the Intel® Distribution of OpenVINO™toolkit?

▪ What is the reason behind and benefits pertaining to the use of Intermediate Representation by
Intel® Distribution of OpenVINO™toolkit?
▪ What role does the Post Training Optimization tool play in the pipeline?
▪ How would you start working on your Deep Learning project? What should ideally be your first
step in starting a Deep Learning project?
▪ Why is it better to start with a pre-trained model instead of training a neural network from scratch?

43
Module 1

System Requirements

44
System Requirements
Developer Platforms

• Processors
▪ 6th to 13th generation Intel® Core™ processors
▪ 1st to 4th generation Intel® Xeon® Scalable processors
▪ Intel Atom® Processors with Intel® Streaming SIMD Extensions 4.2 (Intel® SSE4.2)
▪ Integrated Graphics
▪ Intel® HD Graphics
▪ Intel® UHD Graphics
▪ Intel® Iris® Pro Graphics
▪ Intel® Iris® Xe Graphics
▪ Intel® Iris® Xe MAX Graphics
▪ Discrete Graphics
▪ Intel® Data GPU Flex Series Center (formerly code named Arctic Sound)
▪ Intel® Arc ™ GPU (formerly code named DG2)

▪ Compatible Operating Systems

▪ Ubuntu* 22.04 long-term support (LTS), 64 bit (Kernel 5.15+)
▪ Ubuntu 20.04 LTS, 64 bit (Kernel 5.15+)
▪ Ubuntu 18.04 LTS with limitations, 64 bit (Kernel 5.4+)
▪ Windows® 10 and 11
▪ macOS* 10.15 and later, 64 bit
▪ Red Hat* Enterprise Linux* 8, 64 bit

45
System Requirements
Target Platforms

Compatible Operating Systems

Processors / Accelerator Ubuntu* 22.04
long-term support
Ubuntu 20.04
LTS, 64 bit (Kernel
Ubuntu 18.04 LTS
with limitations, 64
Windows® 10 and 11
(64 bit)
macOS* 10.15 and
later, 64 bit
Red Hat* Enterprise
Linux* 8, 64 bit
(LTS), 64 bit 5.15+) bit (Kernel 5.4+)
(Kernel 5.15+)

▪ 1st to 4th generation Intel® Xeon® Scalable processors

▪ 6th to 13th generation Intel® Core™ processors

CPU ▪ Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with

Intel® HD Graphics
▪ Intel Atom® Processors with Intel® Streaming SIMD Extensions
4.2 (Intel® SSE4.2)

Discrete Graphics
▪ Intel® Data GPU Flex Series Center (formerly code named
Arctic Sound)
▪ Intel® Arc ™ GPU (formerly code named DG2)

GPU Integrated Graphics in Processors

▪ Intel® HD Graphics
▪ Intel® UHD Graphics
▪ Intel® Iris® Pro Graphics
▪ Intel® Iris® Xe Graphics
▪ Intel® Iris® Xe MAX Graphics

▪ Intel® Vision Accelerator Design with Intel® Movidius™ Vision

VPU* ▪
Processing Unit (VPU)
Intel® Neural Compute Stick 2

▪ Intel® GNA

GNA

*VPU supported in OpenVINO™ toolkit version 2022.3 LTS and before only

46
Module 1

Installation Methods

47
Choose how you install

Choose and download free directly from Intel

Intel® Distribution of OpenVINO™ Toolkit

Also available from these sources:

Intel® Developer Cloud | PIP Docker Hub |
Dockerfile Anaconda Cloud | YUM | APT

Build from source:

GitHub | Gitee (for China)

48
Github Install via PIP
https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide

• Select your desired

OS/environment for instructions
▪ Install prerequisites as instructed
▪ Python
▪ Git
▪ Microsoft Visual C++ (Windows)
▪ Follow instruction commands
▪ Create & Activate environment
▪ Clone Git repo
▪ Install packages
▪ Launch any Jupyter* Notebook

49
Module 1

Open Model Zoo

50
What is in the Open Model Zoo Repository?

▪ Intel Pre-Trained Models (pre-converted)

▪ Public Pre-Trained Models
▪ Model Downloader and other automation tools
▪ Demos that demonstrate model usage with OpenVINO™ Toolkit

51
How to obtain OpenVINO™ IR Models

Public
Model Downloader Model Converter
Open Model Zoo 300+ open Pre-trained Models
sourced and optimized
Tool: Tool:
pretrained models available
omz_downloader omz_converter

User Trained Models

Supported Frameworks Models

Tensorflow* Model
Tensorflow Lite* Optimizer Intermediate Representation
PyTorch* (.xml + .bin)
Keras*
Tool: mo
ONNX*
PaddlePaddle*
Kaldi*
MXNET*
Caffe*

52
Types of Models

▪ Object Detection ▪ Question Answering ▪ Image Translation

▪ Object Recognition ▪ Machine Translation ▪ Optical Character
▪ Reidentification ▪ Text To Speech Recognition
▪ Semantic Segmentation ▪ Speech Noise Suppression ▪ Deblurring
▪ Instance Segmentation ▪ Time Series Forecasting ▪ JPEG Artifacts Removal
▪ Human Pose Estimation ▪ Action Sequence Modeling ▪ Salient Object Detection
▪ Image Processing ▪ Face Recognition ▪ Text Prediction
▪ Text Detection ▪ Monocular Depth Estimation ▪ Named Entity Recognition
▪ Text Recognition ▪ Image Inpainting ▪ Vehicle Reidentification
▪ Text Spotting ▪ Style Transfer ▪ Background Matting
▪ Action Recognition ▪ Colorization ▪ Speech Recognition
▪ Image Retrieval ▪ Sound Classification ▪ Compressed

Open Model Zoo docs:

https://docs.openvino.ai/latest/model_zoo.html

53
Object Detection and Recognition

Several detection models can be used to

detect a set of the most popular objects -for
example, faces, people, vehicles.

Object recognition models are used for

classification, regression, and character
recognition. Use these networks after a
respective detector (for example,
Age/Gender recognition after Face
Detection).

54
Reidentification Models
Precise tracking of objects in a video is a common application of Computer Vision (for example, for
people counting). It is often complicated by a set of events that can be described as a "relatively long
absence of an object". For example, it can be caused by occlusion or out-of-frame movement. In such
cases, it is better to recognize the object as "seen before" regardless of its current position in an
image or the amount of time passed since last known position.

55
Semantic Segmentation Models

Semantic segmentation is an extension of object

detection problem. Instead of returning bounding
boxes, semantic segmentation models return a
"painted" version of the input image, where the "color"
of each pixel represents a certain class.

These networks are much bigger than respective

object detection networks, but they provide a better
(pixel-level) localization of objects, and they can detect
areas with complex shape (for example, free space on
the road).

56
Instance Segmentation Models

Instance segmentation is an extension of

object detection and semantic segmentation
problems.

Instead of predicting a bounding box around

each object instance segmentation model
outputs pixel-wise masks for all instances.

57
Human Pose Estimation Models
Human pose estimation task is to predict a
pose: body skeleton, which consists of key
points and connections between them, for
every person in an input image or video. Key
points are body joints, i.e. ears, eyes, nose,
shoulders, knees, etc. There are two major
groups of such methods: top-down and
bottom-up.

The first detects persons in a given frame,

crops or rescales detections, then runs pose
estimation network for every detection. These
methods are very accurate. The second finds
all key points in a given frame, then groups
them by person instances, thus faster than
previous, because network runs once.

58
Text Detection and Recognition

Examples of text that can be detected and recognized

with different models in the Open Model Zoo

59
Natural Language Processing Models

For example, you can use a fine-tuned Are the meanings of these two sentences similar?
HuggingFace BERT PyTorch*model trained
on the Microsoft Research Paraphrase The company said it would check all ofits million-
Corpus (MRPC) to build a model that plus domestic workers to ensure they were legally
compares the meaning of sentences. employed.
The example is in the OpenVINO™
Notebooks on Github*: It has also said it would review all ofits domestic
https://github.com/openvinotoolkit/openvi employees more than 1 million to ensure they have
no_notebooks/tree/main/notebooks/105- legal status.
language-quantize-bert
The same meaning: yes

60
Live Action Recognition with OpenVINO™

Human action recognition finds actions over

time in a video. The list of actions in this
notebook is extensive (400 in total) and
covers Person Actions, (e.g., drawing,
drinking, laughing),Person-Person
Actions(e.g., hugging, shaking hands), and
Person-Object Actions(opening present,
mowing the lawn, playing "instrument").

You could find several parent-child groupings

on the label's list, such as braiding hair and
brushing hair, salsa dancing, robot dancing, or
playing violin and playing guitar. For more
information about the labels and dataset, see
"The Kinetics Human Action Video Dataset“
research paper.

61
Image Processing Models
Single Image Super Resolution

An Attention-Based Approach for Single

Image Super Resolution but with reduced
number of channels and changes in network
architecture. It enhances the resolution of
the input image by a factor of 4.

62
Model Downloader Usage

The model downloader allows the developer to easily fetch both publicly available models and Intel
provided models. It conveniently free developers from constant searching for the online locations of
models.

omz_downloader–all# Download all the models. This takes a while.

This will download all models. The --all option can be replaced with other filter options to
download only a subset of models. See the “Shared options” section.

Download a specific model:

omz_downloader--name {model} --output_dir{base_model_dir} --

cache_dir{omz_cache_dir} --precision {precision

63
Model Converter Usage

omz_converter–all

This will convert all models into the OpenVINO™ IR format. Models that were originally in that
format are ignored. Models in PyTorch* format will be converted in ONNX* format first.
The—all option can be replaced with other filter options to convert only a subset of models.
See the “Shared options” section.

omz_converter--name {model} --precisions {precision} --

download_dir{base_model_dir} --output_dir{base_model_dir}

This command will convert a specific model in the download_dir to a specific format in the output_dir

64
Model Quantizer Usage
Quantization is using lower-bit representations than higher bits for a given real-valued number. For
example, you can represent a continuous real number such as a 32-bit (or, 4 bytes) floating-point
number with a discrete number such as an 8-bit (or, 1 byte) integer number.

In this way, users will get a much smaller sized model, with the benefit of improved inference speed .

Before you run the model quantizer, you must prepare a directory with the datasets required for the
quantization process. This directory will be referred to as <DATASET_DIR> below.

omz_quantizer--name {model}--dataset_dir<DATASET_DIR>

65
OpenVINO™Notebooks -104-model-tools

Practice with key tools

▪ Access Open Model Zoo
▪ Download
▪ Display model info
▪ Convert
▪ Benchmark

https://github.com/openvinotoolkit/openvino_notebooks/
tree/main/notebooks/104-model-tools

66
Module 1

Intel® DevCloud for the Edge Workloads

67
Intel® DevCloud for the Edge Workloads
A Development Sandbox for Developers, Researchers, Faculty, Students and Startups

Instant No Multiple 24/7

AI Lab Cost Hardware Options Access

With the Intel® DevCloud for the Edge users can: How It Works
▪ Prototype on the latest hardware and software to
future proof the solution
▪ Benchmark the customized AI application
▪ Run AI applications from anywhere in the world
▪ Reduce development time and cost

Deep Learning Workbench + Intel® DevCloud

▪ Graphically analyze models
▪ Compare, visualize and fine-tune models
▪ Test on multiple remote hardware configurations

For more information visit https://www.intel.com/content/www/us/en/developer/tools/devcloud/edge/learn/overview.html

68
Sample Applications by vertical

Smart Cities Healthcare

▪ People Counter ▪ Pneumonia Detection
▪ Accelerated Object Detection ▪ Brain Tumor Segmentation

Safety Industrial
▪ Safety Gear Detection ▪ Weld Porosity
▪ Restricted Zone Notifier ▪ Low Power Object Detection
▪ Intruder Detection

Retail General
▪ Benchmark
▪ Store Traffic Monitor
▪ Time Series Analysis
▪ Store Aisle Monitor
▪ Noise Suppression
▪ Shopper Gaze Monitor
▪ Neural Network Compression Framework

69
Discussion Points

▪ What is the development environment provided by Intel® DevCloud for Edge Workloads?
▪ What are some of the features of Intel DevCloud for Edge that are useful for remote teams?
▪ Why is it necessary to generate your application's output in addition to benchmarking its performance?

70
Summary

▪ We learned about the concepts and applications of artificial intelligence in this module.
▪ We discussed Deep Learning, the motivation behind its use, and the various challenges that
developers face during the development phase.
▪ We also looked at various tools in the Intel® AI Ecosystem and learned about the
OpenVINO™ toolkit.
▪ Additionally, we explored and learned about Intel® DevCloud for Edge Workloads, as well as
how to create Deep Learning applications seamlessly.

71
Next Steps

▪ Register with Intel® DevCloud for Edge Workloads

▪ Download and install the OpenVINO™toolkit

72
Module 1

Hands-on Lab

73
Lab 1: Open Model Zoo

▪ This lab demonstrates how to download models from Open Model Zoo, which are validated
with OpenVINO™ toolkit.

▪ In this lab you will be using Intel® DevCloud for Edge Workloads which is a remote
development environment, which is designed to help you develop, test and run your
workloads for free on a cluster of the latest Intel® hardware and software.

74
Assignment 1.1

▪ Download a pretrained model from the Open Model Zoo and obtain
model information.
▪ You must accomplish the tasks listed below for this assignment.
1. Download the models listed below.
▪ handwritten-english-recognition-0001

▪ yolo-v4-tf

2. After downloading the model, use Model Information Dumper to view the
model's information.

75
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and
brands may be claimed as the property of others.
 System Configuration

1) Memory is installed such that all primary memory slots are populated.
2) Testing by Intel as of September 9, 2020.

77
Compounding effect of hardware and software configuration
See the compounding effect

78