1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.

ai

Contents

EDGE AI (HT TPS://VISO.AI/BLOG/EDGE-AI/)

AI Hardware: Edge Machine Learning Inference

ttps://x.com/viso_ai)
din.com/company/visoai/)

With the growing demand for real-time deep learning workloads, today’s standard
cloud-based Artificial Intelligence approach is not enough (https://viso.ai/deep-
learning/edge-computing-for-computer-vision/) to cover bandwidth, ensure data
privacy, or low latency applications. Hence, Edge Computing (https://viso.ai/deep-
learning/edge-computing-a-practical-overview/) technology helps move AI tasks to the
edge. As a result, the recent Edge AI (https://viso.ai/deep-learning/edge-ai-
applications-and-trends/) trends drive the need for specific AI hardware for on-device
machine learning inference.
Computer vision and artificial intelligence are transforming IoT devices at the edge. In
this article, you will learn about specialized AI hardware, also called AI accelerators,
created to accelerate data-intensive deep learning inference on edge devices
(https://viso.ai/edge-ai/edge-devices/) cost-effectively. Particularly, you will learn:

1. Machine learning inference (Basics)

2. The need for specialized AI hardware

3. List of the most popular AI accelerators

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 1/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

Contents
Machine Learning Inference at the Edge
AI inference is the process of taking a Neural Network Model (https://viso.ai/deep-
learning/artificial-neural-network/), generally made with deep learning, and then
deploying it onto a computing device (Edge Intelligence (https://viso.ai/deep-
learning/edge-intelligence-deep-learning-with-edge-computing/)). This device will then
process incoming data (usually images or video) to look for and identify whatever
pattern it has been trained to recognize (https://viso.ai/deep-learning/pattern-
recognition/).
While deep learning inference occurs in the cloud, the need for Edge AI
(https://viso.ai/deep-learning/edge-ai-applications-and-trends/) grows rapidly due to
bandwidth, privacy concerns (https://viso.ai/deep-learning/privacy-preserving-deep-
learning-for-computer-vision/), or the need for real-time processing.
Installing a low-power computer with an integrated AI inference accelerator close to
the source of data results in much faster response times and more efficient
computation. In addition, it requires less internet bandwidth and graphics power.
Compared to cloud inference, inference at the edge can potentially reduce the time for
a result from a few seconds to a fraction of a second (https://www.steatite-
embedded.co.uk/what-is-ai-inference-at-the-edge/).

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 2/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

People Detection with Edge AI Inference, here with privacy-preserving Face Blur
(https://viso.ai/deep-learning/face-blur-for-privacy-aware-deep-learning)

Contents

The Need for Specialized AI Hardware

Today, enterprises extend analytics and business intelligence closer to the points
where data generation occurs. Edge intelligence (https://viso.ai/deep-learning/edge-
intelligence-deep-learning-with-edge-computing/) solutions place the computing
infrastructure closer to the source of incoming data. This also places them closer to
the systems and people who need to make data-driven decisions in real time. In short,
the AI model (https://viso.ai/deep-learning/ml-ai-models/) is trained in the cloud and
deployed on the edge device.
Computer vision (https://viso.ai/deep-learning/what-is-computer-vision/) workloads
are high and computing tasks are highly data-intensive.

Edge Device Advantages

1. Speed and performance. By processing data closer to the source, edge computing
greatly reduces latency. The result is higher speeds, enabling real-time use cases.

2. Better security practices. Critical data does not need to be transmitted across
different systems. User access to the edge device can be very restricted.

3. Scalability. Edge devices (https://viso.ai/evaluation-guide/how-to-use-any-edge-

device-and-hardware-for-computer-vision/) are endpoints of an AI system that can
grow without performance limitations. This allows starting small and with minimal
costs. The development of cloud-based technology and edge computing has made
it easier than ever for businesses to scale their operations.

4. Reliability. Edge computing distributes processing, storage, and applications

across various devices, making it difficult for any single disruption to take down the
network (cyberattacks, DDoS attacks, power outages, etc.).

5. Offline-Capabilities. An Edge-based system can operate even with limited network

connectivity, a crucial factor for mission-critical systems.

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 3/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

6. Better data management. Fewer bottlenecks through distributed management of

edge nodes. Only processed data of high quality is sent to the cloud.

7. Privacy.
ContentsSensitive data sets can be processed locally and in real time without
streaming it to the cloud.

AI accelerators can greatly increase the on-device inference or execution speed of an

AI model and can also be used to execute special AI-based tasks that cannot be
conducted on a conventional CPU.

Most Popular Edge AI Hardware Accelerators

With AI becoming a key driver of edge computing, the combination of hardware
accelerators and software platforms is becoming important to run the models for
inferencing. NVIDIA Jetson (https://viso.ai/edge-ai/nvidia-jetson/), Intel Movidius
Myriad X (https://viso.ai/deep-learning/intel-neural-compute-stick-2/), or Google Coral
Edge TPU (https://viso.ai/deep-learning/google-coral/) are popular options available to
accelerate AI at the edge.

1.) VPU: Vision Processing Unit

Vision Processing Units (https://viso.ai/deep-learning/vision-processing-unit-vpu-for-
inference/) allow demanding computer vision and edge computing AI workloads to be
conducted with high efficiency. VPUs achieve a balance of power efficiency and
compute performance.
One of the most popular examples of a VPU is the Intel Neural Computing Stick 2
(https://viso.ai/deep-learning/intel-neural-compute-stick-2/) (NCS 2), which is based
on the Intel Movidius Myriad X VPU
(https://www.intel.com/content/www/us/en/products/processors/movidius-
vpu.html). Movidius Myriad X creates an architectural environment that minimizes
data movement by running programmable computation strategies in parallel with
workload-specific AI hardware acceleration.
The Intel Movidius Myriad X VPU is Intel’s first VPU that features the Neural Compute
Engine – a highly intelligent hardware accelerator for deep neural network
(https://viso.ai/deep-learning/deep-neural-network-three-popular-types/) inference.

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 4/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

The Myriad X VPU is programmable with the Intel Distribution of the OpenVINO Toolkit
(https://docs.openvinotoolkit.org/latest/index.html). Used in conjunction with the
Myriad Development Kit (MDK), custom vision, imaging, and deep neural network
Contents
workloads can be implemented using preloaded development tools, neural network
frameworks, and APIs.

2.) GPU: Graphics Processing Unit

A GPU is a specialized chip that can do rapid processing, particularly handling
computer graphics and image processing. One example of devices bringing an
accelerated AI performance to the Edge in a power-efficient and compact form factor
is the NVIDIA Jetson (https://developer.nvidia.com/embedded/jetson-modules) device
family.
The NVIDIA Jetson Nano (https://viso.ai/edge-ai/nvidia-jetson/) development board,
for example, allows neural networks to run using the NVIDIA Jetpack SDK. In addition
to a 128-core GPU and Quad-core ARM CPU, it comes with nano-optimized Keras and
Tensorflow libraries, allowing most neural network backends and frameworks to run
smoothly and with little setup.
With the release of the Xe GPUs (“Xe”), Intel is now also tapping into the market of
discrete graphics processors. The Intel GPU Xe (https://viso.ai/edge-ai/intel-xe-gpu-
architecture/) is optimized for AI workloads and machine learning tasks while
focusing on efficiency. Hence, the different versions of the Intel GPU XE family achieve
state-of-the-art performance at lower power consumption.

3.) TPU: Tensor Processing Unit

A TPU is a specialized AI hardware that implements all the necessary control and logic
to execute machine learning algorithms, typically by operating on predictive models
such as Artificial Neural Networks (ANN) (https://viso.ai/deep-learning/artificial-
neural-network/).
The Google Coral Edge TPU (https://viso.ai/deep-learning/google-coral/) is Google’s
purpose-built ASIC to run AI at the edge. The Google Coral TPU is a toolkit for Edge
that enables production with local AI. More specifically, Google Coral TPU onboard
device inference capabilities build and power a wide range of on-device AI
applications. Core advantages are the very low power consumption, cost-efficiency,
and offline capabilities.

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 5/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

Google Coral devices can run machine learning frameworks (https://viso.ai/deep-

learning/deep-learning-frameworks/) (such as TensorFlow Lite (https://viso.ai/edge-
ai/tensorflow-lite/), YOLO (https://viso.ai/deep-learning/yolov3-overview/), R-CNN
Contents
(https://viso.ai/deep-learning/mask-r-cnn/), etc.) for Object Detection
(https://viso.ai/deep-learning/object-detection/) to detect objects in video streams
from connected cameras and perform Object Tracking (https://viso.ai/deep-
learning/object-tracking/) tasks.

Google Coral (https://viso.ai/deep-learning/google-coral/) AI Accelerator TPU – USB version

(Source: Google Coral)

What’s Next?
Interested in reading more about real-world applications running on high-performance
AI hardware accelerators?

Everything you need to know about Edge AI (https://viso.ai/deep-learning/edge-ai-

applications-and-trends/)

Most popular computer vision applications (https://viso.ai/applications/computer-

vision-applications/) for AI accelerators

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 6/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai

Read about Vision Processing Units (https://viso.ai/deep-learning/vision-

processing-unit-vpu-for-inference/) to power Computer Vision Applications

Google Coral (https://viso.ai/deep-learning/google-coral/) Edge TPU for real-time

Contents
vision inference

Learn about the VPU Intel Neural Compute Stick 2 (https://viso.ai/deep-

learning/intel-neural-compute-stick-2/)

All-in-one platform to build, deploy, and scale computer vision

applications

Show me more
(https://viso.ai/)

viso.ai

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 7/8
1/27/25, 4:29 PM AI Hardware: Edge Machine Learning Inference - viso.ai
Product (https://viso.ai/platform/) Features (https://viso.ai/platform/)
Overview (https://viso.ai/features/) Computer Vision
Evaluation Guide (https://viso.ai/evaluation- (https://viso.ai/platform/computer-vision/)
guide/)Contents Visual Programming
Feature Index (https://viso.ai/feature-index/) (https://viso.ai/platform/low-code-computer-
vision/)
Academy (https://viso.ai/academy/)
Cloud Workspace
Security (https://viso.ai/security/)
(https://viso.ai/platform/cloud-workspace/)
Privacy (https://viso.ai/privacy/)
Analytics Dashboard
Solutions (https://viso.ai/solutions)
(https://viso.ai/platform/data-analytics/)
Pricing (https://viso.ai/pricing/)
Device Management
(https://viso.ai/platform/device-management/)
End-to-End Suite (https://viso.ai/platform)

Industries (https://viso.ai/solutions/) Resources (https://viso.ai/blog/)

Agriculture Blog (https://viso.ai/blog/)
(https://viso.ai/solutions/agriculture/) Learn (https://viso.ai/academy/)
Healthcare Evaluation (https://viso.ai/evaluation-guide/)
(https://viso.ai/solutions/healthcare/)
Support (https://support.visoai.com/)
Manufacturing
Whitepaper (https://viso.ai/viso-suite-
(https://viso.ai/solutions/manufacturing/)
whitepaper/)
Retail (https://viso.ai/solutions/retail/)
Security (https://viso.ai/solutions/?
tx_industry=security)
Smart City (https://viso.ai/solutions/smart-city/)
Technology
(https://viso.ai/solutions/technology/)
Transportation
(https://viso.ai/solutions/transportation/)

About (https://viso.ai/company/)
Company (https://viso.ai/company/)
Careers (https://viso.ai/jobs/)
Terms (https://viso.ai/terms-of-service/)
Contact (https://viso.ai/contact/)

https://viso.ai/edge-ai/ai-hardware-accelerators-overview/ 8/8