Contenido

1.1 Introducing Hyperconvergence and Cisco HyperFlex.............................................................7

Introduction...........................................................................................................................7
1.2 Introducing Hyperconvergence and Cisco HyperFlex.............................................................7
Traditional Data Center Design...................................................................................8
1.3 Introducing Hyperconvergence and Cisco HyperFlex...........................................................13
What Is Hyperconvergence?......................................................................................................13
1.4 Introducing Hyperconvergence and Cisco HyperFlex...........................................................22
What Is Cisco HyperFlex?...........................................................................................................22
1.5 Introducing Hyperconvergence and Cisco HyperFlex...........................................................28
1.6 Introducing Hyperconvergence and Cisco HyperFlex...........................................................38
Cisco HyperFlex Deployment Options........................................................................................38
1.7 Introducing Hyperconvergence and Cisco HyperFlex...........................................................41
Evolution of Cisco HyperFlex......................................................................................................41
2.1 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................48
Introduction.........................................................................................................................48
2.2 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................49
Cisco Server Deployment Models: Standalone vs. Managed.....................................................49
2.3 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................57
Cisco UCS M5 Overview.............................................................................................................57
2.4 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................63
Cisco UCS M5 Server Types........................................................................................63
2.5 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................68
Cisco UCS GPU Acceleration......................................................................................68
2.6 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................69
Cisco VICs and Their Benefits.....................................................................................................69
2.7 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................71
Cisco UCS Fabric Interconnects..............................................................................71
2.8 Describing Cisco UCS: Foundation of Cisco HyperFlex..........................................................80
Cisco UCS Manager.........................................................................................................80
3.1 Describing Cisco HyperFlex Software Components..............................................................84
Introduction.........................................................................................................................84
3.2 Describing Cisco HyperFlex Software Components..............................................................85
Virtual Machine Hypervisor.........................................................................................85
3.3 Describing Cisco HyperFlex Software Components..............................................................90
Log-Structured File System.........................................................................................................90
3.4 Describing Cisco HyperFlex Software Components..............................................................94
Cisco HyperFlex Snapshots vs. VMware Snapshots....................................................................94
3.5 Describing Cisco HyperFlex Software Components............................................................101
Cisco HyperFlex vs. Regular Virtualized Server.............................................101
3.6 Describing Cisco HyperFlex Software Components............................................................107
Cisco HyperFlex Data Distribution............................................................................................107
3.7 Describing Cisco HyperFlex Software Components............................................................119
Writing and Reading Process....................................................................................................119
3.8 Describing Cisco HyperFlex Software Components............................................................131
Data Optimization Overview.....................................................................................131
3.9 Describing Cisco HyperFlex Software Components............................................................138
Cisco HyperFlex vs. Other HCI Solutions...................................................................................138
3.10 Describing Cisco HyperFlex Software Components..........................................................143
Investigate Software Components of Cisco HyperFlex.............................................................143
4.1 Describing Cisco HyperFlex Hardware Components...........................................................161
Introduction.............................................................................................................................161
4.2 Describing Cisco HyperFlex Hardware Components...........................................................161
Introducing Cisco HyperFlex Servers........................................................................................161
4.3 Describing Cisco HyperFlex Hardware Components...........................................................170
Storage Technologies in Cisco HyperFlex.................................................................................170
4.4 Describing Cisco HyperFlex Hardware Components...........................................................173
Storage Components of Cisco HyperFlex Converged Nodes....................................................173
4.5 Describing Cisco HyperFlex Hardware Components...........................................................188
Other Components of Cisco HyperFlex Nodes.........................................................................188
4.6 Describing Cisco HyperFlex Hardware Components...........................................................207
Cisco UCS Fabric Interconnects................................................................................................207
4.7 Describing Cisco HyperFlex Hardware Components...........................................................214
Compute-Only Nodes...............................................................................................................214
4.8 Describing Cisco HyperFlex Hardware Components...........................................................217
Investigate Cisco UCS Part of HyperFlex...................................................................................217
5.1 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................240
Introduction.............................................................................................................................240
5.2 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................241
Installation Summary...............................................................................................................241
5.3 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................248
Software Prerequisites................................................................................................248
5.4 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................254
Hardware Prerequisites..............................................................................................254
5.5 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................256
Cisco HyperFlex Networking....................................................................................256
5.6 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................271
Required Deployment Information.........................................................................271
5.7 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................274
Installing Physical Components................................................................................................274
5.8 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................283
Configuring Upstream Switches...............................................................................................283
5.9 Installing and Expanding Standard ESXi Cisco HyperFlex....................................................285
Preparing Fabric Interconnects..............................................................................285
5.10 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................290
Deploying the Installer VM.......................................................................................................290
5.11 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................295
HyperFlex Installation...............................................................................................................295
5.12 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................298
Post-Installation.......................................................................................................................298
5.13 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................302
Intersight-Based HyperFlex Installation....................................................................................302
5.14 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................305
HyperFlex Expansion................................................................................................................305
5.15 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................319
Additional Installation Options.................................................................................................319
5.16 Installing and Expanding Standard ESXi Cisco HyperFlex..................................................333
Install Cisco HyperFlex..............................................................................................................333
6.1 Managing Cisco HyperFlex in vSphere Environment..........................................................371
Introduction.............................................................................................................................371
6.2 Managing Cisco HyperFlex in vSphere Environment..........................................................371
Management Interfaces Overview...........................................................................................371
6.3 Managing Cisco HyperFlex in vSphere Environment..........................................................375
Cisco HyperFlex Plug-In for vCenter.........................................................................................375
6.4 Managing Cisco HyperFlex in vSphere Environment..........................................................384
Cisco HyperFlex Connect..........................................................................................................384
6.5 Managing Cisco HyperFlex in vSphere Environment..........................................................387
Storage CLI...............................................................................................................................387
6.6 Managing Cisco HyperFlex in vSphere Environment..........................................................391
REST API Overview...................................................................................................................391
6.7 Managing Cisco HyperFlex in vSphere Environment..........................................................393
iSCSI and Cisco HyperFlex LUN Management...........................................................................394
6.8 Managing Cisco HyperFlex in vSphere Environment..........................................................398
Cisco HyperFlex Container Storage Interface...........................................................................398
6.9 Managing Cisco HyperFlex in vSphere Environment..........................................................401
ReadyClones.............................................................................................................................401
6.10 Managing Cisco HyperFlex in vSphere Environment........................................................405
Cisco HyperFlex Snapshots.......................................................................................................405
6.11 Managing Cisco HyperFlex in vSphere Environment........................................................411
Manage Cisco HyperFlex..........................................................................................................411
7.1 Maintaining Cisco HyperFlex..............................................................................................488
Introduction............................................................................................................................488
7.2 Maintaining Cisco HyperFlex..............................................................................................488
Cisco HyperFlex Upgrade Overview.........................................................................................488
7.3 Maintaining Cisco HyperFlex..............................................................................................493
Cisco HyperFlex Online Upgrade..............................................................................................493
7.4 Maintaining Cisco HyperFlex..............................................................................................520
Cisco HyperFlex Offline Upgrade..............................................................................................520
7.5 Maintaining Cisco HyperFlex..............................................................................................524
Cisco HyperFlex Maintenance Mode........................................................................................524
7.6 Maintaining Cisco HyperFlex..............................................................................................532
ESXi Upgrade............................................................................................................................532
7.7 Maintaining Cisco HyperFlex..............................................................................................534
Moving Cisco HyperFlex Group to Another vCenter.................................................................534
8.1 Designing Cisco HyperFlex..................................................................................................536
Introduction.............................................................................................................................536
8.2 Designing Cisco HyperFlex..................................................................................................536
Group Resiliency: VM-Level Final del formulario.....................................................................536
8.3 Designing Cisco HyperFlex..................................................................................................545
Group Resiliency: HXDP-Level..................................................................................................545
8.4 Designing Cisco HyperFlex..................................................................................................553
Cisco HyperFlex Scalability.......................................................................................................553
8.5 Designing Cisco HyperFlex..................................................................................................561
Cisco HyperFlex Storage Capacity............................................................................................561
8.6 Designing Cisco HyperFlex..................................................................................................566
Multiple Systems on One Cisco UCS Domain...........................................................................566
8.7 Designing Cisco HyperFlex..................................................................................................571
Cisco HyperFlex and External Storage......................................................................................571
8.8 Designing Cisco HyperFlex..................................................................................................578
Exporting Cisco HyperFlex Storage with iSCSI..........................................................................578
8.9 Designing Cisco HyperFlex..................................................................................................579
Licensing Tiers..........................................................................................................................579
8.10 Designing Cisco HyperFlex................................................................................................586
Cisco Smart Licensing...............................................................................................................586
8.11 Designing Cisco HyperFlex................................................................................................593
Cisco HyperFlex Use Cases.......................................................................................................593
8.12 Designing Cisco HyperFlex................................................................................................599
Graphical Processing Units on Cisco HyperFlex........................................................................599
8.13 Designing Cisco HyperFlex................................................................................................610
Designing Multicloud Data Center with Cisco HyperFlex.........................................................610
8.14 Designing Cisco HyperFlex................................................................................................638
HyperFlex: Releases Beyond 4.5(1a)........................................................................................638
9.1 Protecting Your Data..........................................................................................................644
Introduction............................................................................................................................644
9.2 Protecting Your Data..........................................................................................................645
Disaster Recovery Overview..................................................................................................645
9.3 Protecting Your Data..........................................................................................................650
Third-Party Data Restore Solutions..........................................................................................650
9.4 Protecting Your Data..........................................................................................................652
Cisco HyperFlex Native Replication Solutions...........................................................................652
9.5 Protecting Your Data..........................................................................................................660
Cisco HyperFlex One-to-One Native Replication......................................................................660
9.6 Protecting Your Data..........................................................................................................682
9.7 Protecting Your Data..........................................................................................................686
Cisco HyperFlex Many-to-One Replication...............................................................................686
9.8 Protecting Your Data..........................................................................................................698
Data at Rest Encryption............................................................................................................698
9.9 Protecting Your Data..........................................................................................................705
Data at Rest Encryption: Remote Key Management................................................................705
9.10 Protecting Your Data........................................................................................................707
Protect Your Cisco HyperFlex VMs...........................................................................................707
10.1 Introducing Cisco HyperFlex Stretched Deployment........................................................730
Introduction............................................................................................................................730
10.2 Introducing Cisco HyperFlex Stretched Deployment........................................................730
Stretched Deployment Overview.............................................................................................730
10.3 Introducing Cisco HyperFlex Stretched Deployment........................................................736
Prerequisites...........................................................................................................................736
10.4 Introducing Cisco HyperFlex Stretched Deployment........................................................739
Data Distribution......................................................................................................................739
10.5 Introducing Cisco HyperFlex Stretched Deployment........................................................754
Datastores and VM Affinity......................................................................................................754
10.6 Introducing Cisco HyperFlex Stretched Deployment........................................................757
Installation Process..................................................................................................................757
10.7 Introducing Cisco HyperFlex Stretched Deployment........................................................761
Monitoring and Maintenance..................................................................................................761
10.8 Introducing Cisco HyperFlex Stretched Deployment........................................................768
Investigate Stretched Group....................................................................................................768
10.9 Introducing Cisco HyperFlex Stretched Deployment........................................................777
Install and Manage Stretched HyperFlex Group.......................................................................777
11.1 Introducing Cisco HyperFlex Edge....................................................................................808
Introduction............................................................................................................................808
11.2 Introducing Cisco HyperFlex Edge....................................................................................808
Cisco HyperFlex Edge System Overview...................................................................................808
11.3 Introducing Cisco HyperFlex Edge....................................................................................816
Edge Design Considerations.....................................................................................816
11.4 Introducing Cisco HyperFlex Edge....................................................................................827
Cisco HyperFlex Edge Network Topologies..............................................................................827
11.5 Introducing Cisco HyperFlex Edge....................................................................................836
Prerequisites and Recommendations.......................................................................................836
11.6 Introducing Cisco HyperFlex Edge....................................................................................843
Installation Process..................................................................................................................843
11.7 Introducing Cisco HyperFlex Edge....................................................................................864
Management and Monitoring..................................................................................................864
11.8 Introducing Cisco HyperFlex Edge....................................................................................877
Upgrades and Maintenance.....................................................................................................877
11.9 Introducing Cisco HyperFlex Edge....................................................................................880
Investigate HyperFlex Edge......................................................................................................880
12.1 Troubleshooting Cisco HyperFlex.....................................................................................896
Introduction.............................................................................................................................896
12.2 Troubleshooting Cisco HyperFlex.....................................................................................897
Cisco HyperFlex Best Practices.............................................................................897
12.3 Troubleshooting Cisco HyperFlex.....................................................................................900
Cisco HyperFlex Security Hardening...................................................................900
12.4 Troubleshooting Cisco HyperFlex.....................................................................................902
Troubleshooting Guidelines......................................................................................902
12.5 Troubleshooting Cisco HyperFlex.....................................................................................911
Generating Tech Support Bundles.........................................................................911
12.6 Troubleshooting Cisco HyperFlex.....................................................................................916
Common Troubleshooting Issues...........................................................................916

1.1 Introducing Hyperconvergence and Cisco HyperFlex

Introduction
A data center, like all other parts of IT, is constantly evolving and adapting to
today's businesses—from faster processors running demanding applications, to
larger disks that store more data, and better protocols delivering data faster.

The core considerations for the data center remain since its inception—redundancy,
scalability, accessibility, security, and manageability.

Hyperconvergence is a data center hardware deployment model that redefines the

architecture of the data center and promises an easy setup and operations, with a
pay-as-you-grow approach to scaling, and tight integration with tools that enable
the infrastructure to become a private cloud solution.

The concept of hyperconvergence is not something new. It is a solution that large

companies such as Google and Facebook have been running for years, and now the
technology has matured and is available to general enterprise customers.
1.2 Introducing Hyperconvergence and Cisco HyperFlex

Traditional Data Center Design

In this video, let's examine traditional data center designs. Traditionally, most data
centers are designed in a modular fashion. They consist of servers that offer comput
e capabilities and either bare metal or virtualized servers.

Centralized storage devices for maintaining data, storage area network, SAN
switches, providing lossless connectivity using Fiber Channel to the storage devices
, and lastly, local area network, or LAN switches, which provide the ethernet
connectivity. The problem is the separation of the storage devices storage networks
and servers create silos, which several teams with different expertise need to
maintain.
This requires much communication and coordination, when it comes to
implementing changes, which can be expensive and time consuming. Traditional
Cisco converged solutions are reference architectures using Cisco Unified
Computing System, or UCS servers, Cisco MDS SAN switches, and a storage
solution from a third party vendor. Storage solutions include Pure Storage, Dell,
EMC, IBM, and NetApp.

All three parts of the design are required to make the solution complete. And they
have to be documented in form of Cisco validated designs. Cisco HyperFlex is an
optional part of the converged solution, and you will actually see this in some of the
marketing materials. However, there is a big difference between Converged
Solutions and HyperFlex, which we're going to discuss as we continue.

But before we get into HyperFlex, let's take a look at some validated designs for
converged infrastructures. FlashStack is a converged infrastructure solution from Cisco
and Pure Storage, which can scale without disrupting as the business grows. FlexPod is a
pre-validated data center platform built on Cisco UCS, the Cisco Nexus family switches
and NetApp data management solution. VBlock systems are built with compute and
networking from Cisco storage from Dell EMC and virtualization technologies from
VMware.

Lastly, VersaStack combines a Cisco UCS integrated infrastructure with the IBM store wise
storage solution. This includes Cisco UCS, Cisco Nexus, as well as Cisco MDS switches and
UCS director. Thanks for watching.

Most data centers that you see today are designed with storage arrays and storage
area networks (SANs) providing storage:

 Servers that offer compute capabilities (either bare metal or virtualized and
using hypervisor technology such as VMware ESXi or Microsoft Hyper-V).
 Centralized storage that offers the capacity to reliably store the data.
 SAN switches, which offer connectivity between servers and centralized
storage.
 LAN switches, which offer connectivity from the data center to the company
users and the internet.

The approach shown in the figure is just one design, although it is very commonly
used in enterprises. Typically, SAN topologies use a specific set of SAN-enabled
devices in a dedicated network design as shown in the example.

There are additional options available within the SAN design:

 Servers can be either rack-mounted or blade servers.

 Storage can be direct-attached storage (DAS), network-attached storage (NAS),
or SAN. Applications that require large amounts of fast storage typically use
SAN with Fibre Channel or Internet Small Computer System Interface (iSCSI)
protocols.
 You can eliminate a dedicated SAN network from your design by running iSCSI
or Fibre Channel over Ethernet (FCoE) instead of Fibre Channel Protocol, but
Fibre Channel is still more popular.

The separation of storage, storage networks, and servers often leads to silos. Silos
are sections of the infrastructure managed by one team. Since the various
infrastructure components are managed by a different team, the silos can pose
challenges even when implementing minor changes. Each individual team must be
involved in the change and usually each team has their own process structure and
specific skill sets, causing delays and even severe bureaucratic overhead. Silos also
significantly increase the necessary communication in the design phase, making the
design process take much longer than it normally would.

Traditional Converged Solutions

Cisco converged solutions are the reference architectures of Cisco Unified
Computing System (UCS) Servers, Cisco Multilayer Director Switch (MDS) SAN
Switches, and a storage solution from a third-party vendor such as PURE Storage,
Dell EMC, IBM NetApp, and others.

A typical build-it-yourself three-part data center solution with servers, switching,

and storage requires experts for all three parts of the solution to make the solution
complete and adapted for a given application. In comparison, Cisco Converged
Infrastructure solutions are validated systems that are documented in the form of
Cisco Validated Designs.

Core benefit of a converged infrastructure is its validated design. For example:

 If you are looking for a validated design with Cisco and Pure Storage to run
Citrix XenDesktop, you can buy a FlashStack bundle and follow the Cisco
design guide to set up the entire solution (servers, SAN and LAN switching,
centralized storage, and applications).
 If you are looking for a validated design with Cisco and NetApp to run SAP
solutions, you can buy a FlexPod bundle and follow the Cisco design guide to
 set up the entire solution (servers, SAN and LAN switching, centralized storage,
and SAP).

You can learn more about the Cisco converged infrastructures at

https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/data-
center-design-guides-all.html (cisco.com login required).

Here are short descriptions of key converged infrastructure characteristics:

 FlexPod is a pre-validated data center platform that is built on Cisco UCS,

Cisco Nexus Series Switches, and NetApp data management solutions. FlexPod
configurations and workloads are published as Cisco Validated Designs that
allow you to deploy applications and converged infrastructure faster, using
proven solutions.
 VersaStack combines the innovation of the Cisco UCS-Integrated Infrastructure
with the efficiency of the IBM Storwize storage system. The Cisco UCS-
Integrated Infrastructure includes Cisco UCS, Cisco Nexus and Cisco MDS
Series Switches, and Cisco UCS Director.
 FlashStack, a converged infrastructure solution from Cisco and Pure Storage,
brings the power and efficiency of all-flash storage to your entire Cisco UCS-
powered data center. With industry-leading components from Cisco and Pure
Storage, FlashStack is a powerful and resilient converged infrastructure solution
that will dramatically reduce your time to value. It scales without disruption as
your business grows and provides all-flash speed and reliability for every
workload.
 VxBlock Systems provide a wide range of solutions to meet your requirements
for size, performance, and scalability. VxBlock Systems are built with industry-
leading compute and networking from Cisco, storage from Dell EMC, and
virtualization technologies from VMware.
 Hitachi Adaptive Solutions for Converged Infrastructure (CI) solutions
propose a powerful and scalable architecture, applying the strengths of both
Cisco and Hitachi Vantara. Cisco devices include Cisco UCS, Cisco Nexus
Series Switches, and Cisco MDS SAN Series Switches. Hitachi Vantara part of
the solution is a storage system.
1.3 Introducing Hyperconvergence and Cisco HyperFlex

What Is Hyperconvergence?

Now, let's take a closer look at Hyperconvergence and Cisco HyperFlex. HyperFlex is Cisco's
interpretation of what a hyper-converged solution should look like. A hyper-verged
infrastructure is built using x86-based servers and connecting them to a LAN network. Each
of the servers has its own processor, memory, and local disks. Each server in this figure is
virtualized using a hypervisor, such as VMware ESXi.

The servers also run HyperFlex, which is an intelligent piece of software for hyper-
convergence. HyperFlex itself is a controller virtual machine, or CVM. It's also referred to as
SCVM or STCVM, both of which mean the Storage Controller Virtual Machine. This hyper-
converged infrastructure is also referenced using the acronym HCI.
The controller virtual machines, or CVMs, these are used to create one great big pool of
resources for the VM to consume. All of the disks in all of the servers are grouped together
using the Hyperconvergence software to present one NFS 3 volume. The VM administrator
defines one or more data stores and then uses them when provisioning the virtual machines.
All of the storage is then shared between the servers within the hyper-converged cluster.

The software-defined storage, or SDS, is a computer program that manages the data storage
resources and its functionalities. It has no dependency on the underlying physical storage
hardware. HyperFlex basically takes a bunch of disks in a bunch of different servers and
abstracts them into one single storage pool.

Hyperconvergence transforms rack servers into a single pool of compute and storage
resources that you can see in the figure. The top section illustrates three nodes connected to
a LAN, and those three nodes contain a pool of resources to the VM, which include CPU,
memory, and storage. In the middle section, this one illustrates the expansion of a three-node
cluster to five nodes when the cluster is running out of resources. Adding new nodes just seam
lessly expands the resources that are available to the virtual machines.

Lastly, the bottom section illustrates the intentional retirement of a server, or possibly a server
that failed in the cluster. In either case, the pool of resources available will shrink in the
cluster but it shouldn't impact the workload itself. The hyper-converged solution is really the
most flexible and simple architecture in your data center. This is because it does not require
any fibre channel or iSCSI and you don't have to worry about fibre channel zoning, device alias,
or LUN configurations.

With the hyper-converged solution, you start small and then expand as needed, usually with
a minimum of three nodes. And then expand when more resources are required for the cluster
, basically meaning you're running out of disk space. This includes scaling up by adding more
disks and then scaling out by adding more servers. Both expansion options should have no
effect on the performance of the hyper-converged clusters.

Hyper-converged system revenue has drastically increased since 2016. And Cisco HyperFlex is
one of the major players in the market. There's a couple reasons for this rapid growth for Cisco
. One reason is the improvements in the processors. Now, this is important because
Hyperconvergence relies on compute power for running in both the hypervisor and the hyper-
convergence software. Also, mechanical hard drives are being phased out in favor of solid
state disk, or SSDs, because the prices have dropped consistently-- and sharply—over the last
few years.
Cisco HyperFlex tightly integrates both the software and hardware to control the entire stack
of the hyper-converged solution. The hardware platforms used in HyperFlex as are marked as
HX servers, or HyperFlex servers, but actually they're just Cisco UCS servers. The only
difference is the product ID in the front bezel-- they look different. The software used in
HyperFlex came from the acquisition of Springpath, which means that Cisco has complete
control over both the hardware and the software portion of the solution. Also, HyperFlex is
tightly integrated into the Cisco Network portfolio providing powerful service integrations.

This figure here shows the Cisco UCS family portfolio, which offers diverse types of
infrastructure options. All of them are built using a single API for management. Cisco UCS is the
foundation for Cisco HyperFlex, Cisco UCS is already built with a single point of management
and connectivity, and it's designed as a single virtual blade server chassis that can span
multiple chassis and racks of blade and rack-server-based nodes. This puts Cisco in a unique
position of being able to deliver hyper-converged solutions that can incorporate blade and rack
system in its architecture.
The Cisco HyperFlex systems are more agile because they perform scaling, inter-operate better
with complexities that are completely hidden from the user. The cluster ships with the
hypervisor and data platform pre-installed and ready to launch through the installation wizard.
It's interconnected with 10 or 40 gigabits of network bandwidth, the system automatically will
discover new hardware when it's installed, and the management capabilities enable you to
install and operate your Cisco HyperFlex solution in the data center you already have.

The Cisco HyperFlex systems combine software-defined computing, with Cisco UCS Servers,
software-defined storage, using powerful Cisco HyperFlex Data Platform software, and
software- defined networking, using the Cisco UCS Fabric Interconnects. The result is a cluster
that comes up and configures itself quickly and can scale easily when more resources are
required.
Finally, let's take a look at the physical topology. Here you see the Cisco UCS HX Series Servers
are connected to a pair of Cisco UCS 6000 Series Fabric Interconnects forming a cluster that
can have between four to 64 servers. The Fabric Interconnects are deployed in a highly-
available cluster pair. The Fabric Interconnects provide high-speed, low-maintenance
connectivity, server orchestration using the built in Cisco UCS Manager software. Lastly, Cisco
Nexus switches are recommended for the topper row switches, which are not really part of the
HyperFlex bundle, but they provide connectivity to the network and needs support of 10 or 40
gigabits as well as jumbo MTUs. All right, thanks for watching.

Hyperconverged infrastructure is a deployment model that utilizes the speed of

modern Ethernet networks to allow communication between individual servers of a
hyperconverged server group. Since this network is used for storage operation and
the workload running on the server group, it needs to have extremely low latency,
reliability, and throughput.

Hyperconverged infrastructure is built by taking x86-based servers and connecting

them to a high-performance LAN network. Each server has its own processor,
memory, and local disks. The figure shows three such servers connected to a LAN.
Each server is virtualized using a hypervisor (the example in the figure is VMware
ESXi) and each server also runs an intelligent piece of software for
hyperconvergence. This intelligent piece of software is, in the case of Cisco
HyperFlex, a controller virtual machine (CVM).
Note
You will also see CVM being referred to as SCVM or StCVM in the solution
documentation. All of the terms refer to the same storage controller virtual machine
(CVM).

People often refer to hyperconverged infrastructure by the acronym HCI.

CVMs communicate between each other over the Fast Ethernet network to create
one large pool of resources for the VMs to consume. All the disks in all the servers
are bundled together using hyperconvergence software to present it to VMs as a
shared volume. The VM administrator can then simply define one or more
datastores and use them when provisioning the VMs. All the storage is shared
between all the servers in a hyperconverged group, so you could consider Cisco
HyperFlex as a software-defined storage (SDS) solution.

Note
SDS is a computer program that manages data storage resources and functionality
and has no dependencies on the underlying physical storage hardware. Cisco
HyperFlex takes a bunch of disks in different servers and, using the software,
abstracts them into one single storage pool.

On top of storage, there is a hypervisor group. In this example, this hypervisor

group is built using VMware vSphere and its ESXi hypervisor:

 The VM (any of the 9 VMs shown in the figure) runs on one of the hosts, and
you can move that VM between the hosts manually (using vMotion) or move it
dynamically, based on VMware’s distributed resource scheduler (DRS).
 If a host or hypervisor fails, VMware high availability will make sure that the
VM runs on another host with no more than a few pings lost.
 From the perspective of the vSphere group, a hyperconverged datastore is just
one giant storage (shared volume).
Motivations for Hyperconvergence
The hyperconverged solution transforms rack servers into a single pool of compute
and storage resources. The individual hardware components act as system building
blocks and can be added or removed depending on the resource requirements.

The topology of an HCI solution is designed to be as uniform as possible. This

makes expanding the solution relatively easy and minimal additional design phase
is required to increase the available resources. Since the design of the infrastructure
is modular, the size changes can be performed without interrupting the workload
already running on the infrastructure, which is key for the availability requirements
of modern applications.

There are some inherent characteristics and benefits provided by the

hyperconverged infrastructure design:

 In most hyperconverged solutions, the minimum group size is three servers

(commonly referred to as nodes). The top section of the following figure shows
three nodes connected to a LAN, and those three nodes give a pool of resources
(CPU, memory, and storage) to the VMs.
 The middle section of the figure shows the expansion of your three-node group
to five nodes. When you are running out of resources in your group, you can
always add more nodes, which would seamlessly expand your pool of resources.
Expanding the group in a hyperconverged solution usually means buying new
servers, connecting them to the upstream LAN connection, and running an
expansion wizard.
 The bottom section of the figure shows the retirement of one server. When you
retire a server, the pool of resources for your VMs becomes smaller. Similarly,
if one of the servers fails, the pool of resources would shrink.

Deploying a hyperconverged solution consists of preparing the infrastructure, such

as top-of-rack (ToR) switch configuration, Domain Name System (DNS), Network
Time Protocol (NTP), and Active Directory setup, connecting servers to an
upstream Ethernet connection, and running the installation software. There is no
Fibre Channel configuration to configure—no Fibre Channel zoning or device
aliases, no logical unit number (LUN) creation, and no LUN masking.

Because of the easy scalability and large maximum resource availability of a single
hyperconverged server group, HCI is the natural choice for private cloud
deployments. Hyperconverged solutions enable you to start small and expand when
needed, so you will not over- or under-provision resources for your applications and
business.

When considering hyperconverged solutions, look for a vendor that not only offers
a hyperconverged platform, but also offers tightly integrated tools to build your
private cloud solution. As the solution scales over time, the infrastructure
management can add additional management overhead.

State of Hyperconvergence
According to the International Data Corporation (IDC) Worldwide Quarterly
Converged Systems Tracker, revenue from hyperconverged systems sales grew
consistently over the last years. The hyperconverged systems also take up an
increasing relative part of modern data centers. The IDC reports can be found on
their website.

Major players on the hyperconverged market are:

 Cisco HyperFlex
 Nutanix
 VMware virtual SAN (VSAN) and Dell EMC
 HPE SimpliVity
 Huawei FusionCube

Hyperconvergence has really taken off in the past couple of years. Here are some of
the reasons:

 Improvements in processors: You need to have capable processors, because

hyperconvergence relies on compute power for the purposes of running the
hypervisor and the hyperconvergence software (either in the kernel of a
hypervisor for VMware VSAN, or as a CVM as for Cisco HyperFlex or
Nutanix).
 SSD price drop: Hard drives are one of the last mechanical parts of the
computer and are being phased out in favor of solid-state disks (SSDs). Flash is
a superior technology in every way, but its high price slowed adoption until
about 2015. Since then, the prices have dropped consistently and sharply. All-
Flash hyperconverged solutions accelerate the move to Flash storage with off-
the-shelf pricing and rapid support of the latest Flash storage technologies, such
as Intel Optane.
 Improvements in Ethernet network capabilities: Most data centers run equipped
with Fast Ethernet networking and it is not uncommon to find 100 Gbit/s
connectivity used. The latency and reliability has also improved over the years
with the introduction of Ethernet enhancements such as Data Center Bridging
(DCB).

1.4 Introducing Hyperconvergence and Cisco HyperFlex

What Is Cisco HyperFlex?

The Cisco HyperFlex solution is a Cisco interpretation of what a hyperconverged
solution should look like. Cisco HyperFlex tightly integrates software and hardware, for
an easy-to-install and easy-to-operate solution.

Some vendors define hyperconverged solutions as software only, which means that
when you buy hyperconverged software, you can use any hardware. However, this does
not mean that you can actually deploy HCI effectively on any hardware. The software-
only HCI offerings come with a hardware compatibility list (HCL) specifying which
hardware the solution works well with. Not all of the hardware on the list is equally
suitable, and there are also often huge disadvantages when using mixed hardware.
Uniformity is very important in distributed systems to ensure consistent performance
and avoid bottlenecks.

A major side-effect of all the hardware and deployment considerations that software-
only HCI introduces is more difficult design, installation, and management. It is also
more difficult to integrate these systems into broader solutions as their individual
components are not predictable.

Cisco HyperFlex takes a more complete approach to hyperconverged infrastructure

deployment:

 HyperFlex is a Cisco single-vendor solution for hyperconvergence.

 It is a complete solution: a bundle of hardware, software, and management.

o The hardware is tried-and-tested Cisco UCS C-Series Servers and Cisco UCS
Fabric Interconnects.
o The underlying software was developed by a company called Springpath that
was acquired by Cisco and is part of Cisco since September 2017.
o Installation is greatly simplified and can be performed remotely from the cloud.

 Cisco HyperFlex is deeply integrated with the rich Cisco ecosystem and public
cloud, making it the perfect multi-cloud solution.
Cisco controls the entire stack of the HyperFlex solution: hardware and software.
Hardware platforms used in Cisco HyperFlex are marked as HX-Series but are
essentially Cisco UCS C-Series Servers. The only differences between the HX-Series
and C-Series platforms are the different product IDs (PIDs) and the front bezel. The
software used in Cisco HyperFlex comes from a company called Springpath. Springpath
was a visionary in hyperconvergence software, based in Sunnyvale, CA. In the
beginning of 2016, Cisco reserved exclusive partnership with the company and in the
fall of 2017, Cisco bought the company outright. This acquisition means that Cisco has
complete control over both the hardware and software part of the HyperFlex solution.

Cisco HyperFlex is tightly integrated into the Cisco portfolio, so it allows powerful
service integrations:

 HyperFlex cloud management platform: Cisco Intersight is a Software-as-a-

Service (SaaS) cloud management, installation, and upgrade platform capable of
full-stack management and updating of Cisco HyperFlex. Cisco Intersight is the
core management platform for a Cisco data center.
 Application optimization: Cisco Intersight Workload Optimized (IWO) analyzes
workloads from the Cisco SaaS cloud to assist with application performance
optimization on HyperFlex, general data center systems and in the public cloud.
 Optimization of application services: Cisco AppDynamics enables performance
monitoring of the hybrid applications running application tiers on HyperFlex and
across clouds.
 Cloud mobility: Cisco CloudCenter enables workload mobility between HyperFlex
and public and private clouds, including private cloud self-service Infrastructure-as-
a-Service (IaaS) capabilities.
 On-premise container platform: Intersight Kubernetes Service integrates with
Cisco HyperFlex on a fundamental level and enables deployment and management
of Kubernetes systems globally.
Cisco HyperFlex: Part of the Cisco UCS Family
Cisco HyperFlex uses the Cisco UCS hardware as the hardware part of its solution.
Cisco UCS is a tried-and-tested data center system that is running in many data centers
across the world. The Cisco HyperFlex solution, therefore, benefits from the tried-and-
tested design of its hardware.

Note
The following figure shows the Cisco UCS family portfolio. Mainstream computing
includes blade-based systems. Hyperconverged infrastructure includes Cisco
HyperFlex. Converged infrastructure includes converged solutions that Cisco built in
partnership with third-party vendors. Scale-out infrastructure includes Cisco UCS C-
Series and S-Series-based systems with third-party software on top. You can manage all
previously mentioned infrastructure types using Cisco UCS Manager, Cisco UCS
Director, and Cisco Intersight. The Cisco UCS portfolio offers diverse type of
infrastructure options, but all are built with a single application programming interface
(API), making potential integration with your existing management straight-forward.

Cisco UCS, the foundation for Cisco HyperFlex systems, is built with a single point of
management and connectivity for the entire system. The system is designed as a single
virtual blade server chassis that can span multiple chassis and racks of blade and rack
server-based nodes. Thus, Cisco is in the unique position of being able to deliver a
hyperconverged solution that can incorporate blade-and-rack systems in its architecture,
offering greater flexibility than any other solution. Because Cisco develops the servers
on which Cisco HyperFlex nodes are based, you can count on having the latest
processor technology available to speed your performance.

Cisco HyperFlex systems are more agile because they perform, scale, and interoperate
better:

 Deployment is fast and easy: Your HyperFlex ships with the hypervisor and data
platform preinstalled and ready to launch through the installation wizard.
 Integrated networking brings high performance: Your HyperFlex is
interconnected with low, consistent latency and with 10-G, 25-G, and 40-G network
bandwidth.
 Scaling is fast and simple: The system automatically discovers new hardware when
it is installed. Adding it to the group takes only a few mouse clicks. New logical
availability zones (LAZs) allow groups to grow to up to 64 nodes while minimizing
the impact of multiple node failures.
 Interoperability is straightforward: Management capabilities enable you to install
and operate your Cisco HyperFlex System in the data center you have today, with
high-level management tools that support operations across your hyperconverged
and your traditional infrastructure.

If you want to make a configuration change after the installation, you have full access to
do so:

 The same Cisco Fabric Interconnects and Cisco UCS Manager that you are used to.
 Cisco HX-Series servers are just Cisco UCS C-Series servers with a different PID
and different front bezel.
 ESXi and vCenter or Hyper-V and Hyper-V Manager or System Center Virtual
Machine Manager (SCVMM)
 You can use Cisco UCS Director or Cisco Intersight to orchestrate Cisco
HyperFlex-based systems in the same way that you would by using regular Cisco
UCS. You add Cisco HyperFlex as a pod to Cisco UCS Director and then access it
as simply as any other resource.

Note
PID stands for Product ID and is an identifying product designation assigned to a
product.

Cisco HyperFlex: Complete Hyperconvergence

Cisco offers the first hyperconverged platform that is designed as an end-to-end
software-defined infrastructure and eliminates the compromises found in first-
generation products. Cisco designed Cisco HyperFlex to support a broader range of
applications and workloads in the data center, private and hybrid clouds, remote
locations, and edge-computing environments. This new generation extends
hyperconverged system deployment, management, and support beyond your central data
center to multicloud environments and to the network edge.
Cisco HyperFlex Systems combine:

 Software-defined computing in the form of Cisco UCS servers with Intel Xeon
scalable processors.
 Software-defined storage with the powerful Cisco HyperFlex Data Platform
software.
 Software-defined networking with Cisco UCS fabric that optionally integrates
smoothly with Cisco Application Centric Infrastructure (ACI).

Together, these elements comprise an adaptive infrastructure that lets you integrate
easily with your existing infrastructure. The result is a server group that comes up and
configures itself in an hour or less and that scales resources independently to closely
match your application resource needs.

Cisco HyperFlex: Physical Topology

Cisco HyperFlex is delivered as a bundle of hardware and software.
 The hardware portion of the Cisco HyperFlex solution comprises these components:

 Servers

o Cisco UCS HX Series servers connected to Cisco UCS Fabric Interconnects

using two Twinax cables each. Optionally, you can connect each server
northbound with more than two links but use of two uplinks is the most common
configuration.
o Since HyperFlex Version 3.0, the minimum is 3 servers and the maximum is 64
servers. These are limits for the Standard and Stretched vSphere-based
deployment type. Since HyperFlex Version 3.5, the vSphere-based Stretched
deployment can have between 4 and 64 servers, while Hyper-V-based Standard
deployments are limited to 32 nodes. HyperFlex Edge also supports the two
node deployment option that requires Cisco Intersight.

 Fabric Interconnects

o Cisco UCS generation 2, 3, or 4 of Fabric Interconnects (Cisco UCS 6200, 6300,

6400 Series), deployed in group pair.
o Provides high-speed, low-latency connectivity and server-level orchestration
(Cisco UCS Manager).
 ToR switches

o Not part of the Cisco HyperFlex bundle.

o Jumbo maximum transmission unit (MTU) is required.
o Cisco Nexus Series Switches are recommended because they deliver high
density, configurable ports for flexible access deployment and migration.

Note
Cisco UCS Fabric Interconnects of the second generation (6200 series) are in the End of
Sale (EOS) status and cannot be purchased. They are, however, supported until May of
2024, and can be used for HyperFlex deployments on pre-existing Cisco Fabric
Interconnects.
1.5 Introducing Hyperconvergence and Cisco HyperFlex

Cisco HyperFlex Basics

In this video, we'll review a primer of Cisco HyperFlex. First of all, the Cisco HyperFlex
installation is easy because it's deployed using a wizard. Here are the three high-level
steps for installing a standard vSphere-based Cisco HyperFlex solution.

First, racking and stacking, installing the servers and fabric interconnects, and
interconnecting everything all together with the topper row switches. That was the first
step. Second, prepare the infrastructure. And then third, install the cluster. This includes
deploying HyperFlex using that intuitive wizard I mentioned and performing the post-
installation task. Now you're ready to start running your virtual machines.
So let's examine a scenario, here, where you have a three-server system and you're
running out of storage and you want to use HyperFlex to expand the storage. On the left
side of the figure over here, the three servers have empty disk slots, so you can expand
the storage by inserting more disks into each server. The inserted disk will automatically
be added to the storage pool. Each server in the cluster should have the same number of
disk because HyperFlex relies on system symmetry for best performance.

Now, on the right side of the figure, over here, the servers are already full of disks. There'
s no more room to insert anything. So in this case, a new server with the same number of
disks can be added to the system using the expansion wizard.

This next scenario, here, is a three-server system which is running out of compute power
for the VMs. On the left side of the figure, HyperFlex server is added to the system—the
same as when you're running out of storage. However, this will give you extra compute
power, as well as additional storage.

HyperFlex nodes with compute and storage are commonly referred to as converged nodes
. But if you look at the right side of the figure, here shows another option for expanding
the HyperFlex cluster, this time with compute power only. Here, a compute-only node
is added to increase horsepower, but it doesn't add any additional storage capacity.
Compute-only nodes are B series and C series with HyperFlex software installed.
HyperFlex is a resilient system when you size it properly for the workloads. A storage
cluster status described as using both operational and resiliency status. If one disk fails in
the servers, the resiliency status will change to unhealthy but the operational status will
still be online. Once the self-healing process is complete, then the status will return to
healthy. The HyperFlex cluster can even sustain the failure of an entire server.

You should always deploy Cisco HyperFlex with two Cisco UCS fabric interconnects for
high availability. The fabric interconnects are needed for orchestration of Cisco UCS—
using UCS manager-- and to provide a high speed and low latency network.

Cisco HyperFlex works by, once again, distributing the data across all the servers in the
cluster, providing better performance than the other customers, plus more resiliency.
This superior performance is outlined in a Cisco HyperFlex white paper written by ESG Lab
validation. In this paper, they stated that HyperFlex not only produces the typical
benefits of a hyper-converged infrastructure, but also performs the performance that
mission critical virtualized workloads demand.
Finally, for a regular ESXi-based cluster, there's several management points. VMware
vCenter is a data management server application developed to monitor the virtualized
environments. On a day-to-day management basis, vCenter is the only tool that
you'll ever need to manage HyperFlex. HyperFlex Connect is an alternative to VMware
vCenter, and it's available with a HyperFlex without the installation. It is very intuitive
HTML5 interface but is not fully featured with all of the VM workflows.

HXSTCLI, which stands for Storage Controller CLI, this is used for controller VMs and is
available through SSH. Cisco UCS Manager enables server, fabric, and storage provisioning
, as well as device discovery, inventory, configuration, diagnostics, and much more. Lastly,
HyperFlex provides a built-in REST API access tool using the REST Explorer.
Thanks for watching.

Cisco HyperFlex installation is simple and fast because it is a bundle of software and
hardware that you can deploy using a wizard. The appliance model differentiates Cisco
HyperFlex from other HCI vendors.

The high-level steps for installing a standard vSphere-based Cisco HyperFlex are:

 Racking and stacking

o Install the servers and Cisco Fabric Interconnects and connect them with cables
—usually only two Twinax cables from each server. Cisco HX-Series Servers
 use unified connectivity for management and data, on just two high-speed and
low-latency links.
o Connect the Fabric Interconnects to the ToR switches and configure upstream
switch ports.
o Connect the Layer 1 port on the A Fabric Interconnect to the Layer 1 port on the
B Fabric Interconnect, and the Layer 2 port on the A Fabric Interconnect to the
Layer 2 port on the B Fabric Interconnect. Now both Fabric Interconnects will
continuously monitor the status of each other and synchronize configurations.
o Connect the management ports on Fabric Interconnects to the ToR switch.

 Preparing the infrastructure

o Configure a redundant pair of Fabric Interconnects for high availability (Cisco

UCS domain).
o Assign the uplink and server ports in the Cisco UCS Manager.
o Make sure that the following services are available to the HyperFlex (an
example for a Standard vSphere deployment): NTP for time, DNS for name
resolution, vCenter to host the installer VM and the HyperFlex itself, and Active
Directory for domain management. Only the NTP service is mandatory for the
standard vSphere deployment.

 Install your HyperFlex:

o Deploy HyperFlex using an intuitive wizard. The wizard is a VM that you

deploy as Open Virtual Appliance (OVA). You would download the OVA
from https://software.cisco.com.
o Alternatively, use the Cisco Intersight to perform the installation remotely
without an extra VM. In this case, the Cisco UCS domain will need to be
associated with your Intersight account. Association is performed through the
Cisco UCS Manager GUI.
o Perform post-installation tasks such as enabling high availability and DRS.

After the installation is complete, the HyperFlex is ready to run your VMs. You can
provision new VMs or move them (for example, vMotion) from your old infrastructure.

Running Out of Storage

After you have run your workload on Cisco HyperFlex for a while, the storage
resources may start running low. In HyperFlex, this can easily be corrected by
expanding the HyperFlex server group.
If the servers in the HyperFlex group have empty disk slots, you can expand the storage
by inserting more disks into the servers. The system will automatically add disks into
the storage pool.

Note
You still need to follow the storage guidelines for Cisco HyperFlex deployments. It is
important that you insert the same number of disks into all servers in a group to have a
symmetrical storage configuration. This enables the HyperFlex system to run
consistently and provides the ability to distribute data effectively.

If the disk slots on your HyperFlex group servers are already full, you can buy another
server, connect it to the Cisco Fabric Interconnects, then run the expansion wizard. The
new server must have the same number, type, and size of disks as the rest of the servers
in the group.
Running Out of Compute Power
A more common situation for most workloads is that your Cisco HyperFlex system
starts running out of compute resources (memory and CPU). HyperFlex can also
address this situation through its expandability.

To address the lack of compute resources in a Cisco HyperFlex group, you have two
options. One is adding an extra HyperFlex server that also has disks, which will expand
the compute and storage resources of a HyperFlex group.

Note
The HyperFlex servers that are equipped with both storage and compute are called
converged nodes. These are Cisco HX-Series Cisco UCS Servers and need to follow the
storage requirements of HyperFlex systems.

By expanding your HyperFlex deployment with converged nodes when you are running
out of compute resources, you will also need to purchase the storage drives that you
might not actually need, depending on the situation.

The alternative to expanding your HyperFlex with converged nodes is to expand it with
compute nodes. Compute nodes are Cisco UCS servers that you can add to your
HyperFlex system and only provide compute resources. These servers only need to run
the hypervisor. Booting off the SAN is also supported, so the compute nodes do not
actually need to have any disks at all.
Compute-only nodes can be both Cisco B-Series and C-Series (even non-HX-Series)
servers. It is completely possible to run a fully populated Cisco UCS blade chassis for
compute and use HyperFlex to provide storage.

Resiliency
The ability to tolerate failures on the Cisco HyperFlex system is determined by the
system resiliency status. Resiliency determines how many additional failures can the
system sustain before the storage operation is affected.

There are two different elements determining the operational status of a HyperFlex
storage:

 Operational status: Describes whether the storage is capable of providing the

storage services. If the system is online, the storage is available; if it is offline, the
storage is not accessible.
 Resiliency status: Describes the ability of the storage group to tolerate node
failures. It describes how well the storage group can handle disruptions. The
resiliency status that you want is healthy. An unhealthy resiliency status indicates
that something is wrong, (for example, a server has gone down with a fault).
Cisco HyperFlex is fully redundant from the ground up. This includes the storage
resiliency, network resiliency, and management resiliency. Every individual component
of a HyperFlex system can fail and the storage stays available, even on the smallest
HyperFlex deployments.

If a disk fails in one of the servers, the system will recover from that failure. First it will
be declared "unhealthy," then it will perform self-healing, and recover to "healthy." The
system will be functional or in "online" operational status throughout the whole failure
and self-healing process. Similarly, a HyperFlex group can sustain the failure of entire
servers. The system will self-heal as long as you have at least 4 servers in that group and
enough resources on the remaining servers.

When a disk fails, you want the rebuild to be fast and the performance to be predictable,
which is unlikely on many traditional storage systems. However, in a hyperconverged
web-scale environment, the loss of even an 8-TB SATA disk would result in minimum
rebuild time because of the distributed nature of the recovery process.

You deploy Cisco HyperFlex with two Cisco UCS Fabric Interconnects for resiliency
purposes. Cisco Fabric Interconnects are a mandatory feature of Cisco HyperFlex
standard and stretched deployments.

Cisco Fabric Interconnects provide two major benefits to a HyperFlex system:

 Orchestration of Cisco UCS deployment and management.

 And more importantly, high-speed and low latency network.

Plenty of Performance
One of the primary benefits of Cisco HyperFlex over its competitors is its consistent
performance.

The Cisco HyperFlex solution works by distributing its data across all servers in the
group, which allows the storage operations to be fully divorced from the server where
the VM that is making requests is running. A major benefit of this approach is that the
system is designed to use the network for all storage actions.

Because of its fully symmetrical design, HyperFlex data distribution is performed in

real time and is accelerated by the caching drive, which accelerates storage operations.
Even the HyperFlex systems with spinning drives use an SSD cache.

Even the fastest storage can benefit from caching, especially if the cache drive has
better performance. HyperFlex Non-Volatile Memory Express (NVMe) deployments
can be accelerated by Intel Optane drives that provide the best performance of all SSDs
on the market.

Note
Cisco HyperFlex exists in three flavors regarding performance: hybrid, all-flash, and
all-NVMe. All three types have cache and capacity tiers. Hybrid uses hard disk drives
(HDDs) for capacity. All-Flash uses NAND type SSDs for capacity. All-NVMe uses
NVMe-type SSDs for capacity. Hybrid HyperFlex is the most affordable configuration
and all-NVMe is the highest performing configuration.
 HyperFlex uses a distributed data system:

o All servers in the system work together to write and read information.
o System relies on high-throughput, low-latency network of Cisco UCS Fabric
Interconnects to guarantee that the network is not a bottleneck.

 Most other hyperconverged systems rely on data locality.

o For a given VM, reads and writes are limited to the local server that can lead to
hotspots.

Manage HyperFlex Using Intuitive Tools

The system performance is not the only metric that determines the value proposition of
a data center system. Many times, the operational costs of a system can make even the
most powerful solution non-viable.

Cisco HyperFlex offers several options for its management. The guiding principle of
HyperFlex management is intuitive design and the appliance approach to maintenance.
The idea is to eliminate management overhead by removing the need for employee
education through usage of intuitive self-explanatory management tools and global
access through systems such as Cisco Intersight to eliminate travel.

For a Standard ESXi-based HyperFlex group, the management options are:

 Cisco Intersight: Cisco Intersight is the most complete suite of management tools
for Cisco HyperFlex systems, it provides the means to manage both hardware and
software resources of a HyperFlex group with the ability to install and perform full-
stack upgrades of HyperFlex groups. Cisco Intersight is a SaaS system that is
provided by Cisco and uses secure connectivity to the on-premises systems to
manage and orchestrate those systems. It is a broader data center management and
orchestration system and can manage many Cisco and third-party data center
systems.
 VMware vCenter: The Cisco HyperFlex system has VMware vCenter-based
management. vCenter Server is a data center management server application
 developed to monitor virtualized environments. Cisco HyperFlex Data Platform
(HXDP) is also accessed from the vCenter server to perform storage tasks. vCenter
supports key shared storage features such as VMware vMotion, DRS, high
availability, and vSphere replication. VM-based workflows are done like on any
other ESXi-based HyperFlex. You perform datastore management and HyperFlex
monitoring through the HyperFlex vCenter plug-in. On a day-to-day management
basis, vCenter is the only tool that you need to manage HyperFlex. So, if you are
already familiar with vCenter and you are only now diving into HyperFlex, there
should be little learning curve for you.
 HyperFlex Connect: Alternative to VMware vCenter that is available with
HyperFlex without installation. It is a very intuitive HTML5 interface. However,
while all HyperFlex management features are there (datastore management,
HyperFlex monitoring, and so on) it is not a fully featured interface for VM
workflows. VM start, stop, and resume are supported, VM native clone and
snapshot are supported, deployment of VMs is not supported). HyperFlex Connect
might become a fully fledged alternative to vCenter in the future.
 Cisco HXDP storage command line interface (hxcli): CLI for CVMs is available
through Secure Shell (SSH). CLI for CVMs is called the storage controller CLI.
 Cisco UCS Manager: Supports the Cisco UCS server and Cisco HyperFlex
hyperconverged infrastructure portfolios. It enables server, fabric, and storage
provisioning, along with device discovery, inventory, configuration, diagnostics,
monitoring, fault detection, auditing, and statistics collection. You can extend the
benefits of Cisco UCS Manager globally across an enterprise to thousands of servers
in multiple domains with Cisco UCS Central Software. So, if you are already
familiar with Cisco UCS Manager and you are only now diving into HyperFlex,
there should be no learning curve for you.
 REST API Explorer: Cisco HyperFlex provides a robust Representational State
Transfer (REST)-based API for managing and monitoring HyperFlex groups.
HyperFlex provides an in-built REST API access tool using REST Explorer.
HyperFlex REST API Explorer is a useful tool for discovering the various REST
APIs available for HyperFlex. The REST API Explorer tool provides details about
the available API operations, provides an example of each operation, and allows you
to test the operations directly from within the web-based API explorer interface.

1.6 Introducing Hyperconvergence and Cisco HyperFlex

Cisco HyperFlex Deployment Options

Cisco HyperFlex has several distinct deployment types, providing additional specific
features, with the standard HyperFlex deployment being by far the most popular type.
Not all features are available on different deployment types and certain systems function
differently based on the deployment type. The standard deployment is, however, the
reference deployment type that is used to describe HyperFlex operation.

Note
If there is no specific information on deployment type stated, the information is
provided for the standard deployment.

HyperFlex is ordered based on the deployment type, as there are several hardware and
software specifics that are determined by the HyperFlex deployment type:

 Standard Deployment: Can either be purchased with ESXi hypervisor preinstalled

or Hyper-V hypervisor, which does not come preinstalled. This deployment features
two Cisco Fabric Interconnects as a mandatory part of the solution. You can re-use
existing Fabric Interconnects that you already have.
 Stretched Deployment: Only supports the ESXi hypervisor, which comes
preinstalled on the servers. A stretched deployment is deployed in two locations that
need to be interconnected by a fast network link to synchronously replicate data
between sites. If one of the sites becomes unavailable, the other site takes over. This
deployment requires an extra virtual machine to run on the infrastructure separate
from the deployment, to act as an arbiter for split-brain scenarios. Minimum
deployment size is 2 nodes on each site.
 Edge Deployment: Does not require Cisco Fabric Interconnects, which gives it
greater connectivity flexibility. Supports 1G and 10G connectivity depending on the
ordered networking. Also supports 2-node deployments that are not available on
other HyperFlex deployment types. Edge is the fastest growing deployment type and
has received significant updates in the latest HyperFlex versions.

Standard deployment is the only one that supports an alternative hypervisor to ESXi.
Hyper-V HyperFlex deployment can be interpreted as a completely separate deployment
type, since it has several specific features. For the sake of clarity, the standard
deployment is interpreted as an ESXi-based deployment.

Standard Stretched Edge

Group size 3-64 nodes 4-64 Nodes over 2 sites 2-4 Nodes

Hypervisor ESXi and Hyper-V ESXi ESXi

Fabric Interconnects 2 4 (2+2) 0 (Connects to ToR)

Deployment specifics Most common Site-wide resiliency Small deployments

Cisco HyperFlex started at the beginning of 2016 as an ESXi-based Standard
deployment; other deployments were introduced with later HyperFlex releases.

HyperFlex is not just one deployment type. However, Standard ESXi-based HyperFlex
is the most widely deployed by significant margin. Limitations listed in the table apply
to Cisco HXDP 4.5(1a).

Available
HyperFlex Type Description
Since

One group 3-64 servers, connected to one pair of Fabric

Standard ESXi HXDP v1.7
Interconnects.

One group of 2, 3, or 4 servers, connected directly to

HyperFlex Edge
HXDP v2.1 switches (no Fabric Interconnects). Suitable for branch
ESXi
offices and small companies.

One group 4-64 servers, at two different sites. Close to

Stretched ESXi HXDP v3.0
zero recovery time for VMs if there is a failure.

Standard Hyper-V HXDP v3.0 One group 3-32 servers. Hyper-V instead of ESXi.

It is important that you understand the differences between deployment types to best
design your HyperFlex.

Capabilities of Cisco HyperFlex change with each release. For example, the first regular
ESXi-based HyperFlex had a limitation of 8 servers per group. When you are designing
your solution, make sure you check the release notes of the target version to learn about
the more recent changes. The release notes for version 4.5(1a) are available
at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Dat
aPlatformSoftware/HyperFlex-Release-Notes/hx-release-4-5/Cisco-HXDataPlatform-
RN-4-5.html.

The release notes of a feature release (major version) are updated with information on
maintenance and patch releases. You do not need to look for the exact minor version
information to learn about that specific version.

Note
Although there are different HyperFlex deployments available, they are very similar in
installation and operation. Since the standard ESXi deployment of HyperFlex is the
most common, it can be used as a reference and then only the differences between
deployments can be explored. This is how the content of this course is structured.
1.7 Introducing Hyperconvergence and Cisco HyperFlex

Evolution of Cisco HyperFlex

Next, let's review the evolution of Cisco HyperFlex. The first version of HyperFlex was
version 1.7. Before that, the product was part of Springpath. And with that version, you
could run it on any hardware.

After introduced by Cisco as HyperFlex, it has been in rapid development. This includes
increasing the cluster size up to 64 nodes, interoperability features, and performance
enhancements. Each of the versions brings new features. So when you are designing a
new cluster, it's important to understand which version includes which feature. Cisco has
no plans for slowing down the rapid growth of Cisco HyperFlex.
Cisco HyperFlex is a well-tested and proven ecosystem of integrated products. Cisco has
added tools and processes within Cisco HyperFlex to support a full operations cycle,
which includes planning, using HX profiler, HX sizing, and HX benchmark tools, install and
identify using wizard-based installation tool, which you can use to expand the cluster,
manage using vCenter HX Connect, stcli, or through the REST API, protect using a natively
built-in mechanism to protect your data by replicating VMs to other clusters or using
third-party tools, and lastly, monitor using the exact same tools that you use to manage
your HyperFlex.

Cisco HyperFlex started as a regular ESXi-based cluster. However, as of September 2018,

several other flavors are available. This includes HyperFlex Edge ESXi, stretched ESXi, and
regular Hyper-V. They're very similar in installation and operation. So you should become
familiar with the regular ESXi-based cluster and then learn the differences for the other
cluster types as you go along.
Finally, in order for hyperconvergence to move beyond softwaredefined storage, it must
be able to orchestrate tasks, integrate with cloud-based systems, and perform as a self-
service platform.

Cisco HyperFlex is a multicloud platform. It integrates with several solutions, including

Cloud Center, which eliminates the complexity of having to adapt to different public cloud
models, Cisco WOM for HyperFlex, which helps optimize where your workloads run, so
that you get the needed perform and resources when they're necessary, Cisco Intersight,
which is a cloud-based interface that gives you instant access to all your clusters, and
lastly, Cisco Container Platform, which is a lightweight container management platform
for production-grade environments, powered by Kubernetes.

Thanks for watching.

The first Cisco HyperFlex version was 1.7. Before Version 1.7, the product was not
known as HyperFlex because it was part of the Springpath, Inc. company, and you
could run it on any hardware on the hardware compatibility list—not just Cisco. Since
Version 1.7, this HCI software is Cisco only.

Cisco HyperFlex software follows a time-based release model that delivers maintenance
and feature releases. This approach enables Cisco to introduce stable and feature-rich
software releases in a reliable and predictable cadence.

Cisco always lists the current recommended HyperFlex releases on the cisco.com
website. You can find the list of currently recommended releases at
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/release-guidelines-and-support-timeline/b-recommended-hx-data-
platform-sw-releases/m-recommended-releases.html.

The HyperFlex releases follow specific versioning:

 Feature release: Introduces new features and functionalities.

 Maintenance release: Also called long-term release, is a stable release with a
longer release lifecycle. Maintenance releases mainly provide stability
improvements.
 Patch release: Introduces fast fixes for both feature and maintenance releases.
Provides stability, security, and bug fixes.
Patch releases for a long-lived release are delivered approximately every 3 months.
Patch releases are the primary mechanism to deliver critical bug fixes to Cisco
HyperFlex Software releases.

A Cisco HyperFlex software long-lived release or maintenance release (X.Y(Za)) is

signified by the third digit (Z) in the release version number. If this digit is greater than
1, that Cisco HyperFlex software version is considered a long-lived release (for
example, the 2 in Cisco HyperFlex software release 3.5(2d).

Long-lived releases have a release cycle of 30 months and are meant for stable
deployments. This release cadence is used since HyperFlex version 3.5(1a), prior
releases do not follow the described logic for their versioning.

Each version brought these key features.

 Version 1.7:

o Basic ESXi deployment (8-node group)

o HyperFlex native snapshots and clones

 Version 1.8:

o New user interface for installation and expansion—consistent experience from

all versions beyond 1.8
o Compute nodes (8+8-node group)
o Connectivity to external storage
o Improvements in data handling
o Support for a nested vCenter

 Version 2.0:

o All-Flash HyperFlex
o Sharing Cisco UCS Fabric Interconnects between HyperFlex and non-
HyperFlex workloads

 Version 2.1:

o HyperFlex Edge deployment (HyperFlex for small deployments)

 o Larger group size supported

 Version 2.5:

o Full support for data at rest encryption

o NVMe SSDs for cache
o Native replication
o Support for vSphere Version 6.5
o Version 2.6:
o Support for Cisco UCS M5

 Version 3.0:

o Hyper-V based deployment

o Stretched ESXi-based deployment
o Support for Cisco Container Platform
o Logical availability zones
o Support for Intel Optane drives

 Version 3.5:

o Support for Cisco Container Platform

o Single workflow for upgrades of HXDP, ESXi, and Cisco UCS firmware
o Support for 4th generation of Cisco UCS fabric interconnects and VICs
o Support for ESXi and vCenter 6.7
o HyperFlex Hardware Acceleration Engine

 Version 4.0; up to 4.0(1a):

o 2-node and 4-node options for HyperFlex Edge

o All-NVMe HyperFlex
o Support for VMware Site Recovery Manager (SRM) Integration
o Support for Server 2019 with Hyper-V HyperFlex
o C480 ML as Compute-only node
o Container Storage Interface (CSI) plug-in for Cisco Container Platform
o VMware Site Recovery manager integration for HyperFlex native replication

 Version 4.5 up to 4.5(1a)

o M.2 RAID support for Hypervisor drive

o N:1 replication for Edge deployments
o HyperCheck script integration
o Native snapshot scheduling
o HX-CSI and iSCSI support
o 240 full-depth chassis support for HyperFlex Edge
o HTML5 plug in for vCenter

Cisco HyperFlex has been in rapid development since its unveiling at Cisco. Each
version brings features, so when you are designing a new deployment, it is important to
understand which version includes which feature. The rapid innovation with Cisco
HyperFlex platform will not stop with version 4.5.

Complete Operations Cycle

Cisco has also built many tools and processes within and around HyperFlex that enable
you to have a full operations cycle:

 Plan: To profile your existing data center and understand how big your HyperFlex
group is, use the HX Profiler. To size your HyperFlex, use HX Sizer. To evaluate
performance of your HyperFlex, use HX Benchmark. To verify your HyperFlex
configuration before installation, use the HX preinstall tool. All three tools are
available from http://hyperflexsizer.cloudapps.cisco.com.

 Install and Verify: Installation is easy and wizard based. Installation guides and
Cisco Validated Design guides are also available from http://www.cisco.com to
guide you through installations. Use the same tool that you use for installation to
expand your HyperFlex.

 Manage: You can manage your ESXi-based HyperFlex using vCenter, HyperFlex
Connect, CVM command line, or through REST API. You can monitor your Hyper-
V-based deployment using Hyper-V Manager, SCVMM, CVM command line,
HyperFlex Connect, or REST API. You can also manage all HyperFlex types
through Cisco Intersight.

 Protect: HyperFlex has a natively built mechanism to protect your data by

replicating VMs to another HyperFlex system. You can also apply third-party tools,
such as the Veeam Availability Suite to protect your data. HyperFlex can encrypt
data on a hardware level using self-encrypting disks. To protect your VMs using
snapshots, use regular vSphere snapshots or HyperFlex-based snapshots which are
done by HyperFlex at the storage level.

 Monitor: Monitor your HyperFlex system using the same tools that you use to
manage your HyperFlex.
Note
You must be signed in to access the tools. Create a user account if you do not have one.

Beyond Hyperconvergence
Applications are at the heart of the digital business transformation. The changing
landscape demands support for traditional, monolithic workloads and distributed
microservice architectures.

To help your business organization meet this challenge, you need an infrastructure that
is:

 Simplified

o You need a hyperconverged infrastructure that integrates networking, high-

performance computing, and a purpose-built, high-performance, low-latency
distributed file system.

 Multicloud

o You need to support deployment in your private cloud, and also in the public
cloud, with tools for automatic deployment regardless of location.

 Flexible

o Today’s application architectures are evolving. They need support for

monolithic applications as they transition to a more scalable, cloud-like model.
Big data and analytics applications need unprecedented scalability. Cloud-native
architectures need to run containerized applications across multiple clouds, with
automatic deployment and scaling.

Cisco HyperFlex is your multicloud platform that helps you address these challenges:

 Cisco HyperFlex with Cisco CloudCenter is simpler with self-service deployment.

It helps define how an application is to be structured and then deploys it in a Cisco
HyperFlex, a public cloud, or a multicloud environment. CloudCenter eliminates the
complexity of having to adapt to different public cloud models, making it easy to
deploy and move applications based on changing business, workload, or economic
factors.
 Cisco IWO for HyperFlex helps you optimize where your workloads run so that
they get the resources they need for best performance. It continuously analyzes
workload consumption, cost, and compliance constraints to give you the best
possible intelligence on where to deploy your workloads initially and where to move
them should conditions change.
 Cisco Intersight is a cloud-based interface that gives you instant access to all your
data center systems, including installation, upgrade, and management capabilities
for HyperFlex systems. Intersight uses a special secure connection to wherever data
center systems are located, so that you can ship your nodes to any facility and install
them remotely.
 Cisco Intersight Kubernetes Services: Intersight can act as the management,
installation, and maintenance platform for your Kubernetes instances. It allows
administrators to quickly deploy and then manage Kubernetes instances in any
location through Intersight secure connectivity. Because of the intuitive
management capabilities IKS removes the barrier to entry for new Kubernetes users.
 Cisco Intersight Services for Terraform: Intersight connectivity can be used as
the automation and orchestration conduit. Integration with Terraform Cloud allows
automated application of configurations on the infrastructure by simply changing
the code that describes that infrastructure. Terraform is always aware of the state of
the infrastructure so it gives administrators the ability to reverse changes and only
execute the necessary actions required to configure the infrastructure.

2.1 Describing Cisco UCS: Foundation of Cisco HyperFlex

Introduction
Cisco HyperFlex solution is based on a tried-and-tested Cisco Unified Computing
System (UCS) server infrastructure, which provides a very solid and reliable base.
Cisco UCS servers can be deployed in a managed configuration when connected to
Cisco Fabric Interconnects. Fabric Interconnects act as a management platform and a
very high-performance networking solution, providing connectivity to HyperFlex nodes
that create a distributed data platform across the network.

Cisco UCS is a Nextgen data center platform that unites computing, networking, storage
access, and virtualization resources into a cohesive system designed to reduce the total
cost of ownership (TCO) and increase business agility. The system integrates a low-
latency, lossless 10/25/40/100 Gigabit Ethernet unified network fabric with enterprise-
class, x86-architecture servers. The system is an integrated, scalable, multi-chassis
platform in which all resources participate in a unified management domain.
2.2 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco Server Deployment Models:

Standalone vs. Managed

Moving on, let's review Cisco Server Deployment Models and their benefits. Cisco UCS C-
Series servers can be deployed and managed using fabric interconnects or in a standalone
scenario. On the left side of the figure, a pair of fabric interconnects running Cisco UCS
Manager is used to manage the Cisco UCS C-Series servers.

Now, on the right side of the figure, Cisco UCS C-servers are deployed without centralized
management. Instead, the Cisco Integrated Management Controller, or Cisco IMCs, are
used. All Cisco UCS C-Series, these servers come pre-installed with a Cisco IMC, which
allows for management of the server through the IP network.
Cisco IMC capabilities on the C-Series servers includes connecting via a keyboard, video
mouse, or a KVM interface for management. The KVM connection eliminates the need to
physically connect to any of the hardware.

A standalone deployment reduces upfront costs because the Cisco fabric interconnects
are not required. But standalone deployments increase management overhead. So this
particular solution being standalone doesn't scale very well. But the good news is you're
always able to integrate a single deployment into a centrally managed infrastructure.

A managed solution typically includes two Cisco UCS fabric interconnects for high
availability. In this design here, all of the Cisco UCS servers connect to the fabric
interconnects, which then connect to the top-of-row switches.

Management of the servers is performed over the same cable as the data compression,
which reduces the cabling required. Local server traffic doesn't really need to pass
through an upstream top-of-row switch because it's handled by the fabric interconnects.
As a result, the data passing through the network while reading and writing under normal
circumstances does not cause any load on the infrastructure outside of Cisco UCS domain.
Managed deployments with two fabric interconnects provide not only high availability for
Cisco UCS Manager deployment, but it also creates two redundant fabrics for all of the
server communication.

The topology in this figure represents a very common Cisco UCS deployment. There are
four servers here, and they are connected and managed by a pair of fabric—or redundant
pair of fabric interconnects managed by UCS Manager.

Notice how each of the servers are connected to two of the upstream paths. And in
normal circumstances, one path is used for communication, and other the is used for
redundant failure.

And then-- let me get rid of some of these on here. Then the fabric interconnects are
interconnected by the L1 and L2 links. Now, these links between the fabric interconnects
do not forward any data. And they're only used for management information. This
ensures that any of the changes made to the primary interconnect will be propagated to
the other subordinate fabric interconnect.

The upstream switches, these can be connected using the virtual port channels, or what
we call VPCs. This not only provides communication, but it also provides a data path in
case of failure scenarios.

The interconnects, they could be connected without using regular port—without using
virtual port channel by using just regular port channels. Boy, this simplifies a network
configuration, but it is definitely going to decrease resilience.
Moving on, as many of you already know, Cisco UCS Manager introduced the concept of
configuration abstraction, which means that the server and its configuration are kept
separate. Service profiles are used to define the configuration, which then are applied to
the servers.

If you need to replace a server in a cluster for some reason, you just transfer the service
profile from one server to the next. It's a 1 to 1 ratio. Since the Cisco HyperFlex Installer
configures a server infrastructure for you through the centralized management of Cisco
UCS Manager, you don't have to do much more than provide the basic configuration of
the fabric interconnects.

You can still use all of the features of Cisco UCS Manager to manage the use of the actual
Cisco UCS servers. And that comes to the end of the video. Thanks. Thank you very much.

You can choose to deploy Cisco UCS C-Series servers in a managed configuration with
Fabric Interconnects or in a standalone configuration. Both options are equally valid,
and the server hardware is the same, but each deployment has specific features.

When you deploy Cisco UCS C-Series servers as standalone individual servers without
a centralized management, Cisco Integrated Management Controller (IMC) integrated
on each Cisco UCS server is used to control each individual server.

By connecting Cisco UCS servers to a Cisco UCS Fabric Interconnect, the Cisco UCS
Manager is used as the centralized management system for all the connected servers.
Standalone Cisco UCS Server Deployments
All Cisco UCS C-Series servers come preinstalled with Cisco IMC, which allows
management of the server through an IP network. Cisco IMC capabilities include
connecting to the server through a keyboard, video, mouse (KVM) interface, managing
disks, BIOS configuration, and network interface card (NIC) configuration. In
standalone deployments, Cisco IMC manages the server and connects to a management
network, while data is connected separately.

A KVM is a virtual connection over HTTP protocol to a Cisco UCS server, that
emulates a monitor, mouse, and keyboard. This connection makes remote management
of the server much easier, because you do not need to physically connect any hardware
to the server to manage it, nor is there a need for any external KVM solutions. The
KVM solution on Cisco UCS is available through the management network, based on
the ubiquitous TCP/IP and HTML5.

Standalone deployments have these features:

 Reduced upfront cost, but increased management overhead.

 Filly functional deployment. Needs Cisco Intersight or IMC Supervisor for
centralized management.
 You are always able to integrate a single deployment into a centrally managed
infrastructure.
 Profile-based management can be performed through Cisco Intersight.
Standalone deployments of servers do not require Cisco Fabric Interconnects to operate,
which reduces the capital expenses (CAPEX) associated with the deployment. It does
not mean that the long-term total cost of ownership (TCO) is better in standalone
deployment scenarios, because management overhead is much greater than in a
managed deployment scenario, especially in larger deployments.

Since the standalone deployments do not have a centralized management interface, each
individual server needs to be managed individually. This can cause significant overhead
when managing larger deployments, since the administrator needs to connect to each
server and perform the same configuration change, which would be done in one click in
managed deployments.

Since 2020, Cisco Intersight cloud management platform can provide centralized
management and profile-based configuration to Cisco UCS standalone deployments,
this greatly alleviates the management difficulties traditionally associated with
standalone server deployments. Intersight also provides full-stack software upgrades
that in the case of HyperFlex extends to firmware, ESXi hypervisor, and Cisco
HyperFlex Data Platform.

Managed Cisco UCS Server Deployments

A managed solution includes Cisco UCS Fabric Interconnect pair. This redundant
configuration prevents single failures from affecting the system. If one Fabric
Interconnect in a pair fails, the other takes over seamlessly and provides connectivity
and management to the Cisco UCS domain until the failure is resolved.

In managed deployments, Cisco UCS servers connect to the Cisco FIs instead of ToR
switches. Interconnects provide centralized management and very fast connectivity.

There is no need for a separate connection to the management network, when

connecting Cisco UCS to the Fabric Interconnects. Management of servers is performed
over the same cable as the data connection, which reduces cabling and port saturation
on the upstream device.
When you connect individual Cisco UCS C-Series servers to Fabric Interconnects, you
add them to a centralized management platform with network and management
redundancy. The servers themselves can communicate within the Cisco UCS Fabric
Interconnect domain and local traffic does not pass through an upstream device under
normal circumstances, like a top-of-rack (ToR) switch.

The fact that VLANs are terminated within the Cisco UCS domain, benefits the Cisco
HyperFlex solution greatly, as the data passing through the network while reading and
writing under normal circumstances does not cause load on the infrastructure outside the
Cisco UCS domain. Cisco Fabric Interconnects also provide an extremely responsive
high-bandwidth network, which is necessary for a seamless distribution of data across
the Cisco HyperFlex system.

Managed deployment with two Fabric Interconnects provides not only high availability
to Cisco UCS Manager deployment, but also creates two redundant fabrics for all server
communication.

The topology in the figure represents a very common Cisco UCS deployment. Four
servers are connected and managed by redundant Fabric Interconnects. These provide
connectivity, and management through the Cisco UCS Manager. The servers connect to
the Fabric Interconnects through a single cable, usually a Twinax copper cable. Cisco
offers Twinax cables up to 10 meters in length. Each server is connected to each Fabric
Interconnect in the topology. Thus, two fabrics (named Fabric A and Fabric B) are
created, which are two upstream paths.

Both upstream Fabric interconnects are valid communication paths but only one is the
primary for management purposes—the one that carries the management virtual IP
(VIP). In HyperFlex, most networks are configured with active/passive server link
configuration. The active/passive link is configured on the hypervisor level.

Whenever Fabric Interconnects are deployed in pairs (in high-availability mode), they
share management information through L1/L2 links. These links, which run between the
Fabric Interconnects, are not used for data communication even in failure scenarios.
They serve only for the coordination of Fabric Interconnects. This coordination enables
that changes in Cisco UCS Manager on one Fabric Interconnect are propagated
automatically to the other Fabric Interconnect.
The upstream switches in the example are connected by the virtual port channel (vPC)
peer link. This link not only provides communication, but also a unified valid data path
in failure scenarios. The vPC technology allows devices or links in vPC to fail without
affecting the communication. A combination of any switch and any Fabric Interconnect
in the example topology can fail without affecting the operation of the system.

The Cisco UCS Fabric Interconnects can also be connected to upstream switches using
regular port channels, which decrease resilience. The determining factor when deciding
the configuration are the capabilities of the upstream switches.

Managed Cisco UCS deployments can also be added to Cisco Intersight. This provides
similar capabilities as with standalone deployments. Creation and management of server
configuration profiles however cannot be managed through Intersight directly as of
HyperFlex version 4.5(1a) and still needs to be performed through the Cisco UCS
Manager.

Note
On-premises Cisco UCS Manager is accessible without a VPN connection through the
Cisco Intersight cross-launch feature. This enables management features that are not
supported through Intersight directly to still be accessible from one central Intersight
interface.

Cisco UCS-Managed Model Benefits

When you have a lot of servers, it becomes impractical to connect to each one of them
when you need to change something across the server infrastructure. One such case may
be a firmware upgrade, where you would have to connect to each server and upgrade
one-by-one manually. Therefore, the managed model provides several benefits over the
standalone server deployments, contributing greatly to the ease of management of the
Cisco UCS deployments, especially where high availability is required.

Cisco UCS Manager introduces the concept of configuration abstraction to the Cisco
UCS infrastructure. Meaning that a server and its configuration are kept separately. The
configuration can be applied to another server through these service profiles at will. All
the settings that are applicable for that server will be transferred, which is beneficial
when you need to replace a server in the group for some reason. You do not have to
properly configure everything, only transfer the profile from one server to the next.

Another benefit that Cisco UCS Manager brings to the table comes into play when you
need to scale out the infrastructure. Each service profile can be created from a template,
which allows you to access various generic server configurations. If you choose to
create an updating (in oppose to initial) template for server configurations, any change
to the updating template will be propagated to all the service profiles created from that
template retroactively. It can be practical when you have to change a setting due to a
firmware update or a discovered issue, without digging through the entire infrastructure
to make the changes.

Managed deployments provide these features:

 Centralized and simplified profile-based management of the entire Cisco UCS

domain.
 Individual configuration of each redundant fabric or global configuration.
 High availability of the management system and connectivity
 Easy management of large deployments, scalability, and oversight of the Cisco UCS
servers.
 Reduced operating overhead, lowering operating expenses (OPEX).

The Cisco HyperFlex installer configures the server infrastructure for you through the
centralized management of Cisco UCS Manager, so you only need to provide the basic
configuration of the Fabric Interconnects to start using the Cisco HyperFlex solution.
You can still use all the features of the Cisco UCS Manager to manage Cisco UCS
servers.

Although the HyperFlex on-premises installer automatically configures HyperFlex Edge

deployments, these deployments do not use profile-based system configuration. Cisco
Intersight addresses that and provides a profile-based deployment to Edge systems,
similar to Cisco UCS Manager. Networking; however, needs to be configured fully by
the user.

2.3 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco UCS M5 Overview

In this video, let's examine the Cisco UCS M5 and the virtual interface cards. The Cisco
UCS M5 server, these are the latest Cisco UCS C-Series servers platform, and they're
based on the Intel Xeon processors. They support up to 3 terabytes of RAM and 28 CPU
cores per CPU.
The M5 server is about 20% better in performance than its predecessor, the M4. Both the
Cisco UCS M4 and M5 servers can be managed by the Cisco fabric interconnects using
Cisco UCS Manager.

The Cisco UCS B-5108 chassis, as you can see in the diagram here, features 8 slots for the
individual blade servers. The B-Series servers are inserted into the blade chassis, which
provides connectivity and power to the blade.

The server options range from the smallest B-200 blade, which takes up one server slot or
a half of the slot in a chassis, to the B-480 blade, which takes up two server slots, which
runs across the entire chassis.

The server blades also allow mounting of the mezzanine virtual interface cards, or VICs,
which rely on the I/O Module, or IOM, connected to the blade chassis for connectivity.
Those are the cables that came out of the chassis and ran up to the fabric interconnects.

Also, the grid power mode splits the power supplies and provides N+1 redundancy. Lastly,
the B-Series radically simplifies traditional blade server deployment because they're
centrally managed using Cisco UCS Manager Software, which, once again, runs on the
fabric interconnects. This allows extreme compute density while still being very highly
and easily manageable.
Getting back to the C-Series, which are the rack-mounted servers, they're still a very
feasible server solution. The C-480 model supports deep learning with up to six Graphic
Processing Units, or GPUs.

It can be used in a standalone deployment option with Cisco IMCs or central management
using Cisco UCS Manager-- obviously, higher storage capabilities than the B-Series blades.
It's an excellent choice for storage or compute servers, machine learning, and so on.

The Cisco UCS S-Series stored servers offer an absolute maximum of raw storage capacity
in a form factor. They combine the idea of a modularity. They have the modularity from
the B-Series with two slots for server nodes, while the chassis contains the actual physical
hot-swappable drives. And there's also mezzanine slots as well.

The S-Series servers are the natural choice for a backup or a disaster recovery system.
They can easily integrate with the Cisco fabric interconnect domain. And once again, they
can be centrally managed by Cisco UCS Manager.
The Cisco UCS HX-Series, this is the hyperconverged data platform solution based on the
Cisco UCS C-Series servers. And it includes the UCS HX-220 and the UCS-240 rack server
models.

And once again, I mentioned this before. Even though they're very similar to Cisco UCS,
the HX-Series is the only Cisco UCS platform that supports the HyperFlex solution. And if
you'll remember, I said they have a different Product Identification, or a PID, and this
means it's verified when installing HyperFlex. And as you can see in the figure here, it has
a completely different visual bezel. It looks different from the regular C-Series servers.

Now let's look at the Cisco Virtual Interface Cards, or VICs, and their benefits. The Cisco
VICs allow you to create up to 256 PCIe-compliant interfaces that are then presented to
the hypervisor as individual network interface cards. This allows for great flexibility when
configuring software-defined networking components.

The C-Series VICs resemble regular NICs, and they use a PCIe slot to connect to the
system. Cisco VICs that you assign—they enable you to assign hardware directly to
individual VMs. Each virtual interface can be either a network interface card or a host bus
adapter while sharing the same physical VIC.
Finally, this figure shows a list of numerous features provided by the Cisco VIC. Cisco VICs
enable you to assign hardware directly to individual virtual machines. As far as the
hypervisor is concerned, the virtual interfaces provided to VIC are the actual physical
devices.

Cisco VICs also support Fibre Channel and Ethernet processing to facilitate high
performance in the Cisco unified fabric structure. Each of the interfaces can be a—
convert and be a network adapter interface card for Ethernet or a host bus adapter for
a Fibre Channel using the same VIC.

These are called our CA ports, or our Converged Access ports. Or actually, they're not
converged access. There's an, actually, term-- oh, unified ports, that's it. These are what's
called unified ports because they help you converge the network itself. Because a port
can be either a NIC, or a VIC, or a host bus adapter. And that concludes our video. Thanks
for watching.

The M designation in Cisco UCS names designates the server generation. Cisco UCS
M5 servers are the latest Cisco UCS server platform, based on Intel Xeon processors
providing top of the line performance and reliability.

The first Cisco UCS generation that included HyperFlex servers was M4 and there are
M4 HyperFlex systems still running in many data centers today. However, when
purchasing new systems only Cisco UCS M5 servers are orderable as of HyperFlex
version 4.5(1a), with the M6 generation release being almost ready for release.

Both Cisco UCS M4 and M5 Series servers are manageable by Fabric Interconnects,
and Cisco UCS Manager of the same firmware version.

Note
Cisco UCS M4 servers are End of Life and End of Sale.
 M5 servers are about 20 percent better in performance than M4
servers when using a similar hardware configuration.

M5 Series is the first iteration of Cisco UCS to feature

integrated dual M.2 slots on the motherboard, which
replaces the much less effective Secure Digital (SD) cards
as the primary boot media device. SD cards are still
available as an option on Cisco servers.

Some M5 servers, such as the Cisco UCS C240 M5 server,

feature rear-mounted drives next to the power supplies, which
are not featured on a previous generation of the same model.

M4 Generation Features M5 Generation Features

Intel Xeon E5 on C220

Intel Xeon Scalable CPUs (generations 1 and 2) with up
CPU and C240; Intel Xeon E7
to 28 cores/CPU
on C460

Up to 1.5 TB of RAM per CPU with M-type CPUs Up

768 GB per CPU on
to 3 TB of RAM per CPU with L-type CPUs Optional
RAM C220 and 240; 1.5TB on
use of Intel Optane memory modules that can bring
C460
total memory up to 4.5 TB

2x SD FlexFlash + SAS,
Storage Integrated dual M.2 slots + SAS, NVMe, Intel Optane
NVMe storage

Integrated Integrated dual 1G Base-

Integrated dual 10G Base-T Ethernet
mLOM T Ethernet

Up to 2 GPUs on 460
Up to 6 GPUs on C480 M5 and C240 M5 Up to 2 GPUs
GPU M4 model, 1 GPU on
on C220 M5
C240 M4

Additional — Cisco Intersight support through a native device

 M4 Generation Features M5 Generation Features

Features connector

Cisco UCS also supports Intel Optane storage when using 2nd generation Intel Xeon
Scalable CPUs. Intel Optane is a very fast type of persistent storage, similar to SSDs.
Optane can be connected to either the standard NVMe interface or in the memory slots.
The NVMe option looks like a standard SSD, Data Center Persistent Memory Mode
(DCPMM) Optane modules look like RAM sticks and are installed in the memory slots.

Note
Find Cisco UCS M5 memory guidelines in the Cisco UCS C220/C240/B200 M5
Memory Guide specsheet:
https://www.cisco.com/c/dam/en/us/products/collateral/servers-
unified-computing/ucs-c-series-rack-servers/memory-guide-c220-
c240-b200-m5.pdf.

2.4 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco UCS M5 Server Types

The Cisco UCS lineup features storage, blade, and rack server options. Rack servers are
more flexible and support more storage, while blades have a higher compute density and
are adapted to the virtual environments of modern data centers. Storage servers are
specifically designed for massive raw storage capacity. The HX-Series is a specific rack
server version designed for deployment of the Cisco HyperFlex hyperconverged data
platform.

Standalone C-Series Blade B-Series HyperFlex Series

Standalone or managed
Managed through Cisco UCS Manager Integrated HyperFlex storage platform
deployment
The greatest compute density Based on the Cisco UCS C-Series server line
Wide array of workloads
Usually relies on external storage No external storage necessary
The most flexible platform
 At-A-Glance Comparison of Server Characteristics
S3260 With M5
B480 M5 (Chassis Required) C240 M5 (HyperFlex) C480 M5
Nodes

4RU rack with up

Form Factor Full-width blade 2 rack unit (RU) rack 4RU rack
to 2 nodes

Max CPUs 4 2 4 2 per node

Xeon scalable CPUs Xeon scalable

Xeon scalable CPUs (1st and Xeon scalable CPUs
CPU Type (1st and 2nd CPUs (1st
2nd generation) (1st generation)
generation) generation)

Max
28 28 28 44 per node
Cores/CPU

RAM DIMMs 48 24 48 7 per CPU

Max RAM 12 TB 3 TB 12 TB 896 GB per CPU

32 SAS, SATA,
Storage 26 SAS, SATA + 4 54 top 3.5'' + 2
4 SAS, SSD, SATA, NVMe NVMe with aux.
Drive Types NVMe rear 2.5''drives
module

Dual RAID
12-Gbps modular RAID 12-Gbps modular controller with 4-
RAID Built-in 0, 1
controller RAID controller GB RAID cache per
chip

Integrated Dual 10G BASE-T Dual 10G BASE-T

No Onboard Cisco VIC
I/O Ethernet ports Ethernet ports

Mezzanine
5 No No 1 per node
Slots
Note
Some options, such as storage, have different available configurations. Only one is
featured in the table for brevity.

Cisco UCS B-Series Blade Servers

Cisco UCS B-Series feature a chassis and the actual servers, which each feature their
own specifications. The Cisco UCS-B 5108 chassis features 8 slots for individual
servers called blades. These servers are purchased and configured separately.

Cisco UCS B-Series servers are modules that get inserted into a blade chassis, providing
connectivity and supplying the power to the system. The Cisco UCS B-Series servers
radically simplify the traditional blade server deployment as they are centrally managed
through the Cisco UCS Manager. This central management allows the servers to be
highly manageable while reaching extreme compute density.

The servers range from the smallest B200 server (which takes up 1 slot space in the
chassis) up to the B480 blade (which takes up 2 server slots in the chassis).

In M5, the form factor of the blade servers is either half width or full width. Meaning
that a blade takes up half of a row in the chassis (single-server slot) or a full row in a
chassis (two-server slots).

The Cisco B-Series servers are surprisingly powerful for their size and support up to 4
CPUs and up to 12 TB of RAM, much like the rack servers, but Cisco B-Series feature a
much more compact form factor while having less expansion and deployment options.

Cisco UCS B-Series server lineup includes these features:

 Blade server chassis and blade servers, which must be inserted into the chassis.
 Advanced power redundancy options like GRID mode.
 Superb density in a compact yet standardized form factor.
 Excellent choice when very dense compute is necessary, and storage is provided by
SAN or HCI.

The GRID power mode available on Cisco UCS B-Series configurations splits the
power supplies available in the blade chassis into two redundant groups. One group
power the servers while the other provides better than N+1 redundancy, since it
provides two PSU standby. If there is a failure of one GRID group, the other group fails
over.
Cisco UCS B-Series servers support cards, which are attached internally to the server
motherboard using a special slot called mezzanine. This allows mounting of modular
devices like Cisco virtual interface cards (VICs) parallel to the motherboard, since the
height of a blade server is limited.

Mezzanine VIC cards do not have external ports like C-Series VICs that you insert into
regular PCI Express (PCIe) slots but rely on the I/O module (IOM) connected to the
blade chassis for connectivity.

Cisco UCS C-Series Rack Servers

Cisco C-Series server platform is a very flexible server solution. Where the blades shine
in density of compute performance, the Cisco UCS C-Series offers a broad array of use
cases, from deep learning with up to 6 graphics processing units (GPUs) on the
C480ML model, to very small deployments of Cisco C220 1-rack unit servers, and up
to 1.1 PB of all-flash storage in a Cisco HyperFlex data platform group. All these
features are inherent to this extremely flexible platform.

The Cisco UCS C-Series platform supports large form factor and small form factor
spinning drives. The chassis for each option is different and the drive form factor must
be taken into consideration on purchase.

Cisco UCS C-Series server lineup includes these features:

 Great flexibility of use cases and expandability like optional GPU acceleration.
 Standalone deployment option with Cisco IMC or central management, providing
additional deployment flexibility.
 Higher storage capabilities than B-Series, but lower than Cisco UCS S-Series.
 Excellent choice for storage or compute servers, machine learning, virtual, or bare-
metal, and so on.

Cisco UCS S-Series Storage Servers

Cisco UCS S-Series storage servers offer the absolute maximum raw storage capacity
for their form factor. They combine the idea of modularity from the B-Series, as the
solution features two slots for server nodes, while the chassis contains the actual
physical drives. Mezzanine slots are also present on the S-Series nodes. The product
line features the S3260 chassis, which supports M4 and M5 nodes.

Cisco UCS S-Series server lineup includes these features:

 Very high capacity with up to 54 top loading SAS 7200-RPM 12-TB drives
 Two modular compute nodes provide high availability to the solution.
 Hot swappable and easily accessible drives for easier maintenance.
 Great for disaster recovery and backup solutions or when extreme capacity is
needed, like UHD video repository.

Cisco UCS S-Series storage servers are the natural choice for a backup or a disaster
recovery solution. They can easily integrate into a Cisco Fabric Interconnect domain
and are centrally managed by Cisco UCS Manager like all other Cisco UCS servers.

Cisco UCS HX-Series Servers

The Cisco UCS HX-Series is a hyperconverged data platform solution based on Cisco
UCS C-Series servers. The HX-Series share hardware options with the Cisco UCS C-
Series rack servers with Cisco HyperFlex specific caveats.

The HX-Series includes a Cisco UCS HX220c and a Cisco UCS HX240c rack server
model. The HX rack servers are visually distinct from the C-Series by their front bezel.
Cisco HyperFlex data platform solution can only be installed on Cisco HX-Series rack
servers.

Cisco UCS HX-Series server lineup includes these features:

 Centralized management or Intersight managed deployment of Cisco HX Edge.

 Tried-and-tested hardware built on Cisco C-Series solution.
 Easy to deploy, scale, and maintain. Generally, very good, but superb in cloud and
ease of use.
 Cisco HyperFlex hyperconverged data platform solution.

Even though very similar to the Cisco UCS C-Series, the HX Series is the only Cisco
UCS platform that supports the Cisco HXDP HCI solution. The HyperFlex servers have
a different product identifier (PID), which is verified when installing the HyperFlex
Data platform software.

2.5 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco UCS GPU Acceleration

GPUs are commonly used in the data center to accelerate graphical workloads such as
VDI workloads. GPUs are supported on the entire Cisco UCS lineup including Cisco
UCS HX-Series HyperFlex servers and B-Series blade servers.

GPUs can be ordered directly from Cisco and will come preinstalled in the ordered
servers. The required software and support licenses can also be ordered directly from
Cisco. This makes deploying and supporting GPU accelerated workloads very easy.

GPUs are also very useful in artificial intelligence (AI) and Machine Learning (ML)
workloads, especially for Deep Learning (DL). Deep Learning is the process of data
analysis that produces an instruction set for AI decision making. AI/ML/DL workloads
benefit from many small simpler operations that CPUs can handle much better than
GPUs.

NVIDIA offers an extensive GPU lineup that can assist with any GPU optimized
workload.

T4 RTX 8000 / RTX 6000 V100

CUDA Cores 2560 4608 5120

Tensor Cores 320 576 640

Memory 24G 48G / 24G (RTX 6000) 32G

 T4 RTX 8000 / RTX 6000 V100

Use Case Any VDI Workstation AI/ML

CUDA cores are the more traditional graphical acceleration cores, while Tensor cores
are powerful Nextgen cores, which are especially optimized for AI/ML workloads.

The graphical memory is important depending on the size of the analyzed dataset.
Dataset in virtual desktop environments best translates in screen resolution, while in
simulations it translates in the size of the simulation, such as the number of particles
involved in the simulation.

There are mezzanine GPU acceleration cards offered by NVIDIA, such as P6. These
GPU cards are designed for use in Cisco UCS B-Series blade servers.

2.6 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco VICs and Their Benefits

In the heavily virtualized environments of modern data center infrastructures, hardware
no longer represents the actual topology of a software-defined data center, which is also
true for network connectivity. While physical cabling still constructs the physical
topology, how individual hardware components are used can be much more flexible.
When several virtual machines exist on the same server and in their own network
topology, they are still limited by physical network interfaces for communication.

Cisco VICs allow you to create up to 256 PCIe compliant interfaces that are presented
to the hypervisor as individual network interface cards. Allowing for great flexibility
when configuring the software-defined network components while maintaining a simple
physical topology.

Cisco C-Series VICs resemble regular NICs and use a PCIe slot to connect to the
system, while Cisco B-Series VICs are connected internal mezzanine slots and only
process traffic. Mezzanine VIC cards rely on the Cisco B-Series Chassis to provide
physical connectivity through the I/O Modules (IOM).
VIC interface cards provide these capabilities:

 Traffic processing for internal and external communication supporting simultaneous

host bus adapter (HBA) and NIC operation on the same physical hardware.
 Multiple-interface-card virtualization without any additional driver requirements
with integration of the virtualized cards into the Fabric Interconnect infrastructure.
 Management through Cisco IMC or Cisco UCS Manager with dynamic
configuration of virtual interface cards based on the server profile (MAC/WWN).

Cisco VICs enable you to assign hardware directly to individual VMs. Each virtual
interface can either be a network interface card or a host bus adapter while sharing the
same physical VIC. The VIC can provide both functionalities at the same time. As far as
the hypervisor is concerned, the virtual interfaces provided by a VIC are actual physical
devices.

The Cisco UCS VIC 1400 series is based on 4th generation Cisco ASIC technology and
is well suited for next-generation networks requiring 10/25 Gigabit Ethernet for C-
Series and S-Series servers and 10/40 Gigabit Ethernet connectivity for B-Series
servers. Additional features support low latency kernel bypass for performance
optimization. The Cisco UCS VIC 1400 series provides high network performance and
low latency for the most demanding applications such as Big data, High Performance
Computing (HPC), large-scale virtual machine deployments, and high-bandwidth
storage targets and archives.
Cisco VICs function as individual interfaces adding these features:

 If a VIC is physically connected to an upstream device that supports VM-FEX, the

virtual interfaces show up as if they were physical.
 The virtual interfaces created by a VIC can accommodate hypervisor
communication, like VMkernel interfaces on vSphere.
 The virtual interfaces created by a VIC can be passed through to individual VMs,
further emulating direct physical connectivity.
 The VIC supports many advanced features like FCoE, VXLAN, VMDirectPath,
NPIV, and FC-NVMe.

Traffic going to and from the server has to be processed, which causes load on the
system. In desktop computers, the load is usually shouldered by the CPU, since the
amounts of traffic and the consistency of the load are significantly lower than in a server
environment. Since processing on a server would cause an unnecessary load on a server,
the traffic processing is offloaded to the network interface, usually named HBA. HBAs
take over the traffic processing and leave the CPU to perform other tasks.

HBAs feature a highly optimized ASIC, which is optimized to process a very specific
workload well. Traditionally, HBAs supported a specific protocol, like Fibre Channel,
but no other protocols. Cisco VICs support Fibre Channel and Ethernet processing to
facilitate high performance in a Cisco Unified Fabric infrastructure.

In a virtual environment, where all the data generated is virtual, the situation is no
different, since all the traffic must be processed to be compliant with the networking
standards. So, an HBA card is necessary even in entirely virtual environments.
2.7 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco UCS Fabric

Interconnects

Now let's take a closer look at the Cisco UCS Fabric Interconnects with the UCS Manager
software. The Cisco Fabric Interconnects are the management and connectivity platform
for the Cisco UCS data center solution. Cisco UCS Manager software runs on the Fabric
Interconnects and is used through a browser interface to manage the servers. If you
remember, the Fabric Interconnects can be deployed in a single deployment. However,
typically they're deployed in pairs connecting with the L1, L2 links, and that provides high
availability and redundant fabrics.
The Fabric Interconnects are the backbone of the Cisco managed deployment. This figure
here lists the features for the second generation of Cisco UCS Fabric Interconnects, which
support 48 or 96 ports of 10 gigabit Ethernet or Fibre Channel. You remember the unified
ports? With these unified ports feature, you can configure the protocol of the port to
operate any other mode. The Cisco UCS Fabric Interconnects require licenses to be
purchased for the number of ports that you're actually going to use. So basically, you just
pay as you grow with this solution.

This figure lists the features of the third generation of Cisco UCS Fabric Interconnects with
different port configurations, including a breakout feature that allows 40-gig QSF ports to
be connected to four 10-gig ports. So it will break it out. Even though the different
generations of Fabric Interconnects perform similar tasks in the data center, their port
licenses are not transferable between the different generations.
Lastly, this figure lists features of the fourth generation of Cisco UCS Fabric Interconnects.
Notice the Cisco UCS 6454 supports 54 ports and has 3.4 times higher throughput and
two times lower latency than the 6200 series. Be sure to check the latest release notes for
all of the compatibility guidelines.

The Cisco UCS Manager software, as I mentioned before, is a central management and
monitoring point for the Cisco UCS servers. This system allows for large-scale
deployments to be managed through a common interface. This includes managing the
interfaces, monitoring, logging, and GUI management of the actual UCS devices. UCS
Manager is accessed through a web interface by connecting to the IP address of the
Fabric Interconnect and is based on HTML5.

Basically, you have a virtual IP address, and the active Fabric Interconnect is the one, or
the primary Fabric Interconnect, is the one that's actually going to connect. And the
subordinate just kind of keeps track of what's going on. The functionality extends from
managing the Fabric Interconnect itself to then managing the entire Cisco UCS domain. It
automatically discovers servers connected to the Fabric Interconnects and then provisions
them depending on the policy-based settings.
Cisco UCS Manager uses a combination of policies and pools to derive values. Policies are
the settings that are applied when certain conditions are met, such as server type or
firmware version, for example. The pools, on the other hand, contain a range of values
that are assigned to certain Cisco UCS hardware elements. Pools can include what's called
abstract identities, such as management IP addresses, MAC addresses, UUIDs, and so on.
But pools can also include actual server blades.

Now typically, pools and policies are assigned to service profile templates, which kind of
define the standard for a common server. And then from the template, that's where the
individual service profiles are then created. The service profile is then assigned to a server
. And once again, it's a one to one ratio. A service profile can only be assigned to one
server.

Using Cisco UCS Manager, you can manage Fabric Interconnects—you can manage the
Fabric Interconnects on which it is running and then assign port roles for each port. For
example, Ethernet ports are unconfigured by default. So when you configure it as a server
port, this is where the Cisco UCS hosts are connected to. And the minute you say change
this port mode to server port, then each of the hosts are discovered automatically when
the server port is configured.
Once the host is detected, it must be acknowledged to be a usable. So acknowledgment
can either be done manually, or you can set it up to do automatically. The
acknowledgment prepares the server for use and for the service profile assignment. You
can also change the port operation mode to Fibre Channel if you're connecting to a native
Fibre Channel storage area network.

This next figure illustrates the creation of a pool, which, in this case, is a block of IP
addresses. As you remember, defining pools can be used to create a set of abstract
identities, which when they're used with creating the service profile, that's what assigns
the IP address. The service profiles will acquire the next unused value from the pool. I
mean, once it's used by another entity or another node, it can't be used again.

So the service profile goes out and says, give me the next IP address in a pool, or give me
the next MAC address or the UUID or whatever the case may be. And it grabs it when the
service profile is being associated to the actual server. It will also release the value. If you
disassociate the service profile from the server, then it returns those abstract identities to
the pool, saying they're not used anymore.
And once again, remember, I said that individual pools and policies can be configured into
the service profile template. And then the template's used to create service profiles,
create multiple service profiles with common attributes. Maybe I've got my AAA server,
so I build a service profile that says, this is all the attributes you need for a AAA server.
Boy, does this save time when you're rolling out many servers.

There are two types of service profile templates available. The initial templates
generating individual service profiles. If you want to make change, then you just go out
and change it to the individual service profiles. However, updating templates are linked to
the service profiles which they create. This means if you change a template, all of the
service profiles associated with that template will also be updated. So if I'm going to go
out there and make a change to the AAA servers and I want to make sure I change them
all, I don't have to go out to 20 or 30 AAA servers. I just make the change on the updating
template, and it automatically gets pushed out to all the service profiles.

If you decide you don't want it to be associated with that service profile anymore, no
problem. You can disassociate the service profile from the templates if you need to.
Updating templates are very useful when changing settings across several devices. An
another example would be like if you had a BIOS setting that had to be changed on the
entire cluster. The updating template, boy is that going to save you time because you only
need to change it in one place.

Finally, once a service profile template has been created, then you can generate multiple
service profiles, which can be associated with individual servers, which is great because
there's like eight or so steps to create a service profile, well, there's eight or more steps
to create the service profile template. So instead of me going through eight steps every
single time when I create a service profile, I go to the template, say generate x amount of
service profile, bam, it's done.

Associating the service profile with the actual server blades, now that's disruptive and will
cause the server to reboot. So keep this in mind for a production environment. And one m
ore cool feature is that ports can also be automatically configured as service ports when t
he hosts are connected to them. With this feature, it's simply plug and play. You plug in a
nother server into the Fabric Interconnect, and it's ready to go. So that's the end of this vi
deo. Thanks for watching.
Cisco Fabric Interconnects are the management and connectivity platform of the Cisco
UCS data center solution. They feature Cisco UCS Manager management software,
which runs on the Cisco Fabric Interconnect and manages the servers. Cisco Fabric
Interconnects are deployed in highly available pairs, where they provide Cisco UCS
Manager and network connectivity redundancy by providing two redundant fabrics.

Cisco Fabric Interconnects are the backbone of the Cisco-managed deployment.

Common features of the platform include:

 Centralized access to server configuration and server infrastructure through Cisco

UCS Manager platform.
 One connection for management and data traffic, reducing cabling.
 Template-based configuration of servers, easing server replacements and scaling.
 Centralized firmware management for the entire infrastructure and servers.
 Role-based provisioning and accounting.
 API integration support.

Individual generations of Cisco Fabric interconnects (FIs) have different capabilities.

The first and second-generation Fabric Interconnects are End of Life and End of Sale,
but the second generation is still supported for HyperFlex deployments.

Second generation of Cisco UCS Fabric Interconnects have these features:

 48 or 96 ports throughput of 10-Gbps Ethernet or 8/4/2/1 Fibre Channel per port. All
Ethernet ports are FCoE-capable.
 Unified Port Feature, where you can configure the protocol of a port to operate in
Fibre Channel or Ethernet mode.
 Expandable design for pay-as-you-grow deployments including per port licensing.

Licensing of Cisco UCS Fabric Interconnects requires licenses to be purchased for the
number of the ports in use. This requirement prevents exceeding licensing fees to be
paid on purchase if not all ports of the device are needed. If you purchase 32 licenses,
you can use any 32 ports on the device at the same time and the ports that use up
licenses are not fixed.

Cisco UCS 6248UP offers bandwidth throughput of 960 Gbps and 6496UP offers
throughput of 1920 Gbps.

Note
The Cisco UCS 6200 Series supports a wide variety of 10-Gigabit Ethernet connectivity
options using Cisco 10GBASE SFP+ modules. In addition, all fixed and modular
expansion ports on the Cisco UCS 6200 Series support 1-Gigabit Ethernet connectivity
options using 1GBASE SFP modules. Alternatively, 2/4/8-Gbps Fibre Channel
enhanced small form-factor pluggable (SFP+) and 1/2/4-Gbps Fibre Channel small
form-factor pluggable (SFP) interfaces are supported on all ports.
The 6248 comes with 12 prelicensed ports (you can choose which ports are
prelicensed). The 6296 comes with 18 prelicensed (you can choose which ports are
prelicensed). The Gigabit Expansion Modules (GEM) come with 8 ports prelicensed.
Licenses for additional ports are available.

Note
Cisco UCS 6200 Series Fabric Interconnects have been announced end-of-life, and will
not be sold after 31 May 2019, but will be supported until 31 May 2024.

Third generation of Cisco UCS Fabric Interconnects offer these features:

 Cisco UCS 6332: 32 QSFP Ethernet ports

 Cisco UCS 6332-16UP: 24 QSFP Ethernet ports and 16x10G ports with 16-G Fibre
Channel support
 Breakout feature allows 40 G QSFP ports to connect to 4x10G ports
 Per port licensing, but no expansion slots as with the second-generation Fabric
Interconnects

Note
Even though different generations of Fabric Interconnects perform similar tasks in a
data center, their port licenses are not transferrable between different generations.

Only the ports 1/17 to 1/34 are capable of breakout configuration on FI 6332-16UP,
other ports are fixed.

Cisco UCS 6300 offers bandwidth throughput of 2.56 Tbps.

The base unit comes with several prelicensed ports. The 6332 comes with 8 prelicensed
ports (you can choose which ports are prelicensed). The 6332-16UP comes with 4
prelicensed QSFP+ ports and 8 prelicensed Unified Ports (you can choose which ports
are prelicensed). Licenses for additional ports are available.

Fourth generation of Cisco UCS Fabric Interconnects offer these features:

 Cisco UCS 6454 has 3.4x higher throughput and feature 54 Ethernet ports that are
FCoE-capable
o 36x capable of 10/25G Ethernet, 4x capable of 1/10/25G Ethernet. 8x
capable of 10/25G Ethernet or 8/16/32G Fiber Channel, and 6x capable of
40/100G Ethernet.

 Cisco UCS 64108

o 72x capable of 10/25G Ethernet, 8x capable of 1/10/25G Ethernet. 16x
capable of 10/25G Ethernet or 8/16/32G Fiber Channel, and 12x capable of
40/100G Ethernet.

Note
Please check the latest release notes for the latest compatibility guidelines.

The 6454 comes with 18 pre-licensed 10/25 Gbps and 2 pre-licensed 40/100 Gbps ports.
The 64108 comes with 36 pre-licensed 10/25 Gbps and 4 pre-licensed 40/100 Gbps
ports. You choose which ports are licensed when you start using them. Licenses for
additional ports are available for both FI devices.

2.8 Describing Cisco UCS: Foundation of Cisco HyperFlex

Cisco UCS Manager

Cisco UCS Manager is a central management and monitoring point for Cisco UCS
servers. The system allows for large-scale deployments to be managed through a
common interface. It supports management through management interfaces, monitoring,
logging, and GUI management of Cisco UCS devices.

Cisco UCS Manager is usually accessed through a web interface by connecting to the IP
address of the Fabric Interconnect and is based on HTML5, although Secure Shell
Protocol (SSH) access is also supported.

The functionality of Cisco UCS Manager extends from managing the Fabric
Interconnect itself to managing the entire Cisco UCS domain. The Cisco UCS Manager
discovers servers connected to the Fabric Interconnects and provisions them depending
on the policy-based settings.
Device discovery is a process that is initiated when a server is detected in Cisco UCS
Manager. Discovery gathers the information about the device, such as the full device
component inventory. It also configures the device to be compliant with the default
policies configured in the Cisco UCS Manager.

Cisco UCS Manager centralizes Cisco UCS management, including:

 Fabric Interconnect management port assignment, uplink pinning, and firmware.

 Server chassis configurations such as power and fans.
 Server hardware configurations such as disks, BIOS, networking, and KVM.
 Cisco UCS Manager is automatically updated on both Fabric Interconnects in high-
availability deployments.

Since the Cisco UCS Manager manages several servers through assignment of
configurations to servers, it needs a set of rules so that the correct values are derived
from the rules when a service profile is assigned. Cisco UCS Manager uses a
combination of policies and pools to derive values. Policies are settings that are applied
when certain conditions are met, such as server type or firmware version. Pools on the
other hand contain ranges of values that Cisco UCS Manager assigns to certain Cisco
UCS hardware elements, including Universal Unique Identifier (UUID), management
IP addresses, MAC addresses, and Fibre Channel world wide name (WWN) values.

You assign pools and policies to service profile templates from which individual service
profiles are created.

Profiles define a set of configuration values, which get associated to servers. This
association can be moved or expanded on hardware change.

As the entire configuration exists in the Cisco UCS Manager, only the backup of the
Cisco UCS Manager configuration is required to back up the configuration. There are
several backup options available, including an .XML configuration export or binary
export of the entire Cisco UCS Manager installation, which can be redeployed even if
the Cisco UCS Manager installation itself becomes unusable.

Cisco UCS Manager provides a profile-based configuration of servers using these steps:

 Server is physically connected and detected on a host port by the Fabric

Interconnect.
 You define pools and policies, which define configuration options.
 You add pools and policies into a profile template.
 You create a service profile from a template.
 You assign the profile created from a template to a single server.
 If a server fails, the same profile can be assigned to its replacement, drastically
reducing deployment time.

Cisco UCS Manager manages the Fabric Interconnects on which it is running. You can
assign port roles for each port on Fabric Interconnects.

Server ports have Cisco UCS hosts connected to them and each Cisco UCS host is
discovered automatically on a server port. You can assign appliance ports to connect to
NAS or change the port operating mode to Fibre Channel to connect to a native Fibre
Channel SAN.
Defining pools and policies creates a set of configurations to be used. The example
shows a management IP pool being created but other pools exist (WWN, MAC ...).

Every server needs its own service profile, so ranges must be configured for the values
which are valid for assignment. Pools contain the arrays of valid numbers to be
assigned. Similar to when you define DHCP pool for assignment in an IP network
environment.

You combine individual pools and policies into one template by following a series of
configuration steps.
Two types of service profile templates are available: Initial templates generate
individual service profiles and any changes to the template are not carried over to each
service profile that is created from the template; Updating templates are linked to their
service profiles and all the service profiles that are generated from the template change
when the template changes. It is possible to unbind service profiles from their templates
if needed.

Updating templates are particularly useful when changing a setting across several
devices. So, if there is a BIOS setting that must be changed for the entire group, the
updating template will save time, because you will only need to change the setting once.

Once a template exists, you can generate several service profiles and associate them
with servers, which will push the configuration to the server.

The process of service profile association is usually disruptive and will cause the server
to reboot. Keep this information in mind when changing associations in a production
environment.

Servers can be automatically associated from a server pool by the Server Discovery
Policy. Ports that can also be automatically configured as hosts are connected to them.
This feature enables you to simply plug another server into the Fabric Interconnects and
it will be automatically configured without any need for additional configuration.

The flexibility of Cisco UCS Manager provides many options for very complex
deployments, but if even further automation, oversight, and self-service provisioning are
required, the Cisco UCS Manager allows integration into additional management
systems such as Cisco UCS Director, Cisco UCS Central, or Cisco Intersight. Each of
these options uses the Cisco UCS Manager to communicate with the infrastructure.

The Cisco UCS Manager solution supports REST API calls for cases when you need to
combine Cisco UCS Manager with your custom management solution.
3.1 Describing Cisco HyperFlex Software Components

Introduction
The Cisco Unified Computing System (USC) hardware is used as the physical platform
for the Cisco HyperFlex Data Platform (HXDP) installation. Cisco HXDP is the
software component of the Cisco HyperFlex solution and is installed through an
installer on the Cisco HX-Series Servers to create the hyperconverged infrastructure
(HCI) storage platform.

You can install Cisco HyperFlex through Cisco Intersight or through the on-premises
installer virtual machine (VM). In either case, the installed software uses the same
components, functions in the same way, and provides advanced integrated-storage
features to the Cisco UCS Server systems.

There are major benefits of using Cisco HyperFlex Data Platform software on Cisco
UCS hardware:

 Simplified installation: The fully integrated nature of Cisco HyperFlex and its
single-vendor design provide the ability to autoconfigure and autoinstall all
HyperFlex components with minimum user involvement.
 Storage within servers: The storage is not separated from the servers, such as in
SAN systems, so it is easier to scale and connect. At the same time storage is not
reliant on Redundant Array of Independent Disks (RAID) cards, which limit the
scalability and data protection options.
 Modular design: Individual components of the system are repeated many times
making the system much easier to scale and design. This also provides excellent
component redundancy and failure tolerance.
 Distributed storage operations: The modular design combined with the fast
network of the HyperFlex solution allows load to be evenly distributed between
systems in a HyperFlex group.
 Integrated management: The single vendor ecosystem and full-stack integration of
HyperFlex provides the ability to centrally manage the HyperFlex hardware,
firmware, and software.
 Compression: A process of encoding the same data with the minimum possible
information.
 Deduplication: A technique for eliminating duplicate copies of repeating data over
the entire storage system is also always enabled and in tandem with compression
optimizes the storage space available.
 Native snapshots: A set of reference markers for data at a particular point in time is
provided by the native capability of StorFS—the HyperFlex filesystem.
 Extremely fast VM cloning: A clone is a copy of an existing virtual machine. By
using the features of StorFS, you can quickly create many fully storage optimized
copies of a virtual machine.

The Cisco HyperFlex solution differs from other hyperconverged solutions in its
consistent performance, deep integration of networking, ease of installation, and simple
administration.
Note
Many Cisco HyperFlex features are independent of the hypervisor, but there are some
differences. For clarity, this section focuses on the Cisco HyperFlex Standard
deployments with the ESXi hypervisor, unless specifically stated otherwise.

3.2 Describing Cisco HyperFlex Software Components

Virtual Machine Hypervisor

This video explores the virtual machine hypervisor and log-structure file system. The ESXi
hypervisor is the most common deployment and comes pre-installed on Cisco HyperFlex
servers from the factory. The hypervisor management platform provides detailed control
over the virtual machine deployment and management. Most of the hypervisor configurat
ion is done by the HyperFlex installer automatically as you install HyperFlex. The HyperFle
x installer also integrates with the HyperFlex plug in into vCenter for management.
Hypervisor provides
a communication layer between the software and the hardware and passes the physical st
orage to the controller VM, which then integrates it into the HyperFlex data platform. The
storage created by the HyperFlex data platform is then presented back to the hypervisor a
s a network file system version 3, or an FSV3 share. This way, the underlying HyperFlex se
amlessly integrates into the hypervisor and provides all of the benefits for the data platfo
rm.

The storFS is a log-structured file system used by Cisco HyperFlex. The files on the disk are
just strings of binary code and data, which is referenced in a ledger or allocation table, wh
ich describes its location. Obviously, it's impractical to try to reference every single 0 and
1 written to the file system. So as you see here, a reference grid of pieces is created on th
e disk. Now each field on this grid is called a block. These blocks are referenced when you
r computer looks for the files in the file system ledger.

Now the drive does not look for enough empty space but rather just writes the data in the
first available block. The log-structured file system then keeps track of all these pieces of
files written on the physical storage using what's called inode pointers. They point to the i
ndividual blocks that are stored on the physical storage, so it knows how to quickly assem
ble the file when it's reading it. This sequential operation of log-structured file system
speeds up the write process.
When the drive comes to the end of the physical storage, it then goes back and starts at t
he beginning of the storage and then just writes into the empty space between the used b
locks. These blocks contain information which, when combined, produces all the informat
ion on the file. Now an individual block-- and this is a powerful feature-- an individual bloc
k is reusable if two files have identical parts, instead of storing identical copies.

This is great for creating clones, since a clone is mostly identical with minor changes. So in
stead of creating an exact clone, I just need to create the information that's different and
then use pointers to point back to the original block. All the blocks that are the same will
always be reused, while the differences are written to the empty blocks.

This minimizing operations is when you're writing out to the disk, you want to minimize th
e operations because it's highly beneficial when you're writing to solid-state drives, or
SSDs. Now here's a great thing. Since there's no physical components in the SSD, the spee
d of the seek times are not affected by the writing. The problem is writing affects the long
evity of the SSD.

And what that means is each SSD chip has a theoretical limited number of writes in its life
span. It's some astronomical number, but you don't want to keep rewriting over and over
and over again. So this system, well,
a perfect example is, remember how we used to defrag the disks in Windows? You don't
want to do that if you have an SSD. You don't need to defrag the disk because SSDs there'
s no delay. And if you start defragging the disk in Windows, you're just burning up that li
mited number of writes in its life cycle.

So anyway, this system of pointers provides a storFS with a native deduplication mechanis
m. This decreases writes because, once again, identical blocks are going to be linked not c
opied. So you don't have multiple, identical copies, you just have pointer files pointing ba
ck to the original copy. The actual savings depends on the number of identical blocks.

Finally, this figure here lists the benefits of the storFS for all flash and hybrid HyperFlex. O
nce again, storFS pointers not only point to locally-stored information, but they can also
point to blocks across the HyperFlex nodes. This, along with replication capabilities, provi
des data redundancy. This means the individual copies can be lost without affecting the p
erformance of the data platform because they're actually stored in multiple locations for r
edundancy.

All HyperFlex converged nodes also include caching drives, which buffers the information
before writing to the hardware. This will decrease the number of writes, so you can increa
se the longevity of your solid-state drives. And it allows for much faster response times.
That's the end of this video. Thanks for watching.

The virtual machine hypervisor is the base operating system installed on individual
Cisco HyperFlex nodes. Both converged and compute nodes need a hypervisor
installed.

The hypervisor is the hub where all the pieces of the entire Cisco HyperFlex solution to
come together. As of HyperFlex Version 4.5(1a), the VMware ESXi and Microsoft
Hyper-V are supported as hypervisor options in HyperFlex Standard deployments. The
ESXi hypervisor is the only hypervisor available for Stretched and Edge HyperFlex
deployments. ESXi is much more common than Hyper-V on Standard deployments.

ESXi provides several mechanisms used by Cisco HyperFlex, such as Distributed

Resource Scheduler (DRS), high availability, VMware vSphere® Storage APIs – Array
Integration (VAAI) and others.
The ESXi hypervisor needs to be installed on each individual server in a Cisco
HyperFlex group before you can install the HyperFlex solution. The ESXi hypervisor
comes preinstalled on Cisco HyperFlex servers from the factory, so that the hypervisor
installation does not need to be performed manually by the customer.

Note
The Hyper-V hypervisor does not come preinstalled on Cisco HyperFlex servers and
needs to be installed during the HyperFlex installation. Since HyperFlex Version 3.5.2a,
this is a fully automated process performed by the HyperFlex Installer.
The ESXi hypervisor in Cisco HyperFlex:

 Serves as the bare-metal operating system, providing the link between software and
hardware.
 Runs HyperFlex CVMs, which perform HyperFlex storage tasks.
 Enables a virtual environment for running, deploying, and managing of VMs.
 Integrates virtual network components into the physical network.
 Provides the platform for services such as HyperFlex IOVisor, VMware DRS, high
availability, and VAAI.
 Enables the integration of HyperFlex solution into vCenter management system.

The hypervisor provides detailed control over the virtual machine deployment and
management.
Some VM-specific features are also integrated into HyperFlex Connect, which is the
Cisco HyperFlex management platform. This allows administrators to perform
HyperFlex-specific VM tasks such as native cloning, native snapshotting, and native
snapshot scheduling through HyperFlex Connect.
The basic hypervisor configuration is done by the HyperFlex Installer automatically as
you install Cisco HyperFlex. The network, controller VM configuration, vCenter
integration, and ESXi base configuration are all configured automatically from the
information that you provide to the HyperFlex Installer.
The hypervisor provides a communication layer between the software and hardware and
passes the physical storage to the CVM, which integrates it into the HXDP. The storage
created by the HXDP is then presented back to the hypervisor as a Network File System
Version 3 (NFSv3) share. This way the underlying HyperFlex seamlessly integrates into
the hypervisor and provides all the benefits of the data platform.

3.3 Describing Cisco HyperFlex Software Components

Log-Structured File System

Cisco HyperFlex StorFS, a log-structured file system, is the power behind the Cisco
HXDP. It provides advanced file system features and utilizes the network to provide
real-time on-write redundancy and data distribution. The system is highly optimized and
provides distributed reading and writing across the infrastructure, inherently balancing
the load across nodes of the group.

Note
Whenever you interact with the data on your computer, you are usually operating with
files arranged in hierarchical folders. These files have different sizes, and you might
have noticed that larger files take more time to manipulate than smaller ones. This
process is different when you compare it to moving objects in real life. You simply pick
them up in their entirety and move them from one drawer to the next. In the computer
world, files are not complete objects, but streams of information encoded in binary.
Therefore, when you move a file, your computer does not move the entire file at once as
you would an object; it reads the first piece, then the next, and so on, until all the pieces
are moved.

Note
Your files exist on your disk just as strings of binary encoded data. The information
written to the disk is then parceled and referenced in a ledger—an allocation table—
which describes where on the disk the file or its pieces reside. Because it would be
impractical to reference every 0 and 1 written to the file system, a reference grid of
pieces is created on the disk. Each field in this grid is a unit called a block. The blocks
are then referenced when your computer looks for your files in the file system ledger.

There are several ways in which the referencing of chunks works and how chunks are
read and written depending on the file system. You can decide to keep chunks that
constitute a file as close as possible. This system was utilized by the more traditional
filesystems and worked great on spinning disks, as there was little physical distance
between parts of a file. In such a system as the time goes on and files get changed, the
pieces get fragmented, and disorganized, which decreases the performance.

Log-structured file systems take a different approach to data organization. Writes to

storage are performed sequentially. Instead of keeping different blocks of a file all in
one place, the individual blocks that constitute a file are written in sequence as they
come in. The drive, which is performing the write, does not need to look for empty
space on the physical media where to put the next block but writes it in the first
available block.
The log-structured file system keeps track of all the pieces of files written on the
physical storage by using iNode pointers, which point to individual blocks that are
stored on the physical storage. iNodes tell the log-structured filesystem how to assemble
the file from the blocks on the physical storage.

StorFS writes information to a disk sequentially and creates an iNode index where
individual parts of files are:

 All the information is written to the first free space available; the drive does not
have to randomly seek free space.
 Drive does not have to seek on write, improving spinning drive performance.
 Data is not deleted on changes only marked for deletion—write latency is reduced.

You can use the sequential read/write operation of a log-structured file system to speed
up writing on spinning drives significantly. Because the spinning drives read/write
using a physical moving head, they have much slower seek times and any seeking for
free space would increase write latency.

Spinning drives are physically limited when reading by the physical read head, which
has to read information off the physical spinning disk with strings of data written to it.
This read head must physically move to where on the drive platter the data being read is
stored. It also has to relocate to where free space is when writing to storage. If the writes
are sequential, the movement of the read head is minimal, which reduces latency.

Spinning drives are physically limited when reading by the physical read head, which
has to read information off the physical spinning disk with strings of data written to it.
This read head must physically move to where on the drive platter the data being read is
stored. It also has to relocate to where free space is when writing to storage. If the writes
are sequential, the movement of the read head is minimal, which reduces latency.

A log-structured filesystem makes passes of the entire spinning drive sequentially, much
like a gramophone does with vinyl records, once the drive comes to the end of the
physical storage, it starts from the beginning and read/writes in the space between used
blocks.

Whenever a block is changed or deleted it is not actually physically deleted from the
drive, instead it is marked for deletion by the cleaner process. The cleaner process is a
background parallel job performed by the operating system that reclaims the unused
blocks and makes them available for writing on the next pass.
StorFS is a log-structured file system, so it keeps on sequentially writing all the
information to the storage as it comes in and instead of keeping track of the start and
end blocks of file pieces it keeps track of each individual file piece. Whenever you
make a change, that change is written to the disk in sequence, and the pieces of the file
are only referenced in the file system ledger.

Writing changes to the storage is also accelerated by the log-structure:

 The new information is just appended to the first free block (Block 12).
 The pointer in the StorFS index changes from the obsolete block (Block 11) to the
new one.
 Cloning does not create a copy. Cloning only creates a new entry in the file system
pointing to the same blocks.

StorFS uses a system of iNode pointers to construct data. These pointers are basically
linking to data blocks written on storage. The blocks contain information, which when
combined produce all the information in a file.

Log-structured file systems write data sequentially, without any regard which files own
the block. Sequential writing reduces spinning drive seek times and does not require
data to be copied to have file system consistency as the data is already completely
written when the pointer to the block is updated. After the update, the old block is not
deleted, further increasing the performance.

StorFS, like most log structured file systems, uses a separate cleaner process to remove
the obsolete data from storage. This process compacts the storage for subsequent
writing. The cleaner utility constantly runs in the background and goes into aggressive
mode when the system is running out of space. Aggressive mode uses a lot more
resources than a normal cleaner job, which is one of the reasons why you should always
have sufficient storage for normal HXDP operation.

Minimizing operations when writing is also highly beneficial when writing to SSDs.
While reading/writing is not affected by seek times with SSDs, the longevity of an SSD
drive is affected by writing. Each SSD chip has only a limited number of writes in its
life span. StorFS follows the same linear read/write operation on SSDs, utilizing chips
evenly.
StorFS provides these benefits to SSDs due to their limited number of rewrites:

 As few write jobs as possible in the write process by indexing, caching, and
compression.
 Clones are created from pointers instead of copying, reducing writes to hardware.
 Parts of an SSD drive are used more evenly and last longer.

Since there are no moving parts in SSDs, the SSD response times are much more
consistent as data is written across a system of chips instead to a physical platter.

An individual block is reusable by referencing it in iNodes several times. If the two files
have identical parts, the data does not need to be written on the storage twice. This
reusability is great for creating clones, since a clone is mostly identical with minor
changes. All the blocks that are the same get reused, while differences are written to
empty blocks. This applies to any kind of duplicate data on the storage. The iNodes
provide a filesystem native deduplication mechanism.

Since Cisco HyperFlex uses inline deduplication and compression, it reduces the size of
the data that needs to be written, which increases the lifespan of SSDs because fewer
changes are done to the storage.

These common benefits of StorFS apply for all versions of HyperFlex:

 Native deduplication rooted in the design of the file system.

 Superb file system level cloning and snapshot capabilities.
 Pointers can refer to storage that is reachable over the network on another
HyperFlex node and does not exist locally.
 Pointers can take into account redundant copies on HyperFlex storage, providing
redundancy.
 Physical storage on several physical machines can be presented as one contiguous
storage platform.

In StorFS, pointers do not have to refer only to locally stored information but can point
to blocks across the HyperFlex nodes. This ability, combined with the replication
capabilities (creation of replicas/copies of data from one storage location to another) of
HyperFlex, provides data redundancy, where individual copies can be lost without
affecting the performance of the data platform.

All HyperFlex converged nodes also include caching drives, which buffer quickly
changing information before writing to the hardware (destaging), which decreases
writes to SSDs increasing their longevity. In hybrid deployments, the cache is also used
when reading. It caches the information, allowing for much faster response times.

3.4 Describing Cisco HyperFlex Software Components

Cisco HyperFlex Snapshots vs. VMware

Snapshots
In this video, let's examine Cisco HyperFlex snapshots versus VMware snapshots and regul
ar virtualized server. A VM snapshot is a copy of a virtual machine's disk file at a given poi
nt in time. It can be used to restore a VM to a particular point in time when a failure occu
rs. A good idea to have as a fallback mechanism when you're doing a risky operation such
as software upgrade.

VM vSphere provides a snapshot functionality as part of the vSphere solution. All new rig
hts are made to the most recent snapshot leaving the old ones frozen in time, which are b
asically obsolete unless you need to roll back immediately after the upgrade, for example.
Over time, the size of the snapshot can become bigger than the original, which affects the
performance and eats up storage. Fortunately, you can choose to consolidate the snapsho
ts and merge the old one and the new one into one .vmdx file. This new file will contain t
he most current information on each of the files in the virtual drive.

Cisco HyperFlex improves on snapshot functionality. HyperFlex uses file system native sna
pshots, which utilize the pointers of the storFS to automatically optimize a snapshot. Rem
ember we talked about why keep duplicate copies when you can point to the same inform
ation. Since the snapshots are created through the direct file system integration, they are
very responsive, especially when deleting or consolidating snapshots. There is real no dat
a movement. Everything is done through the storFS.

It's important that the first snapshot be a native HyperFlex snapshot. This is the one that
defines all of the future snapshots, which will be HyperFlex native snapshots. HyperFlex n
ative snapshots will show up in vCenter the same as a normal snapshot. And then they ca
n be managed the same. The neat thing is there's no limitation on the age of the number
of snapshots-- age or number, I should say.

Since HyperFlex shares its hardware with regular Cisco UCS C-Series, the underlying
hardware is no different from a regular virtual server. However, the functionality of stora
ge is vastly different including reading, writing, scaling, and resiliency. When several drive
s are striped to create a larger unit, it's usually done through RAID implementation. Now
HyperFlex, on the other hand, utilizes the controller virtual machine, or CVM, which creat
es the data store and shares it with the hypervisor using NFS version 3 network share.

The drives, in this case, are passed through the CVM, and the hypervisor does not write to
the data store independently. The CVM is responsible for merging individual drives into o
ne great, big, large HyperFlex data store. The IOVisor enables the storFS to function acros
s the entire Cisco HyperFlex cluster. And then lastly, the vStorage API for Array Integratio
n, or VAAI, is a VMware API for managing storage in the vSphere environment.
So when you're scaling a regular virtual server deployment, the data store is created from
local server storage or from a centralized storage. The data stored in a
HyperFlex environment spans the network to create a single storage from all of the storag
e to your drives. Also, the setup of the additional node must be performed manually in a r
egular environment, whereas HyperFlex offers an installer to make expanding the infrastr
ucture much easier.

The Cisco HyperFlex Controller Virtual Machine, or CVM, is the main software component
of Cisco HyperFlex data's platform. The CVMs must exist on all of the servers in the Cisco
HyperFlex cluster. Otherwise, the servers would not be able to communicate with the Hyp
erFlex data store. So when you connect to the management IP address of the Cisco Hyper
Flex cluster using HTTP, you're accessing the HyperFlex Connect or HF Connect Managem
ent platform on a CVM. The controller VM is responsible for handling of application progr
amming interface calls and provides an API Explorer. The CVMs are connected by the man
agement network, and they synchronize the configuration between all of the CVMs.
The IOVisor is an ESXi vSphere installation bundle driver. And it intercepts the read-write
request and then forwards them to the CVMs across the cluster. Now when a CVM fails, t
he IOVisor still remains active and functional. And this enables uninterrupted operations
of the system by forwarding I/O to another available CVM in the HyperFlex cluster.

This distributed input/output model also allows reading and writing to have that load
shared amongst all the hosts, meaning that there's no hotspots created where you would
have a single host that would get overloaded with I/O while others would be on standby
or light load. It's completely shared. The IOVisor also enables replication across the data n
etwork.
Finally, the vStorage API for Array Integration, remember VAAI, is a VMware API for mana
ging storage in a vSphere environment. VAAI is not a HyperFlex specific technology. It's a
general API offered by vSphere, which enables ESXi host to offload storage-intensive
operations directly to another handler. So in HyperFlex, it enables the storFS pointer-
based snapshot and the cloning operations. That brings us to the end of this video. Thank
s for watching.

Snapshots are performed on a VM to create a freeze-frame of the VM's state at some

point in time. These snapshots make reverting to a working state very easy and is one of
the major advantages virtual environments have over bare-metal deployments.
Whenever you are performing a risky operation, like a general update of the virtual
machine software, it is beneficial to have a fallback mechanism, such as a snapshot of
the virtual machine in a working state.

Snapshots can be made obsolete by new data stored on the virtual machine disk.
Reverting to a previous snapshotted VM state in time will erase the data performed after
that point.

As a snapshot is created, the current information is frozen and a separate VM version is

created. To avoid creating a full copy, the new information is written to an overlay that
carries only the changes since the snapshot. Whenever the information is read, overlays
need to be consolidated in real time. This consolidation becomes slower and slower with
both the number of overlays and the size of overlays.
StorFS provides native snapshot and cloning capabilities by using its log-structured file
system design. This design increases the efficiency of the regular vSphere-based cloning
and snapshots in several ways. Instead of using overlays that need to be consolidated,
iNode pointers are used to follow changes.

VMware Snapshots
VMware vSphere provides the snapshot functionality as a part of the vSphere solution.
These snapshots are easy and efficient to create but have limitations because they need
to be consolidated in real time.

When you are creating a snapshot in a live environment, you need to take into account
that if you simply copy data, the virtual machine will inevitably change. After the copy,
you will lose the changes that happened between the start of the copy and the end of the
copy.

vSphere addresses this issue by creating a new .vmdk delta disk, while the
original .vmdk is frozen and does not receive any updates if a newer delta disk exists.
So, vSphere can create a snapshot very quickly, because it simply branches the storage.
The downside is that whenever information is written to the disk, it is written to the
latest copy, which acts as an overlay for the old one. Over time, the size of the snapshot
can become bigger than the original virtual machine, which affects performance and
consumes storage.

You can choose to consolidate the snapshots and merge the old and new data in
one .vmdk, which contains the most current information. During this process, you lose
the consolidated snapshots and can no longer revert to a previous point in time.

In a vSphere environment, ESXi-native snapshot capability has these characteristics:

 Each snapshot is a delta virtual disk. Previous snapshot or original gets frozen.
 All writing is done to the latest instance, which can outgrow the original.
 A lot of snapshots or large snapshots can negatively affect performance.
 The data written is not automatically deduplicated across snapshots and a lot of
storage gets used.
 Manual consolidation is needed to merge information between current and previous
snapshot.

Cisco HyperFlex Snapshots

The Cisco HyperFlex solution uses the features of its log-structured file system to
upgrade the regular concept of snapshots in vSphere. HyperFlex creates an initial
snapshot, which enables the following VMware or HyperFlex snapshots to be file
system native snapshots.

Native snapshots use the pointers of StorFS to automatically optimize the snapshotting
process. HyperFlex integrates into vSphere through VMware vSphere Storage API for
Array Integration (VAAI) to create and manage snapshots. Since the snapshots are
created through direct file system integration, they are very responsive, especially when
deleting or consolidating snapshots as there is no real data movement and everything is
done through StorFS.

When you list virtual machine disk images (VMDKs) for an individual VM, there will
still be individual files for each snapshot, but these files are written to the datastore by
referencing old and new data through pointers. The data no longer needs to be
consolidated on read but is optimized while at rest on the storage.

HyperFlex uses file system native snapshots and provides these features:

 Automatically deduplicating data of snapshots through StorFS.

 No impact on the VM performance even after a lot of writes.
 On the management level, snapshots work the same as vSphere snapshots.
 No limitations on the age of the snapshot. First snapshot must be a native HyperFlex
snapshot.

Note
If you want to use Cisco HyperFlex native snapshots, it is important that the first
snapshot is a native HyperFlex snapshot. The first snapshot defines all the future
snapshots, and all future snapshots will be HyperFlex native snapshots. If you create a
vSphere snapshot first, HyperFlex native snapshots will be disabled for that VM.

Additional features of snapshots, like quiescing the virtual machine or memory

snapshots, can be configured in the CVM CLI using the stcli vm
snapshot command. To quiesce is to pause or alter a device or application to achieve
a consistent state, usually in preparation for a backup or other maintenance.

HyperFlex native snapshots will show up in vCenter the same as normal snapshots, as
they can be managed as if they were regular vSphere snapshots, except that there is a
comment added to the snapshot stating it is a native HyperFlex snapshot.

Snapshots can also be created in HyperFlex Connect including scheduled storage native
snapshots introduced with HyperFlex version 4.5(1a).

3.5 Describing Cisco HyperFlex Software Components

Cisco HyperFlex vs. Regular

Virtualized Server
The Cisco HyperFlex Data Platform (HXDP) utilizes a virtual environment for its
operation. Since HXDP shares its hardware type with Cisco UCS-C servers, the
underlying hardware itself is no different from a regular virtual Cisco UCS server.

The differences in the operation come from the hardware configuration and the software
components of the system. The hypervisor and virtual machines are no different
themselves as in a regular vSphere virtualized environment; however, the functionality
of storage is vastly different including reading, writing, scaling, and resiliency.

In traditional server deployments where storage is integrated in the server chassis, you
ensure storage resiliency and data distribution by attaching storage drives to RAID
cards. RAID cards are storage controllers that can take several physical drives and
merge them in different ways to provide additional features.

HyperFlex virtual servers differ from regular servers in these key areas:

 No RAID: No RAID required to consolidate disks into a shared data platform.

 CVM: Virtual appliance, which performs reading/writing, caching, deduplication,
and compression.
 IOVISOR: Hypervisor driver, which mounts HyperFlex storage and distributes
data.
 VAAI: vSphere storage API allowing file-system-level snapshots and cloning.
HXDP merges the individual hosts beyond what is possible with Cisco UCS virtualized
deployments using RAID and distributes the load as evenly as possible across
individual hosts. HXDP utilizes several software components to create the HyperFlex
storage and its features.

Note
There is an extra service active in the ESXi hypervisor: stHypervisorSvc. This vSphere
Installation Bundle (VIB) adds enhancements and features needed for native HyperFlex
data protection.

In RAID-based deployments, individual physical drives are utilized directly through the
hypervisor, which is responsible for reading and writing to the drives on the physical
level.

Cisco HXDP, on the other hand, passes the storage drives to a CVM, which creates the
datastore and shares it with the hypervisor using the NFSv3 network share. The
hypervisor does not write to the datastore independently, although it does provide the
means to configure the pass-through drive operation.

The datastore in a Cisco HyperFlex environment spans the network to create single
storage to which all the HyperFlex servers (converged nodes) contribute. Also, the setup
of the additional node in a regular environment has to be performed manually, while
HyperFlex offers an installer. Both of these features make expanding the HyperFlex
infrastructure much simpler.

If you scale a regular virtual server deployment, the datastore is created from either the
server local storage or central storage. In Cisco HyperFlex, all storage tier drives create
a shared datastore from all the drives of the group.
The CVM is responsible for merging individual drives in one large HyperFlex datastore
instead of RAID. HyperFlex merges drives across servers. This has large implications
for storage resiliency.

Controller Virtual Machine

The Cisco HyperFlex CVM is the main software component of Cisco HXDP. It runs on
the hypervisor in each individual node and provides the node with access to the HXDP
storage. It is also involved in the reading and writing processes and performs advance
data functions like compression and deduplication.

The CVMs must exist on all the converged servers in a Cisco HyperFlex group to make
storage operations on those nodes possible.

Since HyperFlex Version 4.0, the compute-only nodes no longer require a CVM. The
access to the storage is fully integrated into the hypervisor component of the IOVisor,
which is installed as a VIB.

A CVM is an Ubuntu Linux VM that is placed outside the converged data platform on
the housekeeping drive. This is necessary because the CVM needs to be operational to
create the HyperFlex storage datastore.

The CVMs have these features:

 An Ubuntu-based VM running in the hypervisor of each individual server, having

direct access to the server's storage.
 Is installed automatically on HyperFlex installation, configured through the installer.
 Needs network access to ESXi, other CVMs, and management network.
 Performs caching, deduplication, and compression of data.
 Utilizes IOVisor to distribute data across the HyperFlex group.
 Provides HyperFlex Connect, HyperFlex CLI, and REST API for management.
 Provides iSCSI interface to access HyperFlex storage LUNs.

When you connect to the management IP address of the Cisco HyperFlex group using
HTTP, you are accessing the HyperFlex Connect management platform running on the
primary CVM.

The CVM is also responsible for the handling of API calls and provides an API
explorer, which is accessible by connecting to https://[CVM IP]/apiexplorer. CVMs also
provides SSH administrative access capability.

Since CVMs provide critical storage functionality for the HXDP, the HX Data Platform
Installer configures CPU resource reservations for the controller VMs. This reservation
guarantees that the controller VMs have the minimum required CPU resources.
 CVM CPU Reservation
CVM Location Number of VM CPU Shares Reservation Limit

All-NVMe groups 12 Low 10,800 MHz Unlimited

Hyper-V groups 10 Low 10,800 MHz Unlimited

All other 8 Low 10,800 MHz Unlimited

Cisco HyperFlex supports a Turbo mode configuration, which can be enabled to give
CVMs more resources. This improves HyperFlex storage performance under heavy
load.

CVM Memory Reservations

CVM Location Amount of Guest Memory Reserve All Guest Memory

HX220c M4/M5 SFF Hybrid and All-Flash 48 GB Yes

HX240c M4/M5 SFF Hybrid and All-Flash 72 GB Yes

HX240c M5 LFF Hybrid 78 GB Yes

HX220c M5 SFF All-NVMe 72 GB Yes

Note
You will still find the CVMs on compute nodes of older HyperFlex systems. Even if
you upgrade those systems to the latest HyperFlex version, the CVMs will remain. The
compute node CVMs have a tiny footprint with 512 MB and 1-Ghz resource
reservation. The compute node CVMs do not have their own management IP addresses.

The CVMs manage the HyperFlex group and are responsible for the API explorer
(HTTP), HyperFlex Connect (HTTP) and the CLI (SSH).
CVMs run the following:

 Device Connector: A service for management integration into Cisco Intersight

 HyperFlex Connect: HTML5 web interface, for the management of the HXDP.
 HyperFlex CLI: Bash-based command line for managing and troubleshooting
Cisco HyperFlex.
 HyperFlex API: REST automation interface for control of Cisco HyperFlex.

You assign a group IP address to CVMs when installing, but you can connect to any
CVM to have access to configuration options. The CVMs are connected by the
management network and synchronize the configuration between CVMs.

The CVM that carries the HyperFlex group virtual IP (VIP) address is the designated
primary CVM. Always perform management tasks on the primary CVM if possible.

IOVisor
IOVisor is a HyperFlex subsystem that intercepts read and write requests from local
virtual machines and forwards those requests across the network. It enables the StorFS
to function across the entire Cisco HyperFlex group and distributes reads and writes
across the network. It also enables replication of data across the network.

IOVisor provides data distribution and resilience to HyperFlex.

When local writing or reading is performed, IOVisor intercepts the read/write requests
and forwards them to CVMs across the group. This action allows nonlocal CVMs to be
aware of the I/O requests so that they can perform the appropriate I/O action. This
feature enables the entire group to function as one coherent storage using the network.

In vSphere, IOVisor exists in the ESXi hypervisor. When a CVM fails, the IOVisor
remains active and functional, which enables uninterrupted operation of the system by
forwarding I/O to another available CVM in the HyperFlex group.

By its nature, Cisco HyperFlex does not prioritize the local CVM to process or cache
data. In Standard and Edge deployments IOVisor distributes I/O action to any host
which is available and the local CVM is not necessary for IOVisor to function normally.

If a CVM fails, other CVMs take over the load, which does not cause any significant
deterioration of performance for the data platform. Since the load is distributed through
the IOVisor, the more CVMs that are available, the less impact a single failure has.

This distributed I/O model also allows the reading and writing load to be shared among
hosts. Meaning that there are no hotspots created where a single host would get
overloaded by I/O, while others would be on standby or under light load.

VMware VAAI Storage API

VAAI is a VMware vSphere API for offloading storage functionalities to external
systems. Even though Cisco HyperFlex uses StorFS to create snapshots and clones,
these snapshots and clones need to be visible and manageable by the vSphere
deployment.

VAAI is not a HyperFlex-specific technology, it is a general API offered by vSphere,

which enables ESXi hosts to offload storage intensive operations directly to another
handler. In the HyperFlex solution, it enables StorFS pointer-based snapshot and
cloning operations to be managed through vSphere.
VAAI provides these functionalities:

 When a native snapshot is requested, the request is processed by the hypervisor.

 Instead of the snapshot being performed in the hypervisor, it is offloaded to Cisco
HXDP.
 Cisco HXDP creates a file system native snapshot, which is registered in the
vSphere.

3.6 Describing Cisco HyperFlex Software Components

Cisco HyperFlex Data Distribution

Let's take a closer look at data distribution in Cisco HyperFlex. Traditional RAID technolog
y enables individual disks to be combined into a single unit using data-mirroring and data-
striping techniques.

Mirroring duplicates data across several disks, while striping distributes data across disks
to create a larger disk. There are several deployment options available when using RAID.

For example, RAID 0 stripes data across multiple disks, and RAID 1 mirrors data across mul
tiple disks. You can perform hot swapping of drives in a RAID environment, depending on
the configuration, obviously.

However, the RAID rebuild after a swap can take considerable amount of time. If you ever
need to replace the RAID card, you have to take care that you provide a compatible replac
ement.

Different vendors have different implementations of RAID technology, which definitely ca

n be an issue. So it's wise to stick with well-established vendors. Keep in mind, only local
drives can be configured together in using the RAID technology.
Now, Cisco HyperFlex has no local limitation. It can span the entire cluster and include ma
ny drives and many hosts connected to the network. Once again, instead of striping and
mirroring, Cisco HyperFlex introduces the replication factor, which we're going to talk abo
ut more in a moment, and inline distribution through the I/O visor and store FS.

A replication factor is used to determine how many copies of information will be kept in t
he Cisco HyperFlex cluster. Now, the replication factor choice has significant implications
for effective available disk space on a HyperFlex cluster.

The table here displays the difference between replication factor 2 and replication factor
3. RF 3 uses 33% more raw storage than RF 2, but offers additional resiliency and overcom
es its limitations. So Cisco recommends using RF 3 to provide the cluster with the maximu
m data protection.

The replication factor for HyperFlex is chosen while you're doing the installation and cann
ot be changed after the cluster has been installed without completely reinstalling the who
le thing. So make up your mind before you actually install.
Now, when a disk fails, the compute part of that node is marked as unhealthy. But it's still
fully operational. Disk failures only have about a one-minute timer on them before the
self-healing starts.

The self-healing process replicates the pieces of information that were lost when that
drive failed. These pieces are then replicated according to the configured RF, or Replicatio
n Factor, whether you're doing RF 2 or RF 3.

After there's enough copies on the cluster to satisfy the replication factor policy, the clust
er is then marked as healthy again. And the cool thing is the performance impact of the dr
ive is minimal since they just represent such a small fraction of the total storage.

Now, when a node fails, now the system is marked as unhealthy. Again, but once again, it
still remains operational. The node failure initiates the vSphere high-availability event and
moves all the VMs that ran on that failed node, moves them to other nodes in the cluster.
And this is done quickly because there's no data migration. Right? Because VMs continue
reading the data from the remaining copies. Because, remember, we talked about how m
any copies, two copies or three copies?

The self-healing process is much more extensive when a node fails, which is why the
timer, before it starts to do the self-healing, is two hours. And this delay allows the
administrator to respond to the issue and prevents unnecessary self-healing from adding
additional load to the cluster.

If too many nodes fail, obviously the cluster is going to go down. It can't keep tolerating
multiple fails, node fails.

Now, the vSphere Distributed Resource Scheduler, or DRS, follows the usage of the server
s in the cluster and redistributes the virtual machines so that cluster utilization is equally
balanced. When a new node is introduced to the cluster, it is detected by the DRS process
, and the VM migration to the new node is triggered.

Since the underlying data store where the VMs exist in the HyperFlex data store, because
they exist there, the VM is migrated without moving information from node to physical n
ode. The rebalance is a process that redistributes information across the clusters so that a
ll the nodes have equal data usage on their physical storage. And this process is initiated
daily at 5:15 AM, or it can be executed manually.

Finally, Logical Availability Zones, or LAZ, enables larger clusters to reduce the chance of g
oing down and doing multiple failures. OK? Now, LAZ uses these resiliency zones to rearra
nge the data. And it doesn't add any additional storage usage.
Now, why would we use LAZ? Well, the reason we use it for large-- we use it is that the lar
ger cluster itself has a greater chance of the number of failures that might exceed the RF,
or Replication Factor, tolerance. With LAZ enabled in a cluster, the maximum tolerable fai
lures far exceed the two-node limit, which is defined by RF 2.

Keep in mind, LAZ is only available on very large clusters with at least eight
or more nodes and require the configuration of at least three LAZ zones. LAZ can also be a
ctivated while the cluster is active, and the data will be rebalanced into the defined zones
. That's the end of this video. Thanks for watching.

You might have had experience with hard drive failures on your home computer, where
you only had information written on one drive. When the drive failed, you lost it all and
could not get it back. These failures are highly unpleasant in a private setting, but in a
business setting, they are completely unacceptable. Failures are not a new problem; they
were addressed in the 1970s with the introduction of RAID technology.

HyperFlex Data Platform RAID-Based Storage

All drives in all the servers form one storage unit One RAID controller forms a storage unit
All drives are involved in redundancy Redundancy through drive assignment
Hyperconverged approach Traditional hardware-based approach
Server failure reduces redundancy Server failure makes data inaccessible

Cisco HyperFlex does not use RAID for its operation, eliminating several limitations of
RAID deployments. It uses replication for data protection and a distributed log-structure
file system for data distribution.
Traditional RAID
RAID is a technology that enables individual disks to be redundant and combined into a
single unit using data mirroring and data striping. Mirroring duplicates data across
several disks, while striping distributes data across disks to create a seemingly larger
disk. There are several deployment options available when using RAID. For example,
RAID 0 that stripes data, and RAID 1 that mirrors data across several physical drives.

RAID controller cards are usually used when employing RAID, because software RAID
introduces additional load on the system resources (such as the system CPU). Dedicated
cards take over the processing overhead, which allows system resources to be better
spent elsewhere.

Even though RAID cards are a reliable solution and do not add any resource overhead,
there is an inherent limit to the technology where a limited number of drives can be
integrated into one RAID group, depending on RAID configuration (usually 32).
Several RAID groups can exist on a single system, but the maximum size of the
datastore, which can be created out of these drives, is still limited to one server.

RAID cards enable striping, mirroring, and parity, with these features:

 No load on the system resources, drives seem as one drive to the operating system.
 Hot replacement of drives available, depending on configuration.
 Disk replacements require RAID rebuilds, taking a long time.
 On RAID card failure, the RAID card compatibility can be an issue.
 Limited drives in a raid field, depending on solution, limiting scalability.
 Only local drives can be in RAID together.

A RAID card usually includes a BIOS that can be configured at boot by pressing a key
combination, similar to server BIOS. This action enables the card to be configured in
different RAID modes. Each mode provides unique advantages and limitations.
Note
Cisco management systems such as Cisco IMC and Cisco UCS Manager have the
capability to manage the onboard raid controllers without accessing their BIOS during
the reboot.

RAID does not function through a network; therefore, all the drives that are grouped
into a RAID must be attached to the local RAID controller. This restriction limits data
protection possibilities, as it cannot span several servers. If a server gets completely
destroyed (like in a flood), all the information is lost, even though the data is locally
protected.

If you ever need to replace the RAID card, you have to take care that you provide a
replacement that can access information on your drives. Different vendors have
different implementations of RAID technology, which can cause severe problems if you
need to replace your card from a vendor that went out of business many years ago. It is
wise to stick to well-established vendors with years of successful presence in the
market.

Data Distribution in Cisco HyperFlex

The Cisco HyperFlex solution leaves behind the traditional RAID single host data
protection and data distribution. Cisco HyperFlex uses a server-group-level data
protection and distribution. This implementation spans the entire HyperFlex system and
merges many drives and many hosts into one distributed system through the Ethernet
network.

Instead of using parity and striping to provide data protection and distribution, Cisco
HyperFlex introduces the replication factor and network-based data distribution through
IOVisor and StorFS as data protection mechanisms.

The replication factor determines how many copies of information will be kept in the
Cisco HyperFlex group. The replication factor 2 (RF2) will have two copies available,
distributed across two nodes. The replication factor 3 (RF3) means that there will be
three copies available on the group. The example figure is based on RF3.

Cisco HyperFlex replaces the traditional RAID with these

functionalities:

 Many more drives from many systems can be used to create

a single datastore.
 Data is stored across the hosts in the HyperFlex group, not
just on server.
 Failures and maintenance initiate self-healing with minimal
impact.
 No RAID cards required. Fast network is part of the solution.

The replication factor choice has significant implications for

effective available drive space on the HyperFlex group. The RF3
uses up 33 percent more raw storage than RF2 but offers
additional resiliency.

Cisco recommends using RF3 to provide the group with

maximum data protection. To appreciate how much more safe
RF3 is compared to RF2, you only need to review the following
table. The RF3 uses a third more space but it can usually
tolerate 50 percent more failures.

Maximum Tolerable Failures Dependent on Replication Factor

3-4 Nodes 5 Nodes or More (No LAZ)

Replication Factor 2 1 node or 1 drive failure 1 node failure or 1 drive failure

Replication Factor 3 1 node failure or 2 drive failures 2 node failures or 2 drive failures

Replication factor implies:

 RF3 has up to 100 percent better protection of data while using 33 percent more
space and is recommended.
 When performing rolling upgrades, the upgraded node is down for maintenance,
stretching RF2 to the absolute limit.

There is an inherent limitation in three- and four-node groups using RF3, where only
one node can fail at a time. However, if you plan on scaling the HyperFlex system, the
deployment will eventually grow beyond five converged nodes. Then the RF3 will be
even more beneficial, but you cannot change the replication factor on the fly.

Note
The replication factor in a Cisco HyperFlex system is chosen on installation and cannot
be changed after the group has been installed without completely reinstalling the entire
HyperFlex software stack. Reinstallation removes all the data from the storage tier
drives, so choosing the appropriate replication factor when installing is very important.

Disk Failure
When a disk fails, it poses only a minor issue for the Cisco HyperFlex server group.
Since the compute part of the node is still fully operational, no high-availability event is
triggered by a disk failure. When several disks within one node fail, the group is not
disrupted. If you lose all disks, the entire node will be marked as failed.
Disk failure initiates this process:

1. If the replication factor is sufficient for the failure, the system is marked as
unhealthy but remains operational.
2. VM running on the node is not migrated and the I/O continues from copies.
3. Performance is almost unaffected. Sets 1-minute timer until self-healing starts.
4. After 1 minute, the missing pieces are re-created from the remaining instances.

Disk failures only have a 1-minute timer on them before self-healing starts. The self-
healing process replicates the pieces of information that were lost when the drive failed.
These pieces are then replicated according to the replication factor. After there are
enough copies on the group to satisfy the replication factor policy, the group is marked
as healthy.

When two disks fail on two nodes, the replication factor 2 is not sufficient anymore and
the group would go down. If your group is using the replication factor 3, then the same
timer is triggered as when one drive fails. After one minute, the self-healing will
reproduce the missing pieces and store them on the remaining drives of the group after
which the group will be healthy again.

When you lose more components than it is tolerable, the Cisco HyperFlex system goes
offline. Going offline does not mean that all the information on the drives is lost, but the
operation stops. In this case, your workload will experience downtime. Therefore, it is
important to keep on the safe side of the resiliency line, even during upgrades, when one
node is down. Hence, the recommended replication factor is 3.

The performance impact of a drive failure is minimal, since they represent only a small
fraction of the total storage and the HXDP is optimized to perform reads and writes
from anywhere in the group. Basically, it is business as usual for the HyperFlex group,
except it has to find information elsewhere instead of reading it from the failed drive.

Node Failure
Failure on a node initiates the following process:

A node failure initiates the vSphere high-availability event and moves the VMs that ran
on the failed node to other nodes in the group. This action is done as quickly as
possible, but still no data migration happens, since the StorFS performs storage
operations of the VM on other copies on the group.

1. The system is marked unhealthy but remains operational.

2. The VMs on the failed node are moved to another node by vSphere high
availability.
3. VMs keep reading from the remaining copies with minimal impact to performance.
4. A 2-hour countdown initiates before the self-healing process.

Node failure on a three-node server group reduces the available amount of compute and
physical storage in the group significantly, however the storage operations are offloaded
to the remaining nodes and spread among the remaining nodes. This keeps the workload
performing as efficiently as possible, typically the impact is minimal. HyperFlex server
groups experience less impact the larger they are, since one node represents a smaller
percentage of the complete HyperFlex group.

The self-healing process is much more extensive when a node fails, which is why the
timer before it starts is set to 2 hours. This duration can be changed, if necessary, by
contacting Cisco Technical Assistance Center (TAC). The delay allows the
administrator to respond to the issue and prevents unnecessary self-healing from adding
load to the system.

When too many nodes fail, the group will go down. Remember that full 2-node failure
resiliency only comes into effect on replication factor 3 and group size of five or more
nodes.

Expansion and Hardware Replacement

When you replace a node or expand a group, the following happens:

1. vSphere DRS migrates the virtual machines to the new node to balance the load.
2. On node replace, the self-healing has to finish for the group to be healthy.
3. The new node is already used for writing, but the old data is not migrated until the
rebalance process.
4. Rebalance is initiated daily at 5:15 AM or can be executed manually with the stcli
cluster rebalance command.

vSphere DRS is the vSphere functionality that follows the usage of servers in the group
and redistributes the virtual machines so that the group utilization is as even as possible.
As a new node is introduced to the group, it is detected by the DRS and VM migration
to the new node is triggered. Since the underlying datastore, where the VMs exist is the
HyperFlex datastore, the VM is migrated without moving information from physical
node to physical node.
When adding new hardware to the group, the new hardware will be used for current
writes only and subsequent reads of written information, but the data distribution will
never be even until a rebalance happens.

Rebalance is a process that redistributes information across the group so that all the
nodes have equal data usage on their physical storage. Due to the distributed nature of
HyperFlex, it is the most optimal and desired state.

Logical Availability Zones

A logical availability zone (LAZ) enables larger groups to have statistically less chance
of going down when more failures happen. The larger the group, the greater the
statistical probability that the number of failures will exceed the tolerances defined by
the replication factor.

LAZ function segments the group into resiliency zones to increase tolerable failures:

 The function does not introduce any additional storage usage, it only rearranges the
data.
 Replication factor defines how many complete zones can be lost before the system
goes down.
 In RF2, you can lose 1 complete zone, in RF3 2 complete zones.
 Only available on larger groups of 8 or more nodes, with three zone minimum.
 Can be activated while the group is active, and the data will be rebalanced into
zones.

While the worst-case resiliency of a LAZ-enabled group is no different than if LAZ was
not enabled, the maximum tolerable failures far exceed the two-node limit defined by
the RF3.
While failures of servers are localized within one or two zones with RF3, the group will
remain online. In a worst-case scenario, three servers could fail in three different zones
and the group would go down the same as without LAZ.

Statistically, the chance of this happening is low, while there is no performance impact
in enabling LAZ beyond initial rebalancing of information. The functionality also does
not increase storage space taken on the group.

3.7 Describing Cisco HyperFlex Software Components

Writing and Reading Process

In this video, let's review the reading and writing process. To provide optimal performanc
e, Cisco HyperFlex employs several hardware layers of storage, each which have different
roles. For example, random access memory is the most responsive storage tier. Since RA
M is volatile and doesn't persist to reboots, it's used for functions like file system indexing
, deduplication, and so on.

To overcome the limitations of random access memory, the caching tier of storage offers
the next best thing in storage responsiveness, and it is persistent. The caching tier is also i
nvolved in deduplication and compression of the data.
All information is redundantly cached on the caching drives before being written to the ca
pacity drives. They function very much like random access memory, but they don't lose th
e information when they're powered off, making them especially well-suited for caching
random input/output loads where significant gains can be achieved.

Lastly, the capacity tier is the persistent and redundant tier. The capacity tier is for long-
term storage distributed across the individual HyperFlex nodes over the network.

Cisco HyperFlex uses a distributed file system, which writes several copies of the data on
several nodes at the same time. Now, the number of copies that get written is determine
d by the replication factor. For maximum performance, HyperFlex uses caching extensivel
y.

Now, the cache drive is not one uniform drive, but is instead segmented into functional p
arts, each one used for a special use. First, the primary write cache hosts the information
of the main data copy.
Next, each level of the write cache is separated into two parts. One is currently active and
the other is passive. Writes are performed to the active part and passives are read-only.
The second and third levels are reserved for the primary and secondary redundancy copy
of information, where the primary controller VM mirrors information across the cluster. N
ow, on hybrid Cisco HyperFlex deployments, most of the cache drive is dedicated to the r
ead log.

Now, this figure here shows the steps performed when writing in Cisco HyperFlex. First, t
he application and operation system initiates a write request. Then the VM writes it to its
.vmdk file. The IO Visor intercepts the write requests and forwards it on to the primary co
ntroller VM for a given block. This decision is made on a per block basis. Each block is cho
sen from the entire cluster, providing distribution of blocks across the converged nodes in
the entire cluster.

Once the IO Visor forwards the information to the primary CVM, it performs compression
on the information and commits it to the primary write log while simultaneously forwardi
ng the compressed block based on the replication factor. Once the data is written into the
write log of an individual node, the CVM sends a write acknowledgment to the virtual mac
hine confirming its access.
When a primary write cache of a node is full, it then goes into the passive mode, and the
destaging process is initiated. The information stored in the primary log is then deduplicat
ed on the primary CVM. This prevents duplicate data to be written several times across th
e same storage.

The primary node then initiates the distribution of the deduplicated blocks across the enti
re cluster. Lastly, the contents of the primary cache are then mirrored to other nodes, pas
sed to their controller VMs, and written to the capacity drives.

This figure here shows the steps performed when reading in the Cisco HyperFlex solution.
First, the application or operating system that initiates the read request. The VM then rea
ds from its .vmdk file. This is when the IO Visor intercepts the request and forwards it to t
he controller VM, which starts to locate the data.

The read process tries to locate the requested data from all of the available sources befor
e initiating a read from the capacity tier. This process increases the speed of the entire re
ad process and prioritizes recently written and most commonly read information.
After the IO Visor intercepts the read request, it checks the active write log in case the inf
ormation was recently written. If nothing was found, it then checks duplicate write logs o
n the server. If that doesn't find any results, then it checks the random access memory ca
che. On the hybrid deployments, it will also check the read log. Finally, if the data is not lo
cated on any of these locations, it will then read from the capacity.

So this figure lists the three options for the read caching in Cisco HyperFlex. The write-
back cache, which is the default, contains only write information and mostly used informa
tion. It stores the application write requests prior to the data being stored on the hard dis
k.

Next, the write-through cache, which is the install option for the Virtual Desktop
Infrastructure, or VDI, this contains only the most commonly used data optimizing VDI per
formance.

The no caching SSD is used with all flash nodes. This is because there's little difference in
read speeds between SSDs. All flash deployments read from the primary copy if it's availa
ble.

This brings us to the end of the video. Thanks for watching.

Whenever you deploy a data center solution, performance and data protection are the
main concerns. You often have to make compromises and choose how much of one you
will sacrifice for the other. For example, when you calculate the storage I/O operations
per second (IOPS), whenever you have to write data twice to allow for data protection,
it negatively impacts the effective IOPS of the system. The problem is compounded
when both writes have to be performed locally. It is further compounded when one of
the copies has to be moved to another server for resiliency if there are failures.

VMware with HyperFlex VMware with local storage

Group use HyperFlex distributed storage. Each server uses its local drives only.
 VMware with HyperFlex VMware with local storage

VM runs on a single node. VM runs on a single node in the group.

Storage operations are performed on all nodes at Storage operations are performed only on the
once. server VM runs on.
Servers share the storage workload of the VM. Single server performs all the workload of the VM.

All these issues are alleviated by a truly distributed system. The entire Cisco HyperFlex
system is designed from the ground up to avoid data locality, it performs all the reads
and writes across all the nodes of the system and, by design, does not prefer one node
over another. Even when you encounter failures, the system will be minimally affected.
There is no single point of failure, and the server group operates in the same way as
without failures.

Note
Many of the writing and reading processes depend on the chosen replication factor,
which determines the number of data copies that the Cisco HyperFlex system uses. For
this demonstration, the replication factor of 3 is used in the examples and figures.

Tiers of Storage in Cisco HyperFlex

To provide optimal performance, several hardware layers of storage are employed in the
Cisco HyperFlex solution. Each hardware layer, or tier, performs an integral part of the
reading and writing process, and includes RAM, a high-performance SSD-based
caching drive, and the storage tier, which consists of drives that are connected to all
converged nodes of a Cisco HyperFlex group.

Different storage tiers play different roles in operation:

 Memory (RAM): Memory contains file system metadata (index) and helps in
handling rapid data bursts.

 Fastest storage available but limited in capacity.

 Cache (SAS SSD, or NVMe SSD or Optane NVMe SSD): Caches writes before
writing to the capacity drives, helps speed up writing and improves capacity SSD
longevity.
  On hybrid HyperFlex, reads are also cached, but not on the All-
Flash or All-NVMe HyperFlex, since the gains are too small.

 Capacity (HDD or SSD or NVMe SSD): Main storage for storing large amounts of
data.

 Largest but slowest storage in HyperFlex.

Although RAM is the most responsive storage option, it is limited in its capacity. RAM
is also volatile, which means it does not persist through reboots. Therefore, memory is
used in cases where the most responsiveness is required, such as file system indexing,
deduplication, and compression.

To overcome the limitations of RAM, the caching tier of storage offers the next best
thing in storage responsiveness and longevity; it is also persistent and is involved in
caching writes and, if system uses HDDs for capacity tier, reads. The caching tier is also
involved in deduplication/compression of data. All information is redundantly cached
on the caching drives before being written (destaged) to the capacity drives.

While SSDs are available for the caching tier, the highest performance option for the
cache is Intel Optane drives, which are non-volatile memory drives. They function
much like RAM but do not lose information when powered off, making them especially
well-suited for caching random I/O loads, where significant gains can be achieved.
Optane drives can only be attached by Peripheral Component Interconnect Express
(PCIe), attaching Optane modules to memory slots is not yet supported on HyperFlex as
of version 4.5(1a).

The capacity tier is also persistent and redundant. It supports both spinning disks
(HDDs) and Flash drives (SSDs). The main advantage of HDDs is the lower price. LFF
HDDs offer more capacity per drive than small form factor (SFF) spinning disks. The
capacity tier is the long-term storage distributed across the individual HyperFlex nodes
over the network. The HyperFlex system can offer up to 1.1 petabytes of usable storage.
This already sizable usable space is further extended by deduplication and compression,
whose gains vary with the use case.

Note
Aside from the storage tiers, two other drives are present in Cisco HyperFlex M5
servers, which are not part of the actual HyperFlex datastore reading and writing
process, but provide the underlying infrastructure for the operation of a HyperFlex
system. The housekeeping drive holds the HyperFlex controller VM and HyperFlex
Data Platform logs, while the internal M.2 drive provides storage for the Hypervisor,
replacing the FlexFlash SD cards.

Writing in Cisco HyperFlex

Cisco HyperFlex uses a distributed file system, which writes several copies of data on
several nodes at once. The number of copies that get written is determined by the
chosen replication factor, but the underlying writing system remains the same whether
you chose RF2 or RF3. Since the copies are written across the network in real time, a
potential general failure of the network connecting the nodes would result in a short
Recovery Point Objective (RPO), meaning that the last persistent write before failure
was very recent and not a lot of data has been lost.

The extremely fast and responsive network created by the Cisco Fabric Interconnects in
the topology provides all these features. Cisco Fabric Interconnects are the backbone of
the read and write process.

To achieve maximum performance, the HyperFlex system uses caching extensively, as

all the information gets checked for redundancy in the cache before being written to the
capacity (destaged).

The cache drive is not one uniform drive, but it is instead segmented into functional
parts, each dedicated to a specific use, which you can see in the following figure.

The cache drive is segmented into these functional parts:

1. Primary write cache, which holds the information of the main data copy.
2. Each of the write cache levels has a passive and active part; one is actively written
to while the other is read only.
3. Write cache level 2 and 3 are reserved for mirrored copies of the primary data to
ensure resiliency.
4. On hybrid Cisco HyperFlex deployments, most of the cache drive is dedicated to the
read log.
In all-Flash nodes, the write cache uses up the entire cache drive. Hybrid nodes also
have read cache on the same physical drive.

Write cache is separated into three levels. The first is used for blocks for which the local
CVM is chosen as the primary CVM. It is the only level of write cache, which initiates
destage when full. The second and third levels are reserved for the primary and
secondary redundancy copy of information when the primary CVM mirrors information
across the group. The same information is always cached in different levels of write
cache and always across different servers in the group.

Each level of write cache is again separated in two parts. One is the currently active and
the other is passive. Writes are performed to the active part; the passive is read only and
is most notably used when destaging data to the capacity tier.

Even though all these parts get used in the writing process, this process is a part of the
underlying infrastructure and the VM, which initiated the write, does not see any
difference in underlying storage, while the writing process gets automatically triggered
in the Cisco HyperFlex infrastructure.

Writing is performed in these steps:

1. The application and operation system initiate a write.

2. VM writes to its .vmdk file.
3. IOVisor intercepts the write and forwards it to the primary CVM for a given block.
4. Primary data is compressed, sent to write log of the primary node, then mirrored to
others.
5. Acknowledge is sent back to the VM.
6. The primary node write log is full and goes into passive mode.
7. The information stored in the primary log is deduplicated on the primary CVM.
8. The primary node initiates distribution of the deduplicated blocks across the group.
9. The contents of the primary cache are mirrored to other nodes.

First, the application and the operation system initiate a write. When a write is initiated,
the IOVisor is the primary tool of data distribution, every read or write is passed
through the IOVisor, including local writes. IOVisor chooses a CVM, which will be the
primary for an individual block. This decision is made on a per block basis. There is no
preference for the local CVM, each block is therefore chosen from the entire group,
providing the on-write distribution of blocks across the converged nodes of the group.

Once the IOVisor forwards information to the primary CVM, the CVM performs
compression on the information and commits it to the primary write log, while
simultaneously forwarding the compressed block according to the chosen replication
factor. If replication factor 2 has been chosen when installing the Cisco HyperFlex
group, the information will be sent to one additional CVM. If the replication factor of 3
was chosen, two additional copies will be forwarded.

When the primary CVM mirrors the information, the additional copies are written to
write logs on different nodes of the group. There are separate write logs for primary,
secondary, and tertiary copies. Primary write log is used on the primary CVM for the
block, the secondary is used for the first mirror, and the tertiary is used for the second
mirror. The redundant copies are only used if the primary copy is lost.

Once the data is written in the write log of an individual node, the CVM sends a write
acknowledgment to the virtual machine, confirming the write has been successfully
committed. The data is not yet written to the capacity tier at this moment, it only exists
in the cache, once the primary cache of any converged node is full, destaging is
initiated.

The following figure illustrates the first five steps of the writing process.

The destaging is the process of writing cached data to the capacity tier. The destaging is
initiated when a primary write cache of a node is full and needs to be written to the
capacity drives. The process again distributes the information to additional mirrors
according to the replication factor. It is important to note here, that the information
mirrored on the initial write does not get used in destaging. The contents of the cache
are again distributed to other nodes over the network.

The same way as with the initial write, the destaging creates secondary and tertiary
mirrors on converged nodes, except that here these copies get written to the capacity tier
and are not only created for redundancy. Once the data to be written is cached, the write
log becomes passive and is not used for writing. The data in the now passive write log
gets written to the capacity drives.

The blocks that would be written to the capacity drives, but exist on the capacity drives,
only get the metadata update in the file system, so that more than one file uses the same
block by using pointers to individual blocks comprising a file. This process is called
deduplication because it prevents duplicate data from being written several times on the
same storage.

This system of references to individual blocks of files allows the information to be

written consecutively rather than in specific locations. On destage each block is written
in sequence, and the following destage will again continue where the previous one left
off. This feature is part of the StorFS log-structured file system.

Since the data is now stored on the drives, the cache which held it before it was passed
to the capacity tier, is now available to be used again. The data in the capacity tier is
also ready to be read on demand.

The following figure illustrates steps 6–9 of the writing process.

Reading in Cisco HyperFlex

The long-term stored information in Cisco HyperFlex resides on the storage tier drives,
which are the slowest storage in the system. The speed difference between the SSD
cache and the SSD capacity tier is minor, since high performance (such as the ones used
as cache) and normal performance SSDs (such as the ones used as capacity) mainly
differ in writing capabilities, caching does not produce any significant performance
gain.

The situation is different with spinning drives, which read much more slowly than SSDs
due to seek times. Therefore, the Hybrid Cisco HyperFlex deployments also use cache
to speed up reading. There is still only one caching drive present on hybrid
deployments, but this drive is split between read and write cache. The latest and the
most commonly used information in hybrid HyperFlex is stored on the read cache.

The steps involved in reading in Cisco HyperFlex are:

1. The application or operating system initiates the read.

2. The VM reads from its .vmdk file.
3. IOVisor intercepts the request and forwards it to the CVM, which starts to locate the
data.

After it intercepts the read, the IOVisor locates data in this sequence:

A. Check the active write log, in case the information was recently written.
B. Check duplicate write logs on other servers.
C. Check RAM cache.
D. Check read log (hybrid only).
E. If the data is not located anywhere else, read from the capacity.

The read process tries to locate the requested data from all the available sources before
initiating a read from the capacity tier. This process increases the speed of the read
process and prioritizes recently written and most commonly read information.

Local information is not preferred when reading; however, in all-flash deployments, a

primary copy is used if possible. All-Flash and all-NVMe deployments use this rule
strictly and always read from the primary copy if it is available. Hybrid deployments
read from any copy depending on which copy is more readily available to speed up the
process.

There are three options for read caching in Cisco HyperFlex:

 Write-through (default): Most commonly used and most recently read data is
cached in the read log.
 Write-back (install option for VDI): Only the most commonly used data is
cached, optimizing VDI performance.
 No read caching: With All-Flash/All-NVMe nodes, because there is little
difference in read speeds between SSDs.

In normal hybrid deployments, the most frequently used information and the most
recent information are both cached in the read cache. This process improves
performance as the most often used information is available on a much faster SSD
storage, while there is a high probability that something that just got written will also be
read soon.

When installing the HyperFlex solution, you can choose a VDI optimization option in
the installer. This option will optimize the HyperFlex installation for VDI and omit the
most recently written data to be cached on write. This write-through approach allows
more of the most commonly used data to be held in cache directly. VDI applications
usually do not benefit from having access to the most recently used information
provided in the read cache.

All-Flash and All-NVMe deployments do not use any read caching and therefore do not
have any VDI optimization option available.

3.8 Describing Cisco HyperFlex Software Components

Data Optimization
Overview
In this video, we will look at an overview of data compression with Cisco HyperFlex. Comp
ression replaces duplicate information within the data being stored. And it encodes the d
ata in order to use as little space as possible.

Now, compression can significantly reduce the raw storage space needed for representing
the data. Compression is only good if the procedure can be reversed. This means you mus
t be able to retrieve the same information you put in.

Some compression procedures which use approximation, like the very common JPEG form
at file images, unfortunately, this type of compression leads to some information being lo
st, which is completely unacceptable for storage.
Cisco HyperFlex uses a lossless compression algorithm which allows for 1-to-1 information
conversion. If you remember, deduplication in Cisco HyperFlex is based on the STOR-FS
file system. It uses a framework of pointers to individual parts of a file, which can be reus
ed for the same blocks in multiple locations.

For example, if two files would require blocks with identical information, only one block g
ets written, and both files reference that block. This process eliminates duplicate informat
ion from being unnecessarily written more than once.

Data optimization, this process is tied tightly to the writing process, as it is performed in li
ne when the writing process is performed. The system is designed so that the deduplicati
on and compression are done only once by that primary controller VM.

This figure here lists the sequence of data optimization process. First, on the write reques
ts, the local IOVisor sends the information to the primary CvM for that particular block. N
ext, the CvM compresses the data, writes it to its cache drive, and mirrors the information
.
Then it sends an acknowledgment to the virtual machine confirming that the write reques
t has been successfully performed. Lastly, when the write log is full, a destage is initiated.
This means that the primary CvM performs the best-effort deduplication and writes the
information across the nodes.

Since both data compression and deduplication rely on removing duplicate data, the actu
al savings depends on the type of workloads. Workloads that have a lot of shared informa
tion across the files will definitely benefit from data optimization.

For example, compression and deduplication benefit operating system files in a Virtual De
sktop Infrastructure, or VDI. Or it benefits clones in test-and-development scenarios.
However, databases such as Oracle, they don't gain very much. That brings us to the end
of video. Thanks for watching.

Cisco HyperFlex not only provides up to 1.1 petabytes (PB) of raw storage, it also
offers compression and deduplication, which can increase the actual data stored
significantly. Compression is a process of encoding data so that it uses as little actual
storage as possible, while deduplication re-uses the same blocks for several purposes
(such as two files or two VMs).

The actual efficiency of deduplication and compression is highly dependent on the

information that is being stored on the Cisco HyperFlex storage. While some datasets
have a lot of duplicate information, others may not. If there is no repetition, no
deduplication can take place, as there is no data to be reused. Compression also relies on
duplicate information, but instead works by building the glossary of a shorthand to write
the information.

The data optimization reduces the data written to storage:

 Only unique information is written.

 Repeated information is written only once.
 Especially useful when a lot of information is duplicated (VM clones).
 Compression and deduplication are different approaches to solve the same problem.
Both processes are CPU and memory intensive, since the information has to be
analyzed, patterns extracted and rearranged as efficiently as possible. The resources
used in compression and deduplication processes are provided by the CVMs.

The compression and deduplication in Cisco HyperFlex cannot be turned off and is
performed in real time as the information gets written.

Compression
Compression mechanism tries to reduce the size of a file by removing redundant data
chunks within the file. When information is being compressed, any patterns, which are
found in the information, get written in a way that uses up the least space. Compression
can significantly reduce the raw storage space needed to represent the data.

Of course, the procedure of compression must be reversible, so that you get out the
same information as you have put in. There are also certain compression procedures,
which use approximation, like the very common .jpeg format for images. This type of
compression inevitably leads to some information being lost, which is sometimes
acceptable, but is completely unacceptable in storage, where you would have real data
loss. The Cisco HyperFlex data platform uses a lossless compression algorithm, which
allows 1:1 information conversion.

The compression replaces duplicate parts of the information and encodes them in a way
that they use up as little space as possible.
Deduplication
Deduplication in Cisco HyperFlex is based on the StorFS file system. It uses a
framework of pointers to individual parts of a file, reusing the same blocks in several
files. This process removes duplicates and the need for the same blocks to be written
more than once.

Deduplication, like compression, aims to reduce the amount of data that needs to be
written to storage. If two files require blocks with identical information, only one block
would get written, and then both files would reference that block. This further increase
storage efficiency and speedup the writing process.

The downside of referencing and non-consecutive writing is that the storage metadata
takes up considerable space. In Cisco HyperFlex, this metadata is stored in the memory
of each individual node, so that each node knows where to find a block. However, not
all duplicates on the HyperFlex storage can be found in real time, so the deduplication is
a best effort process and information existing on the storage is not strictly unique.

Deduplication works on the file level using the StorFS log structured file system by
pointing to blocks which are parts of the file on the storage.

Data Optimization Process and Actual Data Savings

The process of optimizing information is tightly tied to the writing process, as it is
performed inline as the writing process is being performed. The system is designed so
that deduplication and compression are done only once by the primary CVM. The
IOVisor determines which CVM is primary when the initiated write is intercepted,
before it is forwarded to the chosen CVM.

The process of data optimization is performed in this sequence:

1. On write, the local IOVisor sends the write to the primary CVM for that block.
2. The primary CVM compresses the data, writes it to its cache drive and mirrors it.
3. ACK is sent to the virtual machine that the write has been successfully performed.
4. Once the write log is full, a destage is initiated, where the primary CVM performs a
best effort deduplication and writes the information across nodes.

Compression is done before the information is first cached on the primary node.
However, deduplication is performed after the successful write acknowledgment has
already been sent to the VM that initiated the write. Latency is reduced in the VM
operation as the acknowledgment is received as soon as possible.

The fact that a CVM exists on every converged node, makes the compression and
deduplication scalable, as the load is always shared between all the nodes. IOVisor
chooses a CVM from all the CVMs in the group, automatically distributing the
workload. When you are scaling the Cisco HyperFlex system with additional nodes, you
are also providing additional CVMs for the IOVisor to choose from.

Since both data optimization processes rely on removing duplicate data, the actual
savings are very dependent on the type of workload.

Workloads that have a lot of shared information across files, such as the operating
system files in VDI or clones in test and development scenarios, gain more from the
compression and deduplication, while databases, such as the Oracle database, gain
almost no benefits.

Note
When you migrate data between different storage systems, the deduplication and compression
savings can change. A great example is copying a large VM that is a HyperFlex native storage
clone. While both exist on the same storage, the data optimization is 100%, when you move the
clone to another storage that clone will take up much more space, because its source VM does
not exist on that storage. Even if it does the clone was not created through the native cloning
process and does not have perfect deduplication.

3.9 Describing Cisco HyperFlex Software Components

Cisco HyperFlex vs. Other HCI
Solutions

Now let's compare Cisco HyperFlex against other Hyper-Converged Infrastructures, or HCI,
solutions. This figure here lists some of the differences between HyperFlex and other solutions.

First of all, the Cisco Fabric Interconnects used in the HyperFlex solution provide a very
responsive and highly available network. This enables a system to be completely independent
from the local storage as all the information is available through the network.

Regular vSphere installations and other hyper-converged solutions still rely on the data
locality, which is good for small deployments, but it just doesn't scale very well as it only
creates load on the local node.

In Cisco HyperFlex, all information is automatically distributed, and the read and write process
requests go across the entire network. Since the blocks of information are distributed across
the servers, HyperFlex will eventually use all of the copies when reading. This speeds up the
process and distributes the workload to all of the nodes.
This next figure lists the steps for moving a virtual machine to another server on a regular HCI.
First, the initial server on the left is overloading. So the Dynamic Resource Scheduler, or DSR,
moves the virtual machine 1 to a less-used server.

Second, it creates a suboptimal scenario because now the reads are remote since the
destination server does not have the VM data. Third, in order to reach optimal performance,
the VM data must be copied over to the new server. Now, this creates a load on the source
and the destination server as well as the connecting network.

So localized loads on specific parts of the infrastructure is what are called hotspots. HyperFlex
is designed for distributed operation from the ground up. It doesn't care where the individual
part is, just as long as there's another part to which the workload can be offloaded.

Now, in this figure, we illustrate the steps for VM migration without relying on data locality.
First, the initial server on the left is overloaded. And the DRS decides to move virtual machine
1 to a less-used server.

Second, the VMs read the stored information across the network from all the servers. And
third, the VM is transferred to the new machine, and still reads are done across the network
just as normal.
And fourth, since there is no local data required for optimal performance, there is no data
migration. It's not necessary. And this means that there is no additional load on the servers or
the network. This gets rid of all of our hotspots.

Finally, this last figure just lists the HyperFlex advantages over regular HCIs. Keep in mind,
regular HCI solutions appear to approve-- to provide, I should say, nearly all the same features.
But when, really, when you look closely, there are major differences which will have a
significant impact when the system scales to a very large size.

This includes no I/O hotspots. There's no migration of data and no situations where the VM
will not fit into the data stores. It's actually easy to scale, and no RAID groups or special
hardware are required. That brings us to the end of the video. Thanks for watching.

The Cisco HyperFlex solution is a system that spreads the workload among its units
(nodes). The reading, writing, data optimization, and the user space workloads are all
distributed between these nodes resulting in a highly distributed system.

To connect all the individual parts of the group, a very responsive and reliable network
is used. This network is provided by the Cisco Fabric Interconnects. This Cisco
HyperFlex feature enables the system to be completely independent from local storage,
as all the information is available through the network. Therefore, Cisco HyperFlex
always performs reads across the network infrastructure by default.

Since the data that VMs use is no longer local to the VM, you can say that HyperFlex
has no data locality; data is chunked-up and written in copies across the entire group
without any regard to having all VM's data available on VM's server.

The HCI with data locality is different from a HyperFlex model in these points:

1. Data local model performs data operations locally or it performs poorly.

2. Optimally, VMs in HCI with data locality do not read from non-primary copies,
creating load only on the local node.
3. On HyperFlex data operations are always done through the network.
4. Blocks of information on HyperFlex are distributed across servers. Hyperflex uses
all the blocks available for storage operations, distributing the load to all nodes.
Even though regular hyperconverged infrastructures provide data protection by creating
copies of information, the operating which is actually used is local to the system where
the VM resides, other instances of information are merely copies, which cannot be used
for a virtual machine to read, until they are local to the virtual machine.

The process of writing in these cases writes the information locally, then that
information is copied over to another node, creating a cold copy. This process may not
seem as an issue in normal operation, as the data is still protected and the workload is
still shared between the nodes, however there are some serious implications when using
this approach.

In Cisco HyperFlex, all information is automatically distributed and all the reads and
writes go across the network. The hybrid deployments of HyperFlex actually use all the
copies when reading, to speed up the read process. All-Flash and all-NVMe
deployments read from the primary blocks if they are available, since the hardware read
limitations are mitigated greatly by the use of Flash storage.

The copies on a HyperFlex all-Flash or all-NVMe system are not actually cold copies
but are just not used if it is not necessary. The VMs can exist anywhere, without having
to move any information as in regular hyperconverged deployments.

VM Migration with Data Locality

Moving a VM to another server on a regular HCI follows these steps:

1. The initial server is overloaded, and DRS moves the VM to a less used server.
2. The reads are remote since the destination server does not have the VM data, which
is suboptimal.
3. To reach optimal performance, VM data is copied over to the new server, creating a
lot of load on the source and destination servers, and the connecting network.

If the VM in the example would move to a node where information exists, there would
be no data migration. Still, the example in the figure shows only a four-node group, if
you scale the group to many nodes, there is less and less chance that the target host
would have a copy ready for VM migration.
The amount of information being moved between servers can be significant and this
causes read load on the source host, write load on the destination host and high
utilization of the network transferring data from one host to the other. This localized
load on specific parts of the infrastructure is called hotspots.

VM Migration Without Data Locality

The following figure shows the process of VM migration without data locality:

1. First the server is overloaded, and DRS decides to move the green VM to another
server.
2. The VMs read the stored information across the network from all the servers.
3. The VM is transferred to the new server—reads are still done across the network.
4. Since there is no local data required for optimal performance, there is no data
migration and no additional load on the servers or the network (hotspots).

Cisco HyperFlex is designed for distributed operation from the ground up, so it does not
care where any individual part is, as long as there is another part to which the workload
can be offloaded. If a CVM fails, that CVM is not chosen by the IOVisor; if a node
fails, the replication factor ensures that the information is still available, and the file
system can read from any of the copies, whether they are local or not, with the same
speed.

Because the file system does not care where information is when it has to provide data
to the requesting application, HyperFlex is able to avoid preferring local copies
completely. HyperFlex does not have any data stored locality, all information is
distributed in 8K blocks to all the nodes.

Cisco HyperFlex Advantages Over Other HCI Solutions

In summary, Cisco HyperFlex has these advantages over the regular HCI solutions due
to its data distribution:

 No I/O hotspots, the load of reading and writing is distributed across the system.
 No data migration when moving virtual machines manually or through automation.
 No situations where a large VM does not fit onto the datastore, even though the
group has enough space total.
 Easy to scale, since the information is automatically rebalanced when a new node is
introduced, freeing up space on all nodes.
 No RAID groups or special hardware required, since the system is composed of
standard hardware components. Hardware is uniform and easy to scale.
At first glance, the regular HCI seems to provide nearly the same features. But in actual
operation, there are major differences in the way minor differences reverberate through
many areas, which will have significant impact when the system scales to a large size.

3.10 Describing Cisco HyperFlex Software Components

Investigate Software Components of

Cisco HyperFlex
Investigate Software Components of Cisco HyperFlex
Cisco HyperFlex consists of the underlying data platform, which provides storage and
the VM environment, which runs the workload off the storage.

In ESXi-based HyperFlex, you primarily manage and monitor VM components (VM

deploy, VM start, VM stop, enable DRS, and so on) through vCenter. To manage the
storage component of Cisco HyperFlex, you would on the other hand primarily use the
HyperFlex Connect management interface.

Even though each of the interfaces is the primary tool to manage a specific part of
HyperFlex, there is integration between the two, which allows certain actions to be
performed across both management systems.

You can perform management and monitoring of storage components through either
vCenter or HyperFlex Connect. VMware vCenter allows you to monitor and manage
HyperFlex through the HyperFlex plug in. As of HyperFlex Version 4.0(2a), HyperFlex
HTML5 plug-in for VMware vCenter support was introduced on vCenter 6.5U2 and
later. HyperFlex Connect uses an HTML-based interface and allows certain VM-based
actions to be taken, such as VM power on, ReadyClones, and others.

From a day-to-day workflow point of view, the vCenter HyperFlex plug-in offers access
to most storage-based tasks, but you will need to use the HyperFlex Connect to
configure and monitor the HXDP in more detail.

Note
There are also three other interfaces for the HyperFlex storage management and
monitoring that are available to you: CLI, REST API, and Intersight. These interfaces
are beyond the scope of this lab.
Explore vCenter

Step 1

Log in to vCenter at https://vcenter.hx.lab/

Answer
Using the Firefox browser, navigate to vCenter by typing https://vcenter.hx.lab into
the URL bar. When you connect to the URL, choose LAUNCH VSPHERE CLIENT
(HTML5) and log in using student as the username and 1234QWer* as the password.

Note
The user information that you are using in this lab has been created in vCenter and
assigned a read-only role.

If you are experiencing problems with the virtual desktop being too big for your screen,
or if you experience keyboard issues, refer to the Instructions.pdf document that can be
found in the /home/student/DCIHX/Instructions/ directory. The virtual machine that
you are using runs Xubuntu 18.04 Linux distribution. You can find information about
setting up this environment on the page 26 of the Instructions.pdf document.
Step 2

Discover the vCenter hierarchy of a standard HyperFlex group.

Answer
Once in vCenter, expand the HyperFlex Lab data center and then the Standard group.

Note
There should be four different HyperFlex groups in the vCenter (one edge group, two
standard groups, and one stretched group). In this lab, you will focus on the ESXi-based
HyperFlex group with six nodes.

Choose the Summary tab of the Standard group.

A standard group can have anything between 3 and 64 servers as of HyperFlex Version
4.5.1a.

Step 3

Discover the existing group datastores.

Answer
Click the Datastores tab for the Standard group to list all the group datastores.

Notice that the group lists 6 datastores. Five of them are designated as
the Springpath datastores and one of them is named Standard Datastore. The five are
local datastores on each individual server and are not part of the HyperFlex distributed
storage. Keep also in mind that a local Springpath datastore is not present on the
compute-only node. The Standard Datastore is the only HyperFlex distributed
datastore. You can recognize it by its NFS 3 storage type. HyperFlex shares its storage
with the ESXi hypervisors as the NFS 3 file share. You should always use the
HyperFlex distributed datastore when deploying VMs on a
HyperFlex group.

Step 4

Discover CVMs.

Answer
Keep the Standard group chosen and then click the VMs tab. Type wzp in the filter
search box, which is located on the top-right side, and press Enter.

You will notice that there are 6 hosts in the Standard group and only 5 CVMs. This is
because CVMs for compute-only nodes are not needed anymore since Hyperflex release
4.0(2a). The WZP... suffix is the serial number of the server that the CVM runs on.

Choose one of the CVMs and click the Summary tab.

If not expanded, expand the VM Hardware section of the Summary tab.

Guest OS shows that the CVM is an Ubuntu Linux VM. However, a CVM is not a
regular VM but a storage appliance running on each individual server. It must not be
moved off the server or shut down while the HyperFlex storage is active.

Step 5

Investigate the virtual networking of a host.

Answer
In the Navigator menu, scroll up and choose the first server from the Standard group.
Then go to the Configure tab and click the Physical adapters option from the middle
menu.
The HyperFlex installer creates eight vNICs, four of them for each side of the fabric.
These vNICs are assigned in pairs as uplinks to vSwitches. They provide redundant
upstream connectivity to each vSwitch and connect to each of the upstream Fabric
Interconnects.

Step 6

Try to access the HyperFlex plug-in in vCenter.

Explore HyperFlex Connect

HyperFlex Connect is the main interface in which you manage the HXDP. You can use
vCenter credentials to log in.

Step 7

Log in to HyperFlex Connect

Answer
In your browser, open the URL https://hxconnect.standard.hx.lab. Log in using
credentials student / 1234QWer*, which are the same credentials that you used in
vCenter.
Note
The user that you are using in this lab has been created in vCenter and assigned a read-
only role. HyperFlex Connect allows you to log in using a vCenter user. As of HXDP
Version 4.5.1a, there are two roles that HyperFlex Connect recognizes from users
defined in vCenter—vCenter administrators are given the administrative role in
HyperFlex Connect while other uses have read-only access.

Step 8

Discover the HyperFlex Connect Dashboard.

Answer
The Dashboard pane opens immediately after you log in to HyperFlex Connect.
Group operational state should be Online, which means that the group is available. If it
is Offline, the group has lost too many nodes or the group has been shut down.

Group resiliency health should be Healthy. If the group was unhealthy, there was
recently a node or a drive failure.

The group should have six servers: five of them converged nodes and one compute-only
node. Converged nodes contribute compute and storage to the group while compute-
only nodes, which, as the name implies, contribute only compute resources to the group.

The Dashboard also lists the number of VMs in the group. This number does not
include CVMs since they are ESXi agents. At the bottom of the Dashboard, you should
see IOPS, throughput, and latency graphs for the group.

Step 9

Discover the Alarms, Events, and Activity panes of HyperFlex Connect.

Answer
Choose the Alarms pane.
You might or might not see alarms in this tab since the lab environment is live.

Choose the Events pane.

The Events tab will almost certainly list many notifications. While the Alarms tab
displays critical messages, the Events tab will list warnings and informational
notifications.

Choose the Activity pane.

The activity tab displays actions performed on the group by the administrator or users.

Step 10

Discover storage graphs within HyperFlex Connect.

Answer
Choose the Performance pane.

Notice that HyperFlex Connect shows IOPS, throughput, and latency graphs. The black
line is for writes, and the blue line is for reads. By default, graphs are shown for the last
day; you can change the view to observe historic data for the last 1 hour, 1 day, 1 week,
1 month, or 6 months. By clicking the date range, you can also specify a custom interval
for the graphs.

Click Cluster to show individual group components.

You can choose a group component to display its detailed performance graph, which
follows the same color coding as the overall performance graph.

Step 11

Discover the Replication pane within HyperFlex Connect.

Answer
Choose the Replication pane.
The Replication tab is reserved for HyperFlex native replication. Since replication is
not configured on this group, the tab is empty.

Step 12

Discover the System Information pane in HyperFlex Connect.

Answer
Choose the System Information pane.

The System Overview subtab provides several details:

 Status of the group (online in the example above)

 Registered vCenter (https://vcenter.hx.lab in the example above)
 Uptime (33 days, 3 hours, 6 minutes, 19 seconds in example above)
 Hypervisor (6.7.0-17167734in the example above)
 HXDP version (4.5.1a in the example above)
 Total usable capacity of the group before deduplication (8.03TB in the example
above)
 Available capacity (7.38TB in the example above)
 Data Replication Factor (3 in the example above)
 DNS servers (192.168.10.40 in the example above)
 NTP servers (192.168.10.253 in the example above)

Each server within the group is represented with a square. Within that square, you can
find this information (examples given for the first server in the list):

 The type of server (HXAF220C-M5SX)

 Disks (1 caching, 6 persistent)
 HXDP version (4.5(1a))
 Hypervisor status (online)
 Hypervisor address (192.168.10.111)

Click the Disk View Legend.

You can use the Disk Legend to get familiar with different types of drives that are used
by HyperFlex nodes and the states they are in.

Click the Nodes subtab.

You should see four ESXi nodes. All of them should have a hypervisor address, be
online, have server model, and version of HXDP listed.

Click the Disks subtab.

The Disks subtab should list persistent and cache disks in all servers in the group. Disks
that are not shown here are housekeeping and boot disks.

The capacity of the disks is displayed in binary rather than the decimal base. In the
example above, you see six capacity disks per server, each having a capacity of 894.3
GB in the binary base, equivalent to 960 GB in the decimal base.

Step 13
Discover the Datastore view of HyperFlex Connect.

Answer
Choose the Datastores tab.

You should see one datastore present. The datastore has the following characteristics:

 The name of the datastore is Standard Datastore.

 This datastore is Mounted to all servers in the group.
 Datastore Unpaired because the group is not configured for HyperFlex native
replication to/from another group.
 Status of the group is Normal, which means there are no health warnings about the
health of the storage.
 Provisioned size of datastore is 500 TB.
 Used storage: 369.84 GB.
 Free storage: 499.65 TB.

Note
As you can see, the provisioned size of the datastore is bigger than the total group
capacity. This is possible because HyperFlex is always using compression and
deduplication which allows you to store more data than there is total group capacity.

Step 14

Discover the iSCSI view of HyperFlex Connect.

Answer
Choose the iSCSI pane.
This tab is used for configuration of iSCSI network, LUNs, targets, and initiators. Since
iSCSI network is not configured, the tab is empty.

Step 15

Discover the Virtual Machines view of HyperFlex Connect.

Answer
Choose the Virtual Machines tab.
VMs listed in this view are the workload machines running on the group. You can see
how much storage is provisioned and used for a particular VM, a number of VM
snapshots, and its protection status. HyperFlex Connect will not list CVMs.

Step 16

Discover the Kubernetes view of HyperFlex Connect.

Answer
Choose the Kubernetes pane.

This tab is reserved for Kubernetes group configuration. This tab is also empty, since
Kubernetes group and iSCSI network (required for Kubernetes group) are not
configured.

Note
Since you are accessing HyperFlex Connect using a read-only user account, HyperFlex
will give you a limited view. If you would log in using an administrative account, you
would also have access to the Upgrade tab (for upgrades) and the Web CLI tab (for
running CLI commands on CVMs directly from HyperFlex Connect).
4.1 Describing Cisco HyperFlex Hardware Components

Introduction
Cisco HXDP runs on the Cisco UCS server platform, which is a tried-and-tested
solution, providing the underlying hardware, centralized management, and networking.
Cisco UCS servers utilize the top-notch performance of the latest Intel Xeon processors.

Many hardware features of the Cisco HyperFlex Series stem from the C-Series rack
server platform and there is a significant overlap of the features set; however, the
standard Cisco UCS C-Series do not support the Cisco HXDP. Standard Cisco UCS
servers can however be used as compute nodes in HyperFlex deployments.

4.2 Describing Cisco HyperFlex Hardware Components

Introducing Cisco HyperFlex Servers

Now in this video, we're going to introduce Cisco HyperFlex Servers and Storage Technologies.
Cisco HyperFlex supports the latest two versions of Cisco UCS servers, if you'll remember, the
M4 and the M5.
Now, Cisco stopped selling the M4 servers in February of 2019, and the M5 servers are the pat
h forward as for the space support and better processors, memory, et
cetera, et cetera. But the M4s are still supported if you have them.

The figure here lists the form factor for the different models, as well as the number of support
ed drives. This includes both
the spinning capacity or flash capacity drives. If you remember, you can always integrate comp
ute-only nodes to increase horsepower if desired without adding additional drives if you don't
need them.

When deciding what to purchase for your deployment, you will need to decide on the server ty
pe, the generation, storage type, and loadup. For new deployments, you use the M5 servers. Y
ou can choose the HX240
servers to maximize the storage pool or choose the HX220 servers to ensure high compute po
wer relative to storage.
For environments where storage performance is crucial, use all NVMe nodes. Now, NVMe stan
ds for Non-Volatile Memory Express. And it's a transfer protocol specification that defines the f
eatures for memory access.

Now, this figure here lists the guidelines when mixing HyperFlex servers. If you try to install a C
isco HyperFlex cluster using an unsupported mix of servers, such as different form factors, amo
unt of storage, or types of drives, the installation will not go through, and the validation for the
installation is going to fail.

Full data distribution requires that all of the servers deployed in a single cluster are the same. F
rom the standpoint of storage performance, your cluster is only as fast as the weakest server i
n the cluster-- weakest link in the chain. Right?

Now, you can expand an M4 cluster with M5 nodes, but you can't go the other way. You can't
go backwards.

The Cisco UCS M5 generations of servers features the latest hardware available on the platfor
m. And as I said before, the M5 servers supersede the M4 generation. That was the first to sup
port Cisco HyperFlex and offer some very distinct advantages over the M4. For example, the M
4 now support Microsoft Hyper-V.

The table in this figure describes high-level differences between the M4 and
the M5 server generations. Once again, this includes form factor, supported CPUs, memory, an
d so on. Notice how the memory capacity doubled from 1.5 terabytes to 3 terabytes. Also, the
number of maximum supported Graphic Processing Units, or GPUs, on the 240 jumped from o
ne to six.

The Cisco HyperFlex solution features a wide range of storage components. Their properties di
ffer based on the manufacturing technology and applications. Newer technologies have been a
dded to the HyperFlex product line to meet the performance expectations of very demanding
environments.

The most common SSDs consists of memory cells that are made of floating-gate transistors. Th
ese regular SSDs are also known as NAND flash SSD and are the preferred option because of th
eir faster write and erase times.
The Non-Volatile Memory Express, remember, NVMe's, is an optional high-performance and sc
alable transfer protocol interface that's designed to work with current and next generation MV
e technologies.

NVMe is defined to enable host software to communicate with non-volatile memory over PCIe'
s. It has been designed from the ground up to offer a large number of queues, streamlined sim
ple command sets, and advanced features for future deployments and development.

NVMe was designed to improve the performance of flash storage many times over in ways tha
t were not possible with the traditional technologies for many operating systems.

Finally, Intel Optane technology can be used to replace or extend the DRAM memory pool and
provide much faster read-write I/O operations. This performance boost was implemented with
HyperFlex version 3.5.1. It also improves better endurance over drives.

Now, due to its competitive cost and excellent I/O performance, Intel Optane cache SSD is now
recommended-- is the recommended choice for all new developments. Be aware, however, th
 at you cannot mix cache types within a given cluster. That brings us to the end of this video. Th
anks for watching.

As the power of computers keeps growing, new updated versions of the hardware are
introduced. Different generations of Cisco UCS servers are indicated by "M," followed
by the generation number. Cisco HyperFlex supports the latest two versions: the older
M4 and the newer M5.

Cisco made end-of-life announcements for Cisco UCS M4 Series Servers in August
2018 (https://www.cisco.com/c/en/us/products/collateral/hyperconverged-
infrastructure/hyperflex-hx-series/eos-eol-notice-c51-741237.html). Cisco stopped
selling Cisco UCS M4 servers in February 2019. Cisco UCS M5 servers are the only
orderable generation as of May 2021, with the M6 generation to be released later in the
year.

Note
In this section, references to Cisco UCS M4 Series Servers are only for context and
compatibility discussion.

Different servers have different features, which determine their utility:

 Generation: M4 or M5

 M5 servers support better processors with more memory, NVMe and Optane
drives, faster networking, and more GPU options.

 Form factor: HX220c (1 rack unit [RU]) or HX240c (2 RU)

 HX220c supports up to 8 SFF drives and HX240c supports up to 23.
 Both HX220c M5 and HX240c M5 have support for GPUs.
 HX240c M5 Chassis has a short depth (SD) option for Edge deployments.

 Storage type: Spinning capacity drives or Flash/NVMe capacity drives

The converged nodes can only be HyperFlex rack servers. In addition to the converged
nodes, you can use 3rd, 4th and 5th generation Cisco UCS B and C series servers as
compute nodes. It is recommended to use the same generation of servers in a group to
get the most out of the system.

Choosing Cisco HyperFlex Server Platform

When you are designing your HyperFlex deployment, you will need to define which
server type, generation, storage type, and load out to purchase for your deployment.

When you evaluate the server configuration most appropriate for your environment,
consider these general guidelines:
  Choose HX240c M5 servers to maximize the storage pool. The variant with Large
Form Factor (LFF) drives gives you the largest capacity
 Choose HX220c M5 servers to ensure high compute power (relative to storage).
 Choose All-Flash platforms to increase I/O performance.
 For workloads that require top-tier storage performance, use All-NVMe option.

Testing shows that an All-NVMe storage offers these performance benefits over All-
Flash:

 With general workloads:

 Between 28 percent and 57 percent higher IOPS

 Between 22 percent and 40 percent lower latency

 With complex mixed workloads:

 Between 66 percent and 90 percent higher IOPS

 Between 31 percent and 39 percent lower latency

 With databases:

 Between 57 percent and 71 percent higher IOPS

 Between 34 percent and 37 percent lower latency
For more information on the above numbers, read the ESG Technical Validation:
Mission-critical Hyperconverged Workload Performance Testing on Cisco HyperFlex
All NVMe with Intel Optane DC SSD at https://www.esg-global.com/validation/esg-
technical-validation-mission-critical-hyperconverged-workload-performance-testing-on-
cisco-hyperflex-all-nvme-with-intel-optane-dc-ssd.

Note
NVMe is a transfer protocol specification that defines features of data communication.
It uses the PCIe interface and could be combined with different form factors for
physical connectivity, such as M.2.
Mixing Cisco HyperFlex Servers
Cisco HyperFlex systems spread data equally across all converged nodes by design. Full
data distribution requires that the servers deployed in a single group use a uniform
storage configuration.

The CPU and RAM configuration of a server is not as strict, but mixing different
component generations additional considerations. The most notable component is the
CPU, which later also defines the memory that can be used in the server.

When mixing HyperFlex servers, follow these guidelines:

 You cannot combine:

 HX220c and HX240c chassis

 Servers with different drive sizes
 Servers with different number of drives
 Hybrid, All-Flash, and All-NVMe capacity drives
 Nodes with self-encrypting and nodes with standard drives

 You can expand an M4 server group with M5 nodes:

 Enable Enhanced vMotion Compatibility (EVC)

 Adding M4 servers to M5 group is not supported
 Initial mixed generation installation in not supported

If you try to install a Cisco HyperFlex deployment using an unsupported mix of servers
(such as different form factors, amount of storage, or types of capacity/cache drives),
the installation will fail during the validation stage. Both Cisco Intersight and on-
premises installer check the hardware setup at the beginning of the installation process.

Note
EVC (Enhanced vMotion Compatibility) is a setting in vSphere that reconfigures the
CPU instruction set to a common denominator for the server group. This effectively
causes newer CPUs not to use functionalities that older CPUs in the group do not
support. This enables full VM compatibility across all the servers, but decreases the
performance of the newer CPUs.

Cisco HyperFlex M5 Servers at a Glance

The Cisco UCS M5 generation of servers features the latest hardware available on the
platform. It also offers some distinct advantages over the previous M4, which enables
the Cisco HyperFlex to support more functionalities on the M5 generation, like
Microsoft Hyper-V hypervisor support, and more than one GPU.
HyperFlex M5 servers include these important features as of
HXDP 4.5(1a):

 HDD or SSD or NVMe SSD drives for capacity storage

o Self-encrypting drive options are available.

 SSD cache drive (serial-attached SCSI [SAS], NVMe, or NVMe

Optane)
 M.2 SATA drives as boot drives for the hypervisor (ESXi or
Hyper-V)
 All nodes use Intel Xeon Scalable CPUs and DDR4 memory.

M5 servers supersede the M4 generation of Cisco UCS servers that was the first to
support Cisco HyperFlex. M4 nodes used the Intel Xeon processor E5-2600 v4 family
CPU. M4 servers did not support M.2 drives for the hypervisor boot. The reliance of the
M4 generation on SD cards prevents Hyper-V installations on M4 servers and Hyper-V
is not supported on the M4 server generation.

With the HyperFlex version 4.5(1a), an extra RAID option is available for the
hypervisor boot drive, adding additional resiliency to the HyperFlex hypervisor.

Comparing Server Generations at A Glance

The table describes high-level differences between the M4 and M5 server generations.
Usually, M5 generation surpasses the capabilities of M4 by a factor of 2.

Feature M4 M5

Form factor 220 / 240 220 / 240

Supported configurations Hybrid / All-Flash Hybrid / All-Flash / All-NVMe

CPU Intel Xeon E5 Intel Xeon Scalable CPUs

Max memory 1.5 TB 3 TB

Supported number of GPUs 1 in HX240c, 0 in HX220c up to 6 in HX240c, up to 2 in HX220c

 Feature M4 M5

Boot medium SD card M.2 SSD (RAID option)

Most commonly 2x 10 Gbps 2x 40 Gbps (VIC 1387) 2-4x 10/25 Gbps

Network interface speed
(VIC 1227) (VIC 1457)

M4 HyperFlex deployments use 2nd generation Fabric

Interconnect devices, while the M5 generation uses both third
and fourth generation of Fabric Interconnects, depending on the
connectivity configuration of the chosen Cisco VIC cards.

4.3 Describing Cisco HyperFlex Hardware Components

Storage Technologies in Cisco HyperFlex

The Cisco HyperFlex solution features a wide range of storage components. Depending
on their purpose, they are made according to specific standards, which can describe the
communication and form factors. Their properties will differ based on the
manufacturing technology and application.

These various specifications describe storage operation:

 Interfaces: PCIe, SAS, and SATA

 Transfer protocols: NVMe
 Disk form factors: small (SFF), large (LFF), and M.2 (RAID)
 Storage disks for capacity:
o Hard (magnetic) disks with 10k rotation per minute (rpm)
o NAND SSD: 1 WPD

 Storage disks for cache:

o SAS SSD: 10 WPD endurance

o NVMe SSD: 10 WPD endurance, higher IOPS, and lower latency than SAS
SSD
o Intel Optane SSD: 30 WPD endurance, higher IOPS, and lower latency than
regular NVMe SSD

Traditional technologies, such as SAS and serial ATA (SATA), are still available as
optional features when you order Cisco HyperFlex servers.

The most common SSDs consist of memory cells that are made of floating-gate
transistors. These regular SSDs are also known as NAND Flash SSD. The term NAND,
along with NOR, describes the logic of storing the information in the transistors. NAND
and NOR are mutually exclusive. NAND is usually the preferred option because of
faster write and erase times.

Different types of SSD storage are used in HyperFlex as capacity and/or cache drives,
depending on their performance and endurance.

Newer technologies, such as the NVMe and Intel Optane SSD, have been added to the
HyperFlex product line to meet the performance expectations of the most demanding
environments.

Note
Writes per day (WPD) is an endurance rating that manufacturers of SSD drives provide.
SSDs have a limited number of write/erase cycles before the oxide layer within the
storage device's floating-gate transistor breaks down, which is commonly referred to as
Flash wear out. WPD varies between different disks. If the drive is declared to have 3
WPD, then you are guaranteed to be able to rewrite each sector on disk 3 times per day
for the warranted lifespan of the drive (usually 5 years). Generally speaking, you get
better endurance if you pay more. HyperFlex cache layer SSDs perform a lot of writing
and erasing because they consolidate I/O before it is de-staged to the capacity layer.
Therefore, higher endurance is beneficial for the lifetime of cache SSDs in write-
intensive environments.

Transfer Protocol: NVMe

The traditional data center storage used the Serial Attached SCSI (SAS) storage for
communication between the CPU and the drives. The communication was inherently
limited by its serial design, which prevented parallel communication with the storage.
This was not a major issue on spinning drives, because their speed was limited, while
their reading was still mostly linear due to how the drives physically read and written
the information using a read/write head.

Because earlier technologies (such as SAS and SATA) were architected for hard disk
drives, they are unable to take full advantage of the SSD potential that can parallelize
reading, so a new architecture was needed that could take advantage of the SSD storage.

NVMe was introduced to provide the features needed by next-generation storage

devices. NVMe standard uses a much larger queue depth and allows multiple CPU cores
to have access to different queues. This allows better distribution of data processing
between CPU cores, which is especially beneficial in the data center where CPUs with
many cores are standard.

NVMe is a transfer protocol with these features:

 Uses the PCIe interface.

 Does not need an I/O controller and communicates directly with the CPU.
 Uses many more and deeper queues for command submission than SAS/SATA.
 Streamlines the command set to generate less CPU overhead.
 In HyperFlex:

o NVMe (regular or Optane) disks can be used for cache in the All-Flash
version.
 o Regular NVMe is ca pacity and Optane is cache in All-NVMe version.

NVMe has been designed from the ground up to offer:

 Large number of queues (up to 64,000) for command submission and completion.
Every CPU core can have its own independent I/O queue pair without any need for
locking and contention.
 Each queue is much deeper (up to 64,000 commands) compared to SAS or SATA.
The high parallelism of the NVMe interface is designed to handle ultra-fast CPU
and storage.
 Streamlined and simple command set, resulting in less CPU overhead needed to
process I/O requests, providing more IOPS per CPU instruction cycle, and lower I/O
latency in the host software stack.
 Advanced features for future software and hardware development.

Due to these and multiple other enhancements of the storage I/O stack, NVMe has
helped to improve the performance of Flash storage many times over in ways that were
not possible with the traditional technologies.

NVMe drives are especially good at mixed or write-heavy workloads. This makes them
a very good choice for database workloads that also benefit from the low storage latency
of NVMe storage, which can further be improved by using Intel Optane as a cache.

Storage Media: Intel NVMe Optane SSD Cache

For years, there was an enormous void in memory and storage technologies, forcing
massive trade-offs between placing data on cheap, but extremely slow hard disk drives
(in cold storage)—or placing it in very fast, but very expensive and volatile DRAM (or
hot storage).

Performance in the data center increases as technologies get closer to the processor—
and so does the cost per gigabyte of capacity. But now, recent Intel innovations are
rapidly filling the gaps, delivering an unprecedented balance of performance, cost, and
capacity.

Intel's P4800X is the world's most responsive data center SSD. It brings an industry-
leading combination of low latency, outstanding quality of service (QoS), and high
endurance, delivering the performance needed for both memory and storage workloads.

 Intel Optane SSDs provide these features:

 o Offers faster read/write I/O than other SSDs:

 Performance boost is implemented since HXDP 3.5(1a).

o Better endurance than SAS or regular NVMe SSDs.

o It is the recommended choice for new deployments.
o It is based on the 3-D XPoint project:

 st by changing the resistance across bits in large batches.

o BooIn HyperFlex, Intel Optane can be used as cache in All-Flash systems and is
the only choice for cache in All-NVMe systems.

Intel Optane technology is based on chalcogenide glass materials, which ensure a better
endurance than the Flash technology based on the floating-gate transistors.

Starting in HyperFlex 3.5, the Intel Optane SSD cache offers a significant read and
write speed boost compared to Flash cache.

Due to its competitive cost and excellent I/O performance, Intel Optane cache SSDs are
the recommended option for deployment types that have Intel Optane listed as a
compatible cache drive. As of HXDP 4.5(1a) all deployment types are supported except
for hybrid deployments and deployments with self-encrypting disks (SEDs).

4.4 Describing Cisco HyperFlex Hardware Components

Storage Components of Cisco HyperFlex

Converged Nodes
Now let's review the storage components of Cisco HyperFlex converged nodes. Cisco
HyperFlex converged nodes differ from the compute-only nodes based on their internal
storage resources.

Now, the storage resources include the capacity drives and cache drives which contribute to
the overall storage pool. The figure here illustrates a high-level diagram of hardware
components of HyperFlex servers.

Capacity drive types depend on the type of converged node, the hybrid, all-flash, or all-NVMes.
Capacity drives belong to the storage pool, whereas the cache drives are used to temporarily s
tore data and accelerate the I/O operation.
Capacity drives in a Cisco HyperFlex environment are the internal drives installed on the
converged nodes. And they can be equipped with different types of drives in various form
factors and support various technologies.

The capacity drives are attached to the PCIe bus. And as I remember before, as I mentioned
this before, they belong to the storage pool. Remote storage resources are not considered or
supported as capacity drives. These include drives accessed via NFS, iSCSI, Fiber Channel, or
Fiber Channel over Ethernet.

The form factor of a disk drive actually describes its physical compatibility with disk bays in the
servers and the storage arrays. Most of the HyperFlex servers use the Small Form Factor, or
SFF, drives. For example, the HX240 model can house up to 23 capacity drives with a maximum
of 3.8 terabytes per drive.

A newer option on the XX240 hybrid model uses Large Form Factor, or LFF, drives. The use of t
he LFF drives in the HX240 offers a very high-density solution at a lower cost.
This next figure here illustrates the front views of the three Cisco HyperFlex server types. The
HX220c supports hybrid, all-flash, or NVMe's. The first two slots are used by housekeeping and
cache disks.

Now the HX240c-M5SX support hybrid and all-flash. Now here, only the first slot is used for the
housekeeping drive. Lastly, the HX240c-M5L, well, that one only supports—that supports the
hybrid only.

Now, depending on the server type, you will find the SSD cache drives either in the front or the
rear panel. Now, on the HX220 node, the cache drive is installed in slot 2 on the front of the
chassis, whereas on the HX240 nodes, the cache drive is installed on one of the rear bay slots.
This includes the LFF chassis on the HX240c-M5L.

Now, you can have different types of cache drive depending on the platform. You must follow
the guidelines listed in the figure here when selecting drives for each node within a cluster.
Now, these rules here will guarantee that data is spread equally across all levels in the nodes
within the cluster. And it also guarantees that each data chunk can be read and written using
the same SLA format. This includes the same type of size of capacity drives, the same number
of capacity drives, and the same type of cache drive.

Performance will definitely be affected if you have an unequal number of types of disk across t
he nodes in the cluster. I mean, HyperFlex will be slowed to the server with the slowest
hardware. This is the reason why Cisco only supports homogeneous clusters.

The M2 dedicated boot drive is directly plugged into an M.2 slot on the motherboard. And it all
ows for the hypervisor software to boot. It is used for the system files of the Controller Virtual
Machine, or Controller VM.

Now, the M.2 connector has various notches that describe its purpose and the module capabili
ties. This feature prevents undesired plugging of M.2 modules in an incompatible slot. In all
Cisco HyperFlex M5 servers, the servers come as an internal expansion card, which provides
much better write endurance and throughput.
Finally, there's a few more things to remember about the 240 gigabyte SSD housekeeping in
system drive. In addition to the boot drive, there's also a second drive used by the controller
VM. Let's see. But various system operations, the drive that are going to be used will
include saving and restoring program state, removing unneeded software, executing disk
maintenance utilities, garbage collecting for
freeing up local memory, and, of course, file backups.

And one more time on the HX240C.M5L, the system drive is installed in one of the two rear SFF
drive slots. For all other servers, the system/log drive is installed in one of the first front-drive
slots. That brings us to the end of this video. Thanks for watching.

Cisco HyperFlex converged nodes differ from compute-only nodes by the internal storage
resources that they contribute to the overall storage pool. These storage resources include the
capacity drives and cache drives.
The figure illustrates the hardware components of HyperFlex servers at a high level.
The capacity drives can be hard disks (hybrid nodes), SAS SSDs (All-Flash nodes), or NVMe
SSDs (All-NVMe nodes). Capacity drives are assigned to the storage pool that defines the
storage type of a HyperFlex node.
Cache drives can be SAS SSD, NVMe SSD, or Intel Optane SSD. The cache is used to
temporarily store data before it is written to the capacity drives in a destage process. This
accelerates the writing process by using better performing storage for the initial write.

Types of Capacity Drives

Capacity drives in a Cisco HyperFlex environment are internal drives installed in the converged
servers. Capacity drives are attached to the PCIe bus.

Note
Remote storage resources accessed through NFS, iSCSI, Fibre Channel, or FCoE are not
considered or supported as capacity drives.

HyperFlex M5 servers can be equipped with different types of capacity drives:

 Various form factors
o SFF in all HX220c M5SX and HX240c M5SX

o LFF in HX240c M5L

 Various technologies
o Spinning drives in hybrid nodes

o SAS NAND Flash SSDs in All-Flash models

o NVMe NAND Flash SSDs in All-NVMe platforms

Small and Large Form Factor Capacity Drives
The form factor of a disk drive describes its physical compatibility within disk bays in servers
and storage arrays. The most common form factors found in enterprise solutions are SFF and
LFF.

Note

SFF corresponds to 2.5-inch, and LFF to 3.5-inch drive diameter.

Different storage options offer different advantages. While the HX240c M5SX model can house
up to 23 capacity drives with a maximum of 7.8 TB per drive, offering most storage
performance and high capacity, the small form factor HX220c M5SX has less storage space,
with up to 8 capacity minimum drive size of 1.2 TB, but has the same compute power and RAM
capacity as the HX240c models.

Note

There is also a HX240c M5SD chassis available for HyperFlex Edge. The SD chassis only has 4
SFF capacity drive slots and offers the least storage. It is not covered here to avoid confusion
and will be fully covered with HyperFlex Edge.

An LFF drive option was introduced on the Cisco HyperFlex Version 3.0 with the LFF HX240c
M5L Hybrid Model. This storage type offers the most storage density of the HyperFlex lineup
but also the lowest storage performance, as seek times increase with the drive size. The use of
the LFF drives in the HX240 M5L chassis offers a high-density solution at lower cost.
To mitigate the effect of the slower storage tier, LFF servers feature a higher-capacity caching
drive to better optimize the larger storage.

Identifying Capacity Drives

In all server types, the capacity drives are installed on the front.
Capacity drives are installed in:
 All HX220c M5 (Hybrid/All-Flash/All-NVMe)
o Front slots 3–10

o First two slots used by housekeeping and cache drives

 HX240c M5 (Hybrid/All-Flash/All-NVMe)
o Front slots 2–24

o First slot used by housekeeping drive

 HX240c LFF M5 (hybrid only):

o Front slots 1–12

The figure illustrates the front views of the three Cisco HyperFlex server types (HX220c M5,
HX240c M5, and HX240c LFF M5), and the slots in which capacity drives are installed,
independently of the capacity drive type (HDD, SSD, and NVMe SSD).

Capacity Drive Options

The following table describes the maximum number, type, and size of the capacity drives that
you can install in every converged node. SED options are also listed.
Depending on the platform, you can include different types of capacity drives: HDD, SAS SSD,
or NVMe SSD. All but NVMe (regular and Optane) are available in SED variation.

Capacity Drive Options

Drive
Server Capacity Drive Type
s

HX220c M5SX Hybrid 2.4-TB, 1.8-TB, or 1.2-TB SFF HDDs

HX220c M5 Hybrid with SED 2.4-TB, 1.2-TB SED SFF HDDs

HX220c M5 All-Flash 6–8 7.6-TB, 3.8-TB, 1.9 TB or 960-GB SSDs

HX220c M5 All-Flash with SED 3.8-TB, 960-GB, or 800-GB SED SFF SSDs

HX220c M5 All-NVMe 8-TB, 4-TB or 1-TB NVMe SSD

HX240c M5 Hybrid 2.4-TB, 1.8-TB, or 1.2-TB SFF HDDs

HX240c M5 Hybrid with SED 2.4-TB or 1.2-TB SED SFF HDDs

6–23
HX240c M5 All-Flash 7.6-TB, 3.8-TB, 1.9 TB or 960-GB SSDs

HX240c M5 All-Flash with SED 3.8-TB, 960-GB, or 800-GB SED SFF SSDs

HX240c M5L Hybrid 6–12 12-TB, 8-TB, or 6-TB LFF HDDs

Cisco HyperFlex servers support these capacity drive options:

 Cisco HyperFlex HX220c M5 hybrid node: This small footprint Cisco HyperFlex
hybrid model contains a minimum of six, and up to eight 2.4 TB or 1.8 TB or 1.2 TB
HDDs that contribute to storage capacity. For configurations requiring self-encrypting
drives, the capacity disks are 1.2TB SED HDDs.
 Cisco HyperFlex HX220c M5 All-Flash node: This small footprint Cisco HyperFlex
All-NVMe model contains six to eight 960 GB, 3.8 TB, or 7.6 TB SATA SSD drives
for storage capacity. For configurations requiring self-encrypting drives, the capacity
disks are either 800 GB, 960 GB, or 3.8 TB SED SSDs.
  Cisco HyperFlex HX220c M5 All-NVMe node: This small footprint Cisco HyperFlex
All-Flash model contains six to eight 8 TB, 4 TB, or 1 TB NVMe SSD drives. All-
NVMe nodes are supported starting with HXDP 4.0(1a).
 Cisco HyperFlex HX240c M5 hybrid node: This capacity optimized Cisco HyperFlex
hybrid model contains a minimum of six and up to twenty-three 1.2 TB or 1.8 TB or 2.4
TB SAS SFF hard disk drives that contribute to system storage. For configurations
requiring self-encrypting drives, the capacity disks are replaced with 1.2 TB SED
HDDs.
 Cisco HyperFlex HX240c M5 All-Flash node: This capacity-optimized Cisco
HyperFlex All-Flash model contains six to twenty-three 960 GB, 3.8 TB, or 7.6 TB
SATA SSD drives for storage capacity. For configurations requiring self-encrypting
drives, the capacity disks are replaced with either 800 GB, 960 GB, or 3.8 TB SED
SSDs.
 Cisco HyperFlex HX240c M5L hybrid node: This density optimized Cisco
HyperFlex hybrid model contains a minimum of six and up to twelve 6 TB or 8 TB or
12 TB SAS LFF HDD that contribute to system storage. Large form factor nodes cannot
be configured with self-encrypting disks.

Identifying Cache Drives

Depending on the server type, you will find the SSD cache drives either in the front or the rear
panel.
Cache drives are installed in these locations:
 Second front slot in the HX220c M5 nodes
 One of the rear bay slots in the HX240c M5 nodes
o Including the LFF chassis (HX240c M5L)

Cache Drive Options

Depending on the platform, you can choose different types of cache drives: SAS or regular
NVMe, or Optane. SAS SSDs exist in SED variation.
 Cache Drive Options

Server Cache Drives

All HX220c/HX240c M5 hybrid configurations SAS SSD

All HX220c/HX240c M5 All-Flash configurations Optane NVMe SSD or NVMe SSD or SAS SSD

All HX220c M5 and HX240c M5 configurations with

SAS SED SSD
SED

HX220c M5 All-NVMe Optane NVMe SSD

Cisco HyperFlex servers support these caching drive options:

 Cisco HyperFlex HX220c M5SX hybrid node: This Cisco HyperFlex hybrid model
contains a 400-GB, 480-GB, or 800-GB SSD caching drive. For configurations
requiring self-encrypting drives, the caching SSD is the type 800-GB SAS SED SSD.

Note
Either a 400-GB, 480-GB, or 800-GB caching SAS SSD may be chosen. There is no
performance, capacity, or scalability benefit in choosing the larger disk. It is just a matter of
disk supply availability.

 Cisco HyperFlex HX220c M5 All-Flash node: This Cisco HyperFlex All-Flash model
contains either a single 375-GB Optane NVMe SSD, a 1.6-TB NVMe SSD, or 400-GB
SAS SSD cache drive. For configurations requiring self-encrypting drives, the caching
SSD is an 800-GB SAS SED SSD.
 Cisco HyperFlex HX220c M5 All-NVMe node: This Cisco HyperFlex All-NVMe
model contains a single 375-GB Optane NVMe SSD.
 Cisco HyperFlex HX240c M5 hybrid node: This Cisco HyperFlex hybrid model
contains a single 1.6-TB SSD caching drive installed in a rear slot. For configurations
requiring self-encrypting drives, the caching SSD is replaced with a 1.6-TB SAS SED
SSD.
 Cisco HyperFlex HX240c M5 All-Flash node: This Cisco HyperFlex All-Flash model
contains either a single 375-GB Optane NVMe SSD, 1.6-TB NVMe SSD, or 400-GB
SAS SSD cache drive installed in a rear slot. For configurations requiring self-
encrypting drives, the caching SSD is an 800-GB SAS SED SSD.
 Cisco HyperFlex HX240c M5L hybrid node: This Cisco HyperFlex hybrid model
contains a single 3.2-TB SSD caching drive, installed in the rear hot-swappable slots.
Large form factor nodes cannot be configured with self-encrypting disks.

Drive Selection Rules

Similar to the limitations in mixing different nodes in a group, you must follow these guidelines
when choosing drives for each node within a group:
Every node in Cisco HyperFlex group must be equipped with the following:
 The same type and size of capacity drives
o SFF HDD: 1.2, 1.8, 2.4 TB

o LFF HDD: 6, 8, or 12 TB

o SSD: 960 GB, 1.9, 3.8 or 7.6 TB

o NVMe SSD: 1, 4 or 8 TB

 The same number of capacity drives

o 6–8 in HX220 M5

o 6–23 in HX240c M5SX

o 6–12 in HX240c M5L

 The same type of cache drive

o SAS SSD, NVMe SSD, or NVMe Optane SSD

o Cache disk size is not very important, but type is

These rules guarantee that the data is spread equally across all disks and all nodes within the
group and that each data chunk can be read and written using the same SLA level. If you would
have an unequal number and/or types of disks across nodes in the group, HyperFlex would
always be slowed down to the server with the slowest hardware. This reason is why Cisco only
supports homogenous server groups.
When ordering, you have multiple choices for cache disks on some servers. For example, on the
HX240 M5 All-Flash your options will be 375-GB Optane NVMe SSD, 1.6-TB NVMe SSD, or
400-GB SAS SSD. You should ignore the disk sizes and focus on the type because whichever
disk you pick, the same amount of that disk will be utilized. In the case of All-Flash HX240 M5,
that number is approximately 360 GB. The following advice applies to any other system; focus
on the type of disk and ignore the capacity.

M.2 Boot Drive SSD

The boot drive is directly plugged in a M.2 slot on the motherboard expander. The M.2 boot
drive provides storage to boot the hypervisor. Both ESXi and Windows are installed on the boot
drive in HyperFlex deployments.
A dedicated boot drive provides these functions:
 Allows hypervisor boot for both VMware vSphere and Microsoft Hyper-V.
 You can order two M.2 drives to set up a RAID 1 redundancy configuration.
 In all Cisco HyperFlex M5, servers come with an internal expansion card.
o Allows attaching 240-GB M.2 form factor SSD

o A single drive and dual RAID expander option is available since HXDP
v4.5(1a)
 Provides much better write endurance and throughput than SD cards in M4.
M.2 form factor specifies the interface for internal mounting of cards and connector. More
flexible than its predecessor, the mini-SATA (mSATA) standard, which uses the PCIe Mini
Card, the M.2 specification permits variable module widths and lengths.
The RAID option for the M.2 hypervisor boot drive provides additional system resiliency. If
one of the drives fails, the hypervisor is still fully operational. This removes the previous single
point of failure.
The default expansion module does not support RAID configuration and you cannot add
additional M.2 drives to servers that do not have the HWRAID type expander installed.

Note
RAID 1 is a RAID configuration where all data is mirrored between two disks. Anything that is
written to one disk is written to the other in parallel. If one of the disks fails, the storage is fully
usable.

The RAID option has two requirements when configuring the system:
 A different motherboard expander card needs to be ordered (UCS-M2-HWRAID)
 An extra 240GB M.2 storage drive
Note
A mixed configuration of RAID and non-RAID configured servers can be used in a HyperFlex
group. This allows expansions with RAID-enabled systems.

Cisco Mini Storage FlexFlash SD Cards

Every Cisco HyperFlex node includes two Flexible Flash (FlexFlash) SD cards, but their
purpose differs between the server generations, M4 and M5.
The SD cards are hosted by the Cisco Flexible Flash storage controller, a PCI-based controller,
which has two SD card slots. When FlexFlash is enabled, the cards appear as a single partition,
visible as a USB drive to both the BIOS and the host operating system.
You can populate one or both SD card slots that are provided. If two SD cards are populated,
you can use them in the mirrored mode.
In M4 servers, the SD cards are used as boot drives with factory preinstalled ESXi.
In the In M5 nodes, the SD cards are still present but are not used for boot anymore. Instead,
they are available for utilities such as the Cisco Host Upgrade Utility (HUU).

Housekeeping/System Drive
The housekeeping, or the system drive is an SSD drive installed on all HyperFlex systems. It
provides the storage for the CVM.
The CVM cannot run off of the HyperFlex storage that it is creating, because it is necessary to
create that storage. An extra dedicated drive is therefore needed to boot the CVM and create the
HyperFlex storage.
The housekeeping drive is a 240-GB SSD drive:
 Also known as system drive
 Drive is used by the controller VM
 Used for various system operations
o Saves and restores program state

o Removes unneeded software

o Executes disk maintenance utilities

o Collects garbage

o Frees local memory on the stack upon exit from a function

o Backs-up files
Consider these points when identifying the housekeeping drive across various HyperFlex
platforms:
 In the HX240c M5L, the system drive is installed in one of two rear SFF drive slots.
 System drive is installed in the first front drive slot in all other server chassis.

4.5 Describing Cisco HyperFlex Hardware Components

Other Components of Cisco HyperFlex

Nodes
This video describes the HyperFlex converged node storage components. Each Cisco HyperFlex
M5 node is powered by two Intel Xeon Scalable processors. They deliver a significantly
improved performance and serve a much wider range of application needs than the prior
servers. In fact, the benchmarks show improvements of up to 86% over the prior generations
of servers.

They have the following characteristics. They must be installed in pairs of identical CPUs,
belong to the Intel Xeon Scalable processor family, support hyper-threading, and be based on
the Intel C620 series chipset. They're available in a wide range of variants, with cache sizes up
to 38.5 megabytes and up to 28 cores.
Now, the Cisco HyperFlex M5 servers come with the v5 Skylake generation of Intel Xeon proces
sors. Now, the table here in the figure lists the different CPU classes, which differ in the
number of cores, clock rate, cache size, amount of supported memory, and instruction sets.
There are several dozen CPU variants that are available with the Cisco HyperFlex M5 servers.

Enhanced vMotion Compatibility, or EVC, is a feature that should be enabled on the cluster
level when mixing servers with different CPU generations. EVC ensures that all the hosts in the
cluster present the same CPU feature sets to the virtual machines, even if the actual CPUs on
the host are different. This is done by determining the highest common CPU feature set.
And it presents the same CPU feature set preventing migrations with vMotion from failing
because of incompatible CPUs.

Now, Enhanced vMotion Compatibility works only for different CPUs in the same family. You
can configure EVCs between older and newer CPUs from the same vendor, but not between
two CPUs or different vendors.
Memory is used by Cisco HyperFlex servers to serve the internal hypervisor process and
expedite VM-related functions. Now, this figure here lists the benefits provided by memory in
the HyperFlex M5 nodes. They allow up to two vMotion per memory channel.

They are organized with six memory channels per CPU. They come in 128, 64, 32, and 16
gigabyte DIMMs. And they also support 3 terabytes of maximum memory on the M processors
In a dual CPU configuration, you need to select from four to 12 DIMMs per CPU.

When selecting the most appropriate CPU for your cluster, you need to consider the overhead
consumed by the controller VM. You also need to know the RAM support limits. So as a result,
the figure here lists the resources which are reserved for the CVM. The controller VM gets a
dedicated number of processors, cores, or vCPUs and enough memory that allows it to deliver
consistent performance.

The controller can access all storage without hypervisor intervention through the VMware VM
DirectPath feature. So be sure to use the M processors for your larger deployments.
This next list here, this one lists the CPU and memory guidelines that you should follow. The
memory DIMMs must be all of the same type. The number of DIMMs must be equal for both
CPUs. And lastly, you should install six or 12 DIMMs per CPU. Make sure that the compliant
DIMMs are installed in the appropriate slots.

The modular LAN on Motherboard, or mLOM, slot is used for a Cisco Virtual Interface, or VIC,
Card. It does not consume a PCIe slot. These cards can present the 256 PCIe standards-
compliant interfaces to the host, each dynamically configured as a NIC, or a virtual host bus
adapter.

Now, the personality of the interface is set programmatically using the service profile
associated with that server. The figure lists the variance for both the M4 and the M5 servers.
Starting with HyperFlex version 3.5.1, multiple NICs are supported per server. Multiple NICs
are only supported on the Cisco UCS M5. Multiple NICs increase resiliency and enables
features such as offline streaming as well as backups.

The primary mLOM-placed NIC is still mandatory. The other NICs fit into the available PCIe
slots. This is supported on fresh installations only. You cannot upgrade an existing cluster with
additional cards. And of course, all of the converged nodes must have the exact same
configuration of NICs. If you try to install HyperFlex version 3.0 or prior with multiple NICs,
guess what? Installation is just going to fail.

Finally, with HyperFlex version 3.5.1, third-party NICs are supported, as shown in the details
listed in the figure. Third-party NIC support is designed to give customers maximum flexibility
in configuring their servers.

Popular network adapters are available to order preinstalled from Cisco when configuring
HyperFlex converged or compute-only nodes. However, you need to order spare adapters, if
desired, and install them in the server before beginning the HyperFlex installation. Also, be
sure to verify the list of supported third-party network adapters that are approved by Cisco.
And that brings us to the end of this video. Thanks for watching.

Cisco HyperFlex converged servers include, apart from the storage components
(capacity, cache, boot, and housekeeping drives), CPU processors, RAM, and network
adapters.

Hardware component HXDP Function Workload Function

Memory StorFS operation, write caching Workload VM memory

CPU Data optimization VM workload processing

VIC Storage and management traffic Uplinks / traffic processing

Acceleration Engine Card Storage optimization offloading More resources left for workload

CPU Overview

Each Cisco HyperFlex M5 node is powered by two Intel Xeon scalable processors.
These processors deliver significantly improved performance and can serve a much
wider arrange of application needs than prior servers.

The family delivers highly robust capabilities with outstanding performance, security,
and agility. The CPUs provide top-of-the-line memory channel performance and include
three Intel Ultra Path Interconnect (UPI) links across the sockets for improved
scalability and intercore data flow.

CPU processors in HyperFlex M5 servers have these characteristics:

 Typically installed in pairs of identical CPUs

 Intel Xeon Scalable processors
 Support hyper threading
 Based on the Intel C620 series chipset
 Available in a wide range of variants

o Cache size up to 38.5 MB

o Up to 28 cores per processor
The benchmarks show improvements of up to 86 percent over the prior generation of
servers. Each HX-Series node includes a range of processor choices so that you can best
serve your application requirements.

Note
ESXi-based Standard HyperFlex deployments and HyperFlex Edge deployments
support single CPU deployments as an ordering option. Stretched and Hyper-V
deployments require 2 CPUs per HyperFlex converged node.

Intel Xeon Scalable CPU Classes

Cisco HyperFlex M5 servers come with Intel Xeon Scalable processors. This CPU 14-
nm microarchitecture offers more cores, allow higher CPU speeds, and support more
RAM than its predecessors.

Intel Xeon Scalable processors are available in different classes, which differ in the
number of cores, clock-rate, cache size, amount of supported memory, and instruction
sets.

The table lists the examples of CPU capabilities for each CPU tier. The name of the
listed CPU is in parentheses.
 Intel Xeon Class Max. Number or Cores Max. Base Frequency Max. Turbo Frequency

Platinum (I8276) 28 2.2 GHz 4.0 GHz

Gold (I6248) 20 2.5 GHz 3.9 GHz

Silver (I4216) 16 2.1 GHz 3.2 GHz

Bronze (I3206R) 8 1.9 GHz No Turbo

The letters at the end of the CPU designation have different meanings. R, for example,
denotes that the CPU is a refreshed version. You can take a look at the full Intel CPU
lineup and CPU naming scheme at
https://www.intel.com/content/www/us/en/products/docs/processors/xeon/2nd-gen-
xeon-scalable-processors-brief.html.

Note
Please note that a single CPU may not support both maximum number of cores and
maximum speed. The table displays example CPUs that are most representative as of
May 2021.

CPU Architecture

The performance of the Intel Xeon Scalable processors is

enhanced by other architectural components, such as the
workload accelerators Lewisburg Platform Controller Hub (PCH),
which work in tandem to maximize the overall system
performance.

The Lewisburg PCH family offers these features:

 Multiples of the system bandwidth back-hauled to the CPU.

 Integration of up to 4x 10GbE on the PCH.
 New and fast generation of QuickAssist.
 New Innovation Engine available to Original Equipment
Manufacturers (OEMs).
Certain applications can benefit greatly from workload
accelerators made by Intel. Some of these accelerators are:

 Intel AVX-512 (2X FLOPs/core) for HPC, Analytics, Database,

and Security.
 Intel QuickAssist Technology: 2x faster crypto and
compression.
 FPGA: 5X speedups on server, storage, and network
algorithms.

EVC in Mixed CPU Groups

Different CPU generations and even CPU versions can use different CPU instruction
sets. These instruction sets define which actions a VM can request from the CPU to
process data. The Xeon Scalable CPOUs use the AVX-512 instruction set.

When you start a VM, it verifies the available CPU instructions on the host and starts
using that instruction set. The instruction set used by a running VM cannot be changed
without rebooting that VM.

When you are mixing older and newer CPUs you cannot freely migrate VMs started on
a newer CPU to a host with an older CPU, because the older CPU does not support the
instruction set that the VM is using. This can lead to a failed failover event, problems
with VM migrations and uneven VM distribution, because DRS cannot distribute VMs
to less used hosts.

EVC (Enhanced vMotion Compatibility) is a feature that you can enable on a vSphere
group level. It defines the highest common instruction set for supported instruction sets
for all the servers in the group. This effectively disables the newer features of CPUs and
lowers their performance. The greater the difference between CPU generations, the
greater the decrease in performance.

EVC should be enabled on the group level when mixing servers with different CPU
generations (such as Skylake and Broadwell, or Haswell). EVC ensures that all hosts in
a group present the same CPU feature set to virtual machines, even if the actual CPUs
on the hosts differ. Presenting the same CPU feature set prevents migrations with
vSphere vMotion from failing because of incompatible CPUs.

EVC provides the following features:

 Once enabled, prevents vMotion failures between server generations

 Group determines the highest common CPU feature set.

o EVC level associated with each CPU generation

o Host on a given level supports all lower levels.

 Makes all hosts present identical features to the VMs

 EVC makes it easier to add hardware to existing server groups. No manual CPUID
masking is required. New CPUs are automatically configured to be compatible with
earlier versions.

EVC works only for different CPUs in the same family. You can configure EVC
between older and newer CPUs of a vendor but not between two CPUs of disparate
vendors. This limitation is irrelevant for Cisco HyperFlex servers, because they are
Intel-only.

Note
Be aware that enabling EVC on a working server group will require you to restart the
servers in the group to enable the feature.

vMotion capabilities are required when you desire to move a VM from one host to
another, and they also provide a foundation for more advanced features, such as
vSphere High Availability and vSphere DRS.

CPU EVC Level Added Features (Compared to Previous Generations)

M4 (Intel "Haswell" ABMX2, MOVBE, FMA, PERMD, RORX/MULX, INVPCID,

L6
Generation) VMFUNC

Advanced Vector Extensions 512, Persistent Memory Support

M5 (Intel "Skylake"
L8 Instructions, Protection Key Rights, Save Processor Extended States
Generation)
with Compaction, Save Processor Extended States Supervisor

The table describes the EVC levels associated with the two HyperFlex CPU
generations, M4 and M5. It lists some of the CPU features added in the respective
generation.

Therefore, if you enable EVC in a mixed M4/M5 setup, all hosts will advertise the L6
level. The VMs will know that they can apply the features encoded in the L6 level but
not in the L8 level.

Memory
The Server memory configuration is important for running the workload on any server,
however on a hyperconverged infrastructure it is also used for the storage operation and
the HCI components on servers reserve a certain amount of the system memory for their
operation.
Operating system memory is used by the Cisco HyperFlex CVMs to perform
compression and deduplication jobs as well as cache the storage operations and storage
metadata. Memory performance has a significant impact on overall system operation.

Memory in HyperFlex M5 nodes with Intel Skylake processors:

 Allows up to two DIMMs per memory channel

 Is organized with six memory channels per CPU
 Comes in 128-, 64-, 32-, and 16-GB DIMMs
 Permits 3-TB (3072-GB) maximum memory

o 2 x 128-GB x 6 channels x 2 CPU = 3072 GB

 Requires specific processor type for maximum memory

o Example: Intel I8276L (Platinum)

o One processor supports up to 1.5-TB memory.
o Dual CPU setup supports 3-TB memory.

HyperFlex M5 servers with Intel Xeon Scalable processors support these DIMM types:

 Registered ECC DDR4 DIMMs (RDIMMs)

 Load-Reduced DIMMS (LR-DIMMs)
 Through-silicon-Via DIMMs (TSV-DIMMs)

The DIMMs have the following technical specifications:

 Clock speed: 2933 MHz

 Ranks per DIMM: 1, 2, 4, or 8
 Operational voltage: 1.2 V

In a dual-CPU configuration, you need to choose from 4 to 12 DIMMs per CPU. Intel
recommends that you have entire channels populated for maximum efficiency of
performance, so either 6 or 12 DIMMs per CPU.
DDR4 DIMM Choices
The HyperFlex servers support up to 24 DIMMs (2 CPUs x 6 channels x 2 DIMMs per
channel), all of which must be of the same size and type. The available DIMM sizes and
descriptions are given below.

Product ID Description Voltage Ranks/DIMM

HX-ML-128G4RT-H 128GB DDR4-2933-MHz LRDIMM/4Rx4/1.2v 1.2 V 16

HX-ML-X64G4RT-H 64GB DDR4-2933-MHz LRDIMM/4Rx4/1.2v 1.2 V 16

HX-MR-X64G2RT-H 64GB DDR4-2933-MHz RDIMM/2Rx4/1.2v 1.2 V 8

HX-MR-X32G2RT-H 32GB DDR4-2933-MHz RDIMM/2Rx4/1.2v 1.2 V 8

HX-MR-X16G1RT-H 16GB DDR4-2933-MHz RDIMM/1Rx4/1.2v 1.2 V 4

Memory Enhancements
The overall memory performance differs between the M4 and M5 server generations.
This improvement has been brought about by the increase in clock frequency and fewer
number of DIMMs per channel.

Memory clocking and per-channel density between M4 and M5

Server Generation DIMM Clock Frequency Number of DIMMs per Channel

M4 2400-MHz 3

M5 2933-MHz (18% increase) 2 (50% per-DIMM bandwidth increase)

CPU and Memory Guidelines

When selecting the most appropriate CPU for your HyperFlex group, you should
consider the overhead consumed by the CVM and RAM support limits.

Considerations when choosing hardware:

 Resources are reserved for the CVM:

o 8 vCPUs, shared; 10.8 GHz of CPU power
o 48-GB memory on all HX220c M5, except All-NVMe where 70 GB is
reserved
o 72-GB memory on each HX240c M5, except LFF where 78 GB is reserved

 Use appropriate processors for large memory deployments.

o 2nd generation L processors support 1.5-TB memory.

 o Total maximum memory for a dual CPU is 3 TB.
o Different processors support different memory capacities and features.

The CVM gets a dedicated number of processor cores (vCPUs) and amount of memory
allowing it to deliver consistent performance and not affect the performance of other
virtual machines in your system. The controller can access all storage without
hypervisor intervention through the VMware VM_DIRECT_PATH feature.

In contrast to capacity storage or networking connectivity, the CPU resources cannot be

upgraded without replacing the CPU processors. It therefore recommended to choose
the best available or at least highly sufficient CPU power for your Cisco HyperFlex
servers.

With the first generation of Intel Xeon Scalable Processors, follow these CPU/memory
guidelines:

 Memory DIMMs must all be the same type.

 Number of DIMMs must be equal for both CPUs.
 Install 6 or 12 memory modules for best performance.
To guarantee smooth system operation, you must ensure that compliant DIMMs are
installed in the appropriate slots. Although different configurations are supported,
installing 6 or 12 RAM sticks will give the best results. Be careful since memory
bandwidth can be negatively affected by other memory slot population options.

The minimum requirement for memory capacity is 192-GB DRAM. This will not leave
much memory for your workload due to CVM reservations.

Note
Memory mirroring is not supported for HyperFlex solutions.

HyperFlex Graphical Acceleration

Graphics cards can also be installed in the PCIe modules of HyperFlex nodes.
Depending on the chassis and the graphics card form factor, up to 6 cards can be
installed in a single Cisco HX-Series chassis.

GPUs can be ordered for new or existing HyperFlex deployments:

 HX220c Chassis supports only single-width GPUs, such as T4

 HX240c Chassis also supports up to two double-width GPUs, such as RTX 8000
 HX220c Chassis supports up to 2 single-width GPUs, while HX240c supports 6
 Consider the power supplies, power delivery, and heat dissipation when using GPUs
The GPU number and size are determined by the physical space available in the chassis.

There is a power calculation tool available on Cisco.com that helps you determine the
optimal power supply unit to use with your GPU-enabled
deployment: https://ucspowercalc.cloudapps.cisco.com/public/.

Large power usage also brings with it a lot of heat. Cisco UCS servers have been
designed with GPU support in mind, so they dissipate heat very effectively compared to
other server solutions. But even the best cooling solution depends on the environment
where it runs. Make sure to provide sufficient cooling for your data canter. It is also a
good practice to have hot/cold rack separation to optimize cooling.

Not all HyperFlex servers need to be equipped with a GPU to run a GPU workload,
however, to provide consistency, it is best practice to install GPUs in all the servers.
Live vMotion is supported with NVIDIA GPUs, allowing you to migrate accelerated
VMs while they are powered on. To enable migration, the target system needs to meet
the migration requirements.

Network Adapters: mLOM

A modular LAN-on-Motherboard (mLOM) slot is used for a Cisco VIC. It incorporates
next-generation converged network adapter (CNA) technology from Cisco, providing
investment protection for future feature releases.

Important information about Cisco UCS VICs:

 Installed in the mLOM slot do not consume a PCIe slot

 Can present up to 256 PCIe standards-compliant interfaces to the host
 Available in two variants, for M4 and M5 servers:

o Cisco UCS VIC 1227

 Dual-port SFP+ 10-Gbps adapter

 Supported only on UCS M4

o Cisco UCS VIC 1387

 Dual-port QSFP+ 40-Gbps adapter

o Cisco UCS VIC 1457

 Quad-port SFP28 10/25-Gbps adapter

The cards enable a policy-based, stateless, agile server infrastructure that can present up
to The cards enable a policy-based, stateless, agile server infrastructure that can present
up to 256 PCIe standards-compliant interfaces to the host, each dynamically configured
as either a NIC or HBA. The personality of the interfaces is set programmatically using
the service profile associated with the server. The number, type (NIC or HBA), identity
(MAC address and WWN), failover policy, adapter settings, bandwidth, and QoS
policies of the PCIe interfaces are all specified using the service profiles.

Network Adapters: Multi-NIC Support

Multiple NICs are supported with HyperFlex hardware to add additional link resiliency
to the solution. This can help overcome the policy or environment restrictions of a given
HyperFlex deployment.

Starting with HXDP 3.5(1a), multiple NICs are supported per server:

 Increases resiliency and enables use cases, such as offline streaming and backup
 Primary, mLOM-placed NIC is still mandatory; other NICs fit into PCIe slots.
 Only supported on fresh installations; no upgrade of existing deployments with
additional cards

o All converged nodes must have the same configuration of NICs.

Multi-VIC support in HyperFlex is designed to tolerate any single link failure, fabric
failure, or a hardware VIC failure. As shown in the following illustration, all HyperFlex
services are pinned to port 1 on VIC #1 and port 2 on VIC #2. This spreading of vNICs
across the two VIC cards enables seamless failover during a hardware failure. The
aggregate bandwidth available to HyperFlex services (management traffic, VM traffic,
storage traffic, and vMotion traffic) is therefore the same whether there are one or two
physical VICs installed. However, the additional unused ports, may be used as needed
for other use cases. User vNICs can be created on port 2 of VIC #1 and port 1 of VIC
#2. You may design your own failover strategies and virtual networking topologies with
this additional bandwidth available to each HyperFlex server.

You must place only new customer defined vNICs on these unused ports to avoid
contention with production HyperFlex traffic. Do not place them on the ports used for
HyperFlex services.

Multiple NICs are only supported on Cisco UCS M5. Multiple NICs are not supported
on HyperFlex Edge systems. You may not combine VIC 1300 and VIC 1400 series in
the same node or within the same group. Interface speeds must be the same: either all
10-Gbps (with enough QSAs if using VIC 1300) or all 40-Gbps. All VIC ports MUST
be connected and discovered before starting installation. HyperFlex installer will
validate that all VIC ports are properly connected to the Fabric Interconnects.

Note
Refer to the Cisco UCS HCL Tool (https://ucshcltool.cloudapps.cisco.com/public/) for
the complete list of network adapters supported. HyperFlex servers follow the same
certification process as C-series servers and will support the same adapters.

If you try to install HXDP v3.0 or prior with multi-NIC nodes, the installation will fail
due to validation not allowing it to proceed because multi-NIC nodes were not
supported.

Network Adapters: Third-Party Adapter Support

Third-party NIC support in HyperFlex is designed to give customers maximum
flexibility in configuring their servers as needed for an expanding set of applications
and use cases.
Third-party NICs are supported on all HyperFlex converged and compute-only systems,
including HyperFlex running under Fabric Interconnects and HyperFlex Edge.

Starting with HXDP 3.5(1a), third-party NICs are supported:

 Example use cases are RDMA over Converged Ethernet (RoCE) and Single Root
I/O Virtualization (SR-IOV).
 Primary, mLOM-placed NIC is still mandatory; other NICs fit into PCIe slots.
 Interface speeds must be the same. All VIC ports must be connected and discovered
before starting installation. Installation will check to ensure that all VIC ports are
properly connected to the Fabric Interconnects.
 The maximum quantity of NICs supported is based on the physical PCIe space in
the server. Mixing of various port speeds, adapter vendors, and models is allowed,
because these interfaces are not used to run the Cisco HyperFlex infrastructure.
 Different Intel and Mellanox NICs are supported, among others.

Note
RoCE is a network protocol that allows remote direct memory access (RDMA) over an
Ethernet network.

Note
SR-IOV allows a single physical PCIe to be shared between many VMs in a virtual
environment.

The most popular network adapters will be available to order preinstalled from Cisco
when configuring HyperFlex converged or compute-only nodes. However, any
supported adapter may be ordered as a spare, and installed in the server before
beginning HyperFlex installation.

Note
Refer to the Cisco UCS HCL Tool (https://ucshcltool.cloudapps.cisco.com/public/) for
the complete list of network adapters supported. HyperFlex servers follow the same
certification process as C-Series servers and will support the same adapters.

Special consideration must be taken when using third-party adapters on HyperFlex

systems running under Fabric Interconnects. Manual policy modification is required
during installation.

Follow these steps to manually alter the policies:

 Launch Cisco UCS Manager and login as an administrator.

 In the Cisco UCS Manager GUI, go to the Servers tab.
 Navigate to the HX Cluster org that you wish to change and fix the issue when you
get a configuration failure message on each Service Profile, such as "There is not
enough resources all for connection-placement."
 On each Service Profile template that is listed (hx-nodes, hx-nodes-m5, compute-
nodes, compute-nodes-m5), change the vNIC/vHBA policy to Let System
Placement. Click Modify vNIC/vHBA Placement.
 Change the Placement back to Let System Perform Placement.
 This action will trigger all the Service Profiles that are associated to any of the 4
Service Profile Templates to go into a pending acknowledgment state. You must
reboot each affected system to fix this issue. This process should only be done on a
fresh deployment and not an existing one during upgrade or expansion.

4.6 Describing Cisco HyperFlex Hardware Components

Cisco UCS Fabric Interconnects

Now let's examine the Cisco UCS Fabric Interconnects, as well as the compute-only nodes. The
Cisco Fabric Interconnect, once before I mentioned, is the core of the Cisco UCS solution. It
provides network connectivity and management capabilities for the Cisco HyperFlex and the
Cisco UCS servers.

You can use a redundant pair of fabric interconnects in a cluster configuration for high availabil
ity with the management plane and this is recommended. Data redundancy depends on the
user configuration and might require a third-party tool for fiber channel host support.
Now by default, like I said, the fabric interconnects are non-blocking and provide wire rate
speed with less than two microseconds port-to-port latency. So the fabric interconnects come
with also with a fixed number of ports that you can optionally install expansion modules on.
Lastly, G2 and G3 fabric interconnects are supported as of HyperFlex version 3.5.1.

Now, you connect the Cisco HyperFlex servers to the fabric interconnect similar as you would
cable any other rack-mounted servers. The HyperFlex servers are connected via using the two
cables to both fabric interconnects and a high-availability cluster. All of these C-Series as well
as the B-Series chassis, along with their blades, then become part of a single management
domain.

Now, this figure lists some details and limitations for connectivity. HX-Series converged servers
are connected to the Cisco UCS fabric interconnects in what we call the direct connect mode.
This means that Cisco UCS Manager can be managed to direct servers using a single cable for
both the management traffic and data traffic. In the past, you had to run two separate cables.
This figure lists the fabric interconnect models along with supported M5 connectivity options
for the VIC 1387 and VIC 1457 cards. The Cisco QSFP to SFP or the SFP-plus adapter is now a
40-to-10 gigabit converter. This enables you to convert the VIC 1387 to 10 gig ports on the
fabric interconnects. Breakout cables are not supported with HyperFlex due to QoS
restrictions.

This next figure here lists the details for building large clusters. Cisco HyperFlex is a scalable
system. And as of HyperFlex version 3.5.1, 64 servers are supported on a standard ESXi base
cluster.

But be sure to note some of the limitations. I mean, if you want to connect Fiber Channel
storage to the same Cisco UCS domain, almost all fabric interconnects support those unified
ports that I talked about, which can be configured as Ethernet or Fiber Channel. The only
exception is the Cisco UCS 6332. And also keep in mind, large clusters, such as eight servers or
more, they're not that common.
Cisco HyperFlex has two types of servers, as I mentioned before, converged and compute-
only-- when the HyperFlex cluster runs out of resources, you've got to make a decision. Do you
want to add converged nodes, which add compute CPU, and memory, and storage resources
to the common pool? Or do you just want to add compute-only nodes, which, as the name
implies, only adds compute resource and just basically gives more horsepower to the common
pool?

This option, like I said, is used when you want to add additional CPU horsepower, and you just
don't need the unnecessary storage. Since the compute-only nodes do not have storage, they
utilize resources available from the converged nodes.

The key point in this figure here describes the guidelines when expanding a Cisco HyperFlex
cluster with compute-only nodes. Obviously, you can't build an initial cluster with only
compute-only nodes. You need storage. Right?

The compute-only nodes can be either B-Series or C-Series servers. If there's local storage
installed, it will be ignored. The compute-to-converged nodes ratio is based on the HyperFlex
license. You can get a 1-to-1 ratio with a Standard license and a 2-to-1 ratio with the Enterprise
license. Also, the compute-only nodes do not require an extra HyperFlex license.
Finally, this figure lists the points you should consider when expanding a Cisco HyperFlex
cluster with compute-only nodes. You should be aware of the supported blade servers and the
B-Series chassis. You should recognize the supported C-Series rack-mounted servers.

And lastly, be aware of the supported boot options. Also, you can mix compute-only node
models and generations if you enable the enhanced vMotion compatibility or the EVC mode.
That brings us to the end of this video. Thanks for watching.

The Cisco Fabric Interconnect is a core part of the Cisco UCS. The Cisco Fabric
Interconnect provides network connectivity and management capabilities for the Cisco
HyperFlex and Cisco UCS servers.

Cisco UCS Fabric Interconnects offer these benefits:

 Provide the management and communication backbone.

 Enable non-blocking, wire-rate connectivity.
 Comprise a fixed number of unified ports with optional expansion modules.

o Expansion modules available only in G2 Fabric Interconnects

 G2, G3, and G4 of Fabric Interconnects are supported with HyperFlex.

Cisco Fabric Interconnect provides the management and communication backbone for
the Cisco HyperFlex and B-Series blade servers and chassis. When you connect servers
to the Fabric Interconnect, all servers become part of a single and highly available
management domain.

Cisco UCS Fabric Interconnect supports unified fabric, which means that you can
provide storage and LAN networking for all servers within the domain. This solution
reduces the need for multiple parallel physical networks, different types of adapter
cards, switching infrastructure, and cabling within racks.

Cisco UCS Fabric Interconnect comes with a fixed number of ports. You can optionally
install expansion modules.

You will use a redundant pair of fabric interconnects in a HyperFlex configuration. If

one Fabric Interconnect becomes unavailable, the other takes over.

In addition, a high availability configuration actively enhances failover recovery time

for redundant virtual interface connections. When an adapter has an active virtual
interface (VIF) connection to one Fabric Interconnect and a standby VIF connection to
the second, the learned MAC addresses of the active VIF are replicated but not installed
on the second Fabric Interconnect. If the active VIF fails, the second Fabric
Interconnect installs the replicated MAC addresses and broadcasts them to the network
through Gratuitous Address Resolution Protocol (GARP) messages, shortening the
switchover time.

The high availability configuration provides redundancy only for the management
plane. Data redundancy depends on the user configuration and might require a third-
party tool to support data redundancy.

To use the high availability configuration, you must directly connect the two fabrics
interconnects using Ethernet cables between the L1 (L1-to-L1) and L2 (L2-to-L2) high-
availability ports, with no other fabric interconnects in between. Also, you can connect
the fabric interconnects directly through a patch panel to allow the two fabric
interconnects to continuously monitor the status of each other and quickly know when
one has failed.

Wiring Cisco HyperFlex Servers to Fabric Interconnects

You connect the Cisco HyperFlex servers to the Fabric Interconnects in the similarly as
you wire other rack-mount servers.

Use unified wire to connect each HyperFlex server to both Fabric Interconnects.

 HyperFlex UCS M5 as of HXDP 4.5(1a), supports mLOM-based VIC 1387 and

VIC 1457.

o VIC 1387 (40G) is a single wire to each Fabric Interconnect.

o VIC 1457 (10G or 25G) supports one or two wire connectivity to each
Fabric Interconnect.
 Use of Fabric Extender (FEX) between server and Fabric Interconnects is not
supported.
 When connecting VIC to Fabric Interconnects, make sure port numbers match on
both fabric Interconnect devices.

o If ports do not match, HyperFlex installation will fail.

Cisco UCS Fabric Interconnect Supported Connectivity Options

The HX-Series converged servers are connected to the Cisco UCS Fabric Interconnects
in Direct Connect mode. This option enables Cisco UCS Manager to manage the HX-
Series Rack-Mount Servers using a single cable for both management traffic and data
traffic.

Cisco HyperFlex M5 generation servers can be configured with Cisco VIC 1387 or VIC
1457. With VIC 1387, you would connect port 1 of the VIC card (the right-hand port) to
a port on Fabric Interconnect A, and port 2 of the VIC card (the left-hand port) to a port
on Fabric Interconnect B. If you are using VIC 1457 with HyperFlex make a note to
connect ports 1 and 2 to FI-A and ports 3 and 4 to FI-B, otherwise the server will not
discover them. The HyperFlex installer checks for this configuration, and that all
servers’ cabling matches. Failure to follow this cabling practice can lead to errors,
discovery failures, and loss of redundant connectivity.

Fabric Interconnects UCS M5 with VIC 1387 UCS M5 with VIC 1457

UCS 6248 2x QSA + 10 Gbps 2x or 4x 10 Gbps, min. HXDP 3.5(1a)

UCS 6296 2x QSA + 10 Gbps 2x or 4x 10 Gbps, min. HXDP 3.5(1a)

UCS 6332 2x 40 Gb Not supported as of HXDP 4.0(1a)

UCS 6332-16 2x 40 Gb Not supported as of HXDP 4.0(1a)

UCS 6454 2x QSA + 10 Gbps 2x or 4x 10/25 Gbps, min. HXDP 3.5(2a)

UCS 64108 2x QSA + 10 Gbps 2x or 4x 10/25 Gbps, min. HXDP 3.5(2a)

Cisco QSFP to SFP/SFP+ Adapter (QSA) is a 40- to 10-Gbps converter.

Breakout cables are not supported when connecting HyperFlex servers to Fabric
interconnects due to QoS restrictions. Breakout cables can be used for Fabric
Interconnect uplinks.

As of HXDP 4.5(1a), the maximum size of the standard ESXi-based system is 64

servers.

 Except for the stretched deployment, a HyperFlex deployment cannot be a part of

more than one Cisco UCS domain.
 You can only achieve this size with Cisco UCS 6296 or Cisco UCS 64108, other
fabric interconnects do not have enough ports.
 If you want to connect Fibre Channel storage to the same Cisco UCS domain, all
Fabric Interconnects, except Cisco UCS 6332, support unified ports.

4.7 Describing Cisco HyperFlex Hardware Components

Compute-Only Nodes
Compute only nodes are Cisco UCS servers that can be added to HyperFlex groups and
share the storage provided by the converged nodes. Compute nodes do not have any
capacity disks and do not provide storage to the shared HyperFlex storage.

Compute only nodes can have an internal Hypervisor drive or can alternatively boot
from SAN. Aside from booting the Hypervisor, no additional storage is necessary.

When a HyperFlex system runs out of resources, perform these actions:

 Add more converged node resources.

o Add disks or memory to existing nodes.
o Add nodes (CPU, memory, storage).

 Add B-Series or C-Series servers to scale only compute.

o No capacity is provided by compute-only nodes.
o Compute-only nodes must be in the same Cisco UCS
domain.
Compute-only nodes are part of the same vSphere group as the converged nodes. Since
compute-only nodes do not provide storage for VMs, they utilize resources available
from the converged nodes.

Note
Prior to HyperFlex Version 2.5, a single boot policy was shared between HyperFlex and
compute-only service profile templates. Starting in HXDP 2.5(1a), a dedicated boot
policy is available for use with compute-only nodes.

Adding Compute Nodes

These key points describe the guidelines when expanding a Cisco HyperFlex
deployment with compute-only nodes:

 You cannot build the initial system with compute-only nodes.

 Compute-only nodes are either B-Series or C-Series servers.
o If compute-only nodes have local storage, that storage can only be used
for boot.

 Compute-to-converged node ratio is limited.

o 1:1 with HXDP Datacenter Advantage license
o 2:1 with HXDP Datacenter Premier license
The two Fabric Interconnects both connect to every HX-Series rack-mount server, and
both connect to every Cisco UCS 5108 blade chassis and Cisco UCS rack-mount server.
Upstream network connections, also referred to as "northbound" network connections,
are made from the Fabric Interconnects to the customer data center network at the time
of installation.

Supported Compute-Only Nodes

Consider these points when expanding a Cisco HyperFlex deployment with compute-
only nodes:

 Supported blade servers:

o B200 M4/M5
o B260/B420/B460 M4, B480 M5

 Supported rack-mount servers:

o C220/C240 M4/M5
o C460 M4, C480 M5, and C480 M5 ML

 Supported boot options:

o FlexFlash SD Cards or local disk (HDD/SSD)
 o SAN LUN

 You can mix compute-only models and generations.

o Enable VMware EVC when mixing generations.

Note
With HyperFlex, it is recommended that you enable vMotion to use the full automation
potential of the deployment. You can mix servers with different generations of
processors and still keep the vMotion functionality as long as you enable the EVC.
However, it is recommended that you keep processors in a group no more than one
generation apart (such as Skylake and Haswell).

Note
HXDP 4.0(1a) introduced support for the C480 M5 ML as a new compute-only node for
Deep Learning/Machine Learning Workloads. Data scientists can now use the power of
up to eight NVIDIA SXM2 V100 GPUs to accelerate deep learning workloads. VMs
running deep-learning workloads will need to use PCIe pass-through for access to
GPUs.

4.8 Describing Cisco HyperFlex Hardware Components

Investigate Cisco UCS Part of

HyperFlex
Investigate Cisco UCS Part of HyperFlex
The Cisco Unified Computing System (UCS) is the server foundation for Cisco
HyperFlex. The Cisco HyperFlex installer wizard abstracts the configuration of the
Cisco UCS Manager, so you (as the installer) do not need to manually create templates,
create pools, derive service profiles, and so on. All that is done for you by the
HyperFlex installer.

You will use an account with read-only credentials to access the Cisco UCS Manager.
The infrastructure you are looking at is the actual physical infrastructure that your lab
environment runs on.

The Cisco UCS Manager that you are accessing in this discovery lab, sits on a physical
pair of fabric interconnects and all are a part of this course lab environment. Because
changes are being made on the infrastructure that you are accessing, the configuration
on screenshots will differ to the configuration that you are seeing throughout the lab
exercise.

Investigate a Server in Cisco UCS Manager

Step 1

Log in to the Cisco UCS Manager.

Answer
Open the Firefox browser and navigate to https://ucsm-a.hx.lab. Choose Launch UCS
Manager on the Cisco UCS Manager introduction page to open the login prompt.

Note
The Cisco UCS Manager lets you monitor and manage the Cisco UCS servers,
including connecting to the server KVM of an individual server. You can also connect
to the server management IP directly to access the keyboard, video, mouse (KVM)
Manager of that server, which only gives you KVM access to it.
Log in using the student / 1234QWer credential combination.

Note
The credentials you are using provide you with read-only access to the system. You will
be able to view the configuration, but you will not be able to change it.

When the Cisco UCS Manager loads, you will be redirected to the Equipment tab of
the Cisco UCS Manager interface.

The Equipment tab gives you access to hardware information of the Cisco UCS
domain. It offers a topology overview at the top level and general Cisco UCS domain
information.

Note
Cisco UCS Manager objects are in a hierarchical structure. Click the black triangle next
to an object to reveal its subordinate items.
Step 2

Investigate one of the servers in the Cisco UCS domain.

Answer
Navigate to Equipment > Rack-Mounts > Servers > Server 1. Server 1 is one of the
servers connected to this pair of Fabric Interconnects and is the first node of the six
node standard HyperFlex group.

The General tab of a server provides you with the general information about that
chosen server:

 The server type is HXAF220c (under Product Name).

 The service profile applied to this server is org-root/org-standard-cluster/ls-rack-
unit-1.
 The product ID of the server is HXAF220C-M5SX. You can only use servers with
HyperFlex PIDs for converged HyperFlex nodes.

Note
The Actions menu provides you with quick access to common actions for a server. To
configure the server in more detail, use the tabs above the server image. Usually, there
are several ways to accomplish the same task in the Cisco UCS Manager.

Step 3

Explore the inventory of a server.

Answer
Choose the Inventory tab, and then choose the CPUs subtab.

Notice that the server has the Xeon Silver 4114 type of Intel processor. Each processor
has 10 cores at a 2.2-GHz clock rate.

In the Inventory tab, choose the Memory subtab. Each CPU has 12 slots for RAM
DIMMs. This server should be fully populated with 64GB DIMMs—1.5TB altogether
in the entire server.

Note that you can also see the DIMMs' layout on the server's motherboard. This can be
helpful when the replacement of a faulty DIMM is needed.
In the Inventory tab, choose the Adapters subtab.

Notice that the network adapter that the server has is a one single mLOM-based 40-
Gbps device. The server is equipped with a 2-port VIC 1387, which provides a
redundant physical link to each of the Fabric Interconnects. The downstream network
connecting to each of the Cisco UCS Fabric interconnects is called a fabric.

In the Inventory tab, choose NICs subtab.

Notice that the same network adapter that you observed in the Adapters subtab is
segmented into 8 virtual network cards: four for each fabric (fabric A and fabric B).

Note
These virtual cards are seen as a physical Network Interface Cards (NICs) on the ESXi
or in the vCenter.

Within the Inventory tab, choose Storage, then Disks. You should see several storage
controllers. If you expand a storage controller, by clicking the triangle next to its name,
the disks connected to the expanded controller will be listed. Click one of the disks.

 Server has one boot SSD (M.2 format and size of 240 GB).
 Server has one housekeeping SSD (SFF format and size of 240 GB).
 Server has six capacity SSDs (SFF format and size of 960 GB).
 Server has one cache SSD (SFF format and size of 400 GB).
Note
Feel free to explore the rest of the sub-tabs of the Inventory tab. Some tabs will be
empty (such as GPUs, host bus adapters [HBAs], Internet Small Computer Systems
Interface [iSCSI] virtual network interface cards [vNICs]), because this group does not
have those components installed.

Step 4

Examine the finite state machine (FSM) and faults.

Answer
Choose the FSM tab.
The FSM Status should be Success and the bar should show 100 percent, indicating that
the previous action performed on the server has finished successfully.

The FSM tab is very useful when you want to follow events on the server. The FSM
system lists individual states of performed actions and is therefore a great resource for
troubleshooting. It also displays a completion percentage for performed actions.

Choose the Faults tab.

When something goes wrong on a server, the error will be listed in the Faults tab.
Errors are classified according to severity and can be filtered to quickly review errors of
a particular severity.

If any faults are displayed, investigate them. As shown in the example, you should see a
warning-level fault because the server is not redundantly connected to a power supply.
Investigate a Fabric Interconnect
Step 5

Investigate the Fabric Interconnect type.

Answer
In the Cisco UCS Manager, navigate to Equipment > Fabric Interconnects > Fabric
Interconnect A. Note that the Fabric Interconnect product type is Cisco UCS 6332.

Note
A standard Cisco HyperFlex server connects to one Cisco UCS domain, and one Cisco
UCS domain comprises two Fabric Interconnects: A and B.

The General tab is similar to the server General tab, except that it lists actions and
values specific to the chosen Fabric Interconnect.

Step 6

Investigate ports.
Answer
Navigate to the Physical Ports tab, choose Ethernet Ports, and expand the Fixed
Module entry to reveal the list of ports available on the module.

You are looking at the Cisco UCS Fabric Interconnect 6332 device. Note that some
ports are configured as Server, some are configured as Network, and some
are Unconfigured.

On the Cisco UCS domain installation, all the ports are Unconfigured, which means
they are administratively down. You then configure the ports to serve as Server ports
leading down to the servers, or Network ports, which are the uplinks.

Cisco UCS 6332 Fabric Interconnect has 32 40-Gbps ports. Some of these ports can be
broken up into 4x10 Gbps or converted to 1x10 Gbps. Ports that are broken up are
called breakout or scalability ports. On the Fabric Interconnect, you are investigating
one or more ports that are marked as Scalability ports. These ports use a special cable,
which splits out: on one side, it has a 40-Gbps QSFP+ connector; on the other side it
splits out into four 10-Gbps SFP ports. The cable enables you to connect Fabric
Interconnect 6332, which only has 40-G ports to a 10G switch while maintaining the
throughput. In the example below, you can see one of the scalability ports.
Note
When you want to convert a 40-Gbps port into a breakout port, you must restart the
Fabric Interconnect. Because there are two Fabric Interconnects, you can configure one,
wait for the restart, and then configure the breakout port on the second one and wait for
that restart. This designed redundancy is also utilized on upgrades and QoS changes.
The connectivity handover on failover from one Fabric Interconnect to the other is
automatic.

Step 7

Investigate FSM and faults.

Answer
Choose the FSM tab. It serves to show the progress of tasks being performed on fabric
interconnects, for example upgrades or restarts.

Choose the Faults tab and browse through the faults. Some faults are just informative
messages (severity of Info), while other faults have higher severity. The example below
shows a fault that has the severity of Warning because the Fabric Interconnect is only
connected to one power supply.
Investigate a Server Service Profile
Step 8

Investigate the server configuration.

Answer
Navigate to the Servers pane in Cisco UCS Manager.

The Servers pane includes service profile templates, service profiles, policies, pools,
and schedules.

Different configuration policies available in this and other panes are assembled into a
service profile template, which defines which policy to use. Then, the service profile
created from the template is applied to server, which is configured automatically by the
Cisco UCS Manager to comply with the policies that are defined in its service profile.
 Step 9

Investigate service profile templates.

Answer
Expand Service Profile Templates > root > Sub-Organizations and then
expand standard-cluster. Click Service Profile Templates to see all of them.

The Cisco UCS Manager allows policies and configuration to be hierarchically

structured through sub-organizations. As of HXDP v4.5.1a, the HyperFlex installer
creates 11 service profile templates:

 Service Template compute-nodes-anyld: Generic service profile template for compute

servers.
 Service Template compute-nodes-m5-anyld: Generic service profile template for M5
compute servers.
 Service Template compute-nodes-ldr1: Service profile template for compute servers
with the Megaraid controller.
 Service Template compute-nodes-m5-ldr1: Service profile template for M5 compute
servers with the Megaraid controller.
 Service Template compute-nodes-m5-m2pch: Service profile template for M5
compute servers with the PCH controller.
 Service Template compute-nodes-m5-m2r1: Service profile template for M5 compute
servers with the M.2 raid controller.
 Service Template compute-nodes-m5-sd: Service profile template for M5 compute
servers with an SD card.
 Service Template compute-nodes-sd: Service profile template for compute servers
with an SD card.
 Service Template hx-nodes: Service profile template for Cisco UCS M4 converged
nodes.
 Service Template hx-nodes-m5: Service profile template for Cisco UCS M5
converged nodes.
 Service Template hx-nodes-m5-m2r1: Service profile template for M5 HX servers
with the M.2 raid controller.

Note
Cisco HyperFlex uses different server profiles because different configurations are
needed for different node types. For example, M4 and M5 nodes require a different
firmware configuration, and compute nodes have a different local disk policy than
converged nodes.
The templates shown here are used to create service profiles for individual servers,
depending on the generation and the role of the server in the cluster. Changing a
template will change all the service profiles created from it, because HyperFlex creates
updating service profile templates for its nodes.

Step 10

Investigate service profiles.

Answer
Choose Service Profiles. Observe the All subtab on the right.

All servers are in an Associated state, since all are in service. Other states include
the not associated state (yellow) or the error state (red).

Find the service profile Service Profile rack-unit-1 and click it. This service profile is
associated with the server that you explored earlier in this lab exercise.
The HyperFlex installer derives service profiles from templates and you can explore
what was configured on this specific server by exploring its service profile. For
example, click the Boot Order tab. You should see that server will first try to boot from
a CD/DVD. If the first try is not successful, try to boot from the embedded disk.

Note
Associate servers carefully; association will cause an immediate reboot of the server
after a reboot warning.
Investigate a LAN Configuration
Step 11

Explore the LAN configuration.

Answer
Open the LAN pane.

The LAN pane enables configuration of network elements of the devices in Cisco UCS
domain, such as VLANs, QoS, vNICs, and so on.

Step 12

Explore configured VLANs.

Answer
Expand LAN > LAN Cloud > VLANs.

The HyperFlex installer automatically created these VLANs based on the information
provided during the installation. However, these VLANs are not in use until the
HyperFlex installer applies them to vNICs.
Note
Although the default VLAN (1) is defined on the Cisco UCS domain by default, it is not
used by Cisco HyperFlex. Never use the default VLAN for any of the Cisco HyperFlex
networks.

Step 13

Explore configured vNICs.

Answer
Navigate to LAN > Policies > root > Sub-Organizations > standard-
cluster > vNIC Templates. Expand vNIC Template storage-data-a.

You will notice that this vNIC template is associated with the hx-storage-data VLAN,
which is used for Cisco HyperFlex data transport across all nodes in the group.

The vNIC templates correspond to the redundant network configuration of Cisco

HyperFlex. These vNICs are applied to each server, but have unique MAC addresses
defined from the MAC prefix, which is defined during the Cisco HyperFlex installation.
Note
If you ever use several Fabric Interconnect domains, do not use the same MAC prefixes
on different domains, as the assigned MAC addresses can overlap, causing Layer 2
connectivity issues.

The VLANs you define during the Cisco HyperFlex installation must also be configured
on the upstream switches. Cisco HyperFlex storage traffic will not leave the Cisco UCS
domain under normal circumstances, but under certain failure conditions, the traffic will
have to pass the upstream switches.

Investigate an Upstream Switch Configuration

Step 14

Connect to the CLIs of both upstream switches.

Answer
Click the blue and white icon in the top-left corner of the screen to open
the Start menu. Choose Terminal Emulator from the menu to start a terminal session.
Open two sessions so that you can connect to both upstream switches at the same time.
Note
You can drag the terminal window to the edge of the screen to maximize it over half of
the workspace (just like in Windows), so you can work with two terminals side by side.

Type ssh -l student 192.168.10.3 into one of the terminals to connect to the
first upstream switch. Log in with the password 1234QWer.

student@student-vm:~$ ssh -l student 192.168.10.3

Pre-authentication banner message from server:
| User Access Verification
End of banner message from server
Key
board-interactive authentication prompts from server:
| Password:<... output omitted ...>
93180-A#

Type ssh -l student 192.168.10.4 into another terminal to connect to the

second upstream switch. Log in with the password 1234QWer.

student@student-vm:~$ ssh -l student 192.168.10.4

Pre-authentication banner message from server:
| User Access Verification
End of banner message from server
Keyboard-interactive authentication prompts from server:
| Password:
<... output omitted ...>
93180-B#

Cisco UCS fabric interconnects, that you are investigating in this lab, connect to two
upstream switches—each Fabric Interconnect into both switches for better redundancy.

Step 15

Discover where Cisco UCS Fabric Interconnects are connected to upstream.

Answer
Use the show interface description command to identify where Fabric
Interconnects are connected to.

93180-A# show interface description

<... output omitted ...>
Eth1/1 eth 10G FI-MAIN-A E1/15 Sub-1
Eth1/2 eth 10G FI-MAIN-A E1/15 Sub-2
Eth1/3 eth 10G FI-MAIN-A E1/15 Sub-3
Eth1/4 eth 10G FI-MAIN-A E1/15 Sub-4
Eth1/5 eth 10G FI-STR-A E1/15 Sub-1
Eth1/6 eth 10G FI-STR-A E1/15 Sub-2
Eth1/7 eth 10G FI-STR-A E1/15 Sub-3
Eth1/8 eth 10G FI-STR-A E1/15 Sub-4
<... output omitted ...>

93180-B# show interface description

<... output omitted ...>
Eth1/1 eth 10G FI-MAIN-B E1/15 Sub-1
Eth1/2 eth 10G FI-MAIN-B E1/15 Sub-2
Eth1/3 eth 10G FI-MAIN-B E1/15 Sub-3
Eth1/4 eth 10G FI-MAIN-B E1/15 Sub-4
Eth1/5 eth 10G FI-STR-B E1/15 Sub-1
Eth1/6 eth 10G FI-STR-B E1/15 Sub-2
Eth1/7 eth 10G FI-STR-B E1/15 Sub-3
Eth1/8 eth 10G FI-STR-B E1/15 Sub-4
<... output omitted ...>

Servers of both Cisco HyperFlex Standard groups to two Fabric Interconnect domains
(site A and site B), each consisting of two Fabric Interconnects (A and B). The fabric
interconnects are connected to two upstream switches (93180-A and 93180-B),
specifically, the first eight ports on each switch.

The breakout cable, from 40 Gbps to 4x10 Gbps, is used to connect the Fabric
Interconnects to the upstream switches (one cable from each Fabric Interconnect).

Note
In the lab topology, each Fabric Interconnect pair is connected to a single upstream
switch. In this topology, failure of one Cisco Nexus Switch would result in a full loss of
connectivity for the entire downstream Cisco UCS domain. A more resilient
configuration would be to connect each Fabric Interconnect to each of the upstream
switches in a vPC configuration.

The less redundant option was chosen to better emulate the two sites for the Cisco
HyperFlex Stretched cluster configuration.

Step 16

Discover the upstream switch port configuration.

Answer

Use the show interface interface trunk command to discover the

configuration on the upstream switch ports that connect to fabric interconnects. The
example below features Ethernet 1/1 interface.
93180-A# show interface eth 1/1 trunk
<... output omitted ...>
Port Vlans in spanning tree forwarding state and not
pruned
----------------------------------------------------------------
----------------
Eth1/1 Feature VTP is not enabled
1,3089-3096,3191-3192,3292,3392,3492

93180-B# show interface eth 1/1 trunk

<... output omitted ...>
Port Vlans in spanning tree forwarding state and not
pruned
----------------------------------------------------------------
----------------
Eth1/1 Feature VTP is not enabled
1,3089-3096,3191-3192,3292,3392,3492

If you chose to investigate other ports (Eth1/2, Eth1/3, and Eth1/4) on both switches,
you would discover the same configuration; the ports are trunking and trunking VLANs
needed for Cisco HyperFlex management, Cisco HyperFlex storage, vMotion, and all
guest VLANs.

Investigate the MTU size configuration on one of the ports with the show
interface interface command.

93180-A# show interface eth 1/1

Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address:
003a.9c33.1b68 (bia 003a.9c33.1b68)
Description: connection to site A - node A E1/25 Sub-1
MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec

Ports should be configured for jumbo MTU frames if the Cisco HyperFlex was installed
with the jumbo frames option enabled. Jumbo frames can be enabled or disabled during
the Cisco HyperFlex installation and are enabled by default for all but the Cisco
HyperFlex Edge installations. If you enable Jumbo frames during installation but do not
enable them on the upstream switches, the HyperFlex installation will fail.

Note
Each Fabric Interconnect also has a management port connected to a management
switch in the topology. The management switch has ports configured in
the access mode for the management VLAN.

Step 17

Check the proxy Address Resolution Protocol (ARP) configuration.

Answer
Open another terminal session and use the ping 172.16.92.131 command to test if
you can ping the controller VM located on one of the nodes in the four-node Standard
Cluster. After a couple of seconds, simultaneously press the Ctrl+C keys to terminate
the command.

student@student-vm:~$ ping 172.16.92.131

PING 172.16.92.111 (172.16.92.111) 56(84) bytes of data.
^C
--- 172.16.92.111 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time
12273ms

student@student-vm:~$

You will notice that all pings failed. This is because the HX storage traffic is isolated
and not routed.

Use the arping 172.16.92.111 -I eth0 command to verify if the same

interface is responding to ARP requests. After a couple of seconds, simultaneously
press the Ctrl + C keys to terminate the command.

student@student-vm:~$ arping 172.16.92.131 -I eth0

ARPING 172.16.92.131 from 192.168.11.114 eth0
^CSent 8 probes (8 broadcast(s))
Received 0 response(s)
student@student-vm:~$

As expected, you did not receive any responses to ARP pings. This is because the
student VM that you are using is not on the same local network as the mentioned
controller VM.

However, keep in mind that if a proxy ARP feature is enabled on your router/switch you
would be able to see ARP responses.

Note
Proxy ARP is the technique in which one host, usually a router, answers ARP requests
intended for another machine. It can help machines on a subnet reach remote subnets
without the need to configure routing or a default gateway.

To verify if proxy ARP is enabled, you need to connect to the management switch.
Type telnet 192.168.10.254 into the terminal to connect to it. When prompted,
use the 1234QWer password.

student@student-vm:~$ telnet 192.168.10.254

Trying 192.168.10.254...
Connected to 192.168.10.254.
Escape character is '^]'.
User Access Verification

Password:
mgmt-sw>

Type show ip interface Vlan3090 to check if proxy ARP is enabled.

mgmt-sw>show ip interface Vlan3090

Vlan3090 is up, line protocol is up
Internet address is 192.168.11.254/24
Broadcast address is 255.255.255.255
Address determined by non-volatile memory
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing Common access list is not set
Outgoing access list is not set
Inbound Common access list is not set
Inbound access list is not set
Proxy ARP is disabled
<... output omitted ...>
mgmt-sw>

You can see that proxy ARP is disabled in this case. But keep in mind that many
switches and routers have this feature enabled by default which can cause problems
with the Cisco HyperFlex installations and upgrades. It is recommended that proxy ARP
is set to disabled if not required by a specific use case.

5.1 Installing and Expanding Standard ESXi Cisco HyperFlex

Introduction
The Cisco HyperFlex solution exists in different deployment types: ESXi Edge, ESXi
stretched, Hyper-V standard, and ESXi standard. Most of the installations in the field
are standard ESXi HyperFlex deployments. A standard HyperFlex deployment refers to
Fabric Interconnect-based deployments that are not stretched.

Here you will learn the steps to install a Standard ESXi deployment using an on-
premises (on-prem) installer. Once you are familiar with the installation of a Standard
ESXi deployment, you will quickly understand installations of other HyperFlex
installation types.
You can use two types of installer to install a HyperFlex system: an on-prem installer in
the form of a VM and a cloud-based installer that is a hosted service by Cisco and can
be found at https://www.intersight.com. As of HXDP Version 4.5(1a), the on-prem
installer is still the most popular option with Cisco Intersight installation gaining
significant momentum.

Since the on-premises installer is the more extensive of the two installation options, it is
covered here. The workflow in Cisco Intersight is very similar, with different user
interface and no need to deploy the installer VM. Instead, the Cisco UCS Fabric
interconnects are claimed to Cisco Intersight, which also adds the servers attached to
these Fabric Interconnects.

Note
All the HyperFlex deployment considerations from the on-premises workflow also
apply to the Intersight-based deployment. The final HyperFlex installation is equivalent
in both cases.

5.2 Installing and Expanding Standard ESXi Cisco HyperFlex

Installation Summary

This video provides an installation summary as well as software and hardware prerequisites.
While the installation of Cisco HyperFlex itself is pretty straightforward, you must properly
understand the sequence needed to install a cluster. First, check the prerequisites. The
preinstallation checklist for Cisco HyperFlex Data Platforms is a template document that you
can follow to choose the appropriate variables for the installation.
Next, you need to rack the fabric interconnects and connect them to the upstream switches.
Then you configure the upstream switches. Now you're ready to create the UCS cluster,
connect the fabric interconnect ports, upgrade the infrastructure. And afterwards, deploy the
HyperFlex Installer VM and a vCenter. You can use an existing vCenter Server if it's already
configured.

Next, you run the wizard to install HyperFlex and perform the postinstallation tasks. Lastly,
verify the installation. Make sure that you're meeting the software version prerequisites
before you proceed with the installation. This list here in the figure cautions you to make sure
that you understand the takeaways for the recommended HyperFlex Installer version, the
Cisco UCS version, and the vSphere version as well as the edition. Keep in mind that different
versions of vSphere, Cisco HyperFlex software, and hardware firmware must be compatible
with each other for the installation to succeed.
Since Cisco HyperFlex version 3.0, you will find two installers, one for ESXi and the other for
Hyper-V clusters. All cluster types, regular, stretched, and HyperFlex Edge are supported from
one single installer virtual machine. The HyperFlex Installer is a virtual machine that Cisco
distributes in an OVA format. Now, you download the installer from software.cisco.com.

When you're on the website, the recommended version is marked with a star. Don't forget to
carefully read the release notes to determine if the recommended version is suitable for your
system. You will not be able to download the software if you don't have a valid contract for
HyperFlex on your CCO account or if it's just not properly associated.

Next, the HyperFlex software and Cisco UCS firmware must be compatible. This figure here
lists the requirements and caveats for firmware. Be sure to carefully read the release notes
and installation guide for the latest information. The firmware versions are not statically
defined for a HyperFlex release, and they can change at any point. Always consult the
documentation for the latest and greatest information.
This figure describes vSphere versions and editions. Keep in mind, each vSphere release is
compatible with specific versions of vCenter and ESXi. Also, you must verify that all HX servers
have a compatible version of vSphere preinstalled. Lastly, verify that you have a vCenter
administrator account with root-level privileges, because you've got to have those privileges
before you can actually begin.

As you can see, the licensing for Enterprise, Enterprise Plus, Standard, Essentials Plus, and
remote offices and branch offices supported for vSphere on a standard ESXi—based cluster.
How you purchase the vSphere licensing determines how it applies to the HyperFlex system.

The list in the figure describes the details for the following scenarios—if you purchased your
vSphere license with HyperFlex, if you did not purchase the vSphere license with HyperFlex,
and if you purchased your vSphere license from Cisco without a HyperFlex system. Be sure to
check each one of these to make sure you understand them carefully.
vCenter is the management platform of the vSphere environment and is a vital part of the
vSphere-based Cisco HyperFlex installation. This is because regular, stretched, and edge ESXi-
based clusters require vCenter to run. Now, vCenter should be on an external server to the
HyperFlex cluster. And it could be one that's already been installed. However, sometimes that
may not be possible. If you can't have an external server for vCenter, you can always install it
on the HyperFlex storage cluster. This is what's referred to as a nested vCenter. Now, be aware
that if you nest vCenter on HyperFlex, its use will be limited.

Once again, if you don't have a vCenter Server, you should deploy—I mean, if you don't have
the vCenter Server, then you should just go ahead and deploy it outside the HyperFlex cluster.
If it's already present, you've already got one, then you should double-check to make sure it
meets the HyperFlex interoperability requirements. Got to make sure it's the right version.

Make sure that vCenter version is equal to or higher than the ESXi version on the HyperFlex
node. Verify that vCenter meets the requirements to manage all the host and VMs in the
environment-- tiny, small, medium, and large. And lastly, verify that the TCP ports used by
HyperFlex are not used by any other third-party software or plugins.
This next figure here lists hardware prerequisites, which consist of component availability, rack
space, cooling, cabling. The image shows a four-node Cisco HX240, all flash HyperFlex cluster.
Now, for this cluster, you need 12 rack units of space. Two rack units for each server, one rack
unit for each fabric interconnect, and one rack unit for each upstream switch. Although there's
no requirement for contiguous rack space, it just makes the installation easier if you do that.

Finally, this last figure here lists additional hardware prerequisites for the fabric interconnect
base cluster. This includes the power cord and power circuit requirements. You can always
consult with the Cisco UCS power calculator on cisco.com. This can be used to verify that you
meet the required based on your configuration.

Also, the figure lists the twinax cable requirements for connecting each HyperFlex node. A
minimum of two cables are required. And then lastly, the figure details the number and type
of uplink connections for the fabric interconnect. That brings us to the end of this video.
Thanks for watching.

While the installation of Cisco HyperFlex itself is relatively straightforward, you must
understand the sequence of steps to install it.
Note
Here you will learn how to install a standard ESXi-based deployment using Cisco
HXDP Version 4.5(1a) and Cisco UCS Version 4.1(2b). While the installation
procedure has been nearly the same since Cisco HXDP Version 1.8(1a), there are subtle
differences. It is recommended that you use the installation guide that is specific to the
version of Cisco HXDP that you are installing. The installation guide can be found on
the Cisco website.

Remember these steps for a standard ESXi-based HyperFlex installation through an on-
premises installer:

1. Check prerequisites.
2. Rack Fabric Interconnects and servers.
3. Connect servers to Fabric Interconnects and connect Fabric Interconnects to
switches.
4. Configure upstream switches.
5. Create UCS domain, configure Fabric Interconnect ports, and upgrade the
infrastructure.
6. Deploy a HyperFlex installer VM and a vCenter (or use the existing one).
7. Run the wizard to install HyperFlex.
8. Perform post-installation tasks.
9. Verify installation.

The best resource for installation of a standard ESXi-based HyperFlex deployment is

the Cisco HyperFlex Systems Installation Guide for VMware
ESXi at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
 HyperFlex_HX_DataPlatformSoftware/Installation_VMWare_ESXi/4-5/b-hx-install-
guide-for-vmware-esxi-4-5.html.

The Preinstallation Checklist for Cisco HyperFlex Data

Platform (https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/HyperFlex_Preinstall_Checklist/
b_HX_Data_Platform_Preinstall_Checklist.pdf) is a template document that you can
follow to choose appropriate variables for the installation. Download it, print it, and use
it to document settings before you proceed with your installation.

With the HyperFlex version 4.5(1a), a new interactive tool was introduced that allows
preparing system configurations before installation. The HX Pre-install tool is available
on Cisco.com https://hxpreinstall.cloudapps.cisco.com/#/deployments.

You can push the configuration prepared in the Pre-installation tool to Intersight and
create a HyperFlex installation profile from the configuration. You can also prepare the
installation profile in Cisco Intersight.

Comparison of Steps in On-Premises and Intersight Installation

Installer VM Installation Intersight Installation

Review the prerequisites. Review the prerequisites.

Rack and stack the equipment. Rack and stack the equipment.

Configure the Fabric Interconnects. Configure the Fabric Interconnects.

Configure the upstream network and Internet

Configure the upstream network.
connectivity.

Deploy an installer VM on the infrastructure. Register UCS Manager and servers to Intersight.

Run the installation wizard on the Installer VM. Run the installation wizard on Intersight.

Perform the post-installation tasks. Perform the post-installation tasks.

HyperFlex is registered to vCenter and ready to

HyperFlex is registered to vCenter and ready to use.
use.

5.3 Installing and Expanding Standard ESXi Cisco HyperFlex

Software Prerequisites
Make sure that you are meeting the software version prerequisites before you proceed
with the installation. Keep in mind that different versions of vSphere, Cisco HXDP
software, and hardware firmware must be compatible with each other.

Before proceeding with Cisco HyperFlex installation, make sure that you understand the
following:

 Recommended HyperFlex installer version.

o You would use the same installer for any ESXi-based deployment (standard,
stretched, and Edge).

 Recommended Cisco UCS firmware version.

o Cisco UCS firmware consists of software/firmware bundles: A (infrastructure),

B (blades), and C (rack servers).

 Recommended vSphere version and edition.

o Applicable to ESXi and vCenter versions

Cisco HyperFlex on-prem installer is a VM that is deployed as an OVA in a vSphere

environment that you run to install the initial HyperFlex deployment or expand it with
additional nodes. An extra vSphere environment is needed for the Installer VM
deployment.

VMware vCenter is deployed in a vSphere environment as the vSphere management

platform, but it is also a vital part of an ESXi-based HyperFlex deployment installation.
For the most automated HyperFlex installation, you will need to preinstall vCenter on a
separate infrastructure.

The vCenter deployment is optional when installing HyperFlex. You can choose to
install HyperFlex without vCenter and then register the HyperFlex hosts separately.
This enables a nested vCenter installation.

Cisco UCS firmware is a software bundle that Cisco UCS Fabric Interconnects use to
run Cisco UCS Manager and servers (B-Series, C-Series, or HyperFlex). Server bundles
include firmware updates to BIOS, chipset, controllers, and other hardware components,
such as blade chassis.

Cisco HyperFlex Installer Version

Since Cisco HyperFlex Version 3.0, you will find two installers: one for ESXi and the
other for Hyper-V deployments. All ESXi deployment types (standard ESXi, stretched,
and HyperFlex Edge) are supported from the same ESXi installer VM, while a separate
installer VM is used for Hyper-V and is available since HXDP Version 3.0.
Note
The only HyperFlex deployment type that you cannot install using the on-prem installer
is the 2-Node HyperFlex Edge, because it relies on Intersight to provide it with an
arbitration witness in case the communication between the HyperFlex nodes goes down

HyperFlex installer is a VM that Cisco distributes in an OVA format:

 Get the installer from http://software.cisco.com.

 On the site, the recommended version is marked with a star. Carefully read the
release notes to determine if the recommended version is suitable for your system.

Cisco HyperFlex is a rapidly evolving product, so Cisco releases HXDP software

updates at frequent intervals. When installing your HyperFlex deployment, check the
latest information on the recommended version at http://software.cisco.com. You will
not be able to download the installation software if you do not have a valid contract with
Cisco for HyperFlex or if your Cisco.com account is not properly associated.

If the version of HXDP that you are looking for is not listed on the download page, it is
probably not supported anymore, because it was deferred due to a bug or the natural
lifecycle of software support has expired.

Cisco UCS Firmware Version

Cisco HXDP software and Cisco UCS firmware must be compatible:

 Make sure that Cisco HyperFlex servers have a desired version from the factory or
upgrade the infrastructure Cisco UCS firmware before installation.
 M4 and M5 servers may have different recommended software versions.
 Deployments with SEDs may have different recommended software versions.
 Consult with release notes and the installation guide for the latest information.
It is recommended that you upgrade servers to the target firmware version ahead of the
HyperFlex installation process. The firmware upgrade process can sometimes take
significant time (more than 2 hours), causing the HyperFlex installer to time out.

The following table applies to Cisco UCS M5 and M4 and shows the versions of the
HXDP installer and recommended Cisco UCS firmware. The following table only
shows a few of the Cisco HXDP versions that you might be interested in deploying.

M4 Recommended UCS Fabric M5 Recommended UCS Fabric

HyperFlex Release
Interconnect Firmware Interconnect Firmware

4.5(1a) 4.0(2h) 4.1(2b)

4.0(2e) 4.0(4k) 4.0(4k)

3.5(2i) 4.0(4k) 4.0(4k)

3.0(1i) 3.2(3h), 3.1(3j) 3.2(3h)

Note
Compatible versions can change rapidly and are provided only as an example of
compatible versions at a point in time. The firmware versions are not statically defined
for a Cisco HyperFlex release and can change. Always consult the documentation for
the latest information.

vSphere Versions and Editions

Each release is compatible with specific versions of vCenter and ESXi:

 HXDP 4.5(1a) supports ESXi Versions 6.5 U3, 6.7 U3, and 7.0 U1d.

o vCenter Versions 6.5 U3, 6.7 U3, and 7.0 U1d are supported and the vCenter
build number must be the same or higher than vSphere ESXi.

 Verify all HyperFlex servers have a compatible version of vSphere installed.

o For example, if the servers have ESXi 6.5 U3 installed and you want to run 6.7
U3, then reimage the servers before installation or upgrade ESXi after
installation.

 Verify that you have a vCenter administrator account with root-level privileges and
the associated password.

ESXi images used for re-imaging the HyperFlex servers need to be Cisco Custom.
These images properly set up the ESXi hypervisor for HyperFlex installation. You can
download images from http://software.cisco.com.

HyperFlex Release ESXi Versions vCenter Versions

4.5(1a) 6.5 U3, 6.7 U3, 7.0 U1d 6.5 U3, 6.7 U3, 7.0 U1d

4.0(1a) 6.0 U3, 6.5 U3, 6.7 U3 6.0 U3, 6.5 U3, 6.7 U3, 7.0 U1d

3.5(2i) 6.0 U3, 6.5 U3, 6.7 U3* 6.0 U3, 6.5 U3, 6.7 U3

3.0(1i) 6.0 U3, 6.5 U1, 6.5 U2 6.0 U3, 6.5 U1, 6.5 U2

Note
There is no HXDP Flash plug-in for vCenter 6.7 U2, since the support for Flash user
interface was dropped, 6.7U3 had the Flash support returned, but Flash support is not
available with vSphere 7.0 or above.

Enterprise, Enterprise Plus, Standard, Essentials Plus, and remote office-branch office
(ROBO) licensing is supported for vSphere on a standard ESXi-based HyperFlex
deployment.
How you purchase your vSphere license determines how your license applies to your
Cisco HyperFlex system:

 If you purchased your vSphere license with HyperFlex: Each HyperFlex server has
the vSphere license preinstalled at the factory OEM or licenses are delivered as
Product Authorization Keys (PAKs).
 If you did not purchase your vSphere license with HyperFlex: The HyperFlex nodes
have a vSphere Foundation license preinstalled. After initial setup, the licenses can
be applied to a supported version of vSphere.
 If you purchased your vSphere license from Cisco without a HyperFlex
system: Contact Cisco TAC for a spare vSphere license at no additional cost.

Consider the following details if you bought licenses from Cisco:

 HyperFlex nodes have OEM licenses preinstalled. If you delete or overwrite the
content of the boot drives after receiving the HyperFlex servers, you also delete the
factory-installed licenses.
 All factory installed HyperFlex nodes share the same OEM license key. With
vSphere OEM keys, the Usage count can exceed the Capacity value.
 When you add a HyperFlex host to vCenter through the Add Host wizard, in the
Assign license section, choose the OEM license.
 Standard, Essentials Plus, and ROBO editions are not available preinstalled on
HyperFlex servers.
 Since Version 2.6(1a), HyperFlex supports VMware PAC licensing. Existing
VMware embedded licenses will continue to be supported.

vCenter for Cisco HyperFlex

vCenter is the management platform of the vSphere environment and a vital part of the
ESXi-based Cisco HyperFlex installation.

vCenter has two main roles In ESXi-based HyperFlex deployments:

 Management of VMs (deploy, start, stop, configure storage, configure network, and
so on) and VMware group (high availability, DRS, start order rules, and so on).
 Management of HyperFlex as software-defined-storage through HyperFlex plug-in
in vCenter.

ESXi-based deployment (Standard, Stretched, Edge) requires vCenter to run.

 Having vCenter on a server that is external to the HyperFlex system is

recommended.
 If you cannot have an external server to the vCenter server application, install a
vCenter VM on the HyperFlex storage system (Nested vCenter).
Deploy vCenter outside of the HyperFlex system if one is not present. If a vCenter is
present, double-check that it meets HyperFlex interoperability requirements:

 vCenter version must be equal to or higher than the ESXi version on the HyperFlex
node.
o Administrator-level credentials are needed for installation of HyperFlex
and subsequent management of HyperFlex through vCenter.

 vCenter meets the requirements to manage all the hosts and VMs in the
environment: tiny, small, medium, and large.
 TCP ports used by HyperFlex are not used by other third-party software or plug-ins
(for example, Port 8089, Port 80 for vCenter on Windows).

Use the local single sign-on (SSO) account during the HyperFlex installation. The local
SSO account has root-level privileges to vCenter, and it is a member of
the SystemConfigurationAdministrators group by default.

5.4 Installing and Expanding Standard ESXi Cisco HyperFlex

Hardware Prerequisites
There are specific hardware configuration and deployment considerations that need to
be taken into account when deploying Cisco HyperFlex. These considerations need to
be fulfilled for a successful hardware installation and later a successful installation of
Cisco HyperFlex.

Hardware prerequisites consist of component availability, rack space, cooling, and

cabling.

Keep in mind these rack space and cooling considerations:

 HX220c nodes are 1 RU each; for example, for a 3-node deployment, 3 RU are
required.
 HX240c nodes are 2 RU each; for example, for a 3-node deployment, 6 RU are
required.
 FIs are 1 RU each, so you need 2 RU of space.
 If ToR switches need to be installed, usually you need 2 RU for 2 switches.
 Consider airflow:

o Mount FIs with ports on the same side as the ports of servers.
 o The FIs exhaust air on the side with ports, which is the rear.

The figure shows a 4-node Cisco HX240c M5 All-Flash HyperFlex deployment. For
such a deployment, you need 12 RU of space: 2 RU for each server, 1 RU for each
Fabric Interconnect, and 1 RU for each upstream switch.

Note
The figure shows the Fabric Interconnects facing in the wrong direction. In images,
Fabric Interconnects are presented this way so that they are recognizable. In an actual
deployment, the Fabric Interconnect power supplies would be facing the same way as
server front bezels visible in the image.

Although there is no requirement for contiguous rack space, it makes the installation
easier, since it reduces cabling clutter and cable length considerations. Cables that are
not optimally routed can interfere with airflow and cause overheating.

Due to cost-effectiveness, you will most often use copper twinaxial (Twinax) cables to
connect the devices, which are limited to up to 10 m in length. You will need to use
more expensive optical cabling if the devices are farther than 10 meters apart.

Note
Twinax cables up to 5 meter in length are passive cables and are even more cost
effective. Twinax cables above 5 meters in length are active cables and are priced
accordingly.
The following are some additional airflow considerations:

 Maintain ambient airflow throughout the data center to ensure normal operation.
 Consider the heat dissipation of all equipment when determining air conditioning
requirements. When evaluating airflow requirements, take into consideration that
hot air generated by equipment at the bottom of the rack can be drawn in the intake
ports of the equipment above.
 Ensure that the intake and exhaust airflow is unobstructed.
Keep the following in mind regarding Fabric Interconnect deployments:
 The system requires two C13/C14 power cords connected to a 15 A circuit, per
device in the HyperFlex deployment. At a minimum, three Cisco HyperFlex nodes
and two Fabric Interconnects are in a Standard deployment.
 Each HyperFlex node requires two Twinax cables for connectivity.

o Optionally, you can use 4 Twinax cables per node for connectivity with VIC
1457.
o Optionally since HXDP 3.5(1a), you can use more than one NIC for redundancy,
but aggregate throughput will not increase, only resiliency.

 Use 2 to 4 uplink connections for Cisco UCS Fabric Interconnects:

o With the 6300 series Fabric Interconnect, use 1–2x 40G uplinks per Fabric
Interconnect and connect each HyperFlex node to Fabric Interconnects with two
40G Twinax cables.
o With the 6400 series Fabric Interconnect, use 1–2x 40G or 100G uplinks per
Fabric Interconnect and connect each HyperFlex node to Fabric Interconnects
with two or four 10G or 25G Twinax cables.

When you are planning to deploy Cisco HyperFlex, do not forget to consult the Cisco
Power Calculator at https://ucspowercalc.cloudapps.cisco.com/public/index.jsp.

Consider that Fabric Interconnects usually have a finite number of licensed ports pre-
bundled. UCS 6332 includes eight 40-Gbps port licenses. UCS 6332-16 includes four
40-Gbps and 8 Unified port licenses. UCS 6454 includes 18x10/25 Gbps and 2x40/100
Gbps licenses and UCS 64108 includes twice that amount. If you want to use more than
the number of base licensed ports, you have to buy additional port licenses.

5.5 Installing and Expanding Standard ESXi Cisco HyperFlex

Cisco HyperFlex Networking

Let's take a closer look at HyperFlex networking and some of the required deployment
information. The distributed nature of Cisco HyperFlex dictates that the networking
considerations are the key components of the HyperFlex operation.

Network considerations are a key component because of its distributed architecture. The
HyperFlex system is split into several VLANs which are assigned to specific traffic. The installer
installs four vNICs and assigns port groups in the vCenter accordingly.

The groups include HX in-band management for management purposes, HX storage data for
HyperFlex storage devices, vMotion for vMotion between ESXi hosts in the cluster, and, lastly,
HX VM network for the guest VLANs.

The Cisco HyperFlex installer automatically creates vSwitches and assigns the interfaces to
port groups. It's important to remember all storage traffic traverses the fabric interconnects.
However, the storage traffic is never passed to the upstream switches.
There is one exception, one exception for the storage not traversing the upstream switches.
And this is really rare if this occurs. When link A fails on one host and link B fails on another
host, then the fabric interconnect has to pass the storage traffic up to the upstream switches.

Now, for this to work successfully, the upstream switches must have high throughput and
sufficient links to prevent a drop in performance. This includes configuring a storage VLAN and
jumbo MTU on the upstream switches. Once again, it's very rare that this event occurs.

Also, make sure all appropriate ports that are open if your network is behind the firewall.
Right? You don't want to block any ports. This is to ensure that the HyperFlex installation and
subsequent operations will be successful.

If you have a whitelisted environment where you explicitly allow certain traffic, the table in the
figure here lists the services that you must verify that are open. This list shows the port
requirements for regular ESXi base clusters. And additional ports will apply for other types of
installations.
This next figure lists some network best practices, such as separate subnets and VLANs for
each network, 10 or 40 gig cables between each host and the fabric interconnects. Never use
the VLAN 1, which is the default VLAN.

The installer sets the VLANs as non-native by default. Make sure that all the upstream switches
will accommodate non-native VLANs. And lastly, you will need to manually configure high
availability. DRS and vMotion configuration is optional, but it is recommended.

Now, before deploying Cisco HyperFlex, make sure that you've met all the prerequisite
information listed in the figure. The preinstallation checklist can help you assemble this
information, which includes the cluster name, the three IP addresses, the virtual IP for the
cluster, and an IP address on each fabric interconnect.

Remember, you browse to the virtual IP. A large pool of KVM IP addresses sufficient for all
possible nodes in the cluster-- you never want to run out. So make sure you've got plenty
of addresses when you create your pools.
For future growth you should reserve more IP addresses than you need for the initial cluster
deployment. For simplicity, Cisco UCS Manager and HyperFlex management IP space can be in
the same pool. And lastly, choose the admin password for UCS Manager.

Now, this figure here lists some VLAN requirements for deployment. This includes a VLAN for
VMware ESXi and Cisco HyperFlex management, HyperFlex storage traffic, VM VMware
vMotion, and VLANs for the VM guest networks.

And then this next figure lists the IP address ranges required for servers in a regular ESXi
cluster. This includes the management network IP address requirements as well as the data
network IP address requirements. And also keep in mind, management and data must have
different IP address ranges.
This next figure lists the services that are needed for a regular ESXi cluster. It includes a
vCenter IP address with fully qualified domain name along with the username and password,
as well as both a primary and secondary IP address for the DNS and the NTP services.

Finally, here's some other information required for HyperFlex cluster deployment. You've got
to have the root password on the ESXi hosts, configure a MAC address pool for the HyperFlex
cluster.

Be sure to use the required format as shown here--and, optionally, enabling the connected
services as well as email for service request notification. That brings us to the end of this video.
Thanks for watching.

The distributed nature of Cisco HyperFlex dictates that the networking considerations
are a key component of the HyperFlex operation. For security and segmentation
reasons, the networks that are used by the HyperFlex system are split into several
VLANs that are dedicated to specific traffic. Different components of the HyperFlex
solution have specific network requirements for their operation.

Note
The networking considerations along with proper network addressing and setup are
typically the most difficult part of the first HyperFlex system installation.
Each ESXi host needs the following networks:

 Management traffic network: From the vCenter, handles the hypervisor (ESXi
server) management, and HyperFlex storage management.
 Data traffic network: Handles the hypervisor and storage data traffic.
 vMotion network: Handles live migration of VMs between nodes in the group,
or to/from outside the server group.
 VM network: VLANs for your guest VMs. Configured in active/active mode.

HyperFlex installer installs 4 vNICs in Cisco UCS Manager and corresponding port
groups in the vCenter for each fabric:

 hx-inband-mgmt: For management of ESXi hosts and CVMs.

 hx-storage-data: For HyperFlex storage on CVMs and ESXi hosts.
 vmotion: Dedicated vNIC for vMotion between ESXi hosts in the group.
 hx-vm-network: vNIC for guest VLANs.
By default, the hx-vm-network vSwitch is configured as active/active. All other
vSwitches are configured as active/standby.

vNICs and port groups are automatically created with a "vlan" suffix:

 All "a" connect to Fabric Interconnect-A and all "b" connect to Fabric Interconnect-
B.
 MTU of 9000 (jumbo frames) needed for storage-data and vmotion networks.
 By default, all VLANs are tagged.

Note
You can add or remove VLANs on the corresponding vNIC templates in Cisco UCS
Manager, and all the service profiles that are associated with the template will update
automatically.

The Cisco HXDP Installer automatically creates the vSwitches and assigns the
interfaces to port groups.

Remember these takeaways regarding traffic flow under normal operations:

 All storage traffic goes through Fabric Interconnects.

 No storage traffic passes upstream switches.
The default vSwitch NIC teaming policy and failover policy are set to yes, to ensure
that all management, vMotion, and storage traffic are locally forwarded to the Fabric
Interconnects and keep the flow in steady state. When vNIC-a fails, ESXi computes the
load balancing and all the virtual ports are re-pinned to vNIC-b. When vNIC-a comes
back online, repinning is applied and virtual ports are again distributed across vNIC-a
and vNIC-b, which reduce the latency and bandwidth utilization upstream of the Cisco
UCS Fabric Interconnects.

In most failure scenarios, the VLANs will still be able to terminate within the same
fabric and no traffic will need to pass both Fabric Interconnects. In these cases, the
upstream switches will still not be used for HyperFlex storage traffic.

In the rare event that Link A fails on one host, and Link B fails on another host, the
traffic can no longer reach the destination only through one Fabric Interconnect:

 Storage traffic now passes upstream switches.

 To protect against drop in performance, ensure that upstream switches are high
throughput and enough links between Fabric Interconnects and switches.
 To protect against failures, make sure that storage VLAN with jumbo MTU is
defined on upstream switches.
If Host 1 loses connectivity to Fabric A, while Host 2 loses connectivity to Fabric B, the
traffic must go through the upstream switches to allow communication between hosts
experiencing the failures.

If you forget to configure storage VLAN on Switch A or Switch B, you would initially
not see any problems in the functionality or performance of the system until the specific
described multiple failures happen. If the communication through the upstream switches
is not configured, certain servers will not be able to communicate and would be marked
as failed. Depending on the number of simultaneous failures, this can lead to a full
storage system down event.

Port Requirements

Certain parts of a HyperFlex system can communicate over a Layer 3 routed network.
This is most commonly the case with the management network.

If your Layer 3 configuration is firewalled, make sure that all appropriate ports are open
so that the HyperFlex installation and subsequent operations are flawless.

Note
Additional ports apply when you are installing a stretched deployment or configuring
native replication between two HyperFlex systems. Entirely different ports apply for
Hyper-V based installations. This topic only discusses port requirements for Standard
ESXi-based installation.

If you have an allow-listed environment, make sure that you open ports for the
following services:

 Time server
 HXDP installer
 Mail server
 Monitoring
  DNS server
 vCenter
 User
 SSO server
 UCS Manager

Note
For the latest information regarding required ports, refer to the installation guide for the
Cisco HXDP version that you are looking to install on your HyperFlex System.

Time Server Port Requirements

Port
Service/Protocol Source Port Destinations Essential Information
Number

Each ESXi Node Each SCVM Node

123 NTP/UDP Time Server Bidirectional
UCSM HX Data Platform Installer

HXDP Installer Port Requirements.

Port Essential
Service/ Protocol Source Port Destinations
Number Information

Each ESXi Node

Management
addresses
Each SCVM Node

22 SSH/TCP HX Data Platform installer

CIP-M Group management

UCSM management
UCSM
addresses
 Time Server Port Requirements
Port
Service/Protocol Source Port Destinations Essential Information
Number

Each ESXi Node

Management
addresses
Each SCVM Node

80 HTTP/TCP HX Data Platform installer

CIP-M Group management

UCSM management
UCSM
addresses

Management
Each ESXi Node
addresses

Management
Each SCVM Node
addresses
443 HTTPS/TCP HX Data Platform installer

CIP-M Group management

UCSM management
UCSM
addresses

Management
8089 vSphere SDK/TCP HX Data Platform installer Each ESXi Node
addresses

902 Heartbeat/UDP/TCP HX Data Platform installer vCenter

Internet Control
Management
None Message Protocol HX Data Platform installer ESXi IPs CVM IPs
addresses
(ICMP)

9333 UDP/TCP HX Data Platform installer CIP-M Group Management

Mail Server Port Requirements

Essential
Port Number Service/ Protocol Source Port Destinations
Information

25 SMTP/TCP Each SCVM Node CIP-M UCSM Mail server Optional

Requirements for Monitoring UCS Infrastructure

Essential
Port Number Service/ Protocol Source Port Destinations
Information
 Mail Server Port Requirements
Essential
Port Number Service/ Protocol Source Port Destinations
Information

161 SNMP Poll/UDP Monitoring server UCSM Optional

162 SNMP Trap/UDP UCSM Monitoring server Optional

Name Server Port Requirements

Port
Service/ Protocol Source Port Destinations Essential Information
Number

Each ESXi
Name Server Management address
Node

Each
SCVM Name Server Management address
53 DNS/TCP/UDP Node

CIP-M Name Server Group Management

UCSM Name Server

vCenter Port Requirements.

Port
Service/ Protocol Source Port Destinations Essential Information
Number

80 HTTP/TCP vCenter Each SCVM Node CIP-M Bidirectional

Each ESXi Node Each

443 HTTPS (Plug-in)/TCP vCenter Bidirectional
SCVM Node CIP-M

HTTPS (VC Each ESXi Node Each

7444 vCenter Bidirectional
SSO)/TCP SCVM Node CIP-M

Each ESXi Node Each

9443 HTTPS (Plug-in)/TCP vCenter Bidirectional
SCVM Node CIP-M

5989 CIM Server/TCP vCenter Each ESXi Node Bidirectional

9080 CIM Server/TCP vCenter Each ESXi Node Introduced in ESXi 6.5

This port must be accessible from

HX Data Platform each host. Installation will result in
902 Heartbeat/TCP/UDP vCenter
Installer ESXi servers errors if the port is not open from the
HX Installer to the ESXi hosts.
 User Port Requirements
Port
Service/ Protocol Source Port Destinations Essential Information
Number

Each ESXi Node Management addresses

Each SCVM Node Management addresses

CIP-M Group management

HX Data Platform
22 SSH/TCP User Installer

UCSM management
UCSM
addresses

vCenter

SSO Server

Each SCVM Node Management addresses

CIP-M Group management

UCSM
80 HTTP/TCP User

HX Data Platform
Installer

vCenter

Each SCVM Node

CIP-M

UCSM management
UCSM
443 HTTPS/TCP User addresses

HX Data Platform
Installer

vCenter

7444 HTTPS (SSO)/TCP User vCenter SSO Server

 User Port Requirements
Port
Service/ Protocol Source Port Destinations Essential Information
Number

9443 HTTPS (plug-in)/TCP User vCenter

Virtual
UCSM management
2068 keyboard/video/mouse/(vKVM) User UCSM
addresses
/TCP

SSO Server Port Requirements.

Port
Service/ Protocol Source Port Destinations Essential Information
Number

SSO Each ESXi Node Each SCVM Node

7444 HTTPS (SSO)/TCP Bidirectional
server CIP-M

UCS Manager Port Requirements

Port Number Service/ Protocol Source Port Destinations Essential Information

Bidirectional for each

443 Encryption etc./TCP Each CVM Node CIMC OOB
UCS node

81 KVM/HTTP User UCSM OOB KVM

743 KVM/HTTP User UCSM OOB KVM encrypted

Cisco Intersight Port Requirements

Port Number Service/ Protocol Source Port Destinations Essential Information

svc.intersight.com
80 Encryption etc./TCP Each CVM UCSM Bidirectional
(54.164.186.11)

svc.intersight.com
443 Encryption etc./TCP Each CVM UCSM Bidirectional
(54.164.186.11)

The listed requirements for Intersight-based installation are listed for direct Internet
access. The same requirements apply when using an HTTP proxy, with the source being
your HTTP proxy. Do not forget that all the listed systems will still need access to the
HTTP proxy over the Management network. Ports depend on your proxy configuration.
The usual HTTP proxy ports are 3128 and 3129.

Note
Requirements for Auto Support, and post installation script are covered in other sections
of this course.
Networking Best Practices
Remember the following networking best practices:

 Use separate subnet and VLANs for each network.

 Directly attach each host to a Cisco UCS Fabric Interconnect using:

o Two 40G cables if using VIC 1387.

o Two or four cables if using VIC 1457. You can use 10G or 25G cables.

 Do not use VLAN 1 (the default VLAN), as it can cause networking issues,
especially if you use a disjoint Layer 2 configuration.
 Installer sets the VLANs as non-native by default. Ensure to configure the upstream
switches to accommodate the non-native VLANs.
 Cisco HXDP automatically creates vSwitches, but you will need to configure DRS
separately (optional), vMotion (optional), and high availability.

o For the highest degree of automation of your HyperFlex deployment, having

DRS and vMotion configured is recommended.

Note
When you have uplinks from a Fabric Interconnect connected to two different upstream
switches, you may encounter a condition called disjointed Layer 2. This situation occurs
when the Fabric Interconnects are running in the end host mode and the upstream
network is not configured properly. Refer to Deploy Layer 2 Disjoint Networks
Upstream in End Host Mode at https://www.cisco.com/c/en/us/solutions/collateral/data-
center-virtualization/unified-computing/white_paper_c11-692008.html.

5.6 Installing and Expanding Standard ESXi Cisco HyperFlex

Required Deployment
Information
Before deploying Cisco HXDP, make sure that you have all the prerequisite information
on hand. A preinstallation checklist can help you assemble this information in a concise
manner.

Here is the information needed to set up the Cisco UCS domain and (later) provide
KVM connectivity to hosts:

 Cisco UCS system name

 3 IP addresses: 1 for the VIP and 1 for each Fabric Interconnect

o Including subnet mask and default gateway IP

 Pool for KVM IP addresses—one per Cisco HyperFlex node is required. Take
possible scaling into consideration.
 Cisco UCS management IP space can be part of HyperFlex management IP space
for simplification of addressing.

o Cisco UCS IP management cannot be isolated from HyperFlex management

since HXDP controllers need to talk to Cisco UCS.

 Desired password for Cisco UCS Manager user admin.

It is recommended that you reserve more KVM IP addresses than you need for initial
deployment. If you start, for example, with a 5-node installation and assign a pool of
only 5 IP addresses for your KVM connectivity, you will run into trouble when
expanding your deployment, because you did not reserve enough IP addresses
additional nodes.
Use separate subnet and VLANs for each of the following networks:

 VLAN for VMware ESXi and Cisco HyperFlex management

o Used for management traffic among ESXi, HyperFlex, and VMware vCenter;
must be routable.

 VLAN for HyperFlex storage traffic

o Used for storage traffic and requires Layer 2. Needs to exist on upstream
switches.

 VLAN for VM VMware vMotion

o Used for vMotion VLAN, if applicable

 VLANs for VM guest networks

o Used for VM/application network

You will need to decide on VLAN IDs for your HyperFlex deployment.

While storage traffic is local to the UCS domain, in a worst-case scenario over ToR
switches, you should choose a VLAN in your network that is not used elsewhere for
clarity and documentation purposes. The same applies to IP addresses for the storage
VLAN range.

You can choose not to define any VLANs for a guest VM/application network. You can
later configure additional VLANs after installation. This can be done manually through
vSphere management or by using the post_install script available on the installer
VM.

The following IP addresses ranges are needed for servers in regular ESXi deployment:

 Management network IP address

 o 2 per server (1 for ESXi, 1 for CVM) + 1 for HyperFlex deployment and 1 for
Installer VM.
o Needs to be Layer 3 IP space with connectivity to vCenter, HyperFlex Installer,
NTP, DNS, and other services.

 Data network IP address

o 2 per server (1 for ESXi, 1 for CVM) + 1 for group

o Data traffic is local to the Fabric Interconnects under normal conditions.

 Management and data must have different IP address ranges.

It is recommended that you reserve more IP addresses for data and management than
you need for initial installation. Imagine that you start with a 5-node system and only
reserve 12 IP addresses for management and 11 for storage. When you will want to
expand for 6 nodes, you might not have the next available IP address, and you will end
up with discontiguous address space at best. Reserve /25 or /24 for each management
and data network if IP addressing space is not an issue for your network.

The following services are needed for a regular ESXi deployment:

 vCenter IP address and a fully qualified domain name (FQDN) are recommended,
with username and password.

o Data center and group name within vCenter hierarchy.

 DNS service IP address (primary, secondary)

o Not required, but highly recommended.

o DNS cannot reside on the HyperFlex system.

 NTP service IP address (primary, secondary)

o The NTP server must be stable, continuous (for the lifetime of the system), and
reachable through a static IP address.
o NTP cannot reside on the HyperFlex system.

Cisco HyperFlex communicates with vCenter through standard ports. Port 80 is used for
reverse HTTP proxy and may be changed with TAC assistance. Port 443 is used for
secure communication to the vCenter software development kit (SDK) and may not be
changed.

DNS and NTP servers should reside outside of the HyperFlex storage system. Use an
internally hosted NTP server to provide a reliable source for the time.

Other information is also needed for deployment of your HyperFlex:

 Password for root username on ESXi hosts

o Factory default is root / Cisco123.

 o Since HXDP 3.5(1a), the installer forces you to change the password from
default.

 MAC address pool for HyperFlex group: 00:25:B5:XX:YY:ZZ

o Provide 2 hexadecimal characters for XX. Must be unique for your data center.
o First 6 characters are vendor-specific. Last 4 (YY:ZZ) are autogenerated by
HyperFlex.

 Optionally, enabling connected services (yes/no)

o Enables Intersight, auto-support, and call-home connectivity

 Optionally, email for service request notification.

o Only possible if you have Simple Mail Transfer Protocol (SMTP) configured.

If you check the Enable connected services check box during the installation, you will
enable connectivity to Intersight, auto-support, and Cisco Smart Call Home services.
You can also enable connected services after you are finished with the installation
process.

Some additional configurations might be required to take full advantage of the

connected services, such as the claiming devices in Cisco Intersight.

5.7 Installing and Expanding Standard ESXi Cisco HyperFlex

Installing Physical Components

This video describes the installation of physical components, configuring the upstream
switches and preparing the fabric interconnects. The first step in deploying a HyperFlex
solution is to install the physical components of the solution servers. This includes the fabric
interconnects, the upstream switches, and cabling the components. This is necessary before
you actually deploy the HCI software on the hardware to create the HyperFlex solution.

This figure here shows the bezels from a regular Cisco UCS installation. And that was because
the photos were taken before HyperFlex MFI nodes were shipped for the HyperFlex bezel. So
just be aware in case that kind of confused you. The power cords must be attached to each
power supply in the node. This system will actually take a couple of minutes to boot up.

Now, this next figure here, this is an important one. The first one shows the connection of the
L1 to L1 and the L2 to L2 ports between the fabric interconnect. Remember, they're located on
the front of the chassis. And if you'll remember, these are used for the management high
availability. They don't carry any data, just information between the fabric interconnects to
keep them synchronized.

Next, the second image, you connect the VCI ports on the back of each Cisco M5 server, one to
each of the fabric interconnects. And then lastly, the last diagram connects the uplink ports to
the fabric interconnects, to the upstream switches. The two upstream switches should be used
for redundancy. These upstream switches need to be configured to support Jumbo MTU.

The commands depend on the type of switch. They should also have either 10 or 40 gig
downlinks to the fabric interconnects. And lastly, you really should use Cisco Catalyst or Nexus
series switches. This is necessary if you want to optionally use a virtual port channel
configuration.

Now you're ready to create all the necessary VLANs, one VLAN for management, another for
storage, and a third one for vMotion. You can choose to create guest VLANs during the
installation or afterwards using a post-install script. The figure here shows a configuration
example if you're using a Cisco Nexus 9300. However, your actual configuration depends on
what type of switch you're using. And of course, the switches should be configured in pairs.
This next figure illustrates an example of configuring the links on the Nexus 9300, which
connects to the fabric interconnects. As you can see, the downlink is configured as a trunk with
six VLANs allowed. And it's using port channeling. Also, another port is configured as an access
port connecting to the management port of the fabric interconnect. This configuration should
be duplicated on the other fabric interconnect as well, using the appropriate ports.

Here, we're examining the CLI procedure for preparing a Cisco UCS cluster. But be aware, a
GUI-based procedure is also available. You need to manually configure Cisco UCS cluster
before you begin the HyperFlex installation. You connect to the console port of fabric
interconnect A and configure it as the primary device in the cluster. Then you connect the
console to fabric interconnect B, which will be configured as a subordinate interconnect.
Let's take a look at an example in the next figure. Notice as you connect the console port to
fabric A and power up the device, and after it boots, you enter the console to use the CLI. Then
you enter the command setup for the initial setup script. After you enter the admin password,
you'll be prompted to create a new cluster. Enter Yes, choose A as the switch fabric and
complete the configuration.

This includes the system name, the IP address, including the virtual IP address, which will be
used by both the primary and the subordinate. Now you continue with the DNS configuration.
And when you're prompted, enter Yes to apply and save the configuration. So now we're done
with the primary fabric interconnect.

Next, you connect the console to fabric interconnect B and power up the device. Now, when it
boots up, you enter the console to use a CLI. The installer will detect the presence of the peer
fabric interconnect via that L1 and L2 link between the devices. So it's going to ask you, do you
want to be a subordinate device? You say, Yes. I want fabric interconnect B to be the
subordinate device.
Now, the only thing you need to do is add an IP address for fabric interconnect B, because it
will learn the virtual IP address and everything else it needs to know from the primary fabric
interconnect across the L1, L2 links. Lastly, you enter Yes to apply and save the configuration.

Now that we've finished the configuration on both fabric interconnects, you can verify the
operational state using the show cluster state command. The example in the figure shows the
output of the command on fabric interconnect A. Now, notice it shows A is the primary and
B is the subordinate. It shows that it's up and high availability is ready. For more detailed
information, you could use the same commands, as show cluster and then say extended dash
state. That would give you more details.

After you've created the Cisco UCS cluster, now you're going to navigate to the virtual IP
address of the cluster using a browser. And of course, the primary fabric interconnect will
respond to whatever requests are being sent to that virtual IP address. You log in as admin,
along with a password that you assigned when you configured the cluster. Now, by default, the
Ethernet ports will be unconfigured and down.

So notice in the example here it shows in Cisco UCS Manager, port 1 connected to the
upstream switch. We configure it as an uplink port, which will enable the port. Now, notice
ports 17 through 19 are connected to the HyperFlex servers. These, we configure as server
ports one by one so that they will be labeled correctly. It's much better if you do it that way so
you know which servers are connected to which ports.

You will need to individually configure both uplink and server ports on both fabric
interconnects. After you've configured all the ports, then wait a few seconds. Then verify that
all of the connected ports on both fabric interconnects are now in the up enabled state. Also,
make sure you have the correct interface role assignment, such as whether it's a server or a
network port.

Finally, when all of the appropriate ports are configured as server ports, Cisco UCS starts the
discovery process for the servers. After the discovery process completes, the servers will be
listed as unassociated, because they're not assigned to service profiles yet. That comes later
on.

You are now done preparing Cisco UCS Manager for the HyperFlex installation. If you forget to
configure uplink and server ports, don't worry about it, because when you run the HyperFlex
installer, it will definitely remind you. That brings us to the end of this video. Thanks for
watching.

The Cisco HyperFlex solution is a bundle of hardware and software. You will first
install the physical components of the solution: servers, Fabric Interconnects, upstream
switches, and cabling between these components. Then you will deploy the
hyperconverged software on the hardware to create the Cisco HXDP solution.

Rack and stack servers and Fabric Interconnects:

 Example shows four Cisco HX220c M5 servers and two Cisco UCS 6332 Fabric
Interconnects.
 Attach a power cord to each power supply in your node, and to a grounded AC
power outlet. During initial boot up, wait for approximately two minutes to let the
node boot in standby power.

Note
Notice that the Fabric interconnects are installed with their power supplies facing
towards the front bezels of the servers. The figure shows front bezels from a traditional
Cisco UCS. At the time this photo was taken, HyperFlex M5 nodes were shipped with
Cisco UCS instead of HyperFlex bezels.

Connect Fabric Interconnect heartbeat: L1-L1 and L2-L2 ports. Optionally connect
console management cables to terminal server.

Connect VIC ports on each server to Fabric Interconnects. One port to Fabric
Interconnect A, one to Fabric Interconnect B.

Connect uplink both Fabric Interconnects to upstream switch. And connect the IP OOB
(out-of-band) management to an access port.
Note
The first figure shows the front of two Cisco UCS 6332 Fabric Interconnects. Orange
Ethernet cables connect to out-of-band (OOB) IP management ports and blue Ethernet
cables connect to the terminal server for remote access to consoles.

The second figure shows the backs of four Cisco HX220c M5 servers. Each server has
one VIC 1387. From each VIC, there are two 40G Twinax cables connected, one to
each Fabric Interconnect. The port on the right connects to Fabric Interconnect A and
the port on the left connects to Fabric Interconnect B.

The last figure shows two Fabric Interconnects from the back. The first two ports on the
right are the RJ45-based Layer 1 and Layer 2 ports that interconnect the two Fabric
Interconnects for the Cisco UCS heartbeat. The first two 40G ports are used as uplinks
to upstream switches in this instance. Ports 3–6 (with orange bands) are used to connect
to the VICs of servers.

The Cisco HX220c M4/M5 and Cisco HX240c M4/M5 servers connect directly to the
Fabric Interconnects. The direct connection enables Cisco UCS Manager to manage the
HyperFlex Series servers using a single cable for both management traffic (in-band
management) and data traffic. When you use direct connect mode, all Cisco UCS-
managed adapters must be connected to the server ports on the Fabric Interconnects.

The Fabric Interconnects OOB IP management port connects to an upstream access

port, which must have the correct VLAN configuration for OOB management network.

Do not connect dedicated Cisco IMC ports to the network for integrated nodes. Doing
so causes the server to not be discovered in Cisco UCS Manager. If the server is not
discovered, reset Cisco IMC to factory settings for each server.

Set the Cisco IMC server to the factory default settings before integrating with Cisco
UCS Manager. You can set it to factory defaults by attaching a mouse and a monitor to
the server and then pressing F8 when prompted during boot to access Cisco IMC setup.
If you just bought a new HyperFlex system, Cisco IMC will be in factory-default
setting.

The ports where the individual server is attached to Fabric Interconnects must be
consistent on both Fabric Interconnects or the HyperFlex installation will fail.

Complete the physical cabling of servers to the Fabric Interconnects and configure the
ports on the Fabric Interconnects as server ports in order for the servers to be detected.
5.8 Installing and Expanding Standard ESXi Cisco HyperFlex

Configuring Upstream Switches

It is recommended that you use two upstream switches for redundancy. Those upstream
switches need to support jumbo maximum transmission units (MTUs) and have either
10-Gbps, 40-Gbps, or 100-Gbps downlink (to Fabric Interconnects) connectivity. It is
recommended that you use Cisco Catalyst or Cisco Nexus Series Switches, where you
can optionally employ a vPC configuration for maximum resiliency and throughput.

To configure upstream switches, you will need to perform the following:

 Create VLANs for Cisco HyperFlex management, HyperFlex storage, vMotion, and
all your guest VLANs that will exist on the HyperFlex system.

Note
You will also need Cisco UCS management VLAN if separated from HyperFlex
management VLAN.
 Configure ports as trunks, put them into port channel if using more than one port per
switch. Allow appropriate VLANs onto the trunk.
 Configure default gateways for HyperFlex and Cisco UCS management, vMotion,
and guest VLANs if upstream switches serve as the default gateway.

o If upstream switches are, for example, Catalyst stack, then configure one single
switch virtual interface (SVI) for the stack.
o If the upstream switches are, for example, a vPC-based Cisco Nexus setup, then
you will need to configure First Hop Redundancy Protocol (FHRP).

Configure jumbo MTU and disable ARP proxying:

 Standard HyperFlex deployments use jumbo MTU, which needs to be enabled on

Upstream switches.

o For example, on Cisco Nexus 9300 Series Switches, you can use the system
jumbomtu 9216 command to configure the default jumbo MTU on all Layer 2
interfaces.
o Do not forget to configure both switches in the pair.

 ARP proxy should be disabled on the HyperFlex networks.

The MTU size specifies the maximum frame size that an Ethernet port can process. For
transmissions to occur between two ports, you must configure the same MTU size for
both ports. A port drops any frames that exceed its MTU size. By default, each port has
an MTU of 1500 bytes, which is the IEEE 802.3 standard for Ethernet frames. Larger
MTU sizes are possible for more efficient processing of data with less overhead. The
larger frames, called jumbo frames, can be up to 9216 bytes in size, which is also the
default system jumbo MTU size.

For Cisco IOS switches, configure the jumbo MTU on the global level:

SwitchCat(config)# system mtu 9000

ARP Proxy allows communication between VLANs without specific Layer 3 rules. This
can cause issues with HyperFlex internal system communication and ARP table
population. This can lead to severe issues such as storage unresponsiveness. Make sure
that ARP proxying is disabled.

SwitchCat(config)# ip arp proxy disable

Create all the needed VLANs:

 One VLAN for management, one for storage, one for vMotion
 None, one, or more as guest VLANs

o You add guest VLANs after HyperFlex installation (manually or using post-
install script).
 Example shown is on Cisco Nexus 9300 Series, but configuration depends on the
platform.

o You could use one single VLAN for HyperFlex and Cisco UCS management.
Example separates them.

 Do not forget to configure both switches in the pair if they are not stacked.

vlan 10
name HX-mgmt
vlan 11
name UCS-management
vlan 12
name HX-storage
vlan 200
name HX-vmotion
vlan 201
name data-engineering
vlan 202
name data-marketing

Configuration on a Catalyst type switch would be similar to the one on a Cisco Nexus
Switch.

HyperFlex management VLAN is needed for IP addresses on ESXi and CVM

components. Cisco UCS management is needed for Fabric Interconnects and OOB
Cisco UCS management of HyperFlex servers. HyperFlex and Cisco UCS management
can be the same subnet if simplicity of IP addressing is desired.
Configure links to Fabric Interconnects:

 Example shown is on Cisco Nexus 9300 Series, but configuration depends on the
platform.

o Example shows the configuration of one downlink as trunk with 6 VLANs

allowed. No port channeling.
o Example also shows the configuration of one port as access–that port connects to
the management port of Fabric Interconnect.

 Do not forget to configure also connectivity to other Fabric Interconnect.

! Configure downlink to FI as trunk and allow only VLANs

that are in use
interface eth1/1
switchport mode trunk
switchport trunk allowed vlan 1,10,12,200,201,202
no shutdown
! Configure port that connects to fabric interconnect
management port
interface eth 1/30
switchport mode access
switchport access vlan 11
no shutdown

Configuration on a Catalyst type switch would be similar to the one on a Cisco Nexus
Series Switch.

For better redundancy, you can use more than one cable from each Fabric Interconnect.
In that case you need to configure port channeling on both upstream switches and Fabric
Interconnects.

5.9 Installing and Expanding Standard ESXi Cisco HyperFlex

Preparing Fabric Interconnects

There are two primary ways to configure a Cisco UCS domain: using either GUI or
CLI. Here you will examine the CLI procedure, but a GUI-based procedure is also
available.

Cisco UCS domain provides high-speed connectivity with an orchestration base for
your Cisco HyperFlex deployment:
 You need to manually create a Cisco UCS domain before you start HyperFlex
installation:

o Connect with the console to Fabric Interconnect A, choose the console

configuration option, and configure the parameters.
o Repeat the procedure on the console to Fabric Interconnect B. It will find Fabric
Interconnect A, and you can join it to the group.
o Verify the setup.

 Configure uplink and server ports on both Cisco UCS Fabric Interconnects.

You will need these settings on the console port to successfully connect to Fabric
Interconnects:

 9600 baud
 8 data bits
 No parity
 1 stop bit

Creating a Cisco UCS Domain

Configure the primary Fabric Interconnect (Part 1):

Enter the installation method (console/gui)? console

Enter the setup mode (restore from backup or initial setup)

[restore/setup]? setup

You have chosen to setup a new switch. Continue? (y/n): y

Enter the password for "admin": <password>

Confirm the password for "admin": <password>

Do you want to create a new cluster on this switch (select 'no' for
standalone setup or if you want this switch to be added to an existing
cluster)? (yes/no) [n]: yes

Enter the switch fabric (A/B): A

Enter the system name: UCS1

Mgmt0 IPv4 address: 10.10.1.2

Mgmt0 IPv4 netmask: 255.255.255.0

IPv4 address of the default gateway: 10.10.1.254

Virtual IPv4 address: 10.10.1.1

Follow these steps to configure a primary Fabric Interconnect, also shown in the
previous example:

 Connect to the console port.

 Power up the Fabric Interconnect. You will see the power-up self-test messages as
the Fabric Interconnect boots.
 When an unconfigured system boots, it prompts you for the setup method to be
used. Enter console to continue the initial setup using the console CLI.
 Enter setup to continue as an initial system setup.
 Enter y to confirm that you want to continue the initial setup.
 Enter the password for the admin account.
 To confirm, re-enter the password for the admin account.
 Enter yes to continue the initial setup for a domain configuration
 Enter the Fabric Interconnect fabric (either A or B).
 Enter the system name.
 Enter the IPv4 or IPv6 address for the management port of the Fabric Interconnect.
If you enter an IPv4 address, you will be prompted to enter an IPv4 subnet mask. If
you enter an IPv6 address, you will be prompted to enter an IPv6 network prefix.
 Enter the respective IPv4 subnet mask or IPv6 network prefix, then press Enter.
You are prompted for an IPv4 or IPv6 address for the default gateway, depending on
the address type you entered for the management port of the Fabric Interconnect.
 Enter either IPv4 or IPv6 address for the default gateway.

Configure the Primary Fabric Interconnect (Part 2):

Configure the DNS Server IPv4 address? (yes/no) [n]: yes

DNS IPv4 address: 10.10.1.8

Configure the default domain name? (yes/no) [n]: yes

Default domain name: cisco.com

Join centralized management environment (UCS Central)? (yes/no) [n]:

no

Following configurations will be applied:

<... summary of configuration omitted ...>

Apply and save the configuration (select 'no' if you want to re-
enter)? (yes/no): yes

Perform the following (continued) steps to configure the the primary Fabric
Interconnect, which you can also see in the example above:

 Enter yes if you want to specify the IP address for the DNS server, or no if you do
not.
 (Optional) Enter the IPv4 or IPv6 address for the DNS server. The address type
must be the same as the address type of the management port of the Fabric
Interconnect.
 Enter yes if you want to specify the default domain name, or no if you do not.
 (Optional) Enter the default domain name.
 Review the setup summary and enter yes to save and apply the settings or
enter no to go through the setup wizard again to change some of the settings. If you
choose to go through the Setup wizard again, it provides the values you previously
 entered, and the values appear in brackets. To accept previously entered values,
press Enter.
Configure the subordinate Fabric Interconnect to join the primary one:

Enter the installation method (console/gui)? console

Installer has detected the presence of a peer Fabric Interconnect.

This Fabric Interconnect will be added to the cluster. Continue
(y/n) ? y

Enter the admin password of the peer Fabric Interconnect: <password>

Peer Fabric Interconnect Mgmt0 IPv4 Address: 10.10.1.3

Apply and save the configuration (select 'no' if you want to re-
enter)? (yes/no): yes

Follow these steps to configure the subordinate Fabric Interconnect, which you can also
see in the previous example:

 Connect to the console port.

 Power up the Fabric Interconnect. You will see the power-up self-test messages as
the Fabric Interconnect boots.
 When the unconfigured system boots, it prompts you for the setup method to be
used. Enter console to continue the initial setup using the console CLI. The Fabric
Interconnect should detect the peer Fabric Interconnect in the domain. If it does not,
check the physical connections between the Layer 1 and Layer 2 ports, and verify
that the peer Fabric Interconnect has been enabled for a domain configuration.
 Enter y to add the subordinate Fabric Interconnect to the group
 Enter the admin password of the peer Fabric Interconnect.
 Enter the IP address for the management port on the subordinate Fabric
Interconnect.
 Review the setup summary and enter yes to save and apply the settings or
enter no to go through the Setup wizard again to change some of the settings. If you
choose to go through the setup wizard again, it provides the values you previously
entered, and the values appear in brackets. To accept previously entered values,
press Enter.

Verifying Cisco UCS Cluster Creation

After you are done configuring the Cisco UCS domain, you will need to wait few
minutes for the cluster to finish up the setup. Then you can verify if both of the
domain's Fabric Interconnects are up and that high availability is ready.

show cluster state displays the operational state and leadership role for both
Fabric Interconnects in a high-availability domain.

UCS1# show cluster state

Cluster Id:
0x4432f72a371511de-0xb97c000de1b1ada4
A: UP, PRIMARY
B: UP,
SUBORDINATE HA READY
To get more info typically used when troubleshooting issues, use the show cluster
extended-state command.

You can also log in to Cisco UCS Manager available via browser at the group IP
address.

Configuring Uplink and Server Ports

After you have created a Cisco UCS domain, use a browser, and navigate to the IP
address of the group. The login page to Cisco UCS Manager should open. Log in with
the username admin and the password that you assigned to the admin during Cisco
UCS domain creation.

The example shows Cisco UCS Manager with Port 1 connected to the upstream switch
and Ports 17–19 connecting to HyperFlex servers. To configure ports, navigate
to Equipment > Fabric Interconnect A (primary) > Physical Ports.

By default, all these ports are unconfigured and disabled. Right-click on the uplink port
and choose Configure as Uplink Port. Right-click on the downlink port where a server
is connected and choose Configure as Server Port. Perform server port configuration
one at a time and in order of ports so that servers will be discovered and numbered in
physical order in the rack.

You will need to individually configure both uplink and server ports on both Fabric
Interconnects.
After you have configured all ports, wait a few seconds, and verify that all connected
ports on both Fabric Interconnects are now in Up/Enabled state.

After Cisco UCS successfully discovers all servers, you are done preparing Cisco UCS
for HyperFlex installation.

If you forget to configure the uplink and server ports, and proceed directly into
HyperFlex installation, the installer will remind you that you need to configure them.

You can investigate the process of server discovery in the FSM tab of an individual
server.

Note
Do not connect the dedicated Cisco IMC ports to the network for nodes that are being
managed by the Cisco UCS Manager. Doing so causes the server to not be discovered in
Cisco UCS Manager. If the server is not discovered, reset Cisco IMC to factory settings
for each server.

5.10 Installing and Expanding Standard ESXi Cisco HyperFlex

Deploying the Installer VM

Now let's deploy the installer VM, begin the HyperFlex installation, and then perform the post-
installation tasks. The Cisco HyperFlex installer can be installed on a VMware Workstation, on
VMware Fusion, VirtualBox, or VMware vSphere. In this topic, we're going to discuss the
vSphere as an example.

The Cisco HyperFlex installer can be downloaded from software.cisco.com. The installer
requires a separate ESXi which is not a member of the vCenter HyperFlex cluster.

Now, this figure shows the initial screen when deploying the OVA template where you view
the OVF template details. From this screen, you would click Next, and you would proceed
through the Wizard, where you would configure the name, the location, disk format
information, network mapping, and then complete all of the deployment process.

The preferred settings for the installer virtual machine are three virtual CPUs and 4 gigs of
memory. After the HyperFlex installer, VM will boot up. Open the installer VM console. The
initial console will list all the IP address, or the IP address.
From here, you would log in with root and password that you provided during the OVA
deployment. And at this point, you can start the installation of HyperFlex. However, it is
recommended you go back and check again to make sure you've met all of the prerequisites.

After you've met the prerequisites, you're ready to proceed with the installation. This figure
here shows you the steps that you should follow. This includes browsing to HTTPS address of
the installer.

You would provide the information about UCS manager, hypervisors, and vCenter, and, lastly,
starting the installation. Then the installer does everything else. Be prepared to wait for
about an hour to let the installer complete.

During the installation, you can follow the progress, as you can see. And you will clearly see
each one of the tasks being performed one at a time. You might get some validation errors. Be
sure to read them carefully, and then try to resolve the issues. Don't ignore them if you're not
familiar with the potential consequences.
Then don't forget the post-installation tasks. You can accomplish most of the post-installation
tasks by executing the post_install script located on the installer VM. To start this script, you
would SSH to the installer VM with the same credentials used in the HyperFlex installation.

The post-installation script allows you to perform the following actions. You enable high
availability and distributed resource scheduler on the cluster, disable SSH warnings in vCenter,
add vMotion interfaces and add VM network VLANs for the guest VLANs, and, lastly, run the
health checks to verify the health and connectivity in the cluster.
Lastly, keep in mind, syslog is not really part of the installation, is not built into the post-
installation scripts. This needs to be configured and verified manually. You'll need to make sure
the syslog server is set up and running, and make sure that the TCP and UDP ports are open
to receive log messages from the ESXi servers.

The figure here shows the commands to execute using the ESXi shell. You need to repeat this
verification for all of the ESX hosts in the cluster. Also, make sure the remote syslog server is
receiving the log messages. That brings us to the end of this video. Thanks for watching.

You can deploy Cisco HXDP Installer on a VMware Workstation, VMware Fusion,
Virtual Box, or VMware vSphere. Here you will use vSphere as an example.

Download the Cisco HyperFlex Installer from software.cisco.com.

A separate ESXi, which is not a member of the vCenter HyperFlex deployment, is

required for the installer:

 Open a virtual machine hypervisor for VMware, for example, vSphere.

 Deploy OVA:

o Provide IP addressing, NTP, DNS, map network adapter, and provide the
desired password.
o Installer VM needs to have connectivity to both HyperFlex and Cisco UCS
management networks.
The preferred settings for the Cisco HXDP Installer VM are 4 vCPUs and 4 GB of
memory. Reducing these settings can result in 100 percent CPU usage and spikes for the
host.

Note
The default password of Cisco123 is no longer valid with HXDP 3.5(1a) and later.
When you deploy the OVA, you must specify a password. If you try logging in
with Cisco123 and it does not work, that means you had previously set a password
when you deployed it.

Wait for the HXDP installer VM to boot up:

 Open the installer VM console. The initial console display lists the IP address.

o Accept the self-signed certificate.

o Log in using the username root and the password you provided as part of the
OVA deployment.

Data Platform Installer.

****************************************

You can start the installation by visiting

the following URL:

http://10.10.1.108

****************************************

Cisco-HX-Data-Platform-Installer login:

At this point, you can start installation of HyperFlex. However, it is recommended that
you check back to see if you have really met all the prerequisites and pre-installation
tasks for the installation to go smoothly.

5.11 Installing and Expanding Standard ESXi Cisco HyperFlex

HyperFlex Installation

After you have met the prerequisites listed in the pre-installation checklist guide, racked
the Fabric Interconnects and servers, connected the servers to the Fabric Interconnects,
connected the Fabric Interconnects to the switches, configured the upstream switches,
created a Cisco UCS domain, configured the ports on the Fabric Interconnects, and
deployed the installer VM, you are ready to proceed with the installation of Cisco
HyperFlex.

Follow these steps after you have met all prerequisites:

 Browse to the HTTPS address of the installer. Before you start installation, double-
check the installation guide.
 Provide information about Cisco UCS Manager, hypervisors, and vCenter.
 Start installation. The installer does the rest.

o Installation can take 1 hour or more if updating the Cisco UCS firmware of the
servers.

Note
Sometimes, Cisco UCS Fabric Interconnects ship from the factory with a Cisco UCS
firmware version that does not fit the version required by the HyperFlex installation. In
this case, upgrade the Cisco UCS infrastructure firmware first, and then, during
installation of Cisco HXDP, choose that version to be applied to HyperFlex rack
servers.

If the firmware version that you have chosen to be used on servers is different than the
current one that the servers are using, the installation includes a firmware update.

During installation, follow the progress, and you will see each task that is being done
during the installation process.

If you get validation errors, read them carefully and try to resolve the issues. Do not
ignore them if you are not familiar with the potential consequences. For example, if you
are installing HyperFlex on Fabric Interconnects that either do not have the quality of
service (QoS) setup (clean Fabric Interconnects) or the QoS settings are not as
HyperFlex expects them, the installer wizard will notify you that configuration of QoS
settings is disruptive. If your HyperFlex system is connected to Fabric Interconnects
that are not serving other workloads, you can safely skip the validation:

Note
The installation steps are not described in this topic since you will do the installation
yourself in the lab.
After successful installation, do not forget to perform post-installation tasks and test
whether the installation is working as intended.

Note
The figure shows the summary screen after a successful installation of a 3-node
HyperFlex deployment that is online and healthy.

You can accomplish most post-installation tasks by executing the post_install script
located on the installer VM. To start the script, SSH into the installer VM with the same
credentials that you used in the HyperFlex installation.

After you successfully create the initial group of converged nodes, you can add the
compute-only nodes. Note that the compute-only nodes cannot be a part of the initial
group.

5.12 Installing and Expanding Standard ESXi Cisco HyperFlex

Post-Installation
You need to perform various (mostly vSphere-related) tasks outside the workflow of
Cisco HyperFlex installation. These tasks are simplified because they are integrated into
a script that you run from the installer VM.

The post_install script is available in two locations:

 Installer VM: Mostly used with on-premises installations.

 CVM: Since there is no Installer VM available in Intersight-based installations you
can log in to the CVM using SSH.

o Prior to 4.5(1a): Run post_install.sh as root on a CVM to initiate script.

o 4.5(1a): Run hx_post_install as admin on a CVM to initiate script.
o The resulting dialogs are equivalent. Use the -h switch with the script for
additional options.

Since HXDP 4.0(1a), when you run the post_install script, you are given three options:

 New/Existing Cluster: Use this option only for new deployments.

 Expand Cluster: Use this option for expanded nodes. After you choose this option,
you must provide details to be configured for the expanded node.
 Generate Certificate: Use this option to generate a new self-signed certificate, and
copy it to all the SCVMs. You are prompted to register the certificate with the
vCenter again.

You would use the second and third option after an upgrade procedure and the first one
after the installation procedure.

To complete the post-installation tasks, you can run a post-installation script on the
installer VM. The script allows you to perform the following actions:

 Enable High Availability/DRS on cluster: Enables vSphere high availability and

DRS.
 Disable SSH warning: Suppresses the SSH and shell warnings in the vCenter. SSH
is needed for HyperFlex snapshots and upgrades.
 Add vMotion interfaces: Configures vMotion interfaces. Requires IP address and
VLAN input.
 Add VM network VLANs: Adds additional guest VLANs on all HyperFlex hosts.
 Run Health Check: Verifies health and connectivity within the deployment.

Configure Syslog for ESXi Servers

Configuration of syslog is not part of the installation or built-in post installation
procedure. You will need to perform it manually:

1. Verify that the syslog server is up and running and TCP/UDP ports are open to
receive logs from ESXi servers.
2. SSH to the ESXi shell and execute:

o esxcli system syslog config set --loghost=<udp://remote-syslog-server-

ip> esxcli system syslog reload esxcli network firewall ruleset set -r syslog -e
true esxcli network firewall refresh

3. Repeat steps 1 and 2 for all ESXi hosts in the group.

4. At the remote syslog server, verify if the logs are being received in the designated
directory.

It is best practice to send all logging information to a centralized syslog repository.

 Changing Passwords on ESXi and CVM
During the HyperFlex installation, you can set root passwords for ESXi servers and
CVMs.

If the policy requires you to change passwords, the procedure is as follows:

 ESXi host

o Log in into ESXi host using SSH.

o Acquire root privileges: su -
o Change the root password: passwd root

 CVM

o Log in into CVM using SSH.

o Change the password on all converged nodes: hxcli security password set

If you add new compute nodes and try to reset the system password using the hxcli
security password set command, the converged nodes get updated, but the
compute nodes may still have the default password. To change the compute node
password, use the following procedure:

1. Use VMotion to move all the user VMs off the ESXi hosts.
2. Launch the CVM console from vCenter and log in as the root user.
3. Run the passwd command to change the password.
4. Log out and re-login to confirm that the password changed successfully.
5. Run the hxcli node add -f command to add the node back into the group.

Auto Support and Smart Call Home for HyperFlex

Cisco Auto Support is the alert notification service provided through HXDP. If you
enable Auto Support, notifications are sent from HXDP to the designated email
addresses or email aliases that you want to receive the notifications. Typically, Auto
Support is configured during HyperFlex storage system creation by configuring the
SMTP mail server and adding email recipients.

Cisco Smart Call Home is an automated support capability that monitors your
HyperFlex storage systems, flags issues, and initiates resolution before your business
operations are affected, which results in higher network availability and increased
operational efficiency. Cisco Smart Call Home is a product feature embedded in the
operating system of Cisco devices that detects and notifies the user of various fault
conditions and critical system events. Cisco Smart Call Home adds automation and
convenience features to enhance the basic Call Home functionality. After Cisco Smart
Call Home is enabled, call-home messages/alerts are sent to Cisco Smart Call Home.

You can configure the HyperFlex storage system to send automated emails regarding
documented events. You can use the collected data to help troubleshoot issues in your
HX storage system.
 Choose the Enable Auto Support check box during the installation or later through
HX Connect, CLI, or API.
 You can disable Smart Call Home for Data Collection on installation and enable it
later.

Note
Only unauthenticated SMTP is supported for Cisco Auto Support.

Auto Support can also be used to connect your HyperFlex storage system to monitoring
tools.

Data about your HyperFlex is forwarded to Cisco TAC through a direct or proxied
HTTPS.

If you have not enabled Cisco Auto Support or Smart Call Home during the installation,
refer to the Cisco HyperFlex Systems Installation Guide for VMware
ESXi at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/Installation_VMWare_ESXi/4-5/b-hx-install-
guide-for-vmware-esxi-4-5.html for post-installation configuration procedures.

Using Cisco Smart Call Home requires a device and Cisco.com ID to be registered with
the support service.
5.13 Installing and Expanding Standard ESXi Cisco HyperFlex

Intersight-Based HyperFlex Installation

The Intersight installation workflow is almost the same as when installing through the
on-premises Installer VM, however, there are differences in the user interface and
device registration.

Note
Only the differences are covered here to avoid an overlap. The procedure explained here
replaces the deployment of the Installer VM and connecting to the Installer VM
interface.

There are several advantages to the Intersight-based HyperFlex installation:

 The installation is profile-based, and the profiles are re-usable.

 No Installer VM required, any available versions of HyperFlex can be installed.
 No VPNs or jump hosts, multiple installations can be performed at the same time.
 The HyperFlex system is registered for management from Intersight.

Registering HyperFlex Nodes to Cisco Intersight

To register devices to Intersight, you will need to enable connectivity
to svc.instersight.com through an HTTP proxy or directly. Once the connectivity is
established, you can use the device connector on Cisco UCS Manager to register the
UCS domain to Intersight.

Registering the UCS domain also registers all the servers that are part of the domain to
Intersight. This includes HX-Series and C and B Series servers.

The registration is performed through the Intersight claim interface:

1. Go to intersight.com and log in using your credentials.

2. Access the ADMIN > Targets interface to list available registration targets.
3. Click on the Claim Target button and select Cisco UCS Domain (UCSM
Managed) from the list.
4. The Intersight interface will ask you to provide Device ID and Claim Code.
You do not need any special license to do this action. Claiming devices and installing
HyperFlex is always available in the free licensing tier.

You can find the Device ID and Claim Code in the UCS Manager device connector:

1. Connect to the UCSM group IP address (VIP) and log in as admin.

2. In the UCSM Interface, navigate to the Admin > Device Connector.
3. The device connector will list the network connectivity status, Device ID,
and Claim Code.
4. Copy the values from Device connector to Intersight and click Claim this will make
servers available for HyperFlex installation.

There are handy copy to clipboard buttons available next to the Device ID and Claim
Code in the UCSM Device Connector interface.

Once you click Claim in Intersight, click the Refresh button in the UCSM Device
Connector to verify that the connection to Intersight has been established. If the
connection was successful, the figure will change to display a green line from the
system to the Intersight icon.

To configure HTTP proxy in the device connector, click the Settings button, this will
display an extra dialog, where you can provide proxy information.

After successful registration of the servers connected to the Cisco UCS domain that you
registered, will be visible in the Intersight inventory. You will be able to choose the
unassociated servers from the list as the targets for HyperFlex installation.

Installing HyperFlex from Intersight

After the servers have been registered to Intersight, you can create a HyperFlex profile
in the Intersight interface. This profile can either be used immediately, or can be applied
later by selecting available servers from the Intersight inventory.
Although the user interface of the installer is different, the same configuration values
need to be provided to install a HyperFlex system:

1. Go to CONFIGURE > Profiles and click on Create HyperFlex Cluster

Profile button.
2. Select the deployment type, HXDP version, and name of the HyperFlex system.
3. Provide the installation configuration. Same values apply as with Installer VM.
4. Intersight does not ask for Data network IP range. Always uses 169.254.0.0/24.
5. Choose the servers for HyperFlex installation and start the installation.

The Intersight installer will provide feedback during the installation, similar to the
Installer VM:

 You can follow the installation in widgets, global notifications, and mobile app.
 You can troubleshoot issues through the Intersight cross-launch feature.
 You can share the installation information with other users.
 The HyperFlex system will automatically register to Intersight

You can install the Standard and Edge ESXi-based deployments from Intersight.
Intersight is the preferred option for HyperFlex installation because it is future-proof.
New functionalities and system capabilities can be added to your existing installation
very easily through the simple full-stack Intersight upgrade.

Note
Intersight lifecycle management is only available for HyperFlex systems installed from
Intersight. As of HyperFlex version 4.5(1a), adding existing HyperFlex installations to
Intersight does not enable full-stack update capabilities.
5.14 Installing and Expanding Standard ESXi Cisco HyperFlex

HyperFlex Expansion

This video explores how to perform a cluster expansion. With Cisco HyperFlex, you can start
with a small deployment of three nodes, and then expand as needed. This keeps the upfront
costs low while providing a clear path for an upgrade.

The cool thing is that the cluster expansion can be performed while the system is operational.
This figure lists the high-level steps for adding a node to an existing HyperFlex cluster.

This includes making sure that the HyperFlex storage cluster is healthy. The new nodes must
meet the prerequisites, preparing the new nodes, and then running the HyperFlex installer
with the Expand workflow. When done, perform the post-installation tasks.
Keep in mind that converged nodes come with ESXi pre-installed. You must make sure that the
version of the new device matches the version of the initial cluster. Next, the compute-only
nodes are just regular UCS C- or B-Series servers. They don't come pre-installed with the ESXi.
It must be installed manually.

Lastly, the ESXi image needs to be the Cisco custom image that was downloaded from Cisco,
software.cisco.com. So let's take a look at the expansion process using the installer.

First, you create a service profile to get to its KVM. In UCS Manager, you would navigate to
Servers, Service Profile Templates, Sub-Organizations, and then select your cluster. During the
initial cluster creation, four different service profile templates are created-- compute-nodes,
compute-nodes-m5, hx-nodes, and hx-nodes-m5. Right click the appropriate service profile
template, and then choose Create.

Then you provide the name of the service profile to be created from the template. Now, this
is a temporary name just for reimaging only.
After the service profile has been created, you need to apply it to the server. So back in USC
Manager, you navigate to Equipment, Rack-Mounts, and Servers. Right click the new server,
and choose Associate Service Profile.

Then choose the service profile that you just created, and confirm your choice. You can
monitor the progress in the Finite State Machine, or FSM, tab, just in case you were wondering
what FSM stood for.

Now this next figure here describes the steps to load the ESXi image to the Virtual Media of
the new server. In Cisco UCS Manager, you navigate to Equipment, Rack-Mounts, Servers.
Right click the server, and choose KVM Console.

In the KVM for the server, you need to mount the ESXi image as a virtual CD. So you choose
Virtual Media and activate Virtual Devices. Then you choose that Virtual Media and, this time,
CD/DVD.

Pick Choose File, and then find that Cisco custom ESXi image. Then, lastly, you click Map Drive.
Finally, go back to UCS Manager. Right click your server. Choose Reset, and then choose Power
Cycle, and click OK to begin.
You will see the server rebooting in the KVM console. When you are prompted to select an
install option, choose the one appropriate for your new server. ESXi installation is destructive,
and it cannot be reversed.

After you complete the installation, go back and delete that temporary service profile that you
created. And then wait for the disassociation to complete.

Now let's look at the cluster expansion workflow, which is the same for clusters with either
converged or compute nodes. In the HyperFlex installer, choose Expand Cluster, and then
choose Standard Cluster. You just complete the details like you did during the installation
process.

Obviously, you don't need to configure IP addressing for storage controller if you're on a
compute-only node. Once again, compute nodes do not have local storage.
Let's look at what the-- let's see what this looks like in the actual GUI interface. All right, in the
figure, you'll notice that you have two options—Create Cluster and Expand Cluster. When you
expand the dropdown for Expand Cluster, you can choose either Standard Cluster or Stretch
Cluster. In this figure, you'll see we're choosing the Standard Cluster.

You can watch the cluster expansion progress on the display. When it completes, you should
see the Cluster Expansion Successful. And then don't forget to perform the post-installation
task. And finally, test the cluster functionality.
Finally, the benefits gained from expanding storage resources do not take effect until the
scheduled auto-rebalancing takes place. And I think I mentioned that once before. The default
schedule for rebalancing is at 5:15 AM, early in the morning.

Rebalancing only applies to converged nodes since compute nodes don't have any additional
storage to the cluster. You can either wait for the scheduled time or initiate an on-demand
rebalancing using the command illustrated in the figure. There we go. That brings us to the end
of this video. Thanks for watching.

An important advantage of the Cisco HyperFlex solution is that you can start with a
small deployment of 3 nodes and expand as needed. The upfront costs are low, but you
have a clear upgrade path. The HyperFlex expansion can be done while the system is
operational.

For example, if you have a 3-node HyperFlex installation, are running out of disk space,
and do not have any empty slots, you can add another node to the group. Similarly, if
you are running out of processing power, you can add another converged node. If you
have plenty of storage but not enough CPU, you can add a compute-only node.

The following are high-level steps for adding a node to an existing HyperFlex
installation:

 Ensure that the HyperFlex storage system is healthy.

 Ensure that the new node meets installation prerequisites.

o If it is a converged node, it needs to be the same factor (220/240) and type

(Hybrid/All-Flash/All-NVMe and SED/regular) as the original installation.
o If it is a compute-only node, it can be any supported server.

 Prepare the new node. Connect to Fabric Interconnects, configure Fabric

Interconnect ports, and ESXi reimage (if necessary).
 Run the HyperFlex Installer with the Expand workflow.

o If you upgraded the HyperFlex installation, you must download and install a
new installer VM that matches the current version of HXDP running on the
system.

 Perform post installation tasks.

When you add a node that has a different CPU family from what is already in use in the
HyperFlex group, enable EVC.

Note the following when you are building a mixed M4/M5 system:

 Carefully consult on HXDP, UCS, and vSphere version combination.

 If necessary, upgrade the HyperFlex system version of the installed system. Older
versions do not support M5 nodes.
 Download and deploy the HXDP installer to run the expansion workflow. Installer
must match the HXDP version of the existing installation.
 Upgrade vCenter to v6.5 or later. Without vCenter v6.5, the Broadwell EVC mode
cannot be enabled. Only the vCenter upgrade is required, ESXi can remain on an
older version subject to the VMware software interoperability matrix. Proceeding
with EVC mode off is not supported and will cause operational issues in the future.

The table lists the supported compute-only node configurations for compute-only
functions:

 B200 M4/M5
 B260 M4
 B420 M4
 B460 M4
 C240 M4/M5
 C220 M4/M5
 C460 M4
 C480 M5 and C480 M5 ML
 B480 M5

After you connect the new node or nodes, you need to configure the ports on both
Fabric Interconnects as Server. So, the configuration is the same as on initial
installation:
Imaging a New Server with ESXi
Compute nodes are standard UCS servers, which do not come preinstalled with the
ESXi hypervisor from the factory. Before starting the expansion workflow, you need to
install the ESXi on the compute node.

To be able to install ESXi on a UCS server of any type, you need to properly configure
the server. In a managed UCS deployment configuration is done through service
profiles in UCS Manager.

To associate a service profile with a server you need to define most of its parameters,
which are not optional, which can take time and is not self-evident to new users.

Since compute nodes do not come preinstalled with ESXi, you need to install it:

 You can use the existing HyperFlex UCS templates.

o In Cisco UCS Manager, navigate to Servers > Service Profile

Templates > Sub-Organizations > your cluster.
o Use the compute-nodes-m5 to install ESXi on M5 compute nodes.
o Use the hx-nodes-m5 to reimage converged nodes (expansion).

 Name the service profile. You will only use it temporarily.

Apply the previously created service profile to the new server:

 In Cisco UCS Manager, navigate to Equipment > Rack-Mounts > Servers.
 Find your server, right-click, and choose Associate Service Profile.
 Choose the service profile you previously created and confirm your choice.

o Be very careful that you do not accidentally reimage a working server.

 Wait for service profile association to finalize.

o You can monitor progress in the FSM tab.

Note
The described procedure is useful when you already have an installed HyperFlex system
in the same UCS domain as the new server. For re-installations of ESXi on converged
nodes (such as HyperFlex re-installations), you can run the advanced workflow in the
HyperFlex Installer VM and choosing Run Cisco UCS Manager Configuration. This
creates service profiles without finishing the HyperFlex installation.

By default, no user intervention is required if you are booting from FlexFlash (SD card).
However, if you are setting up your compute-only node to boot from a local disk,
complete the following steps in Cisco UCS Manager:

 Click the service profile created by the HyperFlex installer.

 On the General tab, click Unbind from the Template.
 In the working pane, click the Storage tab. Click the Local Disk Configuration
Policy sub tab.
 In the Actions area, choose Change Local Disk Configuration Policy > Create
Local Disk Configuration Policy.
 Under Create Local Disk Configuration Policy, enter a name for the policy, and
keep the rest as default. Click Ok.
 In the Change Local Disk Configuration Policy Actions area, select the newly
created local disk configuration policy from the drop-down list. Click Ok.
 Now, go back to the HyperFlex installer, and click Retry UCSM Configuration.

If the vCenter group has EVC enabled, the deploy process fails with a message saying,
"The host needs to be manually added to vCenter." To successfully perform the deploy
action, do the following:

 Log in to the ESXi host to be added into vCenter.

 Power off the controller VM.
 Add the host to the HyperFlex group in vCenter.
 In the HyperFlex Installer, click Retry Deploy.

Load the ESXi image into the virtual media of the new server:

 In Cisco UCS Manager, navigate to Equipment > Rack-Mounts > Servers, find
your server, right-click it, and choose KVM Console.
 In KVM:

o Choose Virtual Media and Activate Virtual Devices.

o Again, choose Virtual Media, and, this time, choose CD/DVD. Pick Choose
File, and find your Cisco custom ESXi image.
o Click Map Drive.

 Go back to Cisco UCS Manager, right-click your server, choose Reset, and then
choose Power Cycle as the reset option. Click OK.
The figure shows an example of a custom Cisco ESXi image version 6.5 U2 being
mounted to the server. The important thing to remember is that the ESXi version must
match the version that is deployed on the ESXi hosts on the existing installation.

Note
Be very careful not to restart or reimage one of the production servers.

Return back to KVM:

 Depending on your profile configuration, you may need to press F6 during boot and
select vKVM media from the boot menu.
 Wait for ESXi prompt and choose the appropriate option.
 Installation of ESXi is a destructive process and cannot be reversed.
 The rest of the installation process is hands-off, wait for it to complete.
 After ESXi is successfully installed, delete the temporary service profile you have
previously created.
Note
The process described is not suitable for upgrades.

If you fail to get the prompt shown in the image, you are probably installing from a non-
customized version of the ESXi.

HyperFlex Expansion Workflow

In the HXDP installer, choose Expand Cluster and then Standard Cluster:

 The same workflow applies to expanding a deployment with either converged or

compute nodes.
 Fill out all the details similarly as you would in the installation process.

o With compute nodes, you do not need IP addressing for storage controllers
(neither management nor data) since there is no local storage.
If you skipped the imaging of compute-only nodes before running the expansion
process, the expansion will fail, and the installer will prompt you to image the server
with ESXi:

If the node you are trying to add to the installation is still associated with another
service profile, the new server will not be listed as an option for expansion. The
associated server will be listed in the associated tab. You can disassociate the server
from the installer by choosing the Actions option or going into the Cisco UCS Manager
and disassociating the profile manually. In either case, the server will be listed as
available for expansion.
After successful expansion, do not forget to perform post-installation tasks and test the
functioning of the expanded system.

To perform post-install tasks, connect (using SSH) to the installer VM, log in using root
credentials, and execute the post_install command. You will be prompted to choose one
of three options. Choose the Expanded Cluster option. The configuration of post-
installation tasks is the same procedure as configuring them with the initial installation;
the only difference is that the configuration this time around is only for the node that
was used to expand the system.

Rebalancing Storage After Expansion

Rebalancing re-distributes the storage across the HyperFlex nodes, so that they are
utilized evenly. Usually, you do not need to perform this task after expansion, since it is
done automatically every night.

Rebalancing manually makes sense when you are running very low on storage and you
expand your HyperFlex because you need space urgently the very moment that the
expansion is complete.

Rebalancing only applies to converged nodes, since compute-only nodes do not add
storage to the storage system.

When you add a converged node to an existing storage system, the system continues to
have the same high-availability resiliency as the original storage system until auto-
rebalancing takes place at the scheduled time:

 Rebalancing is typically scheduled during a 24-hour period, either 2 hours after a

node fails or if the storage is out of space.
 If you need to rebalance the storage before the scheduled time, initiate
the rebalance storage command.

o stcli rebalance start --force

o stcli rebalance status

The default scheduled rebalance time is 00:15AM, every night.

5.15 Installing and Expanding Standard ESXi Cisco HyperFlex

Additional Installation Options

This video reviews some additional installation options. Let's start with Enhanced vMotion
Compatibility, or EVC, which is disabled by default. EVC limits the CPU instruction set on the
newer processors in the cluster to the lowest-common denominator.

Now, if you enable EVC when expanding an M4 cluster with M5 nodes, that's not disruptive.
However, if you enable EVC later, it will be disruptive.
This figure here lists the steps you must perform. First, you power off all the guest/application
VMs and controller VMs. Shut down HyperFlex using the cluster shutdown command. And
put the hosts into the maintenance mode.

Next, enable EVC on the cluster using the vSphere web client. Then exit the maintenance
mode on all the servers. The CVMs will come online automatically. Lastly, start HyperFlex using
the cluster start command. Wait, and when the cluster is healthy, power on all the
guest/application VMs.

If you remember, vCenter can either be installed as an external server, which is the
recommendation, or you can run a Cisco HyperFlex cluster—within the HyperFlex cluster, I
should say, and that's referred to as a nested vCenter.

The nested vCenters remove the requirements for an extra host machine. However, there are
some certain requirements that you've got to meet. First of all, you must use installer version
2.6-1a or later. vCenter must be installed inside a VM, and cluster expansion can only be
performed after it is registered to a vCenter. Lastly, when installing vCenter, choose the option
Embedded Platform Services Controller.
There is also some limitations associated with a nested vCenter deployment. vCenter has
limited autostart capability. Only online upgrade is supported. And if the storage cluster is
down, then vCenter supported-related operations must be performed directly on the ESXi
hosts. Lastly, snapshots of a nested vCenter, they're not supported.

So let's take a look at the installation workflow. The first step here is to install the HyperFlex
cluster without registering vCenter. This is accomplished by leaving the just vCenter credential
fields blank. You'll get a warning message that said, hey, wait a minute. vCenter is required.
But just ignore it, and press Continue to acknowledge that you will register vCenter after the
installation is complete.
After the installation of the HyperFlex cluster is complete, then you need to perform the
following steps. Deploy vCenter on one of ESXi hosts in the cluster. Create a data center and
cluster with the names that you specified during the installation. Lastly, register vCenter with
HyperFlex from the st CLI.

The figure shows an example of the command syntax that you would use.

A specific BIOS policy change is when you install Cisco HyperFlex nodes with the Graphic
Processor Units, or GPUs. A BIOS setting that allows for more than 4 gigs of memory, mapped
I/O is required for all supported GPU cards.

The UCS server service profile for HyperFlex servers does have this setting enabled by default
when a GPU is present. However, if the GPU card is installed before the cluster was created,
then, during the cluster creation, choose the Advanced Workflow.

This figure lists the following steps. You've got to choose "I know what I'm doing. Let me
customize my workflow." That's scary. Next, check Run Cisco UCS Manager Configuration, and
click Continue.
Then verify that the BIOS setting for MMIO is above 4 gigs, and set it to Enable. If you have to
enable the setting, reboot the servers. Lastly, you go back to the Advanced Workflow and
create the HyperFlex cluster.

During the installation, all of the networks are configured with standard vSwitches. Using
Distributed Virtual Switches, or DVS, is an optional step. It's not required.

But distributed virtual switches ensure that each node is using the same configuration. What it
does is it helps prioritize traffic and allows other network streams to use all of the available
bandwidth.

This figure here lists the steps for migrating non-HyperFlex networks to a DVS after the
installation. First, you would create the DVS switch and port groups inside of vCenter. Then
you would migrate the vSwitch VM network. Next, migrate the vSwith from the Standard to
the DVS. And lastly, verify that there is no impact to the virtual machines.

Finally, the Cisco HyperFlex installer offers a simple workflow that you can use to reimage ESXi
servers before you actually start the installation. This is accomplished using the Advanced
Workflow option.
First, in the installer, click "I know what I'm doing. Let me customize the workflow." Next,
select Only Run UCS Manager Configuration. Then mount the Cisco custom version of the ESXi
ISO on each server.

When you restart the servers, select the appropriate installation workflow. After all ESXi
images are reinstalled, you're ready to finish the HyperFlex installation. But by this time, you're
just going to use Create HX Cluster option. That brings us to the end of this video. Thanks for
watching.

There are several additional options available when installing

HyperFlex. These options can depend on your deployment and infrastructure
needs, such as nested vCenter, or hardware specifics, such as the presence of GPUs in
the HyperFlex system.

Additional options listed here are described in steps to give you a reference point when
you are installing your HyperFlex deployment. None of the additional options are
mandatory.

This topic discusses a few non-mandatory installation options and considerations:

 Enabling VMware EVC

 Deploying Cisco HyperFlex with nested vCenter
 Installing NVIDIA GPUs in HyperFlex
 Migrating non-HyperFlex networks to VMware vSphere Distributed Switch (VDS)
 Re-imaging ESXi hosts before installation
Installation Option: VMware EVC
Cisco Hyperflex Series Servers have had three different generations of Intel Xeon
processors. HyperFlex UCS M4 had Intel Xeon, code name Haswell, and later Intel
Xeon, code name Broadwell. Cisco UCS M5 uses Intel Xeon scalable CPUs. The
second generation of Xeon Scalable CPUs carries the codename Cascade lake.

You can mix Cisco HyperFlex Series Servers with different generations of processors
within the same server group. However, you need to enable VMware EVC for vMotion
to work between servers with different generation of processors. Same considerations
apply when you are adding a compute node to the system. You need to make sure that it
is using the same generation of CPU as the nodes in the group you want to add the node
to. Alternatively, you need to make sure that VMware EVC is enabled (recommended).

EVC limits the CPU instruction set on the newer processors in the group to the highest
common denominator. Therefore, VMs can run on any node using the same instruction
set.

EVC is disabled by default. When expanding your M4 system with M5 nodes, you can
enable EVC without disruption. If enabling EVC later, it will be disruptive:

1. Power off all guest/application VMs and CVMs, shut down HXDP using the hxcli
cluster shutdown command, and put all hosts into maintenance mode.
2. Enable EVC on the system through the vSphere Web Client.
3. Exit maintenance mode on all servers; CVMs will come online automatically.
4. Start HXDP using the hxcli cluster start command. Wait and make sure that the
storage is healthy, then power on all guest/application VMs.
Migration of VMs using vMotion may fail, even if they are within an EVC group, under
these scenarios:

 When a host is not connected to a vCenter Server system.

 When a host is not configured for vMotion.
 If the VM does not reside on storage shared by the source and destination hosts.

Note
Upgrade vCenter to 6.5 or later. Without vCenter 6.5, Broadwell EVC mode on
disparate CPU generations cannot be enabled. Only vCenter upgrade is required, ESXi
can remain on an older version (subject to the VMware software interoperability
matrix). Proceeding with EVC mode off is not supported and will cause operational
issues in the future.

Installation Option: Nested vCenter

ESXi-based HyperFlex deployments (regular, stretched, Edge) require vCenter to run.
vCenter can be either external (recommended) or run on Cisco HyperFlex system,
which is commonly referred to as nested vCenter.

Using a nested vCenter removes the requirement for an extra host machine where
vCenter can be deployed but introduces certain limitations. Other common
infrastructure requirements, such as NTP, must still be provided.

Nested vCenter requirements:

 vCenter to be installed as a VM.
 HyperFlex expansion with additional nodes may be performed only after the
HyperFlex storage system is registered to vCenter.
 When installing vCenter, choose the Embedded Platform Services
Controller option. The External Platform Services Controller option is not
supported.

You can use either of two methods to install a vCenter VM on an HX storage system
(for example, nesting vCenter on Hyperflex):

 Installing a vCenter VM as part of the HXDP installation. This method allows you
to install a vCenter VM after the HyperFlex storage system is installed. It does not
require any additional hardware as compared to the second option.
 Using a vCenter VM stored on an external server. This method uses the typical and
recommended external vCenter to install and configure the HX Data Platform and
storage system, and then migrates the vCenter server to reside on the HyperFlex
storage system. In this context, an external server means a server that is not included
in the HyperFlex storage system. For example, it could be on an NFS mount.

Since an external vCenter is the recommended option and you would only retreat to
having a nested one if external infrastructure is not available, the first out of the above
options would be more popular. This topic focuses on that procedure. The second way
to install vCenter on HyperFlex, as listed just above is to use the vCenter VM stored on
an external server and then migrate it onto HyperFlex. You would deploy the vCenter
VM on an external server, follow the regular installation procedure for HyperFlex, and
finally migrate the vCenter from the external server onto HyperFlex using VMware
vMotion.
 Note
The minimum version to support nested vCenter is HXDP 2.6(1a).

The following are the limitations of hosting vCenter on HyperFlex:

 vCenter has limited auto-start capability.

o Properly configured high availability and DRS are needed for vCenter to move
gracefully around.
o If the entire HyperFlex system suffers a power interruption, high availability
does not restart the vCenter.
o If the system is gracefully shut down, manually power on vCenter from the
ESXi host.

 Only the online upgrade is supported while hosting a nested vCenter.

o As each HyperFlex node is brought down during a rolling upgrade, the vCenter
VM migrates automatically if DRS is enabled. If DRS is not enabled, the
vCenter VM must be manually migrated during upgrades.

 When the HyperFlex storage system is down or must shut down, then any support-
related operations completed through vCenter, must be performed directly on ESXi
hosts.
 HyperFlex snapshot of nested vCenter is not supported.

Also, snapshots cannot be deleted when the HyperFlex storage system is in the
ENOSPC (means that there is no space on storage) state. Deleting snapshots is
frequently used to reclaim space for an HyperFlex storage system. When a HyperFlex
storage system enters ENOSPC, the VMs hosted on that HyperFlex storage system can
no longer perform writes—which includes the vCenter VM. Therefore, the vCenter
hosted on the HyperFlex storage system cannot perform actions, including deleting
snapshots to help bring the storage system out of ENOSPC. Use an ESXi host
command-line option to delete snapshots.

Note
For more about why you should not take snapshots of a nested vCenter, read the
VMware KB article at https://kb.vmware.com/s/article/2003674.

Nested vCenter: Installation Workflow

If you want to install vCenter on HyperFlex, perform these steps:

 Install HyperFlex without registering vCenter.

 Deploy vCenter onto HyperFlex.
 Manually create the Data Center–Group–Host hierarchy in vCenter and register
vCenter with HyperFlex.

Proceed with standard installation workflow, but leave vCenter credentials fields empty.
Acknowledge that you will register vCenter after HXDP is installed.
After you perform initial installation and you log in into HyperFlex Connect, you will
see the following message:

After successful installation of HyperFlex, follow these steps:

 Deploy vCenter onto one of the ESXi hosts in the HyperFlex deployment.
 Create data center and group with the names you specified during the installation.
 Register vCenter with HyperFlex from using stcli from the CVM with CIP-M:

stcli cluster reregister --vcenter-datacenter <DC_name> --vcenter-cluster

<Cluster_name> --vcenter-user <User_name> --vcenter-url <IP>
Note
You must perform registration of your HyperFlex system to a vCenter from the system
IP manager (CIP-M) VM.

Installation Option: Cisco HyperFlex with GPUs

A specific BIOS policy change is required when installing Cisco HyperFlex nodes with
GPUs. All supported GPU cards require enablement of BIOS setting that allows greater
than 4 GB of Memory Mapped I/O (MMIO). Because HyperFlex servers are integrated
with Cisco UCS Manager and controlled by a service profile, this setting is enabled by
default in the service profile when a GPU is present.

If the GPU card is installed before HyperFlex is installed, then, during installation,
choose the Advanced workflow:

 On the HXDP installer page, choose I know what I’m doing, let me customize my
workflow.
 Check Run Cisco UCS Manager Configuration and click Continue. This action
creates the necessary service profiles for the HyperFlex nodes
 Verify that BIOS Setting by setting MMIO Above 4-GB configuration is set
to Enabled.

o If it is not, enable it and you will need to reboot the servers.

 Go back to the Advanced workflow on the HX Data Platform Installer page to

continue with Run ESX Configuration, Deploy HX Software, and Create HX
Cluster to complete system creation.

Installation of HyperFlex needs to be split if using GPUs because you need to make sure
that, as previously discussed, special BIOS settings need to be configured.

Note
If you want to install the GPU after the HyperFlex system is created, then you must
modify the service profile. Enable MMIO Above 4 GB. You will need to reboot the
server.
Installation Option: vSphere Distributed Switch
Using VDS is an optional and not a required step.

During installation, all the networks are configured with standard vSwitches. You can
migrate non-HyperFlex networks to VDS after:

 The HXDP can use VDS for non-HyperFlex dependent networks.

o vMotion networks
o Guest networks

 Steps summary to migrate to VDS:

1. From vCenter, create VDS and port groups.

2. Migrate the vSwitch, VM Network.
3. Migrate the vSwitch from standard to VDS.
4. Verify that there is no impact on the VMs regarding I/O, Network connectivity, and
VM Migration.

Distributed switches ensure that each node is using the same configuration. It helps
prioritize traffic and allows other network streams to utilize available bandwidth when
no vMotion traffic is active.

For detailed steps for migrating to VDS, please follow the Cisco HyperFlex Systems
Network and External Storage Management Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/Network_External_Storage_Management_Guide/
b_HyperFlex_Systems_Network_and_External_Storage_Management_Guide_3_0.pdf)
available on the Cisco site.

The HXDP has dependency that these networks use standard vSwitches:

 vswitch-hx-inband-mgmt: Storage Controller Management network

 vswitch-hx-inband-mgmt: Management network
 vswitch-hx-storage-data: Storage Hypervisor Data network
 vswitch-hx-storage-data: Storage Controller Data network

Re-Imaging ESXi Hosts Before Installation

Cisco HyperFlex Installer offers a simple workflow to reimage ESXi servers before you
start the installation. You can do that by going with the advanced workflow within the
installer and breaking up the installation into:

 Running the Cisco UCS configuration from the installer

 Re-imaging the ESXi hosts
 Completing the installation

If you want to reimage hosts before you begin the HyperFlex installation, use the
advanced workflow:
 On the first page of the installer, click I know what I'm doing, let me customize
my workflow.
 Choose only Run UCS Manager Configuration to configure and apply service
profiles to servers and gain access to their KVMs.
 Mount the Cisco-custom version of ESXi ISO on each server, restart the servers,
and choose a suitable installation workflow.
 Once each ESXi is re-installed, finalize the installation. This time, by clicking Run
Hypervisor Configuration, Deploy HX Software, and Create HX Cluster.

Why would you reimage ESXi hosts before installation as described above?
  One reason would be if servers have been used in a HyperFlex installation before.
For example, if you are cleaning up this node from a Proof of Concept (PoC), then
you should also clean up the Fabric Interconnects by erasing the old HyperFlex sub-
org and VLANs. When installing HyperFlex, do not forget to choose the option to
clean up disk partitions.
 Another reason may be a preference of workflow, where you associate servers first,
then perform ESXi upgrades, and finally install HyperFlex.

Hardware Acceleration Card Installation

Remember that, as of HXDP 4.5(1a), HyperFlex hardware acceleration cards are only
supported when:

 Using HX240c M5 servers that are either hybrid (SFF or LFF) or All-Flash.
 Using standard ESXi-based HyperFlex installations.
 Hardware acceleration cards are present in all nodes and only with initial
installations. Cards cannot be added after installation of HyperFlex.
 You are not using HXDP native replication.

To install a HyperFlex system with hardware acceleration cards:

 Make sure that you are familiar with the prerequisites and limitations of these cards.
 Perform the regular installation procedure.
 After installation is complete, verify that the hardware acceleration cards are
successfully installed:

o Log in to CVM using the root user.

o Verify /opt/springpath/config/offload.tunes is present. If it exists, the
installation succeeded.
o Check for any potential error messages or events in HX Connect.

 You should observe the compression gain value in HX Connect.

Note
If installation is operational, but you are getting generic alerts in vCenter and HX
Connect, contact Cisco TAC for assistance. If an error is detected during installation
when the card goes offline, and an event is thrown in the user interface and the vCenter,
contact Cisco TAC for assistance.

5.16 Installing and Expanding Standard ESXi Cisco HyperFlex

Install Cisco HyperFlex

Install Cisco HyperFlex
So far, you performed lab exercises on the physical server installations that the labs are
running on. In this lab exercise, you will explore Cisco HyperFlex functions, which
require you to have full administrative privileges.
You will use a virtual Cisco HyperFlex environment that is running on the infrastructure
that you have already explored. You will access the virtual environment through your
Student VM.

Because Cisco HyperFlex is a software and a hardware solution, some functionalities

that are available in physical HyperFlex deployments will not be available in this
software environment since the underlying hardware is virtualized.

The virtual environment has the following characteristics:

 HyperFlex nodes are emulated using VMs running nested ESXi installations.
 Cisco UCS Manager is emulated using Cisco UCS Platform Emulator, which
emulates Cisco UCS Manager running on Fabric Interconnects.
 You will use a vCenter installation, which is a standard version without any
customizations.
 The HyperFlex Installer is heavily modified for the virtual environment, but it
simulates an actual installation as closely as possible.
 The infrastructure is connected by virtual switches in a vSphere environment. The
connectivity is truly working and is configured by the HyperFlex Installer.

Get Acquainted with the Virtual Environment

The lab environment is split into two parts. The first part is an actual hardware-based
installation of HyperFlex where you only have read-only access. The second part is the
virtual environment pods, running on the underlying HyperFlex installation, where you
have administrator access to the environment.

The virtual pods use IP subnetwork 10.1.100.0/24 and the DNS domain suffix .pod. The
vCenter virtual pod is located at https://vcenter.hx.pod, while
the https://vcenter.hx.lab is the vCenter physical infrastructure, where you have read-
only access.

Your individual pod infrastructure includes one Cisco HyperFlex UCS domain with a
set of nested ESXi installations.

The Student VM is your access point to both the physical and virtual infrastructure.
Your Student VM also serves as the NTP and DNS source for the HyperFlex
installation.

Step 1

Explore the management interfaces of HyperFlex Cluster A.

Prepare Preinstallation Information

Before you install Cisco HXDP, you must first set up the environment according to the
HyperFlex requirements. You will need to choose VLANs, define the IP and MAC
address space, set up DNS and NTP, and choose other prerequisite values.
The entire process of installation is described in the Cisco HyperFlex Installation
Guide, which is available on the Cisco website
at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Dat
aPlatformSoftware/Installation_VMWare_ESXi/4-5/b-hx-install-guide-for-vmware-
esxi-4-5.html. Before the installation process, the main considerations that you need to
resolve are condensed in the Preinstallation Checklist document, which is also available
on the Cisco website
at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Dat
aPlatformSoftware/HyperFlex_Preinstall_Checklist/
b_HX_Data_Platform_Preinstall_Checklist.html.

The preinstallation checklist is available in a modifiable .pdf format, and it lists the
requirements for HyperFlex installation, along with all the information that you will
need to provide to the installer.

You can find the example of the preinstallation checklist for the Cluster B installation in
the DCIHX folder of your Student VM.

Step 2
Open /home/student/DCIHX/Preinstallation Checklist/ and review
the Preinstallation Checklist document.

Step 3
Open and review the filled-out version of the Preinstallation Checklist.
Another useful tool that can be used for preparing preinstallation information is called
the HX Pre-Install Tool. It is accessible on the Cisco
website https://hyperflexsizer.cloudapps.cisco.com/an/hx-preinstaller/ and requires
CCO (Cisco account) login to be accessed.
HX Pre-Install Tools allow you to create, validate, and store many HyperFlex group
configurations before the start of HyperFlex group deployment. You simply create a
new group profile and input group configuration parameters such as type of
deployment, number of nodes in the group, VLANs, IPs, vCenter details, and DNS
parameters. Compared to the preinstallation checklist that was already mentioned, the
HX Pre-Install tool is much better since it also validates configuration parameters so
that no necessary parameters are missing, and, for example, that there are no duplicate
IPs and VLANs. If there are any inconsistencies or miss configurations, you will get a
warning as well.

Once you are done, the tool allows you to push the configuration directly to Cisco
Intersight by using the Intersight API. In case you are deploying your HyperFlex group
with Intersight, or by exporting the JSON installation file, that can be imported to the
on-prem version of the HyperFlex installer.

The tool also comes in an on-prem OVA version that can be deployed locally and used
with a combination of Intersight Virtual Appliance or Private Virtual Appliance.

Note
Keep in mind that the website where this tool is located also offers other tools such as
HyperFlex Bench and HyperFlex Profiler.

Since the group that you will install in the following tasks will be a virtualized one, you
will not be using the HX Pre-install tool to create the JSON installation file. Instead,
you will follow the filled-out preinstallation checklist, which is already provided.

Perform Preinstallation Tests

The first part of the Preinstallation Checklist contains an extensive list of fixed
requirements that you need to fulfill.
After you deploy the HyperFlex installer on your infrastructure, you need to verify that
all the services are reachable and available on the network.

Note
Both vSphere and Hyper-V installers are Ubuntu-based Linux VMs, so the commands
are identical in both installers.

Step 4
Verify that the NTP server is operational in the management VLAN.

Answer
Click the white and blue icon in the top-left corner of the screen to choose Terminal
Emulator from the menu and open the terminal of your virtual machine.

The VM has access to the management network of your HyperFlex group. Type ntpdate
-d 10.1.100.3 in the command line of the Student VM and press Enter to verify the NTP
server on IP 10.1.100.3 is active.

student@student-vm:~$ ntpdate -d 10.1.100.3

3 May 11:16:35 ntpdate[14775]: ntpdate 4.2.8p10@1.3728-o (1)
Looking for host 10.1.100.3 and service ntp
10.1.100.3 reversed to student-vm.hx.pod
host found : student-vm.hx.pod
transmit(10.1.100.3)
receive(10.1.100.3)
transmit(10.1.100.3)
receive(10.1.100.3)
transmit(10.1.100.3)
receive(10.1.100.3)
transmit(10.1.100.3)
receive(10.1.100.3)
10.1.100.3: Server dropped: strata too high
server 10.1.100.3, port 123
stratum 16, precision -25, leap 11, trust 000
refid [10.1.100.3], delay 0.02565, dispersion 0.00000
transmitted 4, in filter 4
reference time: 00000000.00000000 Thu, Feb 7 2036 6:28:16.000
originate timestamp: e43a5a19.2c8420d5 Mon, May 3 2021 11:16:41.173
transmit timestamp: e43a5a19.2c7dd9c2 Mon, May 3 2021 11:16:41.173
filter delay: 0.02567 0.02570 0.02565 0.02567
0.00000 0.00000 0.00000 0.00000
filter offset: 0.000010 -0.00000 0.000006 0.000003
0.000000 0.000000 0.000000 0.000000
delay 0.02565, dispersion 0.00000
offset 0.000006

3 May 11:16:41 ntpdate[14775]: no server suitable for synchronization

found

The NTP configuration is very important in the HyperFlex installation. An incorrectly

synchronized time can cause various problems, so make sure to check the NTP server
before installing HyperFlex.

If you have problems with the HyperFlex installation or SSO logins, check that the time
and date on all the devices is the same.
Note
The error returned at the end of the ntpdate output is caused by the fact that you are
testing NTP from the same machine the NTP server is running on. The receive and
transmit entries tell you the server is active, much like the ping command would.

To check the time on a windows machine, use the w32tm command to verify that the
NTP server is functional. The syntax to query NTP server 10.1.100.3 is w32tm
/stripchart /computer:10.1.100.3.

Step 5
Connect to the HyperFlex Installer VM by using SSH.

Answer
Now type the ssh -l root installer.hx.pod command in the terminal window and
press Enter. Log in by using the password C1sco12345.

student@student-vm:~$ ssh -l root installer.hx.pod

root@installer.hx.pod's password:
Last login: Mon May 3 03:31:58 2021 from 10.1.100.3
root@HyperFlex-Installer-4.5.1a:~#

Now you are connected to the command line of the HyperFlex Installer VM.

Note
By using the HyperFlex installer to verify the availability of required services, you not
only verify that the services are available, but also that the HyperFlex Installer is
properly configured. A misconfigured DNS in the Installer can, for example, lead to
installation issues even though the DNS server itself is properly configured.

Step 6
Verify that the DNS server is operational in the management VLAN.

Answer
Type nslookup vcenter.hx.pod 10.1.100.3 in the command line of the Installer VM and
press Enter.

root@HyperFlex-Installer-4.5.1a:~# nslookup vcenter.hx.pod

Server: 10.1.100.3
Address: 10.1.100.3#53

Name: vcenter.hx.pod
Address: 10.1.100.5

The command queries a specific DNS server (10.1.100.3) to derive an IP address from
the A record in DNS and returns the IP address for the vcenter.hx.pod domain name.
The DNS is not required for the vSphere installation of Hyperflex, but it is required in
Hyper-V installations. If you want the HyperFlex infrastructure to resolve domain
names, you still need to provide a working DNS server.

The A and PTR records of the Hyperflex infrastructure need to be configured on the
DNS server before the names are available for resolution.

Note
The nslookup syntax is the same on Linux and Windows.

Step 7
Verify that the service port is reachable.

Answer
Type nc -zv esx-b-1.hx.pod 80 in the command line of the installer VM and press Enter.

root@HyperFlex-Installer-4.5.1a:~# nc -zv esx-b-1.hx.pod 80

Connection to esx-a-1.hx.pod 80 port [tcp/http] succeeded!

The procedure can be performed for each individual host and port on the requirements
list. The HyperFlex infrastructure is already preconfigured to allow appropriate ports, so
port testing is most often performed through a router, to verify the router configuration.

Note
The necessity of port testing is mostly determined by the deployment configuration in
your specific case.

Step 8
Verify that the vCenter is reachable.
Answer
Type ping vcenter.hx.pod in the command line of the installer VM and press Enter.
Press the CTRL+C keys simultaneously after a couple successful pings to terminate the
command.

root@HyperFlex-Installer-4.5.1a:~# ping vcenter.hx.pod

PING vcenter.hx.pod (10.1.100.5) 56(84) bytes of data.
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=1 ttl=64 time=0.825 ms
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=2 ttl=64 time=0.341 ms
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=3 ttl=64 time=0.384 ms
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=4 ttl=64 time=0.384 ms
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=5 ttl=64 time=0.389 ms
64 bytes from vcenter.hx.pod (10.1.100.5): icmp_seq=6 ttl=64 time=0.373 ms
^C
--- vcenter.hx.pod ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 4999ms
rtt min/avg/max/mdev = 0.341/0.449/0.825/0.169 ms
vCenter is reachable by the installer and there is no packet loss. The HyperFlex Installer
will be able to register the HyperFlex group and install the HyperFlex vCenter
management plug-in.

Install the Cisco HyperFlex Data Platform

Preparing the information for the Cisco HyperFlex installation is only one of the steps
that are required before you begin the HyperFlex installation.

Though you will not perform these tasks in this virtual lab environment, you will
perform the following steps in an actual standard group installation scenario:

1. Rack the equipment and cable it.

2. Configure the upstream network to allow connectivity and provide shared services.
3. Set up the Cisco UCS domain and discover the servers in Cisco UCS Manager.
4. Upload the HyperFlex platform-compatible version of the firmware packages to the
Cisco UCS Manager. Packages A, B, and C are required.
5. (Option) Reimage the ESXi installations on the HyperFlex servers if the version that
shipped with the HyperFlex servers does not match your desired version.
6. Deploy the installer VM on the shared infrastructure to install HyperFlex.

Step 9
Connect to the Cisco HyperFlex Data Platform Installer and choose the installation
workflow.

Answer
Connect to the Cisco HXDP Installer by typing https://installer.hx.pod/ in a new tab in
your browser. Log in by using credentials root / C1sco12345. Check the I accept the
terms and conditions checkbox and click Login.
Click the Create Cluster drop-down menu on the left side of the Workflow installer
page. Choose Standard Cluster in the drop-down menu.

The same installer can be used to install nearly all the VMware flavors of HyperFlex:

 Standard deployment requires a pair of Fabric Interconnects. You can use pre-
existing supported Fabric Interconnects. The Fabric Interconnects will be
reconfigured by the HyperFlex Installer, which is potentially disruptive. Consult
with Cisco TAC before attempting to install HyperFlex on pre-existing Fabric
Interconnects.
 Edge installations do not use Fabric Interconnects. Therefore, no Fabric Interconnect
configuration is necessary during install.
 Stretched deployment allows you to spread a HyperFlex group across two sites, each
site with its own pair of Fabric Interconnects. You need to configure individual sites
during installation.
 As of HyperFlex Version 4.5.1a, Edge deployments support 2, 3, or 4 node
deployments, which cannot be expanded. You can install all the Edge deployments
from a cloud-based installer. The 2-node Edge deployment requires cloud installer
and cannot be installed from a local HyperFlex Installer. The 3 -and 4-node Edge
options can be installed locally.

Note
Below the standard workflows, you can enable the advanced installation. This advanced
option allows you to perform individual steps of the installation. Although these
advanced workflows are useful in certain situations, they require advanced knowledge
of HyperFlex and can cause overlaps, failed installation checks, or other installation
issues, if used improperly.

Step 10
Provide management credentials to the installer. Use the Preinstallation
Checklist document as a reference.

Answer
Return to the filled out Preinstallation Checklist document and scroll down to
the Deployment Information section. Follow the filled-in information to fill out the
required installer fields.

Provide the credentials for Cisco UCS Manager and vCenter.

Note
If you have just reimaged your HyperFlex servers with ESXi installations and the ESXi
hypervisors do not have the administrative password Cisco123, you have not deployed a
Cisco custom image. You will need to install the Cisco custom ESXi image on your
servers before you can install HyperFlex. The Cisco custom ESXi image can be
downloaded from https://software.cisco.com/. vSphere HyperFlex installations require a
custom Cisco ESXi installation.

Click Continue to proceed to the Server Selection step.

Step 11
Choose the servers for the HyperFlex installation.

Answer
Choose the first three servers (Server 1–3) in the Server Selection step to build your
group. If you do not see all four servers, refresh the installer by clicking
the Refresh button on the top right.

The Associated tab, which is next to the Unassociated tab, lists the servers that have a
service profile associated in the Cisco UCS Manager. Those servers are not available for
HyperFlex installation and are already associated with a service profile in Cisco UCS
Manager.

Click the Configure Server Ports link on the top right.

To configure server ports, you need to know to which ports your HyperFlex servers are
connected and you need to know how many slots the Fabric Interconnects have, to
properly define the fabric / slot / start_port - end_port syntax. Third- and fourth-
generation Fabric Interconnects only have one slot. Please note that you must define the
server ports on both fabric interconnects even though HyperFlex requires the cabling to
be consistent on both Fabric Interconnects (same servers to same ports).

Click Cancel to go back to Server Selection step, then click Continue to proceed
to UCSM Configuration step.

Step 12
In the UCSM Configuration step, provide the VLAN information. Use
the Preinstallation Checklist as a reference.

Answer
Fill in the required installer fields according to the filled-out Preinstallation
Checklist document. You can also refer to the Job Aids.
In actual installations, the VLANs you provide must be properly configured on the
upstream network:

 Management VLAN: Must have access to all the management interfaces of ESXi
hosts, CVMs, vCenter, the HyperFlex Installer, Cisco UCS Manager, and your
computer.
 Storage data VLAN: Should always be unique to a single HyperFlex group and
used exclusively for HyperFlex data traffic.
 VM network: The VLAN that you will use for your workload VMs deployed on
the HyperFlex group. This VLAN should be reachable by the end user or service.
 vMotion: The vMotion network must be shared between hosts in the same group for
high availability and VM migration. It can also be shared by all the hosts in a
vCenter, so you can move VMs between groups.

Click Continue to proceed to Hypervisor Configuration step.

Step 13
Configure the hypervisor addressing.
Answer
Fill the Hypervisor Configuration step by referring to the filled out Preinstallation
Checklist document. Use ESXi hostnames HX-01, HX-02 and HX-03, if checked,
uncheck The hypervisor on this node uses the factory default password option and keep
the suggested password.

The unchecked option for hypervisor default password is a limitation of the virtual
environment. Whether or not you would check this check box in a real installation,
depends on whether the ESXi hypervisors on individual nodes have the default
password Cisco123 or not. In the lab environment, the ESXi password was changed
to C1sco12345.

In the Hypervisor Configuration step, you define the management IP addresses of the
ESXi hypervisors. These interfaces are assigned to the management VLAN that you
defined in the previous step.

The hostname you configured will be assigned to the hypervisor, and the domain that
you provide in the Cluster Configuration step is added as the suffix to the hostname.
The hostname and domain name do not significantly affect the operation of the
installation, because the vSphere based HyperFlex installation is independent of DNS.
The IP addresses, their subnet masks, and gateway that you provide here are configured
in the management VLAN of the ESXi hosts automatically by the installer and do not
need to be configured manually.

Note
Be careful that the IP addresses you assign are reachable by all the Cisco HyperFlex
components in the management VLAN. If you are accessing parts of the infrastructure
through a router, configure your routing and ACLs to allow connectivity to the IP
addresses that you assign in this step. Also, be careful not to cause IP address conflicts
on the network.

Click Continue to proceed to IP Addresses step.

Step 14
Configure the storage data VLAN and CVM addressing.

Answer
Follow the filled-out Preinstallation Checklist document to fill out the IP
Addresses installer step. You can also refer to the Job Aids.

In the IP Addresses step, you provide the addressing in the management and storage
data VLANs. You already configured the ESXi host management IP addresses in
the Hypervisor Configuration step, so they are grayed out. You have also already
defined the management VLAN subnet mask and gateway.

The addresses within a VLAN need to be reachable. While the IP addresses in the
management VLAN can be routed through a router, the data VLAN IP addresses cannot
be routed. The HyperFlex Installer will warn you if you provide the wrong subnet mask
or IP addresses in the storage data VLAN.

The Cluster IP Address field in the management VLAN is where the HyperFlex
Connect will be accessible. The IP address will be assigned to one of the CVMs as a
virtual IP address (VIP). If a node fails, another node will be reconfigured with this
same IP automatically, so that the HyperFlex Connect IP is always reachable. You can
also access any of the management IP addresses of the CVMs to access HyperFlex
Connect on a specific node.

The Gateway Address field is only applicable to the management network, since data
network only supports Layer 2 connectivity.

The storage controller IP addresses will be assigned to CVMs after they are deployed on
individual hypervisors.

Note
In a real installation, you can reorder the servers by clicking the sandwich icon at the
beginning of each server row. Be careful to order the node addresses so that they reflect
their sequence in Cisco UCS Manager if that is important to you. It is best to compare
the serial numbers of nodes with those in Cisco UCS Manager.

Because this is a virtual environment, the order was already set.

Click Continue to proceed to the Cluster Configuration step.

Step 15
Configure the Cisco HyperFlex group information.
Answer
Follow the information in the filled-out Preinstallation Checklist document to fill the
required installer fields. You can also refer to the Job Aids.

To avoid issues with the virtual environment, follow these guidelines:

 Do not change the options in the Advanced Configuration submenu.

 Choose UTC (London) as the time zone and select Replication Factor 2.
 Do not start the installation immediately after you fill out the information—do
not press Continue.
Here is some information on specific options in the Cluster Configuration step:

 The Controller VM username and password will be used to access the HyperFlex
Connect and SSH into the CVMs.
 The vCenter Configuration provides the data center name and group name for the
HyperFlex group. The data center is the parent category for the group. If you enter a
data center name that exists in the vCenter, the HyperFlex group that is being
installed will be added into the existing data center. Choose a unique group name,
because group names must not overlap.
 You tested the DNS and NTP servers in a previous step. They are provided here, so
that the installer configures them on the hypervisors and the CVMs.
 DNS Domain Name provides the domain suffix. If you are not using a DNS, leave
this field blank.
 Connected Services allow automated reporting and remote management to be
enabled during installation. Connected services can be configured later with
additional options, such as proxy servers.
 Jumbo Frames option in the Advanced Configuration subsection enables packet
frame size greater than the default of 1500. Enable jumbo frames only if your top-
of-rack switches support this feature and the feature is enabled on the ports where
you connect the Fabric Interconnect uplinks to the top-of rack switches.
 Clean up disk partitions is used when you are reinstalling HyperFlex. It is highly
recommended to also reimage the ESXi installations before reinstallation.
 Optimize for VDI only deployment changes how read caching works on the
server. Use this option if your HyperFlex group will mainly be used for VDI.
 vCenter Single-Sign-On (SSO) Server is a backwards-compatibility feature. You
will almost never use this option, unless it is specifically necessary. The SSO
between HyperFlex Connect and vCenter will be configured even if you leave this
field blank.
 The Replication Factor (RF) defines the redundancy of your installation. Choosing
this option depends on the workload that you will be running on the HyperFlex
group. While RF2 is more storage efficient, RF3 offers better redundancy, which is
very welcome when nodes are shut down for upgrade. Cisco recommends RF3. In
this exercise, choose RF2, since the virtual environment is for testing only, and will
carry no important data.

Note
During installation, all the information that you entered has been added to the
configuration list on the right. You can now compare the information from the
configuration list with the settings listed in the job aids.

If you want to change a setting from one of the previous pages, click the Back button in
the bottom-right corner to go back a step. Only passwords will be lost when going back
a step.

Step 16
Apply the saved configuration to the installer.
Answer
You can download the configuration you have entered by clicking the arrow down icon
in the top-right corner. You can also restart the installation and then apply a
configuration.

Click the gear icon in the top-right corner of the installer and click Start Over in the
menu. You will load a preconfigured version of the configuration to explore the import
function of the installer.
After you restart the installation, you will be presented by the workflow selection step.
Click the Create Cluster drop-down menu and choose Standard Cluster, which
forwards you to the first step of the standard group installation.

Click the Select a JSON File button on the right side to choose the configuration file.

Choose the ../Student/DCIHX/Configurations/Standard Cluster B.json file as the

configuration and click Open.
Click Use Configuration in the bottom-right corner to apply the configuration from the
JSON file.
Provide the passwords in all the steps that require them. Passwords are not saved in the
JSON file for security reasons.

 In the Credentials step, use password 1234QWer for the UCS Manager
Credentials.
 In the Credentials step, use password 1234QWer* for the vCenter Credentials.
 In the Cluster Configuration step, use password C1sco12345 as the Controller
VM admin password.

Step 17
Start the HXDP installation.
Answer
Go through all the installer steps by clicking Continue until you reach
the Cluster Configuration step. You will be prompted to switch to the Replication
Factor 3. Since this is a virtualized environment, click Keep Replication Factor as 2 to
proceed. Then provide the necessary passwords.

Click Continue in the Cluster Configuration step to start HyperFlex installation. You
will be asked to download the configuration before the deployment starts.
Installation will take approximately 15–20 minutes. During the installation, the installer
will report detailed actions taken. Follow the actions taken in various steps to learn how
the installer installs the HXDP.

After the installation is completed, the Installation Summary screen lists all the
information of the installed cluster.
Note
If the group installation summary page appears to be blank refresh the web page in your
browser until group details are visible.

Step 18
Open HyperFlex Connect of the group that you have installed.
Answer
On the Summary report, click the blue Launch HyperFlex Connect button in the
bottom-right corner of the screen to connect to the HyperFlex Connect of the group that
you have just installed.

Step 19
Review the group that you have installed in vCenter.
Answer
Open your browser, type https://vcenter.hx.pod/ in the URL bar, and then press Enter.

If prompted, choose the LAUNCH VSPHERE CLIENT (HTML5) interface, and log
in to the vCenter. Use credentials administrator@hx.pod / 1234QWer* to log in.

After the interface loads, click the black triangle next to the HyperFlex Datacenter in
the Navigator pane on the left. Then click the similar black triangle next to
the Standard Cluster B in the same pane.
Note
The hosts that you can see in the vCenter are running VMware Evaluation license and
because of that you will see an orange warning message displayed once you log in.
Ignore the message or close it by pressing the X in the right corner of the warning.

The group that you have just expanded is the group you have installed. If you click one
of the servers in Standard Cluster B, you will see a warning with a blue background in
the right pane.

The warning indicates that the HyperFlex installer enabled the SSH on the HyperFlex
ESXi hosts. SSH is always enabled on HyperFlex nodes, since it is used in certain
management actions.

In vCenter, click one of the Standard Cluster B hosts and navigate to

the Configure tab on the right. Choose Virtual switches under Networking in the
middle menu to see the configured vSwitches on the ESXi of the selected HyperFlex
node. Click the vmotion vSwitch in the Virtual switches list.
No port groups are defined on the vmotion vSwitch after the installation of the group.
To configure the vMotion interfaces and port groups on HyperFlex nodes, you need to
use the post_install script from the HyperFlex Installer command line. This will be done
in the last lab procedure, after you will expand the group.

Note
Not configuring vMotion will prevent online upgrades of the group. During the online
upgrade process while a node is offline, VMs on nodes are migrated to other nodes.

Expand Cluster B
You have installed a three-node group, but the number of nodes in a HyperFlex group is
not fixed. A major feature of Cisco HyperFlex is its ability to scale the infrastructure
either by adding hardware components like drives to nodes or by adding nodes to the
infrastructure.

Adding nodes to a group follows a similar procedure as the installation. The expansion
does not cause any downtime or additional load to the system. The additional node is
attached to a live group as it is performing its workload.

You will expand the group by an extra converged node.

Note
Normally, you would not expand the group with converged nodes immediately after
installing it—you would simply install more converged nodes. However, you can only
add compute nodes after the initial group installation.

When expanding a real group, be careful that the information for the additional nodes
matches to the original group.

 Use the same passwords for ESXi and CVMs you are using on existing group nodes.
 Use identical VLANs as in the original installation.
 Assign IP addresses, which are reachable by the existing interfaces of the same type.
 Follow all the same cabling, configuration, and compatibility requirements for the
nodes that you followed during the original installation.
Note
To help you with the installation, you can use the JSON file from the initial installation
and import it into the installer. The file will populate all the installer fields that are not
specific to the new nodes.

Step 20
Launch the workflow to expand the group in the HyperFlex installer.

Answer
Reopen the HyperFlex Installer by entering https://installer.hx.pod. Log in by using
the credentials root / C1sco12345. The installer will take you to the summary screen.
Click Back to Workflow Selection in the bottom right.

On the workflow selection screen, click the Expand Cluster drop-down menu and
choose Standard Cluster. Cisco HyperFlex Installer will take you to
the Credentials step.
Step 21
Import the .json configuration.

Answer
Click the Select a JSON File button in the Configuration pane on the right, click
the DCIHX link on the left, and open the Configurations folder. Choose the Expand
Standard Cluster B.json file to import the configuration values (except the passwords)
from the file. Click the Use Configuration button on the bottom right to populate the
installer fields.

Use the password 1234QWer for Cisco UCS Manager and the
password 1234QWer* for vCenter. Usernames are already populated by the
configuration file.
Click Continue to proceed to the Cluster Expand Configuration step.

Step 22
Choose the group to expand.

Answer
The Cluster Expand step lists one cluster to expand. Click the Standard Cluster
B square and the Management IP Address field will automatically be populated with the
IP 10.1.100.250, which is the management IP of your group.

Since the password is not saved in the .json file, enter the password C1sco12345 in the
password field.
Click Continue to proceed to the Server Selection step.

Step 23
Choose the server to add it to the group.

Answer

Make sure that the server selection check box is checked.

Note
If the server does not appear in the list, click the Refresh button. You may need to
click Refresh several times before the server appears. This limitation is not present on
hardware deployments, only in this virtual environment.

Click Continue to proceed to the UCSM Configuration step.

You will see that enough of IP addresses are left for group expansion.
Click Continue to proceed.

Step 24
Provide the hypervisor IP address, hostname, and password.

Answer
A unique IP address must be chosen for the hypervisor management interface and a
unique hostname for the hypervisor. The hypervisor IP addresses must be reachable
among themselves.
Click Continue to proceed to the IP Addresses step.

Step 25
Provide the IP addresses for the hypervisor storage data interface and CVM interfaces.

Answer
The management VLAN addresses need to be reachable among themselves. The data
VLAN addresses must be within the same subnet.
Note
It is a good practice to have the IP addresses of nodes assigned sequentially, but while
they need to be reachable, they do not really need to be sequential. To avoid
inconsistent addressing, leave sufficient sequential address space for individual node
interfaces to be used in potential expansions.

Click Start to start installation.

Step 26
Wait for the node to be added to the existing HyperFlex group.

Answer
The installation will go through the same steps as the initial group installation.
After the installation is complete, the installer presents a summary of the expanded
HyperFlex group.
Perform Post-Installation Tasks
The new node has been added to the HyperFlex group, but the post-installation tasks
have not been performed on the new node. Therefore, the new node is unavailable for
vMotion, high availability, and DRS.

You need to rerun the post_install command from the HyperFlex installer.

Step 27
Run the post installation script in the HyperFlex Installer command line.

Answer
Click the blue and white icon in the top-right corner of the student VM and
choose Terminal Emulator from the menu.

Type ssh -l root installer.hx.pod in the terminal command line. Use the
password C1sco12345 to log in to the HyperFlex Installer command line.

student@student-vm:~$ ssh -l root installer.hx.pod

root@installer.hx.pod's password:
Last login: Mon May 3 04:17:50 2021 from 10.1.100.3
root@HyperFlex-Installer-4.5.1a:~#

In the same terminal window that you entered

the ssh command, type post_install and press
the Enter key. Place the terminal window on top of the vCenter
window, so that the Recent Tasks pane is visible.

Execute the post_install command in the Cisco HyperFlex installer command line.
The post_install script will ask you a series of questions. Follow the example to answer
the questions. Use the vCenter username administrator@hx.pod and
password 1234QWer*.

root@HyperFlex-Installer-4.5.1a:~# post_install

Select post_install workflow-

1. New/Existing Cluster
2. Expanded Cluster (for non-edge clusters)
3. Generate Certificate

Note: Workflow No.3 is mandatory to have unique SSL certificate in the

cluster.
By Generating this certificate, it will replace your current certificate.
If you're performing cluster expansion, then this option is not required.

Selection: 2
Expanded Cluster workflow selected

HX cluster to connect to: 10.1.100.250

Logging in to controller 10.1.100.250
Getting ESX hosts from HX cluster...
vCenter URL: 10.1.100.5
Enter vCenter username (user@domain): administrator@hx.pod
vCenter Password:
Found datacenter HyperFlex Datacenter
Found cluster Standard Cluster B

post_install to be run for the following hosts:

esx-b-1.hx.pod
esx-b-2.hx.pod
esx-b-3.hx.pod
esx-b-4.hx.pod

Enter vSphere license key? (y/n) n

Enable HA/DRS on cluster? (y/n) n

Disable SSH warning? (y/n) y

Add vmotion interfaces? (y/n) y

Existing cluster configuration for reference

--------------------------------------------
Netmask Vlan-Id IP-Address
--------------------------------------------
--------------------------------------------

Enter Expanded node configuration-

Netmask for vMotion: 255.255.255.0
VLAN ID: (0-4096) 3093

Unable to query vMotion MTU from existing cluster, Using MTU from tune file :
1500
Do you wish to enter the range of vMotion IPs ?(y/n) y
Please enter vMotion Ip range (format: IP_start-IP_end) 172.16.93.211-
172.16.93.214
Vmotion ip 172.16.93.211 used for esx-b-1.hx.pod
Adding vmotion-3093 to esx-b-1.hx.pod
Adding vmkernel to esx-b-1.hx.pod
Vmotion ip 172.16.93.212 used for esx-b-2.hx.pod
Adding vmotion-3093 to esx-b-2.hx.pod
Adding vmkernel to esx-b-2.hx.pod
Vmotion ip 172.16.93.213 used for esx-b-3.hx.pod
Adding vmotion-3093 to esx-b-3.hx.pod
Adding vmkernel to esx-b-3.hx.pod
Vmotion ip 172.16.93.214 used for esx-b-4.hx.pod
Adding vmotion-3093 to esx-b-4.hx.pod
Adding vmkernel to esx-b-4.hx.pod

Add VM network VLANs? (y/n) n

Run health check? (y/n) n
root@HyperFlex-Installer-4.5.1a:~#

You have chosen the expanded workflow which normally configures just the nodes that
were recently added to the HyperFlex group, but since post installation tasks were
skipped after initial installation of the group, all nodes were configured using the
post_install script.

Step 28
Show Me

Verify actions taken by the post installation script.

Answer
Go into the vCenter interface and click the first host of Cluster B. On the right side of
the vCenter interface, click the Summary tab to see a host overview.

The SSH-enabled warnings that were there before you ran the post-installation script are
now gone. The script disables the warnings since they are irrelevant. HyperFlex ESXi
hypervisors always have SSH enabled.

On the right side of the vCenter interface, under Standard Cluster B, click one of the
four hosts that are listed. Click the Configure tab on top of the host pane on the right.

Near the middle of the screen, you will see a list with a scroll-down bar. Find
the Networking subsection and click VMkernel adapters to see the VMkernel adapter
configuration for the host.
You can configure the vMotion vmk2 interface by using the post_install
command.

You can see the following functions of the VMkernel interfaces:

 Management Network: Attached to the management vSwitch, you access the ESXi
hosts through this interface. Type the IP address of the management network
interface into the Student VM browser to access the ESXi management interface.
 Storage Data Hypervisor Data Network: An interface leading to a network, which
is used to transfer HyperFlex storage data. The data network interface is dedicated to
storage data transfer and is the means by which the ESXi Hypervisor and the
HyperFlex storage (CVMs) are connected.
 vmotion-3093: The vMotion interface that is used in DRS/high availability and
vMotion events. You have just configured this interface.

Now you have set-up your Cisco HyperFlex group, and it is ready for use.

Note
In a real deployment, the data on the group will be redistributed to the new node during
the next rebalancing cycle.

In this discovery, you have learned how to install the Cisco HyperFlex Standard group.
You covered everything from filling out the preinstallation document for the group,
performing preinstallation tests and validations, to the actual installation of the group
with the HyperFlex installer. You also performed expansion of the group, and post-
installation tasks to suppress SSH warning, configure HA/DRS, and add vMotion
interfaces on all nodes in the group.
6.1 Managing Cisco HyperFlex in vSphere Environment

Introduction
Cisco HyperFlex offers a range of tools that you can use to monitor and manage the
overall solution. The management options differ between environments, based on the
type of hypervisor that is used in the Cisco HyperFlex deployment: VMware vSphere or
Microsoft Hyper-V. Cisco HyperFlex is traditionally deployed in a vSphere
environment. Here, you will focus on the vSphere deployment of Cisco HyperFlex.

For a vCenter administrator, the vSphere Cisco HyperFlex deployment does not feel
different from a non-HyperFlex infrastructure. Some specific features are available, but
most of the functionality is still present and the main benefits are running in the
background. Cisco HyperFlex-specific file system-accelerated features add a layer of
performance and security, while providing scalability beyond a regular vSphere
environment.

Cisco HyperFlex also introduces additional management and monitoring interfaces,

which enable automation, remote management, and Cisco HyperFlex Data Platform
(HXDP)-specific functionalities.

6.2 Managing Cisco HyperFlex in vSphere Environment

Management Interfaces Overview

This short video provides an overview of management interfaces. First, Cisco HyperFlex
introduced additional management and monitoring interfaces. These enable automation,
remote management, and some other specific HyperFlex functionalities.
This table lists several interface options that are available, along with the description and their
use cases. For example, Cisco UCS Manager embedded in the fabric interconnects, this is used
to monitor and manage broader networking and hardware functions.

Also VMware vSphere web client can be used for the most management options of the ESXi
host. However, you may want to use the ESXi CLI for troubleshooting or trying to recover
individual hosts from failures.

Finally, Network Time Protocol, or an NTP server, is mandatory when you install and use Cisco
HyperFlex because NTP provides the network synchronizations between the software
components. The HyperFlex installer will check to see if the NTP server IP address that you
provide is reachable. And if it's not, then the installation is just going to fail.

Now, the figure lists some of the requirements that you should verify for your NTP server. The
NTP server must always be operational because, I mentioned before, it's required for certain
 functions, such as preventing the time skew between vCenter and the cluster from rising
above the acceptable level.

By default, active domain authentication requires all clocks to be within five minutes of each
other. You should never use different NTP sources for different parts of the HyperFlex
infrastructure. They should all come from the same source. And that brings us to the end of
this video. Thanks for watching.

Cisco HyperFlex provides a fully contained virtual server platform that combines three
layers: compute, storage, and network. Cisco HyperFlex provides a mix of management
tools, each with a different management and monitoring scope. Some of the same
functionalities are available in different management tools, while others are tool-
specific.

Different management interfaces can also be used to differentiate between users. The
vCenter single sign-on (SSO) integration is available, and the users and permissions
granted in the vCenter propagate to other management platforms as well, which allows
for granular role-based access control (RBAC) between different management options
that are available in various interfaces.

Interface Description and Use Cases

Cisco Intersight Cloud management platform for HyperFlex, UCS, and third-party systems.

VMware vSphere Web VM-based workflows and vSphere features, such as DRS and high availability. Includes
Client the Cisco HXDP plug-in for vCenter.

HXDP plug-in for vCenter plug-in for HyperFlex-based workflows (datastores, nodes, disks, and VMs).
vCenter Installed automatically by the HyperFlex installer.

GUI for HyperFlex-based workflows: datastores, nodes, disks, encryption, native

HX Connect
replication, upgrades, HyperFlex clones, and snapshots.

Command-line management of HyperFlex-based workflows, executed

Storage CLI (hxcli)
with stcli and/or hxcli commands.

REST API HyperFlex-based workflows orchestrated over REST API.

Cisco UCS Manager Management of the HyperFlex hardware infrastructure and networking.

VMware ESXi CLI Command-line management of individual ESXi hosts.

VMware vSphere Web Client and the Cisco HXDP plug-in for vCenter provide a
management bundle that is dedicated to server administrators. The client is the primary
management point for virtual machine (VM) management and VMware features, such
as high availability, machine start order, Distributed Resource Scheduler (DRS), and
machine auto start. These features are very useful in a HyperFlex environment because
the shared storage pool is automatically mounted as one or more datastores on all nodes
in the system, but VMs still run on individual nodes and need a failover and load-
balancing features. vSphere high availability and DRS rely on vMotion and the vMotion
VLAN that is configured during HyperFlex installation to move VMs between nodes
while the VMs file system is kept on the shared datastore.

Cisco HXDP plug-in for vCenter can be installed by users after the HyperFlex
installation. As of Cisco HyperFlex Version 4.5(1a), the HyperFlex installer does not
install the HTML5 version of the Cisco HXDP plug in. The plug-in allows you to
manage HyperFlex elements from the vCenter HTML 5 interface.

Note
When installing Cisco HyperFlex on vCenters that still support Flash, the Flash HXDP
plug-in is automatically installed. The HXDP Flash plug-in is deprecated in favor of the
HTML5 version.

Three management tools cover a similar configuration scope related to native Cisco
HyperFlex features: HyperFlex Connect, Storage CLI, and REST API. HyperFlex
Connect is an HTML5-based web interface. Storage CLI (stcli) lends itself very well to
troubleshooting. REST API offers the optimal solution when you integrate the
HyperFlex system with RESTful orchestration tools.

You can use Cisco UCS Manager that is embedded in the Cisco UCS Fabric
Interconnects to monitor and manage the broader networking and hardware functions.
You can change hardware policies and the server service profiles, upgrade hardware and
firmware, and configure networking. Cisco UCS Manager is the primary management
point for upstream network connectivity.

Cisco Intersight is another interface that can effectively be used for managing and
monitoring of the Cisco UCS infrastructure, including Cisco HyperFlex. Intersight
upgrades the Cisco UCS Manager concept by being able to integrate many Cisco UCS
domains, many devices, and many locations in one interface.

Although the VMware vSphere Web Client covers the most relevant management
options for the ESXi hosts, you may need to use the ESXi CLI when troubleshooting or
recovering individual hosts from failures. SSH access is enabled on all HyperFlex hosts
by default and can be used to access the ESXi CLI.

Time Synchronization
An accessible NTP server is mandatory when installing and using Cisco HyperFlex, and
it provides time synchronization between different software components of Cisco
HyperFlex. The HyperFlex Installer specifically checks the availability of NTP—the
installation fails if it is not reachable.

Time synchronization plays a special role in systems that use SSO infrastructure. Cisco
HyperFlex relies on NTP to provide accurate time information to all system
components. During the HyperFlex installation, you must enter the NTP server IP
address for the wizard to complete.
Note
Synchronized time provides a range of benefits in most IT systems, such as the accuracy
of logging information, operational time-based access control, and identity proof
(certificates).

Time synchronization is critical for multiple services, such as HyperFlex snapshots, so

make sure that these prerequisites are met:

 NTP server configuration allows it to operate with Linux clients.

 NTP service is provided by Active Directory server or another central system.
 NTP service is available throughout the lifetime of the HyperFlex deployment, not
only during installation.
 NTP server is properly configured, accurate, and preferably connected to a lower
Stratum server.

Cisco HyperFlex with VMware vSphere uses SSO authentication against an Active
Directory domain. Active Domain authentication requires all clocks to be within 5
minutes of each other (by default). You must ensure that NTP is always operational, to
prevent time differences between different systems of a HyperFlex installation.

Do not use different NTP sources for different parts of the Cisco HyperFlex
infrastructure, because changes performed on one NTP server might not be changed on
another—even if they are synchronized. The connectivity for time synchronization
could be interrupted, causing inconsistencies in time synchronization.

Note
NTP servers should reside outside of the HyperFlex storage. Nested NTP servers can
cause the HyperFlex system to not start after shut down, such as during offline
maintenance or data center power loss.

6.3 Managing Cisco HyperFlex in vSphere Environment

Cisco HyperFlex Plug-In for vCenter

Now let's examine the Cisco HyperFlex Plug-In for vCenter and Cisco HyperFlex Connect. The
Cisco HyperFlex Plug-In is one of the interfaces that you can use to monitor and manage the
storage cluster. It's automatically installed during the HyperFlex installation and integrated
with the vSphere web client.

The vSphere web client is used for VM-based workflows and provides Role-Based Access
Control, or RBAC, for security. The HyperFlex Plug-In for vCenter is embedded in the vSphere
web client and provides a menu supporting the HyperFlex features.

Most features for the Cisco HyperFlex Plug-In are located, as I said, in vSphere vCenter
inventory lists menu. The various functions are distributed between Getting Started, Summary,
Monitor, and Manage tabs. This figure lists the tabs along with a brief description for their
purpose.
The Summary tab shows an overview of the cluster status. The Monitor tab allows you to
monitor the performance and events. And the Manage tab allows you to manage data stories,
copy, as well to be able to export information.

Not all cluster elements are managed through vSphere vCenter inventory lists menu. For
example, Cisco HyperFlex native actions available for VMs and hosts are found alongside their
vCenter counterparts.

You will find cloning and snapshot functions by right clicking on the item that you want to
manage. And other HyperFlex functions can be found in their own submenu at the bottom of
the item pop-ups menu.
This figure lists some takeaways when using the HyperFlex Cluster vCenter Plug-In. Cisco
HyperFlex Plug-In gives vSphere administrators access to HyperFlex features which are natively
accessible only through the Cisco HyperFlex Native Management interfaces.

However, not all of the features are available directly from vCenter. For example, Cisco UCS
hardware and firmware management, you still have to manage that through UCS Manager.
There are also some browser requirements and limitations that you should be aware of that
are in Firefox and HTML5. So read the read-me documentation.

The Cisco HyperFlex Connect is the main management interface of the Cisco HyperFlex
solution. It supports basic management of the VMs and most functionalities for managing
Cisco HyperFlex. It's a graphical HTMI interface-- HTML5 interface.

It's reachable via HTTPS on the cluster IP address of one of the cluster virtual machines. It
supports native HyperFlex workflows and some limited VM workflows. It also provides web
access to the stcli, but not all of the commands are supported. Some of them, you'll have
to go to the actual CLI.

This figure here lists the functional areas of Cisco HyperFlex Connect. There is a partial overlap
between the features available through HyperFlex Connect and the vSphere web client with
the plug-in.

The choice of the management GUI that you want to use really depends on the feature that
you want to control. HyperFlex Connect offers a broader scope of native HyperFlex functions,
but it does not cover the full VM management like vCenter does.

Finally, to access the HyperFlex Connect, you will need to provide user credentials before you
can begin to manage the installation. Administrator-level access is predefined when HyperFlex
is installed, which includes the local route and admin-user access.
The passwords are assigned during the HyperFlex installation. In HyperFlex 3.5 and later, root
is disabled by default. You can also create users in vCenter using the Role-Based access control,
or RBAC.

Users that you create in vCenter with administrative role will have both read and modify
permissions, whereas users with the read-only role, they just have read permissions, and they
can't make any changes. That brings us to the end of this video. Thanks for watching.

There are two versions of the Cisco HyperFlex vSphere plug-in. The plug-in version is
determined by the type of the vSphere interface you are using. Up until vSphere Version
7.0, most HyperFlex-supported vSphere versions included a Flash web interface, which
has been removed in the latest vSphere versions.

The Flash plug-in is still available if you install Cisco HyperFlex on a vSphere that still
supports Flash user interface. HTML5 version is, however, the recommended version
and replaces the old Flash option.

Note
Flash Player is no longer supported since December 31 2020, after several extensions. It
is recommended that users remove the Flash Player from their machines and use a
newer user interface version, such as HTML5.

The main features of the vSphere Web Client and the Cisco HXDP plug-in are:

 vSphere Web Client:

o Full management for VM-based workflows

o Provides RBAC

 Admin or read-only access to the HXDP plug-in

 Users defined in vCenter.
 HyperFlex plug-in for vCenter:

o Embedded in vSphere Web Client.

o Adds HyperFlex management features to the vSphere interface.
o Used for HyperFlex based workflows, such as:

 Deploy, mount, and delete the HXDP datastore.

The vSphere Web Client, along with the Cisco HyperFlex plug-in, is dedicated to VM
administrators. This single point of management provides access to the relevant
HyperFlex functions together with vSphere-specific features, such as licensing or
RBAC.

RBAC allows you to specify access privileges to different vSphere components and to
the Cisco HXDP plug-in. There are two visibility levels in the HyperFlex plug-in:
administrator and read-only. Typically, these two levels are used to differentiate
between administrators and operators.

Depending on the complexity of your solution, the vSphere Web Client can include
plug-ins for other data center components. For example, if you deploy Cisco HyperFlex
in a Cisco ACI environment, you could also install the Cisco ACI plug-in for vCenter
and use it to manage the application policies on the Cisco Application Policy
Infrastructure Controller (APIC). The vSphere Web Client would then represent a single
point of access for an even broader solution.

Cisco HyperFlex Plug-In: Installation

As of HyperFlex Version 4.5(1a), the HTML5 plug-in is not installed on HyperFlex
installation. To use the HTML5 plug-in, you need to download and install it.

You can download the current version of the HyperFlex vSphere HTML5 plug-in
from https://software.cisco.com/download/home.
Follow these steps to install the HTML5 plug-in on your vCenter:

 After downloading the zip package, upload it to a CVM /tmp directory using Secure
Copy Protocol (SCP).

o You can use Linux command line scp tool or WinSCP on Windows.

 Log in to the CVM, where you uploaded the package and cd in the /tmp directory.
 Unzip the uploaded package and run the install_vc_plugin command.
 Follow the installation prompts to complete the installation.

After the installation, you will find the HyperFlex HTML5 plug-in in your HTML5
vCenter interface under Home > Cisco HyperFlex. It will not be visible in the Flash
interface.

Cisco HyperFlex Plug-In: Managing Storage System

The Cisco HXDP vCenter plug-in allows you to easily manage and monitor Cisco
HyperFlex datastores from the vCenter interface.

Option Description

Summary At-a-glance overview of the system configuration and performance

Performance Monitor storage throughput and latency.

Events Monitor HyperFlex system event messages.

Alarms Monitor HyperFlex system alarms and errors.

Datastore Create, rename, mount/unmount, delete datastores.

The main Cisco HyperFlex functionality when managing VMs on HyperFlex storage is
creating the datastore. After the datastore is created, the VM operations are the same as
in any other vSphere environment except for HyperFlex-native storage functions.

Cisco HyperFlex Plug-In: Considerations

The Cisco HXDP plug-in gives vSphere administrators access to the features of Cisco
HXDP, which are natively accessible only through Cisco HyperFlex native management
interfaces, such as HyperFlex Connect and the stcli command, but not all the features
are available directly from vCenter.

Aside from not being able to manage hardware specifics and networking, which is done
through Cisco UCS Manager on most HyperFlex deployments, several HyperFlex-
native functions are not accessible from the vCenter.

Consider these takeaways when using the HyperFlex vCenter HTML5 plug-in as of
HXDP 4.5(1a):

 Cisco UCS hardware and firmware management, and a part of networking, is still
managed in Cisco UCS Manager.
 Native replication, snapshotting workflows, CLI access, HyperFlex updates, and
SED management are still exclusive to HyperFlex Connect and the stcli command
and API.
 Can be configured with read-only permissions.
 Not supported in vSphere desktop client, deprecated version available in vSphere
Flash client.
6.4 Managing Cisco HyperFlex in vSphere Environment

Cisco HyperFlex Connect

Cisco HyperFlex Connect, most often referred to as HX Connect, is the main
management interface of Cisco HyperFlex. It supports basic management of VMs and
most functionalities for managing Cisco HXDP. It is an HTML5 graphical interface,
reachable by HTTPS on the group management IP address of the HyperFlex system or
any of the CVMs.

HyperFlex Connect is accessible through the in-band management VLAN of the

HyperFlex topology and resides on the CVMs.

Cisco HyperFlex Connect provides:

 HTML5-based access to various HyperFlex storage operations.

 Native HyperFlex workflows: Replication, encryption, datastore, monitoring,
upgrading.
 Limited VM workflows: Cloning, snapshotting, power on/off/suspend, protect
VM.
 Web access to stcli command-line management with limited functionality.

There is a partial overlap between the HyperFlex features available through HyperFlex
Connect and the vSphere Web Client with the HyperFlex plug-in. The choice of the
management GUI depends on the feature that an administrator wants to control and their
personal preferences.

HyperFlex Connect offers a broader scope of native HyperFlex functions, including

encryption and replication, but it does not cover the full VM management capabilities of
vCenter.

Functional Areas of Cisco HyperFlex Connect

Key monitoring pages that include information about the local HyperFlex storage are
listed and described in the table:
Page Description

Dashboard Overall HyperFlex storage system status

Alarms/Events/Activity Used for troubleshooting

Performance Charts for IOPS, throughput, and latency

Replication Native HyperFlex protection and disaster recovery

System Information System overview, HyperFlex maintenance

 Page Description

Datastores Workflows for HyperFlex datastores

Configure the iSCSI network and LUNs including

iSCSI/Kubertnetes
CSI

Virtual Machines Limited VM management functions

Upgrade HXDP, ESXi, and Cisco UCS firmware

Web CLI Access to the CVM CLI (not all functions)

The information that HyperFlex Connect provides in the Alarms, Events, and Activity
areas is collected from multiple sources—the vCenter, and the controller VMs. vCenter
provides details about vSphere-specific features, while controller VMs provide
information about HyperFlex-specific events. The events are synchronized both ways to
be as consistent as possible.

The CLI that is available through HyperFlex Connect only supports stcli commands
and does not allow Linux-native command execution. The HyperFlex Connect CLI also
does not support advanced Bash CLI features.

Users in Cisco HyperFlex Connect

During the Cisco HyperFlex installation, you provide the HyperFlex group IP address in
the in-band management VLAN, which is the IP address where you access the
HyperFlex Connect over HTTPS. If you are using the recommended option and use
DNS entries for the infrastructure, you can also provide the group fully qualified
domain name (FQDN) instead of the group IP.

To access HyperFlex Connect, you must provide user credentials before you can begin
to manage the HyperFlex installation. The administrator-level access is predefined when
HyperFlex is installed, and the password is provided in the HyperFlex Installer.

Consider the following rules when managing your HyperFlex through Cisco HyperFlex
Connect:

Cisco HyperFlex Connect differentiates between two types of users:

 Predefined: local/root and local/admin

 RBAC: Users defined through vCenter in the format name@domain.local

o A vc- prefix required for API login as a vCenter-created user

o Full name and a domain required as a username to log in to HX Connect
 User User Type Description

Default local authentication account. Since

3.5(1a), the root user is disabled by default
root and admin Pre-defined
for HX Connect login. Since 4.5(1a) root CLI
access is only available to TAC.

vCenter user with RBAC (defined in Read and modify permissions. Can modify
Administration role vCenter) the HX Storage system.

vCenter user with Read- RBAC (defined in Read (view) permissions. Cannot make
only role vCenter) changes to the HX Storage system.

vCenter recognizes the session with HyperFlex Connect; therefore, system messages
that originate with vCenter might indicate the session user instead of local/root.

For a successful login, it is vital that the time is synchronized in vCenter and HyperFlex.
Otherwise, you will not be able to log in. Local accounts (admin and root) will still
work.

6.5 Managing Cisco HyperFlex in vSphere Environment

Storage CLI
Now let's take a look at an overview of the storage CLI and REST API. The storage CLI, or stcli, is
accessed by connecting to the Cisco HyperFlex cluster address using SSH. This will connect you
to one of the available controller VMs and presents a bash command line.

The stcli command is the most detailed and advanced tool to manage the Cisco HyperFlex data
platform. This figure here lists the functionalities that are not available in the HyperFlex
Connect GUI or vCenter. The stcli also allows you to differentiate between administrators with
full-access privileges and operators that have read-only permissions.

This figure lists the main takeaways about common storage commands. For example,
administrator-level permissions are required for any modification commands. Both
administrators-- or read-only permissions allow users to issue read commands. And adding
the -h at the end of the command will provide you syntax for some help information.

The table in this figure here provides several examples of stcli commands. These commands
are related to security settings, cluster maintenance, native replication, and services control.
You'll notice that most of the advanced management feature commands will begin with the
stcli cluster.

By the way, never include passwords in command strings. This is because they can be passed
to the log files as plain text. Always wait till the command prompts or before you enter an
actual password.
The REST API is a standardized architecture for API integration. And it's supported by Cisco
HyperFlex. The REST-style architecture consists of clients and servers. The clients initiate
secure, standardized HTTP-based requests to the servers. The servers then process the
requests and return appropriate responses via HTTP, JSON, or XML formatted messages. The
clients can generate different types of requests, which include GET, POST, PUT, and DELETE.

Finally, the REST API Explorer provides an interface for testing and exploring the functionalities
of the Cisco HyperFlex API. This figure here shows you how you can access the API Explorer
and some of its features. The REST API displays a database of operations available to the API
on HyperFlex for issuing a command through the API. It also shows some of the variables that
can be used. The REST API also allows for easy testing of API functionalities with third-party
tools. That brings us to the end of this video. Thanks for watching.

The hxcli is accessed by connecting to the Cisco HyperFlex group address using Secure
Shell (SSH), which connects you to one of the available CVMs and presents a limited
Bash command line.

The hxcli command is the most detailed and advanced tool to manage the Cisco HXDP.
Extensive information about the system can be gained by using the hxcli command,
including advanced snapshot features, cleaner process management, information about
system status, and many others.

The Bash command line also enables access to system logs, which can assist you in
troubleshooting. For example, you can open the logs, manipulate outputs using
the grep command, or verify connectivity between interfaces using the ping command.

Storage CLI provides these functionalities (not available in the HyperFlex Connect GUI
or vCenter):

 Control system state (start or shutdown)

 Advanced snapshot features such as quiescing
 Cleaner process scheduling and rebalancing
 Detail information about the state of the system
 Access to the logs

hxcli [-h] {drnetwork,policy,about,services,vm,dp,snapshot-

schedule,cluster,appliance,node,disk,cleaner,datastore,file
,security,iscsi-whitelist,license,rebalance}

To avoid connecting to a CVM command line through SSH, HyperFlex Connect also
provides access to most of the stcli commands through the web CLI tab.

Common Storage CLI Commands

The hxcli command provides embedded information about different options by adding -
h at the end of the command. As you add additional keywords to a command, each
additional keyword will be summarily explained by adding -h at the end of the
command.

The hxcli command lends itself not only to troubleshooting but to various situations that
occur in everyday operations. You can choose from a range of commands, according to
 your use case. All hxcli commands are either read commands that provide information
or modify commands that perform changes on the system.

Main takeaways about common storage CLI commands:

 Modify commands require administrator level permissions.

 Read commands are permitted with administrator or read-only level privileges.
 Operator -h can be used to provide additional information on a command.

Command Explanation

Used to change a controller

VM password and configure
hxcli security [-h] {password | whitelist | ssh |
access control allow lists
encryption}
(using
the whitelist command)

Data protection commands for

hxcli dp [-h] {vm | group | peer | schedule | vm
replication and disaster
hxtask}
recovery

hxcli services [-h] [smtp | dns | ntp | asxup | sch | System services-related
remotesupport | timezone] operations

hxcli node maintenanceMode {id ID | ip IP address} -- Enter the node into HyperFlex
mode enter maintenance mode.

The table provides several examples of stcli use cases that

relate to security settings, HyperFlex maintenance, native
replication, and services control.

You can control the system security (stcli security command)

by setting the user password for all controller VMs in the
storage system (password), performing the firewall allow-listing
operations (with the whitelist command), resyncing SSH keys
in the storage system, and managing encryption. Firewall allow
lists contain the IP addresses that are permitted to manage the
system.

To change the CVM password, use the stcli security password

set command.

Note
Standard Linux commands are restricted since HXDP Version
4.5(1a). The admin user still has the ability to read logs and
perform some OS-level actions, such as run editor, but the array
 of available commands is much more limited than in previous
versions.

Never include passwords in command strings. Commands, such as login commands

and stcli commands, are frequently passed to the logs as plaintext. Wait until the
command prompts you for the password.

Another use case involves the control of management and network services, such as
SMTP, DNS, NTP, Cisco Automatic Support, and Cisco Smart Call Home.

Cisco Automatic Support enables you to proactively obtain information about failures
and responds immediately. It also helps in planning system performance and capacity.
Cisco Smart Call Home provides continuous monitoring, proactive diagnostics, alerts,
service ticket notifications, and remediation recommendations about the Cisco
HyperFlex storage system to the designated Cisco Automatic Support customer
contacts. It can provide the information through HTTPS and a proxy server, if needed.
You can enable or disable the services, and configure IP addresses of the servers
(SMTP, DNS, NTP) or recipients (Cisco Automatic Support).

Common Storage CLI Commands: System Management

The stcli cluster command offers a range of features that are helpful when performing
system-related operations. You can verify and control the system operational status
(shutdown or start) and maximize insight into the operation of the HyperFlex system.

Common commands for system management are shown in the table. Most advanced
management features are invoked using the stcli cluster keyword.

Command Comment

Shut down the storage system.

hxcli cluster shutdown hxcli cluster start hxcli
Restart the storage system. Check the
cluster info
state.

stcli cluster get-cluster-access-policy

Verify and change the system access
stcli cluster set-cluster-access-policy name
policy level.
{strict | lenient}

Manually initiate a storage system

stcli rebalance start [--force] rebalance. This command should be
stcli rebalance status run after adding or removing a node
to/from the system.

Storage system cleaner operations for

stcli cleaner [-h] {info | start | stop | stats |
removing stale data and releasing
report | get-schedule | set-schedule}
storage

The storage system is rebalanced on a regular schedule—as of HXDP 4.5(1a), the

default setting is 00:15 a.m. (server time). The rebalancing process distributes
information between nodes so that it is evenly distributed across the storage. If you add
or remove a node in the storage system, you can manually initiate a storage system
rebalance using the stcli rebalance command.

Because of the way the Cisco HyperFlex filesystem works, data is not deleted
immediately when it is no longer needed. Actual data reclamation is performed by a
background cleaner process that you can control through the stcli cleaner command.

6.6 Managing Cisco HyperFlex in vSphere Environment

REST API Overview

If you have your own toolset that is used for automation, orchestration, and
management of infrastructure, you can integrate Cisco HyperFlex through an API. This
external software is used to issue commands to the system and use similar
functionalities to the natively installed management platforms of Cisco HyperFlex.

Cisco HyperFlex supports Representational State Transfer (REST) architectural API

style, which abstracts the architectural elements within a distributed hypermedia system.
REST ignores the details of component implementation and protocol syntax to focus on
the roles of components, constraints upon their interaction with other components, and
their interpretation of significant data elements. REST has emerged as a predominant
web API design model.

The REST API is a standardized architecture for API integration:

 The client sends secure standardized HTTP-based requests that the server interprets.
 The server processes the requests and responds through HTTP in JSON or XML
format.
 The client can use several methods:

o GET: Get a list of tasks or a task by ID.

o POST: Create a new task.
o PUT: Add to an existing task by ID.
o DELETE: Delete an existing task by ID.
REST-style architectures conventionally consist of clients and servers. Clients initiate
requests to servers. Servers process requests and return appropriate responses. Requests
and responses are built around the transfer of representations of resources. A resource
can essentially be any coherent and meaningful concept that may be addressed. A
representation of a resource is typically a document that captures the current or intended
state of a resource.

The client begins by sending requests when the client is ready to make the transition to a
new state. While one or more requests are outstanding, the client is considered to be in
transition. The representation of each application state contains links that may be used
the next time that the client chooses to initiate a new state transition.

REST is intended to evoke an image of how a well-designed web application behaves.

Presented with a network of web pages (a virtual-state machine), the user progresses
through an application by choosing links (state transitions), resulting in the next page
(representing the next state of the application) that is being transferred to the user and
rendered for their use.

Cisco HyperFlex System RESTful APIs with HTTP verbs integrate with other third-
party management and monitoring tools that can be configured to make HTTP calls. It
enables authentication, replication, encryption, monitoring, and management of a
HyperFlex system through an on-demand stateless protocol. APIs allow external
applications to interface directly with the HyperFlex management plane.

The supported functionalities and programming schemas can be explored through a web
application that is available on the group IP address: the API Explorer.

Note
DISA STIG Compliance: HXDP Version 4.0(1a) added new HyperFlex REST APIs
for setting, removing, and checking the status of Defense Information Systems Agency
Security Technical Implementation Guides (DISA STIGs) for CVMs, ESXi hosts, and
vCenter. These APIs enable customers to meet DISA security requirements by centrally
and securely applying STIGs, detecting and correcting for drifts in any STIG settings.
Cisco HyperFlex REST API Explorer

The REST API Explorer provides an interface for testing and exploring the
functionalities of the Cisco HyperFlex API. Rest API Explorer shows a database of
operations available to the API on HyperFlex, and the variables that can be used when
issuing a command through the API. It also allows for easy testing of API
functionalities with third-party tools, such as Google Postman.

Note
Cisco DEVNET is a great resource to learn from HyperFlex REST API
examples: https://developer.cisco.com/docs/ucs-dev-center-hyperflex/#!rest-api-
examples/hyperflex-api-examples

You can access API Explorer on https://Group_IP/apiexplorer/ and use these features:

 Authentication, authorization, and accounting (AAA) and access token generation

 Lists API calls, their schemas, and the value types
 Differentiates between calls specific to Hyper-V and vSphere.
 Tests responses by directly providing values and sending the request.
The HyperFlex REST API Explorer covers a wide range of functionalities compared to
either the HyperFlex Connect and the stcli. Most of the user commands are passed
through the API either way.

After testing, the schemas can be implemented into other systems and sent through other
applications to the API over HTTPS. This way, you can send commands to the
HyperFlex system without directly accessing any of its management interfaces.

6.7 Managing Cisco HyperFlex in vSphere Environment

iSCSI and Cisco HyperFlex LUN

Management
Internet Small Computer Systems Interface (iSCSI) is an IP-based storage protocol that
allows block storage communication over Ethernet networks. The evolution of storage
(SAN) networks has moved away from discrete hardware networks, such as the native
Fibre Channel.

In Cisco HyperFlex, there is no native Fibre Channel network. This is one of the major
advantages of the Hyperconverged design; however, exporting block-level storage is
still a useful feature, especially when combined with non-persistent systems, such as
containers.

Since only Ethernet networking is used in HyperFlex, the simplest choice for providing
block storage LUNs to external systems on HyperFlex is iSCSI. iSCSI support has been
introduced with HyperFlex Version 4.5(1a).

iSCSI on HyperFlex provides storage-defined drives with these capabilities:

 Block-level access using the fast HyperFlex network that is part of the solution.
 Automated network and LUN configuration through a GUI.
 Additional capabilities for Kubernetes integration.
 Creation of storage-optimized clones of the entire logical storage drive.
 Full-API integration of iSCSI actions in HyperFlex REST API.

The iSCSI network needs to be configured for HyperFlex iSCSI:

 iSCSI network is configured through HX Connect.

 iSCSI network uses a separate VLAN.
 The VLAN contains an extra set of IP-addressed interfaces and VIP.
 Cisco UCS Manager VLAN creation can be done automatically or manually.
 You can have initiators in the same VLAN or behind a router.
 Initiators can be in the same Cisco UCS domain as HyperFlex.
The iSCSI network configuration is done through GUI and also allows you to
automatically reconfigure the vNIC policy template of the HyperFlex service profiles,
which allows the iSCSI VLAN communication between the Fabric Interconnects and
the servers.

To extend the iSCSI communication the HyperFlex target and the initiators need to be
reachable over IP. To allow this communication, you will need to enable the iSCSI
VLAN on the upstream switches.
Note
As of Version 4.5(1a), the HyperFlex iSCSI network is configured in the same
active/passive configuration as the HyperFlex data network. To protect against certain
failure scenarios, you should always enable the iSCSI VLAN on upstream switches
even when your initiators are in the same Cisco UCS domain for the same reasons you
need to enable the data VLAN for HyperFlex.

iSCSI mostly uses the SAN terminology for its systems:

 Fabric: Is a physical data path in a SAN topology. Usually there are two redundant
fabrics in a typical SAN topology.
 Target: Storage where the LUNs exist to be mounted by initiators. Typically the
term target refers to the target storage address.
 Initiator: Client device that mounts the remote storage. Typically the term initiator
refers to the initiator storage address.
 IQN: Stands for iSCSI Qualified Name. This is the iSCSI addressing used in
protocol communication. IQNs are assigned to both initiators and targets.
 LUN: Stand for Logical Unit Number. This is a software defined storage unit that
can be mounted over a SAN network.
 CHAP: Challenge-Handshake Authentication Protocol. A secure communication
authentication protocol.

Understanding the terminology will help you configure your iSCSI configuration. The
terminology used in the HyperFlex iSCSI configuration is the standard naming used in
iSCSI protocol and applies to all iSCSI protocol communication.

After the network is configured, you can create the target group and LUNs that part of
that group:

 A target group defines the target IQN that is listed under that group. The IQN is
used by the initiator to mount the LUNs in the target group.
 CHAP secret is defined on the initiator group level and can be enabled after the
group is already created.
 LUNs are created as part of an initiator group and are child objects of their target
group.

Allowed initiators need to be defined to allow access to targets:

 Initiator IQN is defined on the system that is mounting the storage (the client).
 You create initiator groups and link targets to those groups to give the initiators in
the group access to linked targets.
 After you have defined the Target IQN, LUN and allowed initiators, you can mount
the LUN on Linux and Windows systems.

As of Version 4.5(1a), only Linux and Windows initiators are supported with
HyperFlex LUNs. Support for additional systems, such as ESXi will be introduced with
later HyperFlex releases.

Since iSCSI provides block-storage to the initiator, you will not be able to start writing
to the storage immediately, you will first need to format the storage and then start using
it.

The logically defined HyperFlex LUNs also allow duplication of block-level storage.
Since the LUN data exists on the StorFS system, HyperFlex can also create duplicates
on storage level.

LUN cloning creates a crash-consistent duplicate of an existing LUN:

 The new LUN is created in the same IQN group as a separate LUN.
 You can create an application-consistent clone by adding VSS system information.
 The clones are performed on the storage level creating a duplicate software defined
storage.
 Can quickly clone many VMs including other information on the LUN.
Volume Snapshot Service (VSS) or Shadow Copy service is a Microsoft Windows
subsystem that creates a call to the cloned systems to voluntarily freeze the application
I/O for up to 60 seconds. This enables the applications to finish the I/O actions and be
storage-consistent.

6.8 Managing Cisco HyperFlex in vSphere Environment

Cisco HyperFlex Container Storage

Interface
Cisco HyperFlex offers a Container Storage Interface (CSI) for integration with the
Kubernetes container platform. CSI enables iSCSI-based LUNs to be mounted in
container environments using the standard Kubernetes tools.

The HyperFlex storage volumes are consumed through standard Kubernetes primitives
such as Persistent Volume Claims and Storage Classes.
 CSI is a two-component system that allows easy provisioning of storage to Kubernetes
pods:

 CSI uses the iSCSI protocol for communication between the container and
HyperFlex storage and the iSCSI network needs to be configured for CSI.
 CSI runs four pods in the Kubernetes deployment for HyperFlex integration.
Attacher, Provisioner, and Resizer run in a single instance systemwide, while the
Node plug-in runs per-worker node.

The HyperFlex plug-in is the communication hub for all the services and provides the
API communication channel. The central Kubernetes API server initiates calls to
different CSI components that then execute supported actions on the HyperFlex system.

CSI Components and Their Functions

Component Function

Attacher Pod (csi-attacher-hxcsi) Required by CSI, but not currently used in Cisco deployment.

Provisioner Pod (csi-provisioner- Watches Kubernetes Persistent Volume Claim objects and triggers CreateVolume
hxcsi) and DeleteVolume operations.

Watches Kubernetes Persistent Volume Claim objects and triggers

Resizer Pod (csi-resizer-hxcsi)
ControllerExpandVolume and NodeExpandVolume operations.

HyperFlex Plugin Container (csi- Discovery and formatting of provisioned HyperFlex iSCSI LUNs on Kubernetes
nodeplugin-hxcsi) worker nodes. Implements NodePublish/NodeUnpublish Volume APIs.

The HyperFlex CSI support is designed according to the Kubernetes CSI integration
specifications, which allows full integration of HyperFlex storage into the Kubernetes
container ecosystem.

As of HXDP Version 4.5(1a) CSI supports the following features:

 Dynamic creation/deletion, attach/detach actions.

 Volume cloning on the same datastore.
 Block-level storage access.
 Kubernetes 1.18 support and CSI 1.2 Spec APIs.
 Multi-writer support (ReadWriteMany).
 CSI Plug-in installation and upgrade through Helm chart.
 Volume statistics reporting per CSI specifications.
 Volume resize with ext3 and ext4 volumes.

Note
You can find more advanced information on HyperFlex CSI functionalities and
integration and a detailed installation guide on Cisco HyperFlex Systems
Administration Guide for Kubernetes, Release 4.5.

Configuring Kubernetes for CSI Integration

On the Cisco HyperFlex system, the iSCSI network needs to be enabled before the CSI
integration will be enabled. To utilize the CSI integration between Kubernetes and
HyperFlex, the Kubernetes components need to be installed.
The Kubernetes storage plug-in is available from the HyperFlex download site. There
are CSI plug-ins listed for older versions, but CSI storage is only supported on Version
4.5(1a) or higher.

To use HyperFlex CSI, you need to meet these prerequisites:

 Cisco HyperFlex deployment is running 4.5(1a) or later.

 Configured an iSCSI network in HX Connect and is reachable from Kubernetes
nodes.
 Verify that all Kubernetes nodes have 2.0.874-5ubuntu2.10 version of the open-iscsi
package.
 That Kubernetes configuration runs the iscsid service and has the primary node
properly configured.

After the prerequisites are met and you have downloaded the CSI package
from https://software.cisco.com/download/home, you need to install the package on
your Kubernetes system.

These are the high-level steps for the CSI plug-in installation:

1. Copy the bundle to the administrator host using scp or WinScp.

2. Extract the bundle on the administrator host using tar -xvzf command.
3. Run the hxcsi-setup command in the setup directory with the appropriate
parameters.
4. A new hxcsi-deploy directory is created after successful setup.
5. Execute kubectl create -f ./hxcsi-deploy/ to start deployment.
6. Confirm deployment with kubectl get pods command.
7. Create a Kubernetes storage class to allow developers to consume the storage.
8. Confirm storage class creation using kubectl get sc command.

Note
For detailed procedure and output examples, go to Cisco HyperFlex Systems
Administration Guide for Kubernetes, Release 4.5.

After the CSI plug-in installation and definition of the storage class, you can call the
storage actions from their container environments through the Kubernetes storage class.

6.9 Managing Cisco HyperFlex in vSphere Environment

ReadyClones
This video provides a brief overview of ReadyClones. Cloning virtual machines is probably one
of the most common tasks in virtual environments. This is one of its major advantages when
comparing virtualized servers to the regular bare-metal servers.

However, Cisco HyperFlex ReadyClones improves on the hypervisor version of native cloning.
ReadyClones upgrade the regular VM duplication functionality by using native storage
functions, which greatly increase its efficiency.

The resulting virtual machine is functionally the same as the one created with the regular
cloning process. But the underlying technology is very quite different.

Clones are created by reusing the parent VM blocks on storage, but they're completely
independent. The clones are not completely identical to the host VM because, obviously, the
cloning operations are going to modify the UUID, and MAC addresses, and so on, but retains all
of the other system data.

Lastly, ReadyClones are extremely fast and more efficient than the legacy cloning operations
because they support VAAI data offloads. Finally, this figure here lists the requirements for
normal ReadyClone operations.
Remember, ReadyClones effectively replaced the hypervisor's normal cloning functionality.
However, this feature still relies on the HyperFlex data store to function. The virtual machines
must reside on a HyperFlex data store and can only have HyperFlex native snapshots. And that
brings us to this short video. Thanks for watching.

Cloning VMs is a common task in a virtual environment and one of the major features
of the virtualized servers when compared to bare-metal deployments. With the cloning
process, you create a near duplicate of the VM, with only minor changes to the
hypervisor-based configuration, such as MAC addresses, allowing you to create any
number of identical VMs on demand.

In Cisco HyperFlex, ReadyClones upgrade the regular VM duplication functionality by

utilizing native storage functionalities, which greatly increases the efficiency of the
cloning process. The resulting VM is functionally indistinguishable from the one
created in a regular cloning process, but the underlying technology is very different.

ReadyClones improve hypervisor-based cloning:

 The cloning process is more efficient at the storage level utilizing VMware VAAI.
 Clones maintain hypervisor integration and have a universally unique identifier
(UUID) and MAC.
 Clones are created by re-using the parent VM blocks on storage.
 Cloning can be initiated from HX Connect, API, and CLI management interfaces.
The clones do not share configuration parameters with the source VM. If they retained
the same UUID and MAC address, you would experience conflicts in the VM
identification and addressing. Both the UUID and the MAC address are meant to be
unique. UUIDs are commonly used as unique keys in databases and other structures.
The cloning operation modifies the UUID and MAC address and retains all other
system data. You can modify a clone at any time, and the changes will not affect the
host VM.

Cisco HXDP ReadyClones are extremely fast and more efficient than legacy cloning
operations because they support VMware vStorage API for Array Integration (VAAI)
data offloads. VAAI, also called hardware acceleration, or hardware offload APIs, is a
set of APIs that enable communication between VMware vSphere ESXi hosts and
storage devices. Use Cisco HXDP ReadyClones to clone VMs in seconds instead of
minutes.

ReadyClone Requirements
Even though ReadyClones effectively replace the hypervisor's normal cloning function,
this feature relies on the HyperFlex datastore to function.

Consider these requirements for normal ReadyClones operation:

 VMs must reside on an HyperFlex datastore.

 Cannot mix HyperFlex native and VMware snapshots
 Only a single vNIC customization template is allowed.
 No support for Win2008 or Win2012 VMs that are powered on
If you need to perform multiple cloning operations, you should start a single batch on a
given VM at a time. Do not create multiple batches of clones simultaneously on the
same VM because it will cause failures or display incorrect status information.

Customization templates in the ReadyClone creation interface in vSphere provide a

powerful tool that allows you to define more complex patterns for the naming and
networking of the cloned VMs. Without customization, the VM names will have the
same prefix and consecutive numbers as the suffix. When using the customization
specification, make sure that the properties will work for the entire batch. If needed,
obtain user-defined parameters from the cloning workflow.

Note
Maximum of 2048 ReadyClones can be created from one base VM. Maximum of 256
ReadyClones can be created in one batch at a time.

6.10 Managing Cisco HyperFlex in vSphere Environment

Cisco HyperFlex Snapshots

This video looks at Cisco HyperFlex snapshots. The first figure here lists the advantages
that HyperFlex native snapshots have over vSphere snapshots.
The HyperFlex native snapshots are high-performance snapshots. They are taken by the
HyperFlex distributed file system rather than using the VMware redo log-based
snapshots. They are very space efficient because they're pointer-based snapshots.

None of the delta data is actually duplicated. Its only referenced. So you don't have to
worry about the physical storage capacity being burned up by snapshots.

HyperFlex snapshots can be created through the vCenter plug-in, the stcli, and the API.
Although the underlying technology is quite different, HyperFlex native snapshots share
some functionalities with vSphere snapshots.

First, HyperFlex snapshots can be created through vCenter. The interface for scheduling
snapshots is very similar. A single interface is used for management of both types of
snapshots. Quiescing the running VMs is performed through the stcli. And lastly, the
vSphere native functionalities for a VM snapshot are available while on HyperFlex.
Finally, this table here in the figure lists specific restrictions for HyperFlex snapshots. In
order for the HyperFlex native snapshots to function on a file system, the first snapshot
of the VM must be a sentinel snapshot.

The sentinel snapshot ensures that the follow-on snapshots are all native snapshots. You
cannot create a sentinel snapshot from a suspended VM. You also cannot schedule
snapshots for storage controller VMs. The VM must reside on a HyperFlex data store in
a HyperFlex storage cluster. And it cannot be in the transient state.

Lastly, if VMs have native snapshots, the storage VM is not supported. You have to
delete the snapshot first. And that brings us to the end of the video. Thanks for watching
.

VM snapshots allow you to create images of the VM state, which are maintained
through time. You can then revert to a certain point in time when a snapshot was
created.

This functionality is useful when you want to revert to the same state but do not want to
create multiple virtual machines. Examples include patching or upgrading the guest
operating system in a VM. Snapshots give you the ability to back out of the patch or
upgrade process if problems occur during patching or upgrading.

Native vSphere snapshots create a delta image file, which is the only instance being
written to and contains all the changes from the previous snapshot. If you create another
snapshot, the new delta file references previous files, which decrease performance.
When a VM has many snapshots, storage usage can also be suboptimal, since each
snapshot can exceed the size of its parent. It is, therefore, wise to keep the number of
vSphere snapshots to a minimum by manually consolidating them, which merges the
latest snapshot with its parent.

Similar to ReadyClones, Cisco HyperFlex native snapshots perform snapshots on the

level of the file system, eliminating most of the issues of snapshots, such as increasing
size over time and the need for consolidation.

HyperFlex native snapshots have advanced features that allow administrators to

effectively and easily manage VMs over time:
 Created snapshots are storage-optimized so they do not degrade over time.
 Delta .vmx files are still created like in vSphere but are written through pointers.
 Because there is no duplicate delta data, the actual storage that is consumed is
minimal.
 Snapshots can be created through HX Connect, stcli, and API.
 Snapshot operations are faster and snapshots can be scheduled.

Cisco HXDP Native Snapshots is a backup feature that saves versions (states) of
working VMs. VMs can be reverted to native snapshots.

A native snapshot is a reproduction of a VM that includes the state of the data on all
VM disks and the VM power state (on, off, or suspended) at the time the native
snapshot is taken. Take a native snapshot to save the current state of the VM, so that
you can revert to the saved state. You can take a native snapshot when a VM is powered
on, powered off, or suspended.

Data can be moved offline or restored from snapshots instantaneously. The main
administrative interface for native snapshots is HyperFlex Connect, but the created
snapshots are visible in the vSphere snapshot interface along with vSphere snapshots.
 Cisco HyperFlex Snapshots: Restrictions
Cisco HyperFlex native snapshots introduce specific restrictions that are related to
snapshot management and creation. Administrators should be aware of these restrictions
to use the snapshots effectively over time.

For the HyperFlex native snapshots to function on the filesystem level, the first
snapshot of a VM must be a sentinel snapshot. If you create a vSphere native snapshot
as the first snapshot, then all subsequent snapshots will not be HyperFlex native. You
can revert to HyperFlex native operation by removing existing vSphere snapshots and
creating a HyperFlex snapshot as the initial snapshot.

When you create the first snapshot of a VM, the Cisco HXDP creates a base snapshot
that is called a sentinel snapshot. The sentinel snapshot ensures that follow-on snapshots
are all native snapshots.

Sentinel snapshots prevent reverted VMs from having VMware redo log-based virtual
disks. Redo log-based virtual disks occur when an original snapshot is deleted and the
VM is reverted to the second oldest snapshot.

Capability or Requirement Description

Must be present as the first snapshot for the HyperFlex

Sentinel snapshot
native snapshot function to work.

Cannot create sentinel snapshot from a Creating the first native snapshot (the sentinel snapshot)
suspended VM from VMs in a suspended state is not supported.

Cannot snapshot SCVM You cannot schedule snapshots for storage controller VMs.

To make a snapshot, the VM must reside on an HXDP

VM must be on HyperFlex datastore.
datastore in an HXDP storage.

The datastores must be accessible. The VMs must be valid

VM must be valid and stable.
and not in a transient state, such as vMotioning.

Storage vMotion is not supported on VMs with If the VM needs to be moved to a different datastore,
native snapshots delete the snapshots before running storage vMotion.

Because HyperFlex snapshots are performed at the file system level, they cannot be
migrated to another file system; therefore, you cannot perform Storage vMotion on
VMs with HyperFlex native snapshots.

Currently, VMware has a limitation of 31 snapshots per VM. This maximum total
includes VMware created snapshots, Cisco HXDP sentinel snapshots, and Cisco HXDP
native snapshots. The sentinel snapshot consumes one snapshot of the total 31 that are
available per VMware limitation.
The lifecycle of native snapshots, similar to VM snapshots, is tied to the virtual
machine. If the VM is deleted, whether accidentally or intentionally, then all the
associated snapshots are also deleted. Snapshots do not provide a mechanism to recover
from a deleted VM. Use a backup solution to protect against VM deletion.

Do not delete the sentinel snapshot. Do not revert your VM to the sentinel snapshot.

Do not use the special characters, dot (.), dollar sign ($), or accent grave (`) in any
guest/user VM names for which you want to enable snapshots.

6.11 Managing Cisco HyperFlex in vSphere Environment

Manage Cisco HyperFlex

Manage Cisco HyperFlex
Your installed Cisco HyperFlex group offers many management options—from the
user-friendly GUI interfaces of HyperFlex Connect and HyperFlex vCenter plug-in, to
the powerful automation-oriented HyperFlex API.

In this lab exercise, you will explore and use different HyperFlex interfaces. You will
see how easy it is to perform everyday jobs on a Cisco HyperFlex group.

Note
Click the Initialize button below to start the lab. Initialization takes 15–60 minutes. A
progress bar will show you when the lab is initialized and active. Please be patient with
the initialization process; several components are loading and getting ready!

Review the Status of Cisco HyperFlex Groups in vCenter

After Cisco HyperFlex is installed and the post installation tasks have been performed,
several interfaces are available to manage the installation. While each group has its own
HyperFlex Connect instance, one vCenter can manage several HyperFlex groups and is
the main interface for virtual machine management.

HyperFlex is deeply integrated into vCenter, offering data platform management and
monitoring functionalities directly in the vCenter interface.

Step 1
Connect to vCenter.
Answer
Open the browser and connect to the pod vCenter at https://vcenter.hx.pod/. Choose
the LAUNCH VSPHERE CLIENT (HTML5) option, and log in using the
credentials administrator@hx.pod / 1234QWer*.
Note
You can always find all lab information and credentials in Job Aids.

Step 2
Examine the HyperFlex groups.

Answer
Expand the HyperFlex Datacenter to reveal the available groups.

If not already expanded, expand the individual group to see the hosts and VMs that are
running on it. Note the CVMs that are running on individual hosts.

Step 3
Review host datastores.

Answer
Click the host esx-a-1.hx.pod in the Navigator pane and go to the Datastores tab in the
top right.
You will see two datastores: VMFS 5-type datastore is the host local drive that is not
part of the HyperFlex storage, and the NFS3 type drive that is a HyperFlex drive that
has already been created for Cluster A.

Now click the host esx-b-1.hx.pod in the Navigator pane and review the Datastores
tab of this host.

Note
The HyperFlex NFS3 share is not present on this host. This group has just been installed
and it needs to be configured for the HyperFlex storage to be available.

Manage Cisco Hyperflex Group Using HyperFlex Connect

Cisco HyperFlex Connect is the primary tool for managing the storage part of Cisco
HyperFlex. Cisco HyperFlex Connect also incorporates some of the VM management
functionalities of the vCenter.
Cisco HyperFlex Connect synchronizes vSphere events to make them visible in its
interface and provides advanced management and monitoring of the Cisco HyperFlex
system. It also allows you to synchronize the logs to an external log server.

Step 4
Examine the interface of Cisco HyperFlex Connect.

Answer
In your browser, open the URL https://hxconnect-b.hx.pod/ and log in using
credentials admin / C1sco12345.

Examine the information in the Dashboard.

You should find that the group is online, is both healthy and able to tolerate 1 node
failure, and has 4 converged nodes.

Step 5
Examine the System Information pane.

Answer
In Cisco HyperFlex Connect, click the System Information menu option in
the Manage section of the menu.
The System Overview tab of the System Information pane offers the most information
about the HyperFlex group at a glance, including various subsystem IP addresses, drive
and node statuses, and maintenance functionalities.

Go to the Nodes tab of the System Information pane, and click a node. Options to
enter and exit HX Maintenance Mode should be available.
The HX Maintenance Mode handles shutdown of the CVM but is otherwise similar to
regular vSphere maintenance mode. Always use the HyperFlex maintenance mode on
converged nodes.

Go to the Disks tab of the System Information pane, and choose one of the disks. You
should see the options to control the Locator LED.

Step 6
Create a datastore with the size of 1 TB, and explore datastore management.

Answer
Click the Datastores option in the Manage section of the menu.
You will notice that no datastores are created so far.

Click Create Datastore in the top-left corner. Name the datastore Cluster B
Datastore, set its size to 1 TB, leave the Block Size at 8K, and click Create Datastore.

After a few seconds, the new datastore will appear. The status should be Normal and
the datastore should be mounted.

Click Cluster B Datastore to view more information.

Here you are able to see on which hosts the datastore is mounted and if the datastore is
accessible. You can also see the capacity of the datastore and various performance
charts. Keep in mind that sometimes performance charts will not be available
immediately after creation since this is a virtualized environment.

For the HyperFlex storage to be usable, all the hypervisors on individual hosts must
have access to the HyperFlex datastore. If any node does not have the datastore
mounted, that is an error.

Note
When a single node is disconnected from the group due to a network issue, it might
refuse to mount the datastore after being reconnected. Reboot the affected node to
resolve this issue.

Select Actions > Edit, and increase the datastore size to 2 TB. Click Edit Datastore to
apply changes.
The size change should succeed, since live size changes of Cisco HyperFlex datastores
are supported. A size change does not require re-initialization of the datastore or impact
the performance of the workload that is running on the datastore during the
reconfiguration.

Note
You can create a datastore that exceeds the raw storage of the group. The system will
notify you when the physical storage nears critical usage.

Edit the datastore again. Click Actions > Edit and try to change its name to Store_B.
You are not able to change the name because the datastore is mounted on the hosts of
the HyperFlex group. To rename a datastore, you should shut down all the VMs on it,
unmount the datastore, rename it, and then remount it.

Step 7
Deploy a VM on the Cluster B HyperFlex datastore using the HTML5 vSphere client.

Answer
Return back to the HTML5 vSphere interface, and choose one of the nodes from
the Standard Cluster B group. Verify that the datastore you created is now visible by
the host that you chose.

Note
If the Cluster B Datastore is not reporting the size of 2 TB, refresh the vSphere
interface.

Right-click the host esx-b-1.hx.pod of the Standard Cluster B. Choose Deploy OVF
Template in the menu.
In the Select an OVF Template menu, choose the Local file check box, click
the UPLOAD FILES button, and choose the Tiny Linux VM.ova file located in
the DCIHX > Virtual Machines folder.

After you have chosen the .ova file to deploy, click Next to continue.

In the name selection step, keep the default name Tiny Linux VM, and
choose HyperFlex Datacenter as the deployment target.
Click Next.

Deploy the VM on the host esx-b-1.hx.pod of Standard Cluster B.

Click Next.

In the Review details step, you can review your current configuration up to this point.
Click Next to continue without changing any options on the review step.

In the Select storage step, choose Cluster B Datastore as the target for the VM
deployment.

Note
Never deploy VMs on the internal server storage. This storage is
not part of the HyperFlex Data Platform. Therefore, it is not
redundant or distributed.
You have just chosen the HyperFlex datastore that you previously created through
HyperFlex Connect.

Click Next.

In the Select Networks step, choose VM Network from the drop-down menu.
Note
The VM network is configured by default in a Cisco HyperFlex environment, but you
can create other networks and use different VLANs for your VMs, as long as the
HyperFlex-specific networking is not changed.

Click Next.

In the Ready to complete step, click Finish to deploy the VM.

You can see that the VM is now successfully deployed on the HyperFlex datastore.
Leave the VM shut down for now.
Step 8
Examine VM management.

Answer
Return back to HyperFlex Connect, click on the Virtual Machines menu option in
the Manage section of the menu.
The interface will list all the VMs that you created on the Standard Cluster B. You can
also manage the VMs from the same interface.

Choose Tiny Linux VM by checking the check box in front of its name. Click Power
On to start the VM.

The system will indicate that the task is started. After a few seconds, if the VM is
not Powered On, refresh the page.
Go to the vSphere interface and confirm that the VM has been started.

A green arrow next to the VM icon indicates that the VM is powered on.

After you confirm that the VM has been started, return to Cisco HyperFlex Connect.

Step 9
Explore the Monitor and Analyze subsections of HyperFlex Connect.
Answer
Click and observe each subsection of the Monitor group of views in HyperFlex
Connect.

The Monitor group displays the notifications for the group: critical alarms are shown
under Alarms, Events displays the warning and info levels of log messages,
and Activity displays actions that are taken on the group. For example, you can see the
power on virtual machine action that you performed in the previous steps.

Explore the Performance view in the Analyze menu subsection.

You should see graphs for IOPS, throughput, and latency. You can change the time
range of the graph, and the parameters to show.

Step 10
Examine the Upgrade pane.

Answer
In Cisco HyperFlex Connect, click the Upgrade menu option in the Manage section of
the menu.

Check the checkboxes in the Select Upgrade Type panel to see what is required for the
automated upgrade of an individual HyperFlex component.
The Upgrade pane allows you to upgrade the components of the Cisco HyperFlex
system. Upgrades are done in a rolling fashion and do not interrupt the workload.

An automated upgrade through Cisco HyperFlex Connect is not the fastest way to
upgrade the HyperFlex installation, but it is not disruptive and is the most user-friendly
as it does not require an in-depth understanding of the Cisco UCS and Cisco HyperFlex
inner workings.

Install and Explore Cisco Hyperflex HTML5 Plug-In in vSphere

Cisco HyperFlex HTML plug-in for VMware vCenter, communicates with HyperFlex
Connect to synchronize monitoring messages and provides HyperFlex functionalities in
vCenter. The plug-in lets you perform many day-to-day tasks, including datastore
creation, directly from the vCenter.

Step 11
Verify if the HyperFlex HTML5 plug-in is installed.

Answer
Connect to the vSphere web client and click Menu to see the items from the drop-down
menu. Look for Cisco Hyperflex.
You will notice that the Cisco HyperFlex plug-in cannot be found on the list.

Step 12
Install HyperFlex HTML5 plug-in for vSphere web client.

Answer
Go to the top-left corner of the screen and click the blue icon. Choose Terminal
Emulator from the drop-down menu.
Once the terminal window opens, use the cd DCIHX/Software/ command to move to
the software directory. Type ls to list all the files in this directory.

student@student-vm:~$ cd DCIHX/Software/
student@student-vm:~/DCIHX/Software$ ls
HyperFlex-VC-HTML-Plugin-2.1.0.zip

You will notice that the Hyperflex plug-in installation file is present.

Note
Since this is a lab environment, the plug-in installation file is already provided. In a real
environment, you would go to the Cisco Software download portal, search for
HyperFlex HX Data Platform, choose the 4.5 version, and look for the HyperFlex
HTML plug-in file under related software.

Copy the installation file to the controller VM that is located on node 1 of the
HyperFlex group A. Type scp HyperFlex-VC-HTML-Plugin-2.1.0.zip
admin@10.1.100.121:/tmp and press Enter. Provide the CVM password when
prompted.

student@student-vm:~/DCIHX/Software$ scp HyperFlex-VC-HTML-Plugin-2.1.0.zip

admin@10.1.100.121:/tmp
Warning: Permanently added '10.1.100.121' (ECDSA) to the list of known hosts.
HyperFlex StorageController 4.5(1a)
admin@10.1.100.121's password:
HyperFlex-VC-HTML-Plugin-2.1.0.zip
Note
SCP stands for Secure Copy Protocol and is used to securely transfer files between two
hosts over Secure Shell (SSH) protocol.

Log in to the controller VM. Type ssh -l admin 10.1.100.121 and provide
CVM password when prompted.

student@student-vm:~/DCIHX/Software$ ssh -l admin 10.1.100.121

HyperFlex StorageController 4.5(1a)
admin@10.1.100.121's password:
---------------------------------------------------------
!!! ALERT !!!
This service is restricted to authorized users only.
All activities on this system are logged. Unauthorized
access will be reported.
---------------------------------------------------------

HyperFlex StorageController 4.5(1a)

This is a Restricted shell.
Type '?' or 'help' to get the list of allowed commands.

Move to the /tmp folder and list all files located in this directory. Use command cd
/tmp and ls to do so.

admin:~$ cd /tmp
admin:/tmp$ ls
generic_disk_info.DGu hxdc_au nginx storfs.pid tmp.3mEpFWHRd1
tmp.ak80InYQwP tmp.k4tFiaxSYR tmp.OMcJOoUP9e tmp.sjSK4tmP4w tmp.XUwgrP8DnC
zookeeper
hsperfdata_root hxdc_tmp scvmclient.pid tmp.2d7rJpxqac tmp.6ndLQZRDoC
tmp.Hl6osVEMRE tmp.KGCXBKojj9 tmp.onLReW9Iu9 tmp.UdyKaWvCHp tomcat8-tomcat8-
tmp
hsperfdata_tomcat8 HyperFlex-VC-HTML-Plugin-2.1.0.zip storfseventsfifo tmp.34fH3IGmph
tmp.7cvFGD75H5 tmp.HtWfScpAiw tmp.Mh85ank3SE tmp.PHtBDzOtaQ tmp.vuEHuB67mt
vmware-root

You will see many files in this folder. Among them should also
be the plug-in installer that you copied. If you cannot find the
file, check and repeat the SCP command used in the previous
step.

Unzip the installer file using unzip HyperFlex-VC-HTML-Plugin-

2.1.0.zip command. You can avoid typing the entire name of
the file by typing a few first letters and pressing the Tab key
for autocomplete.
admin:/tmp$ unzip HyperFlex-VC-HTML-Plugin-2.1.0.zip
Archive: HyperFlex-VC-HTML-Plugin-2.1.0.zip
inflating: hx-html-plugin.zip
inflating: install_vc_plugin.py
inflating: readme.txt

You will notice that three files were unzipped. Use the command install_vc_plugin to
run the plug-in installation script. When prompted, type vcenter.hx.pod for the vCenter
FQDN, administrator@hx.pod / 1234QWer* for vCenter credentials,
and C1sco12345 for storage controller root and admin passwords.

admin:/tmp$ install_vc_plugin
Initiating installation of Cisco HyperFlex vCenter plugin...
Enter vCenter FQDN/IP: vcenter.hx.pod
Enter administrator vCenter username: administrator@hx.pod
Enter password for administrator@hx.pod:
Enter storage controller root password:
Enter storage controller admin password:
Copying plugin to storage controller VMs...
Copying plugin to 192.168.0.52
Copying plugin to 192.168.0.53
Copying plugin to 192.168.0.51
Cisco HyperFlex vCenter Plugin version '2.1.0' is registered successfully with vCenter server
vcenter.hx.pod

You will notice that the HyperFlex plug-in for vCenter is now installed.

Close the terminal window and refresh the vSphere site in your browser.
Click Menu and verify if the Cisco Hyperflex is now visible on the list.
If you still cannot see the HyperFlex plug-in, log out and log in back to the vCenter.
Step 13
Explore the HyperFlex plug-in in the vSphere web client.

Answer
In the vSphere Client, click Menu and choose Cisco HyperFlex.

If you see an empty HyperFlex Clusters list, click Rescan to search for the HyperFlex
groups registered with the vCenter.
The HyperFlex plug-in manages all the HyperFlex groups that are associated with the
vCenter installation. Additional groups are registered automatically during installation.
If you manually move a group, you need to run the stcli cluster
reregister command from the CVM command line to register an
extra group.

Click Standard Cluster B.

You should see group information such as storage utilization, number of VMs, ESXi
and HyperFlex version, uptime, and deployment type.

Expand the Resiliency Status in the Status section.

The group should be Online, Healthy, and report four hosts in the group and one node
failure tolerance.

Choose the Performance tab.

You should see graphs for IOPS, throughput, and latency. The graph might not be
populated yet, because the group was just installed.
Note
You have options to change the time range of the graph, and to
choose different monitoring objects: Storage Cluster, Hosts,
and Datastores.

Choose the Nodes tab.

You should see all nodes in this group and all CVMs residing on those nodes.

Click Datastores.
This tab reveals the HyperFlex datastore management interface. Here you can configure
new datastores, view datastore information, and reconfigure existing datastores. The list
should also include the datastore that you configured in the previous steps.

Click the Cluster B Datastore.

You should see more options and information for the chosen datastore. For example,
you can edit, unmount, or delete the datastore.
Configure iSCSI Target on a Cisco HyperFlex Group

With Cisco HyperFlex release 4.5(1a), the HyperFlex iSCSI Target Service became
supported for Standard HyperFlex groups. This service allows you to present block
storage to applications running inside or outside the HyperFlex group.

Step 14
Configure an iSCSI network.

Answer
To configure iSCSI on your HyperFlex group, click the iSCSI option in
the Manage section of the main menu in HyperFlex Connect.

You will notice that the iSCSI network configuration is needed first.

Click Configure Network.

Enter 10.3.100.0/24 in the Subnet field and insert IP addresses 10.3.100.200 and
10.3.100.210 in the IP Range fields. Click Add IP Range when finished.

The IPs from the IP range that you specified in this step are going to be configured on
the CVMs.

Set 10.3.100.250 as the iSCSI Storage IP, check the Set non default MTU check box,
and choose 1500 from the corresponding list.
This will configure another IP address on your group from where you will be able to
discover LUNs configured in the following steps.

Under VLAN Configuration, click Create a new VLAN and specify VLAN ID,
VLAN name, and Cisco UCS Manager information. You can find all the required
information in Job Aids. Click Configure.

Note
In this case, VLAN 0 was used for encapsulation of the iSCSI data. Keep in mind that
this is not recommended in the real environment.

Configuration of the iSCSI network can take some time. Go to the Activity pane to
monitor the progress of the iSCSI network configuration.
Go back to the iSCSI tab and verify that the iSCSI network is configured.

You should see that now more iSCSI configuration options are available.

Step 15
Configure iSCSI initiator, target, and LUN.
Answer
Go to the top-left corner of the screen and click the blue icon. Open the Terminal
Emulator and type the sudo cat /etc/iscsi/initiatorname.iscsi command.
Press Enter and type 1234QWer password when prompted.

student@student-vm:~$ sudo cat /etc/iscsi/initiatorname.iscsi

[sudo] password for student:
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator. The InitiatorName must be unique
## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:67e135f5d357

With the command that you used, you will display the contents of
the initiatorname.iscsi file. As the name implies, the file holds the configured initiator
iSCSI Qualified Name (IQN). This name is used to identify nodes on the iSCSI network
and should be unique within the operational domain.

Copy the initiator name and return to HyperFlex Connect.

Click Initiator Groups and then Create to create a new initiator group. Name
it student-VM, paste the copied IQN, and click Add initiators.

Click Create Initiator Group to continue. Verify that configured IQN is now visible
under Initiators tab.
Go back to Targets and click Create. Name it Cluster_B and click Create Target.

Click Create LUN. Name the LUN 100GB_LUN, set its size to 100 GB, and
click Create LUN.
Note
LUN stands for logical unit number and represents a dedication partition configured on
a shared storage.

Go to Linked Initiator Groups and click Link.

Choose the student-VM initiator group and click Link Initiator Group(s).
This will allow initiators configured in the initiator group to discover LUNs configured
for a specific target.

Step 16
Mount the LUN to the student VM.

Answer

Open Terminal Emulator and enter the lsblk command. Press Enter.

student@student-vm:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 98.4M 1 loop /snap/core/10823
loop1 7:1 0 175.4M 1 loop /snap/postman/133
loop2 7:2 0 175.4M 1 loop /snap/postman/132
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /
sr0 11:0 1 1024M 0 rom

Note the devices that are available.

Note
The lsblk command is used to list all available block devices in the Linux system.

Enter the sudo iscsiadm -m discovery -t sendtargets -p

10.3.100.250 command.

student@student-vm:~$ sudo iscsiadm -m discovery -t sendtargets -p 10.3.100.250

10.3.100.250:3260,1 iqn.1987-02.com.cisco.iscsi:Cluster_B

Compare the displayed IQN with the IQN visible in the HyperFlex Connect under the
Cluster_B target. The IQNs should be the same.

Use the same command but add an -l option at the end. Press Enter.

student@student-vm:~$ sudo iscsiadm -m discovery -t sendtargets -p 10.3.100.250 -l

10.3.100.250:3260,1 iqn.1987-02.com.cisco.iscsi:Cluster_B
Logging in to [iface: default, target: iqn.1987-02.com.cisco.iscsi:Cluster_B, portal:
10.3.100.250,3260] (multiple)
Login to [iface: default, target: iqn.1987-02.com.cisco.iscsi:Cluster_B, portal:
10.3.100.250,3260] successful.

This command will log in the initiator to the target and allow the initiator to use the
configured LUNs.

Enter the lsblk command again.

student@student-vm:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 98.4M 1 loop /snap/core/10823
loop1 7:1 0 175.4M 1 loop /snap/postman/133
loop2 7:2 0 175.4M 1 loop /snap/postman/132
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /
sdb 8:16 0 100G 0 disk
sr0 11:0 1 1024M 0 rom

You will notice that one more block storage device is now visible by the system.

Use the sudo mkfs.ext4 /dev/sdb command to format the new partition.

student@student-vm:~$ sudo mkfs.ext4 /dev/sdb

mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 26214400 4k blocks and 6553600 inodes
Filesystem UUID: 6f3b38a3-0e58-4399-8a98-a166e237a9af
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done

Writing inode tables: done
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done

The mkfs.ext4 command is used to format the new drive in the ext4 file system
format. By doing so, the partition becomes available for storing data.
Use the mkdir DCIHX_LUN command to create a new directory. Verify that the
directory is created successfully with the ls command.

student@student-vm:~$ mkdir DCIHX_LUN

student@student-vm:~$ ls
DCIHX_LUN DCIHX Desktop Documents Downloads Music Pictures Public snap Templates
Videos

Enter the sudo mount /dev/sdb /home/student/DCIHX_LUN/ command to

mount the iSCSI device to the folder.

student@student-vm:~$ sudo mount /dev/sdb /home/student/DCIHX_LUN/

Now the data can be stored on the device. Use the following command to copy a file to
the iSCSI device.

student@student-vm:~$ sudo cp ./DCIHX/Instructions/Instructions.pdf ./DCIHX_LUN/

If you do not see any message, the command was executed successfully.

Go to the top-left corner, click the blue icon, and choose the File Manager application
from the list. Double-click the DCIHX_LUN directory. Verify that
the Instructions.pdf that you copied in the previous steps is visible in the directory.

Step 17
Edit and clone the LUN.

Answer

Go back to the iSCSI page of the HyperFlex Connect. Choose the configured LUN and
click Edit.
You will notice that you can change the name of the LUN and its size. Click Cancel.

Click Clone LUN. Use Cluster_B_clone for the new destination target and LUN name.
Click Clone.

Verify that a new target with Cluster_B_clone name appeared.

This option allows you to efficiently clone data, for example, for development purposes.

Connect Cisco HyperFlex Group to Remote Syslog Server

Since Cisco HyperFlex Version 4.5.1a, you can connect your HyperFlex group to a
remote syslog server directly from HyperFlex Connect, so you can aggregate system
messages in a centralized location and further integrate the logs into the monitoring
systems.

Step 18
Configure the Export of Audit Logs.

Answer
Click in the gear icon in the top-right corner of HyperFlex Connect, and choose Audit
Log Export Settings from the menu.
Step 19
Enable the Export and apply syslog configuration from the Job Aids.

Answer
In the Audit Log Export Settings window, check the check box to enable syslog
export.

Configure the connection settings to match the configuration of the syslog server that is
running in your pod. Refer to Job Aids for values. When done click, OK to continue.
Note
To make configuration easier, the syslog server has been configured without additional
encryption. In your environment, if you need additional security, use the recommended
Transport Layer Security (TLS) configuration.

Audit log export is now set up.

Step 20
Verify logs that have been exported to the remote syslog server.

Answer
The remote syslog server in this lab was set up on the student VM that you are using.

Click the white and blue button in the top-left corner of the student VM interface, and
choose File Manager from the menu. Navigate to the /home/student/Desktop/Logs/
directory.

You will see that many directories are present that have been automatically generated
using the HyperFlex audit log export functionality. If directories are not present check if
the configuration in HXConnect is correct and wait up to 10 minutes for logs to be
exported.

Double-click on one of the directories, for example, HX-CTL-01, and observe the files
located in the directory. Open one of the files and view its content.
You will notice that there are many messages in the log file that were exported from
Standard Cluster B.

Explore Cisco HyperFlex CLI

The Cisco HyperFlex command line is the most powerful interface of Cisco HyperFlex.
The CLI can perform nearly all Cisco HyperFlex-specific functions of other interfaces,
provides the most feedback, and allows you to access Cisco HyperFlex logs.

Currently two Cisco HyperFlex CLIs are supported: stcli and hxcli. The Cisco
HyperFlex hxcli commands are replacing stcli commands. When possible,
use hxcli commands as they perform much faster than the same stcli commands. Syntax
of both commands is extensive, but you can always get information about individual
commands by appending -h to the command.

Step 21
Get help about the hxcli commands in the CVM command line.

Answer
Click the white and blue button in the top-left corner of the student VM interface, and
choose Terminal Emulator from the menu.

Enter the ssh -l admin hxconnect-b.hx.pod command to connect to the HyperFlex group
command line. Use the same C1sco12345 password as for logging in to the HyperFlex
Connect.

In the CVM command line, type the hxcli -h command and press Enter.

admin:~$ hxcli -h
Hyperflex Command Line Interface
Usage:
hxcli [flags]
hxcli [command]

Available Commands:
about Cluster version, model and other metadata details
cluster Commands supported in the cluster namespace
dataprotection Commands supported in the Data Protection namespace
datastore Commands supported in the datastore namespace
disk Commands supported in the Storage disk namespace
events Commands supported in the events namespace
help Help about any command
iscsi Commands supported in the iscsi namespace
jobs Commands supported in tasks namespace
node Commands supported in the Storage node namespace
security Commands supported in the security namespace
services Commands supported in the services namespace
tasks command.hxtask.desc.short
vcenter Commands supported in vCenter namespace
version HX CLI Version
volume Commands supported in the volume namespace

Flags:
-h, --help help for hxcli

Use "hxcli [command] --help" for more information about a command.

The help flag shows you possible arguments for the stcli command.

Type hxcli cluster -h in the command line to get more information about
the cluster argument.

admin:~$ hxcli cluster -h

Displays the list of commands available in the cluster namespace.

Usage:
hxcli cluster [flags]
hxcli cluster [command]

Aliases:
cluster, cl

Available Commands:
about Cluster version, model and other metadata details
detail Short summary of cluster details
health Health and resiliency information of the cluster
info Detailed information about cluster
shutdown Shuts down the Hyperflex cluster
start Start the Hyperflex cluster
 stats Cluster capacity and space savings
status Cluster Operational Status and Resiliency Health

Flags:
-h, --help help for cluster

Use "hxcli cluster [command] --help" for more information about a command.

You can string together arguments and append -h.

Execute the hxcli cluster info command. You should see basic information
about the group version, state, and capacity.

admin:~$ hxcli cluster info

Cluster Name : Standard Cluster B
Cluster UUID : 7409053511665237662:6119592481993350064
Cluster State : ONLINE
Cluster Access Policy : Lenient
Space Status : NORMAL
Raw Capacity : 8.0 TB
Total Capacity : 3.7 TB
Used Capacity : 76.2 GB
Free Capacity : 3.6 TB
Compression Savings : 0.00%
Deduplication Savings : 0.00%
Total Savings : 0.00%
# of Nodes Configured :4
# of Nodes Online :4
Data IP Address : 192.168.0.250
Resiliency Health : HEALTHY
Policy Compliance : COMPLIANT
Data Replication Factor : 2 Copies
# of node failures tolerable :1
# of persistent device failures tolerable : 1
# of cache device failures tolerable : 1
Zone Type : Unknown
All Flash : No

Keep in mind that the same command might be reporting different health and policy
compliance status in your case.

The hxcli cluster info command only displays information, but you can also
use the hxcli command to manage the group.

Step 22
Create and resize a datastore using the CVM command line.
Answer

Type hxcli datastore create -h in the command line. Try to discern the
syntax to create a 2 GB datastore with the name CLI_Datastore and 8k block size.

admin:~$ hxcli datastore create --name CLI_Datastore --size 2 --unit gb --block-size 8k

Datastore created successfully

+---------------+---------------+------------+--------------+--------------+--------------+--------+
| NAME | MOUNT SUMMARY | STATUS | USAGE TYPE | SIZE | USED | FREE
|
+---------------+---------------+------------+--------------+--------------+--------------+--------+
| CLI_Datastore | MOUNTED | NORMAL | NFS | 2.0 GB | 0 B | 2.0 GB |
+---------------+---------------+------------+--------------+--------------+--------------+--------+

The command will list the information about the newly created datastore and the
datastore will show up in HyperFlex Connect and vCenter.

Go to Cisco HyperFlex Connect, click the Datastore option in the Manage section of
the menu, and verify that the new datastore was created. Wait until the new datastore is
mounted and its status is listed as Normal.

Return to the CVM command line and list information about the newly created
datastore. Take a note of the datastore UUID.

admin:~$ hxcli datastore info --name CLI_Datastore

UUID : 00000000c3982890:0000000000010188
Name : CLI_Datastore
Mount Summary : MOUNTED
 Status : NORMAL
Usage Type : NFS
Size : 2.0 GB
Used :0 B
Free : 2.0 GB
Block Size : 8.0 kB

Change the size of the CLI_Datastore to 5 GB. Attempt to get information about
syntax by using the -h switch.

admin:~$ hxcli datastore update --id 00000000c3982890:0000000000010188 --size 5 --

unit gb

Datastore edited successfully

UUID : 00000000c3982890:0000000000010188
Name : CLI_Datastore
Mount Summary : MOUNTED
Status : NORMAL
Usage Type : NFS
Size : 5.0 GB
Used :0 B
Free : 5.0 GB
Block Size : 8.0 kB
+-----------+---------+------------+--------+
| HOST NAME | MOUNTED | STATUS | REASON |
+-----------+---------+------------+--------+
| HX-02 | Yes | ACCESSIBLE | |
| HX-04 | Yes | ACCESSIBLE | |
| HX-03 | Yes | ACCESSIBLE | |
| HX-01 | Yes | ACCESSIBLE | |
+-----------+---------+------------+--------+

In Cisco HyperFlex Connect, confirm that the CLI_Datastore is now 5 GB in size.

Refresh Cisco HyperFlex Connect if necessary.
Step 23
Use the CVM command line to view the nodes and list the disks that are installed in the
nodes.

Answer

In the CVM command line, type the hxcli disk list command and
press Enter.

admin:~$ hxcli disk list

+-----------+-------------------+------+----------+---------+------------+-------------+
| NODE NAME | HYPERVSIOR STATUS | SLOT | CAPACITY | STATUS | TYPE | USAGE |
+-----------+-------------------+------+----------+---------+------------+-------------+
| HX-01 | ONLINE | 1 | 512.0 GB | Claimed | Rotational | Persistence |
| HX-01 | ONLINE | 2 | 512.0 GB | Claimed | Rotational | Persistence |
| HX-01 | ONLINE | 3 | 512.0 GB | Claimed | Rotational | Persistence |
| HX-01 | ONLINE | 4 | 2.5 GB | Claimed | Rotational | System |
| HX-01 | ONLINE | 5 | 400.0 GB | Claimed | Solidstate | Caching |
| HX-01 | ONLINE | 6 | 512.0 GB | Claimed | Rotational | Persistence |
| HX-01 | ONLINE | 7 | 200.0 GB | Claimed | Solidstate | System |
| HX-01 | ONLINE | 8 | 0 B | Unknown | | |
| HX-01 | ONLINE | 9 | 0 B | Unknown | | |
| HX-01 | ONLINE | 10 | 0 B | Unknown | | |
| HX-01 | ONLINE | 11 | 0 B | Unknown | | |
| HX-01 | ONLINE | 12 | 0 B | Unknown | | |
| HX-01 | ONLINE | 13 | 0 B | Unknown | | |
| HX-01 | ONLINE | 14 | 0 B | Unknown | | |
| HX-01 | ONLINE | 15 | 0 B | Unknown | | |
| HX-01 | ONLINE | 16 | 0 B | Unknown | | |
| HX-01 | ONLINE | 17 | 0 B | Unknown | | |
| HX-01 | ONLINE | 18 | 0 B | Unknown | | |
<... output omitted ...>

The command will display the information about all disks installed in the nodes of the
group. You are able to see the slot in which a particular disk is installed, its capacity and
size, the type of the disk, and the usage of the disk.

Step 24
Use the CVM command line to check the rebalance status and manually trigger a
rebalancing operation.

Answer

In the CVM command line, type stcli rebalance status, then press Enter to
get information about whether the rebalance job is already under way.

admin:~$ stcli rebalance status

rebalanceEnabled: True
rebalanceStatus:
rebalanceState: cluster_rebalance_not_running
percentComplete: 0

Attempt to start rebalancing using the stcli rebalance start command.

admin:~$ stcli rebalance start

msgstr: Rebalance is not needed
params:
msgid: Rebalance is not needed

The stcli command still checks if a command is necessary before blindly executing the
task.

Step 25
Use the CVM command line to manage VMs.

Answer

In the CVM command line, type stcli vm -h to list available VM-related

functions.

admin:~$ stcli vm -h
usage: stcli vm [-h] {clone,schedule-create,count,snapshotInfo,snapshot} ...

Storage vm operations.

positional arguments:
{clone,schedule-create,count,snapshotInfo,snapshot}
Commands supported in the Storage vm namespace.
clone Creates a specified number of clones for the given vm.
 schedule-create create vm snapshot schedule
count VM count having snapshot and schedule snapshot
snapshotInfo Snapshot count and schedule type for all VMs
snapshot Creates a native snapshot for the given vm.

optional arguments:
-h, --help show this help message and exit

The CVM command line can also perform HyperFlex native cloning and snapshotting,
the same as the vCenter and HyperFlex Connect. Some services are exclusive to the
CVM command line and cannot be performed from anywhere else.

Step 26
Use the CVM command line for licensing and management configuration.

Answer

In the CVM command line, type stcli license show status to list the current
licensing status.

admin:~$ stcli license show status

Smart Licensing is ENABLED

Registration:
Status: UNREGISTERED
Export-Controlled Functionality: Not Allowed

License Authorization:
Status: EVAL MODE
Evaluation Period Remaining: 85 days, 2 hr, 19 min, 20 sec
Last Communication Attempt: NONE

License Conversion:
Automatic Conversion Enabled: true
Status: NOT STARTED

Utility:
Status: DISABLED

Transport:
Type: CALLHOME

The group is not licensed. The 90-day grace period is in effect.

Note
To license your HyperFlex group, generate a smart licensing token on the Cisco Smart
Licensing Portal and provide the token in the stcli license register --
idtoken token. The HyperFlex group would then connect to the Cisco Smart
Licensing Portal and register the group to your Cisco account. There are also offline
options available.

In the CVM command line, type stcli services -h to list the services
configuration options.

admin:~$ stcli services -h

usage: stcli services [-h] {smtp,dns,dnssearch,ntp,asup,sch,remotesupport,timezone} ...

system services related operations

positional arguments:
{smtp,dns,dnssearch,ntp,asup,sch,remotesupport,timezone}
stCli system services
smtp Commands supported in the Storage SMTP configuration namespace.
dns Commands supported in the Storage DNS configuration namespace.
dnssearch Commands supported in the Storage DNS Search configuration namespace.
ntp Commands supported in the Storage NTP configuration namespace.
asup Commands supported in the ASUP configuration namespace.
sch Commands supported in the smart-callhome configuration namespace.
remotesupport Commands supported for remote support.
timezone Commands supported in the Timezone configuration namespace.

optional arguments:
-h, --help show this help message and exit

The service configuration and licensing are unique to the stcli command line. For
example, if your DNS server address changes, you would change it using the CVM
command line.

Manage Cisco Hyperflex Group Using the REST API

Though the CVM command line can be used for basic automation, the Cisco HyperFlex
REST API is the best automation tool.

In this task, you will explore the REST API through an integrated GUI that is available
on the Cisco HyperFlex Connect address, by appending /apiexplorer/ to the URL.

Step 27
Connect to API Explorer and log in to the user interface.

Answer

Type https://hxconnect-b.hx.pod/apiexplorer/ into your browser,

and press Enter.
On the list, you will be able to see the different API segments that provide different
management and monitor options for your HyperFlex group.

Step 28
Explore the Group Life-cycle Management API functionality.

Answer
Open the Core APIs [ESX, HYPERV, and HXAP Platforms] to reveal different
functionalities of the API.
Choose Cluster Life-cycle Management API to list all available
operations.

The operations are marked with three different colors which indicate three different
HTTP methods. The GET method (blue) is used to read or retrieve the data, the PUT
(orange) is used to update the data, and the POST method (green) is used to create a
new resource. There is also another HTTP method that is frequently used which cannot
be seen on this list. It is called the DELETE method which, as the name implies, is used
to delete data.

Click on the GET /clusters operation to view more information about the operation.
Here you can see the different parameters that can be used with this operation and
response class.

Scroll down until you see the Response Messages section.

Here you can see the example of bad request response and internal server error
response.

You should now see the Try it out! button. Click the button to send the request and
when prompted, use credentials admin / C1sco12345 to log in.
Note
The example demonstrates a sample request to fetch a list of groups. The Response
Code 200 tells you that the query has been processed successfully and you can see the
actual response data in the Response Body field.

Take note of the Standard Cluster B UUID because you will need it later. Go to the top-
left corner, click the blue icon, and type Mousepad in the search bar, and choose
the Mousepad application. Copy the Standard Cluster B UUID and paste it in the empty
document.

Click Back to REST API Explorer located in the top-right corner, then
choose Authentication Authorization and Accounting APIs [ESX, HYPERV, and
HXAP Platforms].
You will be able to see all REST APIs that are used to authenticate users and
grant/validate access tokens. You are currently interested in the access token since you
will need it in the following steps.

Choose the Obtain Access Token API and expand the POST operation by clicking it.

You will be able to see the response class that you receive if the request is successfully
processed.

To make the request succeed, you need to send the right credentials with your request.
Scroll down until you see the request body section. Populate the body field by clicking
the yellow field with example values. Replace the example values in the payload
with admin and C1sco12345 as shown in the following figure.
Once you are finished, you can fetch the access_token by sending the request. Scroll
down and click the Try it out! button again.

Observe the response body and take a note of the access_token. You will use this token
to view the groups and datastores.

When you are done, click the blue icon in the top-right corner, type postman in the
search bar, and open the Postman application.
Note
Postman is an API testing tool that allows you to construct HTTP requests, interact with
APIs, and inspect the responses.

Once the application loads, click Create a Request under Start something new.

A new tab where you can construct your requests will open.

Postman supports different authentication methods. In this case, an access_token needs

to be sent with the request. To enter the token, go to the Authorization tab, expand the
type drop-down menu, and choose Bearer Token.
Copy the access token that you obtained in the previous steps and paste it in
the Token field.

Fill the request URL field.

Type https://hxconnect-b.hx.pod/coreapi/v1/clusters/cluster_U
UID/datastores, replace the cluster_UUID value with the value that you noted
before in mousepad. Verify that the URL is correct and click the Send button.
You will see that Postman is not able to get the response because of the SSL error. This
is the expected behavior since the self-signed certificate is present on https://hxconnect-
b.hx.pod.

Click Disable SSL Verification and click the Send button again.

Postman will display the response in the lower part of the window. You should see the
list of datastores in the response body. Note that the CLI_datastore that you created
using the stcli datastore create command is present on the list. Copy
the UUID of the datastore in your mousepad.
Note
After some time, the access token expires. If you get an error 401 after you click send
the request, you need to generate the access token again. Use
credentials admin / C1sco12345 to do so.

Click the plus (+) icon located next to the GET request to open a new empty request tab.
Change the request type to DELETE and set the authorization type to Bearer Token.

Copy the request URL from the previous request and copy it to the new one. To make
a request to the correct endpoint, you need to append the path to the chosen datastore.
Add one more / to the URL and then insert the datastore_UUID, which
corresponds to the UUID of the CLI_Datastore. Your request URL should now look
like
https://hxconnect-b.hx.pod/coreapi/v1/clusters/cluster_UUID/d
atastores/datastore_UUID.
Click Send to send the request.

After the request is processed, you should see the empty response body and response
status 200 OK which means that the datastore was successfully deleted. If the response
status does not equal to 200, check your request again and try to find a mistake.

Verify successful deletion of the datastore in HyperFlex Connect by going in

the Datastores pane.
You should see only Cluster B Datastore on the list since you deleted the
CLI_Datastore using REST API.

Note
During this lab task, you discovered only a few HyperFlex API commands. Feel free to
further investigate the HyperFlex REST API explorer to find more useful functions that
are available to manage and monitor the HyperFlex deployment.

Compare Cisco HyperFlex and vSphere Native Cloning and Snapshotting

Cisco HyperFlex offers high performance storage-based cloning and snapshotting. Both
of these functionalities are available in the HyperFlex Connect.

The fundamentals of snapshotting and cloning are similar. You can accelerate
snapshotting and cloning by using the native functionalities of HyperFlex StorFS.

Step 29
Create a vSphere clone.

Answer

In the vSphere client, click Menu on top of the interface, and choose Hosts and
Clusters. Find Tiny Linux VM and click it.
Right-click Tiny Linux VM, and choose the Clone submenu, but do not choose any of
the options yet.

The Clone menu lists the vSphere native jobs, which are not accelerated by HyperFlex
storage. The result of each clone operation in the native vSphere will be a single object:
either a VM or a template (from which further VMs can be created).

Choose Clone to Virtual Machine from the Clone submenu. Name the
clone vSphere Tiny Clone, and choose HyperFlex Datacenter as the target for
deployment.
Click Next.

Choose esx-b-1.hx.pod of the Standard Cluster B as the computer resource for the
cloned VM.

Click Next.

Choose Cluster B Datastore as the target datastore for the clone.

Click Next.

Check the Power on virtual machine after creation check box in the Select clone
options step.

Click Next and, in the Ready to complete step, click Finish to complete the cloning
job.
You have successfully created a vSphere clone of the Tiny Linux VM. You can repeat
his process to create additional clones one by one.

Step 30
Create 10 HyperFlex native clones in HyperFlex Connect.

Answer
Switch back to HyperFlex Connect of the Standard Cluster B. Choose Virtual
Machines in the manage panel and click Tiny Linux VM.
You will notice that options such as Ready Clones, Snapshot Now, and Schedule
Snapshot became available.

Choose Ready Clones to configure a batch cloning job for 10 clones. Add the
prefix HX-Clone- to the VM name, and check the Power on VMs after cloning check
box.

Click Clone to create the VMs according to the information that you entered.

HyperFlex native cloning is performed as a batch job. You can create many clones at
once, instead of just one (as in vSphere native cloning).

Return back to the vSphere Client to verify that 10 Tiny VM clones have been created.
The batch job will produce a series of VMs with the chosen prefix name. The VM
cloning process is accelerated by the HyperFlex Data Platform, so the VMs are created
very quickly. The speed of cloning lends itself naturally to batch VM creation jobs, such
as rapid provisioning of a new VDI, testing, or PoC virtual machines.

Step 31
Migrate a HyperFlex-based clone to non-HyperFlex storage.

Answer
Right-click the HX-Clone-10 VM, and choose Migrate... from the Actions menu.
In the Select a migration type step, check the Change storage only check box.

Click Next.

In the Select storage step, choose SpringpathDS as the target. The suffix of the name
can be different.
Click Next.

In the Ready to complete step, review the configuration that you entered, then
click Finish to complete the migration.

Migration should complete normally, because native clones do not depend on the
HyperFlex storage after creation and can be copied to non-HyperFlex storage.
Step 32
Compare vSphere snapshots and HyperFlex snapshots.
Answer
In the vSphere client, right-click the vSphere Tiny Clone, and choose Snapshots >
Take Snapshot
Name the snapshot vSphere Snapshot.

Click OK to create the vSphere snapshot.

Click vSphere Tiny Clone again, and choose Snapshots.

The snapshot that you created is now in the snapshot timeline, and immediately after
that is the current state of VM in time, which is a delta file on the datastore. The delta
file contains all the changes since the previous snapshot was created.

Switch to the HyperFlex Connect tab, and check the check box near the Tiny Linux
VM. Choose Snapshot now.

Name the snapshot HyperFlex Snapshot, and click the Snapshot Now button to create
the HyperFlex native snapshot.

Return back to the vSphere Client, click the Tiny Linux VM in the Navigator menu,
and choose Snapshots tab on the right. If snapshots are not visible, choose another VM,
and click Tiny Linux VM again.
The HyperFlex snapshotting process created a sentinel snapshot as it created the
HyperFlex snapshot. The sentinel snapshot enables native snapshotting functionalities
on a VM. Do not revert to or delete the sentinel snapshot if any other snapshots exist,
since they are reliant on the sentinel snapshot.

The sentinel snapshot ensures that all subsequent snapshots are native snapshots.
Sentinel snapshots prevent reverted VMs from requiring a VMware redo of log-based
virtual disks. Redo log-based virtual disks occurs when an original snapshot is deleted
and the VM is reverted to the second oldest snapshot. Sentinel snapshots exist in
addition to the revertible native snapshot. The sentinel snapshot consumes one snapshot
of the total 31 that are available, per VMware limitation.

Note
If this VM had a pre-existing vSphere native snapshot, the HyperFlex snapshots would
not function, because the sentinel snapshot must be the first snapshot for a VM.

Step 33
Migrate a VM with HyperFlex native snapshot to non-HyperFlex storage.

Answer
Right-click the Tiny Linux VM, and choose Migrate... from the menu.
In the Select the Migration Type step, choose the Change storage only check box,
and click Next.

In the Select storage step, choose SpringpathDS as the target (suffixes can be
different).
Note the compatibility warning: Cannot move a VM with native delta disk backing.
Unlike native clones, VMs with HyperFlex native snapshots cannot be migrated off the
HyperFlex datastore.

You can only move the VM with a native snapshot off the HyperFlex datastore after
you consolidate the snapshots, and no HyperFlex native snapshots are present.

Click Cancel to cancel the job.

Step 34
Schedule Cisco HyperFlex native snapshots.

Answer
To create a snapshot schedule for Cisco HyperFlex snapshots, check the check box near
the Tiny Linux VM, and choose the Schedule Snapshot option.
Choose Daily Snapshot to enable a daily snapshot policy. Choose Monday, Tuesday,
Wednesday, and Thursday and specify snapshot start time at 9 PM. No more than 5
snapshots should be kept in the system (retention policy).

Click Save to continue.

You have successfully scheduled a snapshotting process that will be performed

automatically every day at 9 PM.
7.1 Maintaining Cisco HyperFlex

Introduction
To extend the functionality of Cisco HyperFlex and improve its security and
performance, you should keep it updated to the recommended release. Cisco Intersight
made upgrading the full Cisco HyperFlex stack easier than ever.

Not skipping upgrades to the recommended HyperFlex versions is highly

recommended, since upgrading from older systems to newer ones becomes more
difficult the greater the version jump.

7.2 Maintaining Cisco HyperFlex

Cisco HyperFlex Upgrade Overview

This video provides an overview for upgrading Cisco HyperFlex. You should keep
Cisco HyperFlex updated to the recommended software release in order to extend
functionality and improve performance.
This figure here lists the three software component groups that you would upgrade in a
standard ESXi-based cluster. This includes the Cisco HyperFlex data platform, vSphere,
both ESXi and vCenter, and Cisco UCS Manager.

This includes upgrading firmware on both the blade and the rack-mounted servers. You
can do a combined installation for all three software components, or you can separate
them into different upgrade tasks. The combined upgrade can take a very long time, and
it could even timeout.

Upgrading the cluster can be performed while it is online or offline. Now, if you choose
the online procedure, guest applications, and guest and application VMs may stay
online. And support is upgraded in what we call a rolling fashion, or one device at a
time.

If you choose the offline procedure, then you have to shut down HyperFlex cluster and
perform the server upgrades in parallel. Obviously, the online upgrade takes
considerably longer to complete. In a three-node cluster, it would be three times the
amount of time to complete because there is a 120-minute timeout per node. I mean, if
the upgrade fails, you can always try it again.

Finally, this figure here lists critical considerations when upgrading Cisco HyperFlex.
You must upgrade vCenter to version 6.0 before upgrading HyperFlex to version 3.5.
You can upgrade HyperFlex version 2.1 directly to 3.5.

For older versions of HyperFlex, you must first upgrade to 2.0. If you're upgrading
HyperFlex from version 2.5-1a or older, then you use the CLI, or you could use the
vSphere web client.

And lastly, for vSphere 5.5, upgrade vSphere to at least version 6.0 before you upgrade
HyperFlex. And that brings us to the end of the video. Thanks for watching

When you are upgrading Cisco HyperFlex, you need to consider the interoperability of
the source and target system and any bugs and limitations of different versions. The
considerations grow with additional new releases; therefore, it is highly recommended
to keep your Cisco HyperFlex systems current with recommended releases.

With older Cisco HyperFlex releases (before Release 3.0), the compatibility
considerations are quite complex and are listed here for reference. As new issues are
discovered, these considerations can change. Make sure to read the upgrade guide and
interoperability considerations before attempting an upgrade.

You would upgrade three software component groups in a standard ESXi-based

deployment:

 Cisco HXDP, vSphere (ESXi and vCenter), and Cisco UCS (UCS Manager, and
blade and rack server firmware).
 You can perform a combined installation of Cisco HXDP, ESXi, and Cisco UCS on
servers.
 o Or separate each component into separate upgrade flow.
o The challenge of a combined upgrade is that the upgrade of an individual server
can take a very long time and even time out.

Your HyperFlex system can be either online or offline during the upgrade:

 Online: Guest/application VMs stay online and are migrated to the remaining nodes
before all systems of a node are upgraded.
 Offline: You shut down the Cisco HXDP system and perform the upgrade on all
servers in parallel.
 The online procedure takes much longer than the offline procedure; however, in
nearly all upgrades, the online procedure is used.
 Cisco Intersight full-stack upgrade supports only the online upgrade option.
Note
The time-out timer for a node upgrade is 120 minutes for each node. If the upgrade fails,
you can re-try. This timer can easily be exceeded when there is a major version jump
performed by the upgrade.

Upgrade Considerations

Remember these critical considerations when upgrading Cisco HyperFlex:

 Required vCenter upgrade: Cisco HXDP 3.5(1a) and later requires TLS 1.2. You
must upgrade vCenter to 6.0 U3f or later to get this feature.
 Minimum Cisco HXDP:

o On 1.8(1e) or earlier, upgrade to Version 2.6(1e) first.

o On 1.8(1f) and 2.0(1x), upgrade to Version 3.0(1i) first.
o On 2.1(1a) or later, upgrade to 3.5(2x) or 4.0(2d) directly.
o On 2.6(1a) or later you can upgrade directly to 4.5(1a).

 Upgrade client: As of HXDP 4.5(1a), upgrade your systems using HX Connect

user interface. ESXi and UCS can be upgraded from HX Connect if the version is
3.5(1a) or later.
 vSphere 5.5 upgrades: Upgrade to 6.0 U3 or 6.5 U1 before upgrading HXDP.
 vSphere 6.0 upgrades: In version 4.5(1a) and later you need to upgrade to vSphere
6.5, since 6.0 is no longer supported.
Note
It is generally best to upgrade HXDP and ESXi together if combining the upgrade
operations into a single optimized reboot. When running a split upgrade, first upgrade
the HXDP, then proceed to upgrade ESXi.

vSphere v5.5 support was deprecated with Cisco HXDP v2.5(1a), and the upgrade will
fail if you attempt it.

 For Cisco HX220c M4 users running v5.5, contact Cisco TAC for upgrade
assistance.

o Upgrade vCenter to v6.0 U3f or later. If upgrading to v6.5, you must upgrade
your vCenter in place. Using a new vCenter v6.5 is not supported for users
migrating from v5.5.
o Upgrade ESXi to v6.0/6.5 using the offline zip bundle.

 During the upgrade, it might be necessary for you to reconnect ESXi host manually
in vCenter after ESXi upgrade and host reboot.

o Upgrade Cisco HXDP (and, optionally, the Cisco UCS firmware).

 Certain HyperFlex functions such as native and scheduled snapshots, ReadyClones,

and Enter/Exit HX maintenance mode will not operate from the time the upgrade is
started until the Cisco HXDP upgrade to 3.5(1a) or later is complete.
 After upgrading ESXi using the offline zip bundle, use the ESX Exit Maintenance
Mode option. The HX Exit Maintenance Mode option does not operate in the
vSphere Web Client until the Cisco HXDP upgrade is complete.

Firefox browser is not supported for upgrades with Flash plug-in, due to an outdated
version of Flash that is bundled with the browser. Manual update of Flash within
Firefox is possible, but the recommendation is to use either Chrome or Internet Explorer
with a modern version of Flash.

For details on upgrade paths of Cisco HXDP and Cisco UCS software, refer to Cisco
HyperFlex Systems Upgrade Guide for VMware ESXi at
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/HyperFlex_upgrade_guide/4-5/b-HX-Upgrade-Guide-for-VMware-
ESXi-4-5.html.

Note
Please do not blindly attempt upgrades from versions below 3.5(1a) without fully
understanding the requirements. Please consult Cisco TAC if unsure about your upgrade
from any version.

7.3 Maintaining Cisco HyperFlex

Cisco HyperFlex Online Upgrade

Let's examine the online upgrade for a Cisco HyperFlex. This first figure here lists the steps
required to perform the Cisco HyperFlex upgrade online. This means that with guest VMs that
are running during the upgrade process. The upgrades can be performed through vCenter, SD
CLI or the REST API. However, we're going to examine the upgrade through a HX Connect. This
is the recommended method for Cisco UCS and HyperFlex version 3.0 and newer and also ESXi
3.5 and later.
Instead of going through each one of these steps, let's look at them in more detail. This figure
lists the prerequisites for upgrading the Cisco HyperFlex system. You should make sure that
you are fulfilling all of the recommended prerequisites and caveats before you actually start
the upgrade. Prerequisites are a combination of compatibility checks, backing up the existing
configuration, and performing health checks. This includes backing up all of the configurations.
The system configurations can be exported to an XML file, which includes all of the system and
logical configuration settings. You should also consult the Cisco HyperFlex System upgrade
guide for VMware ESXi to determine compatibility of Cisco HyperFlex, Cisco UCS, and the
vSphere software bundles. Lastly, make sure the cluster has the access policy set to the Lenient
mode, which is the default setting.

To perform an upgrade of the entire Cisco HyperFlex system, you need the Cisco HyperFlex Up
grade Bundle, which is in a .TGZ file. The Cisco UCS Infrastructure bundle along with a blade
firmware software and the rack firmware bundle. And lastly, the VMware ESXi software bundle
, which is downloaded in a ZIP format.
You should check the release notes or upgrade guides for recommended versions and
compatibility of the software components. Next, you would go to Cisco UCS Manager and
navigate to Equipment, Firmware, Install Firmware, and download the firmware. Download the
bundles onto the fabric interconnect one at a time. You should see all three bundles under the
Packages tab.

The firmware includes Cisco UCS Manager, switch firmware, blade chassis, IOM firmware, and
the Rack Server FEX firmware. To upgrade the Cisco UCS firmware of your HyperFlex servers,
you will first need to upgrade to Cisco UCS infrastructure. To do this without any disruption of
the workloads, you would navigate to Equipment, Firmware Management, Firmware Auto-
Install, and click Install Infrastructure Firmware. Choose the desired service pack and click
Upgrade Now. And then click Finish.
The Cisco UCS Manager Software may be inaccessible for a little while while it's upgraded.
After you log back in, you'll have to wait for the IOM to be upgraded and the subordinate
fabric interconnect to be activated. Click Pending Activities and click the Fabric Interconnects
tab. This displays the tasks requiring user acknowledgment. And then lastly, click Reboot Now
and OK to confirm.

Wait for the second fabric interconnect to reboot and for the IOM upgrades to complete.
During fabric interconnects reboot, all HyperFlex traffic will be now forwarded to the new
primary fabric interconnect. The reboot will cause a brief traffic interruption. However, it will
not cause any storage I/O failures. When it's done, now you're ready to test the upstream
network connectivity.
First, you test connectivity between the storage IPs of the two node switches locally via the
fabric interconnects. Put one of the nodes in HyperFlex Maintenance mode. Then SSH to the
ESXi host and ping the IP address of another host. The figure illustrates the commands for
pinging with and without the use of Jumbo frames.

Next, test the connectivity between the storage IPs of the two nodes to see if the traffic is
forced over the upstream switch. This figure here describes how to first swap the active
interfaces to force traffic over the upstream switch. Then use ping to verify if connectivity is
successful. And then lastly, swap the interface back to the default and exit out of the
Maintenance mode.
This figure describes the bootstrap process. The manual bootstrapping process is only
necessary if you're upgrading from a version earlier than 3.5.1a. For example, if you're going
from version 3.0.1e to version 3.5.1a, now that requires a manual reboot. Cisco HyperFlex
allows you to perform native or storage-level snapshots of the VMs. This figure here illustrates
how to disable the snapshot schedule on the bootstrap storage CVM or optionally pause native
HyperFlex replication if it's being used.

Now you're ready to upgrade the Cisco UCS firmware of the servers, Cisco HyperFlex and ESXi.
First, you log into HyperFlex Connect and navigate to the Upgrade section. This figure here lists
some of the caveats and Cisco recommendations that you should check for the upgrade.
When you're upgrading your HyperFlex cluster, you can upgrade Cisco HyperFlex, Cisco UCS,
and the ESXi software.
You will actually have three checkboxes to choose from. It'll be UCS Server Firmware, HX Data
Platform, or ESXi. After you choose the desired procedures, you select the components that
you wish to upgrade, and when you're ready, then you click Upgrade. HyperFlex will upgrade
servers one by one, placing the server into the Maintenance mode while the upgrade process
takes place. You can actually observe the process in the Upgrade or Activity section. If DRS is
enabled, the VMs are automatically vMotioned to the other hosts. If not, then you will have to
manually continue.
After HyperFlex Connect declares that the upgrade is successful, you should check the
parameters listed in the figure. This includes making sure the cluster is online and healthy, all
of the nodes are running the same version of the software, and restarting the snapshot and
replication schedules if you had previously stopped them.
Finally, confirm that all of the data stores are mounted on the ESXi host. This can be done in
HyperFlex Connect by navigating to Datastores. Then you choose the datastore, and the mount
status should be listed as Mounted for all of the servers. That completes this video. Thanks for
watching.

Online upgrades are the main way HyperFlex systems are upgraded, since the workload
remains unaffected. The prerequisite checks are the same between the online and offline
upgrades.

For HyperFlex systems installed through Cisco Intersight, the online upgrade is always
available directly through the Intersight interface, making upgrades much easier than
ever before.

Note
Please note that as of HyperFlex version 4.5(1a) only HytperFlex systems that were
initially installed from Intersight can be upgraded through the full-stack upgrade process
available in Intersight.

Actions when performing online upgrade through Installer VM:

1. Carefully check all prerequisites.

2. Download HXDP upgrade package, Cisco UCS firmware, and ESXi upgrade
package.
3. Run the HyperCheck script to perform connectivity and configuration tests.
4. Upgrade Cisco UCS Infrastructure if required.
5. Bootstrap all nodes to upgrade Cisco HXDP before 3.5(1a).
6. Disable backups, snapshot schedule, and pause replication.
7. Start upgrade of Cisco UCS, HXDP, and ESXi from HyperFlex Connect.

o One, two, or all of them (correct order with different combinations applies).

8. Confirm that the upgrade task is complete.

9. Enable snapshot schedule and resume replication.
Compared to the Installer VM upgrades, the Intersight upgrade is much simpler and
already includes Hyperchecks and some preinstall verifications.

Actions when performing an upgrade through Intersight:

 Carefully check all prerequisites.

 Run the hypercheck from Intersight > HyperFlex Clusters > ... > HyperCheck.
 Run the upgrade from Intersight > HyperFlex Clusters > ... > Upgrade.
 Through standard or expert mode, choose the versions for HXDP, UCS Firmware,
and ESXi.
 Start the compatibility checks then start the upgrade.

Note
The key steps of the Intersight upgrade are the same as with
the Installer VM upgrades, but a lot of actions are simplified. As
of Version 4.5(1a), upgrades performed through the Installer VM
are more common and are covered in full.

Prerequisites

Before you begin an upgrade of your Cisco HyperFlex system, it is important that you
make sure that you are fulfilling all prerequisites. Prerequisites are a combination of
compatibility checks, backing up the existing configuration, and health checks.
Perform the following before you start an upgrade:

1. See resolved and open caveats before upgrading and review the new features for a
given release. Refer to the latest Cisco HXDP Release Notes.
2. Review supported versions and system requirements. Make sure that you meet
hardware and software requirements as specified in the Upgrade Guide.
3. Make sure that HXDP, UCS, and vSphere software target versions are compatible.
4. Create a Full State and All Configuration backups in Cisco UCS Manager.

5. Verify that SSH is enabled on all hosts and is configured to be started on boot.
6. Make sure vMotion is fully functional. Migrate a VM to all nodes to check.
7. Verify that DRS is set to fully automated.

o If DRS is not enabled, you must manually migrate VMs during the upgrade.

8. Make sure that HyperFlex has the access policy set to lenient mode (default).

admin@CVM1:~# stcli cluster get-cluster-access-policy

lenient

9. Right before starting, verify that the system is healthy by checking the status in HX
Connect.
All configurations can be exported into an .xml file that includes all system and logical
configuration settings. You can use the file generated from this backup to import the
configuration settings to the original Cisco Fabric Interconnect or to a different Fabric
Interconnect. You cannot use this file for a system restore. This file does not include
passwords for locally authenticated users. To backup configuration on Cisco UCS
Manager, navigate to Admin > All > General > Backup Configuration. For a detailed
procedure, refer to the Cisco UCS Manager Administration Management
Guide at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5.html

Consult the Cisco HyperFlex Systems Upgrade Guide for VMware ESXi
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/HyperFlex_upgrade_guide/4-5/b-HX-Upgrade-Guide-for-VMware-
ESXi-4-5.html) to determine compatibility of Cisco HXDP, Cisco UCS, and vSphere
software bundles.

Before you start an upgrade, follow these steps in Cisco HXDP Release
Notes: https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-
data-platform-software/products-release-notes-list.html.
If you have ESXi lockdown mode enabled, you will need to disable it for the upgrade.
Enable lockdown after you are done with the upgrade.

If vMotion does not work, it was either not properly set up during the post-installation
procedure or it was broken after the procedure. Confirm the following parameters from
your vSphere Web Client:

 Verify that the vMotion port group is in an active/standby configuration across all
ESXi hosts in the system.
 Verify that a port group is configured for vMotion, and the naming convention is the
same across all ESXi hosts in the cluster. The name is case-sensitive.
 Verify that you have assigned a static IP to each vMotion port group, and the static
IPs for each vMotion port group are in the same subnet.
 Verify that the vMotion port group has the vMotion option checked in the
properties, and no other port groups (such as management) have this option checked
(on each ESXi host in the group).
 Verify in the settings that the vMotion port group is set to 9000 MTU (if you are
using jumbo frames), and that the VLAN ID matches the network configuration for
the vMotion subnet.
 Verify you can ping from the vMotion port group on one ESXi host to the vMotion
IP on the other host.

vmkping -I vmk2 -d -s 8972 <vMotion IP address of

neighboring server>

To manually set the HyperFlex access policy to lenient, use this procedure.

 SSH to any one of the controller VMs and log in as root.

 Check if lenient mode is already configured. Lenient mode has been the default
policy mode since Cisco HXDP v1.8.
 If set to strict, change to lenient. If already set to lenient, no further action is
required.

stcli cluster set-cluster-access-policy --name lenient

 Verify the change.

Note
Online procedure upgrades servers one by one. During an upgrade, the health of the
system is degraded due to one of the servers not being available. You need the access
policy to support that workflow.

To verify that DRS is enabled and set to fully automated, follow these steps:

 From the vSphere Web Client Navigator, choose vCenter Inventory Lists >
Clusters > choose your cluster > Configure tab. Verify that DRS is Enabled.
 Click the vSphere DRS tab. Check if Migration Automation Level is set to Fully
Automated.
Hypercheck Health Check Utility

As of HXDP 4.0(1a), Cisco provides a command-line tool called HyperFlex

Hypercheck. It is a health check utility script that automatizes most pre-upgrade checks.

As of HyperFlex Hypercheck 4.0, the tool is supported to run on the following

HyperFlex systems:

 Systems on HXDP versions 1.8, 2.0, 2.1, 2.5, 2.6, 3.0, 3.5, 4.0 or 4.5.
 All deployment types.
 Only ESXi-based deployments.

You should run Hypercheck before an upgrade:

 Download it from GitHub and upload it to the CVM with group management IP in
the /tmp directory.
 Extract the compressed file in the /tmp and run the tool.

o You will be prompted to enter credentials for HXDP and ESXi access for the
script.

 The tool will make a series of checks on HyperFlex level (such as system services
and network checks) and on ESXi level (such as vMotion-enabled check).

o The tool will deposit the report in the HyperFlex-Hypercheck directory.

 If the tool runs OK and all tests show PASS, then the system is good for all the
checks that the script performed.
You can download the tool from the HyperFlex-Hypercheck GitHub site
at https://github.com/CiscoDevNet/Hyperflex-Hypercheck. The tool is constantly being
improved, so make sure to download the latest version to run.

To follow detailed steps on how to use this tool, read the Hypercheck : Hyperflex
Health & Pre-Upgrade Check Tool guide at
https://www.cisco.com/c/en/us/support/docs/hyperconverged-infrastructure/hyperflex-
hx-data-platform/214101-hypercheck-hyperflex-health-pre-upgr.html. The guide also
lists all the checks that the script performs.

Test Upstream Network Connectivity

Connectivity tests are the first troubleshooting steps in most failures on Cisco
HyperFlex. Although the connectivity might be fully working on installations, changes
on the network later on may cause connectivity issues that are not visible until an
upgrade is attempted.

When performing connectivity tests, be sure to know which interface you are pinging
from and which interface you are pinging. There are many interfaces involved in
HyperFlex operation on several systems. The Data network is always the most
important one.

Testing connectivity for a network (VLAN) involves these steps:

 Executing a ping between IPs of two hosts or two systems (ESXi/CVM).

 Forcing storage traffic over the upstream switch by failing over vNICs. Again,
executing a ping between IPs of the two hosts.

These two steps confirm that you have full connectivity and that MTUs are consistent
between Cisco HyperFlex system and the upstream network.

You can manually test the connectivity of your HyperFlex installation:

 Verify that the ping is working by pinging the corresponding vmk1 IP interface of
another host.

 If using Jumbo Frames:

vmkping -I vmk1 -d -s 8972 <data IP of address of another

host>

o If not using Jumbo Frames:

vmkping -I vmk1 -d -s 1472 <data IP of address of another

host>

 Swap the active interfaces in vswitch-hx-storage-data while the node is in

maintenance mode to force traffic over the upstream switch.

esxcli network vswitch standard policy failover set -a

vmnic2 -s vmnic3 -v vswitch-hx-storage-data

 Do not forget to swap the interfaces back before exiting the maintenance mode or
the traffic will keep flowing through the upstream switches.

The normal connectivity test is performed with the Hypercheck script, but the inverted
interface check is not. Usually, you would only perform the normal check, unless there
is good reason to test upstream switches.

If the ping with jumbo MTU is not successful, but the ping with regular MTU is, you
have probably set up HyperFlex with regular size MTU.

If you have had a successful connectivity test between nodes previously and now you do
not, here are some of the possible issues:

 Storage VLAN (HyperFlex data) is not defined on the upstream switch.

 Storage VLAN (HyperFlex data) is not allowed on downlinks (towards Fabric
Interconnects) on the upstream switch.
 Upstream switch does not have proper configuration of MTU. If HyperFlex was
installed with jumbo MTU for storage VLAN, this VLAN also needs to have jumbo
MTU configured on the upstream switch.

Download Software

To perform an upgrade of the entire HyperFlex system, you need the following files:

1. Cisco HXDP upgrade bundle (.tgz file)

2. Cisco UCS infrastructure (A) bundle, blade firmware (B) bundle, and rack firmware
(C) bundle
3. VMware ESXi offline bundle (.zip file)

Note
Software versions listed in the images are only examples.
Always check the release notes or upgrade guide for
recommended version and compatibility of software
components.

Log in to Cisco UCS Manager:

1. Navigate to Equipment > Firmware Management > Installed Firmware >

Download Firmware.
2. Download A, B, and C bundles onto Fabric Interconnects, one at a time.
3. When you are done, all three new bundles should appear under the Packages tab.

If Cisco UCS Manager reports that the bootflash is out of space, delete the obsolete
bundles in the Packages tab to free up space. To view the available space in bootflash,
navigate to Equipment > Fabric Interconnect A (or B) > Local Storage
Information, and look in the Work pane area under the General tab.

Upgrade Cisco UCS Infrastructure

The Cisco UCS infrastructure firmware includes Cisco UCS Manager, switch firmware,
blade chassis IOM firmware, and rack server FEX firmware.

If you want to upgrade the Cisco UCS firmware of your Cisco HyperFlex servers, you
will first need to upgrade the Cisco UCS infrastructure. The Cisco UCS infrastructure is
not updated automatically during the HyperFlex upgrade process.

Upgrade the Cisco UCS infrastructure without any disruption to HyperFlex workloads:

1. Choose Equipment > Firmware Management > Firmware auto-install.

Click Install Infrastructure Firmware.
2. Choose the desired service pack. Click the Upgrade Now box to immediately begin
the firmware upgrade. Click Finish.

o If you see warnings, fix them before proceeding.

3. Check the Upgrade Now box.

Cisco HyperFlex nodes are configured to fail over all Ethernet traffic when a Cisco
UCS Fabric Interconnect is rebooted to upgrade the firmware.

Again, ensure that the HyperFlex storage data and vMotion upstream switches are
configured for Jumbo Frames before proceeding forward; otherwise, the HyperFlex
storage will become offline and all datastores will unmount from the ESXi hosts.

Cisco UCS Manager software upgrades first, so it may be inaccessible for some time;
then log in again.

4. Wait for IOM to be upgraded and subordinate Fabric Interconnect to be activated.

5. During Fabric Interconnect reboot, all HyperFlex traffic will be forwarded to
primary Fabric Interconnect.
6. Click Pending Activities. Click the Fabric Interconnects tab to see the tasks that
require user acknowledgment before they can complete.

o Click Reboot Now and confirm with OK.

Also wait for IOM to be upgraded only if you have a blade chassis connected to the
Fabric Interconnects where you are performing the upgrade.

During the upgrade, HyperFlex traffic will be re-pinned to both Fabric Interconnects.
Wait for Cisco UCS Manager vNIC faults to be cleared. The fault clearing indicates
ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned
immediately when the network interface goes up because ESXi has a fail back timer.
But the failback timer is very low by default (100 ms).

During the Fabric Interconnect reboot, all HyperFlex traffic will be forwarded to the
new primary Fabric Interconnect. The reboot will cause a brief traffic interruption.
However, it will not cause storage I/O failures.

Wait for the second Fabric Interconnect to reboot, then wait for the IOM upgrade.

7. Wait until HyperFlex traffic is re-pinned to both Fabric Interconnects.

A. In the Cisco UCS Manager GUI, wait until all server vNIC faults have been
cleared.
B. Verify that the HyperFlex system is online and healthy.-

8. At this point, infrastructure firmware is complete. You will use HX Connect to

upgrade the server firmware.
You can monitor the status of the Fabric Interconnect upgrade under Equipment >
Installed Firmware > Fabric Interconnects.

You can monitor the status of the IOM upgrade under Equipment > Blade Chassis >
IO Module.

Note
IOMs will be in a Pending Next Boot for Activate Status once the update process
completes. When the IOM upgrade has completed, the Update Status of the IOM is set
to Ready.

For Fabric Interconnects, check the Activate Status of the kernel and switch images.
During the upgrade, the Activate Status is set to Activating.

Bootstrap Process

The manual bootstrapping process, as described here, is only necessary if upgrading

from a version earlier than 3.5(1a). So, if you are upgrading, for example, from Version
3.0(1e) to 3.5(1d), manual bootstrap is necessary. If you are upgrading, for example,
from Version 3.5(1a) to 4.0(1a), bootstrapping is performed by the upgrade wizard.

Since the automatic bootstrap process also upgrades the node where you initiate the
bootstrap, the procedure will log you out of the HyperFlex Connect. This may cause an
incorrect interface to be displayed until the page is refreshed.

Bootstrap lets you upgrade Cisco HXDP and the HXDP plug-in in vCenter from
versions earlier than 3.5(1a).

1. SSH to the group management IP address.

2. Transfer the latest Cisco HXDP upgrade bundle to the controller
VM’s /tmp directory.

o You can use Secure Copy Protocol (SCP) directly from your computer or use a
third-party tool such as WinSCP.

3. Uncompress the upgrade package.

o Example: tar –zxvf storfs-packages-3.5.1a-31462.tgz

4. Invoke the cluster-bootstrap.sh script to bootstrap the packages for upgrade.

./cluster-bootstrap.sh

Note
Do not perform bootstrap from other CVM than the one being the owner of the
management VIP. You should know the group management IP address; it is the same IP
that you use to connect to HyperFlex Connect.

CVMs should have enough free space to accommodate the upgrade package. If they do
not, contact Cisco TAC for assistance. This issue will usually be caught by the
HyperCheck script.

Disable Snapshot Schedule and Pause Replication

Cisco HyperFlex allows you to perform native (storage-level) snapshots of VMs. These
jobs take time to create the backup. If a job is underway during the upgrade, it will
interfere with VM vMotion and the upgrade itself.

The same applies to snapshot schedules and third-party replication jobs of any kind.
These third-party systems cannot be disabled from the HyperFlex interface, so you must
disable them manually through their management interfaces.

Disable the snapshot schedule on the bootstrapped storage CVM:

 stcli snapshot-schedule --disable
 It is enough to run this script on this one CVM.
 If you do not have scheduled native snapshots, you can skip this step.

Pause native HyperFlex replication:

 stcli dp schedule pause

 It is enough to run this script on this one CVM.
 If you are not using HyperFlex replication, you can skip this step.

Disable backup jobs or automation jobs that target the HyperFlex system.

 These interactions interfere with the upgrade process and can cause upgrade failures.
After you are done with the upgrade, you will enable snapshots schedules, native
replication, and external backups again. Make sure you do not forget these steps once
you complete the upgrade.

Perform the Upgrade

On newer Cisco HyperFlex versions, you can upgrade your Cisco HyperFlex system
from HyperFlex Connect. This will apply to nearly all users. If you are upgrading from
an older version (HXDP 3.0 or lower) by yourself, please consult Cisco TAC before
attempting the upgrade.

Now you can start the upgrades of Cisco UCS firmware of servers, Cisco HXDP, and
ESXi.

1. Log in to HyperFlex Connect, and navigate to Upgrade section, check the boxes of
systems you want to upgrade.
2. Cisco recommends the following upgrade combinations:

o HXDP only upgrade, followed by Cisco UCS Firmware and/or ESXi

o HXDP and Cisco UCS Firmware
o HXDP and ESXi
o HXDP, Cisco UCS Firmware, and ESXi

3. Note that if you are upgrading ESXi, you need to upgrade vCenter before upgrading
ESXi on servers.
The vCenter version should always be equal or higher than the version of ESXi on
servers that it manages. For guidelines on how to upgrade vCenter, consult with
VMware documentation.

If you are upgrading components one-by-one, it is recommended that you use the
following order for optimizing the upgrade time:

1. Upgrade Cisco UCS Infrastructure.

2. Upgrade Cisco HX Data Platform.
3. Upgrade Cisco customized VMware ESXi.
4. Upgrade Cisco UCS firmware.

To prepare for the upgrade of Cisco UCS firmware on your servers, follow these steps:

1. Choose the UCS Server Firmware check box, and fill-in IP/FQDN and the
credentials of Cisco UCS Manager.
2. Click Discover. HyperFlex Connect should retrieve the current Cisco UCS version
and a list for the desired (target) version. Choose the target version.
3. You previously upgraded the Cisco UCS infrastructure and staged server firmware.
The upgrade wizard will upgrade the firmware of the servers.

To prepare for the upgrade of Cisco HXDP firmware on your servers, follow these
steps:
1. Choose the HX Data Platform check box, and fill-in vCenter credentials that are
needed for the upgrade process of Cisco HXDP.
2. Drag-and-drop the Cisco HXDP upgrade bundle so it uploaded to the CVMs.

To prepare for the upgrade of ESXi firmware on your servers, follow these steps:

1. Choose the ESXi check box, and fill-in the vCenter credentials that are needed for
the upgrade process of ESXi.
2. Drag and drop the ESXi upgrade bundle so it is uploaded to the ESXi servers.

Cisco HyperFlex upgrades servers one by one:

 Upgrade process will enter maintenance mode for a server, upgrade that server, exit
maintenance mode, wait for the system to be healthy, and finally continue with the
next node.
 Observe the progress of the upgrade either in the Upgrade section of HX Connect
or in the Activity section (more detailed).
 If DRS is enabled, the VMs are automatically vMotioned to other hosts, otherwise
you need to do it manually.
 If any troubleshooting is necessary, make sure you check the step the process failed
in.

When the upgrade is in progress, you may see an error message saying "Websocket
connection failed. Automatic refresh disabled." You can either refresh the page or log
out and log back in to clear the error message. You can safely ignore this error message.

Unless you have a HyperFlex system of 5 or more nodes and RF3, you cannot afford a
server failure from the nodes that are in the group besides the upgrading node. Cisco
HyperFlex will shut down in that case. This is why choosing the replication factor 3 is
the recommended option.

Confirm That Upgrade Is Complete

After HyperFlex Connect declares the upgrade successful, check these parameters:

 System is online and healthy.

o Dashboard of HX Connect

 All nodes in the system are running the same version of HXDP, Cisco UCS, and
ESXi.

1. HXDP: stcli cluster version from the web CLI section in HX Connect.
2. UCS: In Cisco UCS Manager, navigate to Equipment > Firmware Management >
Installed Firmware tab, and verify for the correct firmware version.
3. ESXi: In vSphere Web Client, click on individual host, and navigate to Summary >
Configuration > ESX/ESXi Version.

 Restart snapshot and replication schedules if you have previously stopped them.

stcli snapshot-schedule --enable

stcli dp schedule resume

 All datastores are mounted on the ESXi hosts.

o In HX Connect, navigate to Datastores, and choose your datastore. Mount

Status should be listed as MOUNTED for all servers.

Note
You can also verify that the upgrade is complete and successful using the stcli cluster
upgrade-status command.

For each browser interface you use, empty the cache and reload the browser page to
refresh the HyperFlex Connect content.
7.4 Maintaining Cisco HyperFlex

Cisco HyperFlex Offline Upgrade

This video describes the offline upgrade process. The offline upgrade workflow is very
similar to the on-grade workflow, but there's obviously some differences. This figure
lists the steps for an offline upgrade. Just like the online upgrade, you check your
prerequisites, download the packages, test connectivity, and upgrade Cisco UCS, if
required. However, with the offline, you need to gracefully shut down the Cisco
HyperFlex cluster before you actually do the upgrade.

First, you launch the vSphere Web Client and power down all of the virtual machines.
This includes the user VMs on the Cisco HyperFlex Server, the HyperFlex data stores,
and the compute-only nodes. After all of the VMs are shut down, check the cluster
health using the command stcli cluster info pipe grep health. This is followed by the
command stcli cluster shut down, which is going to take a few minutes to complete.
Once you shut down your cluster, Cisco HyperFlex is not accessible anymore. However
, Cisco UCS Manager and ESXi Server will continue to be accessible. This figure here
lists the steps for performing a Cisco UCS host firmware upgrade, if it's required, before
you actually start the upgrade for HyperFlex. This is pretty much the same as we
described in the online upgrade process. It's not much different. This includes manually
staging the correct Cisco UCS software version.
Next, from the vCenter, right-click each HyperFlex Controller VM and choose Power,
Shut Down Guest OS. Then right-click each ESXi host and choose Maintenance mode,
Enter Maintenance Mode. Wait till all the nodes are upgraded, and then confirm that the
correct firmware packages are installed. And then lastly, in vCenter, right click on each
ESXi host and choose Maintenance Mode, Exit Maintenance Mode. The HyperFlex
Controller VMs will automatically come back online. That concludes this video. Thanks
for watching.

Offline upgrade workflow is very similar to the online upgrade workflow; however, it
allows simultaneous upgrade of all the components. The downside is that the workload
is shut down during the upgrade.

In almost all cases, customers choose the online upgrade, even though it takes much
longer. Since there is no interruption, it is easier to schedule even in critical systems.

Note
This guide presumes that you are familiar with the online upgrade at this point.

Offline upgrade workflow is different than online:

1. As with online upgrade, check the prerequisites, download the required packages
(HXDP, UCS, ESXi), test connectivity, and upgrade Cisco UCS infrastructure (if
needed).
2. Gracefully shut down the Cisco HyperFlex system.
3. Upgrade Cisco UCS host firmware.
4. Disable snapshot and replication schedules, bootstrap the CVM, perform upgrades
from HX Connect, and verify the completed upgrade (the same as the online
upgrade).
5. After Cisco HXDP upgrade is complete, start the system and power on VMs.
6. Enable snapshot and replication schedules (the same as online upgrade).

Note
Like with the online upgrade procedure, it is recommended that you use the HyperFlex
Hypercheck tool to speed up the prerequisite upgrade check.
With an offline upgrade, the same considerations apply to combinations of upgrading as
they do with an online upgrade. To be successful, Cisco recommends that you choose
these upgrade types:

 Cisco HXDP only upgrade, followed by Cisco UCS Firmware and/or ESXi
 Cisco HXDP and Cisco UCS Firmware
 Cisco HXDP and ESXi
 Cisco HXDP, Cisco UCS Firmware, and ESXi

Cisco HXDP and Cisco UCS firmware upgrades can be performed from HX Connect
since Cisco HXDP v3.0(1a). ESXi upgrades can be performed from HyperFlex Connect
since HXDP 3.5(1a).

Graceful Shutdown of Cisco HyperFlex System

At this point in the offline upgrade, you have checked the prerequisites, downloaded the
required packages, tested connectivity, and upgraded the Cisco UCS infrastructure (if
needed).

 Launch the vSphere Web Client and power down all user VMs residing on Cisco
HyperFlex servers and all user VMs running on HyperFlex datastores, including
VMs running on compute-only nodes. After the VMs have been shut down, verify
the health state of the system and perform a graceful shutdown.
 It is important that CVMs remain powered on if you are planning to go straight into
upgrade of Cisco HXDP without upgrading Cisco UCS firmware on hosts.
 SSH to any CVM in the group:

o Check HyperFlex health: hxcli cluster info | grep health

o If healthy, shut down the HyperFlex: hxcli cluster shutdown
o Shutdown takes a few minutes, so wait for the prompt to return.

You will not be able to gracefully shut down your HyperFlex system unless you first
shut down all the guest VMs.

Once you shut down your HyperFlex, Cisco HXDP is not accessible anymore. Cisco
UCS Manager and ESXi server will continue to be accessible.

Upgrading Cisco UCS Firmware

Follow these steps if performing Cisco UCS host firmware upgrade during the offline
workflow:

1. Manually stage the correct Cisco UCS version before starting the upgrade process.

A. In Cisco UCS Manager, navigate to Servers > Policies > Sub-Organizations

> your group name.
B. Expand Host Firmware Packages and choose the policy you want to update.
C. In the Work pane, click the General tab.
D. To modify the components in the host firmware package, click Modify Package
Versions and modify to the recommended firmware version.
 E. In the Excluded Components area, check the boxes corresponding to the
components that you want to exclude from this host firmware package.
F. Click OK. Click Yes for all the warnings.

Do not perform any acknowledgment of pending activities at this point.

Continue the Cisco UCS server firmware upgrade:

2. Shut down HyperFlex CVMs and put ESXi hosts into maintenance mode.

A. In vCenter, right-click each CVM, and choose Power > Shut Down Guest OS.
B. In vCenter, right-click each ESXi host, and choose Maintenance Mode > Enter
Maintenance Mode.

3. Acknowledge pending reboot of servers.

o Wait until all nodes are upgraded. Confirm that the correct firmware packages
have been installed before proceeding.

4. Take ESXi hosts out of maintenance mode and CVMs will come back online.

o In vCenter, right-click each ESXi host, and choose Maintenance Mode > Exit
Maintenance Mode.

Shutting down CVMs or Cisco UCS firmware itself and entering and exiting
maintenance mode for ESXi hosts will all take time to finalize. Wait for each individual
action to complete before moving to the next one.

 The vCenter task bar will tell you when the host has successfully entered or exited
maintenance mode.
 In Cisco UCS Manager, under the individual server's FSM tab, you can observe the
progress of the Cisco UCS firmware upgrade.

7.5 Maintaining Cisco HyperFlex

Cisco HyperFlex Maintenance Mode

This video describes the HyperFlex Maintenance mode and ESXi upgrade. The Cisco HX Mainte
nance mode performs HyperFlex data platform-specific functions in addition to the ESXi
Maintenance mode. It's very similar to the ESXi Maintenance mode, but it also performs
checks such as cluster health.

So be sure to select HX Maintenance Mode for tasks performed on the storage cluster nodes.
First, make sure the cluster is healthy before you use the HX Maintenance mode. Next, you'll
need to make sure that SSH is enabled and ESX on all of the nodes on the storage cluster.
Lastly, when you're done performing your maintenance tasks, don't forget to exit the HX
Maintenance mode.
This figure shows the steps for using HyperFlex Connect to enter the HX Maintenance mode
for a host. First, you would navigate to System Information Nodes. Next, you would choose a
node, click Enter HX Maintenance Mode, and confirm your selection. Then monitor the activity
on the page for the progress. After the node has successfully transitioned into the HX
Maintenance Mode, the hypervisor status should be displayed in Maintenance. And the
controller status should be listed as Offline, not Online but Offline.

When you're done with your maintenance task, don't forget to exit the HX Maintenance mode.
Navigate to System Information Nodes, choose the node again, and click Exit HX Maintenance
Mode. After transitioning out of the HX Maintenance Mode, the node should now display both
the status and the controller status as Online.
ESXi is installed as the hypervisor on any given server within a VMware-based Cisco HyperFlex
cluster. This goes for both the converged nodes and the compute-only nodes. There are three
ways you can perform upgrade of the ESXi hypervisor. You can use HX Connect, which is the
recommended option since hypervisor version 3.5.1a. You can upgrade individual ESXi via the
ESX CLI, which is recommended for the older versions of the hypervisor. And you can use
VMware Update Manager or the VUM.

Using VUM is discouraged because there's no guarantee that the cluster will be healthy by the
time VUM moves on to the next node. If you use VUM for an ESXi upgrade, then you need to
do it one node at a time. Then make sure the cluster is healthy before moving on to the next
node.

This figure lists the steps for an ESXi upgrade using ESX CLI. First, complete the preupgrade
validation checks. Next, download the ESXi upgrade package from software.cisco.com. Then
select one of the hosts and put it into the HX Maintenance mode. When it is in the
Maintenance mode, use Secure Copy protocol and copy the upgrade package to the host.
Lastly, upgrade the ESXi using ESX CLI. And use the software update-- well, actually it's the
esxcli software update dash d command. And if you look, it's actually displayed in the example.
After the upgrade is finished, restart the ESXi host using the esxcli system shutdown reboot
command, as shown in the example. When the ESXi host comes up, verify the correct version
with the vmware dash vl command. Next, wait for the ESXi host to auto-reconnect to vCenter.
Sometimes you need to force it by choosing Connection, Connect. Lastly, exit the Maintenance
mode and make sure the cluster is healthy. Then repeat the process for the other nodes in the
cluster in sequence.

Finally, VMware-based Cisco HyperFlex clusters are registered with a vCenter. It's possible to
unregister a HyperFlex cluster from a given vCenter and register it with another one. This
figure here describes how to move a cluster to another vCenter. First, you would delete the
cluster from the current vCenter that it's registered to.

Next, you would create a new cluster on the new vCenter using the same name. Then you
would add the ESX host to the new vCenter in the newly created cluster. Lastly, you would
register the HyperFlex cluster with a new vCenter from the STCLI using the stcli cluster
reregister command. All of the command arguments are actually shown in the example. And
that brings us to the end of this video. Thanks for watching.
Maintenance mode is applied to nodes in a group. You would perform it on a node-per-
node basis. The maintenance mode prepares the node for assorted maintenance tasks by
migrating all VMs to other nodes before you decommission or shut down the node.
Note that if you are planning to perform maintenance online (with guest VM workload
undisturbed), you must not put more than one node into maintenance mode.

There are two types of maintenance modes:

 VMware ESXi maintenance mode.

o This is the traditional ESXi maintenance mode initiated from the vCenter. It will
check for running VMs and give you the option to terminate the running VMs.
Terminating will shut down the CVM improperly and will not check for
HyperFlex storage resiliency status.

 Cisco HyperFlex maintenance mode.

o This procedure performs the storage resiliency status check and then properly
shut down the CVM. After that the standard vSphere maintenance is initiated.

You should use ESXi maintenance mode when putting a node into HyperFlex
maintenance mode is not working. Note that ESXi maintenance mode is not graceful
towards Cisco HXDP.

Both ESXi and HyperFlex maintenance mode show up the same way in vSphere Web
Client; the server icon changes, and it states "(maintenance mode)" next to the
hostname.

You can enter or exit HyperFlex maintenance mode using either CLI or HX Connect.

Entering Cisco HyperFlex Maintenance Mode

Follow these steps to enter HyperFlex maintenance mode for a host from HX Connect:

1. In the menu, navigate to System Information > Nodes.

2. Choose the node you want to put into HyperFlex maintenance mode, and
click Enter HX Maintenance Mode.
3. Confirm your selection.
4. Monitor Activity page for progress.
Using HyperFlex Connect for exiting and entering HyperFlex maintenance mode is
supported as of Cisco HXDP 2.5(1a) and later. Before, the maintenance mode was
initiated through Flash HyperFlex plug-in, which is no longer available on vSphere 7.0
and above.

Note
After a node is successfully transitioned into HyperFlex maintenance mode, it should
have its Hypervisor Status listed as In Maintenance. The Controller Status should be
listed as Offline. Hypervisor status might be listed as Offline for intermittent time of
few minutes.

You can perform the same task of putting a node into HyperFlex maintenance mode
from stcli:

 Log in to the main storage controller command line as a user with admin privileges.
 Identify the node ID and IP address using the stcli node list --
summary command
 Enter the node into HyperFlex maintenance mode using the stcli node
maintenanceMode (--id <ID> | --ip <IP>) --mode
enter command.
 Log in to the ESXi command line of this node as a user with root privileges and
verify that the node has entered HyperFlex maintenance mode using the esxcli
system maintenanceMode get command.

Exiting Cisco HyperFlex Maintenance Mode

Exiting maintenance mode through the HyperFlex Connect workflow is not as critical
as entering it. You can normally exit the HyperFlex maintenance mode by exiting a
node from the vSphere maintenance and then turning on the CVM.

After you complete the maintenance tasks, you must manually exit the HyperFlex
maintenance mode. Follow these steps to do it from HX Connect:

1. In the menu, navigate to System Information > Nodes.

2. Choose the node that you previously put into HyperFlex maintenance mode.
3. Choose Exit HX Maintenance Mode.

Note
After a node is successfully transitioned out of HX maintenance mode, it should have
both Hypervisor Status and Controller Status listed as Online. Both might be listed
as Offline for the intermittent time of few minutes.

You can perform the same task of putting a node out of HyperFlex maintenance mode
from stcli:

1. Log in to the storage controller command line as a user with root privileges.
2. Identify the node ID and IP address using the stcli node list --
summary command
3. Enter the node into HX maintenance mode using the stcli node
maintenanceMode (--id <ID> | --ip <IP>) --mode
exit command.
4. Log in to the ESXi command line of this node as a user with root privileges and
verify that the node has entered HyperFlex maintenance mode using the esxcli
system maintenanceMode get command.
If the operation of exiting HyperFlex maintenance mode fails, an error message is
displayed. Try to fix the underlying problem and attempt to exit HyperFlex maintenance
mode again.

7.6 Maintaining Cisco HyperFlex

ESXi Upgrade
ESXi is installed as the hypervisor on any given server (converged or compute-only)
within a VMware-based Cisco HyperFlex system. You can upgrade the hypervisor in a
few ways.

You can upgrade ESXi through HX Connect, ESXCLI, and VMware Update Manager
(VUM):

 Using VUM to upgrade ESXi is discouraged.

 Upgrade through HX Connect is only available in HXDP 3.5(1a) or later.
 The command-line upgrade uses the Cisco .zip upgrade package:

o Verify HyperFlex health status.

o Put a node into HX maintenance mode.
o Upload ESXi upgrade package from software.cisco.com to ESXi using SCP.

 You can upload the ESXi bundle to the HyperFlex Datastore and it will be
available on all nodes.

o Use the standard ESXi command stack to perform the upgrade.

The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex
workload. This upgrade is achieved by performing an online rolling upgrade of each
node in the HyperFlex system. If performing an offline upgrade with HyperFlex system
shutdown, you can perform ESXi upgrades in parallel on all hosts without disruption.

The use of VUM for upgrading ESXi is discouraged because there is no guarantee that
the HyperFlex will be healthy by the time VUM moves on to the next node. If you
would use VUM for ESXi upgrade, you will need to use it one host at the time and then
make sure that HyperFlex is healthy before moving on to the next node.

ESXi Upgrade Using ESXCLI

Manual upgrade of the ESXi hypervisor can also be performed if the automatic upgrade
fails on newer versions that support automatic upgrading of the ESXi.
After you have uploaded the upgrade package, log in to the ESXi command line of the
node that you have put into HX maintenance mode:

1. Upgrade ESXi using esxcli software command and reboot:

esxcli software sources profile list -d <path_to_ZIP_file>

esxcli software profile update -d <path_to_profile_ZIP_file> -p <profile name>
reboot

2. Exit the HX maintenance mode.

3. Wait for the HX storage status to become healthy.

Note
The procedure needs to be repeated on each node to upgrade all ESXi installations. This
includes compute-only nodes.

The esxcli software sources profile list -d command will show the
available profiles in the .zip upgrade file. Use the output from the list in the esxcli
software profile update -d command after -p switch.

Always provide full paths in the manual upgrade commands, or you will receive an
error.

After you have rebooted the server, you can log in to ESXi again and verify the version
of ESXi by issuing the vmware -vl command.

Note
When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please
contact Cisco TAC.

Do not use the HyperFlex ISO file or any other VMware ISO to attempt an ESXi
upgrade.
7.7 Maintaining Cisco HyperFlex

Moving Cisco HyperFlex Group to

Another vCenter
VMware-based Cisco HyperFlex systems are registered with a vCenter. It is possible to
unregister your HyperFlex group from a given vCenter and register it with another one.

Perform these actions before you begin the process of moving a HyperFlex group from
one vCenter to another:

 Check that the HyperFlex is online and healthy.

 Check that vCenter is up and running.
 Note that snapshot schedules are not moved with the storage group when you move
the storage groups between vCenters. Make a list of the snapshot schedules so you
can then later re-create them.

Note
If your HyperFlex system is running Cisco HXDP version older than 1.8(1c), upgrade
before attempting to re-register to a new vCenter.

Remove a HyperFlex group from the current vCenter, and re-create the group/host
hierarchy in the new vCenter:

 From the current vCenter, delete the HyperFlex group.

 On the new vCenter, create a new group.
 Add ESX hosts to the new vCenter in the newly created group.
 Register the HyperFlex group with the new vCenter from stcli:

stcli cluster reregister --vcenter-datacenter

NEWDATACENTER --vcenter-cluster NEWVCENTERCLUSTER --
vcenter-url NEWVCENTERURL --vcenter-user NEWVCENTERUSER
If you have compute nodes on your HyperFlex system that is running version 4.0 or
earlier, after completing the re-register, re-add the compute nodes:

 If you are running newer HyperFlex versions, you do not need to re-add compute
nodes.

stcli node add --node-ips <computeNodeIP> --controller-

root-password <ctlvm-pwd> --esx-username <esx-user> --esx-
password <esx-pwd>

 If, after your storage system is re-registered, your compute only nodes fail to
register with the ESX Agent Manager (EAM), or are not present in the EAM client,
and not under the resource pool, contact Cisco TAC to complete the compute node
reregister.

You can also remove the EAM agent from the HyperFlex system. EAM is a legacy
subsystem that is no longer necessary. For more information, follow the guide
at https://www.cisco.com/c/en/us/support/docs/hyperconverged-infrastructure/
hyperflex-hx-data-platform/215255-esxi-agent-manager-removal-process.html.

Note
Similarly, if you rename the data center and/or group name in the vCenter inventory list,
you must run the stcli cluster reregister command from a CVM towards the
existing system but with a new group and/or data center name.

8.1 Designing Cisco HyperFlex

Introduction
The Cisco HyperFlex solution is a flexible product with many applications. The features
and their combinations come into play differently, depending on the workload that you
will be deploying on the infrastructure.

Depending on the needs, elements of design can make a significant impact on the actual
performance and feature set of the solution, such as scoping the initial storage space,
choosing the correct resiliency, and calculating the effect of hardware failures into the
design.

To make a sound and viable design, it is important to have good holistic knowledge of
the Cisco HyperFlex solution, how the various features affect each other, and what is
possible with the Cisco HyperFlex solution in general.

Note
Although the Cisco HyperFlex solution supports vSphere and Hyper-V as hypervisors,
you will focus on the ESXi-based version of Cisco HyperFlex, which is far more
prevalent. Many considerations and features remain (such as the replication factor,
resiliency, networking, and most of installation), but the underlying technology is
different.

8.2 Designing Cisco HyperFlex

Group Resiliency: VM-Level Final del

formulario

Let's examine cluster resiliency in more detail. Virtual machines running in a standard
Cisco HyperFlex cluster are managed in the vSphere environment. vSphere already
supports VM resiliency, migration, and management. Cisco HyperFlex uses these tried-
and-tested mechanisms in its own environment.

When a failure occurs on one of the HyperFlex hosts, the VM is started on another
available host by using VMware high availability. The underlying HyperFlex data store
enables the migration without any movement of the storage data from one host to the
other. This is not necessary because it's already distributed.

The hypervisor manages the VM-level resiliency of the Cisco HyperFlex cluster.
However, HyperFlex converged nodes also provide the storage space for the VMs. A
failure will not only cause the loss of computing memory, but it also means that part of
the storage has been removed. In this scenario, the VMs reattach, but no data movement
is necessary because the data is distributed across the cluster. The HyperFlex cluster
can sustain the losses of up to two nodes in a standard deployment.

Data distribution in HyperFlex provides real-time replication of the information across

the entire cluster. The reading and writing are performed over the network for each
input and output operation. This means that losing a node does not cause any
interruption in the storage operations. This is because the HyperFlex system is aware of
the failures. And it automatically resolves them by not choosing those failed elements.
Even when the number of failures exceed the tolerance of the cluster, data will not be
lost. But unfortunately, the system will shut down if you exceed the tolerance of the
cluster.
The Cisco HyperFlex cluster configuration can influence the cluster's resilience. And
once again, these choices are made when you deploy the HyperFlex cluster. These
choices that you make are going to have a significant impact. For example, how many
node failures will affect the storage cluster. How many numbers of nodes in the cluster.
What are you going to set the data replication factor to? And what is the access policy?
All of these will have a significant impact.

Choosing a replication factor, or RF, is probably the most important choice when you're
installing the HyperFlex cluster. RF2 writes every block to two different nodes in the
cluster. For example, data block A in the example, if you look in here. Data block A and
data block B are written to the first two nodes. And then you'll notice that data block C
is written to the second and the third node and so on.

This is OK, but it's risky if you're performing rolling updates on the system. This is
because one of the servers needs to be offline to perform the maintenance job. And if it
has one of those copies of the data blocks, you can't afford another failure. There's just
no room for additional failures. And it would cause the system to go offline if anything
went wrong during the maintenance procedure.
RF3 writes every single data block to three different nodes in the cluster and provides
resiliency to the system, even during maintenance. This is true on a cluster size of three
or more nodes. For example, in this figure here, you notice that data block A is written
on the first three nodes. RF3 is recommended to maximize the resilience of the cluster.

The cluster access policy defines the level of redundancy before the cluster actually
shuts down. Cisco HyperFlex has two access policies to choose from to determine what
to do if the cluster has no redundancy available. When you choose the StrictMode, it
immediately goes into the read-only mode to prevent data loss on an additional failure.
If you choose the lenient mode, which is the default option, by the way, the cluster does
not go into the read-only mode. It's important to note that if the cluster goes offline due
to failures, it does not mean that all of the data on the cluster is going to be lost.
However, the data since the last successful write operation to the persistent storage, that
tiny piece of data will be lost.

Self-healing is the process of restoring the storage-level redundancy to the server after a
failure occurs and when you lose information. What it does is it replaces the pieces of
information that are lost during the hardware failure. This way, cluster resiliency is
restored, as defined by the replication factor. And if you remember, we said if you've
got a replication factor of three, you need five nodes, right?

Because the process creates a load on the system, they've got to put in a timeout
mechanism to prevent what we call hardware flaps or short-term issues. There is a
default timeout of two hours. And it can only be modified with help from Cisco tech
support.
The self-healing after a capacity drive failure is much less disruptive to the cluster
operation. This is because the VMs don't have to be migrated. And the node can still be
used for the file system operations. If the node failed, we had to migrate the VM. I don't
have to. Capacity drive failure is the least impactful drive failure, and it does not
introduce any storage operations offloading to the others. That's because it's distributed.

Therefore, the self-healing timeout for a drive failure is one minute, and then it will
replace the missed pieces of missing data. The failure scenarios are different for
capacity drives than they are for the caching and the housekeeping drives.

Failure of the caching drive on a node prevents the node from being involved in caching
operations. The IO visors on the nodes in the HyperFlex cluster will not choose the
node as a target for caching. However, the destaging process will still send data to the
node with failed the cache being written to the capacity drive. Because other nodes are
taking over this caching operation, the lossing of a cache drive has a minor impact on
the performance. So introducing a replacement drive is a pretty simple solution to the
problem.
The housekeeping drives. The most significant drive failure is when that darn housekee
ping drive fails, because it interrupts the controller VM operation. The IO visor on the
node with a failed housekeeping drive remains operational. This is because it is only
partially dependent on the CVM, and it exists in the hypervisor. Therefore, it can still
access the HyperFlex storage but cannot read or write from any of the local drives.
Replacing the housekeeping drive with a new one requires a manual re-installation of
the CVM from the CLI. So it's a little bit more complicated procedure.

The table in this figure here compares drive failure characteristics for a Cisco
HyperFlex versus the traditional RAID-based storage systems. Obviously, Cisco
HyperFlex data protection is going to be much more responsive than the traditional
RAID systems. This is especially true when it comes to rebuilding the information.
We're talking about less than an hour as compared to many hours, if not days, for the
RAID system. Also, Cisco HyperFlex does not have the overhead associated with the
RAID hardware or its operations.
Finally, you can choose to increase the resiliency of larger Cisco HyperFlex cluster by
enabling something called logical availability zones, or LAZ. Eight or more converged
nodes are required, but no additional licensing is needed. That's good news. LAZ is
enabled after the installation, and it will cause the data migration. It does not consume
any additional space.

It just segments the cluster into server groups, which act as a redundancy mechanism.
The data is distributed across the server groups according to the replication factor that
you've chosen. With RF3, you can lose up to three entire groups of the server before
redundancy is lost. With RF2, only one server group can be lost. And that brings us to
the end of the video. Thanks for watching.

Individual VMs running on a standard Cisco HyperFlex deployment are operated and
managed from a vSphere environment. The usual vSphere systems are in place for VM
resiliency, migration, and management. The Cisco HyperFlex solution uses tried-and-
tested mechanisms in its environment, offering a well-known management environment
as part of its solution.

Cisco HyperFlex offers VM resiliency and provides these advantages:

 VM-level resiliency is based on VMware high availability.

 Quick restart of virtual machines on a different physical server if the hosting server
fails.
 Cost-effective high availability for all applications.
 Dedicated standby hardware unnecessary.
 Cost and complexity of grouping is eliminated.
When a failure occurs on one of the HyperFlex hosts, the VM is started on another
available host by using VMware high availability.

The underlying HyperFlex datastore enables the migration without moving the storage
data from one host to another, as the same storage is mounted on all the hosts of a
HyperFlex group.

8.3 Designing Cisco HyperFlex

Group Resiliency: HXDP-Level

While the hypervisor manages the VM-level resiliency of the Cisco HyperFlex system,
individual HyperFlex converged nodes are not only virtual hosts—they also provide the
storage space for the VMs.

A failure will not only cause loss of the compute and memory—it also means that a part
of the storage has been removed. The data stored on HyperFlex storage relies on the
StorFS solution and can sustain loss of up to two nodes on standard deployments.

The data resiliency is tightly tied to the replication factor that is chosen during the Cisco
HyperFlex Data Platform (HXDP) installation.
Data resiliency is based on Cisco HXDP and has these features:

 Platform can sustain multiple node failures without data loss.

o If using RF3 on a deployment of 5 nodes or more.

 If a node fails, the evacuated VMs re-attach with no data movement required.
 Replacement node automatically configured through Cisco UCS Service Profile.
 Cisco HXDP automatically rebalance data to the new node.

RF3 does not reach full resiliency until the group size reaches five nodes. This result
can be achieved by expanding the deployment to five nodes and not only with the initial
install. In this case, the configuration on initial installation must be RF3, because you
cannot change the replication factor after installation.

Data Distribution and Non-Disruptive Operations

Data distribution in Cisco HXDP provides real-time replication of information across
the group. Information exists on the storage according to the replication factor. The data
distribution feature allows failures without data loss.

Reading and writing are performed over the network for each I/O operation in real time,
so losing a node does not cause interruption in storage operations. The reads and writes
are offloaded to the remaining instances of information.

When a read or write operation is initiated on the Cisco HyperFlex datastore, the system
called IOVisor assigns the operation to an available node. Therefore, the HyperFlex
system is aware of failures and automatically resolves them by not choosing failed
elements as targets.
The system shuts down when the number of failures exceeds the tolerance of the
HyperFlex group. After resolving the failed element (disks or servers), the group can
recover if the missing nodes are re-introduced.

Cisco HyperFlex Storage Resiliency

The Cisco HyperFlex storage configuration can significantly influence group resilience.
The choices made during deployment can have a significant impact on the functionality
of the HyperFlex system.

Some options, such as access policy, are set by default on install, but can be changed
later. However, the replication factor is fixed and cannot be changed after installation.

How the number of node failures affects the storage system depends on the following
parameters:

 Number of nodes in the group

o Number of locations storing data

 Data replication factor

o Number of data copies

 Access policy

o Number of acceptable failures before shutdown

Replication Factor
Choosing a replication factor is one of the most important choices that you make when
installing Cisco HyperFlex. The replication factor directly affects the free storage space
and resilience.

The replication factor is a significant consideration when you are scoping the number
and size of the drives that you will be purchasing in your Cisco HyperFlex servers.

With RF2, every block is written to two different nodes in the group and offers these
functions:

 Ability to survive one failure—server or drives in one server.

 Minimum replication option for HyperFlex.
 Data is written twice to raw storage; therefore, it can carry half the raw data.
 During rolling upgrade, RF2 can sustain no failures as one node is offline.
 Replication factor cannot be changed after installation.

RF2 is especially risky when you are performing rolling updates on the system, because
one of the servers must be offline to perform maintenance jobs. There is no room for
additional failures, and the system will go offline if anything goes wrong during the
maintenance.

With RF3, every block is written to three different nodes in the group and offers these
features:

 Ability to survive two failures, on a group size of 5 or more.

o Increases resiliency by 100 percent compared to RF2 with 5+ nodes.

  Three copies of data mean that the effective storage size is a third of the raw storage
space.
 Default and recommended is RF3 to maximize resiliency.

Effective data stored on deployments with RF3 will be only a third of the raw storage to
provide resilience to the system even during maintenance. However, you will often see
higher optimization rates with RF3 than with RF2. Compression and deduplication are
application-specific, so plan accordingly when designing your HyperFlex solution.

Resiliency on Server and Capacity Disk Failures

The access policy defines the level of redundancy before the HyperFlex system shuts
down. Effectively, the access policy defines whether you want HyperFlex to go into
read-only mode when it still has redundancy or to skip the read-only state before
shutdown.

Cisco HyperFlex has two access policies to choose from:

 Strict: When the storage has no redundancy the storage goes into read-only mode to
prevent data loss on additional failure.
 Lenient: Default option. Storage does not go into read-only mode when no
redundancy is available.

Replication Factor Access Policy Failed Disks Across Nodes or Number of Nodes

— — Read/Write Read Only Shutdown

3 Lenient 2 — 3

3 Strict 1 2 3

2 Lenient 1 — 2

2 Strict — 1 2

Since HXDP 1.8(1a), the default policy configuration is lenient, which means that the
system will operate normally when running without redundancy.

Because HyperFlex data redundancy is based on the replication factor, the number of
failures before the strict policy read-only mode is tied to replication factor setting.
The table shows that when no redundancy is available in strict mode, HyperFlex
initiates read-only mode. With the default lenient access policy, the storage works
normally until the number of failures exceeds the tolerance at which point it goes
offline.

It is important to note that if the system goes offline due to failures, it does not mean
that all the data on the HyperFlex group is lost if the failure is resolved and the missing
data is re-introduced to the group. Only the data since the last successful write operation
to persistent storage is lost.

Note
You can choose to also increase the resiliency of larger HyperFlex deployments by
enabling Logical Availability Zones (LAZ). LAZ segments the Cisco HyperFlex storage
system into server groups, which act as a redundancy mechanism. The data is
distributed across the server groups according to the replication factor. Enabling LAZ
allows you to lose up to two entire groups of servers with RF3, at which point all
redundancy is lost. With RF2, only one server group can be lost.

Storage Self-Healing: Node Failure

Self-healing is the process of restoring the storage level redundancy to the server after a
failure, whenever you lose copies of information. The storage replication factor policy is
no longer satisfied, and the system must re-instate the resiliency state after the failure.

Self-healing replaces the pieces that are lost during hardware failure and prevents them
from remaining as copies on the storage. In this way, storage resiliency is restored,
defined by the replication factor.

Because the process creates a load on the system, failsafe measures are in place to
prevent initiating the process before necessary. The timeout mechanism prevents
hardware flaps or short-term issues.

The system reacts differently when a node fails or when a single disk on a node fails.
The amount of data that needs to be replicated is significantly different when one or the
other happens.

When dealing with node failure, remember the following facts:

 VMware high availability moves VMs to active nodes without data migration.
 The self-healing timeout for a node failure is 2 hours.
 Delay timeout can be modified.

o Only with Cisco TAC Support

 Self-healing delay accounts for temporary failures.

To lose all redundancy, disks can be lost only across two nodes, but if all disks on one
node fail, then an RF3 configuration maintains redundancy.

Storage Self-Healing: Drive Failure

Self-healing after capacity drive failure is much less disruptive to the system operation,
because replicating the amount of data carried by one drive creates far less load on the
system than when a node fails. The self-healing timer is therefore adjusted accordingly.

Because each node has capacity drives in addition to Cisco HyperFlex-specific caching
and housekeeping drives, the scenarios for each of these drives failing are not the same.

The capacity drive failure is the least impactful drive failure and does not introduce any
storage operations offloading to other nodes.

The following are consequences of dealing with a capacity drive failure:

 Self-healing timeout on drive failure is 1 minute.

 The compute on the node is unaffected, and the VMs are not migrated.
 The CVM still runs, and the node is still used for filesystem operations.

Failure of the caching drive on a node prevents the node from being involved in caching
operations. The IOVisors on the nodes in the HyperFlex group will not choose the node
as the target for caching, but the node will be the target for destaging of data from
caches on other nodes to capacity drives and read operations.

Because other nodes take over caching operations, the loss of a caching drive has only a
minor impact on performance. When the HyperFlex group loses a caching drive, less
cache is available.

When dealing with caching drive failure, the following will occur:

 The IOVisors on the system will send data to be cached to other nodes.
 The de-staging process will still send data to the node with the failed cache to be
written to capacity.
 Reads from the failed node will be performed normally, since the local IOVisor and
CVM are active.
 Less total cache, but only minimum impact on the system operation.
 Introducing a replacement drive will resume normal operations.

Cisco HyperFlex always distributes the I/O operations to across nodes in the group.
When a cache failure occurs on a node, that node is not chosen to cache any
information, but the node performs other functionalities normally. Because the basic
operation of the system is unchanged when the cache drive fails, the system is
minimally impacted.

Failure of the housekeeping drive is the most significant drive failure that can occur,
which freezes storage operations on the node experiencing failure. The following are
consequences of a housekeeping drive failure and requirements for replacement:

 Housekeeping drive failure will interrupt CVM operation.

 CVM is involved in all data operations. Failure marks the storage of the converged
node as inactive.
 The compute of the node is unaffected. Node still has access to the HyperFlex
storage.
 Reduced storage operation speed, by a node not involved in the process.
 More impactful on small deployments, less impactful on larger deployments where
one node represents a smaller part of the system.
 Replacing the housekeeping drive with a new one requires manual re-installation of
the CVM from CLI.

The IOVisor on the node with a failed housekeeping drive remains operational, since it
is only partially dependent on CVM and exists in the hypervisor. Therefore, it can still
access the HyperFlex storage but cannot read or write from local drives, as they are
passed through to the CVM, which is not operational.

Comparing Drive Failure in Cisco HyperFlex and Traditional

RAID-Based Storage System
Cisco HyperFlex data protection is much more responsive than traditional RAID
systems, especially when rebuilding the information. There are also no specific
hardware RAID cards involved or RAID field definitions.

The HyperFlex system is made to be scalable and expandable while maintaining

resilience and uniformity.
Characteristics HyperFlex Traditional RAID Group

Data protection Full mirrors Parity-based

Rebuilds <1 hour Hours to days

Parallel rebuilds Yes No

Sequential failures Unlimited if there are resources (after self-healing) Number of hot spares

Overall performance Fast with reserved resources RAID overhead

After self-healing finishes, the system regains the resilience defined by the replication
factor. Failures transfer the load on the remaining nodes, so the system can sustain many
failures while the remaining nodes can carry the load.

8.4 Designing Cisco HyperFlex

Cisco HyperFlex Scalability

This video examines Cisco HyperFlex cluster scalability and capacity. Cisco HyperFlex is
the ideal solution for a private cloud infrastructure because of its online scalability. Standard
clusters can be expanded by adding either converged nodes or compute nodes. Depending on
the license, scaling of compute and capacity can be set up as a one-to-one or a two-to-one
ratio limit.
For converge nodes, all you need to do is provide compatible firmware with the similar
compute specs and the same drive configuration. For compute nodes, there is a wide range of
options which do not need to be as uniform and don't need the actual storage. So it's not as
strict. However, there still is a benefit for being as uniform as possible when you're choosing
your nodes.

This figure here lists some of the nonscaling options for Cisco HyperFlex. Most notably, there is
a maximum of 64 nodes supported in a cluster. There are different converge to compute node
configurations available. However, a maximum of 32 converge nodes are supported. Plus, the
ratio of converge-to-compute nodes, once again, depends on whether you have the correct lic
ense. It depends on whether you have a Standard or an Enterprise license. So be sure to famili
arize yourself with all of the scaling limitations.
This figure lists the considerations when choosing cluster capacity. When sizing a Cisco
HyperFlex cluster, you will have to take into account several parameters. The parameters
define the actual hardware that you will need for your intended workload. The overall usable
cluster capacity is based on the number of nodes, the number and sizes of capacity layer disks,
and the replication factor.

When it comes to user capacity, you need to closely monitor and manage the cluster,
obviously. But keep in mind that if the utilization is above 70%, then the built-in cleaner
runs in the Aggressive mode, and it will actively clean obsolete information from the disk to
make the remaining free space available, which burns up overhead.

This table and this figure lists the different minimum and maximum capacity per node,
depending on the node drive slot and the drive sizes. You can choose to expand your cluster
with additional drives at a later point if you have free drive slots in your servers. All you have
to do is insert a drive into each server. Keep in mind, the new drives must be the same capacit
y as the original drives. Also, don't forget, the number of drives must be the same in all the
nodes to have a supported system.
This next figure describes the formula for calculating HyperFlex cluster capacity. The Store FS
System requires 8% of the raw storage for its operation. This explains why they have that 0.92
multiplier in the formula. This overhead must be calculated before the replication factor.

The replication factor will effectively divide raw storage by the number of copies it will make,
whether you're doing RF2 or
RF3. RF2 cuts the remaining raw storage in half after that 8% reduction, whereas RF3 will only
carry a third of the raw storage after the 8% reduction. But it offers twice the resiliency.

The example in this figure here actually plugs in some values for you to examine. Keep in mind,
the capacity calculated by this formula provides you with the worst-case storage scenario. This
much data will be available even when writing completely random data, which cannot be com
pressed or duplicated. Well, obviously, this would be a rare scenario.
Now let's take a closer look at the capacity savings. Cisco HyperFlex supports compression and
inline deduplication, both of which are performed in real time. Compression is how much of
the data can be compressed. And deduplication reduces the storage space by storing only one
unique instance of the data. Whenever information is written to the capacity, it's always
automatically optimized for size. This means you will have more space available than the
suggested capacity calculations for completely random data that we just looked at.

This next figure here lists terminology for calculating capacity savings. First, total addressable
space is the total raw space on the storage, including metadata. Total unique use size is the
space taken by the information, and total unique bytes is the data that cannot be optimized
using compression or deduplication.
This next figure provides a capacity savings example for two 100-gig VMs, which are exact
clones. It shows the calculation steps with the result. Now, keep in mind, this calculation only
provides a rough estimate. But it's useful for sizing purposes if using conservative estimates.
It's really difficult to determine how much of the information can be optimized through
compression and deduplication, et cetera. So when calculating the capacity savings, it's best
to calculate any current compression and deduplication into the equation.

The Cisco HyperFlex Sizer is an automated tool, which already includes the estimates for
different types of workloads. It also allows you to provide automatically gathered information
from an existing infrastructure to offer any sizing suggestions. Based on the requirements that
you provide, the HyperFlex Sizer performs storage calculations. It then provides you with the
equipment suggestions that suit your input requirements. Makes it a heck of a lot easier.
Finally, proper sizing of your cluster is not the only thing you should consider when purchasing
a HyperFlex System. First, solid state drives are much more reliable than hard disk drives. So
you should really think about using RF2 for all flash systems to achieve better performance and
storage space benefits. Well, once again, RF2 has a negative impact based on resiliency, as
compared to RF3.

So if you're not sure, just choose RF3. That's recommended by Cisco. And lastly, for best
performance, you should always keep a monitor on the cluster and make sure the capacity is
below 70%. That brings us to the end of the video. Thanks for watching.

Cisco HyperFlex is the ideal solution for a private cloud infrastructure. What makes
HyperFlex so uniquely suited to this workload is, among others, flexibility in scaling.

Standard Cisco HyperFlex deployments can be expanded by adding converged nodes

(HyperFlex nodes only) or compute-only nodes (several Cisco M4 and M5 UCS-B and
UCS-C servers).

Compute nodes allow storage-independent scaling of compute up to the 1:1 or 2:1 ratio
limit, depending on licensing.

To scale your Cisco HyperFlex solution, you only need to provide compatible hardware.
You will need similar compute specs and the same drive configuration for converged
nodes. For compute-only nodes, a wide array of supported options is available, where
compatibility considerations regarding storage do not apply. Compute only nodes access
the HyperFlex storage on the HyperFlex nodes and do not contribute any storage into
the HyperFlex storage pool. However, they do provide compute power to run virtual
machines.

It is beneficial (though not necessary) to design your deployment so that the server's
CPUs have approximately the same performance level. CPUs with varied performance
levels can cause trouble in fail-over scenarios. For example, if you have one server that
has two processors with 28 cores each, and the other servers each have two processors
with 8 cores, a failure of the larger-server-by-CPU might mean that you have many
VMs that need to fail over somewhere.

For different compute-to-converged configurations with a maximum of 64 nodes

supported in a group, at most 32 nodes can be converged. For the latest scaling limits,
refer to the latest version of Cisco HyperFlex release notes.

As of Cisco HyperFlex Version 4.5(1a), HyperFlex Standard and Stretched

deployments support the 2:1 ratio of compute only nodes to converged nodes. This ratio
is only available with Cisco HXDP DC Premier license. DC Advantage license supports
1:1 compute-to-converged ratio. You can upgrade from the DC Advantage to DC
Premier license if a larger compute-to converged ratio is required.

When considering node-scaling options, recognize their limitations:

Up to 64 nodes in a HyperFlex group

 32 converged nodes maximum

o Ratio of converged to compute nodes:

 1:1 with Cisco HXDP DC Advantage license

 2:1 with Cisco HXDP DC Premier license

 M4, and M5 compute-only nodes supported

 All limitations for converged nodes must be followed when adding them to the
group.
 Exceptions to standard ESXi deployments:

o Hybrid HyperFlex with LFF drives supports 3–16 converged nodes and
maximum group size of 48 nodes.
o All-NVMe HyperFlex supports 3–16 converged nodes and maximum group size
of 48 nodes.
Since Cisco HyperFlex Version 4.0(2a), compute-only nodes no longer have CVMs
running on them. Instead, the HyperFlex ESXi IOVisor VIB is used to communicate
with HyperFlex storage. The reason is that CVMs on compute nodes do not need to
perform data handling and optimization. You would add a compute-only node to the
group the same way you would a converged node and that is through the HyperFlex
installer.

8.5 Designing Cisco HyperFlex

Cisco HyperFlex Storage Capacity

When sizing a Cisco HyperFlex system, you will have to consider several parameters
that will define the actual hardware you will need for your intended workload.

When deciding HyperFlex capacity, consider these facts:

 Overall usable capacity is based on:

o Number of nodes in the group

o Number of the capacity layer disks
o Size of the capacity layer disks
o Replication factor

 Used capacity should be closely monitored and managed:

o When storage utilization is above 70 percent, the built-in cleaner runs in

aggressive mode, creating overhead.
o Size for failure to be able to run a full workload even with failure (N+1 rule)
 When the cleaner process runs in aggressive mode, it actively cleans the obsolete
information from the drives to make the remaining free space available for writing as
quickly as possible. The cleaner will introduce additional overhead IOPS even during
high load scenarios.

Under normal circumstances, the cleaner process performs cleaning during low usage
periods to have as little impact as possible. You can also schedule the aggressive mode
to be initiated during a period when you expect lower system resource usage. This
process is performed from the CVM command line.

Several combinations of capacity drives and Cisco HyperFlex node sizes are available,
offering different minimum and maximum capacity per node, depending on node drive
slots and drive sizes.

HX-Series Server Node Capacity Disk Size Capacity Disk Usable Capacity Usable Capacity
Model Quantity (Each) Quantity (Per Node) at RF=2 at RF=3

7.6 TB 8 (max) 221.31 TiB 147.5 TiB

HXAF220c-M5SX 8
960 GB 8 (max) 25.7 TiB 17.1 TiB

6 (min) 165.9 TiB 110.6 TiB

7.6 TB 15 414.9 TiB 276.6 TiB

23 (max) 636.2 TiB 424.1 TiB

HXAF240c-M5SX 8
6 (min) 19.3 TiB 12.9 TiB

960 GB 15 48.2 TiB 32.1 TiB

23 (max) 73.9 TiB 49.3 TiB

If you are purchasing HyperFlex nodes and will not be introducing any compute nodes
to the solution, then each HyperFlex node is also a virtual host for VMs. Having only a
few servers fully populated with drives may offer enough storage but not enough
compute power to run your workload.

You can also choose to expand your deployment with additional drives later if you have
free drive slots in your servers. The procedure is simple. You insert a drive in each
server, but the new drives must be of the same capacity as the original drives. The
number of drives must be the same in all nodes to have a supported system.

Cisco HyperFlex Storage Capacity Calculation

Once you decide on the raw storage (defined by the number of capacity drives in the
HyperFlex group and the drive size), consider the filesystem and the replication factor
that you will be using.
The StorFS filesystem requires 8 percent of the raw storage for its operation. This
capacity needs to be deducted from the total storage capacity.

The replication factor will effectively divide the raw storage by the number of copies it
will make. RF2 halves the remaining raw storage after 8 percent reduction, RF3 will
only carry a third of raw storage in information, but offers twice the resiliency.

Remember this example when calculating Cisco HXDP storage capacity:

 Calculate the raw disk space from the number of drives and their size.
 Divide the result by 1024 to get a value in TiB.
 The 0.92 multiplier accounts for an 8 percent reservation set aside on each disk by
the Cisco HXDP software for various internal filesystem functions.

Because all the drives in a HyperFlex system are the same, and all the nodes have the
same number of drives, raw storage calculation becomes much easier.

Note
Be very careful to use the proper unit when calculating space. Tebibyte (TiB) is not the
same as Terabyte (TB). 1 TiB is approximately 1.1 TB. The 10 percent difference is
often significant.

Practical application of the formula yields these results:

 Capacity disk size in GB = 3800 for 3.8 TB disks

 Number of capacity disks per node = 23 for an HX240c M5 model server
 Number of HyperFlex nodes = 8
 Replication factor = 3

The capacity calculated by this formula provides you with the worst-case storage
scenario. This much data will be available even when writing completely random data,
which cannot be compressed or deduplicated. Such datasets are relatively rare.

Note
Keep storage utilization under 70 percent for best performance and resiliency. In the
example above, 70 percent out of 195.02 TiB is 136.54 TiB.

Understanding Capacity Savings

The Cisco HXDP performs inline deduplication and compression, performed in real
time. Whenever information is written to the capacity, it is always automatically
optimized for size. This function cannot be disabled and is a part of the HyperFlex
solution design.

Such data saving measures mean that you will usually have more space available than
the suggested capacity calculation for completely random data.

The total storage capacity saved by the Cisco HXDP system is a calculation of two
elements:

 Compression: How much of the data is compressed.

o You can use the optional Cisco HyperFlex Acceleration card to offload
compression from the CPU, which gives you better compression and storage
performance.

 Deduplication: How much data is deduplicated. Deduplication is a method of

reducing storage space by eliminating redundant data. It stores only one unique
instance of the data.
 Deduplication and compression savings are useful when working with VM clones or
data types that don’t have native compression or encryption features.

Workloads have a significant impact on the savings that can be achieved by

deduplication and compression. Highly compressed data, such as video or highly
compressed databases, will allow little to no optimization, while workloads that have a
lot of duplicate data, such as VDI, will compress and deduplicate well, and take much
less space. Virtual server infrastructure (VSI) is the middle ground where you can
expect to save 30–50 percent.

Note
Utilizing the HyperFlex Acceleration Engine will increase your compression savings up
to 30 percent, compared to native in-CPU compression. If the Acceleration Engine fails
within the server, the server will failback to using native, in-CPU compression.
The actual compression for a dataset is often hard to determine. Be mindful not to
overestimate the compression and deduplication. As a rule, systems that utilize
optimization techniques will compress and deduplicate poorly. A good example is video
encoding.

Cisco HyperFlex Sizer

The Cisco HyperFlex Sizer (https://hyperflexsizer.cloudapps.cisco.com/), an online tool
that includes estimates for different workloads, also allows you to provide automatically
gathered information from an existing infrastructure, to offer sizing suggestions.

Based on the requirements that you provide, the HyperFlex Sizer performs storage
calculations and presents you with the equipment suggestions that suit your input
requirements.

 Can size for many different workload types (such as VDI, VSI, and databases)
 Takes into account storage and compute utilization
 Supports gathering information from an existing infrastructure

o HX Profiler: VM runs for up to 30 days on existing infrastructure and gathers

information
o RV Tools: Uses existing vCenter information

Due to company policies, it is not always possible to deploy a VM on premises to gather

information. RV Tools are much less invasive, or you can enter data manually with
HyperFlex Sizer providing workload estimates.

HyperFlex Bench, which is also available for download from the HyperFlex Sizer
website, is a graphical benchmarking tool for vSphere environments based on VD
Bench. It uses the VD Bench back end to stress test an infrastructure and adds a
graphical front end. It works for any vSphere environment and allows you to effectively
test the difference between old and new infrastructures.
Cisco HyperFlex Capacity Considerations
Properly sizing your deployment is not the only consideration when purchasing a
HyperFlex system. Different choices made at purchase can also introduce operating
specifics to the system.

When choosing your HyperFlex capacity, consider these facts:

 SSD reliability is significantly better than HDD.

o HyperFlex installer no longer defaults to RF3 for an All-Flash or All-NVMe

system, forcing the end user to make a decision.
o RF3 is still the recommended option.

 Consider RF2 for All-Flash or All-NVMe systems that do not require triple data
copies or multiple component redundancies.

o RF2 displays 15–40 percent reduced write latencies in lab testing.

o RF2 can provide up to 2190 TiB maximum capacity, with 32 nodes each with
23x 3.8 TiB disks.

 Keep overall capacity consumed below 70 percent for best performance.

Even though RF2 has performance and storage space benefits, it has negative impact
resiliency compared to RF3. If unsure, choose RF3.

8.6 Designing Cisco HyperFlex

Multiple Systems on One Cisco UCS

Domain
Now let's explore multiple clusters on one Cisco UCS domain. You can deploy up to
eight HyperFlex clusters in one Cisco UCS domain. The clusters can use the same
management vMotion and VM network VLANs. However, each cluster must have its
own unique data VLAN for storage communication. Each cluster can scale up to 64
nodes, with 32 of them being converged nodes, and the rest must be compute-only
nodes.

HyperFlex clusters can run different versions of HyperFlex software. So the fabric
interconnect firmware must be compatible with all of the HyperFlex versions running in
the Cisco UCS domain.

So you've got to verify that all the HyperFlex servers also have the same version of
Cisco UCS server firmware installed. And lastly, you should consult the HyperFlex
installation guide for firmware compatibility just to be safe.
Cisco HyperFlex compatibility with firmware versions of hardware is not the only
versioning consideration. When configuring data replication between the two clusters,
the newer functionality in release 3.0 is supported only when the clusters are on the
same software version.

Also, it can take a while during the upgrade for both the source and the target to be on
the same version. So be sure, once again, to review the functionality limitations that are
listed here in this figure.

Finally, you can introduce Cisco UCS servers which are not part of a Cisco HyperFlex
cluster to the fabric interconnect domain while you're running HyperFlex. These servers
can just be in their own clusters and can coexist on the Cisco UCS domain as long as
the common configuration requirements match.
You can choose to deploy the HyperFlex cluster on pre-existing fabric interconnects
already running another non-HyperFlex cluster. Since the HyperFlex installer changes
certain global settings on the Cisco UCS domain, you need to consult Cisco TAC before
you actually deploy the HyperFlex cluster. That brings us to the end of this video. Once
again, thanks for watching.

Cisco HyperFlex systems can be deployed on pre-existing Cisco Fabric Interconnect

domains. You can also deploy two different Cisco HyperFlex deployments on the same
pair of Cisco Fabric Interconnects.

Cisco HyperFlex deployments can use the same management, vMotion, and VM
network VLANs, but each server group must have its own unique data VLAN for
storage communication.

Remember the following facts when you have multiple server groups in one Cisco UCS
domain:

 Scale up HyperFlex deployment up to 64 nodes.

o Up to 32 converged and the rest compute-only.

 Scale out computing environment up to 8 HyperFlex deployments in one Cisco UCS

domain.

A single HyperFlex group cannot exceed 64 nodes. More than 64 HyperFlex servers can
be on the same Fabric Interconnect domain, but they must be in more than one
HyperFlex group.

Any given HyperFlex deployment must be within a single Cisco UCS domain. Except
stretched deployments, which span two domains.

Cisco HyperFlex Software Versions Requirements

Different HyperFlex deployments can run different versions of Cisco HXDP, so the
Fabric Interconnect firmware must be compatible with all the Cisco HXDP versions that
are running on the Cisco UCS domain.

Verify that all the HyperFlex servers also have the same version of Cisco UCS server
firmware installed and consult the HyperFlex installation guide for firmware
compatibility.
HyperFlex UCS M4: For new hybrid or All-Flash deployments, verify that Cisco UCS
Manager 3.1(3k), 3.2(3i), or 4.0(2b) is installed.

HyperFlex UCS M5: For new hybrid or All-Flash or All-NVMe deployments, verify
that Cisco UCS Manager 4.1(2b) is installed.

Note
The compatibility and recommended firmware matrix can change over time, such as
when issues are resolved on newer versions. The versions provided are just examples.
For actual currently supported versions, consult the online documentation provided by
Cisco, such as installation guides and/or Cisco HXDP release notes.

Mixing Cisco HyperFlex and Non-HyperFlex Servers

You can also introduce Cisco UCS servers, which are not a part of a Cisco HyperFlex
deployment, to the Fabric Interconnect domain running HyperFlex. Cisco UCS servers
can be in their own groups and can coexist on the Cisco UCS domain, as long as the
common configuration requirements match.

You can choose to deploy your HyperFlex on pre-existing Fabric Interconnects, already
running another non-HyperFlex deployment. Since the HyperFlex installer changes
certain global settings for the Cisco UCS domain, consult Cisco TAC before you deploy
HyperFlex.

Consider these facts, when mixing HyperFlex and non-HyperFlex servers:

 You can either add HX or non-HX Cisco UCS servers to the Cisco UCS domain.
 The procedure of adding HX to the existing Cisco UCS domain must be done with
the assistance of Cisco TAC.
 Cisco HX Installer overwrites all QoS global system class settings.

The process of installing Cisco HyperFlex will not reboot the existing Fabric
Interconnects, unless the infrastructure firmware is also updated (option in the installer).
The configuration changes it, although that might make it incompatible with the
previous workload.

Note
Adding HyperFlex nodes to an existing Cisco UCS-Fabric Interconnect domain is
supported since Cisco HXDP 2.0(1a).
8.7 Designing Cisco HyperFlex

Cisco HyperFlex and External Storage

This video describes Cisco HyperFlex and external storage. Cisco HyperFlex systems can
connect to the shared storage directly through the fabric interconnects. This solution supports
up to 16 gigs native Fibre Channel and up to 40 gigs Ethernet for iSCSI and Fibre Channel over
Ethernet.

The Cisco HyperFlex installer can partially configure the external storage for you. But the SAN
configuration like zoning and LUN masking must be configured manually. You can always
choose to configure the external storage after the HyperFlex installation. However,
reconfiguring your network for the SAN may require that you have to restart the hardware
running the HyperFlex.
When you verify the external storage connectivity to the Cisco UCS LAN fabric interconnects,
some of the storage design considerations need to be taken into account. Now this figure here
lists these considerations which include northbound storage limitations, port channel support,
storage redundancy, port consumption, manual SAN configuration tasks and separate domains
for each cluster.

It's extremely important that you analyze these caveats closely before connecting the external
storage. Native Fibre Channel connectivity introduces several additional configuration
requirements. The Cisco UCS domain will need a proper configuration to provide connectivity
to the remote storage. This figure here lists the steps for attaching Fibre Channel storage to
the HyperFlex fabric.

As you log into UCS Manager, you have to create a VSAN for Fibre Channel communications
and configure both the worldwide node name and worldwide port name parameters. Then
you've got to create a pair of virtual host bus adapters using the worldwide port pool that you
created and associate them with fabric A and fabric B, respectively. Finally, you have to create
the HyperFlex SAN Connectivity Policy and then assign it to the service profile template that is
going to be used for the cluster.

Switch-based zoning defines zones on the upstream switches. Fibre Channel zoning can be
partially configured on Cisco UCS Manager, which defines them directly on the fabric
interconnects. A physical fabric can have a maximum of 8000 zones. Each zone consists of
multiple zone members, and members can only access each other if they're in the same zones.
However, the devices can belong to multiple zones.

As I said before, the switched-base zoning defines zones on the upstream SAN switches. The
Cisco UCS fabric interconnects are only going to act as the intermediaries. The end hosts acces
s the LUNs in their respective zones defined on the Cisco UCS domain. They then will have
access to other members in their zones, including the external storage in the storage area
network.

Finally, this last figure list the steps for attaching iSCSI storage to the Cisco HyperFlex fabric
interconnect domain. First, iSCSI is an IP-based storage protocol. It basically utilizes the
Ethernet network to provide access to the external storage. After you log in to Cisco UCS
Manager, you create a VLAN, create a pool of MAC addresses for the iSCSI storage.

Next, create a pair of vNIC templates associated with both fabric A and fabric B. Then create a
HyperFlex LAN connectivity policy, assign it to the HyperFlex Servers Profile template used for
the cluster. And lastly, add the network and the storage adapters. That concludes this video.
Thanks for watching.

Cisco HyperFlex Installer can partially configure the external storage for you. The SAN
configuration, such as zoning and LUN masking, must be configured manually.

The installer configures the VLANs for iSCSI and enables iSCSI storage. With Fibre
Channel, the installer defines the VSANs and World Wide Node Name (WWNN) or
World Wide Port Name (WWPN) pool for Fibre Channel address assignment, which
makes external storage configuration easier, but does not completely automate it.

You can always choose to configure the external storage after the HyperFlex installation
but reconfiguring your network for SAN may require you to restart your hardware
running HyperFlex. Usually, this can be done in a rolling fashion.

Cisco HyperFlex can connect to the shared storage directly through the Fabric
Interconnects at these speeds:

 Up to 32-Gbps Native Fibre Channel

 Up to 100-Gbps Ethernet-based protocols (iSCSI and FCoE)
Native Fibre Channel is only available on devices with unified ports (for example, Cisco
UCS 6332-16 and 6454), Cisco UCS 6332 Fabric Interconnect only supports Ethernet.
If you are changing the unified port configuration the Fabric Interconnect you are
configuring will reboot, which could affect an existing workload if you are adding a
HyperFlex system with external storage to an existing Fabric Interconnect domain.

Storage Design Considerations

When configuring external storage connectivity to the Cisco UCS Fabric Interconnect
domain, the individual specifics of the external storage solution will need to be taken
into account.

Consider these design characteristics for SAN connectivity:

 Northbound storage physical connectivity does not support vPC.

 Port channels or trunking is supported to combine multiple uplink ports
 Storage handles the redundancy of storage resources and it varies by vendor.
 Connecting the external storage directly to the Cisco UCS domain increases the FI
port consumption.
 Software configuration including VSANs and zoning is required for providing
access to storage resources.
The storage LUNs that you present to HyperFlex through vCenter should be only used
for migration purposes. While it is possible to use compute from HyperFlex to run VMs
that are utilizing storage that is not HyperFlex, it is not recommended.

When utilizing external storage connectivity, it is imperative to have each group

connecting to the storage in its own domain because the LAN connectivity policy can
differ.

Attaching Fibre Channel Storage to Cisco HyperFlex Fabric

Interconnect Domain
Native Fibre Channel connectivity introduces several additional configuration
requirements, such as world wide name configuration and SAN specifics. Part of the
configuration will have to be performed on the SAN infrastructure, but the Cisco UCS
domain will need a proper configuration to provide connectivity to the remote storage.

Steps for attaching the Fibre Channel storage to the Cisco HyperFlex Fabric
Interconnect Domain:

1. Log in to Cisco UCS Manager GUI.

2. Create a VSAN for Fibre Channel communication.
3. WWNN pool and WWNN block for HyperFlex.
4. Create a fabric-specific (hx-a and hx-b) WWPN pool and WWPN block.
5. Create a pair of vHBA templates with the previously created WWPN Pool
associated with Fabric-A and Fabric-B, respectively.
6. Create a HyperFlex SAN connectivity policy.
7. Assign the HX SAN connectivity policy to the HX Service Profile Template that is
used for the HyperFlex nodes.
Note
You can also attach the Fibre Channel storage to the Cisco HyperFlex Fabric
Interconnect Domain through the HyperFlex Installer during the initial setup.
The Fibre Channel zoning can be partially configured on the Cisco UCS Manager.
Cisco UCS Manager-based zoning defines zones directly on the Fabric Interconnects.
The end hosts can access the LUNs in their respective zones defined directly on the
Cisco UCS domain.

Switch-based zoning defines zones on the upstream SAN switches. The Cisco UCS
Fabric Interconnects only act as intermediaries.

Attaching iSCSI Storage to Cisco HyperFlex Fabric Interconnect

Domain
iSCSI is an IP-based storage protocol that can utilize Ethernet network to provide access
to the available LUNs. iSCSI does not require any special port mode, as the native Fibre
Channel does.

iSCSI configuration for importing storage is configured during the installation and sets
up the iSCSI VLAN configuration where the HyperFlex system acts as the initiator in
SAN communication. You need to configure the target storage appropriately, to enable
HyperFlex access to it.

The Cisco UCS Manager configuration alone will not suffice to enable you iSCSI
mounting on HyperFlex. You will also need to configure the upstream network with one
or more iSCSI VLANs and configure network connectivity to the storage. iSCSI uses
the Ethernet network and requires a valid configuration between initiator and target for
connectivity to establish.

Consider these steps for attaching the iSCSI storage to the Cisco HyperFlex Fabric
Interconnect Domain:

1. Log in to the Cisco UCS Manager GUI.

2. Create a VLAN.
3. Create MAC pool addresses for iSCSI storage.
4. Create a pair of vNIC templates associated with Fabric-A and Fabric-B,
respectively.
5. Create a HyperFlex LAN connectivity policy.
6. Assign the HyperFlex LAN connectivity policy to the HyperFlex Service Profile
Template (SPT) used for the deployment.
7. Add network and storage adapters.

Note
You can also attach the iSCSI storage to the Cisco HyperFlex Fabric Interconnect
Domain through the HyperFlex Installer during the initial setup.
8.8 Designing Cisco HyperFlex

Exporting Cisco HyperFlex Storage with

iSCSI
Cisco HyperFlex can also act as the storage providing LUNs to external initiators over
iSCSI. This type of connectivity is not configured during the installation, but is instead
configured through a workflow in HyperFlex Connect.

You will still need to provide connectivity from the initiators to the Cisco UCS domain
over the iSCSI VLANs.

The HyperFlex iSCSI network needs to be configured after installation:

1. Usage of an iSCSI-dedicated VLAN is recommended.

2. The HX Connect workflow configures the Cisco UCS domain and CVM interfaces.
3. The iSCSI network is attached to the Storage vSwitch (iSCSI Primary).
4. You define CVM IP range and Storage IP for the HX Primary node.
5. The iSCSI network is configured with MTU 9000.

While the Cisco UCS Fabric Interconnects are configured automatically, the upstream
network is not. You need to configure it yourself, before you can connect to the Cisco
HyperFlex iSCSI LUNs.

The iSCSI connection is initiated from the initiator:

1. Initiate the iSCSI connection to the HyperFlex group Storage IP.

2. Initiator IQNs need to be allowed in HX Connect to gain access to a LUN.
3. Windows and Linux initiators are supported as of HyperFlex 4.5(1a).
4. The LUN needs to be created and is stored on the HyperFlex storage.

o All the advantages of HyperFlex storage apply (compression, deduplication,

distribution, resiliency).

5. Direct login to HyperFlex nodes requires MPIO configuration for iSCSI resiliency.
6. You can attach LUNs to workloads running on HyperFlex.

o You still need to enable the iSCSI VLAN on the upstream switches if there are
failures.

7. The same iSCSI configuration is needed for CSI integration. The initiator in the
container pod will need to have access to HyperFlex storage.

The direct login option for accessing the iSCSI LUNs opens a direct connection to the
individual nodes. The initial connection is made through the virtual IP, but the storage
communication channel is directly to the nodes.

The Multi Path Input Output driver provides the ability to use redundant paths for
storage communication. In a MPIO scenario, two iSCSI connections are established
directly to two HyperFlex nodes if one of the nodes fails, the MPIO will failover to the
other node.

The MPIO is provided by the initiator operating system and is controllable from within
the system configuration.

8.9 Designing Cisco HyperFlex

Licensing Tiers
Now let's look at Licensing Tiers and Smart Licensing. Cisco HyperFlex is a combination of
hardware and software, and both must be properly licensed.

HyperFlex Software can be licensed in three incremental tiers-- the Edge Edition, which is the
least expensive, which only includes a HyperFlex Edge deployment; Standard Edition, which is
the most common and includes more features, including the previous tier; and Enterprise
Edition, which is the most expensive and covers specific deployment scenarios. This tier is
required for stretch clusters and for deploying more than a one-to-one converge-to-compute
ratio.

Now, there are also two types of ordering options available for vSphere—the embedded
license, which is a factory-installed product activation code, or PAC, the PAC license, which is a
server-enclosed PAC code.
This next figure here describes a Cisco HyperFlex Software Exchange License feature.
HyperFlex Software Exchange Licenses are intended for users who want to exchange from a
lower edition to a higher edition-- for example, going from the Standard Edition to the
Enterprise Edition, which contains more features. You cannot exchange from a higher license
back to a lower license. Exchanged licenses are supported on Smart Accounts.

You can manage Cisco's Smart Licensing through a Central Customer Portal that is available on
the Cisco Commerce website. A Smart Account is required to access the portal, where you can
manage and activate licenses across your entire Enterprise. The Cisco HyperFlex licenses are
subscription-based, and they are licensed per node and can be transferred between nodes if
needed. If the device has connectivity to the portal, then you can license it directly from the
device. There is a 90-day grace period for the Cisco HyperFlex Cluster License, but this gives
you plenty of time to log in to your Smart Account, generate a token, and register the cluster
before it actually shuts down.

A Renew Authorization Request from the cluster is required every 30 days by a Smart Licensing
, and the authorization is then valid for 90 days. If a cluster is noncompliant, a notification is
sent to the account manager. Smart Licensing also includes the option to provide a local
licensing solution, where Internet access is restricted.

First, if you have to create a Smart Account for your organization, you're going to have to
create one if you don't already have one. Next, the purchase license must be assigned to the
Smart Account for them to actually show up. Also, keep in mind, HyperFlex licenses prior to
version 2.5, those won't work with Smart Licensing. And lastly, make sure all of your licenses
actually show up in the Licensing Portal.

After the licenses are available on your Smart Account and visible in the Licensing Portal, you
can then register your HyperFlex cluster. This includes configuring your HyperFlex
Management VLAN across the Internet and configuring the remote support configuration for
connectivity to the cluster. This can be done via HyperFlex Connect or from the controller VM
command line.
Now you're ready to register your Cisco HyperFlex cluster with Smart Licensing. First, log in to
a controller VM, and then use the stcli license show command to verify that the storage cluster
is in the Smart Licensing mode. Then, on the Cisco Commerce website, log in to the Software
Manager. Choose General and then New Tokens. No matter how many nodes are in the
HyperFlex cluster, only one token usage will be used to register the entire cluster.

The Creation Token dialog box will appear. From the New ID Token row, click the Actions drop-
down list, and click Copy. Then, click Create Token. Now you're ready to go back to the
controller VM and use the CLI to register your HyperFlex storage cluster. Issue the command
stcli license register, two dashes-- that's very specific—and the keyword idtoken. And then
paste in that ID token string. After entering registration token into the device, it then will
connect to the Smart Licensing Portal and attach the device to your account.
Lastly, you should confirm that your HyperFlex storage cluster is registered. From the CLI of the
Controller VM, you can issue the command stcli license show summary. You can also verify the
registration and the portal by navigating to Cisco Smart Software Manager, Inventory, Product
Instances. The license should show up as assigned to the device. And by the way, licenses can
be moved from node to node or in between the clusters if necessary.

If you remember, I mentioned earlier that an organization may have restrictions that do not
allow the clusters access to the Internet. This is typically due to security reasons. So, the Smart
Software Manager Satellite is an on-premise solution. And what it's used for is proxying the
license information. The satellite software is an ISO file downloaded from software.cisco.com,
and it's actually installed on the customer premise and has a subset of the Smart Software
Manager functionality. The satellite periodically updates itself by connecting to the Smart
Software Manager at Cisco.
Finally, in some cases, access to the licensing server at Cisco may not be possible at all. This is
when the security policy of the organization requires air-gapped segmentation of the
infrastructure. In this very extreme scenario, a Permanent License Reservation solution is
available from Cisco. However, it's administered on a case-by-case basis, so it requires a formal
business case review and approval.

That brings us to the end of this video. Thanks for watching.

Cisco HyperFlex is a combination of hardware and software, and both must be properly
licensed. Cisco HXDP software can be licensed in three incremental tiers; the most
common, by far, is the Standard edition.

The less expensive Edge license only includes HyperFlex Edge deployments. The most
expensive Enterprise tier covers specific deployment scenarios and is required for
Stretched deployments, to deploy HyperFlex with a compute-only to converged ratio
beyond 1:1, and HyperFlex Acceleration Engine.

There are four licensing tiers available as of HyperFlex Version 4.5(1a):

 Edge Advantage: HyperFlex Edge HX220c platform

 Edge Premier: HyperFlex Edge HX240c platform and N:1 centralized backup
 DC Advantage: Standard ESXi or Hyper-V Hybrid or All-Flash deployments with
1:1 compute to converged ratio
 DC Premier: Stretched deployments, All-NVMe, 2:1 compute to converged ratio,
acceleration cards
 Subscription is for 1, 2, 3, 4, or 5 years
 There are two types of ordering options available for vSphere:

o Embedded license: product activation code (PAC) factory installed.

o PAC license: Server enclosed PAC code.

The licensing tiers have been reconfigured with HyperFlex version 4.5(1a). An extra
Edge licensing tier has been added (Edge Premier) and the licensing tiers have been
renamed.
The following features apply if you use the Cisco HXDP Software exchange license:

 Upgrade a lower license to a higher license but not vice versa.

 The exchange license is supported on smart accounts.
 When many lower tier licenses are upgraded, the lower license with the shortest
duration defines the validity of the upgraded license.

8.10 Designing Cisco HyperFlex

Cisco Smart Licensing

Cisco Smart Licensing allows the available licenses to be managed through a central
customer portal that is available on the Cisco Commerce website.

Purchased smart licenses are automatically listed in the Smart Licensing Portal. Smart
licenses are then consumed by linking the licensed item with a license on the smart
license portal.

With Cisco Smart Licensing, you can utilize these features:

 Smart licensing automates procuring, deploying, and managing licenses across your
entire organization.
 Cisco HyperFlex licensing is licensed per node and is subscription-based.

o 1–5 year licenses are transferable between nodes.

o Licensing is initiated from the device if the device has connectivity to the smart
licensing portal.
Smart licensing connectivity supports HTTP proxying for added security. Cisco Smart
Licensing also includes the option to provide a local licensing portal, which is synced
with the central Cisco portal once every 90 days. Web connectivity must be available
during the update process.

License Registration
To register a Cisco HXDP license, a grace period will be initiated for your deployment.
The product activation must be completed within this 90-day grace period.

Since Cisco HXDP Version 2.5, Cisco Smart Licensing is auto-enabled and offers a 90-
day grace period:

 During the trial period of 90 days, go to your smart account, generate a token, and
register the system.
 HXDP 2.5(1a) or later PAK keys are convertible to smart licenses by contacting
Cisco TAC.
 The system will periodically renew authorization every 30 days, registration every
90 days.
 After the trial ends, no action is taken by HyperFlex—such as a shutdown.
 Cisco account manager gets a notification of system non-compliance.
After the 90-day trial period expires, your HyperFlex system will be non-compliant.

Note
When you are quoting in Software Subscriptions and Services, you can change the end
dates of your contract. You can also align the end dates of your services to a specific
contract while you add the lines to that contract in a single step, which is known as co-
termination (co-term). For more information on co-terminating licenses, refer to
the Cisco HyperFlex Systems Ordering and Licensing Guide at
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/b_Cisco_HyperFlex_Systems_Ordering_and_Licensing_Guide.pdf.

To register your HyperFlex with smart license, follow these prerequisites:

 Active Smart account for your company.

 The licenses must be assigned to your Smart account.
 Verify that you have the number and type of licenses you have purchased listed in
the licensing portal.
Procedure of Registering Cisco HyperFlex with Smart Licensing
After the licenses are available on your Smart account and visible in the Smart
Licensing portal, you can register your Cisco HyperFlex.

First, you must provide connectivity to the smart licensing portal:

1. Set up your HyperFlex management VLAN to have access to the Internet directly or
through HTTP proxy.
2. Configure the remote support configuration to provide connectivity to the
HyperFlex.
3. Remote support can be configured in the HyperFlex Connect or CVM command
line.
To register your Cisco HyperFlex with Smart licensing, follow these steps:

4. Log in to HyperFlex Connect

5. Click the blue warning link Cluster license not registered in the Dashboard pane
6. From your virtual smart licensing account, choose General and then New Token.
7. Copy the token into the HyperFlex Connect interface prompting you for
the Product Instance Registration Token
The token you generate on the Cisco Smart Licensing portal can be used several times
but will expire in three months. If you have a token registered in your HyperFlex CVM
that is invalid, use the stcli license deregister command from the CVM
command line to deregister the wrong or expired token and enter a new one.

You can have many tokens active at the same time, but each token has limited usage, as
defined upon creation of the token. No matter how many nodes are in the HyperFlex
group, only one token usage will be used up after registering the entire group.

After entering the registration token into the device, the device will connect to the Smart
Licensing portal and attach the device to your account.

You can confirm the license status in several ways:

8. HyperFlex Connect will no longer show a warning on the Dashboard

9. Using the stcli license show summary command from the CVM command
line.
10. You can confirm that your HyperFlex storage is registered in Cisco Smart
Software Manager > Inventory > Product Instances.

Once the registration process is complete, the licenses will be assigned to the devices
that you associated with your account using the generated token.

Licenses can be moved from node to node or between the HyperFlex systems, if
necessary.

Smart Software Manager Satellite

The Smart Software Manager (SSM) satellite is an on-premises (on-prem) solution for
proxying licensing information. The satellite is periodically updated by connecting to
the Smart Software Manager available on the web.

When using the SSM satellite, remember the following key facts:

 Smart Software Manager satellite is installed on the customer premises and provides
a subset of Cisco SSM functionality.
 The SSM satellite needs to periodically connect to SSM online, but otherwise
functions independently.
 You can use SSM satellite in case of:

o Security regulations
o Absence of permanent direct Internet connection
The SSM Satellite is installed from an .iso file that is available on Cisco Software
Central at https://software.cisco.com,

Permanent License Reservation

In certain cases, access to the licensing server is not possible at all. The access is usually
not possible because of a security policy requiring air-gapped segmentation of
infrastructure. In these extreme cases, a permanent license reservation solution is
available, but is administered on a case-by-case basis.

Keep in mind these facts when considering permanent license reservation:

 In an air-gapped environment, there is no way to access Cisco Smart License

servers.
 Permanent License Reservation (PLR) is required for air-gapped Cisco HyperFlex
installations.
 The process of obtaining a PLR license requires a formal business case review and
approval.
8.11 Designing Cisco HyperFlex

Cisco HyperFlex Use Cases

This video describes Cisco HyperFlex positioning. Cisco HyperFlex relies on tried-and-tested
hardware. As we mentioned before, it's based on the Cisco UCS solution, and it uses stable
hypervisors such as ESXi and Hyper-V. HyperFlex inherits the benefits of these solutions and
then supports a wide range of application. It also provides its own benefits to the overall
solution as well.
The Cisco HyperFlex Edge lowers hardware requirements by not including networking in the
solution. It also supports a lower minimum hardware specification when it comes to the
servers. The benefits of using the Cisco HyperFlex Edge solution include reduced cost and
complexity, always-on applications and data protection, faster deployments, and improved
performance.

Although it's optimized for small deployments or environments, it's not really supported with
the Cisco Fabric Interconnects UCS domains, and it does have limited scalability. Cisco
HyperFlex provides several benefits for virtual environments. Not only does it integrate into an
existing infrastructure, which includes vSphere and Hyper-V environments, it can be managed
with VMware vCenter Server and other vendor platforms. Lastly, it provides great
performance, load balancing, and data protection. HyperFlex actually takes over the storage
operations and eliminates the need for a storage area network or SAN storage provider.
The Cisco HyperFlex all flash high-performance storage has great benefits for database
workloads. This solution provides many advantages for the Microsoft SQL Server. It increases
scalability and flexibility. It's going to enhance the database backup and recovery functions as
well as instant space-efficient data cloning.

Cisco HyperFlex All Flash, this system is important for the Microsoft SQL Server because of the
low performance impact, real-time data protection paired with flash storage. All of these
features provides a very stable and redundant database environment.
Cisco HyperFlex is also certified for the SAP HANA environment as well. It provides a single
hardware platform, simplified management, and rapid deployment. The Cisco Container
Platform deployed on HyperFlex makes it easy to configure, deploy, secure, scale, and manage
containers as well, which we'll be talking about later. This makes it easier to take advantage of
modern Enterprise applications like the SAP Data Hub.

Cisco HyperFlex actually offers several benefits to container environments. It supports

persistent mounting of storage through iSCSI proxy directly into the container. This provides
persistence while maintaining data protection features of HyperFlex. It also integrates with the
Cisco Container Platform providing automated deployment of the Kubernetes Container
Management platform. And lastly, Cisco is partnering with Google to offer a unified private
cloud with Google's Public Cloud experience and with multiple containers, which can be move
d between these infrastructures.
Finally, in developmental environments, it's important to have an easy to deploy and manage
infrastructure for testing and development. File-system-based snapshots and clones allow for
very quick provisioning of new virtual machines and quick restoration back to a working state.
This, without the storage and performance overhead that you would actually have using
vSphere-based snapshots. This brings us to the end of our video. Thanks for watching.

Cisco HyperFlex is based on the Cisco UCS solution and hyperconverged software and
uses hypervisors such as ESXi and Hyper-V. The Cisco HyperFlex solution inherits the
benefits of the solutions that it utilizes, and supports a wide range of applications.

Note
As one of the workloads that can be classified under VDI, file services are a use case
validated to run on Cisco HyperFlex. You can offer SMB/CIFS file services on Cisco
HyperFlex solution through third-party solutions like Microsoft Windows and Nexenta.
Use cases to run file share on HyperFlex are home directories and profiles in a VDI
environment. To learn more about file services using Microsoft Windows, read
the Implementing SMB File Services On Cisco Hyperflex Using Microsoft Windows
Server Virtual Machine white paper
(https://www.cisco.com/c/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/white-paper-c11-741863.html). To learn more about file services
using Nexenta, read the Implementing SMB File Services on Cisco HyperFlex Using
NexentaStor white paper
(https://www.cisco.com/c/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/white-paper-c11-742543.html). The HyperFlex Sizer tool can take
into account the space required for the home directory when sizing for VDI.

Hyper-V version of Cisco HyperFlex also supports native SMB share exporting. The
procedure to configure such a HyperFlex instance is available in the Cisco HyperFlex
4.0 for Virtual Server Infrastructure with Microsoft Hyper-V
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hx40_vsi_microsoft_hyperv.html).

In all its varied workloads, Cisco HyperFlex Data Platform adds resilience, security,
scalability, and high performance to the infrastructure.

Use Case Design Considerations

Your use case will determine the type and flavor of deployment you need:

 Different applications (VDI, VSI, test and development, databases/ERP, and

containers) can coexist on the same deployment.

o Might make financial sense to have more than one deployment if a small subset
of applications requires extremely high IOPS, while others do not. In that case,
you might have a small All-NVMe and a larger Hybrid deployment to run your
other workloads.
o Optionally, you can design your deployment with GPUs to accelerate image,
video, or large dataset processing.

 For remote offices and branch offices HyperFlex Edge offers a cost-efficient
solution of 2-, 3-, or 4-node deployment. There are no Fabric Interconnects, servers
connect directly to the top-of-rack switch.
 Cisco design guides and white papers are a good starting point to designing your
own solution on Cisco HyperFlex.

Note
Most of the documents listed below are not on recommended software versions.
However, the guides are still a very valuable tool to deploy and manage Cisco
HyperFlex. Make sure to check for recommended releases and supported software
matrix before you start the design of your own Cisco HyperFlex.

A good, general, design guide on how to deploy and manage the ESXi-based Cisco
HyperFlex solution is the Cisco HyperFlex 4.0 for Virtual Server Infrastructure with
VMware ESXi
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hx_4_vsi_vmware_esxi.html).

For details on deploying VMware Horizon 7 on Cisco HyperFlex solution, read Cisco
HyperFlex 4.0 for VMware Horizon VDI with VMware ESXi for up to 5000 Users
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hyperflex_4_vmware_vdi_esxi_5000.html).

For details on deploying Citrix Virtual Apps and Desktops on Hyper-V HyperFlex,
please read Cisco HyperFlex M5 All-Flash Hyperconverged System with Hyper-V 2019
and Citrix Virtual Apps and Desktops
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hx_af_hyper_v_2019_citrix_4_1b.html).

For detail how to deploy Microsoft SQL on HyperFlex NVMe configuration read Cisco
HyperFlex All-NVMe Systems for Deploying Microsoft SQL Server 2019 Databases
with VMware ESXi
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hyperflex_All_NVMe_4_0_1b_esxi_mssql.html)

When planning to deploy Oracle Database on your HyperFlex NVMe system,

read Cisco HyperFlex All-NVMe Systems for Oracle Database: Reference Architecture
White Paper (https://www.cisco.com/c/en/us/products/collateral/hyperconverged-
infrastructure/hyperflex-hx-series/white-paper-c11-743596.html).

You can find a high-level view about running SAP HANA here: Deploy a Cisco
HyperFlex All-Flash Hyperconverged Infrastructure Solution for SAP HANA
(https://www.cisco.com/c/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/cisco-hyperflex-hci-solution-cascade-lake.html).

For Cisco HyperFlex stretched deployment integration with Cisco ACI, you should look
at Cisco HyperFlex 4.0 Stretched Cluster with Cisco ACI 4.2 Multi-Pod Fabric Design
Guide (https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hx_40_vsi_aci_multipod_design.html).

If you want to learn more about deploying Cisco Tetration on Cisco HyperFlex, read
the Deploy Cisco Tetration Virtual Appliance on Cisco HyperFlex Systems white paper
(https://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-
infrastructure/hyperflex-hx-series/whitepaper-c11-741621.pdf).

Note
Cisco Tetration has recently been rebranded to Cisco Secure Workload. You will still
find references to the solution by old name until the documentation is fully updated.

8.12 Designing Cisco HyperFlex

Graphical Processing Units on Cisco

HyperFlex
Now let's explore the graphical processing units on Cisco HyperFlex. Certain workloads rely
heavily on the unique processing that's offered by graphical processing units or GPUs. Cisco
HyperFlex supports GPUs from two different vendors, AMD and NVIDIA. Consider these facts
when choosing a CPU and a GPU. Typically, CPUs consist of a few cores that are optimized for
sequential processing, whereas GPUs have thousands of smaller cores designed for handling
many calculations simultaneously. This is really important for large data analysis and
calculations. Offloading from the CPU to the GPU just basically adds better performance.
The virtual desktop infrastructure, or VDI, mainly relies on the GPUs for graphical processing of
their user desktops. Without a GPU, graphics are processed by the CPU, which creates more
overhead, and provides a suboptimal user experience. Even if the CPU produces the same
quality of graphic rendering, it's still much lower. With a GPU, CPU cycles are not used for
graphical processes. This means that you get better performance and a much better user
experience.

When it comes to big data and artificial intelligence workloads, this is another area where
GPUs excel. The many parallel processing of a GPU, can analyze much more information than a
more versatile CPU would. This benefits artificial intelligence, machine learning, and deep
learning. In fact, certain solutions rely exclusively on GPUs to produce data. And they don't
support CPU information analysis. Then GPUs become a mandatory part of the solution if this
is the case.
Different workloads require different amounts of GPU processing power. For example,
designing and testing a deep learning application and then running the application when the
data has been analyzed is less GPU-intensive task, while deep learning, which finds patterns in
the data and learns them, now this is more intensive. HyperFlex can be utilized in the
deployment and final application of the GPU acceleration roles where moderate GPU
acceleration is sufficient.

Different workloads from server form factors require different GPU types and different
licenses as well. There are also differences between the HX220 and the HX240 node for GPU
support. So when considering support and compatibility of NVIDIA and Cisco HyperFlex, keep
these main points in mind. All GPU cards must be procured from Cisco. The Cisco HX240
supports one GPU in M4 servers and two GPUs in the M5 server model. And as of version
3.5.1a, HX220s, they just don't support the GPUs.
Each of the hardware graphics cards also need to be licensed to operate properly. So consider
these facts when you're licensing the NVIDIA GPUs. The GPUs obtain licenses on demand from
a locally hosted grid service. The NVIDIA grid provides the virtual CPU sharing, and NVIDIA
vGPU Manager software actually divides the physical GPUs for use by multiple virtual machines
. And if a license is not currently used, it is returned to the pool for other GPUs to use.

Finally, only the number of concurrent users defines the number of necessary licenses. You do
not need to provide a license for each individual user. GPUs are purchased through Cisco but
must be registered with NVIDIA based on their PACs. So when you're obtaining and using
NVIDIA grid licenses, you should follow these steps.

First, you should register your product activation keys with NVIDIA. Next, download the grid
software suite. Then install the grid license server software to a host. Generate licenses on a
NVIDIA licensing portal and then download them. Lastly, manage your grid licenses. And that
brings us to the end of this video. So thanks for watching.

For specific use cases, such as VDI and artificial intelligence application, Cisco
HyperFlex servers support for graphical processing units (GPUs), which can add
additional performance to specific workloads that are running on the HyperFlex
infrastructure.

Certain workloads do not benefit from graphical acceleration, while others rely heavily
on the unique processing that GPUs offer. The key factors are the type of operation
performed and software support.

GPUs are better at processing many less complex calculations than CPUs:

 With a GPU: You can expect better graphical performance especially in 3-D space,
higher resolutions, and better data analysis capabilities.
 Without a GPU: Less responsive interfaces, 3-D graphical workloads are choppy,
lower resolutions, and slow data analysis.
 If applications support GPU acceleration, there is much greater benefit from
integrating GPUs in your design.
 GPUs excel in virtual desktop environments, simulations or the real world (for
example, particle physics), or artificial intelligence (AI) and machine learning (ML)

Graphics cards can also have extreme performance gains in encoding scenarios and
video editing. This specific workload uses the graphical and the data processing
capabilities of GPUs.

Enabling Virtual Desktop Workstations with GPUs

Virtual workstations are the highest tier of GPU workloads. In the NVIDIA lineup,
there are four licensing tiers corresponding to different workloads on the NVIDIA GPU
lineup. Three of which are dedicated to Virtual Desktop Infrastructure (VDI) licensing.

There is direct correlation between licensing and the designs of a GPU enabled
environment:
GRID vApps GRID vPC GRID vDWS NVIDIA vCS

Specific tier dedicated to

Single VM that runs several AI/ML workloads. This type
With Virtual PC Powerful workstation
user instances. Individual of NVIDIA license does not
configuration, each user has machines with up to 48 GB
graphically accelerated use the GRID vGPU
their own full OS, but the of dedicated RAM and full
desktops or applications configuration and is
visual RAM is limited to 2 GB. pass-through support.
can be delivered to users. intended for AI/ML
workloads.

Great for most graphically Great for workloads such

Many small on-demand non-demanding jobs, such as as CAD design, video
All dedicated AI/ML
users sharing the basic email, office word editing, desktop
workloads. Very powerful
infrastructure. Usually for processing, and other office simulations, and other
100 GPU lineup (V100, A100)
a very specific use-case. tasks. Browsing the web can graphically intensive
already be slow. workloads.

Even when a CPU would produce the same quality of a graphical render, the speed with
which it processes the image is much lower. Therefore, it makes a desktop unresponsive
to user input, such as windows closing with delay, window frames rendering before the
window content, and similar anomalies. This option makes for a very sub-par end-user
experience.

The performance difference between a system with GPUs and without it is most
noticeable with fluid movement, such as 3-D animation and video. Here GPUs can take
an unusable system and make it very responsive.

Cisco HyperFlex can offer great benefits to a VDI environment:

 Easily add physical infrastructure for users by adding nodes to HyperFlex.

 Storage is distributed and failure tolerant native VM-level replication is available
 Up to six GPUs can be deployed in a HX240c chassis, two in the HX220c chassis
 There are GPU accelerated blade options available for deployment as compute-only
nodes
 Having data on the same system as the user desktop removes the need to migrate
data to and from the data center
The ability to never move data to the local computer has several significant advantages:

 The data is always physically secure, it cannot be physically stolen, lost, or

damaged.
 All data is permanently in one place and easy to back up and maintain
 Because the data rests on HyperFlex storage, it is resilient, high performance and
fully distributed
 Different devices can be used to access the same virtual desktop and instantly have
data everywhere
 There are specific caching optimization options for HyperFlex storage specifically
designed for VDI

The Virtual Desktop Workstation design can be significantly simplified by using Cisco
RapidDeploy bundles:

RapidDeploy bundles make it possible go from the design phase to a fully functional
HyperFlex VDI environment in under a month. The bundles include different starting
configurations that include HyperFlex storage and additional expansion options with
blade HX or UCS-C series servers.
RapidDeploy program uses the same Cisco and NVIDIA components that can be
configured individually, but remove the need to go through the design and testing phase.
The solution options are validated for a specific number of users running a specific
workload type.

Big Data and Artificial Intelligence Workloads

Processing of large datasets is another workload where GPUs excel. The many parallel
processes of a GPU can analyze much more information than a more versatile CPU
would.

Certain solutions rely exclusively on the GPUs to process data and do not support CPU
information analysis. GPUs are a mandatory part of the solution in these cases.

Several terms are used with data analysis, each referring to a specific application of the
analyzed data:

 Artificial intelligence: Intelligence exhibited by machines.

 Machine learning: An approach to achieve artificial intelligence.
 Deep learning: A technique for implementing machine learning.

o Deep learning is a machine learning technique that learns features and tasks
directly from data.
o Data can be images, sound, or text.
o There is finally enough data, storage, and GPUs for its use.

Different workloads require different amounts of GPU processing power. Designing and
testing a deep learning application and then running the application when the data has
been analyzed is a less GPU intensive task, while deep learning, which finds patterns in
the data and learns them, is much more intensive.

You can use HyperFlex with compute-only nodes for the full application lifecycle from
development to production.

You can use HyperFlex HX-Series nodes for developing your AI/ML algorithm then
you can use dedicated systems such as C480ML as compute nodes for deep learning.
Finally you can apply the learned data on HyperFlex HX-Series nodes.

NVIDIA GRID
GRID is an NVIDIA GPU virtualization technology that allows a single CPU to be
segmented into many smaller GPUs (called vGPUs) that can be assigned to individual
VMs. This process has very little overhead and provides an interface with the VM that
is similar to the physical GPUs being available.

An analogy to vGPUs in Cisco world are VICs. Just like VICS segment a single
physical GPU into several virtual NICs, vGPUs segment one physical GPU into several
vGPUs.

The amount of resources you can assign to each vGPU and its feature set are defined by
the licensing tier that assigned to that vGPU instance.

NVIDIA GPU licenses are assigned on the fly from a licensing pool:

 GPUs obtain licenses on-demand from a locally hosted GRID licensing service that
holds the license pool
 NVIDIA GRID allows dynamic allocation of physical GPUs to VMs as vGPUs
 The NVIDIA vGPU Manager software, which is installed onto the hypervisor,
divides the physical GPU for use by multiple VMs on the host.
 If a license is not currently used, it is returned to the pool and is available to other
VMs.
There are four different licensing tiers for NVIDIA GPUs (vApp, vPC, vDWS, and
vCS). Only the first four are licensed as vGPUs.

The GPUs are purchased through Cisco but must be registered with NVIDIA based on
their product activation keys (PAKs).

When obtaining and using NVIDIA GRID licenses, follow these steps:

1. Register your product activation keys with NVIDIA.

2. Download the GRID software suite.
3. Install the GRID License Server software to a host.
4. Generate licenses on the NVIDIA licensing portal and download them.
5. Manage your GRID licenses.

With each NVIDIA software license, you also need to purchase support during the
software license. There are two types of software licences: per-year and perpetual. 5
years of support needs to be purchased with each permanent license.

You can purchase the GPUs and their licenses from Cisco. You only need to choose the
duration and type of the software license; the support will be assigned automatically
according to the software licensing you chose.
8.13 Designing Cisco HyperFlex

Designing Multicloud Data Center with

Cisco HyperFlex
The Cisco HyperFlex solution is often deployed in complex environments with
multivendor equipment or in multi-cloud deployments. The emerging data center
architecture owes most of its success to the automation of routine tasks. Automation
tools allow you to effectively manage and quickly provision the capacity that you have
installed. There are just too many components to care for each one manually.

With the rise of the Intersight cloud management system, a lot of the cloud
functionalities of the Cisco data center have been moved to Intersight, although not all
systems have a full feature set of the on-premises systems. Most notably this applies to
the Cisco UCS Director.

Note
Customers get a free Cisco UCS Director license with the highest tier of Cisco
Intersight license to resolve the feature disparity. Intersight is developing rapidly and is
supporting more and more of the orchestration capabilities.

 Cisco Intersight-based systems:

 Intersight Workload Optimizer: Application optimization engine

 Intersight Kubernetes Service: Kubernetes deployment and management
system
 Intersight management platform: Central management and automation system
for hardware and software

 Stand-alone systems:

 Cisco CloudCenter: Hybrid multi-cloud management system

 Cisco UCS Director: Complete end-to-end management and orchestration
solution

Automation extends to the software layer where complex systems can be configured
once and then rolled out in real time as needed, using cloud automation tools. Intelligent
system architecture can balance the load among compute, network, or storage resources,
bringing systems online or offline as demand dictates.

Note
You will only explore the stand-alone solutions here, the Intersight solutions are
addressed in the Intersight content.
Cisco UCS Director Overview

This video provides an overview of Cisco UCS Director. Cisco UCS Director is a 64-bit virtual
appliance for VMware VSphere, or Microsoft Hyper-V deployments. Cisco UCS Director
is used for a wide array of data center infrastructure components. It also automates and
simplifies the entire IP process.

This is done by abstracting hardware and software into programmable tasks. Organizations can
actually build automated workflows. This removes silos and allows the IT to manage the entire
infrastructure just using a single pane of glass. Cisco UCS Director also increases operational
efficiency by reducing the amount of time is spent on troubleshooting and bug fixes.
This figure here shows the high level tasks that any typical server administrator will perform
on a day-to-day basis. UCS Director provides the availability to automate a wide array of these
tasks and use cases. It actually focuses on delivering a secure end-to-end management
orchestration and automation solution.

This is performed across a broad variety of supported Cisco and third party hardware, as well
as software data center components. It also supports provisioning and management of
physical and virtual servers, as well as dynamic network technologies. Cisco UCS Director
allows you to manage different types of network, compute, and storage platforms.

You can create and deploy different policies that are related to the network compute and
storage to control and manage them. Both virtual and physical accounts are associated with
sites and pods. These sites and pods define the aggregation points for the tenant resources.
Virtual and physical accounts are designed for a given site and pod and the virtual data center.
So some example of physical accounts include Cisco UCS Manager and Cisco HyperFlex. Lastly,
another example of virtual accounts would be VMware vCenter, which manages the Virtual
Computing environment and the virtual machines. The virtual data center, or VDC, not to be
confused with virtual device contexts, but this Virtual Device Center Creation Wizard
automatically discovers a newly installed HyperFlex System.

It creates VDC resources by assigning policies for a group of users and management of multiple
HyperFlex clusters. With configured physical account, UCS Director discovers the HyperFlex
components. Cisco UCS Director provides a self-service web portal, where the VMs are
provisioned from a pool of assigned resources.

It will use pre-defined policies set by administrators. This allows infrastructure and tenant
administrators, as well as end users to perform various actions on the physical and virtual
infrastructure. You can create service logs, self-service provisioning, self-service dashboard,
and management to manage the servers and applications.
Finally, Cisco UCS Director Orchestrator enables cloud automation and standardizing of the IT
services. Orchestrator is used to create a workflow, which is a list of tasks. And this enables the
IT administrator to automate cloud deployment and provisioning, as well as standardized IT
services. That brings us to the end of this video. Once again, thanks for watching.

Let's continue with part two of the overview of Cisco UCS Director. UCS Director provides a set
of simple, predefined workflows for HyperFlex. These workflows are designed to perform a
single task or to automate more complex provisioning or management tasks. The figure here
illustrates the tasks that are available right out of the box in Cisco UCS Director Workflow
Designer.
These building blocks allow you to perform operations on various levels in the HyperFlex
environment, including data stores, hosts, VM, system, and networking. The simplest workflow
consists of two connected tasks, OK? A task represents a particular action or operation. The
workflow determines what order in which Orchestrator executes these tasks. The figure
illustrates an example workflow, which has performed two tasks.

First, a Cisco HyperFlex data store is created, using predefined tasks. Then a VM is deployed on
the provision data store output, using the output from the first step. So first you need to add a
custom workflow. It's not really shown in this figure. But you can specify the workflow library
folder in which you want to save the workflow or create a new folder. You can define the work
flow inputs and outputs when adding the workflow or do it later within the Workflow Designer
. In this case, you don't have to define any input or output.
To add the desired tasks to the workflow, you should first find them in the pre-configured task
library. Now, you can either use the search feature or just navigate to the appropriate folder
and move them to the right pane in the Workflow Designer. The HyperFlex related tasks are
found by navigating to physical storage tasks and HyperFlex Tasks folder.

Cisco UCS Director offers dozens of pre-configured HyperFlex related tasks for data store and
VM operations, such as cloning and making VM snapshots and so on. When you drop a task
into the Workflow Designer, a Task Configuration Wizard will prompt you for the input and
output definitions. You can map the inputs to the values returned from the other tasks. You
can configure fixed values or you can prompt the users for input information, so that when
they run the task, it'll say what do you want? And they have to put in the value.
In this example, the first task is configured with fixed parameters such as the data store name,
sales-ds, and the size, 100 gigs. You can find the task VMware-provisionablankvm, you can find
that in the virtualization VMware folder. When you drop this task into the Workflow Designer,
the two tasks will exist side by side. And they will not be connected into any task sequence.
You're going to have to later link them into the desired order.

Once you drop that second task in the Workflow Designer, this is when the Configuration
Wizard will open, prompting you for the input output definitions. In this scenario, we want to
route the output of the first task, which is the data store ID, as well as the input of the second
task. Next, you should check the respective map to user input checkboxes and choose the
desired variable from the dropdown list.
In this case, only one variable is available for selection, as you can see in the figure. You should
also check the map to user input checkbox if the workflow should prompt users for
information at the time of execution.

Each task has two outcomes, completed success and completed failed. You should use the
mouse to connect each outcome to the respective next step. The simplest workflows, like the
one shown in this figure, route the completed success to the next task and from the final task
to the Completed Success tab, while the completed failed outcomes are routed to the
Completed Failed tab.
Now, obviously, you can design more complex blows with forks and nested branches and so on
and so forth. But that's not the goal of this video. We just wanted to provide you with an
overview. Now, having validated the workflow, you can execute it and observe it's processing
progress in the workflow status of the respected service request.

A service request is created whenever a workflow is executed. If you have any defined user
inputs, you would first need to provide that input information. The Workflow Status window
includes other tabs that can help you identify the source of any potential problems in your
workflow, such as log, objects created and modified, and input and output, and so on.

Finally, after the workflow example completed successfully, you can verify the provision data
store and VM in the HyperFlex cluster. You can use any tool of your choice for the verification.
As you can see in the figure, data stores sales-ds, and VM sales front end have been
successfully provisioned. That brings us to the end of this video. Thanks for watching.
Cisco UCS Director is a 64-bit virtual appliance that focuses on delivering IaaS through
a highly secure, end-to-end management, orchestration, and automation solution for a
wide array of Cisco and non-Cisco data center infrastructure components.

Cisco UCS Director automates and simplifies IT processes. It allows the IT staff to
provision, pool, and schedule resources for delivery within minutes, delivering
significant cost and time savings. By abstracting hardware and software into
programmable tasks, IT organizations can build automated workflows for virtual,
physical, converged, and hyperconverged infrastructure solutions. Cisco UCS Director
increases operational efficiency by validating configurations and resource assignments,
reducing the time spent on troubleshooting and bug fixes. This holistic management
helps to make processes consistent, reliable, and predictable.

Cisco UCS Director has the following characteristics:

 64-bit appliance for VMware vSphere or Microsoft Hyper-V

 Cisco UCS Director provides the automation and orchestration foundation for
private cloud:

o Provisions physical/virtual compute, network, storage, and hypervisor resources

o Enables self-service in compliance with IT policies and approvals.

 It improves IT operational efficiency.

o Replaces managing each layer individually with automated workflows

o Removes silos by allowing IT to manage infrastructure through a single pane of
glass
o Reduces manual activities to allow resources to focus on value-add services for
business

Because your data center comprises diverse technologies, Cisco UCS Director delivers
heterogeneous infrastructure management with the following benefits:

 Ability to automate a wide array of tasks and use cases across a broad variety of
supported Cisco and third-party hardware and software data center components
 Automated setup of VCE VBlock, NetApp FlexPod, IBM VersaStack, and Pure
FlashStack converged infrastructure
 Day-0 setup and day-1 definition and deployment of Cisco UCS servers, Cisco
HyperFlex hyperconverged infrastructure, and Cisco Nexus switches
 Provisioning and management of physical and virtual switches and dynamic
network technologies
 Automated provisioning and management of storage virtual machines, filers, virtual
filers, LUNs, and volumes
 Discovery, mapping, and monitoring of physical and logical data center topologies
 Unified management of Cisco UCS domains

Note
For a complete list of supported infrastructure components and solutions, see the Cisco
UCS Director Compatibility Matrix (https://www.cisco.com/c/en/us/support/servers-
unified-computing/ucs-director/products-device-support-tables-list.html).
Cisco UCS Director helps deliver IT infrastructure through a single interface with point-
and-click simplicity, which reduces the overall deployment and provisioning time and
operating costs. The figure lists the high-level tasks that any typical server administrator
will perform on a day-to-day basis.

Cisco UCS Director provides out-of-the-box integration with virtual and physical
components:

 Hypervisors such as VMware vSphere, Microsoft Hyper-V, and RedHat KVM

 Compute servers and devices such as Cisco UCS, HP, and Dell servers
 Network devices such as Cisco Nexus and Brocade
 Storage components such as NetApp, EMC, and IBM Storwize
 Hyperconverged storage solutions, such as Cisco HyperFlex, VMware VSAN

Without infrastructure automation, consistency and compliance are difficult to achieve.

Human error is the number one reason for downtime in the data center. Downtime
happens when administrators divert their attention to too many tedious and repetitive
manual tasks that could be performed better through automation.

Managing Infrastructure in Cisco UCS Director

Cisco UCS Director allows you to manage different types of network, compute, and
storage platforms. It provides a single pane of glass to integrate and utilize different
resources by adding them as physical or virtual accounts. Once the devices/accounts are
added in Cisco UCS Director, you can create and deploy different policies that are
related to network, compute, and storage to control and manage them. Virtual and
physical accounts are associated with the site and pod.

Three logical entities of Cisco UCS Director:

 Sites and pods

o Aggregation points for tenant resources—hierarchy can reflect the physical

organization and locations of your DC.
 Physical account

o Example: Cisco UCS and Cisco HyperFlex

o Manages physical servers, storage, and networking

 Virtual account

o Example: VMware vCenter

o Discovers and monitors virtual computing environments
o Applies policy-based configurations
o Performs trending and capacity analysis
o Manages VM lifecycle

You create sites and pods to define the aggregation points for the tenant’s resources.
Virtual and physical accounts are configured for a given site, pod, and virtual data
center (VDC).

Physical server management includes Cisco UCS Manager and other physical server-
related accounts. Cisco UCS Director allows you to discover and collect information
from these accounts. Virtual computing management allows the discovery and creation
of public and private clouds.

A physical account for Cisco UCS provides the foundation for physical compute,
network, and storage services for virtual infrastructure. A virtual account handles the
cloud services and virtual infrastructure.

Cisco UCS Director performs the discovery and collects all the data that is related to the
computing environment—for example, from a VMware vCenter. Once the virtual
account is added, you can perform policy-based provisioning and manage the VM
lifecycle—for example, add vNICs or delete virtual disks.

Cisco HyperFlex Converged Platform in Cisco UCS Director

The integration between Cisco UCS Director and Cisco HyperFlex System begins after
you first install the Cisco HyperFlex HX-Series node. Next, you install and configure
the Cisco HyperFlex System software. Then you create the Cisco HXDP deployments
in VMware vCenter.

After you have configured the integration, Cisco UCS Director can communicate with
the components of a supported Cisco HyperFlex System:

 VMware vCenter
 Cisco UCS servers that are managed by Cisco UCS Manager
 Cisco HXDP

The VDC Creation Wizard automatically discovers a newly installed HyperFlex

System. It creates VDC resources by assigning policies for a group of users and
management of multiple HyperFlex deployments. With configured physical account,
Cisco UCS Director discovers HyperFlex components.

You can use Cisco UCS Director to manage these areas for a supported and integrated
Cisco HyperFlex System:

 Inventory collection
 Discovery of systems, disks, datastores, and controller VMs
 Datastore provisioning and management
 Automation and orchestration of VM provisioning

o You can use a VDC to provision a VM on a HyperFlex pod through the

standard Cisco UCS Director VM provisioning process
o You can use ReadyClone VMs to provision multiple VMs at a time based on
the same VM template
 Status reporting for a pod (for example, active VM distribution per pod and total
storage capacity) and/or its components and performance over time

Self-Service Portal
Cisco UCS Director provides a self-service portal where VMs are provisioned from a
pool of assigned resources, using predefined policies set by administrators. It is a web
application that allows infrastructure administrators, tenant administrators, and end
users to perform various actions on the physical and virtual infrastructure.

Self-service portal provides service catalogs, self-service provisioning, self-service

dashboard, and management to create, deploy, and re-configure servers and
applications.

With the self-service portal, users can access VMs that are provisioned from a pool of
assigned resources using predefined policies.

After you request one of the services that are available to you, the end-user portal
completes the service request workflow that is configured by your administrator. This
workflow may include approval of your self-service provisioning request, assignment of
the necessary compute, storage, and network resources, and configuration of security
and performance settings. After your service is provisioned, you can track the status of
your services through the summary dashlets and reports on your landing page, and
through the service request report that is available within the end-user portal.

As an end user, based on your administration setup, you can perform one or more of
these operations:

 Provision VMs, application-specific infrastructure, and bare-metal servers.

 Review and manage your service requests.
 Upload and deploy OVFs and other images.
 Monitor and create reports for your provisioned virtual and physical resources.
 Approve service requests to provision infrastructure.
 Additional functionality may be available to you if your administrator provides you
with the necessary permissions.

Self-service portal has these attributes:

 Enables Cisco UCS Director to achieve the key objective of the self-service and
reduce the administrative burden by pushing the tasks to end users
 Enables IT organizations to create IaaS and offer it as a self-service to end users in
end-user departments
 Allows the creation of complete RBACs for the members of groups and departments
automating the request and approval process
 Allows end users to create catalogs and manage the lifecycle of the provisioned
resources, minimizing the manual intervention of data center processes

System administrators create catalogs, which allow self-provisioning of VMs. A catalog

belongs to a particular group and only the users of that group can view the particular
catalog in their portal. Catalogs can be of two types: standard and advanced. Standard
catalogs work with provision templates that are available in a particular virtual account.
With advanced catalogs, you can actually have workflow items in the form of self-
service packages.

Self-provisioning of VMs is done by using predefined catalog items. Self-provisioning

is based on VM templates—for example, images of the VMs that are used to provision
new VMs.

Cisco UCS Director Orchestration

Cisco UCS Director enables you to automate a wide array of tasks and use cases across
various supported Cisco and non-Cisco hardware and software data center components,
including physical infrastructure automation at the compute, network, and storage
layers.

Here are a few examples of the use cases that you can automate:

 VM provisioning and lifecycle management

 Network resource configuration and lifecycle management
 Storage resource configuration and lifecycle management
 Tenant onboarding and infrastructure configuration
 Application infrastructure provisioning
 Self-service catalogs and VM provisioning
 Bare-metal server provisioning, including installation of an operating system

For each process that you decide to automate with orchestration workflows, you can
choose to implement it in any of these ways:

 Use the out-of-the-box workflows provided with Cisco UCS Director.

 Modify the out-of-the-box workflows with one or more of the tasks provided with
Cisco UCS Director.
 Create your own custom tasks and use them to customize the out-of-the-box
workflows.
 Create your own custom workflows with custom tasks and the out-of-the-box tasks

Cisco UCS Director Orchestrator enables cloud automation and standardizing of IT

services. Orchestrator is used to create a workflow, which is a list of tasks.

Use cases for Cisco UCS Director Orchestration are listed here:

 Tenant onboarding
 Infrastructure provisioning
 Application provisioning
 Capacity scale
 DR automation

The Cisco UCS Director Orchestrator (also called Cisco UCS Director Orchestration
Engine, or just Orchestration) enables IT administrators to automate cloud deployments
and provisioning and to standardize IT services.

At the scale of cloud computing, performing actions manually, such as creating VMs
and provisioning networks, is prohibitively time-consuming. In Cisco UCS Director,
these actions or tasks are executable elements that can be run from the GUI. Cisco UCS
Director Orchestrator further automates these complex tasks by organizing them into
workflows.

Cisco UCS Director Orchestrator works by executing a series of scripted actions called
tasks. Each task performs one action. By connecting tasks so that the input of one task is
the output of a previous task, you build up a workflow to automate administrative
processes, such as creating VMs, provisioning bare metal servers, setting up storage,
compute, and network resources, and many others.

Workflows can be rolled back to a state identical or similar to the state before the
workflow was executed. You can perform a rollback, for example, to remove virtual
components that were created in error.

Cisco UCS Director Preconfigured Cisco HyperFlex Tasks

Cisco UCS Director includes orchestration workflows and tasks that enable you to
automate common VM provisioning and Cisco HyperFlex management tasks in one or
more workflows. You can create a workflow that combines HyperFlex tasks with
VMware host tasks and Cisco UCS tasks for Cisco UCS Manager.

Depending on the permissions that are required to perform a task, you can create
workflows to be executed in Cisco UCS Director by an administrator or in the end-user
portal by a user. For example, a workflow to provision ReadyClone VMs requires
administrator permissions and cannot be executed by an end user.
Cisco UCS Director provides a set of simple, predefined workflows for HyperFlex.
These workflows are designed to perform a single task, such as creating ReadyClone
VMs or creating a datastore.

If you want to automate more complex provisioning or management tasks, make a copy
of a predefined workflow and add tasks to the copy of that workflow. You can also
create your own custom workflows that include HyperFlex tasks.

The figure illustrates the tasks that are available out of the box in the Cisco UCS
Director Workflow Designer. These building blocks allow you to perform operations on
various levels in the HyperFlex environment: datastores, hosts, VMs, system, and
networking (both VMware vSphere and Cisco UCS).

Cisco UCS Director: Cisco HyperFlex Workflow Example

The simplest workflow consists of two connected tasks. A task represents a particular
action or operation. The workflow determines the order in which the Orchestrator
executes your tasks.

You build workflows using Workflow Designer, which is a drag-and-drop interface. In

Workflow Designer, you arrange tasks in sequence and define inputs and outputs to
those tasks. Outputs from earlier tasks are available to use as inputs to any subsequent
task. Complex workflows are created by connecting multiple tasks to a sequence.

As an example, create a workflow that performs these two tasks:

 Create a Hyperflex-backed datastore.

o Apply the task preconfigured in the HyperFlex task library.

o Input values provided by the user or hardcoded.

 Deploy a VM on the provisioned datastore.

o Feed the returned datastore ID as the task input.

o Use one of the tasks from the VMware library.
In this example workflow, you create a Cisco HyperFlex datastore and then deploy a
VM on it. You build the workflow with predefined tasks, so that the output of the first
task (the datastore ID) is the input to the second task.

A custom workflow has these characteristics:

 May contain one or more preconfigured or custom tasks

 Can be exported for general use

First, you need to add a custom workflow. Although not shown in the figure, you can
specify the workflow library folder in which you want to save the workflow or create a
new folder. You can define the workflow inputs and outputs when adding the workflow
or do it later (within Workflow Designer) when adding specific tasks to the workflow.
In this case, you do not define any inputs or outputs.

To add the first task to the custom workflow, perform these two steps:

1. Drag and drop the task from the appropriate library folder to the workflow designer.
2. When the task is dropped, the task configuration wizard will prompt for more data.
To add the desired tasks to a workflow, first you should find them in the preconfigured
task library. Either use the search feature or navigate to the appropriate folder and move
them to the right pane, to Workflow Designer. HyperFlex-related tasks are located in
the Physical Storage Tasks > HyperFlex Tasks folder. Cisco UCS Director offers
dozens of preconfigured HyperFlex-related tasks, for datastore and VM operations, such
as cloning and making VM snapshots.

In the third step, you need to provide user input mapping and task inputs:

 User input mapping: Define if users will be prompted for input data.
 Task inputs: Define fixed input data (users will not be prompted).

When you drop a task in the Workflow Designer panel, a task configuration wizard
prompts you for input and output definitions. You can map inputs to values returned
from other tasks, configure fixed values, or prompt the users for input information. In
this example, you configure the first task with fixed parameters, such as the datastore
name (Sales-DS) and size (100 GB).

Next, you add the second task to your workflow:

1. Choose a VM provisioning task from the Virtualization folder.

2. Drag and drop the task from the library to the workflow designer.
You can find the task (VMware—Provision a Blank VM) in
the Virtualization > VMware folder. When you drop this task into the workflow
designer, the two tasks will exist side by side, and they will not be connected into any
task sequence. Later, you will link them in the desired order.

User input mapping chooses the output from the previous task as input. All other inputs
can be user-provided or hardcoded.

Once you drop the second task into the workflow designer, the configuration wizard
will open, prompting you for I/O definitions. In this scenario, you want to route the
output of the first task, the datastore ID, as the input of the second task. Next, you
should check the respective Map to User Input check box and choose the desired
variable from the drop-down list. In this case, only one variable is available for
selection, as shown in the figure. You should also check the Map to User Input check
box if the workflow should prompt users for the information at the execution time.

Having added the tasks to the workflow designer, you need to define the order in which
they are executed.

When you link tasks, remember:

 Use the connectors to define the order of tasks.

 Define the sequence of events to reach:

o Completed (Success)
 o Completed (Failed)

 Workflow designer allows you to validate the workflow syntax.

Each task has two outcomes: Completed Success and Completed Failed. You should use
the mouse to connect each outcome to the respective next step. The simplest workflows,
such as the workflow shown in the figure, route the Completed Success to the next task,
and from the final task to the Completed Success tab, while the Completed Failed
outcomes are routed to the Completed Failed tab. You can design more complex flows
with forks or nested branches.

When the workflow design is complete, use the validation feature to check for any
syntax errors in the I/O routing.

After you create a workflow, use Cisco UCS Director to execute it.

You can monitor the execution progress in the Log, Objects, and Input/Output tabs.
Having validated the workflow, you can execute it and observe its processing progress
in the Workflow Status of the respective service request. A service request is created
whenever a workflow is executed. If you had defined any user inputs, you would first
need to provide the input information. The workflow status window includes other tabs
that can help you identify the source of any potential problems: Log, Objects
Created/Modified, and Input/Output.

You can go to the service request list at any time, browse through the workflows
executed in the past, examine their results, and perform other functions, such as
rollback. These operations can also be made available to the end users through the self-
service portal.

After the workflow example completed successfully, you can verify the provisioned
datastore and VM in the HyperFlex system. You can use any tool of your choice for the
verification. As you can see from the following figure, Datastore (Sales-DS) and VM
(Sales-Frontend) have been provisioned.

You can verify the workflow results using a tool of your preference. Cisco UCS
Director can also be used to display the discovered components. The figure illustrates
how the datastore and VM appear in HyperFlex Connect.
Cisco CloudCenter Suite Overview

This video provides an overview of Cisco CloudCenter. Cisco CloudCenter features a secure
hybrid management platform. From here, you can quickly and easily model, deploy, and
manage applications in any environment.

You can migrate existing applications or onboard new applications to your cloud infrastructure.
And lastly, CloudCenter automates HyperFlex workload between public and private resources,
such as HyperFlex.
CloudCenter can be integrated with Cisco HyperFlex in a bundle that provides a single portal to
provide and manage the workloads. In turn, HyperFlex-- well, it actually turns HyperFlex into
what we call a double scale-out architecture.

It also delivers a cloud experience, including deployment and management of VMs. You
actually get a dropdown menu that lets you pick the workload and choose where you want to
deploy the workload.

Finally, the Cisco HyperFlex solution with Cisco CloudCenter can optimize a hyper-converged
infrastructure in two very powerful ways-- first, self-service, which includes automated
workflow deployments in different environments. This includes HyperFlex, public cloud
environment, and a cross-cloud application lifecycle deployment; and second, capacity
optimization, which includes load balancing and to automate temporary workloads.
Each of these use cases includes an easy starting point and additional levels of capacity. That
brings it to the end of this short video. Thanks for watchi

Cisco CloudCenter Suite features a secure hybrid management platform that delivers a
full lifecycle approach to deploying and managing your data center or your public or
private cloud environment. You can quickly and easily model, deploy, and manage
applications in any environment, whether you are deploying simple or complex
workloads to one or many environments.

Your multi-layered applications deployed on Cisco HyperFlex storage are abstracted

from the underlying cloud environment, enabling your infrastructure to adapt to meet
your deployment and management requirements.

Cisco CloudCenter Suite is a virtual appliance that performs these six tasks:

 Allows you to model, deploy, and manage applications with unified administration
and governance of clouds and users
 Automates workloads across private and public clouds
 Cloud-vendor agnostic user interface
 Allows workload sharing between public and private resources, such as HyperFlex
 Hides cloud-specific settings from the user
 Mainly targeted at cloud admins and developers

The easiest way to install Cisco CloudCenter Suite components is to use the virtual
appliance method. Most Cisco CloudCenter Suite customers use virtual appliances to set
up all the CloudCenter Suite components, because they can use pre-packaged and
Cisco-certified virtual appliances.

Cisco CloudCenter Suite Model enables you to build and manage cloud-agnostic
application profiles and repositories. You can then determine and confirm your
architectural requirements and environment type profile. Cisco CloudCenter Suite
Deploy enables you to migrate existing applications or onboard new applications to
your cloud infrastructure. Cisco CloudCenter Suite Manage enables you to manage
application deployments and perform ongoing operations using Cisco CloudCenter
Suite. The Administer and Govern feature of Cisco CloudCenter Suite enables you to
manage and govern multiple tenants (organizations) and users (or groups of users). This
type of multi-tenant, multi-user governance provides for setting granular permissions
and controls for objects within the Cisco CloudCenter Suite framework.

Your application is abstracted from the underlying cloud environment, therefore,

enabling your infrastructure to adapt to meet your deployment and management
requirements. Cisco CloudCenter Suite allows you to manage cloud accounts and
permissions, set financial and use plans, and report on use and costs. You can also
manage tenants and users through multitenant management capabilities and RBAC.

Integrating Cisco CloudCenter Suite with Cisco HyperFlex

CloudCenter Suite can be combined with Cisco HyperFlex in a bundle that provides a
single portal to deploy and manage workloads in HyperFlex and the cloud.

HyperFlex is known for its scale-out architecture that makes it easy to add the compute
and storage nodes, and resources to workloads within a node. But that can be limited to
on premises infrastructure, and isolated from a hybrid IT service delivery strategy.

CloudCenter Suite turns HyperFlex into a double scale-out architecture. The ease of
adding compute and storage resources on premises in HyperFlex is combined with
"Scale out" by deploying workloads to the public cloud. Since the Cisco CloudCenter
Suite architecture abstracts the underlying nuances of both vSphere on HyperFlex and
your choice of public cloud, you can choose where to deploy workloads. IT can set
policies that tap public cloud capacity on demand, on schedule, or based on compliance
or security rules. You can optimize a changing mix of infrastructure utilization and your
cloud bill.

HyperFlex with Cisco CloudCenter Suite delivers a cloud experience—including

deployment and management of VM or full application stack—without the cumbersome
IT consumer experience of the past. You get a drop-down menu that lets you pick the
workload and choose where to deploy it. A simple tag-based policy engine provides
guardrails that guide automation and user decisions as they work across both HyperFlex
and public cloud environments.
 Cisco HyperFlex with Cisco CloudCenter Suite Use Cases
The HyperFlex solution with Cisco CloudCenter Suite can optimize a hyperconverged
infrastructure in two powerful ways. Each use case includes an easy starting point and
additional levels of capability:

 Self-service and automated workload deployment in different environments:

o HyperFlex
o Public cloud environment
o Cross-cloud

 Capacity optimization

o CloudCenter Suite application-centric automation and orchestration enable

additional capacity optimization:
o Load balance legacy workloads
o Automate temporary workloads
o Burst to cloud
o Automate hybrid topology

Self-Service
Users get the on-demand experience that they expect in the cloud era. Users avoid the
IT ticket-and-wait processes that are often used to consume infrastructure services.
Instead, they can help themselves by using automated workload deployment and
common day-2 management tasks.

You can use the automated workload deployment in the following three environments:

 HyperFlex environment: Self-service deployment of a simple virtual machine or a

fully configured application in your Cisco HyperFlex environment. Users can also
perform common day-2 management actions on their workloads without the need for a
helpdesk ticket. The IT department can also apply policy-based usage controls to limit
use to fit available resources and terminate workloads after a pre-set time period.
 Public cloud environment: Users can deploy the same virtual machine or application
in the public cloud. The IT department can apply policy-based placement and cloud cost
controls, along with getting roll-up and drill-down use and cloud cost reporting.
 Cross-cloud application lifecycle deployment: Users can start development and test
efforts in the cloud, and then redeploy the workloads back to the Cisco HyperFlex
environment.

Capacity Optimization
You can use application orchestration and management automation to optimize the
Cisco HyperFlex platform for both long-running and temporary workloads. Cisco
HyperFlex systems alone enable users to easily scale computing and storage resources
separately, by adding more nodes or scaling up the cache or capacity within a node.

CloudCenter Suite application-centric automation and orchestration offer additional

capacity optimization:

 Load balance legacy workloads: Deploy more instances of a workload or

workload tier during periods of heavy use. Automatically load balance between
them and then deprovision as use wanes. The IT department maintains workload
performance while avoiding overprovisioning resources.
 Automate temporary workloads: Automate both deployment and end-of-life
actions based on policies. Stand up and tear down workloads quickly and easily
based on user request, API call from another tool, or run-time policy.
 Burst to cloud: Temporarily deploy workloads to the public cloud during periods of
heavy use. Supplement on-premises resources with pay-per-use cloud resources only
when needed.
 Automate hybrid topology: Use multiple environments when deploying an
application placing the web or application tier in a pay-peruse cloud, while
deploying the database tier back in the Cisco HyperFlex environment.

8.14 Designing Cisco HyperFlex

HyperFlex: Releases Beyond 4.5(1a)

Now let's look at HyperFlex releases beyond 3.5(1a). HyperFlex Acceleration Engine is an
optional card that helps offload the compression function and can save controller VM CPU
cycles.

This card is not a single point of failure as the controller VM is, and it is able to pick up the
functions should the card fail. Of course, then the performance benefits would be lost in that
particular case. But also, you can provision multiple cards for redundancy. This optional option
needs to be installed in all the nodes and can be used only in new installs.
The HyperFlex Acceleration Engine is supported by generation for fabric interconnects, plus
the servers can use the VIC 1457 with 10 or 25-gig links. Testing has shown about 15% faster
writes at 30% compressable data and 100% percent writes. Also, compression is 3% to 30%
better than with regular software-based compression. So you're going to get a lot of extra
performance.

Some other enhancements include support for VMware ESXi 6.7 U1, cluster expansion for
Hyper-V compute nodes only, Citrix Cloud Services, and EMC RecoverPoint.
Before HyperFlex 4.0, Edge edition was a three-node cluster that connected directly into a
switch instead of fabric interconnects. With 4.0, Cisco is introducing two more options, a two-
node and a four-node solution. The HyperFlex Edge four-node solution gives you more
resiliency since it has one more node added. The HyperFlex two-node solution gives you a
lower price point than the three-node solution.

For two-node clusters, an Invisible Cloud Witness is offered. So this eliminates the need for the
witness VM, which includes the infrastructure and overhead required to deploy. This figure
here shows two M5 servers connected back to back to each other using two 10-gig links. And
the two servers connect upstream to a switch with either 1-gig or 10-gig networking. The two-
node HyperFlex Edge solution mandates connectivity to Intersight.
As the figure states, HyperFlex All-NVMe platform is built for best-in-class performance. The
All-NVMe platform is a significant step forward in terms of storage performance because, in
addition to an increase in performance, it supports higher VM density and much lower latency.

HyperFlex All-NVMe is, as I said, a significant step in terms of storage performance. However,
the installation and maintenance workflows and the tools are the same as with a hybrid or an
all-flash cluster. The figure here lists the enhancements for Hyper-V as well as the support
provided for VMware Site Recovery Manager.
Finally, this last figure lists other supported options, which include new capacity drives for both
small and large form factor hybrid cluster. Initially, you will still not be allowed to mix different
size and capacity options within this cluster.

Also, there are two more GPUs now supported. NVIDIA T4 is attractive because it's also
supported to be fitted into the HX220c nodes. And lastly, there is support for the C480 ML as a
compute-only node, Kubernetes CSI plug-in with Kubernetes version 1.13, and a centralized
audit log export. That brings us to the end of this video. Thanks for watching.

Additional features offered by integration into Cisco portfolio are not the only source of
the advanced features of the Cisco HyperFlex. Cisco HyperFlex solution is a fast-
evolving product, providing many new features with each release.

HyperFlex Innovations in 4.5(2a)

This course is based on the HXDP 4.5(1a) release. During the content update cycle, new
features were released for HyperFlex in the version 4.5(2a). These features are covered
here and are not listed anywhere else in the course.

There are several improvements that were brought to HyperFlex in version 4.5(2a)

 ESXi 7.0U2 snapshot functionality enhancements:

o Sentinel snapshots are not needed anymore and are not created for native
snapshotting.
o Automatic VAAI offload for all snapshots.
o Native snapshotting of VMs spanned over several HX datastores now supported.
o Existing Sentinel snapshots are automatically removed.

 1:1 replication with 2-Node Edge deployments support added.

 Single CPU socket configuration support for Stretched deployments.
Note
HyperFlex is a constantly developing system with regularly released features. To follow
HyperFlex releases and new features refer to Cisco HyperFlex HX Data Platform
Release Notes https://www.cisco.com/c/en/us/support/hyperconverged-systems/
hyperflex-hx-data-platform-software/products-release-notes-list.html.

9.1 Protecting Your Data

Introduction
Protecting data on your Cisco HyperFlex deployment is essential. You do not want the
data to be lost in a disaster or stolen. Here you will learn about your options for backing
up data and replicating VMs from your Cisco HyperFlex deployment. You will also
learn that you can order HyperFlex as a data-at-rest-encryption system that encrypts all
data on a hardware level.
9.2 Protecting Your Data

Disaster Recovery Overview

This video examines Disaster Recovery Overview and Third-Party Data Restore Solutions.
Protecting the data on your Cisco HyperFlex cluster is essential. This is because you do not
want it to be lost in a disaster or if it's stolen, you know. Disaster recovery is actually one of the
smartest insurance policies that you can ever invest in as far as being a business. There are
three different types of data protection scenarios. You back up the system to protect from lost
data, you can replicate data to another data center, and disaster recovery as a service, which is
provided by a service provider.
The backup, replication, and disaster recovery as a service actually are complementary. You
will have the best chance of ensuring that your business maintains continuity if you use all
three of them. So, backup is basically a long-term approach, which can take a long time to
restore. This comes from the good old days. I remember slapping in those old tape backup
drives. Replication ensures that you can restore the data more quickly. And finally, disaster
recovery as a service, this is just a guaranteed backup and replication solution that you're
outsourcing.

There are a couple of terms that we want to make sure we understand. Recovery Point
Objective, or RPO, describes the acceptable amount of data loss measured in time, and
Recovery Time Objective, or RTO, describes the acceptable amount of time it takes to
completely restore a VM.

Obviously, an RPO and an RTO of 0 or as close to 0 seconds, that's probably the most desirable
scenario. So, which disaster recovery solution is the best? There is no single good answer for
that question. It really depends on the type of business you're running, your service level
agreement requirements, how much money you have, how smart are your IT people,
administrators. So basically, a good disaster recovery solution should be reliable with good
performance, well optimized, and of course, simple enough for your technology people to use.
As of Cisco HyperFlex version 3.5, the native replication solution supports basic replication and
restoration workflows. However, replication does not support workflows such as recovery
from backup files, file-level recovery, or application-based recovery. Also, Cisco has validated
their third-party solutions for Cisco HyperFlex, and we'll talk about these validated solutions in
a moment. Other solutions will work. But the validated solutions give you much more chance
of success the first time around.

Finally, the three HyperFlex supported third-party solutions include Veeam Software, which
develops backup disaster recovery and intelligent data replacement software for virtual,
physical, or multicloud infrastructures; Cohesity, which offers a platform to consolidate all
secondary data, including backup and replication, test and development, and analytics; and
lastly, Commvault, which offers an enterprise-level data platform with modules to back up,
restore, archive, replicate, and search the data.
That brings us to the end of the video. Thanks for watching.
Disaster recovery is one of the smartest insurance policies that you can invest as a
business.

Data protection can save the day:

 Backup: Copy files/databases that you can restore if there is data loss, such as:

o A database is corrupted, and you need to restore it.

o An Outlook user deleted an email from the Inbox and Deleted Items, and you
need to restore it.

 Replication: Have a replica of a VM ready to run on another site if there is a

disaster, such as:

o The entire data center burns to the ground and operations must be restarted in
another data center.

 Disaster recovery as a service: A service provider replicates and hosts servers to

provide a failover if there is a disaster, such as:

o You need to operationalize data recovery costs.

o You lack the facility or knowledge to house or maintain a disaster-recovery
solution.

Backup, replication, and disaster recovery as a service (DRaaS) are complementary—

you will have the best chance of ensuring business continuity if you employ all three of
them. Backup is a long-term approach to ensure that your business continues to run, but
if you need to restore data from a backup, it will take minutes, hours, or even days.
Replication ensures that you can restore operations faster than from a backup, but
backup allows you to restore farther back in time. With DRaaS, backup and replication
are guaranteed by a cloud provider. The more levels of data protection that you have,
the more likely that you will be able to meet business SLAs when there is data loss or a
disaster.

Data recovery solutions are usually measured in RPO and RTO:

 Recovery Point Objective (RPO):

o Describes an acceptable amount of data loss expressed in time since last backup.
o For example, if you do backups every 15 minutes and you lose all data, you will
have a saved copy that is 15 minutes old or less, the RPO is 15 minutes.

 Recovery Time Objective (RTO)

o Describes the acceptable amount of time it takes to complete VM recovery.

o For example, if you have a website and the server where it is hosted fails, the
VM is restarted on another server within 2 minutes or less, the RTO is 2
minutes.

Everyone wants their RPO and RTO to be as close to zero as possible. However, it is
usually not possible. For example, the link between your main site and your disaster
recovery site might not be fast enough to transfer all changes of data in real time. Also,
some processes (such as quiescing VMs) take time.

What is a good disaster recovery solution?

 Reliable: Can the data on the remote site be trusted for recovery?
 High Performance: How fast can the data reach the remote site? How much does it
impact the primary site performance?
 Optimized: How well are the resources utilized (compute, network, and storage)?
 Simple: How easy is it to use? Can it integrate with my ecosystem?

Which disaster recovery solution is the best? There is no simple answer to that question.
It depends on the type of business you are running, your SLAs, your budget, and the
competency of your IT administrators. A sophisticated disaster recovery solution is not
helpful if administrators don’t know how to use it to restore business in a disaster.

Replication and Disaster Recovery Options for Cisco HyperFlex

Like with any storage solution, disaster recovery is key on HyperFlex storage as well.
Even though HyperFlex resiliency can tolerate failures it cannot protect from accidental
deletions or failures of the entire system.

To address these scenarios, you can use the two HyperFlex native replication options
and third-party options:

Cisco HyperFlex native one to one (1:1) replication:

 Built-in replication to protect from site failure

 Snapshot-based protection on VMs only
 Suitable for basic VM replication—not a backup solution
 As of HXDP 4.5(1a), included in all licensing tiers, no need for additional licenses

Third-party disaster recovery solutions:

 Comprehensive solutions from companies that specialize in disaster recovery and

related solutions
 More sophisticated replication workflows than with the HyperFlex native solution
 Available features include backup, backup to cloud, file-level restoration, and item-
level restoration in a given application (such as email, database).

The one-to-one replication option has been available for many years now and allows
symmetrical VM-level replication between two HyperFlex systems. Full backup
solutions are complementary with the native replication that HyperFlex provides.

Cisco HyperFlex native many to one (N:1) replication:

 Available on HyperFlex Edge deployments

 Replicates VMs on Edge systems to a central HyperFlex system on a snapshot level
 Protects against logical corruption or accidental deletion of virtual machines
 Allows migration of VMs between Edge deployments involved in the N:1
replication
 Available since HXDP 4.5(1a), requires Edge Premier licensing tier

As of Cisco HyperFlex Data Platform (HXDP) Version 4.5(1a), the HyperFlex native
replication solutions support basic replication and restoration workflows. HyperFlex
native replication solution does not support workflows such as recovery from backup
files, file-level recovery, or application-based recovery (for example, recovery of a table
in an Oracle database or email from Outlook).

9.3 Protecting Your Data

Third-Party Data Restore Solutions

Cisco has validated four third-party backup solutions to work with Cisco HyperFlex:
Veeam Availability Suite, Cohesity, Commvault, and Dell EMC RecoverPoint. Other
restore solutions probably work with Cisco HyperFlex, but the validated solutions are
much more likely to provide a working solution the first time around.

Note
Here you will be provided with reference information on third-party solutions that are
supported with Cisco HyperFlex. Detailed view and configuration steps are out of scope
for this course.
The figure shows a typical backup solution setup. You would spread out your data
centers (HyperFlex systems in the example) between your headquarters and your branch
offices. You would use a backup solution (such as Cohesity) to backup VMs from all
locations into one solution, usually at headquarters. The backup that is located at
headquarters can be used for data recovery if the data from the primary system is
accidentally deleted, or if the primary system is lost in a disaster. However, if the entire
headquarters site burns to the ground, for example, then you really should have a
secondary copy of the data stored somewhere. In the figure, data is being copied to two
additional locations: a physical disaster recovery site and a cloud solution (such as
Microsoft Azure or Amazon AWS).

Veeam Software is a privately held IT firm that develops backup, disaster recovery, and
intelligent data-placement software for virtual, physical, and multi-cloud infrastructures.

Cohesity is a privately held enterprise storage company. Cohesity offers a platform to

consolidate all secondary data; that includes backup and replication, test/dev, and
analytics. For the Cohesity white paper
visit https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
ucsc240_cohesity_dp.html.

Commvault is a publicly traded data protection and information management software

company. Commvault offers an enterprise-level data platform that contains modules to
back up, restore, archive, replicate, and search data.

RecoverPoint is a data protection product from Dell EMC. The RecoverPoint product
has been validated for use with Cisco HyperFlex as of HXDP 3.5(2b). The synchronous
replication feature of RecoverPoint for VMs is not supported with HyperFlex.

Note
With Veeam, Cohesity, and Commvault, you can integrate with Cisco HyperFlex for
more efficient backup procedure. Using native snapshots instead of ESXi-based
snapshots alleviates the impact of VM stunning and I/O penalties. If you run a backup
on a VM without HyperFlex integration, the backup job will create a VMware snapshot
with a redo-log, and when the backup is complete, the snapshot will be deleted.
Deleting a standard, ESXi-based snapshot is slow and consumes a lot of resources.
Deleting a storage-native snapshot has almost no impact on the production deployment.

9.4 Protecting Your Data

Cisco HyperFlex Native Replication

Solutions
The Cisco HyperFlex Data Platform disaster recovery feature allows you to protect
VMs from a disaster by setting up replication of running VMs between network-
connected systems. VMs running on the initial group where VM was deployed (Source
Group) are replicated in intervals to the backup HyperFlex group (Target Group). The
two groups are typically located in two different geographical locations.

In 1:1 replication, each HyperFlex system can serve as the disaster recovery site for the
VMs that are running on the other system. Both locations are running their own VM set.

In N:1 replication, the central HyperFlex deployment acts as the target group for many
Edge deployments. When restoring a VM from the central HyperFlex deployment, you
can choose on which Edge system you will deploy the restored VM.

Once protection has been set up on a VM, Cisco HXDP periodically takes a replication
snapshot of the running VM on the local HyperFlex system and replicates (copies) the
snapshot to the paired remote system.

Version HXDP 4.5(1a) has this native replication scope:

 Replication between two HyperFlex systems (1:1 replication)

o Bidirectional (each site running VMs and replicating to the other site) and
unidirectional deployment (one site runs VMs, the other is strictly a disaster
recovery site) are possible.

 Replication between many HyperFlex Edge deployments (N:1 replication)

o One central HyperFlex system provides the centralized replication repository for
many Edge deployments.

 Both functions are based on native HyperFlex snapshots.

o Application-consistent snapshots are possible.

 o RPO between 5 minutes and 24 hours

The figure shows two HyperFlex systems: one in Site A and another in Site B. You are
using native HyperFlex replication to protect VM2 that runs on Site A to Site B. So, if a
disaster brings down the system on Site A, you can recover VM2 and run it on Site B,
ensuring business continuity. In this example, you could also protect VM1 to HyperFlex
system on Site B, and VM3 to system on Site A.

Data replication and backup systems use specific terminology. You need to familiarize
yourself with this terminology to effectively understand the data replication and backup
systems.

These are some of the terms used in data replication:

 Failover: Part of the manual VM recovery process in the event of a disaster on the
source system. Failover, in this context, is converting a replication snapshot on the
target system into a working VM.
 Interval: Part of the replication-schedule configuration. This term describes how
often the protected VMs replication snapshot is taken and copied to the target
system.
 Local system: One of a VM replication system pair. The system you are currently
logged into through HyperFlex Connect. From the local system, you configure
replication protection for locally resident VMs. The VMs are then replicated to the
paired remote system.
 Migration: A routine system maintenance and management task where a recent
replication snapshot copy of the VM becomes the working VM. The replication pair
of source and target system does not change.
 Primary group: An alternative name for the source group in VM disaster recovery.
 Protected virtual machine: A VM that has replication configured. Protected VM.
 Protection group: A means to apply the same replication configuration on a group
of VMs.
 Recovery process: Manual process to recover protected VMs in the event the
source system fails, or a disaster occurs.
 Recovery test: A maintenance task that ensures the recovery process would be
successful in the event of a disaster.
 Remote group: One of the VM replication system pair. The remote system receives
the replication snapshots from the local protected VMs on the local system.
 Replication pair: Two systems that together provide a remote system location for
storing replication snapshots of local VMs. Systems in a replication pair can be both
a remote and a local system. Both systems in a replication pair can have resident
VMs. Each system is local to its resident VMs. Each system is remote to the VMs
that reside on the paired local system.
 Replication snapshot: Part of the replication protection mechanism. A type of
snapshot taken of the protected VM. Copied from the local system to the remote
system.
 Secondary group: An alternative name for the target system in VM disaster
recovery.
 Source group: One of a VM recovery system pair. The source system is where the
protected VMs reside.
 Target group: One of a VM recovery system pair. The target system receives the
replication snapshots from the VMs on the source system. The target system is used
to recover the VMs in the event of a disaster on the source system.

Both the source and target systems must be on the same HXDP version—or pair a
HyperFlex system on HXDP 3.0(1x) with a system on 3.5(1x) or later. Mixed version
systems have two limitations:

 Planned migration will not be possible from HyperFlex Connect—only

through hxcli, similar to the HXDP v3.0 method.
 Re-protect will not be possible from HyperFlex Connect—only through hxcli,
similar to the HXDP v3.0 method.

All nodes in both systems take part in replication:

 TCP-based transmissions with robust fault handling retries/timeouts and checksum

 Snapshots are incremental, only the first read is whole.
 I/O for replication are large to optimize replication.
 Data on the wire is compressed.
 Intelligent pattern recognition

The figure shows two HyperFlex systems engaged in 1:1 HyperFlex native replication:
one with 4 servers and the other with 5 servers. You configured the systems to perform
bidirectional native replication; in other words, System A is protecting its VMs to
System B and vice versa. All nodes take part in native replication, so there are 20
connections between the two sites. Each server is assigned a special IP just for native
 data replication purposes. So that the replication traffic between systems can cross the
site-boundary and traverse the internet, each node on System A should be able to
communicate with each node on System B across the site boundary and the internet.

Note
The example is from a 1:1 replication scenario, but the N:1 configuration functions the
same, except that one system is the Source and the other is the Target. Replication in the
N:1 configuration is not bidirectional.

Requirements for Native Replication

Both types of HyperFlex replication use the same data connectivity between location for
the data transfer. The actual implementations and management interfaces are very
different.

Network requirements and considerations for native HyperFlex replication follow:

 Set maximum bandwidth in HyperFlex to less than the throughput of WAN link.
 Make sure that all necessary ports are open between the two sites if using a firewall.
 Use IPsec or MACsec on WAN—HyperFlex data plane is not authenticated.
 Note the IPsec header—the default MTU of the replication network is 1500.

The table lists and describes ports that need to be open between the two systems for
native HyperFlex replication.

Port Port
Service/Protocol Source Essential Information
Number Destinations

9338 Data services manager peer / TCP Each CVM Each CVM Bidirectional, include
node node system management
9339 Data services manager / TCP IP addresses

3049 Replication for CVM/TCP

4049 System Map/TCP

4059 NR NFS/TCP

9098 Replication Service

8889 NR Controller for coordination

/TCP
Port Port
Service/Protocol Source Essential Information
Number Destinations

9350 Hypervisor service/TCP

443 HTTPS/TCP

Design requirements and considerations for native HyperFlex replication follow:

 Replication will lower the I/O capacity of the source system—consult HyperFlex
sizer.
 The systems involved in replication can be different size.

o Be aware of any size discrepancy when managing replication, so you do no run

out of resources.

 Systems can have different storage type (All-Flash / NVMe / Hybrid)

 When configuring 1:1 replication, keep in mind that a smaller deployment might not
be able to failover the larger one
 1:1 replication requires HXDP 2.5(1a) or higher, N:1 requires 4.5(1a) or higher

Note
As of HXDP 4.5(1a), a 2-node HyperFlex Edge deployment does not support 1:1 native
HyperFlex replication.

Ensure that you have sufficient space on the remote system to support your replication
schedule. The protected virtual machines are replicated (copied) to the remote system at
every scheduled interval. Though storage capacity methods are applied (deduplication
and compression), each replicated virtual machine will consume some storage space.
Not having sufficient storage space on the remote system can cause the remote system
to reach capacity usage maximum. If you see Out of Space errors, pause all replication
schedules until you have appropriately adjusted the space available. Always ensure that
your HyperFlex storage capacity consumption is below the space utilization warning
threshold.

Protected virtual machines are recovered with thin provisioned disks irrespective of how
the disks were specified in the originally protected virtual machine.

If you have protected a VM with storage on a non-HyperFlex datastore, periodical

replication will fail. Either unprotect the VM or remove its non-HyperFlex storage. Do
not move protected VMs from HyperFlex datastores to non-HyperFlex datastores. If a
VM is moved to a non-HyperFlex datastore through storage vMotion, then you must
unprotect the VM and reapply the protection.

Protection and recovery of VMs with snapshots:

 VM with no snapshots: When replication is enabled the entire content of the VM is

replicated.
 VM with VMware redo log snapshots: When replication is enabled, the entire
content, including the snapshot data, is replicated. When a VM with redo log
snapshots (vSphere) is recovered, all previous snapshots are preserved.
 VM with HyperFlex snapshots: When replication is enabled, only the latest data is
replicated, and the snapshot data is not replicated. When the VM is recovered,
previous snapshots are not preserved.

Here are some operational requirements and considerations for native HyperFlex
replication:

 Be aware that in 1:1 replication only the latest snapshot can be recovered as of
HXDP 4.5(1a)
 In N:1 replication, any snapshot stored on the Target system can be used for the
restore
 Reduce the RPO if the change rate is greater than the WAN bandwidth
 Do not reboot any nodes in the HX system during any restore, replication, or
recovery operation

Let's take a closer look at the Cisco HyperFlex Native Replication Solution. The Cisco HyperFlex
Data Platform Disaster Recovery feature allows you to protect virtual machines from a disaster
. This is accomplished by setting up replication and running virtual machines between a one-to-
one pair of network-connected clusters.
The protected VMs running on a cluster replicate to the other cluster in the pair and vice versa.
The paired clusters are typically located at a distance from each other so that one location will
actually survive a disaster. Each cluster serves as a disaster recovery site for virtual machines
running on the other cluster. And the recovery point objective is between 5 minutes and 24
hours.

This figure here illustrates two HyperFlex clusters—one with four servers and the other one
with five. Clusters are configured to perform bidirectional native replication. This means that
cluster A and cluster B are protecting each other's virtual machines. Notice, there are like 20
connections between the two sites, because all of the nodes are participating in this native
replication. Each server is assigned a special IP just for the native data replication process.
Now, to enable replication traffic, both clusters should be able to communicate with each
other across the site boundaries and the Internet. There are some network requirements and
considerations for native replication. The MTU replication network is 1500 by default. The
maximum HyperFlex bandwidth should be less than the throughput of the wide-area network
link. All necessary ports should be open if you're using a firewall between the sides. And lastly,
for IPsec, MACsec on the WAN, the HyperFlex data plane is not authenticated.

In your design, you should take into consideration that replication will lose I/O capacity on the
cluster, so be sure to consult the HyperFlex Sizer. Also, two paired clusters can be different size
. You should ensure that you have sufficient space on the remote cluster. This is so you won't
run out of resources. Also, the clusters can be different types, and you should be aware of the
limitations this presents to avoid any problems with the failovers. And lastly, you must have
HyperFlex version 2.5 or higher for the native replication feature to work.
Finally, some optional requirements and considerations include that you should be aware that
as of HyperFlex version 3.5, only latest snapshots can be recovered. You should have the
Disaster Recovery Runbook, which is a documented list of procedures. These procedures are
either automated or manually performed by the administrator in case a disaster recovery
scenario occurs so that you can restore the system. And lastly, remember, the recovery point
objective should be reduced if the change rate is periodically greater than the WAN bandwidth
. That brings us to the end of this video. Thanks for watching.

9.5 Protecting Your Data

Cisco HyperFlex One-to-One Native

Replication
This video describes protecting and restoring your virtual machines with native replication.
Cisco HyperFlex Connect uses the term "protect," which really means replicate this VM
through the other cluster. The simplest way to replicate a VM is to protect it independently.
And the procedure is simple. In HyperFlex Connect, you navigate to Virtual Machines, select
the VM that you want to replicate, and click Protect. There you go.

The protect dialog sets the schedule for replicating the VMs and when to start the first
replication. You can optionally choose which VM tools to quiescent the VM, but be aware this
increases the replication time. Lastly, click Protect Virtual Machine.

The second method for replicating VMs is with protection groups. Remember, we talked about
that earlier. First, you create a protection group in the Replication view of HyperFlex Connect.
The replication group defines the settings for the replication frequency, when to start, and the
optional quiescence if desired. In the Virtual Machines view, choose the VMs and click Protect
Virtual Machines. Next, place the VM into the group, and it will automatically inherit the
protection settings from the group.
In Replication view of HyperFlex Connect, you can observe the summary of replication
configuration. The possible summary status includes Protected. This is the number of VMs that
were successfully replicated. Replication Failures—these were the VMs that were
unsuccessful. And Exceeds Interval, which is the number of VMs with an exceeded protection
interval. You can always go back and edit the protection group by navigating to Replication,
Protection Groups, and Edit Schedule.

You can view replication status of individual virtual machines in the Replication view of
HyperFlex Connect. The VM status will either be Active during the replication process or
Protected when the replication has succeeded. HyperFlex cluster maintenance activities
include adding or rebooting a node or upgrading the HyperFlex cluster. This typically requires a
node be put into the HyperFlex Maintenance mode.
You should pause the replication schedule before maintenance and then resume the schedule
after you're done. From the Controller VM, you would stop HyperFlex Native Replication with
the stcli dp schedule pause. Then, restart the replication. Use the command stcli dp schedule
resume command. The stcli dp schedule status command will return the status of the
replication.

Disaster recovery is performed when the source site is unreachable and you want to fail over
to the target cluster. Recovering virtual machines is restoring a most recent replication
snapshot from the recovery or target cluster. Now, there are three different restore workflows
available in Cisco HyperFlex Native Replication, as you can see in the figure. You've got the
Test Restore, you've got Migration, which is also called a Planned Restore, and lastly, Recover,
also called the Unplanned Restore.
Let's look at these three workflows in a little bit more detail. The Test Restore tests the
recovery without breaking replication. It brings up your VM workload on the target to verify
that it is going to be recoverable. In the Replication view of the remote cluster, you choose the
VM you want to test, and then choose Test Recovery.

Choose the resource pool, folder, power state, and decide on the VM name, and choose the
networks that will be assigned to the VM adapters. Verify the contents of the recovered VM,
and then manually delete the VM from vCenter after you're done with all the tests. Testing
recovery does not disrupt the running clusters.

Migration: Migration, which is also called planned restore, pauses the replication schedule,
replicates the most recent copy, recovers on the target, switches the ownership from the
source to the target, and resumes replication on the target that is now the new source.
Migration has—this workflow has been available since HyperFlex version 3.5.1, so it's relatively
new.
In the Replication view of the remote cluster, choose the VM you want to move between the
clusters and click Migrate. To finalize the migration of a VM, you choose the resource pool,
folder, power state, decide on the VM name, and choose the networks that will be assigned to
the VM adapters. Then, click Migrate.

From the Events display, you can view the process status of the VM migration. Data is
replicated from the local to the remote cluster. The VM is unregistered on the local cluster and
then registered on the remote cluster. Lastly, the migrated VM is automatically protected
back to the cluster that it was actually migrated from. Finally, the recover, which if you
remember, is also called the unplanned restore, it recovers the VM on the target, switches the
ownership from the source to the target, and resumes replication on the target now that it is
the new source.
When a disaster strikes on the local cluster, you need to start up the VMs on the remote
cluster using the latest snapshot. So from the replication view of HyperFlex Connect on the
surviving cluster, find your VM. Choose the resource pool, folder, power state, decide on the
VM name, and choose networks that will be assigned to the VM adapters, and click Recover.

When the recovery is complete, you can continue as usual. After the first cluster comes back
online, you can either migrate VMs back to that cluster, or you can just continue on running
the second cluster.

That brings us to the end of this video. Thanks for watching.

One to one replication option is configured from HyperFlex Connect and includes
network configuration, creation of protection groups and three recovery workflows that
are all available from the HyperFlex Connect interface.

The result of a 1:1 replication configuration is two HyperFlex Edge, Stretched or ESXi
Standard deployments that are paired. Any combination of the systems can be used.
You can, for example, replicate a stretched HyperFlex deployment to an Edge
deployment.

The 1:1 replication configuration can be bidirectional. In this scenario, both HyperFlex
systems are replicated to each other. These systems cannot be involved in HyperFlex
native replication with any other HyperFlex system.

1:1 replication provides a fully featured VM-level replication option:

 Supports all HyperFlex deployment types except 2-Node Edge and Hyper-V.
 Only two HyperFlex deployments can be involved in replication.
 1:1 replication is not a full backup solution, it is a recovery solution.
 No additional license required for this type of native replication.

Configuring One-to-One Native Replication

Once you have two HyperFlex systems up and running and each has at least one
datastore configured, you can configure replication between them.

First, make sure that you have configured and allowed the VLAN that you want to use
for the replication of traffic on all switches between the two sites. Then you need to use
Cisco HyperFlex Connect to configure the replication network on both systems. Finally,
you pair the two sites, and bind the datastores together for replication.

To configure 1:1 replication, you will need to set up each HyperFlex system first:

1. Create at least one datastore per HyperFlex system

2. Configure the replication network on both systems from HyperFlex Connect
3. Pair HyperFlex systems from HyperFlex Connect.
4. Choose individual VMs to protect or create protection groups and set a protection
schedule.

 VMs in protection groups share their replication attributes, such as replication

interval.

Protection attributes can be created and assigned to protection groups. To assign the
protection attributes to virtual machines, add them to a protection group. For example,
imagine that you have three classes of protection: gold, silver, and bronze. Set up a
protection group for each class with replication intervals, such as 5 or 15 minutes for
gold, 4 hours for silver, and 24 hours for bronze. Most of your VMs can be protected by
simply adding them to one of the three protection groups that are already created.
Configure Replication Network and Pair Systems

To start the configuration, open HyperFlex Connect as an administrator on both

systems:

 Choose Replication > Configure Network.

 You can only configure the replication network once. Once configured, you can edit
the available IP addresses and the networking bandwidth.

Note
You will need to configure a replication network on both systems.

Fill out the VLAN Configuration dialog box:

 The VLAN that you will use to replicate VMs between systems

o Each HyperFlex Storage system requires a VLAN dedicated to the replication

network.

 IP and credentials to access local Cisco UCSM

o HyperFlex Connect can use Cisco UCSM to configure native replication VLAN
on the Fabric Interconnects.
Note
When a value is added, the default VLAN name is updated to include the additional
identifier. The VLAN ID value does not affect a VLAN name that is manually entered.

Fill out the IP & Bandwidth Configuration dialog box:

 The minimum number of IP addresses required is the number of nodes in your

HyperFlex group, plus 1

o For example, if you have a 4-node deployment, enter at least 5 IP addresses

o Include sufficient number of IP addresses to cover any additional nodes. You
can add IP addresses at any time

 Set the replication bandwidth limit lower than the available throughput
Note
Enter the maximum network bandwidth that the replication network is allowed to
consume for outbound traffic. Acceptable value is between 10 and 10,000. The default
value is unlimited, which sets the maximum network bandwidth to the total available to
the network. The replication bandwidth is used to copy replication snapshots from this
local HyperFlex Storage to the paired remote HyperFlex Storage.

You can modify the MTU size. The default is 1500 bytes. You would change this option
in cases where you are using additional encapsulation to create a WAN overlay, such as
IPsec. ISPs usually permit MTU 1500, but if you are using encapsulation on 1500 MTU
packets created by HyperFlex replication, those packets may be dropped by your ISP.

Once you started the configuration of the replication network, HX Connect do the
following:

 Verify the remote network, log in, and obtain a VLAN from Cisco UCS Manager.
Assign an IP address to each node in the local system
 Validate connectivity within the system, over the replication network to ensure that
connectivity is successful, and there is no packet drop.
Configuration and testing of the replication network configuration will take a few
minutes

After the replication network is created on the system, each converged node on the
system would be configured with an IP address on the eth2 interface

If you need to edit the replication network, choose Replication > Edit Configuration

Note
Since HXDP 3.5(1a) the validation of the replication network is built into the
configuration workflow. If you are using prior versions, you need to manually validate
connectivity. From each node in the system, ping the replication gateway from a CVM
using the ping -I eth2 replication-gateway command.

Note
If there is a duplicate IP assignment and it is necessary to remove the replication
network assignments, please Contact Cisco TAC.

Click Pair Cluster. By pairing System 1 with System 2, you are specifying that all
VMs on System 1 that are explicitly set up for replication can replicate to System 2, and
that all VMs on System 2 that are explicitly set up for replication can replicate to
System 1.
Pairing is strictly 1-to-1. A HyperFlex system can be paired with no more than one
other HyperFlex system. A datastore on a paired system can be paired with no more
than one datastore on the other system.

Enter a Name for the replication pairing between two HyperFlex systems. This name is
set for both the local and remote system. The name cannot be changed.

For your pairing to be fully functional, you must first:

 Make sure that you have at least one datastore on each system.

o VMs that you want to protect must reside on one of the datastores in the
replication pair.

 Configured a native replication network on both systems.

 Created VLANs and allowed them on switches between the two systems.

In the Remote Connection dialog box, provide the virtual management IP of the
remote system and credentials to log in to that deployment. Click Pair when done.
Cisco HXDP verifies the remote HyperFlex deployment and assigns the replication pair
name.

Since HXDP 3.5(1a), pairing of HyperFlex systems also adds verification of intersite
node connectivity.
Prior to HXDP 3.5(1a), there is no verification of connectivity. Therefore, in the next
step, datastore pairing would either succeed or fail.

In the Datastore Mapping dialog box, map local datastores to remote datastores. Then
click Done. You are ready to start protecting VMs.

The VMs to be protected must on the datastores that you choose. Moving virtual
machines from the configured datastores for the replication pair also removes protection
from the VMs.

If you do not map a local datastore to the remote datastore, you will not be able to
perform native replication for the VMs that reside on that non-mapped datastore. In the
above example, the datastore "ccp" is not mapped to any remote datastore.

If datastore mapping succeeds, connectivity between systems over the replication

network is confirmed by HyperFlex Connect.

If datastore mapping fails (which would only apply to HXDP prior to 3.5(1a)),
skip Map Datastores, and proceed to validate the Intersite connectivity after pairing as
follows:

 Ping the gateway of the peer system from any of the nodes of a local system. If your
HyperFlex systems are A and B, do this check from System A to System B and
System B to System A. On system A, run the ping -I eth2 replication-
gateway-of-system-B command.
 If the ping check fails (100 percent packet loss is seen), check the firewall and router
associated with systems A and B. They should be configured to allow traffic for the
replication subnet and VLAN. Firewalls at Site A and Site B must allow: 1)
outgoing traffic from A’s replication subnet; and 2) incoming traffic from B’s
replication subnet. The router must allow: 1) outgoing traffic from A’s replication
 subnet to be routed to B; and 2) outgoing traffic from B’s replication subnet to be
routed to A.
 If the ping check succeeds, but datastore mapping still does not work, check the
MTU settings. Run the ping -I eth2 -M do -s <bytes> replication
IP of B's node command. Perform this check from System A to System B,
and then from System B to System A.

o If the test fails, retry with a smaller MTU number, until it succeeds. If the
default MTU of 1500 does not work, contact Cisco TAC.

Note
If you need to edit the replication pair, choose Replication > Replication Pairs > Edit.
If you need to delete the replication pair, consult the HyperFlex Systems Administration
Guide (https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-
5.html).

Cisco HyperFlex One-to-One Replication Workflows

Disaster recovery is performed when the source site is unreachable, and you want to
failover the VMs and the protected groups to the target system. The process of recovery
recovers the VM on the target system. Recovering VMs is restoring the most recent
replication snapshot from the recovery (target) system.

Three different restore workflows are available in Cisco HyperFlex native replication:

 Test restore: Testing VM recovery gives you the ability to test recovery without
breaking replication. It can bring up your VM workload on the target to verify that
the VM is recoverable.
 Migration (also called planned restore): Performing planned migration pauses the
replication schedule, replicates the most recent copy, recovers on the target,
switches the ownership from the source to the target, and resumes replication on the
target that is now the new source.
 Recover (also called unplanned restore): Performing failover recovers the VM on
the target, switches the ownership from the source to the target, and resumes
replication on the target that is now the new source.

Note
Cisco recommends that you perform a test recovery on each protected VM to validate
failover.

With Cisco HyperFlex, as of Cisco HXDP 4.5(1a), only one replication point (the latest)
is available for disaster recovery.

When running recovery on VMs that are individually protected, or that are in different
protection groups, the maximum number of concurrent recovery operations on a system
is 20.
Test Restore
In the Replication view of a remote system, choose the VM you want to test restore,
and choose Test Recovery:

 Choose the resource pool, folder, power state, decide on the VM name, and choose
the networks that will be assigned to VM's adapters.
 Verify the recovered VM’s contents (for example, use Web Console from vCenter).
 Manually delete VM from vCenter after you are done with all the tests.

Testing recovery does not disrupt the running systems. The intent is to verify that the
VMs are recoverable.

Before you begin the test virtual machine recovery process, ensure the following:
 The target system is up and in good health.
 The protected virtual machines completed a recent replication to the target system.
These replicated virtual machines are stored as snapshots on the target system.

Only one copy of the test recovered VM can be made at any point. If you need to have
another test recovered VM, please delete the previously created VM.

The default folder to which test recovered VMs are assigned to is the HX Test Recovery.

In networking, you have two choices:

 Test networks: Network that will be assigned to the adapter of the test-recovered
VM. You would want to select a bubble, an isolated network that is not connected to
the first VM, which is probably in production to avoid duplicate IP and potential
data corruption.
 Map networks: Mapping of networks between source and target system.

You can skip specifying network mapping in the following cases:

 If the source VMs use vSphere Standard Switches and if all ESXi hosts on the
recovery side have standard switch networks with the same name
 If the source VMs use VDS port groups and if the recovery site has identically
named VDS port groups

If you want to specify network mapping, ensure that both the name and the type of the
VM network matches between the source and the target.

Planned Restore
In the Replication view of a remote system, choose the VM you want to move between
systems and click Migrate.

Note
GUI-based (HX Connect) migration workflow is available since HXDP 3.5(1a). Prior to
this version of HXDP, migration was much more manual process, requiring most steps
to be made through hxcli. For more details, refer to the Cisco HyperFlex Data Platform
Administration Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5.html) for the version that
you are running.

To finalize migration of a VM:

 Choose the resource pool, folder, power state, decide on the VM name, and choose
the networks that will be assigned to VM's adapters.
 Click Migrate.

For a single vCenter deployment, the Migrate workflow performed entirely through the
HX Connect user interface is not supported. To perform a planned migration:

 Using the WebCLI, run the following command to prepare for failover on the
source: stcli dp vm prepareFailover -vmid <VMID>
 Log in to the vSphere Web Client Navigator of the primary site and remove the VM
from the primary site to unregister the VM. Right-click on the virtual machine and
select All vCenter Actions > Remove from Inventory.
 Log in to HX Connect of the secondary site. Select Replication > Remote VMs
Tab > select your VM. Click Migrate.

HyperFlex will as part of VM migration:

 Perform final replication of data from local to remote system, unregister VM on

local system, and register it on remote system.
 Automatically protect the migrated VM back to the system it was migrated from.
 Unregistering the VM on the source system before doing a planned migration, ensures
that vCenter no longer has a record of the VM and it stops managing the VM, but it
retains the data for the VM.

If you have one single vCenter to manage both source and target system, you have
previously removed the VM from the inventory of the source system so migration to the
target system was permitted. After the Migrate task has completed successfully, log in
to the vSphere Web Client of the secondary site and manually register the VM.

 Log in to vSphere Web Client Navigator. Select Configuration > Storage.

 Right-click on the appropriate datastore and click Browse Datastore. Navigate to
the VMX file of the VM, right-click on the file and click Add to Inventory. Follow
the wizard to manually register the VM.

If you would be migrating VMs that are part of a protection group, HyperFlex would
move VMs out of the protection group. VMs would now be displayed in
the Standalone Protected VMs subpane and HyperFlex will recover only the selected
VM.

Failover
When a disaster strikes on the local system, you will need to start-up VMs on the
remote system from the latest snapshot.

 Find your VM in the Replication view of HX Connect of the surviving system.

 Choose the resource pool, folder, power state, decide on the VM name, and choose the
networks that will be assigned to VM's adapters.
 Click Recover.
The input fields of the Recover VM on this system window are the same as the fields
for the planned restore workflow.

Note
If you are using one single vCenter to manage both source and target system, HyperFlex
Connect will not let you recover the VM. You will first need to log in to vCenter and
remove the VM that you are trying to recover from the inventory of the source system.
The source system is currently unavailable due to a disaster, so vCenter will see the
VMs on it as “disconnected.” In some versions, vCenter (including 6.5 U2) does not
allow you to remove a disconnected VM. You will need to remove the host from
vCenter instead—and that will, in turn, remove the VM. You wouldn't have these
complications in a scenario where you have two separate vCenters managing the two
systems.

You cannot roll back the process of unplanned migration.

If you are recovering VMs that are in a protection group, all the VMs will be moved
from the protection group and the selected VMs will be recovered. Recovered VMs
show protection status as Recovered and the remaining (protection group) VMs show
protection status as Recovering. The protection group will go in Recovered state and is
not reusable, so you can delete it from the primary site.

After the VM recovers on the surviving system, you can continue your business
operations, even if the source system is down due to a disaster.

 When the first system comes back online, you can either migrate the VM back to
that system and protect the VM to the second system, or continue running the VM
on the second system, but protecting the VM back to the first system by selecting
the VM and then clicking Re-protect.

Note
If you are using one single vCenter to manage both source and target system, then
previously you had to remove the hosts from vCenter to remove the disconnected VMs.
In this case, you need to add the hosts back into the vCenter: right-click the system,
click the Add Host option, and follow the procedure to add the host back into vCenter.
The inputs to add a host to vCenter are IP, username, and password.
9.6 Protecting Your Data

Disaster Recovery Orchestration for

One-to-One Native Replication
Multiple steps are required to successfully execute a disaster recovery routine (test
recovery, planned migration, and disaster recovery):

 Transfer the VM data to the disaster recovery site.

 Create a new VM with the configuration parameters.
 Power up a VM or a bunch of VMs in a specific order.

By default, HyperFlex native replication requires you to perform disaster recovery

routines through HyperFlex Connect. However, you also have other options.

HyperFlex native replication allows you to perform disaster recovery using HX

Connect, REST API, or stcli. To orchestrate disaster recovery, you can also use:

 Integration with VMware Site Recovery Manager (SRM)

o SRM is an extension to vCenter.

o SRM only does disaster recovery orchestration and the storage vendor does the
data transfer.
o SRM interacts with the storage vendor using a plug-in called SRA.
o Storage replication is done at the granularity of a datastore.
o Support for VMware SRM 6.5 and 8.1

 Microsoft PowerShell-based disaster recovery runbooks

o Cmdlets for test recovery, migration, and unplanned failover

o JSON-based runbook functions to simplify multiple steps with a script

VMware SRM is a recovery solution that provides automated failover and disaster
recovery testing. SRM is available since HyperFlex version 4.0(1a).

VMware SRM requires Storage Replication Adapter (SRA) to function. SRA is a

storage vendor-specific plug-in for VMware vCenter. This adapter enables
communication between SRM and HyperFlex CVMs. You must install an appropriate
SRA on the SRM server hosts at the protected and recovery sites. You would install
SRA on a Windows VM. If you use more than one type of storage array, you must
install the SRA for each type of array on both of the SRM hosts. For more information
on SRM with HyperFlex, refer to the Cisco HyperFlex Data Platform Administration
Guide (https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/AdminGuide/4_0/
b_HyperFlexSystems_AdministrationGuide_4_0.pdf).
VMware SRM for Cisco HyperFlex Disaster Recovery Overview
SRM prerequisites for HyperFlex are as follows:

 Two HyperFlex systems with HXDP 4.0(1a) or later.

 Replication network connectivity between HyperFlex systems. Requirements for
SRM support are the same as for native replication.

o Includes already configured replication networks and system pairing.

The following is a summary of SRM configuration steps with HyperFlex:

 Prepare HyperFlex native replication from HX Connect.

o Create replication network.

o Create site pairing.
o Create placeholder datastore for SRM (scratch datastore) and map datastores.

 Install SRM
 Configure SRA and pair SRM sites.
 Add SRM array managers and array pairs.
 Prepare a runbook (SRM protection groups and recovery plan).

Keep in mind that VMware SRM software is licensed separately by VMware. To use
SRM, there is no special licensing tier within HyperFlex to support it; support for SRM
is included in all license tiers, the same as native HyperFlex replication

A placeholder datastore for SRM stores the VM files (.vmx, .vmxf, and .vmsd) that are
very small. You will need to create a placeholder datastore on both sites.
Note
For detailed information on SRM with HyperFlex, refer to the Cisco HyperFlex Data
Platform Administration Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5.html).

PowerShell for Cisco HyperFlex One-to-One Disaster Recovery

HXPowerCLI is the name of the PowerShell module for disaster recovery in Cisco
HyperFlex Systems.

Cisco HXPowerCLI has a set of cmdlets for Windows PowerShell that enables task
automation and serves as a CLI for the high-level API:

 You can use the Cisco HXPowerCLI cmdlets to perform tasks related to native
replication.
 It is a free and less sophisticated orchestration alternative to VMware SRM.
PowerShell-based cmdlets will work with HyperFlex 3.5(1a) or later. Cmdlets are
hosted on at Microsoft's PowerShell Gallery
(https://www.powershellgallery.com/packages/Cisco.HXPowerCLI/).

Note
For more information on usage of cmdlets, read the Cisco HyperFlex PowerShell
Cmdlets for Disaster Recovery
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/HyperFlex_PowerShell_Toolkit/4-5/b-hx-powershell-cmdlets-for-
disaster-recovery-4-5/m_installation.html).

You can use PowerShell to prepare the JSON-based runbook:

 Runbooks are available for test recovery, disaster recovery, and planned migration.
 Optional parameters, such as type of recovery, delay between recovery, and boot
order, are available.
JsonVersion : 1.0
RecoveryMode : TestRecovery
RecoveryExecutionMode : Parallel
ProtectedSiteClusterIP : 192.168.12.10
ProtectedSiteVCenterIP : 192.168.12.1
RecoverySiteClusterIP : 192.168.14.10
RecoverySiteVCenterIP : 129.168.14.1
ResourcePoolName :
FolderName :
NetworkMap :
DelayBetweenRecovery : 0
ParallelRecoveryLimit : 10
BootOrderGroup1 : {TestVM1, TestVM2, Win10_1, Win10_2}
BootOrderGroup2 : {Ubuntu1, Ubuntu2, Ubuntu3}
VMParams : @{TestVM1=; TestVM2=; Win10_1=; Win10_2=; Ubuntu1=;
Ubuntu2=; Ubuntu3=}

The example above shows a runbook that was created for test recovery. Recovery will
be done in parallel for up to 10 VMs at a time, without any added delay between
sequential recoveries. The runbook has two boot order groups defined.

9.7 Protecting Your Data

Cisco HyperFlex Many-to-One

Replication
Many-to-one replication is a new feature introduced in HyperFlex 4.5(1a). It uses Cisco
Intersight instead of HyperFlex Connect as the management front end for configuration
and workflows. This allows administrators to effectively manage native replication of
many systems from one user interface.

The N:1 replication solution is only available on the Edge deployments and needs to be
licensed separately, there also needs to be a central HyperFlex system to which all the
Edge system replicate their VMs.

These are general prerequisite requirements for N:1 deployments:

 Source and Target need to run the 4.5(1a) version and must be registered to
Intersight.
 Source systems must be Edge deployments, Targets ESXi Standard deployments.
 Account Administrator or HyperFlex Cluster Administrator role needed in
Intersight.
 Maximum 20 Edge systems can use a single Target with up to 100 VMs per Edge
system.
 Edge Premier license on all Edge systems.
 Target system needs to be licensed with DC Advantage or DC Premier license.

Note
As of May 2021, Cisco HyperFlex system names must not include spaces or you will
not be able to create policies associated with N:1 replication. You can change your
HyperFlex system name by using vCenter Flash plug-in or HTML plug-in version 2.1.0
or later.

 Systems involved in N:1 replication cannot also be involved in 1:1 replication.

 The Target system needs to be reachable by all Source systems.
 The replication network is designed to work over WAN and Layer 3.
 All the systems involved in N:1 replication should use the same MTU setting.
 If you are using encapsulation over VLAN, adjust the MTU size to match your ISP
limitations.

If the licenses run out on the systems involved in N:1 replication, the existing
replication jobs will continue, but you will not be able to set up or change jobs until you
license the systems.

Similar to the HyperFlex 1:1 replication, the datastores in N:1 replication are paired
between the source and target system. The datastores in N:1 replication are created
automatically on the initial configuration of the HyperFlex system.

Datastore pairing introduces additional considerations:

 VM must not span several datastores (disks on different datastores).

 VM must be on the Backup datastore created by the N:1 workflow.
 If a VM is deployed or moved to the Backup Datastore, it will be automatically
protected.
 The Target system has backup datastores for each of the protected systems.
Configuring Target Cisco HyperFlex System
In many-to-one replication, the central Cisco HyperFlex system is used as a central
backup point where many HyperFlex Edge deployments can deposit their backups. The
central system needs to be accessible from all Edge deployments that you want to back
up to it.

The Target system needs to fill the requirements for backup jobs:

 All Source systems need to be reachable over the replication network.

 Connection bandwidth needs to be higher than the sum of all Source systems.
 IP pool for replication requires 1 IP per converged node plus one VIP address.
 Target system needs enough storage capacity to be able to handle the backups.
 Target can use a different storage type than the Sources (LFF/Hybrid).

The Target system needs to be able to handle the bandwidth and the amount of data that
the Source systems are sending. The best guideline is to have at least the amount of
storage of all Source systems and at least the full bandwidth of all Source systems on
the Target system.

Note that the total bandwidth does not refer to the physical bandwidth available, but the
bandwidth set as the maximum bandwidth for N:1 replication. You can configure the
maximum bandwidth on the Source and Target systems when you configure them.

Before you configure your Target system in Intersight, you will first need to configure
the upstream network of the Target system. This includes the Cisco Fabric Interconnects
and the upstream network.

The ESXi port group is automatically configured by Intersight on the management

vSwitch:

 Configure the Replication VLAN on your ToR switches.

 Configure a gateway on the Replication VLAN.

o The IP gateway is required even if you are performing the replication over Layer
2.

 Provide connectivity to the Source system Replication VLAN.

o You can use network overlays such as IPsec to establish connectivity between
locations.
 To verify the connectivity, you can initiate ping between gateways on the Target and
Source side.

Note
Do not use your Management network for the replication. Replication jobs can create a
lot of traffic and need to be reliable. Although the vSwitch is shared with the
Management VLAN, the Replication VLAN should be different.

The Replication VLAN is not configured automatically on the Fabric Interconnects and
needs to be enabled before the replication network is configured.

Configure the replication VLAN on FI from the Cisco UCS Manager:

1. Click LAN option in the LAN tab and go to VLANs sub-tab.

2. Click the add button (+) on the bottom of the VLAN list.
3. Provide the VLAN name and VLAN ID.

o The convention for Replication VLAN name is hx-inband-repl

4. Clisk OK to confirm the VLAN creation.

Defining the VLAN is only the first step in configuring the Replication VLAN in the
Cisco UCS domain. Once the VLAN is defined, you need to allow it on the server links.
The port group is assigned to the management vSwitch in vSphere, so you need to
enable the VLAN on the vNICs that are uplinks assigned to the management vSwitch.

To enable the VLAN, reconfigure the vNIC template in the UCS Manager:

1. Click the LAN tab in the Cisco UCS Manager and go to LAN > Policies > root > Sub-
Organizations > org. name > vNIC Templates > vNIC Template hv-mgmt-a
2. Click Modify VLANs.
3. Check the check box next to the VLAN you created for the Replication network.
4. Click OK to confirm the VLAN assignment.

After the network is configured, go to your Cisco Intersight account where you have
already registered your HyperFlex systems that you want to configure N:1 replication
for.

In Intersight, go to the HyperFlex Clusters pane to configure N:1 replication:

1. Click the ... icon next to the Target HyperFlex system and select Configure
Backup.
2. Fill out the form with the values you wish to configure on your Target system.

o The IP address pool needs to be equal or greater to the number of converged

nodes in the group plus one.
o The replication bandwidth must be greater or equal to the speeds of all Source
systems that will replicate to Target.

3. Click Next and Validate & Deploy to apply configuration.

The system will notify you if the configuration was successful. You can click individual
task to follow along with the configuration. Configuration is typically finished within a
minute.

If the configuration fails, you can click on a failed job for additional information.
Usually, failures result from a misconfigured network.

Note
The configuration you provide during the Target configuration steps will be saved to
policies automatically. You can manage all your policies in the Policies Intersight pane.

Configuring Source Cisco HyperFlex System

Configuration of the Source HyperFlex system is very similar to the configuration of the
target system, except that the Target used by a specific Source is also defined during the
configuration.

The source system must comply with the replication requirements:

 Target system must be reachable over the replication network.

 Source must be Edge systems licensed with Edge Premier HyperFlex license.
 Target system needs to already be configured.
 Source system and Target system must not be in the same vCenter.
If you place both the Source and Target systems in the same vCenter, you will be able to
configure the N:1 replication normally. Where you will encounter issues is with restores
of VMs. You can have all the Source systems in the same vCenter without an issue.

The network configuration for Source systems is similar to Target:

 Source systems do not include FIs.

 Allow Replication VLAN on all your ToR switch downlinks to Edge systems.
 Gateway on the Replication VLAN is required.
 Provide connectivity to the Target system Replication VLAN.

The replication network configuration is very similar on the Source and on the target
side. The WAN segment is optional. As long as there is connectivity between the
Replication networks of the Source and Target system, replication will work.

In Intersight, go to HyperFlex Clusters to configure Source N:1 replication:

1. Click the ... icon next to the Source HyperFlex system and select Configure
Backup.
2. The IP address pool needs to be equal or greater to the number of Edge nodes in the
group plus one.
3. Choose the backup Target system from the list of compliant HyperFlex systems.

o Set the backup interval according to the connectivity capabilities.

o Set the number of snapshots to retain with storage of the Target system in mind.

The IP addresses do not need to belong to the same subnetwork as on the Target system
if they are routable. You can also use Layer 2 communication for replication, but the
gateway is not optional, since its presence is verified during the replication
configuration.

If you set the backup interval that is too short for the link speed between Source and
Target, the next backup job will start before the first one finishes.

If you retain too many snapshots, you will run out of storage on your Target system and
there will be no space left for backups.

Both values can be changed by altering the backup policy later if you find that there are
issues with replication settings.

Only licensed 4.5(1a) systems will be valid targets for replication. As of May 2021,
Intersight does not check if the Target system is configured with replication. If it is not,
the Source configuration fails.

Click Validate & Deploy to configure your Source system:

1. You can follow the phases of deployment by selecting the deployment from the list.
2. The deployment will configure the replication interfaces of CVMs on the Source
system.
3. On the Target system, a new replication datastore will be provisioned automatically.
4. Replication network port group will be configured automatically on Source hosts.

If there are failures, Intersight will provide possible solutions to the encountered
problem. Most often the issue will be network-related.

Replication Workflows
Once the Source and Target systems are configured, the replication will start
automatically between the source and target datastores. You can manage the replication
jobs by accessing the Backups tab under HyperFlex Clusters in Intersight.

The Backups tab provides you with the options to manage your backups:

1. You can browse protected VMs and systems as well as see recent backup and
restore activity.
2. Potential issues with restore jobs will be listed in the restore and backup activity
screens.
3. Intersight notifications will be triggered on replication job issues.
Note
The name source and target can be relative to the direction VM actions are performed.
Here, references to Source and Target are made from the perspective of initial VM
location to have consistency and avoid confusion. Source and Target are also capitalized
as system names for clarity.

You can initiate restore jobs from the Protected Virtual Machines pane:

1. Select the VM you wish to restore and click the ... button at the end of its row.
2. Select Restore from Backup from the resulting menu.
3. A list of snapshots will be displayed for the selected VM.
4. Select the snapshot from which to restore the VM and click Next.
5. All the Source systems associated with the Target from which you are restoring will
be listed.
6. Source Cluster designates the system where the VM originates.
7. You can deploy the VM on any of the associated Source systems.
8. Select the Source system to which you want to restore the VM and click Next.
9. Set the name for the restored VM and its power state after deployment.
10. Click Next to proceed to the summary overview for the replication job.
11. You can return to any of the steps to change the restore settings.

12. You can follow the restore job in the Intersight Active Requests list.

13. The restored VM will be restored on the selected Source site and registered to
vCenter.
14. The VM will be powered on automatically and named according to your previous
selection.
15. If you restored the VM to a Source system different than the system where it
originated, you need to make sure that the networking is set up appropriately.

The restore functionality is useful in migration scenarios, not only restores from backup,
since you can select any Source system on restore.

9.8 Protecting Your Data

Data at Rest Encryption

This video covers Data at Rest Encryption and D@RE Remote Key Management. Cisco
HyperFlex can be installed in different customer environments. Some of these customers such
as banks and military might have very strict security protocols in place for their data centers.
These kinds of customers might require Data at Rest Encryption. Fortunately, Cisco HyperFlex
supports this solution. It allows the data to be encrypted on the hardware or disk level.

Now, to support this feature, you need to order your HyperFlex system with the self-
encrypting drives or SEDs, which are a little bit more expensive. All of the drives in the clusters
need to be SEDs, and you will see the encryption menu in the HyperFlex Connect.
For hardware-encrypted HyperFlex systems, you will need the SEDs along with a key
management solution. The SEDs have special hardware that encrypts the incoming data and
then decrypts the outgoing data in real time. The data on the disk is always encrypted. A media
encryption key controls this encryption and decryption, but the key is never stored in the
processor or in memory.

A security key, also known as the key encryption key or an authentication passphrase, is used
to encrypt the media encryption key. Now, to enable SED, you must provide a security key.
However, no key is required to fetch the data if the disk is not locked. The Key Management
Server manages cryptographic keys and an encrypted system. It includes generating keys,
storing keys, destroying keys, and replacing keys.

You can manage keys on a HyperFlex system using a dedicated Key Management Solution
using Key Management Interoperability Protocol, or KMIP, which is the recommended
approach. Or, you can use locally stored keys, where if lost, cannot be retrieved, and the data
is lost when the drive power cycles. Obviously, this is not the recommended design.
Let's look at the D@RE Remote Key Management steps. First, you obtain a local self-signed
root certificate or one from a trusted third-party certificate authority, or CA. Enter the root
certificate into the HX Encryption field that asks for the cluster key. Create an SSL server
certificate, and generate a certificate signing request, or CSR. Sign the CSR with a root
certificate, update the KMIP server settings to use the client certificate, and lastly, proceed
with a KMIP service configuration.

In HyperFlex Connect, choose Encryption and Configuration encryption. Provide UCS manager
host name and credentials, choose Key Management Server, configure a server with SEDs in
the cluster using either the self-signed or the CA certificates, and then lastly, generate the
certificate signing request or self-signed certificates.
When it comes to generating certificates, the workflow is same whether you're doing self-
signed or CA certificates. Just choose either use certificate authority, sign certificates, or use
self-signed certificates. To generate the remote encryption key, fill in the email address, the
organization name, and the other fields as well as the validity period and days. Then, click
Generate Certificates. Lastly, download the certificates and get them signed if you're using a
certificate authority. Then, configure the KMIP server.

Finally, to configure the KMIP server on Cisco HyperFlex Connect, choose Encryption and
Continue configuration. Choose Manage certificates to upload the CSRs, choose Upload
certificate authority signed certificates if you use the CA, choose Configure key management
server to configure that KMIP server settings, and lastly, click Save to encrypt the cluster with
the remotely managed keys.

That brings us to the end of this video. Thanks for watching.

Cisco HyperFlex can be installed in different customer environments. Some of these

customers, such as banks and military, might have very strict security protocols in place
for their data centers. This kind of customers might require data at rest encryption and
Cisco HyperFlex supports this option.
Note
Key verticals for data at rest encryption include healthcare (HIPAA), financial services
(PCI-DSS), insurance, legal, federal (NIST, FedRAMP, FISMA), and state laws. The
platform itself is hardened to Federal Information Processing Standard (FIPS) 140-1,
and the encrypted drives with key management comply with the FIPS 140-2 standard.

HyperFlex offers an option to have your data encrypted on the hardware (disk) level:

 You would use this kind of encryption for security or compliance (regulatory)
purposes.

o HyperFlex will allow you to encrypt all data on disks, securely expand the
deployment, disable/enable encryption, and securely erase disks.

 You must order your HyperFlex system with SEDs.

o SEDs are a little bit more expensive than regular drives.

o All drives in the deployment need to be SEDs.
o If your system is encryption-capable (has SEDs), you should see
an Encryption menu option in HyperFlex Connect.

The encryption and decryption for SEDs is done through hardware. Thus, it does not
affect the overall system performance. SEDs reduce the disk retirement and
redeployment costs through instantaneous cryptographic erasure. You perform
cryptographic erasure by changing the media encryption key. When system applies
change to the media encryption key to the disk, the data on the disk cannot be
decrypted, and is immediately rendered unusable.
In a Cisco HyperFlex Edge system, the lack of Fabric Interconnects means that Cisco
UCS Manager also is not used. As a result, SEDs are not supported. Therefore, to
encrypt the system virtual machines, a software solution, VMware VMcrypt, is used.

Note
Not all the capacities and types of drives that are available as
regular drives are also available with data at rest encryption
systems. For example, as of August 2019, data at rest
encryption systems were not available with NVMe or Optane
drives.

Note
When installing a data at rest encryption deployment, make sure that you are using
ESXi, Cisco UCS, and HXDP software versions that are supported. Those versions
might not be the same as for a system with regular disks. Consult with the latest Cisco
HyperFlex Data Platform Installation Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/AdminGuide/4-5/b-hxdp-admin-guide-4-5.html).

Encrypted System Components

If you want a hardware-encrypted HyperFlex system, you need a HyperFlex system

with SEDs and a key management solution.

SEDs have special hardware that encrypts incoming data and decrypts outgoing data in
real time. The data on the disk is always stored in encrypted form. A media encryption
key controls this encryption and decryption. This key is never stored in the processor or
memory.

A security key, also known as key-encryption key or an authentication passphrase, is

used to encrypt the media encryption key. To enable SED, you must provide a security
key. No key is required to fetch the data if the disk is not locked.

Key management server manages cryptographic keys in an encrypted system. It

includes generating keys, storing keys, destructing keys, replacing keys, and so on.
Note
Procedure of replacing SED drives is different than the procedure with regular drives.
To ensure that the encrypted data remains secure, the data on the drive must be securely
erased prior to removing the SED. For more information, refer to the latest Cisco
HyperFlex Data Platform Administration Guide.

Key Management Solution

You can manage keys on the HyperFlex System using the following options:

 A dedicated key management solution:

o KMIP 1.1 software (such as SafeNet KeySecure or Vormetric DSM)

o Third-party solution not supported or maintained by Cisco
o Recommended approach

 Locally stored keys:

o Remember the key. If you forget it, it cannot be retrieved, and the data will be
lost if the drive power cycles.
o Should only be used in a proof-of-concept scenarios
o You can first configure local key storage and then migrate to a Cisco Centralized
Key Management solution.

Note
Key Management Interoperability Protocol (KMIP) is a communication protocol that
defines message formats for the manipulation of cryptographic keys on a key
management server. KMIP simplifies encryption key management.

9.9 Protecting Your Data

Data at Rest Encryption: Remote Key

Management
Data security is vital in certain environments, and Cisco HyperFlex provides the support
for this option. When the data handling policy of your company demands data
encryption, you can order self-encrypting drives when ordering your HyperFlex
deployment. HyperFlex will also provide you with tools to manage the drives and drive
encryption keys. HyperFlex Connect management system integrates with well-
established third-party key-management solutions to make data at rest encryption as
simple to manage as possible.
Note
An alternative to an external key management server is using local keys. However,
storing keys locally is not the recommended design. If you are interested in workflows
using local encryption keys, refer to the latest Cisco HyperFlex Data Platform
Administration Guide.

For remote KMIP certificate handling, follow these general steps:

 If you are self-signing, specify the local certificate authority in the configuration and
get a root certificate.
 If you are using a trusted third-party certificate authority (CA), then specify that in
the configuration and use their root certificate.
 Enter the root certificate in the HyperFlex encryption field that asks for the key.
 Create an SSL server certificate and generate a certificate signing request (CSR).
 Sign the CSR with whatever root certificate you are using.
 Update the KMIP server settings to use the client certificate.
 With the SSL certs and root CAs available, proceed with the KMIP service
configuration specific to the vendor you have chosen.

Note
For remote key management, you will need a KMIP 1.1 server. Cisco has validated the
SafeNet KeySecure and Vormetric DSM key management systems.

Configuring Remote Encryption Key

In the HyperFlex Connect navigation, choose Encryption, and Configure encryption:

 Provide Cisco UCS Manager hostname and credentials.

 To secure the HyperFlex deployment using a remote security key generated by the
key management (KMIP) server, choose Key Management Server.
 You can configure a server with SEDs in the deployment to use one of the following
certificates.

o Use certificate authority signed certificates.

o Use self-signed certificates.

 The next step is for you to generate the certificate signing request or self-signed
certificate.

Note
Certificate authority signed certificates would have you generate CSRs signed by an
external certificate authority.

Generating Certificate Signing Requests

There is no difference in workflow for generating either of the certificates that you
intend to sign with external authority or self-signed certificates:
 Choose either Use certificate authority signed certificates or Use self-signed
certificates.
 To generate the remote encryption key for configuring the KMIP server, you will
need to fill in email address, organization name, organizational unit name, locality,
state, country, and validity period in days.
 Click Generate certificates.
 The next two steps are:

o Download the certificates to get them signed by a certificate authority (not

applicable for self-signed certificates).
o Configure KMIP server.

Note
Workflow of signing of the certificate depends on the solution you are using.

Configuring KMIP Server Using CSRs

On the Cisco HyperFlex Connect navigation pane, choose Encryption, and Continue
configuration:

 Choose Manage certificates to upload the CSRs. Fill in Cisco UCS Manager
hostname and credentials.
 Choose Upload certificate authority signed certificates and upload certificate that
you signed.

o Step does not apply if you are using self-signed certificates.

 Choose Configure key management server to configure the KMIP server. Fill in
the IPs of the KMIP server, port number, and public key.
 Once you click Save, the HyperFlex system will be encrypted with remotely
managed keys.

Note
Ensure that you have signed certificates with a certificate authority if not using self-
signed certificates.

9.10 Protecting Your Data

Protect Your Cisco HyperFlex VMs

Protect Your Cisco HyperFlex VMs
Although Cisco HyperFlex has a lot of redundancy built into its design, there are certain
cases where you will want to have your data protected on another group or another site.
You can employ external tools that integrate with Cisco HyperFlex storage, such as the
Veeam data protection suite, but Cisco HyperFlex offers native replication
functionalities between groups. These functionalities have the advantage of being
automatically available without additional licensing, and you can use them when two
compatible HyperFlex groups are joined in a network.

You will get to know how to set up one such replication between the groups and inspect
the replication jobs that are performed over the network.

Note
Click the Initialize button below to start the lab. Initialization takes 15–60 minutes. A
progress bar will show you when the lab is initialized and active. Please be patient with
the initialization process; several components are loading and getting ready!

Configure Replication Network and Pair the Groups

Native replication enables you to replicate VMs between two Cisco HyperFlex groups
without any additional licensing or external solutions. This functionality is integrated in
the Cisco HyperFlex solution.

To enable native replication between HyperFlex groups, you first need to configure the
network between the two groups.

Step 1

Verify the state of the HyperFlex group.

Answer
Open two tabs in your browser. In one of the tabs, enter https://hxconnect-a.hx.pod/.
In the other tab, enter https://hxconnect-b.hx.pod/. Log in to HyperFlex Connect in
both groups, using the admin / C1sco12345 credentials.

Verify that both groups are Online and Healthy before you proceed.
Step 2

Investigate the datastores and VMs on both groups.

Answer
In HyperFlex Connect of Cluster A and Cluster B, go to the Datastores pane, and verify
that there are HyperFlex datastores available on both.

Go to the Virtual Machines pane of Cluster B, and verify that there are 12 VMs
available on the groups.

HyperFlex native replication is a VM-based replication solution. You will need the
VMs to replicate them to Cluster A.

Step 3

Configure the replication network on Cluster B.

Answer
In HyperFlex Connect of Cluster B, go to the Replication pane, and click Configure
Network.

In the VLAN Configuration step, set VLAN 0. This action will prevent
HyperFlex Connect from configuring the VLAN in the Cisco UCS
Manager.

Note
In nearly all physical deployments, you would use a separate non-native VLAN for the
replication. You are using the native VLAN because the network in the lab environment
is virtual and already uses VLAN tagging on the vSwitch port group to separate
networks.
Click Next to proceed to the IP & Bandwidth Configuration step.

Configure the replication subnetwork and the IP address range for Cluster B. Use the
Job Aids to fill in the required information.

Notice that the replication network uses a different subnetwork than the management
address space. The system will not allow you to use the same subnetwork for replication
and management.

After you add the beginning and end values for the IP Range, click Add IP Range to
add the range to the list.
The network will need to be the same on both group that are engaged in the replication,
but the IP address space will need to be unique on each site; otherwise, the IP addresses
will overlap.

You could also provide a bandwidth limit, which would prevent the replication jobs
from taking up the entire bandwidth. You can also define the MTU size. Use an MTU
size that is lower or equal to the lowest MTU defined on the network connecting the
groups involved in replication.

Note
The minimum number of IP addresses required is the number of nodes in your
HyperFlex group plus one. For example, if you have a 4-node HyperFlex group, enter a
range of at least 5 IP addresses. If you plan to add nodes to your group, include enough
IP addresses to cover any additional nodes. You can add IP addresses at any time, but
the minimum must be satisfied or the system will report an error. The replication
network will take about a minute to come online.

Click Configure to continue.

The Test Configuration tab will become active, and the steps taken when configuring
the replication network will be listed.

After the configuration has been successfully performed, Cluster B is ready for
replication jobs.

Click Close to close the configuration window.

Before you will be able to perform HyperFlex native replication jobs, you will need to
configure another HyperFlex group to act as the replication target.

Step 4

Review the replication network configuration.

Answer
Go to the pod vCenter by typing https://vcenter.hx.pod/ into your browser. Log in to
the HTML5 interface using the
credentials administrator@hx.pod / 1234QWer* credentials.

In vCenter, expand Standard Cluster B in the Navigator menu, and choose stCtlVM-
10.1.100.211 from the list.

Click the Summary tab on the right to display VM properties.

Note that the VM that you chose is running on the first node of Cluster B, which is the
group that you configured the replication network on.

In the Summary pane of the stCtlVM-10.1.100.211 CVM, click on the View all 6 IP
addresses link to reveal CVM IP addresses.

In this case, a primary node of the HyperFlex group was chosen, which can be identified
by the 6 IP addresses instead of only 3 that can be seen on other non-primary VMs. The
primary node has additional addresses assigned, these are virtual IP addresses for
management, data, and replication network.

You will see IP addresses in the 10.2.100.10-20 range configured on the CVM. These
were automatically configured when you configured the management network in
HyperFlex Connect.
Remember an IP address in this range as you will ping it to test the replication network
connectivity of the sites after you configure Cluster A. If the chosen node is a primary
node, there will be two. Choose any of the two.

Step 5

Configure the replication network on Cluster A.

Answer
Open a new tab and go to https://hxconnect-a.hx.pod/. Log in
using admin / C1sco12345 credentials.

In HyperFlex Connect of Cluster A, go to the Replication pane, and click Configure

Network.

In the VLAN Configuration step, set VLAN 0.

Click Next to proceed to the IP & Bandwidth Configuration step.

Configure the replication subnetwork and the IP address range for Cluster A. Use the
Job Aids to fill in the required information.

After you add the beginning and end values for the IP Range, click Add IP Range
button to add the range to the list. Subnet and gateway IP addresses must be the same as
on Cluster B, but the IP Range must be different.

Note
The configuration is nearly identical to Cluster B, but if improper values are provided at
any point the replication will not function, since there will be no connectivity between
the sites.
Click Configure to continue.

The Test Configuration tab will become active, and the steps taken when configuring
the replication network will be listed.

Click Close to close the configuration window.

After the configuration has been successfully performed, Cluster A is also ready for
replication jobs.

Now you have two HyperFlex groups configured for replication.

Step 6

Test the replication network connectivity.

Answer
Click the blue and white button in the top-left corner of the Student VM and
choose Terminal Emulator from the menu.

Type ssh -l root 10.1.100.121 into the command prompt and press Enter to connect to
the first CVM of Cluster A. Use the password C1sco12345 to log in.

student@student-vm:~$ ssh -l root 10.1.100.121

Failed to add the host to the list of known hosts (/home/student/.ssh/known_hosts).
HyperFlex StorageController 4.5(1a)
root@10.1.100.121's password:
---------------------------------------------------------
!!! ALERT !!!
This service is restricted to authorized users only.
All activities on this system are logged. Unauthorized
access will be reported.
---------------------------------------------------------

HyperFlex StorageController 4.5(1a)

root@HX-CTL-01:~#

Type ifconfig to display the network interface configuration of the CVM.

root@HX-CTL-01:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0c:29:a2:55:cc
inet addr:10.1.100.121 Bcast:10.1.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:220432 errors:0 dropped:95 overruns:0 frame:0
TX packets:35889 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:258700157 (258.7 MB) TX bytes:15756885 (15.7 MB)

eth1 Link encap:Ethernet HWaddr 00:0c:29:a2:55:d6

inet addr:192.168.0.51 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4773039 errors:0 dropped:19 overruns:0 frame:0
TX packets:4764245 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1129050557 (1.1 GB) TX bytes:1239003859 (1.2 GB)

eth1:0 Link encap:Ethernet HWaddr 00:0c:29:a2:55:d6

inet addr:192.168.0.150 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

eth2 Link encap:Ethernet HWaddr 00:0c:29:a2:55:e0

inet addr:10.2.100.31 Bcast:10.2.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:78961 errors:0 dropped:50 overruns:0 frame:0
TX packets:80294 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7358137 (7.3 MB) TX bytes:7500294 (7.5 MB)
<... output omitted ...>

The Eth2 interface is configured with one of the IP addresses you have assigned for
replication to Cluster A.

Note
The IP addresses are assigned automatically from the provided pool and the CVM you
are logged into can have a different IP address assigned.

Ping the CVM you have inspected previously in the vCenter.

root@HX-CTL-01:~# ping 10.2.100.13
PING 10.2.100.13 (10.2.100.13) 56(84) bytes of data.
64 bytes from 10.2.100.13: icmp_seq=1 ttl=64 time=2.67 ms
64 bytes from 10.2.100.13: icmp_seq=2 ttl=64 time=0.945 ms
64 bytes from 10.2.100.13: icmp_seq=3 ttl=64 time=1.23 ms
64 bytes from 10.2.100.13: icmp_seq=4 ttl=64 time=0.261 ms
<... output omitted ...>

If you have configured the replication network correctly, the remote CVM of Cluster B
should respond. The network connectivity for replication is configured properly.

Step 7

Create a native replication pair from Cluster A and Cluster B.

Answer

In HyperFlex Connect of Cluster B, go to the Replication pane, and choose Pair

Cluster link under Cluster Pairing.

By pairing Cluster A with Cluster B, you are specifying that all VMs on Cluster A that
are explicitly set up for replication, can replicate to Cluster B, and that all VMs on
Cluster B that are explicitly set up for replication can replicate to Cluster A. Pairing is
strictly 1-to-1. A datastore on a paired group can be paired with no more than one
datastore on the other group.

In the Name step, provide the name for the group pair. Name it Paired_Sites_AB.
Click Next to continue.

In the Remote Connection step, provide the IP address of the HyperFlex Connect
belonging to the remote group (Cluster A) and use the admin / C1sco12345 credentials.

Click Pair to start group pairing. This process should finish in under a minute.

In the Run Tests step, the HyperFlex system will verify that the CVMs on both groups
are reachable over the replication network.

Note
Since you are performing this task using an emulated environment, you may experience
an error while completing this step due to the slow responsiveness of the emulated
environment. In this case, reload the HyperFlex Connect site in the browser and proceed
with the next step. Optionally you can also verify that the task was successfully
completed in the background by checking the recent tasks list. This list can be accessed
by choosing the Activity pane located in the left menu of HyperFlex Connect.

Click Map Datastores to proceed to datastore mapping.

In the Datastore Mapping step, choose to pair Cluster B Datastore, where you have
deployed all the VMs, and HyperFlex Datastore on Cluster A. Leave CLI_Datastore
unpaired.

Click Map Datastores to continue.

The Replication pane should now display the group pair and the datastore pair (but no
VMs that are protected).
In the next task you will need to choose the VMs, which you would like to replicate
between sites.

Protect a VM from Cluster B on Cluster A

Step 8

Set up VM protection by creating a VM protection group.

Answer
In the Replication pane of the Cluster B, click the Local VMs subtab and click on
the Protection Group link to reveal the status of the protection groups.

You will notice that a protection group is not yet created.

Choose +Create Group on the right, name the new group VMProtectionGroup, and
set the protection interval to 1 hour.
Make sure the Start protecting the virtual machines immediately radio button is
chosen.

Click Create Protection Group to continue.

You are ready to tell the system which VMs on Cluster B it should protect by
replicating them to Cluster A.

Step 9

Protect the vSphere Tiny Clone VM.

Answer

On HyperFlex Connect of Cluster B, go to the Virtual Machines pane.

Search for vSphere Tiny Clone and choose it.

Click Protect on top of the VM list to define the protection group for the VM.

Add the VM to the protection group VMProtectionGroup.

Click Protect Virtual Machine to continue.

The Protection Status column will show the vSphere Tiny
Clone as Protected by VMProtectionGroup, which allows you to quickly see the
VMs that are protected and by which protection group.

The vSphere Tiny Clone VM will now be protected according to the protection group
settings.

Note
You could also protect the VM from this same interface without adding it to a
protection group. You would have to provide the configuration values that would
otherwise be inherited from the protection group.

Step 10

Verify that the protected VM has been successfully protected on Cluster A.

Answer
Go to the HyperFlex Connect of Cluster A, open the Replication pane, and click
the Remote VMs tab.

Note
If you are already in the Replication pane, you will need to re-open
the Replication pane to see the changes.
The VM is listed as Protected and has the last successful protection job time listed.
Therefore, the VM has been successfully protected at the specified time. If restored, it
will go back to the state at the last protection time. The information written to the
original VM since the last replication will be lost. The RPO for the VMs in
the VMProtectionGroup is 1 hour.

To learn more about the process of replication and data transfers, click the Replication
Activity tab.

The Replication Pairs tab shows the status of the replicating groups and summary of
VM transfers between replicating groups.
The VM is now protected and its replica will always be kept up to date in 1-hour
intervals until the Standard Cluster A is available.

Perform Test Recovery of a VM

Step 11

Execute a test recovery of the protected VM.

Answer

On Cluster A, go to the Replication pane, and click the Remote VMs subtab and
the All Protected VMs link.
Choose the VM vSphere Tiny Clone and click Test Recovery.

In the resulting menu, select Modify recovery configuration for current

operation and provide the restored VM name Restored Tiny Clone. Input the details
needed for the target in the Recovery Rules according to the image below.

The test recovery workflow is intended to verify the state of a VM on a local group. It
does not restore the VM on the original site, that is done by using
the Recover workflow.

Note
The actual recovery is currently not supported in the emulated environment and cannot
be performed.

Click Recover to initiate the test recovery.

Step 12

Verify that HyperFlex has recovered the VM on Site A and powered it on.

Answer
Go to the vCenter, and expand the Standard Cluster A. You will see the restored and
powered on VM named Restored Tiny Clone in the inventory of the Cluster A.
Note
Other restoration workflows are currently not supported by the emulator but are
initiated in the same way from the HyperFlex Connect.

By completing this discovery, you became familiar with HyperFlex native replication
which can greatly increase resiliency and protect your data if there is multi-group or
multi-site deployment.
10.1 Introducing Cisco HyperFlex Stretched Deployment

Introduction
Cisco HyperFlex Stretched deployment is a site-wide resiliency configuration of a Cisco
HyperFlex system. This deployment type uses a two-side infrastructure setup
interconnected with an intersite link that provides synchronous data replication across
two halves of the HyperFlex deployment.

Cisco HyperFlex Stretched deployments have the highest resiliency of all deployment
types and use the replication factor 4 (2+2). Even on group sizes of 4, this replication
factor ensures that two full nodes can be lost before the system goes down.

Cisco HyperFlex Stretched deployment was introduced in HXDP Version 3.0(1a) and is
an established deployment in mission-critical workload scenarios.

10.2 Introducing Cisco HyperFlex Stretched Deployment

Stretched Deployment Overview

This video provides an overview and prerequisites for a Stretched Cluster. Cisco HyperFlex
Stretched Cluster is the new flavor of HyperFlex that Cisco introduced with version 3.0. It
enables you to deploy an active-active disaster avoidance solution for mission-critical
workloads. The Stretched Cluster is basically a single cluster with nodes geographically
distributed across two locations. It's like taking one cluster and just cutting it in half and placing
each half in different data centers. Both sides of the cluster act as primary for certain
user virtual machines.
A Stretched Cluster enables you to access the entire cluster even if one of the sites goes down.
Now, in order for this all to work, a third site is required for hosting a witness as a tiebreaker in
the event of a split-brain scenario, which I'm going to talk about a little bit more later on. The
motivation for deploying a Stretched Cluster is no downtime, easy site maintenance, and no
complex disaster recovery tools.

Like I said, the data center houses each half of the Stretched Cluster—and this is in two
different data centers. These data centers can be in the same building or they could be in
separate buildings. Literally, how far apart those two buildings, is dictated by the quality of the
link that you have between the two sites. But, typically, the two buildings are on the same
campus, just in different buildings for security purposes-- or disaster recovery purposes, I
should say.
There are supported features and limitations for the Stretched Cluster as of HyperFlex version
3.5.1. Stretched Clusters support a 2-to-8 converged server per site, with a 4-to-16 nodes
cluster size. Compute nodes are supported with up to eight per site, with a total cluster size of
32 nodes. Symmetrical cluster expansion is supported, HyperFlex native replication is
supported, however, SEDs, Hyper-V, and logical availability zones are not supported.

Before you begin planning your Cisco HyperFlex Stretched Cluster, it's important that you
understand the prerequisites. When it comes to Cisco UCS, both sides have to have the same
number and types of servers, each site needs a pair of Fabric Interconnects—and they must be
the same type within a site but they can be different between the two sites—only M5 UCS
servers are supported, and, lastly, Stretched Clusters are for new deployment only-- there's no
upgrades for the regular clusters.
When it comes to the network, the Stretched Cluster requires 10 gigabits dedicated
throughput with five milliseconds round trip latency between the sites. Five milliseconds
latency is a very, very strict guideline. If the round trip latency is consistently higher, then users
are going to experience a significant delay in operations.

The maximum recommended Stretched Cluster site distance is 100 kilometers, or 62 miles, so
that's quite a bit. Existing Fabric Interconnects are supported as long as they support the M5
servers. And, lastly, subnet stretched across the sites are for HyperFlex management and
HyperFlex storage.

If you remember, I mentioned the witness is the tiebreaker for a split-brain scenario. This
condition occurs when both sites are still available but they're unable to communicate with
each other. This is where the witness comes in. The witness is responsible for deciding which
site survives-- flip a coin, one's got to survive.
It's critical that a VM site leader is established so the two instances of the same VM are not
accidentally brought online by high availability. Both cluster sites must have connectivity to
that third witness site. 100 megs, 200 milliseconds round-trip latency is required between all
three sites. And, lastly, that third site must have the capability to run an OVF image.

Finally, vCenter and ESXi requirements include a single vCenter for both sites, vSphere version
6.OU3 or 6.5U1 and 6.5U2, also, is supported. vSphere Enterprise Plus edition is highly
recommended. And, lastly, you must prepare vCenter at a third site—probably the same one
that you're going to put that witness on. That brings us to the end of this video. Thanks for
watching.

Because of its specific design, there are additional considerations where implementing
Stretched HyperFlex deployments. There is also additional hardware required to ensure
fast and low latency connectivity for both sites.

Stretched deployment is one HyperFlex server group with nodes geographically

distributed across two sites:

 Storage will mirror data between sites.

o Sites are connected using a low-latency network.

 It is an active-active solution—both sites run workload VMs.

 A third site is required to host an arbiter witness.

o Responsible for deciding primary site when sites lose the intersite link (split
brain scenario).

In Cisco HyperFlex, a witness is a single VM that you need to deploy on a third site. A
witness is a VM that you import into an ESXi or ESXi group.

The two sites are two data centers, each housing half of your stretched deployment.
These data centers can be in the same building or in two different buildings. How far
apart those two buildings can be is dictated by the quality of the link between the two
sites. Typically, the two buildings are within the same campus.
Benefits of deploying a Stretched deployment over a Standard deployment include the
following:

 No downtime in case one site fails.

o All the data resides on both sites, so you can tolerate a failure of one entire site,
assuming that you have enough processing (CPU/memory) power on one site to
run all needed VMs.

 Easy site maintenance.

o Bring all servers in one site into maintenance mode and the workload will
seamlessly run on the other site until you are finished.

 No need for a disaster recovery orchestration tool or other complex procedures.

o A stretched deployment relies on a native hypervisor (in this case, VMware), a

mechanism to allow for resiliency.
o If a site fails, VMware high availability will make sure to turn on VMs on the
second site. Your VMs should be running on the second site in a matter of
seconds.

If a site fails, you can perform site maintenance without service disruption. You would
migrate (for example, using VMware vMotion) VMs from one site to the other, and
perform site maintenance without any interruption to your business. The upside of a
Stretched deployment over a Standard deployment is its advanced high availability.

However, you also need to be aware of the costs. A Stretched deployment uses
replication factor 4 (RF4) (RF2 per site), which means its usable capacity is 50 percent
less than an RF2 Standard deployment. You will also need two pairs of Cisco Fabric
Interconnects: one pair for each site. There are further constraints when deploying
Stretched deployment.

Constraints of Stretched Deployment in Cisco HXDP 4.5(1a)

Constraints are subject to change in subsequent HyperFlex versions.

 2–16 converged servers per site with SFF drives, which is 4–32 per deployment

o Max ratio compute-only:converged of 2:1 and maximum group size of 64 nodes

 2–8 converged servers per site with LFF drives, which is 4–16 per deployment

o Max ratio compute-only:converged of 2:1 and maximum deployment size of 48

nodes.

 Supports all storage types including NVMe.

 SED drives are not supported.
 Expansion with converged nodes must be symmetrical.
 LAZs are not supported.
 Hyper-V as hypervisor is not supported—ESXi only.
 HyperFlex native replication is supported.
 HyperFlex acceleration engine is not supported.

Cisco continues developing HyperFlex at a rapid pace; you can expect reduced
limitations in releases after HXDP 4.5(1a).

While a Stretched deployment is a disaster recovery mechanism, it should not eliminate

a backup solution. A backup solution, such as Veeam or Cohesity, will allow you to
perform disaster recovery if both your sites are affected, allow you to restore data on a
file level, protect yourself against malware attacks, and much more.

10.3 Introducing Cisco HyperFlex Stretched Deployment

Prerequisites
Before you begin planning your Cisco HyperFlex Stretched deployment, it is important
that you understand the prerequisites concerning these four areas:

 Cisco UCS requirements

 Networking (between sites and to the witness)
 Witness requirements
 vCenter and ESXi requirements

A Stretched deployment has these UCS requirements:

 Both sites must have the same number and type of converged servers.

o Also, all limitations apply, just as they do for one single, Standard deployment.

 Each site needs a pair of Fabric Interconnects.

 M5 UCS converged only—no mixing with converged M4 allowed.
 New deployments only—no upgrade from Standard deployments.
 Two separate, manually synchronized fabric interconnect domains are required.
All server limitations of Cisco HyperFlex deployments apply, including guidelines (as
they do for Standard deployments), such as:

 You cannot mix storage types (HDD/SSD) within a deployment.

 You cannot mix node form factors within a deployment (HX220c SFF, HX240c
SFF, and HX240c LFF).
 You cannot mix cache disk types (SAS SSD, NVMe SSD, and Optane NVMe SSD).
 All servers in the system should have the same number of capacity disks.
 It is not necessary that all nodes in a deployment have the same CPU type and
amount of memory, but it is highly recommended that you keep them similar for
VM failover scenarios.

The required two pairs of Cisco Fabric Interconnects are automatically configured
during Cisco HyperFlex installation; the Cisco UCS Manager management and intersite
upstream connectivity needs to be configured by the user.

Network setup is very important for Stretched deployments:

 Minimum 10 Gbps of dedicated bandwidth and maximum 5 ms RTT latency

between the sites is required.
 Existing Fabric Interconnects, like with standard deployment, are supported if they
support HyperFlex M5 servers.
 Management subnet needs to be reachable from vCenter and the witness VM.
 Storage VLAN needs to be reachable by ESXi and CVMs on both sites through the
intersite link.

Connectivity requirements between sites call for a round-trip-time (RTT) parameter

below 5 ms, which is a very strict guideline. You need to know what this means; data
needs to travel from Site A to Site B, be processed, and then returned to Site A, all
within 5 ms. If RTT is consistently higher, users will experience a significant delay in
operations. The maximum recommended Stretched deployment site distance is 100 km
or 62 miles. However, you need to verify that the RTT is consistent and below 5 ms.
Witness has these requirements:

 Both sites must have connectivity to the third witness site.

 Minimum 100 Mbps, maximum 200 ms RTT latencies between the sites and witness
are required.
 The third site must have the capability to permanently run a vSphere VM.

o As of HXDP 4.5(1a), only ESXi was validated as the third site platform.
o It is recommended that you configure the witness with high availability.

A quorum, or most votes, is the minimum number of votes that a distributed transaction
must obtain to be allowed to perform an operation in a distributed system. A quorum-
based technique is implemented to enforce consistent operation in a distributed system.
The witness node serves this function. In the event of split-brain conditions, where both
sites are still available but unable to communicate with each other, it is critical that a
VM site leader is established, so that two instances of the same VM are not brought
online by high availability.

You should assure high availability of the witness by deploying another instance of the
witness VM in high availability mode on another server or site.

Between the witness and the two sites, you need to have all connectivity allowed, or you
need to make sure that the following communication is allowed:

Port Number Service/Protocol Source Port Destination Essential Information

2181 2888 Bidirectional, management

Witness node/TCP Witness Each CVM node
3888 addresses

Bidirectional, management
8180 Witness node/TCP Witness Each CVM node
addresses

Potential future
80 HTTP/TCP Witness Each CVM node
requirement

Potential failure
443 HTTPS/TCP Witness Each CVM node
requirement

Here are the minimum requirements for a witness node:

 vCPUs: 4
 Memory: 8 GB
 Storage: 40 GB

While user data is not being sent between the sites and the witness, some storage-related
metadata does traverse to the witness site, which is why there is a 100-Mbps
requirement and is in line with competitive products.

Here are the vCenter and ESXi requirements:

 Use a single vCenter for both sites.
 ESXi 6.5 U3, and 6.7 U3 and 7.0U1d versions are supported as of HXDP 4.5(1a).
 vSphere Enterprise Plus edition is highly recommended.

o Standard edition of vSphere is supported, but discouraged since it does not have
DRS.

 You must install vCenter on an external site—usually where you run the witness
VM.

o It is recommended that you configure vCenter with high availability.

vCenter is an important component for non-Stretched deployments, but is critical for

Stretched deployment. vCenter, with high availability and DRS configured
automatically, manages the VM movement in the event of a site failure. Due to the role
of vCenter in Stretched deployment failures, nesting the vCenter is not supported on
Stretched HyperFlex deployments.

10.4 Introducing Cisco HyperFlex Stretched Deployment

Data Distribution
Now let's take a look at data distribution in a stretch cluster. With a stretch cluster, data
distribution, which is the read-write process as well as the resiliency mechanisms, are very
similar to a regular cluster. However, we are going to focus on how the handling of the data is
different between the two. The figure here shows an example with two sites, each site has two
Fabric Interconnects, three servers, and two switches. Plus, the sites are interconnected with 1
0 gigs and they have connectivity to the third site, which you'll remember, that's where the
witness is located to be used for a tie breaker.

Under normal stretch cluster operations, a write is committed into four different copies, two
of them are local and two of them are remote. In this figure, the first server in Site A is running
two virtual machines and they're labeled A and B. And the first server in site B is running one
virtual machine, which is labeled C. Virtual Machine A will write the first two data chunks, copy
A1 and A2, locally into the first two servers in their site. OK? And then it'll also write the third
copy and a fourth copy over on the servers in Site B. OK?
The write process is finished when the controller VM receives an acknowledgment from all
four servers for a given data chunk. So since two copies are kept locally and two copies are
kept remotely, this is considered a replication factor of RF4, which is commonly referred to as
RF2+2. And by the way, this is the only possible replication factor that you can use for a stretch
cluster. Once again, that's RF4, which is RF2+2.

Now, in this next figure, Data Chunk Copy C1 and C2 are written locally into Site B, which is
local to VM. This is followed by acknowledgments and then, of course, the other two chunks
are written over here. And these are all followed by acknowledgments that we'll go back and
say, OK, the write process is complete.

Because two of the copies are being written remotely, each write will cause the delay of the
intercluster link. So if that core intercluster link is worse than 10 gigs, and the 5 millisecond
round-trip latency is less-- or if it's more than the 5 millisecond round-trip latency, then you
will notice a drop in the performance. And, basically, the stretch cluster is the same as a
regular cluster as far as the process is concerned. It will distribute the write requests, or the
data, accordingly.
Now, let's take a look at the reading process. Under normal stretch cluster operations, the
controller VMs will read data for a given virtual machine from the copies in their local site. This
is because the stretch cluster will try to avoid reading data from across the inter-site link to av
oid the delay of the read process. Right? So it always reads from the primary copy of the data c
hunk.

Now in this figure, remember V1-- virtual machine 1-- is in Site A, so it's going to read from the
first chunk. So Site A, it reads the Virtual Machines B, and then, of course, Virtual Machine C is
going to read from the first chunk in its site. So that it's always going to be reading from the
primary copy of the data chunks in the local site.

Let's examine the handling of a capacity drive failure. Now in this scenario, one of the drives o
n the server in Site B has failed. So we got a little X down there, OK? So a Virtual Machine C in
Site B just happened to be reading from the C1 data chunk when the failure occurred. So now,
Virtual Machine C will just start reading from the second data chunk-- wrong place, there—the
second data chunk, which also happens to be in Site B.

Now, if both of those chunks become available, then the read would have to be done from the
server in remote Site A. So it would go over here and go to Data Chunk C3. And then, of course
, if that were to become unavailable, we got one more data chunk to save us. Right?

So once again, remember earlier, I said this was a replication factor of four. Well, basically, it's
just our RF2+2. Now, if a drive comes back online again within one minute, the data is going to
be re-synced. Now, if the drive has actually failed, well, then the cluster will have to self heal.

Now, let's examine the handling of a server failure. In this scenario, the first server in Site A has
failed, as you see. So the VMware High Availability mechanism will kick off the VMs on
another server in the same site. So as you can see, it said, all right, let's go grab another-- well,
we'll put it on a server in the same site. So we kind of put that over there on server three.

The Failure 1 server means that you will have lost one out of the four copies of any given
chunk of data in the cluster. But no big deal because I still have a three remaining copy. So,
really, when you look at RF factor of two, really it's 2 plus 2, which is 4. So I still have three
remaining copies left over.

Now, like with a regular cluster, a two-hour timeout will be in effect

where nothing will happen with the data. The cluster is unhealthy but it is still fully functional.
After the two-hour timeout, HyperFlex will self heal by recreating the missing copies of data.
So you'll notice that we're going to create-- we can't get to these anymore so it recreates those
chunks of data. And that was because they were lost when the first server inside A went down.
So, the system self healed itself by recreating those two chunks on the remaining two servers
in the local site.

Now, [EXHALES] it gets worse. But, hey, we can handle it. Right? What happens if you happen
to lose all three nodes in one site? Holy cow. The figure shows that all three servers in Site A
have failed-- no problem. And that's because the VMware High Availability mechanism kicks
off A and B virtual machines in site B. So, they were just basically going to move them over
here and we're going to say, OK, we'll kick off those two particular virtual machines in here.
And, of course, the data is already written over there.

So a failure can only be successful as long as you have enough resources to failover all of the
VMs. So, obviously, keep an eye on the resources and make sure you have enough. Now, for
the recovery operation to succeed after a site failure, the witness node-- way up here—has to
be online. Once again, the cluster is still online but it's going to be registered as unhealthy.

Now, to make things even worse, what happens in addition to losing all three in the first site,
what if I lose another node on the remaining site, as you see here? Well, guess what? The
cluster can still survive this failure scenario and still be online only if you have a minimum of
three nodes on that site. And also, you got to make sure you still have configured in—or
factored enough CPU and memory resources that you could still failover—the virtual machines
and you've got enough resources all together.

[SIGHS] Wow. Now, I think this is pretty much the last scenario. Next, what happens if a stretch
cluster is cut in half because, guess what, oh, boy, we lost our connectivity between Site
A and Site B? Since both sites are equal, the third party, which is that witness VM up here, has
to make the determination and decide how the cluster will proceed. And that's basically like
flipping a coin. One site's going to live while the other site is going to be suspended.

When Site A and B were split in half by that failure of the intersite link, this is what created the
split-brain condition. So what happened here is the single cluster is now divided into two
smaller clusters and each one of those smaller clusters thinks it's the only surviving cluster. I'm
the sole survivor. Each smaller cluster may try to simultaneously access the same data in their
own site and this is problematic because that can lead to corruption of the data. Now once
again, that's where the witness VM in HyperFlex the third party that decides which of the two
shot sites are going to live on. So let's take a closer look.
In this figure here, notice that the witness site said that, well, you know what? Sorry Site B, I've
decided that you're going to be suspended. And then, of course, you'll notice that it turns on
the Virtual Machine C, since that was over and Site B, it's going to enable that one in Site A so
that we can resume normal operations. From this point on, all reads and writes for Virtual
Machines A, B, and C will use the remaining two copies of chunks that are still available in Site
A. The cluster will remain unhealthy until Layer 2 connectivity between the sites is restored
and HyperFlex is done with what we call the self healing.

Now, finally, the witness is only needed if the Layer 2 connectivity between the sites fails. So
losing the witness by itself will actually have no effect on the cluster because the cluster's still
operational. However, what happens if the witness is unavailable and connectivity fails
between the sites? Here's where the issue begins because we're going to have a split-brain
scenario.

Well, split-brain scenarios are not allowed. So if that witness is not available, then the cluster
will move both sites into a shutdown state. So this will prevent data corruptions but now the
whole data cluster is down. So, boy, I'll tell you what. It's a pretty good idea that you should
deploy a witness in the High Availability manner. And that's because if you lose your witness
VM, for example you accidentally deleted it, then contact Cisco TAC, they will perform the data
recovery for you.

That brings us to the end of this video. Hope you learned a lot. Thanks for watching.

The distribution of data in a Stretched deployment, reading and writing process, and
resiliency mechanisms are very similar to a Standard deployment. This topic explains
how a Stretched deployment functions from the perspective of handling data, but it
focuses on how the mechanisms are different with a Stretched deployment.

The figure shows two sites, each with 2 Fabric Interconnects, 3 servers, and 2 switches.
Sites are connected with 10 Gbps (dark fiber or dense wavelength-division multiplexing
[DWDM]) and have connectivity to the third site.

Each site has a pair of Fabric Interconnects and 2 servers. As of Cisco HXDP 4.5(1a),
each site can have at minimum 2 and at maximum 16 converged nodes (if using SFF
drives; maximum 8 per site with LFF). Both sites need to have the same number of
nodes.

Each site has a pair of upstream switches to where Fabric Interconnects connect to.
VLANs (management, data, vMotion, and user VLANs) are configured on the link
between the two sites—"stretched" across sites.

Both sites have connectivity to the third site where the witness VM resides. You would
also use the third site to host the installer VM and vCenter.

Once you install Cisco HyperFlex Stretched deployment, it appears as one single
deployment (in vCenter, for example); therefore, Cisco HXDP is stretched across the
two sites.

Writing Under Normal Operations

For resiliency of data, Stretched deployment uses RF2+2 (two copies in each site).
The figure shows that Server 1 is running two VMs, that are marked A and B. Server 4
is running one VM, marked C. To simplify the diagram, the other servers are shown
without VMs.

Under normal operation, a write is committed in four copies: two are local and two are
remote. A Stretched deployment uses RF4, which is commonly referred to as RF2+2
because two of the copies are kept in each site.

As of HXDP 4.5(1a), 2+2 is the only option for configuration of the replication factor.

A write is finished when CVM receives an acknowledgment (ACK) from all four
servers for that given data chunk. Each write will cause a delay of the intersite link.

Writes would continue for all VMs until all the data is committed in four copies. A
Stretched deployment, like a Standard deployment, distributes data equally across the
server group. The data distribution concepts still apply: all the data is striped, written in
a distributed manner, and read in a distributed manner for optimal performance.
However, the read process is different.

Each write would cause the delay of the intersite link. So, if your intersite link is worse
than the supported 10-Gbps bandwidth and 5 ms RTT, you will notice a drop in
performance.
Reading Under Normal Operations
CVMs will read data for a given VM from data copies on the local site, because reading
across intersite link would not be optimal for performance.

The figure shows how VM-B reads data from one of the servers that is local to the site.
A VMDK belonging to a VM would be composed of many chunks of data, but this
example just shows one chunk of data per VM.

The figure also shows how VM-C reads data from one of the servers that is local to the
site. The Stretched deployment will try not to read data from a site that is not local to
the VM, to avoid adding the delay of intersite link to the read process. The VM would,
under normal conditions, read from the primary copy of the data chunk. The primary
copy resides on one of the servers in the site that is local to the VM.

Handling Capacity Drive Failure

When a disk fails, it reads the resume from the remaining site-local copy. If a drive
comes online within 1 minute, data is re-synced. If the drive has actually failed, the
storage system self-heals.

The figure shows that a drive has failed in one of the servers on Site B. The VM-C has
been reading data from the C1 chunk copy before the failure, but now it will read it
from the other copy that is available on Site B (C2).
If two failures occurred, where both C1 and C2 would be unavailable, reads for chunk C
would be done from servers in Site A (either C3 or C4).

Note
A failed cache or housekeeping drive would have the same respective consequences as
in a Standard deployment.

Handling Server Failures

When a server fails, VMware high availability moves the VMs onto another server. The
system is unhealthy but fully functional.

The figure shows the first server on Site A failing. The VMware high-availability
mechanism will ensure that the A and B VMs will continue running by turning them on
in another server in the deployment. Failure of one server means that you lost (at most)
one out of four copies of any given chunk of data in the deployment, and you have at
least three remaining. As with a Standard deployment, a 2-hour timeout will be in
effect, during which nothing will happen with the data.

After the 2-hour timeout, HyperFlex will self-heal by recreating missing copies of data.
The storage system is back to healthy.

The example continues by replacing data (A1, B2) that was lost after the first server in
Site A went down. The system self-healed by recreating A1 and B2 onto the remaining
servers in Site A. The system must not place more than one copy of a given data chunk
onto one server. Having more than one copy on a given server would not be resilient
because, if there is a failure of that server, you would instantly lose two copies.

Handling Multiple Server Failures

You can lose all nodes on one site. VMs on the failed site will be started by high
availability on the other site and the storage system will be online but unhealthy.

The figure shows how all servers in Site A failed. VMware high-availability mechanism
started the A and B VMs on the other side to ensure continuation of business. Of course,
a failover can be successful if you have enough resources to failover all VMs.

A functionally equivalent scenario to losing all nodes within a site would be losing both
Cisco Fabric Interconnects in one site.

For the recovery operation to succeed after a site failure, the witness node must be
online.

Handling Failure of Connectivity Between Sites

If a deployment is cut in half by a failure of a Layer 2 connection between sites, the
witness will decide which site lives and which is suspended.

Both sites of a Stretched deployment are equal, so a third entity (in this case, a witness
VM) must decide how the system will proceed.

Having one single deployment in two sites is great for resiliency, but if the connection
between them is broken, you could end up with a split-brain scenario:
 Split brain is when both sites think they are the only surviving site.
 A split-brain scenario would have led to both sites accessing the same set of data
locally, corrupting data, and making it very difficult or impossible to bring back the
two sites into one system.
 A tiebreaker or witness is needed to avoid split brain.

A split-brain scenario is a state in which a group of servers is divided into smaller

groups of an equal number of servers, and each small group thinks it is the only
surviving group. If you have one group that was split in two by an intersite connection
failure, each half may simultaneously access the same data, which can lead to
corruption of data. The witness VM, in the case of HyperFlex, is the third party that
decides which of the two separate, equal sites would live on.

High availability will kick off VMs from the suspended site, and the deployment will
live on one site until you resolve the failure. The split-brain scenario is then avoided.

The figure shows that the witness decided that Site B should be suspended. VMware
high availability turned on the VM-C to Site A for that VM to resume operation. From
this point, all reads and writes for the VMs A, B, and C will be to the two remaining
copies of the chunks available in Site A. The storage is unhealthy, and it will remain
that way until Layer 2 connectivity between the two sites is restored and Cisco
HyperFlex is done with self-healing.
For the recovery operation to succeed after a site failure, the witness node must be
online.

Handling Failure of Witness

A witness is only needed in a scenario where Layer 2 connectivity between the two sites
fails:

 Losing the witness VM by itself has no effect on the deployment.

 However, if the witness is not reachable, the deployment will move both sites into a
shutdown state if the sites cannot see each other, which will prevent data corruption
in any case.
 Recommended that you deploy witness as highly available because it is a mandatory
part of the Cisco HyperFlex Stretched deployment solution.

Note
If you lose your witness VM, for example, accidentally delete it, contact Cisco TAC and
they will perform the recovery process. Witness configuration is lost and needs to be re-
generated from the HyperFlex configuration.
10.5 Introducing Cisco HyperFlex Stretched Deployment

Datastores and VM Affinity

This video reviews datastores, site affinity, and the installation process. Site affinity is a
set of mechanisms that ensures optimal performance. With site affinity, the HyperFlex
installer automatically creates two hosts groups, one for each site, and two VMs groups,
one associated with each host group.

The figure here shows a stretch cluster across two sites. Server 1, 2, and 3 belong to Site
A, and Server 3, 4, and 5 belong to Site B. The servers on Site A belong to Host Group
A, and the servers on site B belong to Host Group B. When you create a datastore on a
stretch cluster, you must select site affinity. So when you deploy a new VM, you select
the datastore, you will deploy it, too.

In this example, the A and B virtual machines were deployed on Datastore A, so those
two VMs are pinned to the A Host Group. If Server A goes down, the virtual machine
will failover to either Server 2 or Server 3. Failover to the remote sites will only happen
if there are no local resources available in the site.

The installation process of Cisco HyperFlex stretch cluster is actually very similar to the
installation of a regular ESXi-based cluster. The differences stemmed from the fact that
this cluster is actually in two different sites. This figure here lists the high-level steps for
stretched ESXi-based HyperFlex. Be sure you meet the prerequisites, which means,
deploying HyperFlex Installer VM, vCenter, and a witness.

Rack fabric interconnects at the servers on both sites. Connect the servers to the fabric
interconnects, and connect the fabric interconnects to the upstream switches. Create
your UCS domains. Configure the ports on the fabric interconnects, for like server ports
and uplink ports. And then run the wizard to install HyperFlex. And you will need to
run this three times.
Lastly, perform the post-installation tasks. After deploying the witness, you can install
your stretch cluster. When creating a HyperFlex stretched cluster, enter the IP address
of the witness node. And it will automatically be used during the configuration of the
stretch cluster.

Finally, as with a regular cluster, prepare Cisco UCS cluster. Configure the server and
the uplink ports on the fabric interconnects, and configure the upstream switches, as I
mentioned before. With a stretch cluster, you will perform this twice, one for each site.
It's almost like configuring two separate clusters.

Also, do not forget that HyperFlex storage VLANs requires jumbo-sized MTU be
configured on the upstream switches. And don't forget you will need a third site for that
witness VM. That brings us to the end of this video. Thanks for watching.

Site affinity is a set of mechanisms that ensures optimal performance. Cisco HyperFlex
Stretched deployment achieves site affinity through vCenter rules and HyperFlex
reading and writing rules.

HyperFlex Installer automatically creates two host groups (one for each site) and two
VM groups (one associated with each host group).

 When you create a datastore, you need to choose site affinity.

 When you create a VM, choose a datastore, which links the VM to a VM group.

o The VM will run and fail over to the local site, as long as there are enough
resources.

 The purpose is to optimize the performance of reading, so it is always done from the
local site under normal operations, not incurring WAN latency.

o Writes are committed to both sites for resiliency.

The figure shows one deployment stretched across two sites. Servers 1, 2, and 3 belong
to Site A. Servers 4, 5, and 6 belong to Site B. The servers on Site A belong to the host
group A and servers on Site B belong to host group B. When you create a datastore on a
Stretched deployment, you must select site affinity. The example shows two datastores:
A and B. When you create a datastore, its affinity will be automatically pinned. In this
example, Datastore A is pinned to the A host group and Datastore B is pinned to the B
host group. When you deploy a new VM, you need to select the datastore that you will
deploy it to. In the example, VMs A and B were deployed on Datastore A, so those two
VMs are pinned to host group A. If Server A goes down, VM A will fail over to Server
2 or Server 3. Failover to the remote site happens only if there are no site-local
resources available.

Note
Strictly speaking, affinity does not work exactly like described. When you provision
VM A to run on Datastore A, which has affinity to Site A, the VM will eventually run
on one of the hosts in the A group. In other words, the vCenter might start the VM on a
host in host group A, or on one of the hosts in host group B. However, in the latter
example, DRS will then move it to the A host group. Similarly, if a host fails, the VMs
that were running on that host until that moment will be restarted by high availability
somewhere else. The VMs might get restarted locally or restarted on the other site, and
then DRS will move them to the local site.

10.6 Introducing Cisco HyperFlex Stretched Deployment

Installation Process
The installation process of a Cisco HyperFlex Stretched deployment is very similar to
the installation of a regular ESXi-based deployment. The differences stem from the fact
that this deployment is in two sites.

Note
Before you begin the installation of a Cisco HyperFlex Stretched deployment, consult
the latest version of Cisco HyperFlex Systems Stretched Cluster Guide, Release 4.5.
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/HyperFlex_Stretched_Cluster/4-5/b-hx-systems-stretched-cluster-
guide-4-5/m_guidelines_and_limitations.html)

These are high-level steps for installing a Cisco HyperFlex Stretched deployment:

1. Meet the prerequisites listed in the pre-installation checklist guide.

 o Make sure to check the release notes for caveats.

2. Deploy the HyperFlex installer VM, a vCenter (or use the existing one), and a
witness.
3. Rack Fabric Interconnects and servers on both sites.
4. Connect servers to Fabric Interconnects and Fabric Interconnects to upstream
switches on both sites.
5. Configure upstream switches and network between the two sites. Also, make sure
that you have connectivity to the witness from both sites.
6. Create Cisco UCS domains and configure ports on Fabric Interconnects on both
sites.
7. Run the wizard to install HyperFlex—you will need to run it 3 times:

o Configuration of Site A, configuration of Site B, and creation of the

Stretched deployment.

8. Perform post-installation tasks.

The first difference with a Stretched deployment is that you need to perform all ESXi
and Cisco UCS Manager-related installations twice (Fabric Interconnects, servers,
switches, and so on), once per site.

The second difference is that you need to deploy a witness VM that will be a tiebreaker
if the connectivity between the two sites goes down.

As with a Standard deployment, you can only add compute-only nodes to the
deployment after you have created the initial group.

Note
Make sure you check the Cisco HyperFlex HX Data Platform Release Notes for caveats
and enhancements: https://www.cisco.com/c/en/us/support/hyperconverged-systems/
hyperflex-hx-data-platform-software/products-release-notes-list.html.

Note
Since the installation of an ESXi Stretched deployment has many of the same steps as
the Standard deployment, you will focus on the differences in the installation procedure.

Deploy Witness VM
After successfully deploying the witness node, you can proceed to install your Stretched
HyperFlex deployment. When prompted, enter the IP address of this witness node on
the IP Address page when creating a HyperFlex Stretched deployment. The witness
node is automatically used during configuration of the Stretched deployment.
Before you begin, perform the following steps:

1. Choose an ESXi host that has a minimum of 4 vCPUs, 8-GB memory, and 40-GB
storage. Pre-allocate the entire space while deploying the witness node.
2. Ensure that the virtual network on this ESXi host is reachable from both Stretched
deployment sites.
3. Download the HyperFlex witness node to your desktop or the host that runs the
vSphere Web Client from https://software.cisco.com. As of HyperFlex v4.5(1a), the
current version for the witness is v1.1.1.
4. High availability is optional for the witness node, but it is recommended to deploy
the witness on a highly available infrastructure.

To deploy the witness on the ESXi server, perform the following steps:

1. Log in to vSphere Web Client. Choose the ESXi server where the witness should be
deployed. Right-click the ESXi host and choose Deploy OVF Template.
2. Browse and choose the .OVA file of the witness. Click Next.
3. Specify a unique name for the witness node in the Virtual Machine Name field.
Choose a location for the virtual machine from the drop-down list. Click Next.
4. From the Select a compute resource drop-down list, choose the ESXi host where
you want to deploy the witness node. Click Next.
5. In the Review details pane, verify the template details. Click Next.
6. In the Select Storage pane, select Thick Provisioned, and choose the datastore
where your VM will reside.
7. In the Select Networks pane, choose a Destination Network port group, where the
witness VM has to connect. Click Next.
8. On the Customize Template page, complete the fields that are relevant for your
configuration. If no values are entered, the VM uses DHCP server provided network
configuration parameters.

o Static IP Address: The IP address for this interface. Leave blank if DHCP is
desired
o Netmask: The netmask or prefix for this interface. Leave blank if DHCP is desired
o Default Gateway: The default gateway address for this VM. Leave blank if DHCP
is desired.
o DNS: The domain name servers for this VM (comma separated). Leave blank if
DHCP is desired.
o NTP: NTP servers for this VM (comma separated) to synchronize time. Leave
blank if DHCP is desired.

9. On the Ready to complete page, verify all the details entered. Click Finish.

Prepare Cisco UCS and Switching

As with the Standard deployment, you would prepare a Cisco UCS domain, configure
the server and uplink ports on the Fabric Interconnects, and configure the upstream
switches. With a Stretched deployment, you would do so twice: once for each site.

With a Stretched deployment, you also need to configure a Layer 2 connection between
the two sites. When you configure intersite connectivity, you must pass all the needed
VLANs for HyperFlex to function. So, make sure that the trunk is allowing
management, storage, and vMotion VLANs.

Note
Do not forget that the Cisco HyperFlex storage VLAN requires a jumbo-sized MTU
configured on upstream switches.

You need a third site for the witness VM. This third site is also the best choice to deploy
vCenter for HyperFlex management and HyperFlex Installer. These three VMs must
have connectivity to both Cisco UCS domains (for example, connectivity to Cisco UCS
Manager) and the management subnet: the IPs that the installer will assign to the CVMs
and ESXi servers. If you are using different subnets for the VMs on the third site, the
Cisco UCS domains, and HyperFlex management, do not forget that you need a Layer 3
device to route between them.
10.7 Introducing Cisco HyperFlex Stretched Deployment

Monitoring and Maintenance

Now let's review monitoring and maintenance. When you think about it, and I've said
this before, a Stretched HyperFlex ESXi-based cluster is just like a regular cluster. All
the monitoring and management tools that you would use with a normal cluster are
basically the same. Just like a regular cluster, you can browse to HyperFlex Connect
using this storage cluster management IP address. Plus, you can use vCenter, the stcli,
and the REST API. However, with the Stretched Cluster, these tools also give you
information about site affinity and witness availability.
This figure shows an example of using the stcli cluster info command on either of the
controller VMs. It will display information about the version of HyperFlex; state of the
witness VM; type, help, capacity, and operational state; uptime; the controller VM
configuration; and the cluster configuration. You can also expand both converged and
compute-only nodes. When you add a converged node, you got to ensure that the
configuration is symmetric across both of the sites.

Perform the expansion using the installer's expand cluster procedure. Here, you enter
the credentials for the UCS Manager, vCenter, and the hypervisor. Then, configure the
server ports and associate the HyperFlex servers. Lastly, configure the HyperFlex IP
address, configure the nodes, and start the cluster expansion process.
Finally, the procedure for performing a HyperFlex upgrade on Stretched Cluster—I
don't know how many times I've said this before—is almost like a regular cluster. You
perform upgrade from HX Connect. You can perform the upgrade either online or
offline. Be sure to check the requirements, though, for the manual cluster bootstrap and
auto bootstrap, and keep in mind, upgrading ESXi and UCS firmware upgrade, that's
not supported directly from HX Connect. It will eventually be supported in future
releases. That brings us to the end of this video. Thanks for watching.

A HyperFlex ESXi-based Stretched deployment functions the same as any HyperFlex

system.

 All the monitoring and management tools are there and ready to use.

o HyperFlex Connect, vCenter, stcli, and REST API

 The major difference is that these tools, now with Stretched deployment, give you
information about site affinity and witness availability.
As with a Standard deployment, you can log in to Cisco HyperFlex Connect by entering
the group management IP address in your browser.

The stcli cluster info command executed on either of the CVMs tells you
the following:

 Version of HXDP running

 State of the witness VM (1 = available)
 Type (hybrid, All-Flash), health, capacity, and operational state of the deployment
 Uptime of the deployment
 Configuration of individual CVM
 Configuration of the system

root@SpringpathControllerIW2ZESJLUH:~# hxcli cluster info

Cluster UUID :
3401107909468570171:6127214684284917993
Cluster State : ONLINE
Cluster Access Policy : Lenient
Space Status : NORMAL
Raw Capacity : 21.0 TB

You should use stcli and hxcli commands for information that you cannot find
from the GUI of HyperFlex Connect.

As with a Standard deployment, you can use other commands from the command
prompt of any CVM:

 stcli cluster summary shows usage of storage, including

replication factor, number of tolerable failures, and storage
savings.
 stcli vm <options> allows you to manage clones and
snapshots on the HyperFlex system.
 hxcli datastore <options> allows you to manage datastores
on the system (mount, unmount, update, list, create, and so
on).

Note
To use stcli and hxcli commands, use SSH to connect to one of the CVMs in the
deployment. When looking to use the stcli commands, help yourself with the Linux
standard -h or --help flag, or use the stcli reference guide for HyperFlex available
at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Dat
aPlatformSoftware/CLIGuide/4-5/b-hx-cli-reference-guide-4-5.pdf.

Expanding Stretched Deployments

Expansion of a Stretched deployment, whether with converged or compute-only nodes,
is not very different than expanding a Standard ESXi-based deployment. The same
prerequisites regarding converged nodes apply and the same compute-only nodes are
supported.

Expansion supported for both converged and compute-only nodes:

 When adding a converged node, ensure that the configuration is symmetric across
both sites. For example, add two nodes to each site.

o When adding compute-only nodes, make sure not to exceed the maximum
supported node count.

 You perform expansion, as with Standard deployment, using the installer’s Expand
Cluster procedure.

o Enter Cisco UCS Manager credentials for both sites, vCenter and the hypervisor
credentials.
o Configure the server ports and associate servers.
o Configure hypervisor, IP addresses, configure nodes, and start the deployment
expansion process.
For a detailed workflow for expanding a HyperFlex Stretched deployment, refer to
the Cisco HyperFlex Systems Stretched Cluster Guide at
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/HyperFlex_Stretched_Cluster/4-5/b-hx-systems-stretched-cluster-
guide-4-5/m_expanding_a_stretched_cluster_.html.

When adding a compute-only node to a Stretched deployment, you will need to install
ESXi on the node beforehand:

 You will need to modify the boot policy for compute-only nodes if you are not using
SD cards for boot (for example, using M.2 for boot).
 Start the expansion workflow and when prompted install ESXi using the ISO image.
Then return to the expansion workflow and complete the ESXi installation.

Upgrading Stretched Deployments

The full and current upgrade procedure is covered in the Cisco HyperFlex Systems
Upgrade Guide for VMware ESXi at
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/HyperFlex_upgrade_guide/4-5/b-HX-Upgrade-Guide-for-VMware-
ESXi-4-5.html.

The upgrade of a Stretched deployment is similar to a Standard deployment, except you

need to also upgrade the witness VM:

1. Perform the upgrade from HyperFlex Connect.

2. Upgrade online (rolling, one server at a time) or offline (all servers at once).
3. Manual system bootstrap is required for upgrades from a pre-3.5 releases. Auto
bootstrap is supported for upgrade from 3.5(1a) to later releases.
4. Make sure that the storage is healthy. Then copy configuration files from the
existing witness VM, shut down the old witness, stand-up the new version of the
witness VM, and copy the files over to the new VM.
5. Confirm that the upgrade is complete, the storage is healthy, and datastores are
mounted properly on ESXi hosts.

Note
HyperFlex Witness Node Version 1.1.1 is supported as of 4.5(1a) release.

Upgrading ESXi and Cisco UCS firmware upgrade is not available from HyperFlex
Connect as of HXDP 4.5(1a), but it will be made available in one of the future releases.

The following are detailed steps for upgrading the witness VM:

1. Log in to the witness VM using SSH and execute the following command to stop
the service exhibitor.

o root@WitnessVM:~# service exhibitor stop

2. Copy the exhibitor.properties file available in the
/usr/share/exhibitor/ path to a remote machine from where you can
retrieve the exhibitor.properties file.

o scp
root@<Witness-VM-IP>:/usr/share/exhibitor/exhibitor.prop
erties user@<Remote-Machine>:/directory/exhibitor.properties

3. Log out from the witness VM. Power off and rename the witness VM
to WitnessVM.old.

o Confirm that the IP address of the old witness VM is unreachable using

the ping command.

4. Deploy a new witness VM and configure the same IP address as the old witness
VM.

o If the IP address is not reachable, the witness OVA deployment may contain
stale entries in the /var/run/network directory. You must manually
remove these entries and reboot the VM to have the assigned IP address become
reachable on the network. To reboot the VM, open the VM console in
vCenter/vSphere and execute rm -rf /var/run/network/*, and
finally reboot.

5. Log in to the new witness VM using SSH and execute the following command to
stop the service exhibitor.

o root@WitnessVM:~# service exhibitor stop

6. Copy the exhibitor.properties file from the remote machine to

the /usr/share/exhibitor/ path of the new witness VM.

o scp /directory/exhibitor.properties root@<Witness-VM-

IP>: /usr/share/exhibitor/exhibitor.properties

7. Verify if the following symlinks are preserved in the new witness VM:

o root@Cisco-HX-Witness-Appliance:~# cd /etc/exhibitor/
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ls -al
total 8 drwxr-xr-x 2 root root 4096 Sep 11 13:00 .
drwxr-xr-x 88 root root 4096 Sep 11 12:55 .. lrwxrwxrwx
1 root root 41 Sep 11
13:00 exhibitor.properties lrwxrwxrwx 1 root root 37 Jul
24 16:49 log4j.properties

8. If the symlinks are not available, execute the following command:

o root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ln -s
/usr/share/exhibitor/exhibitor.properties
 exhibitor.properties
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ln -s
/usr/share/exhibitor/log4j.properties log4j.properties
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ls -al
total 8 drwxr-xr-x 2 root root 4096 Sep 11 13:00 .
drwxr-xr-x 88 root root 4096 Sep 11 12:55 .. lrwxrwxrwx
1 root root 41 Sep 11 13:00 exhibitor.properties ->
/usr/share/exhibitor/exhibitor.properties lrwxrwxrwx 1
root root 37 Jul 24 16:49 log4j.properties ->
/usr/share/exhibitor/log4j.properties

9. Start the service exhibitor by executing the following command:

o root@Cisco-HX-Witness-Appliance:~# service exhibitor

start exhibitor start/running, process <ID>

10.8 Introducing Cisco HyperFlex Stretched Deployment

Investigate Stretched Group

Investigate Stretched Group
In terms of managing and monitoring, the Cisco ESXi-based stretched group is not
much different than a standard group. In this lab exercise, you will focus on the
differences between a standard and a stretched group.

Investigate vCenter Configuration

Step 1

Log in to vCenter.

Answer
Using the Firefox browser, navigate to vCenter by typing https://vcenter.hx.lab into
the URL. On the vCenter greeting page, click the LAUNCH VSPHERE CLIENT
(HTML 5) button, and log in using student as the username and 1234QWer* as the
password.
Step 2

Discover the vCenter hierarchy of a stretched HyperFlex group.

Answer
Once in vCenter, expand the HyperFlex Lab data center and then the Stretched
Cluster.
There should be four different HyperFlex groups in the vCenter (edge, stretched, and
two standard groups). In this lab, you will only focus on the stretched group, which is an
ESXi-based HyperFlex group with four servers.

Regarding the vCenter hierarchy, note that nothing tells you that this group is stretched
—besides the name of the group that the person performing the installation decided on.
The stretched group is stretched from the perspective of HyperFlex storage, but it is just
one group from the perspective of the VMware system.

Investigate HyperFlex Connect Configuration

Step 3

Log in to HyperFlex Connect.

Answer
In your browser, open the URL https://hxconnect.stretched.hx.lab. Log in using
credentials student / 1234QWer*. (The same credentials that you used in vCenter.)

Note
The user that you are using in this lab has been created in vCenter and assigned a read-
only role. HyperFlex Connect allows you to log in using a vCenter user. As of HXDP
4.5(1a), HyperFlex Connect recognizes two roles from users that are defined in vCenter:
vCenter administrators are given the administrative role in HyperFlex Connect, while
other users have read-only access.
Step 4

Discover the HyperFlex Connect Dashboard.

Answer
The Dashboard pane opens immediately after you log in to HyperFlex Connect.

A difference in Dashboard (compared to the ESXi-based standard group) is that,

instead of having four nodes, you have two plus two nodes. Two nodes belong to Site-A
and two nodes belong to Site-B. Note that resiliency health reads that 2 Node failures
can be tolerated, which means that you can lose one entire site with a stretched group
and still be operational. With a standard 4-node group, you would be offline if two
nodes failed.

Step 5

Discover HyperFlex Connect System Information section.

Answer
Choose the System Information section under Manage in HyperFlex Connect.
Note that the witness is listed to have IP address of 10.111.111.111 and should
be Online. As of HXDP 4.5(1a), this flag is the only indicator in HyperFlex Connect
that the witness is reachable. If the witness goes offline, it does not degrade the health of
the group. In this lab environment, the witness is deployed on an ESXi host that is not
visible to you, but it is simply a VM that is reachable from the stretched group.

Observe that the Data Replication Factor is listed as 2 + 2.

Notice that each node is tagged with a site: two nodes with Site-A, and the other two
nodes with Site-B.

Step 6

Discover the HyperFlex Connect Datastores.

Answer
Choose the Datastores section under Manage in HyperFlex Connect.
Notice that there are two datastores: Stretched Datastore A and Stretched Datastore
B. Stretched Datastore A is affiliated with site named Site-A and Stretched Datastore
B is affiliated with site named Site-B.

Click Stretched Datastore A and scroll down to Host Mount Status. Notice that this
datastore, while it is affiliated with hosts on Site A, is mounted on all hosts in the group.
Similarly, Stretched Datastore B is mounted to all hosts in the group.

Investigate Both Cisco UCS Managers

Step 7

Log in to both Cisco UCS Managers.

Answer
Open the Firefox browser and open the following URLs in two tabs: https://ucsm-
a.hx.lab and https://ucsm-b.hx.lab. In both, choose Launch UCS Manager on the
Cisco UCS Manager introduction page to open the login prompt.

Log in using the student / 1234QWer credentials.

The credentials you are using provide you with read-only access to the system. You will
be able to view the configuration but will not be able to change it.

Step 8

Find servers marked as part of stretched cluster connected to Cisco UCS domains A and
B.

Answer
In both Cisco UCS Managers, navigate to Equipment > Servers. You should see two
servers connected to each Cisco UCS domain (Servers 6 and 7 in Cisco UCS domain A
and servers 1 and 2 in the Cisco UCS domain B). Each Cisco UCS domain is separately
configured and then linked automatically during the creation of a HyperFlex group.
Step 9

Investigate VLANs.

Answer
In both Cisco UCS Managers, navigate to LAN > LAN Cloud > VLANs.

Find VLAN 3192, which is storage data VLAN for the stretched group. This VLAN is
stretched between sites; in these labs it is just a single, dedicated link between upstream
switches.

Each HyperFlex group in a data center should use a unique storage VLAN. The
example stretched group uses VLAN 3192 (on both Cisco UCS domains), the standard
group on site A uses VLAN 3092, and the standard group on site B uses VLAN 3095.
Edge group also uses unique storage VLAN, but is connected directly to switches and
not to Fabric Interconnects so the edge storage VLAN is not visible in the Cisco UCS
Manager.

Step 10

Investigate Service Profiles.

Answer
In both Cisco UCS Managers, navigate to Servers > Service Profiles > root > Sub-
Organizations.
In this lab environment, each group was configured within its own suborganization.

You should notice that the stretched group has a unique suborganization on each Cisco
UCS domain. However, it is not necessary for the suborganization names of the
stretched group to match between Cisco UCS domains since it is a locally significant
name.

10.9 Introducing Cisco HyperFlex Stretched Deployment

Install and Manage Stretched HyperFlex

Group
Install and Manage Stretched HyperFlex Group
A stretched HyperFlex group is a HyperFlex installation that spans two campus-area
sites connected by a Layer 2 link. The installation is performed by configuring each of
the sites and then uniting them into one group.
The HyperFlex stretched group is installed through the same installer as a standard
group and requires a lot of the same information. However, it requires some unique
information and provides some additional functionalities.

In the lab, you will install a stretched HyperFlex group and then review the features
unique to the HyperFlex stretched groups. This guide assumes that you are already
familiar with a standard HyperFlex group installation and operation.

You will perform the installation of a stretched HyperFlex group in an emulated

environment and as such, there are some limitations. For example, you will not be
configuring upstream switches, you will not be configuring server and uplink ports on
Fabric Interconnects, you will not create two Cisco UCS groups, and you will not
deploy VMs (vCenter, witness, HyperFlex installer) that are needed to set up
HyperFlex.

Pre-Installation Tasks and Workflow Selection

A stretched HyperFlex group installation is performed with the same installer as a
standard Hyperflex group installation, except a different workflow is chosen.

First, you configure each of the Cisco UCS domains. Then you install the HyperFlex
stretched group onto these domains.

Since the stretched HyperFlex group installation combines two Cisco UCS domains,
installation is segmented into three parts:

1. Configure the Cisco UCS Manager and ESXi installation on the first site.
2. Configure the Cisco UCS Manager and ESXi installation on the second site.
3. Perform the final HyperFlex installation, merging the two sites into a single group
and register the group in vCenter.

The three parts of the installation each have a workflow in vCenter. The site
configuration workflow is repeated twice with different values.

The required information and IP addressing has similar limitations as a standard group
deployment. Some additional mistakes can happen if you are not careful, since you need
to provide correct information for each of the two sites.

Be aware of the following limitations when installing the stretched HyperFlex group:

1. MAC address pools on Site A and Site B should not overlap.

2. Cisco IMC IP address pools should not overlap on the two sites.
3. The data VLAN should match between the sites and the sites should be reachable
over the data VLAN.
4. The management VLAN does not need to match and can be routed, but the sites and
the witness must be reachable through the management VLAN.
A lot of information is provided to the installer several times as you are installing a
stretched HyperFlex group. Make sure that you provide the correct information each
time, or the installation will fail.

Note
Even though you do not go through the Preinstallation Checklist document in this lab
exercise, you need to fill it out before installing the stretched HyperFlex group so that
you have all the information available for the installation.

Step 1

Connect to the pod vCenter.

Answer

Type https://vcenter.hx.pod/ into your browser, choose the LAUNCH VSPHERE

CLIENT (HTML5) as the vSphere web interface, and log in using
credentials administrator@hx.pod / 1234QWer*.

You will notice that the vCenter is empty.

Open a new browser tab and enter https://installer.hx.pod/ in the URL field to open
the HyperFlex installer. Log in using credentials root / C1sco12345. Check the I accept
the terms and conditions check box and click Login to proceed.
On the Select a Workflow screen, click Create Cluster, and choose Stretch Cluster from the
drop-down menu.

Configure Site A
Step 2

Load the configuration file for Site A.

Answer
In the Credentials step, make sure that the Configure Site workflow is chosen.

Note
The stretched HyperFlex group installation workflow has two sub-workflows.
The Configure Site workflow is used to configure each of the two sites and must be
done first for each site. The Create Stretched Cluster sub-workflow is done last, after
the Configure Site workflow has already been performed for each of the stretched
HyperFlex group sites.

Click the Select a JSON File button on the right and choose the configuration
file Stretched Cluster Site A.json that is located in
the ../DCIHX/Configurations/ folder.
Click the Use Configuration button in the bottom right to apply the configuration.

Step 3

Configure the stretched HyperFlex group on Site A.

Answer
In the Credentials step, provide 1234QWer as the password for Cisco UCS Manager.
The Site Name parameter is specific to stretched HyperFlex groups and is not used in
other group types. The installer will use this information to associate servers with sites.

Click Continue.

In the Server Selection step, choose the servers with serials RK93 and RK94. Leave
the other one unselected, it will not be used for the stretched HyperFlex group
installation.

Three servers are available, the first two are pre-selected for you by the JSON script.
Leave them chosen, you will only install a 4-node stretched HyperFlex group. Each site
will have two nodes.

Note
When installing a stretched HyperFlex group site, you can only see the servers on that
Cisco UCS domain. They are listed in the Server Selection step. You will need to
connect to the Cisco UCS Manager of the other site to configure the other half of the
servers. The final step in a stretched HyperFlex group installation configures both sites.

In a real, non-emulated group, if the Hyperflex servers do not appear in this list, check
the Cisco UCS Manager and verify that they have been discovered. If the servers have
not been discovered, you probably misconfigured the Fabric Interconnect ports.

Click Continue to proceed.

The UCSM Configuration step is the most important one. The VLANs that you
provide here will be configured on your Cisco UCS domain of Site A. You need to
make sure that you provide the correct VLANs to each site.

The Mac Pool Prefix and Cisco IMC IP Blocks must be different when you configure
Site B or you may cause an addressing overlap making certain system unreachable
through the network.

Note
In this step, an actual installer would also require you to define the Cisco UCS firmware
version. In a virtual environment, this is not available because no actual hardware
exists. Make sure that the Cisco UCS firmware versions match between sites and are
compatible with the HyperFlex version that you are installing.

Click Continue.

In the Hypervisor Configuration step, you will define the IP addressing and hostnames
of the Site A servers.

The factory preconfigures a server with default ESXi credentials: root / Cisco123.
Cisco123 is also the default password when you are re-imaging servers.

Note
Do not change the hypervisor password that is provided.
Note
When performing an actual installation, take note of the server serial numbers to
provide the correct IP addresses once you will finally create the HyperFlex group. The
installer gathers servers from two sites and shuffles them, so it is hard to discern which
IP address belongs to which server. Ideally, you would address the servers in a logical
and memorable way.

Click Configure Site to start Site A configuration.

Step 4

Wait for the installer to configure Site A.

Answer
The configuration of a stretched HyperFlex group site should finish quickly because
there is no server firmware to install.

When the installation is complete, you can review the individual steps on the following
screen.

Note
In an actual installation, this step can take a very long time, depending on the firmware
version that you are running. When you install a new firmware version on your servers,
the site configuration process can take up to several hours.

Update server firmware ahead of time to avoid waiting in the installer. You can update
the server firmware through the Cisco UCS Manager on each of the sites.

Even if you choose to update the firmware through the installer, you will need to upload
the appropriate firmware in UCS Manager.

Configure Site B
In this activity, you will configure Site B, adapting some of the information used to
configure Site A.
Step 5

Load the configuration for Site B.

Answer
In the Cisco Hyperflex Installer, choose the gear icon in the top-right corner of the
screen, and choose Configure Site from the menu.

Now you are back at the Credentials step. Again, click the Select a JSON File button
on the right, and choose the Stretched Cluster Site B.json file that is in
the ./DCIHX/Configurations folder.
Click the Use Configuration button in the bottom right to apply the configuration.

Step 6

Configure the stretched HyperFlex group for Site B.

Answer
Once you have loaded the JSON file, use 1234QWer as the Cisco UCS Manager
password.
Note that in this step, you will provide the address of the Cisco UCS Manager of Site B
and name the site differently.

Click Continue.

In the Server Selection step, choose the first two servers that are available on the Cisco
UCS domain of Site B if they are not already selected.
These two servers will represent the other symmetrical half of the stretched HyperFlex
group.

Click Continue.

Leave the UCSM Configuration step unchanged because the information was provided
by the JSON file that you loaded. Click Continue.

In the Hypervisor Configuration step, provide the hypervisor management IP

addresses for Site B.
The IP addresses that you chose for the Site B hypervisors do not need to be in the same
subnet as the IP addresses on Site A, but all the interfaces must be reachable. In this lab
exercise, you are using the same Layer 2 connection on both sites and need to provide
IP addresses in the same subnetwork.

Click Configure Site to start the Site B configuration.

Step 7

Wait for the installer to configure Site B.

Answer
Installation of the second site should take the same time as the first site.

Install the Stretched HyperFlex Group

Now both sites are configured individually. The servers have been upgraded with the
latest firmware and the hypervisors have their management addresses configured. The
VLANs that you defined have also been added to each of the Cisco UCS domains.

In this activity, you will merge the sites into one cohesive group that stretches over two
locations.

Step 8

Load the configuration file for the final installation.

Answer
Click on the gear icon in the top-right corner of the HyperFlex installer, and
choose Create Stretch Cluster from the menu.
You have returned to the Credentials step of the installer, but this time, the Create
Stretch Cluster suboption of the workflow is selected.

Click the Select a JSON File button on the right side, and choose the Stretched
Cluster Final.json file that is in the ./DCIHX/Configurations folder.
Click Use Configuration to apply the configuration to the installer.

Step 9

Configure and install the stretched HyperFlex group.

Answer
Provide the passwords for both Cisco UCS Managers 1234QWer and
vCenter 1234QWer*. Do not change the default hypervisor credentials.
Click Continue.

In the Server Selection step, make sure that all four listed servers are selected. These
servers are spread over both sites and were configured in the individual site
configuration workflows.

Note
If you cannot see four servers in the list, click the Refresh button on the top right (a
limitation of virtual environment networking). You may need to click Refresh up to 10
times to see all four servers.
Note
The Site parameter indicates to which site a server belongs. You configured the Site
parameter in site configurations.

Click Continue.

Review the IP addresses in the IP Addresses step. Hypervisor addresses in this step
must be the identical to addresses that were configured when you configured individual
sites.
The Witness IP is a unique field for this step. Only stretched HyperFlex groups use the
witness IP.

Click Continue.

In the Cluster Configuration step, provide the CVM password C1sco12345 to the
installer in the Create Admin Password and Confirm Admin Password fields.
The replication factor 2+2 defines that two copies of information should always be kept
on each of the sites and is specific to a stretched HyperFlex group deployment. The data
is distributed automatically through the Layer 2 stretched HyperFlex group link between
the sites.

Note
As of Version 4.5.1a, the stretched group only supports the replication factor (RF) 2+2,
so you will not be able to change the RF during installation.

Jumbo frames are disabled in the virtual environment. In an actual installation, jumbo
frames are enabled by default.

Click Start to begin the installation.

Step 10

Wait for the installation to finish before proceeding to the next task.

Answer
The installer takes approximately 15 minutes to finish the installation, then it will
present the installation summary of the group.
Note
If the group installation summary page appears to be blank refresh the web page in your
browser until group details are visible.

Perform Post-Installation Tasks

As with other HyperFlex installations, installation of a stretched HyperFlex group does
not configure the vMotion network or enable DRS on the group. The configuration is
done after the installation by running the post_install command.

Step 11

Connect to the installer VM command line using SSH.

Answer
Click the blue and white icon in the top-left corner of the screen, and choose Terminal
Emulator from the menu. Type ssh -l root installer.hx.pod into the
terminal window and press Enter.

Use the password C1sco12345 to log in to the installer VM command line.

student@student-vm:~$ ssh -l root installer.hx.pod

root@installer.hx.pod's password:
Last login: Wed Mar 31 03:30:13 2021
root@HyperFlex-Installer-4.5.1a:~#
Step 12

Run the post_install script.

Answer
Execute the post-install script by typing post_install into the installer VM
command line. Follow the instructions in the student notes to configure the script. When
prompted, use 10.1.100.250 for the HyperFlex cluster IP
and administrator@hx.pod / 1234QWer* credentials for vCenter.

root@HyperFlex-Installer-4.5.1a:~# post_install

Select post_install workflow-

1. New/Existing Cluster
2. Expanded Cluster (for non-edge clusters)
3. Generate Certificate

Note: Workflow No.3 is mandatory to have unique SSL

certificate in the cluster.
By Generating this certificate, it will replace your
current certificate.
If you're performing cluster expansion, then this option is
not required.

Selection: 1
HX cluster to connect to: 10.1.100.250
Logging in to controller 10.1.100.250
Getting ESX hosts from HX cluster...
vCenter URL: 10.1.100.5
Enter vCenter username (user@domain): administrator@hx.pod
vCenter Password:
Found datacenter HyperFlex Datacenter
Found cluster Stretched Cluster

post_install to be run for the following hosts:

esx-a-1.hx.pod
esx-a-2.hx.pod
esx-b-1.hx.pod
esx-b-2.hx.pod

Enter vSphere license key? (y/n) n

Enable HA/DRS on cluster? (y/n) n

Disable SSH warning? (y/n) y

Add vmotion interfaces? (y/n) y

Netmask for vMotion: 255.255.255.0
VLAN ID: (0-4096) 3093
Do you wish to enter the range of vMotion IPs ?(y/n) y
 Please enter vMotion Ip range (format: IP_start-IP_end)
172.16.93.211-172.16.93.214
Vmotion ip 172.16.93.211 used for esx-a-1.hx.pod
Adding vmotion-3093 to esx-a-1.hx.pod
Adding vmkernel to esx-a-1.hx.pod
Vmotion ip 172.16.93.212 used for esx-a-2.hx.pod
Adding vmotion-3093 to esx-a-2.hx.pod
Adding vmkernel to esx-a-2.hx.pod
Vmotion ip 172.16.93.213 used for esx-b-1.hx.pod
Adding vmotion-3093 to esx-b-1.hx.pod
Adding vmkernel to esx-b-1.hx.pod
Vmotion ip 172.16.93.214 used for esx-b-2.hx.pod
Adding vmotion-3093 to esx-b-2.hx.pod
Adding vmkernel to esx-b-2.hx.pod

Add VM network VLANs? (y/n) n

Run health check? (y/n) n

root@HyperFlex-Installer-4.5.1a:~#

Explore Stretched HyperFlex Group Specifics

From a vCenter administration perspective, a stretched HyperFlex group effectively
works the same as a standard HyperFlex group. However, the VMs are distributed
across hosts differently.

From the storage administration perspective, the difference is more pronounced.

Step 13

Confirm the stretched HyperFlex group installation.

Answer
Open a new tab in your browser and go to the https://hxconnect-b.hx.pod/ URL. If you
see the security risk warning, click Advanced and then Accept the Risk and Continue.
Log in using credentials admin / C1sco12345.
Note
Since this is a virtualized environment, there is a chance that you will see a banner
with System information error when trying to fetch resources written in it. Click X to
dismiss the error.

Note
The hosts are now arranged in two groups for Site-A and Site-B, which you provided
during configuration.

Go to the System Information tab in HyperFlex Connect and verify the information on
the screen.

Note
The witness IP address at the top and site assignments for individual nodes include two
nodes on Site-A, two on Site-B. The replication factor of 2+2 is also stated here.
Step 14

Create two datastores with different site affinities.

Answer
Go to the Datastores tab in HyperFlex Connect and click on the Create
Datastore button in the top-left corner of the Datastores panel.

Create the first datastore named Datastore A that is 1 GB in size. Choose Site-A as the
site affinity option for this datastore and click Create Datastore.

Note
You cannot create data stores without a site affinity in stretched HyperFlex groups.

Once you have created Datastore A, click the Create Datastore button again and create
another 1-GB datastore that is named Datastore B assign it Site-B affinity.
You should now have two 1-GB datastores, each with a different site affinity.

Step 15

Investigate the datastores and their affinities in vCenter.

Answer
Open a new tab in your browser, and go to https://vcenter.hx.pod/. Click LAUNCH
VSPHERE CLIENT (HTML5) and log in using
credentials administrator@hx.pod / 1234QWer*.

In the vCenter, expand the HyperFlex Datacenter and Stretched Cluster entries in
the Navigator pane. Choose the host esx-a-1.hx.pod in the stretched cluster. Click
the Datastores tab on the right.
Even though the datastores have different affinities, they are mounted on all servers,
regardless of the site where they are physically located. The server is assigned to Site-A
affinity but has both datastores mounted.

Click the Stretched Cluster in the Navigator menu, and choose the Configure tab on
the right. In the middle menu, scroll down to Configuration subsection and choose
the VM/Host Groups option.

Click Site-B_HostGroup and note the servers in the group. Then click Site-
A_HostGroup and observe the nodes in this group too.
The hosts are split by the site that they are associated with. Affinity rules determine that
VMs deployed on the Site-A associated datastore will be run on the host that is
associated with Site-A if possible. Click the VM/Host Rules entry in the Configure tab
to view the affinity rules.

VMs on Site-B will run on the HyperFlex nodes that are associated with the Site-B
affinity rule if the site B is available, otherwise, VMs will be restarted on Site A
compute, but the data will remain on the Site-B datastore, which is why both datastores
are mounted on all servers.
11.1 Introducing Cisco HyperFlex Edge

Introduction
Distributed data centers are the future. In the past, enterprises had a data center in their
headquarters—and that was all. Now, many enterprises have dispersed their compute
and storage data centers between headquarters, remote offices, and the public cloud.

Cisco HyperFlex Edge is a deployment option for Cisco HyperFlex that is specifically
designed for installation in smaller environments. Cisco HyperFlex Edge deployment
has several cost-saving features that make it the most affordable and most flexible, such
as the absence of Cisco UCS Fabric Interconnect devices and lower hardware
requirements.

The motivation for offering HyperFlex Edge is to provide all the benefits of a
HyperFlex system (such as security, ease of installation, and inline data protection,
compression, and deduplication) in a small environment, such as remote office, branch
office, or other corporate edge environments.

Some features and options of a standard HyperFlex deployment are removed in

HyperFlex Edge; therefore, additional limitations and special situations are associated
exclusively with this deployment option. There are, however, additional features that are
unique to HyperFlex Edge, such as centralized many-to-one backups.

11.2 Introducing Cisco HyperFlex Edge

Cisco HyperFlex Edge System Overview

This video provides an overview of the Cisco HyperFlex Edge Cluster solution. The Cisco
HyperFlex Edge cluster is a three node Cisco HX220 cluster. Since fabric interconnects are not
included in the purchase requirements, this definitely reduces the cost of the solution.

Cisco HyperFlex has some limitations compared to the standard vSphere deployment. It is
fixed to three servers without an option to expand, and the HyperFlex Edge servers also canno
t be reused as converged nodes. Different HyperFlex Edge clusters can also be part of the same
vSphere domain.
Deployment in remote and branch offices is where the Cisco HyperFlex Edge really shines. It
can also be deployed in a single deployment in a small environment where the vCenter
installation is available locally. Cisco HyperFlex Edge is a viable solution for small companies,
for their smaller environments that are easier to deploy, and plus a lower licensing and
minimum hardware requirements provide additional savings when you purchase the HyperFlex
Edge cluster solution.

HyperFlex Edge has these features and limitations. First, replication factor 2 or RF2 is the only
resiliency option. The data is still distributed over the network in a shared data platform as
before. It includes snapshots and ready clones, and there are several deployment options, and
it uses the same management as the standard HyperFlex except for Cisco UCS Manager.
Cisco HyperFlex Edge as of version 3.5.1a is a fixed 3 server solution, which consists of three
Cisco HX 220 servers. Now, the servers can be equipped with either flash capacity drives or
spinning small form factor capacity drives. From a design standpoint, HyperFlex Edge servers
have a caching drive and an internal M.2 drive on the M5 servers.

And this will provide the cash and storage space for the hypervisor installation. Both of these
drives are also present in the standard HyperFlex nodes, along with the housekeeping solid
state drive. The HyperFlex Edge servers have these hardware options and limitations. Both, all
flash and hybrid options, the same capacity drive as the standard HyperFlex, only three edge
servers are supported in the cluster, licensing is required, but hey, it's 25% lower price than
normal.

The boot drive is M.2, and the cash and housekeeping are solid state drives. And lastly,
HyperFlec Edge clusters can be deployed, monitored, and managed from Insight. This figure
lists some of the requirements for HyperFlex Edge servers, which are lower than standard HF
servers. For example, only one CPU per server is required.
The number of CPU limits the number of PCIE lanes on a server. So as a result, some of the
PCIE slots may be disabled if you choose to just install one CPU in your HyperFlex Edge servers.
Depending on the networking options, Cisco, a HyperFlex is connected to either one or two up
stream switches either with the one or two links.

Instead of connecting to a fabric interconnect, remember, an end server must be connected

directly to the switch. Because there are no fabric interconnects, the hardware management is
done using this Cisco IMC. Jumbo frames are disabled by default on edge installations but can
be enabled. In this case, you need to enable jumbo frames on the upstream switch as well.

Even though the upstream switches do not have specific requirements aside from the VLAN
configuration support, they will need to handle all of the data in a VM traffic from the cluster
and Cisco HyperFlex. You can connect to the Cisco IMC HTTPS HDML interface through a per
server configured IP address. Now, the change is made to one server are not going to be
propagated to the other servers.
There are no service profiles, and the firmware updates must be done manually on a per
server basis. It will support hardware status overview, including called home functionality, and
lastly, it has a remote KVM interface. The virtual network setup in Cisco HyperFlex Edge is
nearly identical to the standard clustering. Since the Edge deployment has several possible
deployment options, the actual distribution of the traffic and the uplinks are defined according
to specific deployment choice.
And lastly, it has the same four v switches, HX in-band management, HX storage data,
vMotion, and VM network. Let's take a look at them in more detail. In HX band-- the HX in-
band management provides management access to the HyperFlex connect and the vCenter.
HX storage data is the data platform traffic between the controller VMs, and the ESXI hosts.
VMotion is the virtual machine migration and mobility network, VM network is the network for
the deployed guest virtual machines, and finally, the Cisco IMC port is configured separately
and is not part of the vSphere configuration.
That brings us to the end of this video. Thanks for watching.

Cisco HyperFlex exists in different deployment options: Standard, Stretched, and Edge.
The Cisco HyperFlex Edge satisfies two main desires of customers who are looking to
deploy data centers at the corporate Edge efficiency from a cost and performance
perspective and powerful remote management features.

The main differences of HyperFlex Edge deployments compared to HyperFlex Standard

are focused on the efficiency and agility:

 Systems with either 2, 3, or 4 nodes that connect directly to upstream switches.

 No Fabric Interconnects which brings down the price of the solution.
 25 percent lower cost of Edge Advantage license versus Standard Advantage.
 Deployment is intended for corporate edge and smaller environments.
 You can connect HyperFlex to switches with either 10/25-Gb or 1-Gb connections.
 Since version 4.5(1a), Intersight installation is the default deployment method.

Although deployment at scale in remote environments is where Cisco HyperFlex Edge

really shines, it can also be deployed as a single deployment in a smaller environment
where vCenter installation is available locally. Cisco HyperFlex Edge is very viable for
small companies that want an easy to deploy solution for their environment.

Note
Very soon, HyperFlex Edge deployments will have the ability to scale beyond 4 nodes.
This change will be retroactive for existing deployments after the appropriate
HyperFlex Data Platform (HXDP) upgrade.

You can deploy Edge from an on-prem installer or from Cisco Intersight.

 Cisco Intersight is recommended for deployment of Edge and Standard.

 2-node deployments cannot be deployed from the installer since it needs Intersight
witness.
 2-node deployments installed from Intersight cannot yet be expanded.

Using the Cisco Intersight platform for deployment of your HyperFlex Edge systems
allows you to easily perform installations from a central location, reuse configuration
policies for deployment at scale, deploy in parallel, and have excellent visibility into
your remote HyperFlex systems.

You can deploy HyperFlex Edge systems at remote locations and register them to a
vCenter in headquarters. If the WAN link becomes disconnected, Cisco HyperFlex will
function normally, but you will not be able to perform VM management since the
system will not be available in vCenter. It is still important to design your WAN
resiliently. VMware recommends that the round-trip time (traditionally measured by
the ping command) from vCenter to a managed object (HyperFlex Edge in this case)
should not exceed 100 ms for smooth operation.

Cisco HyperFlex Edge systems retain almost all features of HyperFlex Standard
systems:

 RF2 and RF3 are supported.

 Data is still distributed over the network in a shared data platform.
 HyperFlex native filesystem features are still available (native snapshots, clones).
 Same management options as in Standard, except hardware management is Cisco
IMC-based—there is no Cisco UCS Manager.
The figure shows a 3-node Edge system that is connected to a single switch (just one of
the many configuration options for a HyperFlex Edge).

HyperFlex Edge is much more flexible than Standard deployments. However, it puts the
responsibility of providing an adequate networking configuration on the customer. The
network switches used should have consistently low latency and at least 1-Gbps ports
capable of persistent throughput at line rate.

11.3 Introducing Cisco HyperFlex Edge

Edge Design Considerations

Just like a server in any other Cisco HyperFlex deployment type, Cisco HyperFlex Edge
M5 servers have an internal M.2 drive for boot, one housekeeping disk, one caching
disk, and capacity drives.

The hardware used in HyperFlex Edge is in line with the Standard HyperFlex
deployments, with additional scaling limitations and fewer hardware configuration
requirements.

Edge Hardware Options

HyperFlex Edge supports the Cisco UCS C-Series C240 and C220 server chassis, just
like Standard deployments. There is a lower minimum node count and lower per-chassis
specifications:
 All-Flash and hybrid nodes with 2, 3, or 4 servers per group.
 The same capacity drive options available as on a standard HyperFlex deployment.

o Exception is SED drives because data at rest encryption is not supported on

HyperFlex Edge.
o Supported number of drives is between 3 and 24 per server with no NVMe drive
option.

 1 CPU per server is required, 2 CPUs per server is more popular.

 Minimum 128 GB of RAM per server required, recommended minimum is 192 GB.
 Expansion with additional nodes is supported through Cisco Intersight.

When building a Bill of Materials (BOM) for your HyperFlex system, consider whether
a 1-CPU per server is a viable option for your design. The main advantage is that you
will need one less VMware ESXi license, which is per-CPU.

Note
Starting with VMware vSphere version 7.0 the maximum CPU core count for a single
ESXi per-CPU license is 32.

Note
The usable capacity of HyperFlex Edge deployments is precisely the same as with
Standard deployments with the same node count and configuration. Consult with Cisco
HyperFlex sizer for the latest sizing and design guidance.

Networking configuration of HyperFlex Edge is less restrictive than in Standard

deployments:

 Single or dual switch topologies supported.

 1-Gbps and 10/25-Gbps connections supported.

o 1-Gbps topology using two onboard LOM ports combined with the Intel i350 4-
port Ethernet NIC.
o 10/25-Gbps topology uses a dedicated VIC 1457 mLOM.
o Cisco highly recommends the 10/25-Gbps topology for higher performance and
future node expansion capabilities.
o You cannot change from 1-Gbps to 10/25-Gbps topology or vice versa after
installation.
The 1-Gbps topology is reserved for deployments that will never require node
expansion or instances where the ToR switch does not have 10-Gbps ports available.
This is a very restrictive configuration that does not take full advantage of the
hyperconvergence features.

10/25-Gbps topologies are available for all HyperFlex Edge system sizes including 2-
node. The cost of the 10/25 Gbps VIC 1457 is negligible compared to the benefits it
provides; however, at least a 10-Gbps upstream switch is required when using VIC-
based topologies.

Network options can be used with all chassis types that Edge supports, including the
unique HX240-SD short depth server.

Since Edge environments are often deployed with limited space and cooling
configuration, HyperFlex Edge supports a unique chassis option, that is specifically
designed for this specific use case.

Short depth chassis is designed for tight spaces and features a reduce depth chassis with
all ports and drives mounted on the front panel:

 Minimum 3 and maximum 4 Hybrid or All-Flash capacity mounted horizontally.

 System drive and caching drive mounted vertically with internal M.2 boot drive.
 IPMI, uplinks, and PSUs positioned in front to free up space in the back.
 Airflow and design are specifically adapted to environments with limited space.
Short depth server chassis supports fewer capacity drives, but features the same options
for RAM and CPU as with other HX-Series chassis.

Edge VM Encryption
Cisco HyperFlex Edge supports a VMcrypt VM-level encryption that allows customers
to encrypt their data if that is required.

VM-level encryption is enabled through third-party encryption client:

 Since HXDP 4.0(1a), VMware VMcrypt is supported.

 Cisco validated Gemalto SafeNet and Thales Vormetric as KMIP service providers
for VMware VMcrypt on Cisco HyperFlex Edge.
 You cannot expect any deduplication space savings, because encryption at this level
necessarily makes unique all data sent to the storage subsystem.
 Expect a hit on CPU utilization. The IOPS rate can be at 55–90 percent compared to
the same workload on unencrypted virtual machines at 100 percent benchmark load.

KMIP compliant key managers should work, but they require qualification. Refer to
the VMware Compatibility Guide for a list of supported key managers.
The Cisco HyperFlex environment provides storage for the guest virtual machines
deployed in ESXi using VLAN segmented networking. The virtual machines are
available for external resources, a feature typical of any elastic infrastructure
deployment. VMware VMcrypt works above the HXDP level. It operates at the
hypervisor (ESXi) layer and is implemented through vCenter and the remote KMS.

VMware vSphere VMcrypt native encryption has several requirements:

 At least vSphere Enterprise Plus.

 ESX and vCenter 6.5.
 Creation of an encryption storage policy in vCenter.
 Third-party remote KMS.

VMware VMcrypt has the following characteristics:

 No modification of the guest VM is required.

 The guest operating system used does not matter.
 The hardware version used does not matter.
 Any datastore will work.
 Virtual machine policy is set through vCenter.
 Virtual machine disk (VMDK) and virtual machine files are encrypted.
 Guests cannot access keys.
 vMotion encryption is supported.

VMware VMcrypt has some limitations:

 The default KMS must be acquired separately. You must implement the KMIP key
management protocol to make the key manager compatible with the vSphere of the
chosen version.
 SAN backup is not supported.
 Backup data is not backed up encrypted. The backup solution may provide its own
encryption mechanism. After a virtual machine is restored, it must be re-encrypted.
 vCenter cannot be encrypted.
 The following features are not supported:

o Suspend and resume.

o Encryption of a virtual machine with existing snapshots.
o Snapshots of encrypted virtual machines.
o Serial and parallel ports.
o vSphere replication.

Note
For more information on VMware VMcrypt on HyperFlex Edge read the Cisco
HyperFlex Edge Encryption Using VMware ESXi VMcrypt white paper:
https://www.cisco.com/c/dam/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/hyperflex-edge.pdf.
Witness Options for 2-Node HyperFlex Edge
Witness VM is an extra service that can act as an arbitrator in 2-node HyperFlex Edge
deployments. When the link between the two nodes fails, both nodes will attempt to
take over the workload from the other as primary, where only one should become the
primary. Witness chooses one of the sides as the primary and puts the secondary on
standby.

Witness is vital for 2-side system operation in failure situations to prevent storage
corruption and can be provided to HyperFlex Edge by Intersight cloud SaaS service.
Since HyperFlex version 4.5(1a), a self-deployed option is also available for Edge
deployments.

The default option for 2-node Edge deployments is the SaaS cloud witness:

 Witness is provided as a service in Intersight and is included in the price of

HyperFlex subscription.
 Hosted (invisible) witness is why 2-node HyperFlex cannot be installed using on-
prem installer.
 Both SaaS and appliance Intersight are supported.
 No need for third site or additional infrastructure.

With the invisible witness, there is no user interface to monitor, and no user-based
software updates. There is no setup, configuration, or backup required. There are also
no scale limitations and fault tolerance is built into the system.

The invisible witness is automatically updated, including the latest security patches. All
communications are encrypted using HTTPS over TLS 1.2. No specific firewall
configuration is required.
The witness service was built with efficiency in mind. No periodic heartbeats are sent
across the WAN link, and no deployment metadata or user data is transferred to the
Cisco Intersight platform.

HXDP version 4.5(1a) introduced the option to use an independent witness:

 The witness needs to be provided by the customer.

 Network Management Cards (NMCs) supported.
 The witness configuration is performed through the HX Connect Device Connector.
 Allows arbitration independent of Internet connectivity.
 Can only be implemented after installation.

Schneider Electric UPS Network management cards are currently supported as

independent witness systems. You can read more about available HyperFlex Edge
witness configurations in Cisco HyperFlex Invisible Cloud Witness Powered by the
Cisco Intersight Platform White Paper:
https://www.cisco.com/c/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/whitepaper-c11-741999.html

Physical Setup
Depending on the networking options chosen at the time of purchase, Cisco HyperFlex
is connected either to one or two upstream switches with either one or two links. Links
can be either 1 Gbps, 10 Gbps, or 25 Gbps. With 10/25-Gbps option relying on the same
VICs.

If upstream switches support 25-Gbps Ethernet communication, the 25-Gbps speed will
be used, otherwise the VIC will use the 10-Gbps speed automatically.
Instead of connecting to Fabric Interconnects, an Edge server must be connected to an
upstream switch directly. Jumbo frames are disabled by default on Edge installations
but can be enabled in the installer during installation. If you do enable jumbo frames
during installation, make sure you enable jumbo frames on the upstream switch or the
installation will fail.

Because of the different topology and integrated management capabilities, there are
some differences when deploying HyperFlex Edge compared to Standard deployments:

 There are no Fabric Interconnects, so hardware management is done through Cisco

IMC.
 To have access to the Cisco IMC, you need to connect IP management ports of
servers to access ports on a switch.
 Both on-premises and the Intersight installer use Cisco IMC to install HyperFlex, so
it needs access to Cisco IMC IP.
 Upstream switches need to support VLANs and must handle the traffic.
 Intersight-based installation requires the Cisco IMC to be registered to Intersight
before installation.

Even though upstream switches only require VLAN configuration support, they will
handle all the storage data and VM traffic of your HyperFlex deployment. For best
performance and support, Cisco recommends that upstream switches are either Cisco
Catalyst or Cisco Nexus Series Switches.

If you use a one-switch deployment, links are connected to one upstream switch. In the
example, integrated LOMs are used, and each provides 1-Gbps connectivity to the
upstream switch. You can also use shared LOM Cisco for IMC connectivity when using
Cisco VIC.
Note
Keep in mind that the maximum throughput of the system in the example is effectively
2-Gbps full-duplex per node. This includes data traffic. HyperFlex distributes data to
even out the utilization of links, but be careful during design to properly scope the
workload that such a design can support.

Hardware and Firmware Management

Cisco IMC is an integrated management controller present on each Cisco UCS server.
In a managed deployment, Cisco IMC becomes the configuration interface that
communicates with the Cisco UCS Manager and is not directly accessible. But when
servers are deployed in standalone mode, Cisco IMC becomes the actual on-premises
hardware and firmware management interface.

On HyperFlex M5 servers, the initial IPMI IP address of Cisco IMC is configured

through DHCP, or static IP configuration by connecting a monitor to your server and
providing an IP address after entering Cisco IMC configuration interface and after
pressing F8 on the POST screen. This needs to be performed manually for every server
before installation.

Note
The default Cisco IMC username and password credentials on Cisco UCS C-Series
servers are admin / password. The system will immediately prompt you for a new
password on the first entry in the Cisco IMC IPMI configuration.

In Edge deployments, Cisco IMC runs in standalone mode and provides these features:

 Management interface for HyperFlex installer to configure the servers

 Firmware management interface. Firmware must be updated manually.
 User hardware and firmware management HTML5 interface accessible through
HTTPS.
 Hardware status overview, including call home functionality.
 Remote KVM interface.
 Intersight device connector for Intersight management connectivity.
You can connect to Cisco IMC HTTPS HTML5 interface through a per-server
configured IP address. The changes made to one server are not propagated to other
servers like with Cisco UCS Manager-based configuration.

Cisco Intersight can manage entire Edge groups through an intuitive unified globally
accessible interface, enabling management of many sites and many deployments at the
same time. Intersight also allows full-stack upgrades of HyperFlex Edge systems
including the server firmware, which significantly simplifies firmware management.

The IP connectivity still needs to be configured on Cisco IMC before configuration

through Cisco Intersight. You can use DHCP for management interface
autoconfiguration to simplify Intersight registration, but static configurations are more
common. During HyperFlex installation, static addressing will be configured.

Note
When using DHCP, be very careful not to cause IP conflicts on the IPMI network, since
you may not have full control over which IP addresses will be assigned by the DHCP to
an individual node. You will also not have the ability to choose the sequence in which
Intersight will perform a static IPMI configuration, which can cause IP conflict if you
are assigning the same range of IP addresses in both cases.

The registration to Intersight is performed through the device connector available in the
Cisco IMC interface. The interface of the device connector on Cisco IMC is the same as
on Cisco UCS Manger:

 After IPMI configuration, the device connector is available in the IMC interface.
 After all servers of a HyperFlex group to be installed are registered to Intersight,
you can install HyperFlex on them.
 Standard Intersight connectivity requirements apply.
 HTTP proxy configuration is supported.
Virtual Networking
The virtual network setup in Cisco HyperFlex Edge is similar to Standard deployment
networking. Four networks are defined, but may be distributed differently between links
depending on the chosen topology.

Edge deployment has several possible deployment topologies that you can choose from
when designing Cisco HyperFlex Edge deployment. The most common and
recommended deployment option uses 2-switch 10/25-Gbps connectivity using Cisco
VIC 1457 configuration.

The topology choice also determines the cabling of your HyperFlex Edge deployment.
It is important how you distribute links between switches. However, it is not important
that you connect servers to the same port numbers on both switches in 2-switch
topologies. This is still advisable for administrative purposes.

With VIC-based deployments, the network configuration is the same as other

HyperFlex deployments.

Often, Edge has the same four vSwitches as other HyperFlex flavors:

 hx-inband-management: Management access to HX Connect and vCenter

 hx-storage-data: Data platform traffic between CVMs and ESXi hosts
 vmotion: VM migration and mobility network
 vm-network: Network for the deployed guest VM
 Cisco IMC port is configured separately and is not part of the vSphere configuration
11.4 Introducing Cisco HyperFlex Edge

Cisco HyperFlex Edge Network

Topologies
The Cisco HyperFlex Edge solution supports single- or dual-switch topologies using
either 1 Gbps or 10/25 Gbps links. While 3- and 4-node topologies share design
considerations, the 2-node topology has a few specifics to consider.

The topology that you will use will need to be determined when initially designing your
HyperFlex installation. The topology chosen will determine the network cards that you
will order with your HyperFlex hardware.

In all cases where 10-Gbit upstream switching is necessary, you should choose the VIC-
based deployment option. The benefits of VICs far outweigh the small additional cost of
purchasing the cards with your HyperFlex hardware.

Edge deployments have several uplink configurations:

 Back-to-back deployment (2-node Edge only)
 Single switch with access server uplinks
 Single switch with 1-G trunk server uplinks
 Dual switch with 1-G trunk server uplinks
 Single switch with 10/25-G trunk server uplinks and VIC
 Dual switch with 10/25-G trunk server uplinks and VIC

Note
Always consider the specifics of your workload and potential growth when considering
the network topology to use. Sometimes, it makes more sense to purchase a better
switching option at initial purchase to allow for better scaling. You cannot change from
a 1-Gbps to 10/25-Gbps deployment after installation without migrating data off the
HyperFlex system, changing networking, and re-installing.

3/4-Node with a Single 1-Gbps Switch Edge Deployment

In a single-switch deployment, the Cisco HyperFlex Edge system is connected to only
one upstream switch. The connection is done redundantly over two uplinks that must be
able to transport several VLANs at the same time. Therefore, the upstream switch must
be a managed switch with VLAN capability.

In this setup, there is no upstream switch redundancy. However, both server LOM ports
are connected to the upstream device, so the server is connected redundantly to the
upstream switch. The usual HyperFlex VLANs are recommended, but the minimum
configuration must separate the management network (which also carries vMotion and
VM network traffic) and the storage network.

The single 1-Gbps switch deployment also accommodates a unique feature of this setup
that allows the HyperFlex servers to be connected to upstream access ports instead of
regular trunk ports. Access ports can only carry a single VLAN by consolidating
management, vMotion, and VM network traffic into a single VLAN. Two access ports
can enable connectivity for the Edge deployment.

When connecting the HyperFlex servers to access ports, the VLAN for the traffic is
defined on the switch port—not in the port group on the vSwitch—so a managed
upstream switch is required to define two discrete VLANs for each uplink. One of the
uplinks is reserved for the storage data VLAN, which is used for the HyperFlex storage
traffic. This separation is required.

In single 1-Gbps switch deployments with 3- or 4-node deployment, consider the

following:

 Edge servers are connected to the upstream switch with two LOM 1-Gbps uplinks;
mLOM VIC is not supported.
 Two-VLAN configuration supported; the management VLAN also carries guest VM
and vMotion traffic.
 Four VLANs recommended, two required (management and data VLANs).
 Access and trunk uplink configuration are supported; access links support only 2
VLANs.
 o It is highly recommended that you use the trunk port configuration.

 PortFast should be configured on uplink ports if available.

PortFast causes a switch or trunk port to enter the spanning tree forwarding state
immediately, bypassing the listening and learning states. Use PortFast on the switch or
trunk ports that are connected to a single workstation, switch, or server, to allow those
devices to connect to the network immediately, instead of waiting for the port to
transition from the listening and learning states to the forwarding state.

The following table summarizes the difference between access and trunk port
deployment models. Use this table to determine which ports to use for your deployment.

Trunk Ports Access Ports

Requires more setup and definition of VLAN tags

Provides a simpler deployment process than
within CIMC, ESXi, and HyperFlex Data Platform
trunk ports.
Installer.

Allows logically separate management, vMotion, and Requires that management, vMotion, and
VM guest traffic on separate subnets. VM guest traffic share a single subnet.

Requires a managed switch to configure

Provides flexibility to bring in additional Layer 2 ports 1 and 2 on discrete VLANs; storage
networks to ESXi. traffic must use a dedicated VLAN, no
exceptions.

3/4-Nodes with Dual 1-Gbps Switch Edge Deployment

Two configurations of dual-switch deployment are supported. One is a more cost-
effective configuration, using 1-Gbps uplinks to 1-Gbps switches with an extra PCIe
NIC. The other is an mLOM configuration with two 10/25-Gbps uplinks. The 10/25-
Gbps option requires 10/25-Gbps switches for server uplinks and is, for that reason. The
less cost-effective option. Here, you will explore the 1-Gbps deployment.
Unlike a single switch deployment, a dual switch option does not support access ports,
so trunk ports are required on upstream switches. The configuration is also more
resilient because two upstream switches can carry the deployment traffic. There is no
longer a single point of failure.

The two-switch deployment requires an extra NIC installed on the server. This dual 1-
Gbps switch deployment requires four Ethernet ports to be connected: two to each
upstream switch.

In dual 1-Gbps switch deployments with 3- or 4-node deployment, consider the

following:

 Additional PCIe NIC required to provide two link connectivity to the switches.
 Only trunk ports supported. Uplinks are 1 Gbps.
 VLANs are distributed between two vSwitches in an active/standby configuration.
 The LOM ports connect to one upstream switch, the PCIe NIC ports to the other.
 PortFast should be configured on uplink ports if available.
 VICs are not supported on this configuration.

Each server must be cabled in the same way: the same port numbers should be
connected to the same upstream switch, to ensure maximum redundancy.

Four 1-GE ports are required per server:

 Port 1: Management (ESXi, HyperFlex controller, and Cisco IMC) and VM guest
traffic.
 Port 2: HyperFlex storage traffic (and vMotion standby).
 Port 3: VM guest traffic (and management standby).
 Port 4: vMotion traffic (and storage standby).
 Two ports using LOM and two ports from a PCIe add-in NIC:

o 1 LOM and 1 PCIe port serve management and VM guest traffic in a redundant
configuration.
o 1 LOM and 1 PCIe port serve storage data and vMotion traffic in a redundant
and load balanced configuration.
 The Intel i350 quad port NIC (UCSC-PCIE-IRJ45) must be installed for this
topology:

o The NIC may be selected at ordering time and shipped preinstalled from the
factory.
o The NIC may also be field-installed if ordered separately. Either Riser 1 or 2
may be used, although Riser 1 is recommended.

3/4-Node with Dual 10/25-Gbps Switch Edge Deployment

The Edge deployment model with two 10/25-Gbps uplinks from each server is also
available.

Note
Examples explain networking on a 3-node HyperFlex Edge deployment, but the
explanations also apply to larger 10/25-Gbps HyperFlex Edge deployments. Additional
considerations apply to the 2-node solution.

An mLOM VIC 1457 must be ordered with the servers in this deployment. Because
they are VIC, they support virtual interface configuration, and the actual vSwitch and
uplink configuration is the same as in a standard HyperFlex deployment.

Each physical link on each server must be connected in the same way to ensure
maximum redundancy.

In 10-Gbps switch deployments with 3- or 4-node deployment, consider the following:

 Only mLOM VIC ports are connected. The integrated LOM ports are not used.

o As of HXDP 4.5(1a), only 2x10/25-Gbps physical connectivity per server is

supported.

 mLOM VIC is required, which enables virtual interface creation.

 The network configuration is the same as on a standard HyperFlex deployment.

In a dual 10/25-Gbps switch configuration, the upstream switch ports must be trunk
ports.
As with standard ESXi HyperFlex deployments, additional third-party NIC cards may
be installed in the HyperFlex Edge nodes as needed. But, unlike standard deployment,
only a single VIC is supported per HyperFlex Edge node.

2-Node with 1-Gbps Switch Edge Deployment

The 1-Gbps switch topology provides a fully redundant design that protects against
switch (if using dual or stacked switches), link, and port failures. The 1-Gbps switch
may be one or two standalone switches or may be formed as a switch stack. Switches
must be managed switches.

In 2-node deployments with 1-Gbps switch, consider the following:

 Recommended to have four separate networks, but two are mandatory (management
and data).
 Servers connect back-to-back using two LOM-based 10-Gbps connections.
 Cisco IMC connectivity requires a dedicated management port. Shared LOM mode
is not supported.
Note
This topology is not slated to support future node expansion capability and should be
avoided if adding more HyperFlex Edge nodes may be required in the future.

The following requirements are common to both one- and two-switch 1-Gbps
topologies and must be met before starting deployment:

 Dedicated 1-Gbps Cisco IMC management port per server (required).

 Intel i350 Quad Port PCIe NIC installed in a PCIe slot in each server (required).

o Cisco VIC is not used in this topology, but you need a way to connect the two
nodes back-to-back.

 2 x 10-Gbps DirectConnect LAN-on-motherboard (LOM) connections (do not

consume switch ports).

o 2 x Category 6 straight through Ethernet cables for direct connect links

(customer supplied).
 6 x 1-GE top of rack (ToR) switch ports and 6x Category 6 Ethernet cables
(customer supplied).

Two vSwitches, each carrying different networks, are required:

 vswitch-hx-inband-mgmt: ESXi management (vmk0), HyperFlex storage

controller management network, and VM guest port groups.

o The entire vSwitch is set for active/standby across the two uplinks. All services
by default consume a single uplink port and failover when needed. Failover
order for guest VM port groups may be overridden as needed and to achieve
better load balancing.
 vswitch-hx-storage-data: ESXistorage interface (vmk1), vMotion interface
(vmk2), and HyperFlex storage controller data network.

o HyperFlex storage data network and vmk1 are set to the same active/standby
order. The vMotion VMkernel port is set to use the opposite order when
configured using the post_install script, ensuring full utilization of the direct
connect links.

Deploy with a single ToR (the following figure shows a layout):

 Connect the dedicated 1-Gbps Cisco IMC management port on each server (labeled
M on the back of the server) to the switch.
 Connect the LAN-on-motherboard (LOM) port 1 on one server to the LOM port 1
on the other server using a regular Ethernet cable.
 Connect LOM port 2 on one server to LOM port 2 on the second server.
 Connect any two of the four 1GE ports on the i350 NIC from each server to the
same ToR switch.
 Do not connect additional 1-Gbps ports before installation. After deployment, you
may use the additional two 1-Gbps ports for guest VM traffic.

Connect servers back-to-back with 10-Gbps connections and upstream with 3x 1-Gbps
(2x uplink, 1x OOB).

To deploy with dual ToR switches for extra redundancy:

 Connect the dedicated 1-Gbps Cisco IMC management port on each server (labeled
M on the back of the server) to one of the two switches.
 Connect the LAN-on-motherboard (LOM) port 1 on one server to the LOM port 1
on the other server using a regular Ethernet cable.
 Connect LOM port 2 on one server to LOM port 2 on the second server.
 Connect one of the four 1-Gbps ports on the i350 NIC from each server to the same
ToR switch. Use the same port number on each server to connect to the same
switch.
 o Failure to use the same port numbers will result in an extra hop for traffic
between servers and unnecessarily consume bandwidth between the two
switches.

 Connect a second 1-Gbps port on the i350 NIC from each server to the other ToR
switch. Use the same port number on each server to connect to the same switch.

Note
Do not connect additional 1-Gbps ports prior to installation. After deployment, you may
use the additional two 1-Gbps ports for guest VM traffic.

2-Node with 10/25-Gbps Switch Edge Deployment

The 10/25-Gbps switch topology provides a fully redundant design that protects against
switch (if using dual or stacked switches), link, and port failures. The 10/25-Gbps
switch may be one or two standalone switches or formed as a switch stack.

In 10/25-Gbps switch deployments with 2-node Edge, consider:

 You can use dedicated 1-Gbps Cisco IMC ports (recommended).

o Use of shared LOM is possible (but not recommended).

 Trunk ports are the only supported network port configuration.

Connect servers with 10/25-Gbps connections and 1-Gbps OOB upstream. No back-to-
back connection.
11.5 Introducing Cisco HyperFlex Edge

Prerequisites and Recommendations

In this video, let's review some prerequisites and recommendations. This figure lists the
prerequisites and recommendations for a single one-gig switch deployment. In a single switch
deployment, Cisco HyperFlex Edge cluster is
connected to only one upstream switch. The connection is done redundantly over two links,
which have to be able to transport several VLANs at the same time. Therefore, the upstream
switch bus must be a managed switch with VLAN capability. Both access and/or trunk links are
supported in this solution. PortFast should be configured on uplinks if they're available.
This particular option also allows the HyperFlex servers to be connected to upstream access
ports instead of regular trunk ports, as I mentioned before. Now, the access ports obviously
can only carry a single VLAN. So by consolidating the management, VMotion, and VM network
traffic into a single VLAN, two access ports can provide connectivity with no problem. One of
the uplinks is reserved for the storage data VLAN, which is used for the HyperFlex storage data.

There are two supported configurations for a dual switch deployment. One is the more cost-
effective configuration, by using one-gig uplinks to one-gig switches, using an extra PCI-E NIC.
The other is to use an alarm mLOM configuration with two 10-gig uplinks. The 10-gig option
requires a 10-gig switch for-- obviously—for the server uplinks and that's the reason for the
less cost-effective option.
So let's start off by examining, first, the one-gig deployment. Unlike a single switch
deployment, a dual one-gig gig switch option does not support access trunks, so the trunk
ports are absolutely required. The configuration is also more resilient, as there are two
upstream switches, which can carry the cluster traffic. Therefore, there is no longer a single
point of failure. The two-switch deployment also requires an extra NIC be installed in the
server. This dual one-gig switch deployment requires four ethernet ports be configured, two
of them for each of the upstream switches.

The Edge deployment with two 10-gig uplinks is also available. However, it requires
more powerful upstream switches. They are often not available in remote or branch offices,
where the Edge deployment is typically used. An mLOM VIC 1387 has to be ordered with the
servers when you decide to use this deployment. Because they are VICs, they support the
virtual interface configuration. Plus, the actual vSwitch and the uplink configuration, it's the
same as with a standard HyperFlex deployment.
In a dual 10-gig switch configuration, the upstream ports must be trunk ports. A 10-gig
configuration requires an extra script to be run on the HyperFlex installer before this actual
installation. This is to properly configure the infrastructure. The procedure is described in more
detail in the Edge Field Deployment Guide and it's also in the HyperFlex Edge Install Guide, so p
lease refer to them.

Networking and Edge deployments follows the general Cisco HyperFlex configuration.
However, since the Edge deployment relies on these upstream switches, it's a good practice to
configure the server uplinks with static speed. However, this disables the link speed auto
negotiation. If there is an issue with the link, the speed is not going to downgrade to fast
ethernet speeds, and this would definitely cause some communication problems in HyperFlex.
Also, the use of VLAN 1 is not recommended as it's the default VLAN and it could cause issues.
The Fabric Interconnect firmware, obviously, is not an issue in Edge deployments—we don't
need it. [LAUGHS] However, the individual server hardware should still conform to the
minimum firmware requirements for HyperFlex Edge installations. The servers can be updated
through the Cisco IMC controller, installed on each of the individual HyperFlex server. Be sure
to check the software compatibility information, and it's available on the Edge pre-installation
checklist. This list, here, this lists basically the minimum and recommended versions of
firmware for the Edge deployment.

Finally, there is no Cisco UCS Manager available to update the servers. The .bin firmware files,
available on the HyperFlex download site, they can't be used to update the servers. Instead, a
Cisco UCS C-Series server .ISO file is going to be used to update the servers. The server
firmware version can be verified by logging into the Cisco IMC interface on each individual
server. That brings us to the end of the video. Once again, thanks for watching.
Common Networking Considerations
Networking in Edge deployments follows the general Cisco HyperFlex configuration.
Make sure you fill out the Preinstallation Checklist document and follow the same
design considerations from Standard HyperFlex deployments except for Cisco UCS
Manager considerations.

Edge deployments rely on the upstream switches, so it is good practice to configure the
server uplinks with static speed, which disables link speed autonegotiation. If there is an
issue with the link, the speed will not downgrade to Fast Ethernet speeds, which could
cause communication problems in HyperFlex.

When you consolidate VLANs in a single switch deployment, the VM network can
cause a lot of traffic, which you should consider when you are configuring the upstream
network.

In Edge deployments, keep in mind:

 The Management VLAN will have to be accessible from the HyperFlex installer (if
using the on-prem installer), vCenter, and NTP.
 DNS and NTP should exist outside the HyperFlex deployment using those services.
 On 1-Gbps topologies set port speed to 1 Gbps to avoid autonegotiation issues.
 Very similar port requirements to standard vSphere deployment.
 VLAN 1 and default VLAN should not be used if possible.
 Cisco IMC can be used in in-band (LOM) and out-of-band mode. Out-of-band is
recommended.
 Nested vCenter is supported, but not recommended.

The service requirements of the management VLAN are the same as in standard
vSphere deployment. Except for Cisco UCS Manager (which is not present in Edge
deployments), all other services are present and have the same requirements.

You need to provide access to vCenter installation, HyperFlex installer (on-prem), all
the virtual ESXi hosts, CVMs in the management VLAN, which can also include Cisco
IMC in LOM mode and Cisco Intersight.

Note
In shared LOM extended mode (EXT), single wire management is used and Cisco IMC
traffic is multiplexed onto the 10-Gbps VIC connections. When operating in this mode,
multiple streams of traffic are shared on the same physical link and uninterrupted
reachability is not guaranteed. This deployment option is not recommended. In Fabric
Interconnect-based environments, built-in QoS ensures uninterrupted access to Cisco
IMC and server management when using single wire management. In Hyperflex Edge
environments, QoS is not enforced and, therefore, the use of a dedicated management
port is recommended.

You can register your branch offices with vCenter servers that are local to individual
branch offices. Local vCenters can run on an external server or be nested on HyperFlex.
The latter option is, as with standard HyperFlex, not recommended because it only
allows you to perform online upgrades. Having all your branch systems registered to a
centralized vCenter at your headquarters is fully supported—but for the best experience,
make sure that the RTT between a given site and the vCenter is below 100 ms and
bandwidth is above 1.5 Mbps.

Software and Firmware Compatibility

While Fabric Interconnect firmware is not an issue in Edge deployments, the individual
server hardware should still conform to the minimum firmware requirements for
HyperFlex Edge installation.

The servers can be updated through the Cisco IMC controller that is installed on each
individual HyperFlex server:

The software compatibility information is available on the Edge preinstallation

checklist:

 Minimum and recommended versions of firmware.

 Supported vSphere versions.
 The servers can be updated through the Cisco IMC controller installed on each
individual HyperFlex server.
 Intersight also provides the capability to upgrade the server firmware from a central
location.

If Intersight is not available, server firmware version can be verified by logging into the
Cisco IMC interface on an individual server and updated by booting into an .iso image
obtained from the software.cisco.com.
Note
For more details on HyperFlex Edge prerequisites refer to Cisco HyperFlex Edge
Deployment Guide:
https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataP
latformSoftware/Edge_Deployment_Guide/4-5/b-hx-edge-deployment-guide-4-5/m-
installation.html.

11.6 Introducing Cisco HyperFlex Edge

Installation Process
Now let's review the HyperFlex Edge installation process. This video is part 1 of 2.

The installation process for a Cisco HyperFlex Edge service is performed from the same
installer image used for a standard cluster deployment. The installation follows a very similar
procedure. But it has different networking and hardware management.
For example, there is no Cisco UCS Manager information necessary. Therefore, the installation
workflow is going to be slightly different. Since you only provide the Cisco IMC password once,
it should be used on all three nodes.

The figure illustrates the rack and stack for a single switch, one gig deployment. Cisco HyperFle
x Edge servers are based on the Cisco UCSC series, the same as all of the HyperFlex servers. But
they're available for order without the mLOM cards.

So to connect the Edge servers, you'll be using cabling schemes appropriate for your
deployment option-- single-switch with access or trunk ports, dual switch, and with or without
mLOM. The single-switch deployments are the most common way to connect Edge
deployments. And this is because they require only one switch and do not require an mLOM
card, reducing the overall cost of the solution.
This next figure illustrates a dual 1-gig switch deployment. Dual switch deployments,
remember, require next an extra mLOM NIC and distributed-- and it has to distribute VLANs
across the connections. These switches themselves can be the same as in a single-switch
deployment.

10-gig deployments use a Cisco mLOM VIC on the server side, the same as standard HyperFlex
deployments. The cabling is similar to the standard deployment, except the servers are
connected to the top of rack switches without the fabric interconnects.

This figure illustrates some installation caveats. First, you attached the keyboard and monitor
the server. Then you reboot the server. And when that BIOS option screen appears, you press
F8 to configure the Cisco IMC.

The IMC will prompt you to change the login password. The default password for changing
admin is password. You need to enter it to change the admin password. It's going to say,
what's the current password password? Then you change it.
Cisco IMC has several connectivity options available. By default, the management IP port is
used, which obtains the IP address from the local DHCP server. The actual Cisco IMC
configuration depends on the deployment model.

A single 1-gig with access ports use out-of-band Cisco IMC port without a VLAN. Single 1-gig
with trunk ports-- the same as access, or shared LOM using a VLAN. Dual 1-gig switch—the
same as a single, but with trunk ports with active standby NIC redundancy. And finally, dual
with 10-gig use out-of-band Cisco IMC port, or they use a shared mLOM and a VLAN.

This figure lists the configuration requirements for the top-of-row switch. These requirements
include configuring the management VLAN and requirements for both trunk ports and to
access link deployments. Now, keep in mind, the network in the Cisco HyperFlex Edge
deployments is not configured automatically. This is because HyperFlex Edge configurations
support third-party switches to provide connectivity to the servers. The ports on the switches
need to be configured manually, dependent on the chosen Edge type.
Finally, this figure list the steps to update the server firmware. After you've configured the
network, connect to the Cisco IMC GUI interface through the management VLAN. This is where
you access the KVM console and you map the firmware IOS image as a virtual CD-DVD drive.
Then when you reboot the machine, you override the boot device and point it to the virtual CD
and mount the firmware image for installation. And then lastly, just follow the update process.

This brings us to the end of this video. Once again, thanks for watching.
Let's continue reviewing with part two of the HyperFlex Edge installation process. Once you
have the networking set up and the server firmware updated, now it's time to deploy Cisco
HyperFlex Edge. You will need to deploy the HyperFlex installer in a vSphere environment
using the .ova image downloaded from software.cisco.com. Here, this is the same installer
image for deploying a standard cluster. However, for the HyperFlex Edge deployment, you
choose Edge cluster from the dropdown list.

When you deploy the HyperFlex installer ova file, you will choose a password for the
administrative account. This is used for logging into the installer. Be aware that since version
3.5, the installer VM does not have default credentials defined like the previous versions.
However, you must still configure the password when deploying the ova image.

In the second step, you provide the IP addresses of different HyperFlex components.
Remember, the Cisco IMC IP address has to be pre-configured because it's not assigned by the
installer, and this needs to be done manually before the installation starts.
Next, you configure the cluster-specific information. This includes the cluster name, which
defines how the cluster will be called in the HyperFlex-specific interfaces, the replication
factor, controller VM admin password, vCenter configuration, as well as the system services.
DNS is not mandatory for HyperFlex Edge but Network Time Protocol must be provided since it
is vital for operations.

When entering the VLAN configuration, you only provide the data and management VLANs.
Jumbo frames are disabled by default in HyperFlex Edge installations, which is different from
all of the other HyperFlex flavors. If you're not installing HyperFlex on a refresh ESXi
installation, you can choose to clean up the disks during the actual HyperFlex installation. It's a
good practice to install HyperFlex on fresh installations to avoid any potential issues.
Lastly, after you're satisfied with the information you have provided, you can initiate the
HyperFlex Edge installation by clicking Start. The json configuration file is very useful if there's
any mistakes in the configuration. This is cool because you can restart the installation, reload
the .json configuration, and then modify whatever you messed up. This makes repeating the
install process simpler. However, keep in mind, if you wish to change any of the options during
the install, you can always go back to the previous steps.

During installations, detailed information is provided on the installation procedure. If errors

occur, you can use this information to start resolving the issue. This information includes the
Config installer, which applies the configuration; the Deploy Validation that verifies the
configuration; Deploy, which performs a configuration; Create Validation, which confirms
connectivity; and the Cluster Creation, which actually creates the cluster.
After the installation is finished, an overview of the installation information is presented. You
can access HyperFlex Connect directly from this screen or you can connect later to the IP
address of HyperFlex Connect to confirm that the successful installation occurred. You can't
really tell the difference between the Edge and standard HyperFlex connect GUIs. I mean, they
literally are almost identical. This particular figure shows you the HyperFlex Edge Connect GUI.
After you connect, this is where you verify the cluster is healthy, as well as create a data store
and then verify that it's mounted on vSphere.

Finally, the post-installation script allows you to further configure your Edge installation the
same as you would with a standard installation. You can perform post-installation tasks by
using SSH to connect to the installer VM and execute the command post_install. This is where
you enter the vSphere license key, enable high availability or distributed Resource Scheduler,
disable SSH warnings in vCenter, configure VMotion parameters, and add VM network VLANs.
That brings us to the end of this video. Thanks for watching.

You can install HyperFlex Edge in the same way as Standard HyperFlex deployments
through the on-premises installer or through Cisco Intersight. Because of the globally
distributed nature of HyperFlex Edge deployments, it is very beneficial to install
HyperFlex Edge through Intersight to gain the ability to install, maintain, and monitor
HyperFlex deployments.

Note
Intersight is a cloud platform that is continually being updated, and changes can happen
very rapidly with new features being introduced all the time. Always verify the current
set of features on https://help.intersight.com/help/saas.
You can choose between the following two installer options depending on your
requirements:

 HyperFlex On-Premise OVA Installer: Use this option for Edge deployments of
3- or 4-node deployments when connectivity to external networks is not possible.
 Intersight Installer: Use this option to deploy HyperFlex Edge from the cloud
(recommended).

Note
Installation of a HyperFlex Edge deployment using the on-prem installer is very similar
to installation of a standard deployment. This topic covers only installation of
HyperFlex Edge using Cisco Intersight platform to showcase the specifics of this type
of installation and avoid overlap.

Installation Overview
Summary of steps to install HyperFlex Edge from Intersight:

1. Check prerequisites.
2. Ship servers to site and have them racked and connected to the upstream switch by
the local team.
3. Configure upstream switches and Cisco IMCs.
4. Deploy vCenter (or use an existing one).
5. Register individual servers to Cisco Intersight.
6. Install HyperFlex remotely from Intersight.
7. Perform post-installation tasks.
8. Verify Installation.
Note
Intersight is the preferred option for Edge installations and should be used wherever
possible. The on-premises installer is mainly used for air gapped environments that do
not have cloud connectivity.

Configure Upstream Switch

The network in Cisco HyperFlex Edge deployments is not configured automatically,
because HyperFlex Edge configurations support third-party switches to provide
connectivity to the servers. The ports on the switches must be configured manually,
according to your type of Edge deployment.

The upstream switch must be configured:

 Management VLAN must provide connectivity between Cisco IMCs, CVMs,

ESXis, vCenter, DNS, and NTP, usually through the uplink.
 On trunk port configurations, the data, management, VM network, and vMotion
VLANs should be allowed on ports that lead to servers.
 If using access VLAN configuration, only 2 VLANs can be configured; so
management, VM, and vMotion traffic are on the same VLAN.
Note
The example image is from a Cisco Catalyst switch. Use the appropriate versions of
these commands for your hardware. The example configuration is for 1-Gbps single-
switch trunk port deployment and dedicated Cisco IMC ports. It is a 2-node deployment
configuration with one cable running from each server's Cisco IMC port to a dedicated
access port on the upstream switch. Each node also has two 1-Gbps trunk connections to
the same upstream switch.

Configure Cisco IMC

Cisco IMC has several connectivity options available. By default, the management IP
port is used, which gets an IP address from the local DHCP server.

You can configure Cisco IMC to work in-band, using the LOM interfaces. VLAN
configuration of Cisco IMC is required to match the management VLAN configuration.
 In an access-port ToR switch configuration, you do not need to configure the VLAN,
because it will automatically be configured. In this case, use the single-port
configuration on Cisco IMC, because the other port will be used for the data traffic and
will not be available for Cisco IMC failover.

If you are not using DHCP to configure IP addresses on Cisco IMCs, you need to
connect to the management console port, or connect the keyboard and monitor to the
server, and configure Cisco IMC connectivity from there.

To configure Cisco IMC:

1. If Cisco IMCs connect to a management port and you have DHCP configured, IP
addresses will be assigned automatically.
2. If you are not using DHCP, configure Cisco IMC addresses manually on each
server:

1. Attach a keyboard and monitor to the server.

2. Reboot the server and press F8 on the BIOS options screen to enter the Cisco IMC
configuration interface.
3. On first access, Cisco IMC will prompt you to change the login password.
4. Use password password to change the admin password.

Cisco IMC presents a text-based screen with several configuration options.

The Cisco IMC configuration depends on the deployment model:

 Single 1 Gbps with access ports: Use out-of-band Cisco IMC port, configure Cisco
IMC without a VLAN.
 Single 1 Gbps with trunk ports: The same as access, or shared LOM configuration
using an appropriate VLAN.
 Dual 1-Gbps switch: The same as single with trunk ports, use active-standby NIC
redundancy.
 Dual with 10 Gbps: Use out-of-band Cisco IMC port or use shared mLOM
configuration with an appropriate VLAN.
After you configure the Cisco IMC interface, press the F10 key to save the settings.
Use F5 to refresh the screen. The Cisco IMC IP address should be reachable in the
chosen VLAN before you exit the configuration screen, if the ToR switches have
already been configured with the appropriate VLAN configuration.

Install Your Cisco HyperFlex Remotely from Cisco Intersight

Installation from Cisco Intersight uses the device connector on the Cisco management
systems. With Standard deployments this is Cisco UCS Manager, on Edge the device
connector is provided by Cisco IMC on each of the servers.

Because there is no central on-premises management in HyperFlex Edge deployments,

each server needs to be associated to Intersight individually before it can be used in
HXDP installation process.

Edge systems are claimed to Intersight through Cisco IMC:

 Access Cisco IMC on each server.

 In Cisco IMC, navigate to Admin > Device Connector and locate your Device
ID and Claim Code.
 Copy the Device ID and Claim Code to the Intersight Claim Target interface.
 In the preceding example, two nodes are claimed into Intersight. The examples show the
installation of a 2-node HyperFlex Edge system.

If direct HTTPS connectivity is not allowed between Cisco IMC and Intersight, you can
configure proxy connectivity in each Cisco IMC device connector.

To install your HyperFlex system:

1. Navigate to Configure > Profiles > Create HyperFlex Cluster Profile

2. In General step, you provide the base HyperFlex system information and HXDP
version for the profile.
1. Enter a description and add tags (optional).
You can import the required configuration data from policies that are created
automatically from previous installations. Click Select (next to the configuration task)
and choose the appropriate policy from the list. Policies can be manually prepared and
can also be automatically derived from any profile so that you can reuse them.

In Nodes Assignment step, choose the servers you wish to install HyperFlex on:

 Servers that are already associated cannot be selected.

 Only valid HX servers can be selected.
 This step is optional. The profile can be created without server selection on creation.
You can create profiles and later assign servers to those profiles. This enables you to
have configurations ready and assign new servers to a configuration.

In Cluster Configuration step, provide services and addressing:

 Provide the ESXi and CVM credentials.

 System IP addressing and VLANs.
 NTP, DNS, Proxy, Auto Support (ASUP) service configurations.
 Select the uplink speed.
 Provide vCenter credentials and hierarchy.
In Nodes Configuration step, verify node addressing:

 Intersight will display actual addressing for individual nodes.

 You can verify how assigned pool ranges translate to actual interface addressing.
 The storage network IP addressing is automatically in the 169.254.0.0 range.
Because the storage network addressing is automatic, it is vital that you separate the
individual HyperFlex storage networks inside their own VLANs or you will experience
IP conflicts.

Because of overlapping IP addressing in the storage VLAN, it is vital that you assign no
router interfaces to the storage VLAN. If that is not an option, make sure you disable
ARP proxying.

In the Summary step, review the entered configuration options:

 Verify the notices about the configuration.

 Verify the configuration information before starting the validation and deployment.
If you do not choose auto-support or configure email address for support tickets during
configuration, you can configure both after installation.

After the validate & deploy workflow is triggered, Intersight will attempt to install
HyperFlex:

 Validations will test the connectivity and service availability.

 If there is an issue, Intersight will provide feedback on the issue.
 Amber warnings can generally be skipped, while red warnings will prevent
installation.
The warning in the example shows an issue caused by incorrectly set up DNS. PTR
entries were not added and Intersight could not translate IP addresses into domain
names. This does not prevent you from installing HyperFlex, however all configurations
will be done using IP addresses. This will, for example, result in ESXi servers being
displayed by IP address in vCenter.

Note
You can avoid most issues by flowing the instructions in the installation guide and
filling out the pre-installation documentation.

Post-Installation Tasks and Installation Verification

Like with the on-premises installation, you will need to perform post installation tasks
through a script. Since HyperFlex Installer is not available, the script can be accessed
from the CVM command line.

To finalize and verify installation of your HyperFlex Edge deployment:

 Run the post-install script from the CVM command line that is available through the
HyperFlex group virtual IP.

hx_post_install

o The steps in this interactive script are the same as if they were run from the on-
premises installer for a Standard deployment.

 Verify the installation:

 o System should be online and healthy.
o Create a datastore. That datastore should successfully mount to all group nodes.
o Verify that HyperFlex plug-in is registered in vCenter.

Using the post-install script, configure the following:

 License for the vSphere host.

 Enable high availability/DRS on the group.
 Disable the SSH warning in vCenter.
 Add vMotion VLAN and configure vMotion interfaces.
 Add VM network VLANs.
 Perform HyperFlex Edge configuration check.

11.7 Introducing Cisco HyperFlex Edge

Management and Monitoring

This video explores management, monitoring, upgrades, and maintenance. Management

and monitoring interfaces of the Cisco HyperFlex Edge are similar to the standard deplo
yments. However, as you remember, there is no Cisco UCS Manager.
The Cisco IMC replaces the Cisco UCS Manager in these Edge deployments. It is what
provides the hardware inventory, remote call, home features, hardware configuration an
d management, automation APIs, and Cisco Intersight integration through the device co
nnector.

The vCenter configuration is the same as a standard HyperFlex deployment. The only e
xception is the single-switch deployment with access ports. Using access ports flattens t
he VLAN structure of the network into two separate VLANs. This means only two port
groups are present in the ESXi hypervisor for the HyperFlex hosts.
HyperFlex Connect and the stcli have the same functionalities for both the Edge and sta
ndard deployments. This includes basic VM management functions, replication protecti
on, monitoring and maintenance features, as well as the web CLI.

When upgrading any element of a running cluster, it's important to know how the upgra
de process executes, so you must consider what effect this will have on the system. Hyp
erFlex Edge supports automatic updates of its components through the HyperFlex Conn
ect web interface.

The Upgrade workflows can be found in the Upgrade tab. Here you can update ESXi ho
st and the HyperFlex Data Platform software. However, the hardware firmware-- this ha
s to be upgraded manually.
This figure lists the steps for updating the HyperFlex Data Platform software. You can u
pgrade the software to a newer version through the Upgrade tab in HyperFlex Connect.
A specific upgrade bundle must first be downloaded from software.cisco.com in
order to perform the upgrade. Once you drag and drop the bundle file into the Upgrade i
nterface, click Upgrade.

Keep in mind the underlying hypervisor is done in another workflow. You can choose t
o upgrade the hypervisor manually by installing the upgrade.vib files in the hypervisor.
This manual process can be done in a rolling fashion so that the cluster is active during t
he upgrades. Only one server can be upgraded at a time with the online upgrades.

In an offline upgrade procedure, you shut down the HyperFlex cluster and perform the u
pgrades. The offline upgrade can only be done manually since HyperFlex Connect is no
t available.
Finally, the firmware upgrade is performed in Cisco IMC. The firmware cannot be upgr
aded automatically because Cisco UCS Manager is not present. To upgrade the firmwar
e, you must perform a manual upgrade.

The manual upgrade can be done in a rolling fashion. This is done by putting the server
into the maintenance mode, updating it, and then rejoining it to the cluster before updati
ng the next server.

That brings us to the end of the video. Once again, thanks for watching.

Management and monitoring interfaces of Cisco HyperFlex Edge are similar to the
Standard HyperFlex deployment, but there is no Cisco UCS Manager. Instead, Cisco
IMC is used for hardware monitoring for on-premises installations.

Cisco Intersight provides the Edge deployments with a centralized management

interface that has a much broader scope than Cisco UCS Manager and can manage and
monitor several locations and many HyperFlex installations.

Cisco IMC
Cisco IMC provides a per-server hardware management and monitoring but requires
administrators to log in to each server to make changes.

Cisco IMC can also be integrated into centralized monitoring solutions, such as SNMP
systems on a per-served basis.

Cisco IMC offers these functions:

 Hardware inventory, health status, and SNMP reporting.

 Remote call home features and user management including Lightweight Directory
Access Protocol (LDAP) integration.
 Hardware configuration, including VIC, BIOS, power monitoring, and storage
management.
 Provides advanced features, such as automation API and Cisco Intersight
integration, through the device connector.

The Cisco IMC controller replaces the Cisco UCS Manager in Edge deployments. Cisco
IMC is superseded by Cisco Intersight, but remains an option for an environment
without Intersight access.

Cisco IMC allows certain firmware to be updated directly. However, you cannot update
the entire firmware stack on-premises, it must be done manually by booting into the
Cisco UCS update .iso image. Cisco Intersight allows full-stack upgrades of entire
HyperFlex groups.

vCenter and vSphere

The vCenter configuration is the same as in standard HyperFlex deployment. The only
exception is single switch deployment with access ports.
Because using access ports flattens the VLAN structure of the networking into two
VLANs, only two port groups are present in the ESXi hypervisors of HyperFlex hosts.

vSphere in Edge is similar to Standard HyperFlex:

 The HyperFlex plug-in, ReadyClones, native snapshots, snapshot scheduling, and

notification syncing all function the same as in Standard.
 Create, run, suspend, and power on/off VM are available from HyperFlex Connect.
 Redundant networking in 2-switch deployments

HyperFlex Connect and hxcli

HyperFlex Connect offers similar functionality to Standard HyperFlex deployment:

 Basic VM management: Power on/off, suspend, ReadyClone.

 Replication: VM protection, protection groups, system pairing.
 Monitoring: Performance and event monitoring, integration with vCenter.
 Maintenance: HyperFlex and ESXi upgrades, HyperFlex maintenance.
 Web CLI: hxcli support from the web interface.
HyperFlex Connect and hxcli perform the same functionalities on Cisco HyperFlex
Edge deployments as in standard deployments.

HyperFlex Edge and Cisco Intersight

The advanced features of Intersight such as Intersight Kubernetes Service (IKS),
Intersight Workload Optimizer (IWO) and Intersight Services for Terraform all use the
secure connectivity that Intersight provides. Therefore, all these advanced services are
available to all your HyperFlex Edge systems, regardless of their physical location.

Cisco Intersight is especially well suited for HyperFlex Edge and provides the means to
manage, upgrade, and back up your HyperFlex systems:

 Global and parallel installation: Gives businesses the ability to deploy HyperFlex
anywhere without outsourcing or travel.
 Global management: Many small Edge Systems still need to be managed,
monitored, and inventoried.
 Global Cisco Support: Cisco TAC is available globally and TAC cases can be
opened at will through Intersight.
 Modern workforce: Bring the data center to employees where they need it.
Lowering latency, availability and removing dependence on ISPs.
 Centralized many-to-one backups: Manage and create replication jobs for
replication of many Edge deployments to a central Standard deployment.
Intersight provides a powerful centralized replication solution that can fully be set up
and managed from Cisco Intersight. This many-to-one (N:1) replication solution
provides a way to replicate the data from all the HyperFlex Edge system across the
world in one central location.

Many-to-One Replication for HyperFlex Edge

Many-to-one backup solution for HyperFlex Edge is a unique feature that is only
available to HyperFlex Edge deployments and uses the integrated native replication
capabilities of HyperFlex systems.

N:1 backups enable centralized global replication of VMs:

 Target system is a HyperFlex Standard deployment dedicated to gathering VM

backups from many sources.
 Source systems are HyperFlex Edge systems where VMs normally run and are
regularly replicated to the target system.
 Full configuration and management of the N:1 replication is provided through Cisco
Intersight.
 After configuration in Intersight the HyperFlex systems establishes, replication
connectivity directly to each other.
 Additional network is dedicated to replication jobs, just like with the typical
HyperFlex one-to-one replication.
N:1 backups for HyperFlex Edge are fully configurable after HyperFlex installation and
no additional configuration steps are necessary.

Configuring Many-to-One Backups

Many-to-one replication can only be configured from Cisco Intersight. Both the target
Standard HyperFlex deployment and the individual Edge deployments need to be
registered to Intersight to enable the configuration.

The replication is directed from many HyperFlex Edge systems toward the central
Standard HyperFlex system. Each of the converged nodes requires an additional
network configuration to join in the N:1 replication.

There are several general prerequisites for N:1 replication:

 Configure a replication VLAN for each of the locations.

 Assign IP addressing for the replication VLAN interfaces.
 A Standard HyperFlex deployment as the target.
 All replication VLANs need to be reachable over Layer 3.
 Configure the appropriate MTU for the connectivity. All systems should use the
same MTU.
 Systems involved in N:1 backups cannot be also part of the native 1:1 replication.
 Intersight Essentials license required for N:1 replication.
Note
You need to ensure that the network connectivity between the source and target sites is
available. If data security is a consideration on your specific configuration, make sure to
provide a secure link between locations.
The target system has additional prerequisites:

 The target system needs bandwidth equal to the sum of all source Edge system
replication bandwidths.
 Maximum of 20 Edge systems replicating to one Standard system is currently
supported.
 Maximum of 100 replicated VMs per Edge system is currently supported.
 All the Edge systems need Edge Premier license for N:1 replication.
 Cisco UCS Manager of the target system needs to be configured with the replication
VLAN.
 Just like source systems, the replication network requires IP address per converged
node plus one.
Every HyperFlex server (converged node) is assigned an IP address in the replication
network. One additional address is necessary as the virtual IP replication network
address for the group.

Although the requirements for the target and source systems involved in N:1 replication
are different, the setup is very similar. The network configuration is performed through
Cisco Intersight.

Configure the source and target nodes in Cisco Intersight:

 Access Operate > HyperFlex Clusters and choose Configure Backup for a
HyperFlex group.
 Enter the backup configuration for the system by setting up the network.
 Click Validate and Deploy to apply the configuration.

The IP interfaces on CVMs will be configured automatically without interruption to the

workload. The upstream network needs to be properly configured to allow system
involved in the N:1 replication to be reachable.
You will not be able to configure backups on HyperFlex systems that are not running
HyperFlex version 4.5(1a) or higher. They will simply not have the option to configure
the N:1 replication.

Create the Virtual Machine Backup Configuration policy in Intersight:

 Define backup datastore name prefix, back up interval, and snapshot retention count.
 Choose the backup target, which is the target Standard HyperFlex system to which
backups will be deployed.
 Click Validate & Deploy to apply the configuration.

When protecting virtual machines, a datastore is selected for protection. All virtual
machines that are exclusively on that datastore will automatically be protected. If a VM
uses any other datastore, aside from the designated replication datastore, that VM will
not be protected.

If you move a VM from the protected datastore, the VM will not be protected, since the
datastore defines the protected VMs.

To restore a VM from a backup:

 Choose Operate > HyperFlex Clusters > Protected Virtual Machines.

 Choose the Virtual Machine to restore and the snapshot version to restore.
 Choose the target system to restore the snapshot to and start the restore process.

When restoring virtual machines from backup, you have the ability to customize the
virtual machine settings and select whether the virtual machine should be powered on
after the restore action.
11.8 Introducing Cisco HyperFlex Edge

Upgrades and Maintenance

You can upgrade HyperFlex Edge and Standard deployments from Intersight. All parts
of the HyperFlex Edge systems can be upgraded from Intersight: HXDP, infrastructure
firmware, and ESXi hypervisor.

The following recommendations apply to the HyperFlex Edge components that you will
aim to upgrade:

 You can perform a full-stack upgrade of any M5 HyperFlex Edge deployment.

 ESXi upgrade requires the Intersight Essentials license.
 M4 server generation is not supported for Intersight upgrades.
 Edge running Cisco IMC 4.1(3b) or lower, will temporarily disable secure boot for
upgrade.
 Intersight-based HealthCheck utility will help you identify potential problems.

Once you complete the upgrade of any HyperFlex component, as with the Standard
deployment, verify that the upgrade was a success:

1. Verify that the upgraded systems are online and healthy.

2. Verify that datastores are up and mounted properly on all ESXi hosts. Navigate
to HX Connect > Datastores and select the individual datastore. For each browser
interface that you use, empty the cache and reload the browser page to refresh the
HX Connect content.
3. Confirm that all servers share the same (upgraded) versions of HXDP, Cisco IMC,
and ESXi.

After the upgrade, manually upgrade the vCenter HTML plug-in if you are using
HTML version of vCenter. If you are using Flash version, it will be upgraded
automatically.

Intersight-Based HyperFlex Edge Upgrades

The Cisco HyperFlex automated Intersight upgrade process performs a rolling update of
the chosen system. You can configure several updates in HyperFlex Connect at the
same time through Intersight, and they will be performed in sequence.

The on-premises upgrades are also available for air-gapped HyperFlex Edge
deployments. All the Standard deployment upgrade considerations and procedures
apply, except that the Cisco UCS firmware needs to be upgraded manually server by
server when not using Intersight to upgrade.

During online upgrades, VMs will be migrated from server to server. Make sure all
VMs are able to migrate. This includes proper VMware licensing, except for 2-node
Edge systems, which perform migrations without DRS-enabled VMware license.
Make sure you have sufficient resources on your Edge HyperFlex group to normally run
the workload with one server in maintenance mode. Shut down some of the workload if
possible to free up group resources.

To upgrade Edge, follow the same steps as with Standard upgrades:

1. Consult with the latest version of the upgrade guide.

2. Verify that the HyperFlex is online healthy, and all the nodes and disks are online.
3. Upgrade during a low usage and/or maintenance window.
4. Verify the caveats for the new version combination (Cisco IMC firmware, ESXi
software, and HXDP software).
5. Perform Intersight-based or on-prem health check on the upgraded HyperFlex.

You can find the Intersight upgrade guide and further links to relevant resources are
available in Cisco Intersight
documenation: https://intersight.com/help/saas/resources/upgrade_cisco_hyperflex_syst
ems_with_cisco_intersight#upgrade_prerequisites

Note
Even though this is not strictly necessary, it is very useful to have access to the vCenter
of the upgraded system to follow along with the upgrade process and catch any potential
issues, such as VMs not migrating due to incorrect configuration.

Manual Firmware Upgrades

If you need to perform a manual upgrade of the Cisco IMC firmware for
troubleshooting reasons, you can do so manually through Cisco IMC.

Manual upgrades are also needed for upgrades of air-gapped HyperFlex Edge systems
that cannot be joined to Intersight SaaS. Instead of upgrading the Cisco UCS Manager,
such as on the Standard deployments, you instead upgrade the Cisco IMC.

To perform the manual firmware upgrade, you will require the appropriate Cisco UCS
C-Series firmware ISO and boot into the iso through the Cisco IMC KVM.

You can perform a manual Cisco IMC firmware upgrade:

1. Verify that the HyperFlex is healthy and put a server into HyperFlex maintenance
mode.
2. Mount the Cisco UCS upgrade .iso in the server KVM interface.
3. Reboot the server into the update utility.
4. Update the server and re-join the server into the HyperFlex group.
5. Wait until the HyperFlex is healthy and then repeat the process for the next server.
You should confirm the server firmware version in Cisco IMC of each server once you
upgraded Cisco IMCs of all servers in the group.

Note
The legacy on-premises firmware upgrades give you more control over the upgrade
process, they are however rarely needed and Intersight upgrades are the preferred way
to perform the upgrades.
11.9 Introducing Cisco HyperFlex Edge

Investigate HyperFlex Edge

Investigate HyperFlex Edge
Since HyperFlex Edge connects directly to upstream switches, rather than to Fabric
Interconnects, there is no Cisco UCS Manager available. Instead, the HyperFlex
installer configures servers directly through the Cisco Integrated Management
Controller (IMC).

HyperFlex Edge is functionally very similar to standard HyperFlex deployments, but is

optimized for smaller deployments, which is why the management experience is nearly
identical across the management interfaces.

HyperFlex Edge supports various deployment types with 2 to 4 converged nodes. The
3- and 4-node deployments are identical except for the number of nodes, while the 2-
node deployment has certain specifics. In this lab exercise, you will explore a 3-node
deployment.

Investigate vCenter Configuration

Step 1

Log in to vCenter.

Answer
Using the Firefox browser, navigate to vCenter by typing https://vcenter.hx.lab into
the URL. On the vCenter greeting page, click the LAUNCH VSPHERE CLIENT
(HTML 5) button, and log in using student as the username and 1234QWer* as the
password.
Note
The username you will be using during the lab has read-only permissions assigned in
vCenter. The HyperFlex vCenter integration allows you to use the vCenter credentials
to log in to HyperFlex Connect.

Step 2

Explore the HyperFlex Edge group in vCenter.

Answer
Once you log in to the vCenter web interface, expand the HyperFlex Lab data center
and expand the Edge Cluster and CVMs resource pool.
You can see that each server has a CVM running on it, just like in any other HyperFlex
deployment. To separate the workload VMs from CVMs, the workload VMs are located
in the Virtual Machines resource pool.

Expand the Virtual Machines resource pool on the HyperFlex Edge group.

The HyperFlex Edge deployment is running the shared services for the HyperFlex lab
environment, such as vCenter, AD, NTP, DNS, backup server, and DHCP server. All
VMs are running off the HyperFlex storage of their respective group.

Choose Edge Cluster from the HyperFlex Lab data center and then click
the Datastores tab.

You will see a list of all the datastores of the infrastructure, with the Edge Datastore on
top of the alphabetically organized list. The HyperFlex datastores are of the NFS3 type,
while the local drives of the servers are VMFS 6.

So far, the HyperFlex Edge group appears no different from a standard HyperFlex
deployment. But where they do differ is networking.
Step 3

Explore the HyperFlex Edge Connectivity.

Answer
In the vCenter interface, click the esx1.edge.hx.lab host and choose the Configure tab
on the right. In the middle menu, choose the Physical adapters entry
under Networking and extend the Switch column of the Physical adapters list, so that
the switch names are fully visible.

For this HyperFlex Edge deployment, a design with two upstream switches was used.
As you are going to figure out in one of the next tasks, two controllers were installed on
each of the servers to support connectivity to both upstream switches. One of them is
the integrated 2-port X550 Intel LOM (LAN on Motherboard) controller and the second
one is the PCIe Intel I350 4-port controller. All six ports provided by those two
controllers can be seen as "vmnics" in the figure above. As you can see, only four of
those ports are used (two connections to each of the upstream switches) and they are all
configured for a gigabit speed operation.

You can see that the physical ports are distributed between the vSwitches to allow for
better redundancy. Keep in mind that there are of course no Cisco Fabric Interconnects
present in HyperFlex Edge deployment.

Click the blue and white icon in the top-left corner of your student VM and
choose Terminal Emulator from the menu. Type ssh 192.168.10.3 into the
terminal and press Enter. Use the password 1234QWer to log in.
You have just logged in to one of the top of the rack switches the HyperFlex Edge
group is connected to. Open another terminal in the same way and type ssh
192.168.10.4 into the terminal, then press Enter. Use the password 1234QWer to
log in.

student@student-vm:~$ ssh 192.168.10.4

Pre-authentication banner message from server:
User Access Verification
Password:
<... output omitted ...>
93180-B#

The 93180-B is the second upstream switch in the infrastructure.

Type show interface description in both terminals and press Enter.

93180-A# show interface description

<... output omitted ...>
Eth1/9 eth 1000 EDGE-1 LOM-1
Eth1/10 eth 1000 EDGE-1 LOM-2
Eth1/11 eth 1000 EDGE-2 LOM-1
Eth1/12 eth 1000 EDGE-2 LOM-2
Eth1/13 eth 1000 EDGE-3 LOM-1
Eth1/14 eth 1000 EDGE-3 LOM-2
<... output omitted ...>

93180-B#show interface description

<... output omitted ...>
Eth1/9 eth 1000 EDGE-1 PCIe-1
Eth1/10 eth 1000 EDGE-1 PCIe-2
Eth1/11 eth 1000 EDGE-2 PCIe-1
Eth1/12 eth 1000 EDGE-2 PCIe-2
Eth1/13 eth 1000 EDGE-3 PCIe-1
Eth1/14 eth 1000 EDGE-3 PCIe-2
<... output omitted ...>

The highlighted parts of the output describe the configuration you have explored in the
vCenter. There are two redundant links from each physical NIC to one of the top of the
rack switches.

Click the Virtual switches entry under Networking in the middle menu.
Choose vswitch-hx-inband-mgmt from the drop-down menu.

Notice that the vSwitch has two uplinks that can be seen under Physical Adapters: one
from the port Quad port Intel I350 NIC and another one from 2-port X550T Intel LOM
controller. Compare that to the output of the show interface
description command on the upstream switches, and you will notice these are not
the only uplinks to the outside network.

You can also see port groups and VLANs associated with the particular port group that
are configured on the vSwitch:

 Management: 3091
 Replication: not configured
 vm-network: not created
 Student Access Network: 3090

Note
The vm-network would have its own vSwitch and separate uplinks on a HyperFlex
Standard group, but due to the limited uplinks on NIC cards, this network is not
configured. If you want, you can configure it manually on the management vSwitch.
Never attach workload VMs to the HyperFlex storage vSwitch. This will make
workload VMs share the physical uplinks with the storage traffic and may cause
interference with storage operations under load.
Collapse the vswitch-hx-inband-mgmt vSwitch and expand the vswitch-hx-storage-
data one. Observe the uplinks and port groups that are defined on the vSwitch.

You will notice that the uplinks used for this vSwitch are different compared to the ones
used for vswitch-hx-inband-mgmt vSwitch.

Note the port group VLANs:

 Data Network: 3392

 vMotion: 3093
 iSCSI Primary Network: not configured
 iSCSI Secondary Network: not configured

Data network is configured automatically during the HyperFlex group installation while
the vMotion network can be configured with the post_install script after the installation
is complete. There are also two ISCI port groups that are needed for the HyperFlex
iSCSI Target Service. Those port groups are configured through HX Connect.

Step 4

Investigate HyperFlex Edge Networking.

Answer
The networking in HyperFlex Edge follows the physical connectivity implications of
the individual deployments. The VLANs defined on the HyperFlex port groups need to
be defined on the upstream switches.

Return to the terminal windows connecting to the upstream switch command line or use
the ssh command to re-connect to the upstream switch command line to
addresses 192.168.10.3 and 192.168.10.4 in two terminal windows. Use the
password 1234QWer.
Type show vlan in both terminal windows to list VLANs enabled on individual
ports on the switches. Remember, from the output of the show interface
description command, that on both switches ports 9–14 are assigned to the
HyperFlex Edge group.

93180-A# show vlan

<... output omitted ...>

3091 management-network active Po1, Po2,

Eth1/1, Eth1/2
Eth1/3, Eth1/4,
Eth1/5, Eth1/6
Eth1/7, Eth1/8,
Eth1/9, Eth1/10
Eth1/11,
Eth1/12, Eth1/13
Eth1/14,
Eth1/44, Eth1/45
Eth1/47, Eth1/48
<output omitted>
3392 edge-cluster-data active Po1, Po2,
Eth1/9, Eth1/10
Eth1/11,
Eth1/12, Eth1/13
Eth1/14,
Eth1/44, Eth1/45
Eth1/47, Eth1/48
<... output omitted ...>

93180-B# show vlan

<... output omitted ...>

3091 management-network active Po1, Po2,

Eth1/1, Eth1/2
Eth1/3, Eth1/4,
Eth1/5, Eth1/6
Eth1/7, Eth1/8,
Eth1/9, Eth1/10
Eth1/11,
Eth1/12, Eth1/13
 Eth1/14,
Eth1/44, Eth1/45
Eth1/47, Eth1/48
<... output omitted ...>
3392 edge-cluster-data active Po1, Po2,
Eth1/9, Eth1/10
Eth1/11,
Eth1/12, Eth1/13
Eth1/14,
Eth1/44, Eth1/45
Eth1/47, Eth1/48
<... output omitted ...>

The data VLAN of the HyperFlex Edge group (VLAN ID 3392) is only permitted on
the ports of that group. The management VLAN is shared between all the groups in the
infrastructure, so it is defined on all ports.

Note
Ports 1-8 are assigned to Fabric Interconnect uplinks, while the listed ports in the 44-48
range are part of the upstream virtual port channel configuration.

Type show interface quick in both terminal windows. Press the Space key to
advance the output, q key to quit to command line.

93180-A# show interface quick

<... output omitted ...>

Ethernet1/9 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address:
0027.e393.a170 (bia 0027.e393.a170)
Description: EDGE-1 LOM-1
MTU 9216 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 0/255, txload 0/255, rxload 0/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
<... output omitted ...>

Ethernet1/10 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address:
0027.e393.a171 (bia 0027.e393.a171)
Description: EDGE-1 LOM-2
MTU 9216 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 0/255, txload 0/255, rxload 0/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
<... output omitted ...>

93180-B# show interface quick

<... output omitted ...>

Ethernet1/9 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address:
0027.e3ad.43ae (bia 0027.e3ad.43ae)
Description: EDGE-1 PCIe-1
MTU 9216 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 0/255, txload 0/255, rxload 0/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
<... output omitted ...>

Ethernet1/10 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address:
0027.e3ad.43af (bia 0027.e3ad.43af)
Description: EDGE-1 PCIe-2
MTU 9216 bytes, BW 1000000 Kbit , DLY 10 usec
reliability 0/255, txload 0/255, rxload 0/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
<... output omitted ...>

You will notice that the jumbo frames are enabled on the interfaces facing the
HyperFlex Edge group since they carry storage traffic.

Note
In HyperFlex Edge deployments, the VLANs must only be defined on the ToR switches
and vSwitch port groups, since the Cisco Fabric Interconnects are not present. The
hypervisor and firmware configuration performed by the HyperFlex installer is done
through CIMC directly instead through Cisco UCS Manager.

Investigate Cisco IMC Configuration

The Cisco IMC is an integrated controller installed on each Cisco UCS server. It is
responsible for firmware and bios updates and monitoring as well as system
management. The Cisco UCS Manager connects to Cisco IMC to execute most of the
management functionalities on individual servers. In HyperFlex Edge deployments, the
installer communicates directly with Cisco IMC to perform configuration performed
through Cisco UCS Manager on Standard HyperFlex groups.

Note
For the Cisco IMCs to be accessible, it needs to have an IP address assigned. You can
assign it by connecting a monitor and a keyboard to the individual server and press F8
to enter the configuration interface.

Step 5
Log in to Cisco IMC of the first node of the HyperFlex Edge group.

Answer
In your browser, open the https://cimc1.edge.hx.lab URL. Log in using the
username student and 1234QWer* as the password.

The Summary page will be displayed showing an overview of the server

characteristics.

The Summary page will quickly let you know if there is an issue with the server
hardware if the server is overutilized. To find out more about a specific aspect of a
server, you can access more detailed information from the side navigation menu.

Step 6
Explore the Cisco IMC interface and functionalities.

Answer
In the top-left corner of the Cisco IMC interface, you will see a hamburger button with
an arrow. Click the button to show the navigation menu and navigate to different parts
of the Cisco IMC interface.

Note
Every time you will make a selection in the navigation menu, it will hide automatically.
You will have to access it by clicking the hamburger button again. Alternatively, you
can pin it by clicking the pin icon on top of the menu.

Click the Inventory menu entry in the Chassis section.

The Inventory pane allows you to investigate the hardware characteristics of a Cisco
UCS server by clicking the specific tab.

Click the Network Adapters tab in the Cisco IMC Inventory pane.
In the Network Adapters tab, you will see the two LAN cards available on the server:
the integrated Cisco LOM X550-T2 and the PCIe expansion card I350-T4. These are
the network cards that you saw in vCenter as vSwitch uplinks, and it was their upstream
VLAN configuration that you explored on the upstream switches. Compared to the
10/25G connectivity provided by VIC cards this is a more cost-effective, but less
performant deployment, relying on 1G links.

In the navigation menu, under the Chassis section, choose Faults and Logs.

In this tab, you will see the applicable warnings for the physical infrastructure. Most of
these warnings will also be present in vCenter, but in some situations, you should check
the Cisco UCS message log directly.

In the navigation menu, under Admin, choose the Networking entry.

The Networking pane, under the Admin tab, describes the Cisco IMC connectivity
configuration. The NIC operation mode is Dedicated, which means that the Cisco IMC
will use the dedicated IP port on the back of the server chassis for connectivity. This
configuration is also referred to as out of band management (OOB).

If a VIC is installed in the server, the IMC can operate in a shared mode, where the IMC
communication runs through the VIC as an extra virtual interface. It is also possible to
define VLAN tagging for the Cisco IMC from here.

Note
The configuration interface you see is very similar to the interface you will see when
initiating the Cisco IMC configuration for the first time on server boot by pressing F8.
Remember that the factory default password for Cisco IMC is password.

The Cisco IMC networking configuration is also responsible for smart Call Home and
device connector features. Unfortunately, the device connector connecting the Cisco
IMC to Cisco Intersight cloud management is not available on read-only accounts.

Investigate the HyperFlex Connect of the HyperFlex Edge Group

Step 7

Log in to HyperFlex Connect of the HyperFlex Edge group.

Answer
In your browser, open the https://hxconnect.edge.hx.lab URL. Log in by using the
username student and password 1234QWer*.
You will be forwarded to the Dashboard pane of the HyperFlex Connect.

Aside from only three nodes being present in the group and the name, the interface is
indistinguishable from a Standard HyperFlex group.

Step 8

Compare the Edge HyperFlex Connect interface with the one for the Standard
HyperFlex group.
Answer
Open a new tab in your browser and type the https://hxconnect.standard.hx.lab URL.
Log in as student and use 1234QWer* as your password.

Click through the HyperFlex Connect interfaces of both groups to compare them. You
will notice that there are iSCSI and Kubernetes tabs missing from the left menu on
HyperFlex Connect of the Edge group. This is because as of HyperFlex Data Platform
version 4.5(1a) iSCSI is not supported on HyperFlex Edge groups. Except that there is
no difference between the interfaces which makes management experience seamless
across all types of HyperFlex groups.
12.1 Troubleshooting Cisco HyperFlex

Introduction
When implementing your Cisco HyperFlex solution, it is important to follow best
practices and guidelines to best utilize your Cisco HyperFlex system. If all else fails,
you will also need to do basic troubleshooting of the system.

The Cisco HyperFlex solution is a combined system of several software and hardware
components. To effectively troubleshoot HyperFlex, you need to be familiar with the
system that is used in the HyperFlex solution.

Before you start troubleshooting Cisco HyperFlex, it is important to be very familiar

with seven major parts of HyperFlex:

 Cisco Unified Computing System (UCS) for physical hardware, hardware

virtualization, and top of rack networking.
 Hypervisor (ESXi, Hyper-V) to provide the compute virtualization and virtual
networking.
 Installer VM to provide on-prem deployment of the HyperFlex solution. Cisco
Intersight can be used to cloud based deployment.
 Storage Controller to provide control of the storage layer and present it to the
hypervisors.
 Networks to facilitate communication between the solution components.
 Hypervisor Manager (vCenter, domain controllers) to facilitate coordination of
compute resources between nodes.
 Cisco Intersight to provide global installation, monitoring, and management
capabilities.

Note
For efficiency, only the most common problems are covered. For advanced
troubleshooting, contact Cisco TAC. If you are very familiar with the individual
components comprising HyperFlex, refer to the Cisco HyperFlex Troubleshooting
Guide at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/HX_TroubleshootingGuide/4-5/b-hx-
troubleshooting-guide-4-5.html.
12.2 Troubleshooting Cisco HyperFlex

Cisco HyperFlex Best

Practices
Best practice when troubleshooting is prevention and being informed about potential
issues that are discovered. Cisco offers a notification service that provides automatic
notification through a notification service available
at https://cway.cisco.com/mynotifications.

Cisco Intersight will also include some of the notices in the HCL verification checks if
you have your HyperFlex systems registered to Intersight.

Register to Cisco HyperFlex-related notifications to keep up to date with HyperFlex

information:

1. Got to https://cway.cisco.com/mynotifications.
2. Click Create Subscription.
3. Select the type of subscription (RSS or Email).
4. Define how often you want to receive updates.
5. Provide your email or email group.
6. Select the type of notices and level of importance.
7. Define products (HXDP and HyperFlex HX-Series).
The scale from Informational to Critical notification designations provide information
on how important a notification is. Changing this option will allow you to regulate the
number of messages that you receive. Usually High and Critical levels are
recommended. To be kept up to date on more issues, you can enable a lower importance
level of messages.

You can change the configuration at any moment, add additional notification levels,
additional products, or even additional subscriptions.

Here is the explanation of Alert Types entries:

 End of Sale/Support Announcements: These notifications are relatively rare and

notify you if a product support status changes.
 Field notices: These are notifications that are published for significant issues.
Depending on your notification level, you can expect them more often than EOS
announcements. Minimally, the Critical and High subscription is recommended.
 Cisco Security Advisories: Less common than field notices. They describe
vulnerabilities in Cisco products. Critical and High notification subscription is
highly recommended.
 Example of a moderate-level notification providing information on API request
limits. This issue does not cause a security issue but needs to be followed when
automating the HyperFlex system.

Note
To research best practices further, you can read the Cisco HyperFlex Best Practices
White Paper at https://www.cisco.com/c/en/us/products/collateral/hyperconverged-
infrastructure/hyperflex-hx-series/white-paper-c11-744135.html.

Keeping up to date with best practices for managing and maintaining your HyperFlex
system storage is very important, especially since HyperFlex has its unique features that
you can utilize to optimize both storage performance and utilization.

To keep your storage performing optimally, follow the storage guidelines:

 Keep your system under 76% storage utilization, at 99% utilization the storage will
go into read-only mode.
 Native clones will lose some of their deduplication if moved off HyperFlex storage
and back.
 Remember that you can always solve resource shortage by HperFlex expansion.
 VM-level encryption will prevent deduplication and compression.
 Keep the number of HyperFlex datastores low to get the most out of space and
performance.
Intersight will aggregate warnings from all your HyperFlex systems, including storage
issues. These can be drive failures or running out of capacity. Monitoring functions are
part of the free Intersight license available to all users at no additional charge.

Properly manage snapshots, removing obsolete snapshots to optimize the amount of

data used by VMs. vSphere snapshots can also slow down performance of VMs, even
on HyperFlex storage.

Enable secure boot to improve the system security. Secure boot enables boot
verification, making sure that the system is secure from the moment it is powered on.

Note
To read up on storage management best practices, you can read the Capacity
Management in Cisco HyperFlex White Paper at
https://www.cisco.com/c/en/us/products/collateral/hyperconverged-infrastructure/
hyperflex-hx-series/white-paper-c11-744026.html#Cisco
HyperFlexHXDataPlatformoverview.

12.3 Troubleshooting Cisco HyperFlex

Cisco HyperFlex Security

Hardening
Cisco HyperFlex is a highly secure system by design. The default configuration uses
sound defaults that apply to most users. Depending on your environment, you may want
to upgrade the security configuration of your HyperFlex systems.
You can improve the security of several HyperFlex subsystems:

 SSL certificate signing and distribution

 Disabling weak encryption standards
 Harden ESXi and Cisco UCS subsystems.
 Enable communication through a proxy.
 Secure API communication
 Improve native replication security.
 Configure Syslog

Note
To learn more about HyperFlex system security hardening, refer to Cisco HX Data
Platform Security Hardening Guide available
at https://www.cisco.com/c/dam/en/us/support/docs/hyperconverged-infrastructure/
hyperflex-hx-data-platform/HX-Hardening_Guide.pdf.
12.4 Troubleshooting Cisco HyperFlex

Troubleshooting Guidelines

Now let's review some troubleshooting guidelines and generating tech support bundles. This
figure lists some troubleshooting guidelines. The most vital information is assembled in the
HyperFlex pre-installation checklist document this is found in the Install Guide of a specific
HyperFlex version that you're going to be installing. It is imperative to fill out the pre-
installation checklist document and make sure to follow the installation guide.

Be sure not to skip any of the steps. Quite honestly, most of the issues at Cisco TAC sees are
basically the consequence of users not following the procedures outlined in the documentatio
n. For all customers with valid Cisco service accounts, Cisco technical support provides around
the clock support services, so that when you run into trouble, you can contact Cisco TAC for
help.
If you claim your HyperFlex cluster into Cisco Intersight, then Cisco TAC can pull up each
support bundle directly from this location. If not, you will have to generate a support bundle
and then upload it to TAC, which takes a little bit more time.

TAC uses the support bundles to help troubleshoot issues. This figure lists the support bundles,
which are a collection of logs from several components within and used by Cisco HyperFlex.
They include the HX Data Platform Installer VM, controller VM, VMware ESXi host, and
VMware vCenter.
When you open up a tech support case with Cisco TAC, you may be asked to generate a tech
support bundle and then share this with Cisco TAC. This next figure here illustrates a scenario,
where an engineer is running into an installation issue, and he needs some help. When
creating the tech support case, TAC engineers might ask you to generate the deployment logs.
This is easy to do from the installer GUI. It only takes a few minutes to complete.

Another way of collecting deployment logs is by connecting to the Installer VM using SSH and
issuing the deployment-support command. You can also use HyperFlex Connect to generate a
support bundle that collects the logs. When you use this method, audit logs are automatically
included. However, vCenter logs are not collected. They will need to be generated separately.
But what happens if the cluster is down? In this scenario, you will not be able to access
HyperFlex Connect in order to collect your logs. This means that you're going to have to
manually log in and generate the logs. So for each controller VM, you're going to have to run
the command storefs-support. And for each ESXi run the command vm-support. Be patient.
It takes several minutes. Then retrieve the files from the /bar/support directory.

Lastly, you can collect logs for ESXi hosts, controller VMs and vCenter Server, using the vSpher
e web client. Through the Navigator, you would choose vCenter, Inventory Lists, Resources,
vCenter. Then right click the vCenter and choose Export System Logs from the Source panel,
you choose the servers. And from the Ready to Complete panel, choose the system logs to
include. And then lastly, click Generate Log Bundle. This brings us to the end of the video.
Thanks for watching.

Maintaining and managing your HyperFlex system is mainly performed through its
management graphical interfaces, which will notify you if there are issues on the
system.
In day-to-day operations from the user perspective, Cisco HyperFlex can be used
entirely through the vCenter interface, where the HyperFlex plug-in provides all the
core functionalities for managing the underlying HyperFlex infrastructure.

If something goes wrong, these management interfaces of Cisco HyperFlex will notify
you of the issue. Simple issues such as Intersight connectivity can be resolved with little
to no risk, but since HyperFlex is a storage system, it is very important to have in-depth
knowledge of Cisco HyperFlex before attempting to resolve potential issues yourself.

Troubleshooting during the installation and configuration process is less risky as the
storage is not yet populated with data and critical issues potentially result in re-
installation of the system, without any risk of data loss. When stuck with an issue or
risking data loss, you can turn to Cisco TAC support team that are globally available 24
hours and will help you resolve your issue effectively.

Consider these key points when faced with an issue on your HyperFlex system:

 Ethernet network is a key system of Cisco HyperFlex and can critically affect
storage.
 Most issues during installation are a result of not properly filling out the Pre-
Installation document.
 Do not take steps that you do not understand, even if you find them in official
documentation.
 During verification, be careful not to cause further issues.
 Storage logs are available in the /var/log/springpath directory on CVM.
 Perform all verifications on the primary node (CVM).
 Firmware and software versions of components are key when troubleshooting.
When making changes to the upstream network configuration, you need to be aware that
those changes might affect the HyperFlex storage, not only the connectivity of virtual
machines. This issue is not as prominent in systems with Fabric Interconnects, since the
storage communication is contained within the Cisco UCS domain under normal
circumstances.

The importance of network changes is increased for HyperFlex deployments that do not
use Fabric Interconnects, such as HyperFlex Edge. These deployments rely directly on
the upstream switch configuration for storage operation.

Note
Be especially careful when the upstream networking and the HyperFlex system are in
different department jurisdictions. Because of the design of HyperConverged systems,
network administrators should be kept in the loop on the network configuration
requirements of HyperFlex storage.

One CVM is designated within each HyperFlex system as the primary node. This node
carries the virtual IP (VIP) address of the HyperFlex group. The VIP of the primary
node is available on the same address as the HyperFlex Connect GUI.

When researching issues, always verify the versions of all components, or the issues
that you find might not apply to your system. More importantly certain solutions might
not work or can even be detrimental when executed on the wrong HyperFlex version.

You can review different troubleshooting scenarios and actions in the Cisco HyperFlex
Systems Troubleshooting Reference
Guide https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/HX_TroubleshootingGuide/4-5/b-hx-
troubleshooting-guide-4-5.html.

Installation and Maintenance Troubleshooting Guidelines

Following the recommend guidelines provided by Cisco will avoid most of the issues,
regardless if you are just installing Cisco HyperFlex or if you are already using it in
production.

For the HyperFlex Installer to properly configure your HyperFlex installation, it is vital
that you provide the correct values to the installer and properly configure your network
and shared services.

The most important information is assembled in the HyperFlex Preinstallation

Checklist at https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/HyperFlex_Preinstall_Checklist/
b_HX_Data_Platform_Preinstall_Checklist.html, and can be found in the install guide
of a specific HyperFlex version you are installing

It is imperative to fill out the HyperFlex Preinstallation Checklist and follow the
installation guide. Most issues that Cisco TAC sees are because users have not followed
the procedures that are outlined in the documentation.
Note
To make gathering information easier and more interactive you can also enter the
gathered information from the HyperFlex Preinstallation Checklist document into the
HX Pre-Install web form at https://hxpreinstall.cloudapps.cisco.com. This will allow
you to interactively double-check your information. The HX Pre-Install tool also allows
you to export the installation information into a .json format that can be imported into
the HyperFlex installer. Export works for Intersight and the on-premises installer.

To avoid troubleshooting, follow the HyperFlex guides published on Cisco web page.

 If performing the installation, refer to the installation guide. For the upgrade, refer to
the upgrade guide.
 When expanding the system, refer to the administration and upgrade guides.
 Do not skip the checks suggested by any of the guides.
 Do not skip the pre-installation checklist. Skipping the checklist is the main cause of
most issues.
 Do not skip the post-installation tasks.
 Follow the interoperability information closely. Not all versions of Cisco UCS
firmware, hypervisor, and HyperFlex Data Platform are compatible.

Troubleshooting Intersight Connectivity

Intersight is the most powerful tool for HyperFlex administration and allows detailed
overview of the issues on your HyperFlex systems. It can also proactively report
potential issues through the Hardware Compatibility Check subsystem of Cisco
Intersight.
For Intersight to have access to your HyperFlex systems, the connectivity needs to be
properly configured and established.

If the Intersight connectivity is not available, perform these tests in the HyperFlex
Management VLAN:

 If you are using a proxy, verify the proxy works for other services in Management
VLAN.
 Ping a known good IP address to verify network connectivity, such as Cisco
OpenDNS IP 208.67.222.222.
 Use nslookup to resolve svc.intersight.com and verify the HyperFlex DNS can
resolve the domain.
 Verify the TCP/UDP port 443 is allowed using the nc -zv
svc.intersight.com 443 and nc -zuv svc.intersight.com
443 commands.

admin@cvm:/$ ping 208.67.222.222

PING 208.67.222.222 (208.67.222.222) 56(84) bytes of data.
64 bytes from 208.67.222.222: icmp_seq=1 ttl=54 time=16.3 ms
--- 208.67.222.222 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 16.348/16.348/16.348/0.000 ms
admin@cvm:/$ nslookup svc.intersight.com
Server: 208.67.222.222
Address: 52.72.146.139
admin@cvm:/$ nc -zv svc.intersight.com 443
Connection to svc.intersight.com 443 port [tcp/https] succeeded!

Even if you can open a TAC case for your specific issue, after the Intersight
connectivity is established you can then open a TAC case much more easily since
reporting a case becomes a one-click action that attaches logs automatically.

Reporting Issues to Cisco TAC

If you need to open a TAC case manually, you will also need to gather the necessary
information form the infrastructure, such as gathering logs and support bundles. This
process applies to installation and production issues.

Manual gathering of support files is a more complicated task than doing it with
Intersight, since there is no one central way to gather all the necessary information. You
need to gather the information from individual systems.

When you run into trouble, contact Cisco TAC for help:

 It is recommended that you claim your HyperFlex into Cisco Intersight, because
Cisco TAC can pull a tech support bundle directly from there.
 If you are not allowed or unable to connect to Cisco Intersight, or if your HyperFlex
is down, then you need to generate a support bundle and upload it to the TAC-
specified server for further analysis.
Gathering the necessary information from the underlying system is specific to the
individual system, but once the files are obtained, they need to be manually uploaded to
the Cisco TAC site.

For all customers, partners, resellers, and distributors with valid Cisco service contracts,
Cisco Technical Support provides around-the-clock, award-winning technical support
services. The Cisco Technical Support website provides online documents and tools for
troubleshooting and resolving technical issues with Cisco products and technologies:

Using the TAC Support Case Manager online tool is the fastest way to open S3 and S4
support cases. (S3 and S4 support cases consist of minimal network impairment issues
and product Information Requests.) After you describe your situation, the TAC Support
Case Manager automatically provides recommended solutions. If your issue is not
resolved by using the recommended resources, TAC Support Case Manager
(https://mycase.cloudapps.cisco.com/case) assigns your support case to a Cisco TAC
engineer.

For S1 or S2 support cases, or if you do not have internet access, contact Cisco TAC by
telephone. (S1 or S2 support cases consist of production network issues, such as a
severe degradation or outage.) S1 and S2 support cases have Cisco TAC engineers
(http://www.cisco.com/c/en/us/support/web/tsd-cisco-worldwide-contacts.html)
assigned immediately to ensure that your business operations continue to run smoothly.

The following is a partial list of support topics that should always be handled with TAC
assistance:

 Added disks to a node results in the disk not being recognized.

 Failure to add a node to the HyperFlex storage group.
 Changing the IP address for the HyperFlex storage system.
 Destroying a HyperFlex group, including destroying an encrypted group.
 Downgrading the HXDP version.
 Failing HyperFlex storage group creation.
 Modifying a node rebalance timeout.
 Modifying VDI or VSI optimized deployment for a HyperFlex storage group.
 Removing a node in a HyperFlex group.
 Replacing a node in a HyperFlex group.
 Replacing a housekeeping drive in servers.
 Reusing a removed node in another HyperFlex storage group.
 Reusing disks from a removed node.
 Setting the cleaner schedule for recovering storage.
 Setting the MTU value to something other than 9000 after installation.
 Upgrading the HXDP from a version older than HXDP Version 3.0.
 Using the hxcli commands whitelist or recreate.
 Any hardware-related issues.

Note
This llist is non-exhaustive. Always only perform actions that you are comfortable with,
if you are not sure about any action, please contact Cisco TAC for help.

12.5 Troubleshooting Cisco HyperFlex

Generating Tech Support

Bundles
When you open a tech support case with Cisco TAC, you will almost always be asked
to generate tech support bundles and share them with TAC.

Support bundles are a collection of logs from several components within and used by
the Cisco HyperFlex, including:

 HX Data Platform Installer VM: Logs provide information about installation.

 Controller VM: Logs provide information about the HXDP file system, HyperFlex
creation, and HyperFlex expansion.
 VMware ESXi host: Logs provide information about the nodes in the HyperFlex
storage system.
 VMware vCenter: Logs provide information about the HXDP Plug-in and the
vCenter server.

TAC uses support bundles to troubleshoot issues and learn about your HyperFlex
configuration, so that they may assist you in resolving the case.

Collecting Deployment Logs from the Installer VM

Log collection from the installer VM when you have issues with on-premises
installation is performed directly from the installer interface.

The HX Installer VM allows you to gather the support bundle through its GUI:
 Click on the question mark icon in the top-right corner of the user interface.
 Click the Create New Bundle button to initiate bundle generation.
 Generating deployment logs will take few minutes.
 Click the bundle link that will be listed under Tech Support Bundles.
 Previous support bundles will be listed as well. Always generate a new bundle when
making reports.

Another way of collecting deployment logs is by connecting to the installer VM using

SSH, logging in, and issuing the deployment-support command. Locate
the tar.gz logs in the /var/support directory.

The CLI method is generally not necessary, but it is available to you if you experience
problems with the HX Installer GUI.
Generating a Support Bundle Using HX Connect
HyperFlex Connect also allows you to generate a support bundle similar to HX
Installer. It also offers the same command from the command line that deposits logs in
the same location on the CVM where the log collection script was run.

You can use the HX Connect to generate a support bundle that collects the logs from
every controller VM and ESXi host in the storage group.

 When you generate a support bundle through the HX Connect user interface, audit
logs are automatically included.
 The vCenter logs are not collected through HX Connect. You will need to generate
them separately if required.

Log in to HX Connect:

 In the banner, click Edit settings (gear icon) > Support Bundle.
 Click Generate.

o Creating the support bundle can take 1–2 hours.

 When the support bundle displays, click supportBundle.zip to download the file to
your local computer.

Note
If you are using replication to protect your virtual machines and their data, then when
you need to generate a support bundle, you also need to generate a support bundle from
the remote HyperFlex storage system.

Collecting Logs from CVMs and ESXi Hosts Through CLI

The command-line log collection is the most reliable way to gather logs from any of the
underlying systems. If there are issues with GUI or if there is a significant issue on the
system, you might not be able to use the GUI to generate the support bundle.
To compile logs from the individual HyperFlex CVMs and ESXi:

 Log in to each CVM using SSH, generate logs and download them for each CVM:

o Run the storfs-support command.

o Wait for the script to finish.
o Locate and download the latest generated tar.gz log file in
the /var/support directory.

 Log in to each ESXi using SSH, generate logs and download them for each ESXi
host:

o Run the vm-support command.

o Wait for the script to finish.
o Locate and download the .tgz file in the /var/tmp directory.

You can gather the logs using the SCP command-line tool or by connecting to an
individual node using WinSCP or similar windows graphical tool.

Support bundles can be of significant size. When you are gathering support bundles
repeatedly, it can take up a lot of space on the CVM storage. This way you can run out
of storage space quickly, which will prevent you from gathering additional logs. On
older HyperFlex systems, there can also be issues with space in general.
An out of space error occurs when the storage controller VM does not have sufficient
space left to generate the support bundle, typically due to the size of the core file or
previously generated log files consuming space. The following error displays when you
are using the vm-support command to generate support bundles: error = [Errno 28]
No space left.

To generate a support bundle when you receive this error, follow these steps:

 Delete or move the core file and existing log files to a location outside of the storage
controller VM.
 Log in to the command line of the storage controller VM.
 Generate a light support bundle using the storfs-support command.

Collecting Logs from the vCenter Server

You can selectively collect all or some of the logs for the ESXi hosts, controller VMs,
and vCenter server through the vSphere Web Client.

Log in to the vSphere Web Client:

 Right-click the vCenter and choose Export System Logs.

 From the Source panel, choose the servers from which you want to collect the logs.
 Optionally, to include the vCenter logs, choose Include vCenter Server and
vSphere Web Client logs, and click Next.
 From the Ready to Complete panel, choose the system logs to include.
 Click Generate Log Bundle.

o Creating the support bundle can take 1–2 hours.

o When the logs are completed, download the log bundle.
12.6 Troubleshooting Cisco HyperFlex

Common Troubleshooting
Issues

This video covers some of the most common troubleshooting issues. More than half of Cisco
TAC cases on HyperFlex are from installation, upgrade, and expansion issues. Now, all of these
issues can be avoided by simply following the guides that Cisco publishes for installing,
administering, and upgrading Cisco HyperFlex.

This video points out some of the common issues of the installations, and upgrades, and the
expansions so that, well, hopefully, they're going to be successful.

The first figure lists the most common mistakes, which include DNS and NTP services as well as
vCenter not being functional for the HyperFlex installer. Once again, the HyperFlex installer as i
t's running through the process will catch these mistakes during the validation phase.
The next figure lists the most common mistakes that you can make during your HyperFlex
installation and how to make it successful. But it may be environmental. The environment
itself may be unstable.

In other words, everything looked good, but there were some issues, and the system will be
unstable. This includes using the default VLAN 1, which can cause networking issues.
Everything seems to have worked, but watch out. That may come back to bite you.

Using IP addresses already in use by other devices, so be careful to make sure that the IP
ranges you pick are unused, and, lastly, forgetting to configure the storage VLAN with jumbo
MTU on the upstream switches-- all three of these issues are very common errors.

Here, we have the most common mistake that will cause your upgrade attempt to be
unsuccessful—forgetting to bootstrap the installer when upgrading from HyperFlex 3.0 or
earlier. Also, if the cluster storage utilization is over 70%-- remember we talked about this
before-- you've got to constantly monitor the system to make sure that you don't go above
this threshold-- and, lastly, using the ESXi ISO to upgrade the hypervisor instead of the bundle
zip.
Here are some even more common mistakes that will cause the upgrade attempt to be
unsuccessful. You do not have the vMotion configured when performing an online upgrade.
Obviously, you'd need that.

You forget to enable EVC for vMotion when mixing servers with different generations of
processors. And lastly, you uploaded bundles A and C but not bundle B when you're upgrading
Cisco UCS.

Now, oh, here we have some common mistakes that will make your expansion unsuccessful.
For example, the hypervisor not installed on the nodes that you're trying to add to the cluster,
make sure they're on there first.

The hypervisor installed on the server being added to the cluster is not the same as the ones
on the existing nodes in the cluster, so make sure you've got the right version. And lastly, the
hypervisor was installed using a regular VMware image.
Oh, and even more common issues that you might run into during the cluster expansion,
expanding the cluster with HyperFlex installer which is not the same version as the current
cluster. Another one, password complexity requirements-- you set it for enhanced, but the
cluster was originally installed at a lower level. That's a good one. And SSH on ESXi was
disabled during the cluster expansion.

Finally, if you're using Cisco ACI for networking, don't forget to enable ARP flooding and GARP-
based detection. This is because both of these features are needed. And they're also disabled
by default. That's the end of this lesson. Thanks for watching.

There are several resources available for individual HyperFlex installation types on the
Cisco website. These resources provide extensive information regarding the installation
and upgrade processes and considerations.

There are several resources available to avoid and troubleshoot issues:

 Installer VM installation: Individual documents available for Hyper-V, Edge,

stretched clusters.
 Intersight Installation: Install guide for Edge and standard ESXi clusters.
 Upgrade: Individual guides for ESXi and Hyper-V hypervisors.
 Logging: HyperFlex Installer logs are stored in the /var/log/springpath/ directory
on the Installer VM.
 o Logs are separated into several log files describing different phases of
installation.

More than half of Cisco TAC cases regarding HyperFlex are due to installation,
upgrade, and expansion issues. All issues in this topic can be avoided by simply
following the guidelines that Cisco publishes for installing, administering, and
upgrading Cisco HyperFlex.

 Avoid installation and expansion issues by following the installation guides:

o For standard ESXi-based HyperFlex: Cisco HyperFlex Systems Installation

Guide for VMware ESXi
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_H
X_DataPlatformSoftware/Installation_VMWare_ESXi/4-5/b-hx-install-guide-
for-vmware-esxi-4-5.html).
o For stretched HyperFlex deployment: Cisco HyperFlex Systems Stretched
Cluster Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_H
 X_DataPlatformSoftware/HyperFlex_Stretched_Cluster/4-5/b-hx-systems-
stretched-cluster-guide-4-5.html).
o For Hyper-V deployment: Cisco HyperFlex Systems Installation Guide for
Microsoft Hyper-V
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_H
X_DataPlatformSoftware/HX_Hyper-V_Installation_Guide/4-5/b-hx-
installation-guide-for-microsoft-hyper-v-4-5.html).
o For Edge deployments: Cisco HyperFlex Edge Deployment Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_H
X_DataPlatformSoftware/Edge_Deployment_Guide/4-5/b-hx-edge-deployment-
guide-4-5.html).
o For installs from Cisco Intersight: Cisco HyperFlex Systems Installation Guide
for Cisco Intersight
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_H
X_DataPlatformSoftware/HyperFlex_Installation_Guide_for_Intersight/
b_HyperFlex_Installation_Guide_for_Intersight.html).

 You can avoid upgrade issues by simply following upgrade guides:

o For ESXi-based HyperFlex: Cisco HyperFlex Systems Upgrade Guide for

VMware
ESXi (https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/4-5/b-HX-
Upgrade-Guide-for-VMware-ESXi-4-5.html).
o For Hyper-V-based HyperFlex: Cisco HyperFlex Upgrade Guide for Microsoft
Hyper-V (https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/
HyperFlex_HX_DataPlatformSoftware/
HyperFlex_Upgrade_Guide_for_Hyper_V/4-5/b-hx-upgrade-guide-for-
microsoft-hyper-v-4-5.html).

 You can avoid Cisco ACI and HyperFlex by following CVD solutions for specific
HyperFlex deployment scenarios, such as: Cisco HyperFlex Edge 4.5 with Cisco
ACI and Hashicorp Terraform
(https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/
hx_edge_4_5_aci_terraform.html).

The installer itself is quite verbose. The following install and error logs are in
the /var/log/springpath of the installer VM:

 <cluster-name>/stDeploy.log
 <cluster-name>/ansible.log
 <cluster-name>/network_setup.log
 hxinstaller_#_deploy_output_failed.log
 hxinstaller_#_deploy_validate_output_failed.log
 hxinstaller_#_expand_output_failed.log

Note
For more general troubleshooting information, refer to Cisco HyperFlex Systems
Troubleshooting Reference Guide
(https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_Data
PlatformSoftware/HX_TroubleshootingGuide/4-5/b-hx-troubleshooting-guide-4-
5.html).

Common Troubleshooting Issues—Installation

Common installation failures result mostly from not properly preparing the
environment, incorrect or unreachable services such as DNS and NTP as well as an
incorrectly set up network, especially when MTU is mismatched between upstream
switches and the underlying HyperFlex systems.

The default MTU for Standard HyperFlex deployments is 9000, this is not necessarily
the configuration on the upstream switches, which can have different global defaults or
user configuration.

The following common mistakes will make your installation attempt unsuccessful:

 DNS service is not functional for the HX Installer.

o root@installer# nc -zv <dns_server_ip> 53

 NTP service is not functional for the HX Installer.

o root@installer# nc -zuv <ntp_server_ip> 123

 vCenter (if used for installation) is not functional for the HX Installer.

If you have a DNS server, an NTP server or vCenter functionality missing, the HX
Installer will catch that in the validations phase. That message will also be logged
into /var/log/springpath/stDeploy.log.
To troubleshoot the unavailability of DNS or NTP, first ping those two servers from the
CLI of the installer VM. If the ping is successful, proceed to use the nc command from
the installer VM. If the nc command does not return that the connection succeeded, it
probably means that the target is not functioning as DNS or NTP, or that traffic is not
allowed.

The following common mistakes can result in successful HyperFlex installation, but
make your environment unstable:

 Using VLAN 1.

o It is default VLAN and it can cause networking issues, especially if you use the
disjointed Layer 2 configuration.

 Using IP addresses that are already in use typically ends up in installation failure,
but can also make the HyperFlex system highly unstable:

o If reusing an existing IP pool, check the IPs you are assigning are actually
available.

 Forgetting to configure storage VLAN with jumbo MTU on upstream switch.

o Standard HyperFlex deployment use jumbo MTU by default.

admin@CVM:/# arping -I eth0 192.168.182.2

ARPING 192.168.182.2
60 bytes from 94:e6:f7:90:5f:56 (192.168.182.2): index=0
time=16.047 usec
60 bytes from 00:50:56:e2:ed:35 (192.168.182.2): index=1
time=3.745 usec

When using arping, all devices with the selected IP will respond. You can distinguish
between devices by their MAC addresses. If you see two different MAC addresses
responding, this indicates an IP conflict.

To investigate if IP is not already being used somewhere else in the network:

 Check the ARP tables using show ip arp.

 Ping from the SVI, not a random endpoint.
 ARP ping from the HX Installer arping –I eth0 <ip_address>.

Note
The arping will return responses through a router if you have ARP proxying
enabled. ARP proxying should be disabled on all VLANs used by Hyperflex but must
be disabled on the storage VLAN or it could cause problems.

Common Troubleshooting Issues—Expansion

Common expansion issues very closely resemble the issues you can experience during
installation. It is highly recommended that you document the initial HyperFlex
installation well and save the installation .json configuration for possible future
expansions.

It is also very beneficial to leave additional IP address space available for expansions
when designing the HyperFlex system. This can make IP addressing more consistent
and much easier to manage.

The following common mistakes will make your expansion unsuccessful:

 Hypervisor not installed on the nodes that you are trying to add to the group.

o Before you begin HyperFlex expansion, you will need to install the hypervisor
on compute-only nodes.

 Hypervisor installed on the new node is not the same version as hypervisor installed
on existing nodes.
 Hypervisor installed is from a regular VMware installation image.

o HyperFlex servers need to be imaged using Cisco custom ESXi images.

If you try to add a server without an ESXi installed to your HyperFlex group, the HX
Installer will show you an error message: Server rack-unit-1 config failed with
reason Unexpected prompt got through SoL for ESXi.
If the ESXi on node that you want to add does not match ESXi version on existing
nodes, the error message will read: Failure occurred during Add Nodes process:
Some(Node incompatible).

The difference between a Cisco custom ESXi image and a regular ESXi image is that
the Cisco custom image includes all the necessary drivers for Cisco servers and set up
the ESXi system differently.

The following issues are common during group expansion:

 Expanding groups with HyperFlex installer different version than the current HDXP
of the group.

o HyperFlex system version and HyperFlex installer version must match.

 The password complexity requirement has been enhanced, but system was originally
installed at a lower level and is using the less complex password.

o Change the password before expanding using the stcli security

password set command from a CVM.

 Having SSH on ESXi disabled during expansion.

o If you disable SSH after initial installation, which is supported since 2.5, you
will need to re-enable SSH when performing expansion or upgrade.

Cisco HyperFlex, being a completely distributed system, benefits greatly from

uniformity. Within a group, you will not be allowed to mix incompatible servers. Most
notably:

 Storage type (Hybrid vs. All-Flash vs. All-NVMe).

 Server chassis (220 SFF vs. 240 SFF vs. 240 LFF).
 Type of cache drive (SAS SSD, NVMe SSD, or Optane SSD).
 Size of capacity drives (for example, if one node has a configuration of 12x 960GB
NAND SSD, all the nodes must have this exact same configuration).
 Network speed on the VIC card (for example, if one server is connected using
2x40Gbps, all servers need to be connected using 2x40Gbps).

Password mismatch error will read: Failure occurred during Validate Deploy Nodes
process: Some(Authentication failure). This error can also be caused by you
mistyping the existing HyperFlex system password.

You installed a HyperFlex deployment (for example, with HXDP 4.0(1a)), then
upgraded it (for example, to 4.5(1a)). If you now want to expand, you must download
the installer that is the same version as the current version of the system (in this
example, 4.5(1a)) to perform this expansion.

ESX SSH lockdown mode can be enabled on each ESX node of the HyperFlex system.
This option applies only to a post-install system. SSH traffic must not be blocked during
install. Lockdown of SSH for ESXi is supported in HXDP Release 2.5 and above. The
following constraints apply to the deactivation of remote SSH access to the system for
versions prior to 3.5(1a):

 HyperFlex Snapshots for VMs are disabled (redo-log based snapshots still function).
 The source VM for a ReadyClone operation must remain powered off for a cloning
operation. Once the operation is complete, the source VM can be powered back on.
Clones themselves are unaffected.
 System upgrades or expansions are disabled until SSH is re-enabled.

In HXDP 3.0(1a) and above, snapshots and native replication do not use SSH to
interface with ESXi—meaning that neither root- nor hxuser-based SSH logins are
performed. Regarding logins to hosts (ESXi) for vSphere API access, only hxuser is
used. Root login is only used during HyperFlex group creation, node expansion, and
initial installation. You can read more about the implications of ESXi lockdown modes
in the Cisco HX Data Platform Security Hardening Guide.

Common Troubleshooting Issues—Upgrade

Upgrade issues can be more complex and more difficult to resolve than the installation
issues, since you cannot change the live environment as you would like. Especially
when performing an online upgrade.

Before performing an upgrade, you should always run the HyperFlex HealthCheck
script to verify the HyperFlex system is ready for upgrade.

HyperFlex system will fail to upgrade on the following common upgrade issues:

 Forget to bootstrap the installer when upgrading from the pre-3.5(1a) HXDP
version.
 Use ESXi ISO to upgrade hypervisor instead of bundle zip.
 HyperFlex storage utilization is over 70 percent.
 DRS / vMotion not triggering on maintenance mode because of the system CPU and
Memory overutilization.
 VMs are not migratable, which can be a result of a hardware pass-through or CPU
differences between nodes.
 SSH not enabled on ESXi, not enabled on boot, or ESXi hosts set to lock down
mode.
 Out of storage issues on any of the CVM partitions.

If you are running a HyperFlex release that is earlier than 3.5(1a), you must run the
manual bootstrap process to upgrade the HXDP as outlined in the upgrade guide.
However, if you are running HyperFlex Release 3.5(1a) or later, bootstrap will be done
automatically by the upgrade process itself. Not bootstrapping the system properly,
when this is necessary can cause severe issues.

If you are performing an upgrade on a mixed HyperFlex group (for example, Cisco
UCS M4 with Haswell generation processors and Cisco UCS M5 with Skylake
generation processors) and you forgot to enable EVC, vMotion will not work. You will
get a validation error: The virtual machine requires hardware features that are
unsupported or disabled on the target host. If possible, use a cluster with Enhanced
vMotion Compatibility (EVC) enabled; see KB article 1003212.

As part of the upgrade process, HyperFlex will check if your storage utilization is
higher than the warning level of 70 percent. If storage utilization is higher than 70
percent, it will not allow you to upgrade HyperFlex. You will need to move some of the
workload data to an external system to perform the upgrade.

If the CPU and memory utilization is too high when initiating an upgrade, the upgrade
will fail. Typically this happens because DRS would not migrate the necessary VMs,
since there is not enough resources on the HyperFlex system.

SSH access to ESXi and CVM is mandatory for upgrade jobs. ESXi nodes also must not
be in lockdown mode.

Upgrading using the HyperFlex ISO will result in a complete system wipe of the node
where the ISO was used. If this is done on several nodes, storage will become
unrecoverable and all data will be lost.

Certain user-based actions will cause an upgrade to fail as well:

 Renaming CVMs and attempting upgrade prevents upgrades and requires manual
command-line intervention from TAC.
 Virtual Distributed Switch (vDS) configuration on HyperFlex will cause HyperFlex
not to find appropriate networks. You will need to first migrate back to standard
virtual switches.
 If you are upgrading Cisco UCS as part of the upgrade and did not upload the B
bundle, because you are not using B-Series servers.
 Not disabling backup and replication jobs can cause a VM that needs to move
migrated to not be migratable.
To upgrade Cisco UCS firmware, you need to first upload Cisco UCS bundles A, B, and
C to the Cisco UCS Manager. Even if you do not have blade servers in your system, you
need to upload bundle B to the Cisco UCS Manager—otherwise, the HyperFlex upgrade
will detect that bundle B is missing and will terminate the upgrade. You would run into
an identical situation during installation if you wanted to upgrade Cisco UCS firmware
during the installation process—HX Installer would detect that bundle B is missing and
terminate the upgrade.

If you are doing an online upgrade, hosts in the group will be upgraded one at a time.
The guest VM workload will stay online and vMotion will make sure that the node that
is being upgraded will be vacated of guest VMs before being upgraded.

Common Troubleshooting Issues—HyperFlex Down

A common mistake if you are using Cisco ACI for networking is to forget to enable
ARP flooding and GARP-based detection (both disabled by default):

 HX Connect runs on the node with the HyperFlex management IP (CMIP).

 When the CMIP moves to a new node a GARP will be sent out.

o The default settings for an ACI bridge domain will not update the ARP entry.
o This results in being unable to reach the HX Connect interface and a
communications failure between the HyperFlex system and vCenter.

The CMIP interface is shown as eth0:mgmtip on the corresponding storage controller

VM. CMIP will have the same MAC address as the eth0 interface for the VM it is
currently running on.

Note
Consult Install and Manage Cisco HyperFlex Systems in a Cisco ACI
Environment (https://www.cisco.com/c/en/us/products/collateral/hyperconverged-
infrastructure/hyperflex-hx-series/whitepaper_c11-738327.html) for more information
about ACI integration of HyperFlex.