HCI1691 FINALv2 1600713111682001pujp
HCI1691 FINALv2 1600713111682001pujp
HCI1691 FINALv2 1600713111682001pujp
r ibut
is t
HCI1691
or d
Core Storage Best t i on
l ica
Practice Deep Dive pub
o r
tf
Configuring the New Storage Features
o
t: N
te n
o n
C
Jason Massae – VMware
2 0
Technical Marketing Architect
Cody Hosterman – Pure0Storage
d 2
rl
Technical Director VMware Solutions
w o
Engineering
VM
#vmworld # HCI1691
Disclaimer
i o n
r ib ut
is t under
development. o r d
This presentation may contain product features or functionality that are currently
d 2
presented, have not been determined.
o rl
w
The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no
VM
commitment or obligation to deliver any items presented herein.
o n
0 C
20 2
r l d
wo
VM
©2020 VMware, Inc. 3
i o n
r ibut
is t
or d
t i on
l ica
pub
Reliability is Key fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 4
i o n
ut
• GSS Top Storage Issue
Agenda t r ib
• NVMeoF
d is
or
• Latency PSP
t i on
l ica
• iSCSI
pub
f o r
• Queuing
o t
t: N
• VMFS
te n
o n
0 C • Misc
2 02
orl d
M w
V
©2020 VMware, Inc. 5
What are the Most Common Storage Issues GSS Hears?
Hello This is VMware Support, is This a New or Existing Case?
i o n
ut
r ib
is t
or d
t i on
l ic a
iSCSI FC b
pu UNMAP NAS
o r
otf
t: N
te n
o n
Port Binding
0 C
Configuration Latency VMFS or VM Configuration
Pathing
20 2 Pathing Connectivity HW Version
r l
Connectivity
o d Queuing Pathing Version Connectivity
M w
V
©2020 VMware, Inc. 6
i o n
r ibut
is t
NVMeoF or d
t i on
vSphere 7
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 7
NVMe over Fabric (NVMe-oF)
n
NVMe-oF
ut i o
t r ib
d is
or NVMe Host Software
NVMe-oF provides distance connectivity to NVMe
t i on
a
devices with minimal additional latency over a
c
Host Side Transport Abstraction
native local NVMe device
b l i
Key Features:
r pu
t fo
Fibre Channel
o
InfiniBand
N
• Support multiple transports
RoCE
:
TCP
e n t
• Simple command set
o n t
•
0 C
Up to 64k queues per NVM controller
02
• Up to 64K commands per queue
2
Controller Side Transport Abstraction
•
orl d
Removes SCSI from the storage stack
M w NVMe SSDs
V
©2020 VMware, Inc. 8
NVMe Access Modes
Protocols supported are FC and RoCE v2.
i o n
rut
ib
is t
r d
NVMeoF can be accessed:
o
SCSI ALUA States NVMe ANA States
t i
• Active-Activeon
a
access mode
l ic
Active Optimized Active Optimized
• bAsymmetric
Active Non-Optimized Active Non-Optimized
r pu Namespace Access
o
(ANA) mode
Standby Inaccessible
o t f
Unavailable Persistent Loss
t: N
Offline
te
Persistent Loss
n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 9
NVMe-oF Support in vSphere 7
High performance ESXi access to external storage arrays.
i o n
rut
ib
is t
Host Host
Overview r d
ooptimized controller
Host Multi-Path Host Multi-Path
i o n
NVMe is a highly
t
a
Software Software
interface that significantly improves
b l ic
performance for enterprise workloads
u
NVMe Driver NVMe Driver
o
• FC
t: N
• RDMA (RoCE v2)
n
FC RDMA
Fabric
n te Fabric
o
FC Transport
0 2 0C ROCE
2
Driver Transport Driver
r l d
NVMe NVMe
wo
subsystem
Single/Multipath subsystem
Single/Multipath
VM
Devices Devices
Storage Storage
l ic a
commands and payload between
fo r
t
Prerequisites:
o
Host RNIC (RDMA) Driver
t: N
• Host must have RNICs
te n
n
• Physical switches must be in lossless
o
RDMA
0C
Fabric
config
r l d
Target RNIC Driver
wo
storage array (VCG)
VM
NVMe-RDMA Target
ub
#esxcli system module parameters set -m bnxtroce -p disable_rocev2=0
p
Verify module #reboot
f o r
o t
N
They may be other
t:
#esxcli system module parameters set -p enable_roce-1 -m qedentv
settings depending
on RNIC vendor #reboot
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 12
Screen Shot of Available RDMA adapters
i o n
r ibut
is t
or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 13
Configure Network for NVMeoF RDMA
i o n
r ibut
is t
or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 14
Add NVMe over RDMA Adapter
i o n
r ibut
is t
or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 15
Adding NVMe Storage Controller
i o n
r ibut
is t
or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 16
NVMeoF Initial Release
Allowing partners and customers to evaluate NVMeoF.
i o n
r ibut
is t
o r d
Un-Supportedn
Supported Features
a t i o Features
b l ic
• FC
r pu• RDM
n te
SAME, Locking
C o • VAAI Plugins
0 2 0 • VAAI XCOPY
d 2
• Multipathing with HPP
o rl
• Active/Active
• NVMeoF namespace for core-dump
r
Latency=5ms
o Path
Storage
n
VMW
i o
PSP
a t
RR
Storage considerations
P1: OIO=5
b l ic • Latency
u
Latency=1ms
o r p • Pending IOs
Monitor inflowing IOs as sample IOs
t f
omoving average latency Sampling
Track issue and completion time, calculate
: N
/tP(Sampling
• Number of IOs
•
te n
P(avgLatency) = (Completiont - Issuet) IO count) • Time
n
o ‘m’ sampling
Stop sampling Window after
0 C
2
Select optimal path 0 2
•
o r l
IO Wait Timed = P(avgLatency) * (OIO + 1)
w
VM
Re-start sampling Window after ‘T’ interval
n te
useANO: Default C o0
0 2 0
Valid values are2
r l d 0 or 1
1,o
If set tow
VM
ESXi ignores ALUA settings and uses active non-optimized paths if latency deemed good
enough .
C o
0 2 0
$satpArgs.vendor=“Vendor“
l d 2
$satpArgs.satp="VMW_SATP_ALUA“
r
$satpArgs.psp="VMW_PSP_RR“
o
w
$satpArgs.pspoption="policy=latency“
M
V$result=$esxcli.storage.nmp.satp.rule.add.invoke($satpArgs)
t i on
Isolated/dedicated traffic (VLANs), redundant, dedicated interfaces and
a
connections
Port Binding is best practice
•
b l ic
r p u
o among Service Providers for optimal load
Load Balance
Distribute paths to fLUNs
•
o t
t
balancing
N
: protocols, unsupported
te n
Don’t mix
•
n
oRedundancy
0 C
2 02 • Make sure the initiator is connected to all network adapters used for iSCSI
orl d Storage
or d
Mix of SW & HW adapters
VMkernel VMkernel VMkernel supported
t i on
IPv4c &aIPv6 is supported on all 3
iSCSI network iSCSI HBA
l i
iSCSI initiator config driver
pu b
r SW iSCSI provides near line rate
NIC Driver
o
TCP/IP
f
iSCSI
HBA
o t
t: N
NIC driver NIC
HW iSCSI provide lower CPU
n
iSCSI initiator
iSCSI initiator
n te Utilization
o
0C
TCP/IP TCP/IP
NIC Recommend Jumbo Frames (MTU
2
(TCP Offload Engine) (TCP Offload Engine)
2 0 9000)*
l d
Host Host Host
r
wo
Standard NIC adapter Third party adapter Third party adapter
VM
depends on VMware offloads iSCSI, network
networking processing, and
management from host
f r
oSwitch=1500, vmk=9K
t
t: No
te n
C on vmkping -I vmkX x.x.x.x -d -s 1472
Same
l ic atiLoad balancing over multiple paths
pub
Subnet
t
bound ports to all targets
No
n t:
n te NIC Teaming
02 0 NIC/port
or Subnets
w
VM
Able to use separate gateway
per vmkernel
pu b
r
vmk1
n t:
e
vSwitch2
VMkernel adapters
o n t
C
Physical adapters
02 0
2
iSCSI2 vmnic2
vmk2
orl d
M w
V
©2020 VMware, Inc. 28
How do You Configure Multiple Adapters for iSCSI or iSER
Adapter Mapping on a Single vSphere Standard Switch.
i o n
r iut
b
Singled ist Config
r Switch
VMkernel adapters Physical adapters
n o
i o
vmnic1
t
iSCSI1
Add all NICs and VMkernel adapters
NIC
vmk1 vmnic2
l ic a
to single vSphere switch
Teaming
pu b
iSCSI2
o t
t: N
In vSphere 6.5 you can use single
OR n
VMkernel adapters
n te
Physical adapters switch if each iSCSI VMkernel is
02
vmk1 vmnic2
2
Port
Binding
orl d
w
iSCSI2
V M vmk2
t i on vmnic2
a
MGMT iSCSI1
vmk0 vmnic1
b l ic
vmk3 vmnic3
Physical adapters
r pu Physical adapters
t fo
o
vMotion iSCSI2
N
vmk1 vmk4
n t:
n te
vSAN
C o
02 0
vmk2
d 2
orl vMotion, vSAN, FT, etc.
w
Management, iSCSI only, Port Binding
VM
©2020 VMware, Inc. 30
i o n
r ibut
is t
Queuing or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 31
i o n
r ibut
“Little's Law: The long-term is t
average number of customers or d
t i on
in a stable system L is equal to
l ica
the long-term average pub
fo r
effective arrival rate, λ,
o t
multiplied by the average time t: N
a customer spends innthe te n
C o
system, W.” 0 0 2
2
VM
wo
r l d
L = λW
©2020 VMware, Inc. 32
What is a Queue Depth Limit?
A queue is a line, and a queue depth limit is how “wide” that line is.
i o n
r ut
ib
is t
One grocery clerk Two grocery clerks
o r d
t i o n
Can help one customer at a time
l ic a
Two customers can be helped at a time
• (queue depth limit of 1)
p u b
Neither has to wait (no added latency)
If there are two customers, one must wait
t for
for the first to finish (added latency)
No
n t:
n te
C o
02 0
d 2
w orl
V M
©2020 VMware, Inc. 33
Queue Limits
i o n
rut
ib
is t
r d
HBA o
Application
t i on Device Queue Depth
Application
l ica Limit
b
Latency Virtual Machine
Guest
t
Latency Limit
ESXi Kernel
N o
n t: vSCSI Adapter Queue
e
Driver
o n t Depth Limit
C
Device HBA
0
Latency
02
Virtual Disk Queue Depth
2
Fabric
Limit
orl d Array
M w
V
©2020 VMware, Inc. 34
Want Higher Performance?
Use PARAVIRTUAL SCSI ADAPTER
i o n
r iut
b
is t
Setting Value
o r d
Default Adapter Queue Depth 64
t i on
For high-performance workloads,
l ic a
Paravirtual SCSI adapter is best
fo r workloads
256 o t
Maximum Virtual Disk Queue Depth
t: N Higher default and maximum queue
e n depths
nt 1,024
Maximum Virtual Disk QueueoDepth
0 C VMware tools includes the drivers
2 0 2
o rl d
w
VMhttps://kb.vmware.com/kb/1010398
©2020 VMware, Inc. 35
HBA Device Queue
Depending on your HBA vendor the default value varies
i o n
r ut
ib
is t
Type
Default
Value Name r d
o which controls how
Value
t i on
This is an HBA setting
l ic a
many I/Os may be queued on a device
QLogic 64 qlfxmaxqdepth
u b
rp
Values are configured via esxcli
t f o
Brocade 32 bfa_lun_queue_depth
N o Changing requires a reboot of ESXi
n t:
e
https://kb.vmware.com/kb/1267
Emulex 32
n t
lpfc0_lun_queue_depth
o
C
0 fnic_max_qdepth
Cisco UCS 32
2 0 2
r l d
Software
wo 128
VM
iscsivmk_LunQDepth
iSCSI
n e
t (Scsi.ScsivVolPESNRO)
C o
2 0
l d 20
r
wo
VM
©2020 VMware, Inc. 37
Some Quick Math…
Let’s suppose the latency on average is .5 ms
i o n
r ibut
is t
1 second = 1,000 ms
r d
o These
t i on IOPS values are
a
1,000 ms / .5 ms per IO = 2,000 IOPS per PE (or per VMFS)
b l ic per host
VMFS Default: 2,000 IOPS * 32 max outstanding
r pu
I/Os = 64,000 IOPS
t f o
N o
t:
PE Default: 2,000 IOPS * 128 max outstanding
n
I/Os = 256,000 IOPS
n te
C o
0 2 0
l d 2
o r
w
VM
©2020 VMware, Inc. 38
Difference Between Storage DRS and Storage I/O Control
Latency on a VM or latency on a datastore
i o n
r ib ut
is t
or d
STORAGE I/O n
STORAGE DRS
a t i o CONTROL
b l ic
r pu
t f o
Storage DRS moves VMs around based on
No
hitting a latency threshold
n t:
n te
o
Storage I/O control throttles VMs based on
0 C
This is the VM observed latency (includes hitting a datastore latency threshold:
02
any latency induced by queueing)
u b
r pProtocol Endpoint (PE)
vVols are presented to ESXi via a
t f o
o
t: N
There is a misconception there is no more
Protocolte n
Endpoint
shared queue depth because virtual disks
0 C
20 2 • vVols share queue depth limit of their PEs
vVols r l d
wo
VM
©2020 VMware, Inc. 40
i o n
r ibut
is t
VMFS or d
t i on
l ica
pub
fo r
o t
t: N
te n
o n
0 C
2 02
orl d
M w
V
©2020 VMware, Inc. 41
Evolution of VMFS
i o n
r ib ut
is t
2011 2016 2018
or d
2020
t i on
l ic a
ub
vSpherer6.7p
vSphere 5.0 vSphere 6.5
t f o vSphere 7.0
t: No
VMFS-6: e
t n VMFS-6:
VMFS-5:
o n VMFS-6:
• VMFS used for boot device
C
• 64TB datastore • 512e & 4K sector native • VMFS-6 default FS
0
• - >= 2TB disk support • Auto unmap for Sesparse
2
• Enhanced Regional allocation (Affinity 2.0)
0
• LVM2 • Automatic unmap snapshot
2
• RDM, ATS locking • Support. • Configurable unmap rate • “Shared vmdk” support
•
orl d
performance and
scalability
• 512/2k lun/path support
• Affinitized Resource
• 1k/4k lun/path support
• VMFS3 EOL • XCOPY support for SCSI VPD page.
V
• VOMA fix mode for
VMFS-5
a
Resource Cluster to write new IO to space
b l ic
r pu Can add additional IO overhead with back
and forth operations
o
Affinity Resource
Manager Manager
o t f
t: N
te n
o n
0 C
2 02
or l d Storage
M w
V
©2020 VMware, Inc. 43
Affinity 2.0
More efficient first write IO, reducing “cost” of operation.
i o n
r ibut
is t
VMFS File Layer
o r d
t i onNew Affinity Manager
b l ic
u
Resource
Clusters
Allocation Logic
o r p
Manager
Existing RC New RC
o t f No more back and forth
N
Region Map trying to locate free RC
:
Pool
t
Pool
0C
Async Async
Async New
2
Region Existing
0
Map SFBC Pool
RC Pool
2
Mgmt. Mgmt.
Mgmt.
r l d
wo
VM Storage
o t
N
WSFC clusters 3 • Set Windows Cluster Parameter
n t: QuorumArbitrationTimeMax to 60
e
Clustered VMDKs per host 128
o n t
0 C • FC storage supporting SCSI-3 PR type WEAR
02
and ATS
d 2
orl
• Enable Clustered VMDK feature on Datastore
M w
V • Use EZT on shared VMDKs
t: N
How many VMs? nt e n Is there a performance limit to each
C o volume (queue depths)?
0 2 0
l d 2
r
wo
Are there array features you plan on
VM
using?
VM
©2020 VMware, Inc. 48
General Best Practices
Important things to remember
i o n
r ib ut
is t
Syslog/NTP or
UEFI d
t i on
l ic a
pu b
Important. The more integrated applications
fo r
UEFI is a recommended best practices for
are, the more important various logs become.
o t Windows VMs.
t: N
• This causes a 7 MB read that many arrays don’t
Having all logs SYNCED and STORED is
te n support
o n
extremely helpful (ESXi, vCenter, Array,
VASA).
0 C Be on VMware HW version 14 or modern
2 2
0 (Log Insight for releases of ESXi
o
example). Use l d
Deploy a syslog target
r NTP sources everywhere.
• HW 14 doesn't issue large read
• New versions of ESXi split large I/Os by looking
w
VM
at Maximum Transfer Length of device VPD
• Or set Disk.DiskMaxIOSize
t i on
l ic a
VCF and vVols: Empower Your External Storage [HCI2270]
p u b
f o r
vSphere Virtual Volumes (vVols): Modernize tYour
N o External Storage [ HCI1692]
n t:
te
vSphere Cloud Native Storagenwith
C o Virtual Volumes and SPBM: Better Together [HCI2089]
0 2 0
Virtual Volumes 2
o rl d and Site Recovery Manager Deep Dive with Dell EMC Storage [ HCI1451]
w
VMTogether: Site Recovery Manager with Virtual Volumes [HCI1455]
Better
n te
C o
Technical Deep Dive on Cloud Native Storage for vSphere [HCI2160]
0 2 0
l d 2
Storage Management - How to Reclaim Storage Today on vSAN, VMFS and vVols with
o r
John Nicholson [HCI2726]
w
VM
©2020 VMware, Inc. 52
i o n
r ibut
is t
or d
Please Fill Out t i on
l ica
Your Survey pub
fo r
o t
t: N
te n
o n
0 C
202
orl d
M w
V
i o n
r ibut
is t
or d
t i on
l ica
Thank you! pub
fo r
o t
t: N
te n
o n
0 C
202
orl d
M w
V