NTS-2001-I:

Nimble Introduction and Administration

Module 5:
Pre-installation, Racking
and Cabling

Course Revision 5
Module 05 Objectives

At the end of this module, you will be able to:

– Use the pre-installation workbook
– Understand rack mounting principles
– Cable an array
– Understand networking and fibre channel concepts
Pre-Installation Workbook
Pre-Install Checklist

– Collect all necessary data to perform an installation

– Organized in the same order that you will be entering in the data
– Can be left with the customer
Pre-Install Checklist
Pre-Install Checklist
Pre-Install Checklist
Pre-Install Checklist
Pre-Install Checklist
Racking
Use the Quick Start Guide
Download the Installation Guide and Quick Start Guide from InfoSight before starting
 Racking

AF = 4U CSx000 = 4U

AFS2 = 4U AFS = 4U

ES2 = 4U
Racking

Precaution
– Ensure there is at least 24” of cable slack at the back of
the array or shelf.
– Allow enough room to slide the array or expansion shelf out
12” from the front of the rack in order to replace a center
chassis fan.
Cabling
Power

Connect one to
commercial power and
one to backup power.

Controller A Controller B
Networking Terminology

–Interface Pairs
–Controller A eth1 & Controller B eth1
–IP addresses float between

Interface Pairs

Controller A Controller B
Ethernet Ports
eth5 eth6

eth1 eth2 eth3 eth4 SAS Out Ports

When cabling to switches ensure you cable even

ports to one switch and odd ports to the other.
Fibre Channel Ports

fc5 fc9 fc10 fc6

eth1 eth2 fc1 fc2 SAS Out Ports

Networking Concepts
Understanding IPs
–Management Network
–The Array Management IP address
–The 2 Controller Diagnostic IP addresses

–Data Network
–The Data IP addresses

–Discovery Addresses
–iSCSI Discovery Address

–Diagnostic Addresses
Discovery Addresses
iSCSI
Send Targets
–Typical iSCSI Discovery Discovery

– Returns a list of Targets from the Array

– Makes it easy to find Volumes from the Host

Targets
Volume1
Volume2
Volume3

–Nimble iSCSI Discovery (per Subnet)

– Acts as a load balancer (Automatic Mode) Connect to Data IPs
– Redirects to a Data IP address to be used to connect to the array Discovery
.20 .21
(172.16.0.20) .22
– Used to virtualize connections to each array in the group
2 times
– Used by the array to distribute/balance hosts across several data IPs
– Need a Discovery for each Data Subnet
– Discovery cannot redirect across subnet boundaries Data
Addresses
Redirects & Returns 172.16.0.21
172.16.0.22
from Array:
172.16.0.21 & 17.16.0.22
 Typical Subnets
One Shared Network Two Dedicated Networks Advanced Networks
Management Subnet Management Subnet
• Array Management Address • Array Management Address
• Acts as Discovery IP • Acts as Discovery IP
• 2 Controller Diag Addresses • 2 Controller Diag Addresses
Management • Replication (by default, 2.x) • Replication (by default, 2.x)
• Array Management Address
• Acts as Discovery IP
• 2 Controller Diag Addresses
• Replication (by default, 2.x)

Data Subnet Data Subnet Data Subnet

Data • All Data Addresses • 1 Set (Data) • 1 Set (Data)
• All Data Addresses • Discovery IP • 1 Discovery IP • 1 Discovery IP

Note: in 2.x, every array in the group must have access to each subnet
iSCSI Switch Selection Guidelines
Network Design
–Best if dedicated, redundant iSCSI network
–Otherwise, use VLANs to keep iSCSI traffic separate

Switch Attributes
– Good Quality Layer 2 or Layer 3 Managed
– Stacked preferred, but be aware of issues around stacking such as what happens when the master switch in
the stack fails and upgrading of switch firmware may require an outage
– ISLs at least, concern is under or over specifying the total bandwidth required
– ISLs – may want to use IP Address Zones to prevent iSCSI traffic from crossing the ISL
– Support for Jumbo Frames (with Flow Control) is Desirable
– Non-Blocking Backplane
– Bandwidth of the backplane >= (# of ports) * (bi-directional port speed)
 Optimizing Switch Links: IP Address Zones

–Challenge
–How to minimize Inter-Switch Links for traffic?
172.16.0.81 172.16.0.82
–Especially if connections are automatic? eth1
Discovery redirects
eth2
Discovery redirects
to 172.16.0.21 to 172.16.0.22

Single Data Subnet

– Answer: IP Address Zones
– Discovers Data Address via Discovery IP
– Uses Host IP address to select Data Address
Avoid
– Examples: Traffic?
– Even/ Odd: Odd vs Even #s kept together
– Bisect: Low vs High #s are kept together 172.16.0.21 172.16.0.22
(Data IP) (Data IP)
– Default: Single (no division)

Using Odd/Even zone .21 Management

.22
Subnet
172.16.0.20 .20
(Discovery)
Best Practices
Practice Notes
Do not use Spanning Tree Do not use STP on switch ports that connect to iSCSI initiators or
Protocol (STP) or use PortFast the Nimble storage array network interfaces.
Configure flow control on each Configure Flow Control on each switch port that handles iSCSI
switch port connections. If your application server is using a software iSCSI
initiator and NIC combination to handle iSCSI traffic, you must also
enable Flow Control on the NICs to obtain the performance benefit.
Disable unicast storm control Disable unicast storm control on each switch that handles iSCSI
traffic. However, the use of broadcast and multicast storm control is
encouraged.
Use jumbo frames when You must have jumbo frames enabled from end to end for them to
applicable work correctly
Testing Network connectivity Use the Ping command to test network connectivity and to help
determine if jumbo frames are enabled across the network,
example: vmkping -d -s 8972 x.x.x.x
Fibre Channel Concepts
Fibre Channel Components
Array (Target)
Dual Fabric Design

Configure one FC port on

controller A to FC Switch A and
one FC port on Controller A to FC
Switch B
2 3 4 2 3 4

Configure one FC port on CISCO UCS

ID

STAT
6296 UP

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
CISCO UCS

ID

STAT
6296 UP

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

controller B to FC Switch B and FC Switch A FC Switch B

one FC port on Controller B to FC
Switch A

S
A
S
S
A

PCIe3
PCIe4

PCIe1
CIMC

PORT 1

PORT 0
S

S
A

PCIe5

PCIe 2
PSU
2

All FC ports must be in the same 650 W AC

1 2

PSU
1

zone

M

Server (Initiator)
Fibre Channel Config Steps

1. Verify Fabric/Zoning
– Single Initiator
– Single Target
2. Configure Initiator Group
3. Create volume and link to the initiator group
4. Verify connection from volume to assigned initiators
Thank you
Contact information

29