0% found this document useful (0 votes)

284 views

DCMDS20SG Vol1

Uploaded by

Kv142 Kv

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

284 views

DCMDS20SG Vol1

Uploaded by

Kv142 Kv

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 454

DCMDS

Configuring Cisco
MDS 9000 Series
Switches
Volume 1
Version 2.0

Student Guide

Text Part Number: 97-3312-01

Americas Headquarters Asia Pacific Headquarters Europe Headquarters
Cisco Systems, Inc. Cisco Systems (USA) Pte. Ltd. Cisco Systems International BV Amsterdam,
San Jose, CA Singapore The Netherlands
Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco Website at www.cisco.com/go/offices.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this
URL: www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a
partnership relationship between Cisco and any other company. (1110R)

DISCLAIMER WARRANTY: THIS CONTENT IS BEING PROVIDED “AS IS” AND AS SUCH MAY INCLUDE TYPOGRAPHICAL,
GRAPHICS, OR FORMATTING ERRORS. CISCO MAKES AND YOU RECEIVE NO WARRANTIES IN CONNECTION WITH THE
CONTENT PROVIDED HEREUNDER, EXPRESS, IMPLIED, STATUTORY OR IN ANY OTHER PROVISION OF THIS CONTENT
OR COMMUNICATION BETWEEN CISCO AND YOU. CISCO SPECIFICALLY DISCLAIMS ALL IMPLIED WARRANTIES,
INCLUDING WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT AND FITNESS FOR A PARTICULAR PURPOSE,
OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE. This learning product may contain early release
content, and while Cisco believes it to be accurate, it falls subject to the disclaimer above.

Student Guide © 2014 Cisco and/or its affiliates. All rights reserved.
Students, this letter describes important
course evaluation access information!

Welcome to Cisco Systems Learning. Through the Cisco Learning Partner Program,
Cisco Systems is committed to bringing you the highest-quality training in the industry.
Cisco learning products are designed to advance your professional goals and give you
the expertise you need to build and maintain strategic networks.

Cisco relies on customer feedback to guide business decisions; therefore, your valuable
input will help shape future Cisco course curricula, products, and training offerings.
We would appreciate a few minutes of your time to complete a brief Cisco online
course evaluation of your instructor and the course materials in this student kit. On the
final day of class, your instructor will provide you with a URL directing you to a short
post-course evaluation. If there is no Internet access in the classroom, please complete
the evaluation within the next 48 hours or as soon as you can access the web.

On behalf of Cisco, thank you for choosing Cisco Learning Partners for your
Internet technology training.

Sincerely,

Cisco Systems Learning

Table of Contents
Volume 1
Overview 1
Learner Skills and Knowledge 2
Course Goal and Objectives 3
Course Flow 4
Additional References 5
Cisco Glossary of Terms 6
Introducing Cisco MDS 9000 Series Switches 1-3
Overview 1-3
Objectives 1-3
Cisco MDS 9000 Series Platform 1-4
Cisco MDS 9700 Series Director-Class Switches and Components 1-5
Cisco MDS 9500 Series Director-Class Switches and Components 1-20
Cisco MDS 9200 Series Multiservice Switch and Service Modules 1-31
Cisco MDS 9100 Series Fabric Switches 1-42
Summary 1-48
Implementing Integrated Management 1-51
Overview 1-51
Objectives 1-51
Cisco NX-OS 1-52
Cisco NX-OS CLI Command Set and Structure 1-56
Cisco Prime DCNM 1-65
Cisco Prime DCNM Components 1-73
Licensing Cisco Prime DCNM 1-78
Cisco MDS Device Manager 1-81
Cisco Prime DCNM Federated Server 1-83
Cisco Prime DCNM VM Awareness 1-87
Summary 1-89
Module Summary 1-91
Overview 2-1
Module Objectives 2-1
Performing the Initial Switch Configuration 2-3
Overview 2-3
Objectives 2-3
Switch Boot Sequence 2-4
Completing the Initial Setup Routine 2-6
CLI Show Commands 2-12
Lab Environment 2-17
Summary 2-18
Installing and Licensing Cisco NX-OS Software 2-19
Overview 2-19
Objectives 2-19
Software Licensing 2-20
Cisco NX-OS Software Installation and Upgrade 2-37
Version Downgrade Procedure 2-43
Summary 2-45
Module Summary 2-47
Module Self-Check 2-49
Overview 3-1
Module Objectives 3-1
Using FLOGI and FCNS Databases 3-3
Overview 3-3
Objectives 3-3
Fabric Login Sequence 3-4
Device Registration 3-7
FCID Persistence 3-10
Summary 3-15
Configuring Interfaces 3-17
Overview 3-17
Objectives 3-17
Configuring Fibre Channel Interfaces 3-18
Configuring Bandwidth 3-31
Slow Drain Detection 3-36
Explain Interface BB_Credits 3-40
Configuring Trunking 3-45
Summary 3-48
References 3-48
Configuring Port Channels 3-49
Overview 3-49
Objectives 3-49
Port Channel Overview 3-50
Port Channel Configuration for E and TE Ports 3-54
Summary 3-63
Configuring Cisco NPV and NPIV 3-65
Overview 3-65
Objectives 3-65
Cisco NPV 3-66
N-Port ID Virtualization 3-71
F and TF Port Channel Configurations 3-73
Summary 3-76
Configuring VSANs 3-77
Overview 3-77
Objectives 3-77
VSAN Overview 3-78
Creating VSANs 3-86
Displaying VSAN Information 3-92
VSAN Recommended Practices 3-94
Summary 3-95
Managing Domains 3-97
Overview 3-97
Objectives 3-97
Domain Configuration Process 3-98
Configuring the Principal Switch Priority 3-107
Configuring the Domain ID 3-108
Displaying the Domain Database 3-112
Merging Fabrics 3-115
Configuring Fabric Merge Options 3-121
Summary 3-123

ii IP Telephony, Part 1 (IPT1) v9.0 © 2013 Cisco Systems, Inc.

Configuring Distributed Device Aliases 3-125
Overview 3-125
Objectives 3-125
Distributed Device Alias Overview 3-126
Existing Zone Alias Overview 3-129
Distributed Device Alias Database Configuration 3-131
Configuring Device Aliases in a Multiswitch Fabric 3-133
Distributed Device Alias Database Verification 3-136
Summary 3-138
Implementing Zoning 3-139
Overview 3-139
Objectives 3-139
Zoning Overview 3-140
Configuring Zones and Zone Sets 3-146
Zone Configuration Verification 3-150
Configuring Zone Set Distribution 3-154
Merging Zones Without Disruption 3-157
Recovering from Zone Merge Failures 3-160
Managing Zone Sets 3-165
Enhanced Zoning 3-172
Modifying the Enhanced Zone Database 3-177
Smart Zoning 3-180
Recommended Zoning Practices 3-187
Summary 3-188
References 3-189
Module Summary 3-191
Module Self-Check 3-193
Overview 4-1
Module Objectives 4-1
Implementing Cisco MDS Data Mobility Manager 4-3
Overview 4-3
Objectives 4-3
Cisco DMM Overview 4-4
Cisco DMM SAN Topologies 4-16
Cisco DMM Software installation and Configuration 4-24
Use of the Cisco DMM GUI for Data Migration 4-28
Summary 4-32
Monitoring Traffic Flow 4-33
Overview 4-33
Objectives 4-33
SPAN Overview 4-34
RSPAN Overview 4-40
Cisco Fabric Analyzer 4-41
Wireshark Overview 4-44
Summary 4-48
Module Summary 4-49
Module Self-Check 4-51
Overview 5-1
Module Objectives 5-1

 2013 Cisco Systems, Inc. IP Telephony, Part 1 (IPT1) v9.0 iii

Describing FCoE 5-3
Overview 5-3
Objectives 5-3
I/O Consolidation 5-4
Ethernet Enhancements 5-8
FCoE Protocol 5-14
FCoE Addressing 5-18
FCoE Initialization Protocol 5-20
Summary 5-25
Configuring FCoE on Cisco MDS 9500 and 9700 Series Multilayer Directors 5-27
Overview 5-27
Objectives 5-27
Fibre Channel over Ethernet 5-28
VE Interfaces 5-32
FCoE Configuration on the MDS 9500 and 9700 Series Multilayer Directors 5-34
Summary 5-39
Module Summary 5-41
Module Self-Check 5-43

iv IP Telephony, Part 1 (IPT1) v9.0 © 2013 Cisco Systems, Inc.

DCMDS

Course Introduction
Overview
Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 is an instructor-led course
presented by Cisco Learning Partners to their end-user customers. This course is a five-day
product training course. This comprehensive hands-on experience familiarizes data center
systems engineers, field engineers, architects, and Cisco partners who implement storage
networking solutions with the Cisco MDS 9000 Series switch platform.
This course covers features on each of the MDS product family of switches including the Cisco
MDS 9100, 9200, 9500, and 9700 Series models and the Fibre Channel, Fibre Channel over
Ethernet (FCoE), and service modules that are supported. Fundamental topics covered by the
course include 8-Gb Fibre Channel, 16-Gb Fibre Channel, Fibre Channel over IP (FCIP),
Internet Small Computer Systems Interface (iSCSI), multihop FCoE, upgrading the Cisco
Nexus Operating System (NX-OS), and the common management tool Cisco Prime Data
Center Network Manager (DCNM).
The course provides comprehensive SAN configuration for features such as interface
configuration, Cisco N-Port Virtualizer (NPV), N-Port ID Virtualizer (NPIV), virtual storage
area network (VSAN) and domain setup, SAN zoning, and SAN extension using FCIP and
Inter-VSAN Routing (IVR).
Topics introduced also include centralized SAN services using the Cisco MDS 9222i
Multiservice Modular Switch and Cisco MDS 9250i Multiservice Fabric Switch for Cisco
MDS 9000 Input/Output Accelerator (IOA) and Data Mobility Manager, management security,
and role-based access control (RBAC).
The goal of this course is to help you understand how you can apply these new technologies to
optimize the scalability, agility, performance, and operational efficiency of your SAN.
Learner Skills and Knowledge
This subtopic lists the skills and knowledge that you must possess to benefit fully from the
course.

• Develop a basic understanding of data storage hardware components

and protocols, including Small Computer Systems Interface (SCSI) and
Fiber Channel
• Understand networking protocols, including Ethernet and IP

2 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Course Goal and Objectives
This topic describes the course goal and objectives.

To install, configure, and

manage the Cisco MDS
9000 Series Switch
platform in a scalable,
highly available
environment

Upon completing this course, you will be able to meet these objectives:
 Identify the components, services, and features of the MDS 9000 Series switch platform
that can be used to improve the availability, scalability, performance, and manageability of
the SAN
 Describe how to install and configure the MDS 9000 Series switch and perform the initial
software configuration process
 Explain how to implement the logical topology that is specified by a SAN design, so that
connectivity between end devices can be verified
 Describe the traffic management features associated with intelligent network services for
the MDS 9000 Series switches in order to configure basic traffic management services for
the SAN
 Describe how to configure FCoE modules on the Cisco MDS 9500 Series Switch
 Explain how to implement security so that management access is secure and that only
trusted devices are allowed to connect to the fabric
 Describe how to use FCIP to implement appropriate solutions for SAN extension

© 2013 Cisco Systems, Inc. Course Introduction 3

Course Flow
This topic presents the suggested flow of the course materials.

Day 1 Day 2 Day 3 Day 4 Day 5

Course
Introduction
AM
Cisco MDS Building a SAN Intelligent SAN Security FCIP
9000 Series Fabric Fabric Services Implementation Implementation
Switch
Platforms
Lunch
PM System
Installation and Building a SAN FcoE Security FCIP
Initial Fabric Implementation Implementation Implementation
Configuration

The schedule reflects the recommended structure for this course. This structure allows enough
time for the instructor to present the course information and for you to work through the lab
activities. The exact timing of the subject materials and labs depends on the pace of your
specific class.

4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Additional References
This topic presents icons and symbols that are used in this course, as well as information on
where to find additional technical references.

Cisco MDS 9500 or 9700

Multilayer Director Cisco Catalyst 6500
Series Switch
Cisco MDS 9200
Multilayer Switch
Layer 2 Ethernet Switch
Cisco MDS 9100
Fabric Switch

Router
Cisco Nexus 5000
Series Switch
Firewall

Cisco Nexus 7000

Series Switch

Fibre Channel JBOD

Application Server

RAID
PC Subsystem

Laptop Tape
Subsystem

Blade Server Chassis

LUN or Disk

© 2013 Cisco Systems, Inc. Course Introduction 5

Cisco Glossary of Terms
For additional information on Cisco terminology, refer to the Cisco Internetworking Terms and
Acronyms glossary of terms at
http://docwiki.cisco.com/wiki/Internetworking_Terms_and_Acronyms_%28ITA%29_Guide.

6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 1

Introducing Cisco MDS 9000

Series Switches
Overview
In this lesson, you will learn to describe the Cisco MDS 9000 Series switch platform including
the Cisco MDS 9100, 9200, 9500, and 9700 Series models along with various modules that run
at 1-, 2-, 4-, 8-, and 16-Gbps and FCoE modules.
 Cisco MDS 9000 Series Multilayer Switches are high-performance Fibre Channel switches.
These switches support the following:
— Native 1-, 2-, 4-, 8-, and 16-Gbps Fibre Channel ports
— Gigabit Ethernet ports with support for Internet Small Computer Systems Interface
(iSCSI)
— Fibre Channel over IP (FCIP)
— 10 Gigabit Ethernet ports with support for Fibre Channel over Ethernet (FCoE)
This lesson is an overview of the MDS 9000 Series fabric switches and the line card modules
that provide device connectivity.

Objectives
Upon completing this lesson, you will be able to describe the MDS 9000 Series switch
platform. This review includes the MDS 9100, 9200, 9500, and 9700 models along with
various modules that run at 1-, 2-, 4-, 8-, and 16-Gbps and FCoE modules. This ability includes
being able to meet these objectives:
 Identify components of the MDS 9000 Series platform, including the MDS 9100, 9200,
9500, and 9700 models running 1-, 2-, 4-, 8-, and 16-Gbps Fibre Channel and FCoE
modules
 Identify the Cisco MDS 9700 Series director-class switches and components
 Describe the Cisco MDS 9500 Series director-class switches and components
 Describe the Cisco MDS 9200 Series Multiservice Switch and service modules
 Describe the Cisco MDS 9100 Series Fabric Switches
Cisco MDS 9000 Series Platform
This topic describes the components of the MDS 9000 Series platform. The topic includes
descriptions of the MDS 9100, 9200, 9500, and MDS 9700 Series models running 1-, 2-, 4-, 8-,
and 16-Gbps Fibre Channel and FCoE modules.

Multilayer and Multiservice Multilayer Directors

Fabric Switches

MDS 9222i MDS 9250i

MDS 9148 MDS 9506, 9513 MDS 9710

48-Port 16-Gb
Fibre Channel Module
32-port 1/2/4/8/10-Gbps 18/4 MSM 4 Gb 8-Port FCoE
Advanced Fibre Channel Module MDS 9513

Sup-1 MDS 9710

48-Port
48-port 1/2/4/8/10-Gbps Supervisor-2A 10GE FCoE Module
Advanced Fibre Channel Module
SSN-16 Available
MDS 9506, 9513
1HCY14

Multilayer switches are switching platforms with multiple layers of intelligent features, which
include the following:
 High availability
 High performance
 Scalable architecture
 Comprehensive security features
 Ease of management
 Advanced diagnostics and troubleshooting capabilities
 Seamless integration of multiple technologies
 Multiprotocol support

The Cisco MDS 9000 Series offers industry-leading investment protection across a
comprehensive product line, featuring a scalable architecture with highly available hardware
and software. Based on the Cisco Nexus Operating System (NX-OS) and a comprehensive
management platform in Cisco Prime Data Center Network Manager (DCNM) for SAN
(DCNM-SAN Client), the MDS 9000 Series offers various application line card modules and a
scalable architecture from an entry-level fabric switch to director-class systems.

1-4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco MDS 9700 Series Director-Class Switches
and Components
This topic describes the Cisco MDS 9700 Series director-class switches and components.

• The MDS 9710 Multilayer

Director is designed for the
following:
- Scalability
• More line-rate 16-Gb Fibre
Channel ports
14 RU
- Highest availability
24.35"
• N+1 fabric module protection*
provides zero impact to
application bandwidth if a fabric
card or supervisor card fails
- 1.536 Tbps per slot for Fibre MDS 9710 Director
Channel • 10 chassis slots
• 2 half-width supervisors
• 24 Tbps per-chassis capacity
• 8 line cards
• Up to 6 fabric cards
• Fibre Channel and FCoE
17.3" ports
* with an optional fourth fabric module installed
**Note that the FCoE module will be available 1HCY14 34" Deep • 384 line-rate 16-Gb ports
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-6

The Cisco MDS 9710 Multilayer Director provides high performance and is designed with
room for growth.

Performance with Investment Protection

Three fabric cards are required to support the 384 line rate 16-Gb Fibre Channel and FCoE
ports that the platform supports. Because there are slots for six fabric cards, the platform can
ultimately support 1.536 Tbps per slot, for 24 Tbps of total switching capacity. This feature
enables the switch to adapt to support future bandwidth requirements, like 32-Gb Fibre Channel
at full line rate. Importantly, because all Cisco platforms for the data center run on Cisco NX-
OS Software and are managed by Cisco Prime DCNM, customers who choose to migrate to the
MDS 9710 Multilayer Director will be able to use similar processes and procedures, without
retraining staff.

Resiliency
N:N fabric redundancy design eliminates loss of bandwidth even if a fabric card fails.
Competitive switches lose 50 percent of bandwidth if a fabric card fails. The switch also
enables N:N grid redundancy. You can connect to two separate power grids to keep operating
even if a power grid fails. This switch supports hitless and nondisruptive In-Service Software
Upgrades (ISSUs) to keep the platform fully operational, even while performing updates to
Cisco NX-OS.

Multiprotocol Flexibility
The MDS 9710 Multilayer Director can support both Fibre Channel and FCoE, simultaneously,
in the same chassis. This ability allows customers the flexibility to mix and match, as their

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-5
requirements change. For example, the 48-port 16-Gb line card supports 2-, 4-, 8-, 10-, and 16-
Gb optics.

1-6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Standard Maximum
MDS 9710 Configuration Configuration
Line Cards 8

Supervisor-1 Modules 2

Fabric Modules 3 6

Power Supplies 6 8
14 RU
Fan Trays 3

Chassis Height 24.35" (14 RU)

Chassis Width 17.3"

Chassis Depth 34"

17.3"
Line-rate 16G Ports per
384
System

Airflow Front to Back

The MDS 9710 Multilayer Director chassis has built-in redundancy to make sure that
performance and throughput are not impacted by any failure. This ability is true provided you
have fully loaded chassis, N:N fabric module redundancy, N:N and N+1 grid redundancy for
power supplies, N:N supervisor modules, a redundant control path, a data path for fan trays,
and so on. Additionally, all hardware components use LED indicators that are easily accessible
and seen. You do not need to remove the cover or unscrew hardware to see LEDs.

Fan Tray 2
Fan Tray 3
Fan Tray 1
Line Card Modules 1–4

Supervisor
Module 5

Supervisor
Module 6

Chassis Handles Fabric Modules 1-6 (Left to Right)

Located Behind Fan Trays

Line Card
Modules 7–10 Power
Modules

The figure describes the MDS 9710 Multilayer Director chassis.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-7
• New half-width form factor
• 10/100/1000 management port, RJ45 console port, 2 x USB 2.0 ports

Feature Description
Number of Cores 4
Clock Speed 2.1 GHz
Instruction 64 bit
Memory 8 Gb
USB Ports 2

The figure describes the Cisco MDS 9710 Multilayer Director Supervisor-1 Module.
The MDS 9710 Multilayer Director supports a supervisor module that is designed specifically
for the Cisco MDS 9700 Series. This supervisor module provides control and management
functions for the switch and enables high-performance switching.
The supervisor module for the MDS 9710 Multilayer Director supports the following features:
 Nondisruptive software upgrades
 Stateful process restart and failover
 Fully redundant operation
 Support for up to 384 Fibre Channel ports in a single chassis and 1152 Fibre Channel ports
in a single rack
 Support for up to 24 Tbps of Fibre Channel system bandwidth
 Multipathing based on Fabric Shortest Path First (FSPF)
 Ability to dynamically reroute traffic in the event of a switch failure
 Network management through the CLI and through Cisco DCNM
 Extensive security features including RADIUS and TACACS+, Fibre Channel Security
Protocol (FC-SP), Secure FTP (SFTP), Secure Shell (SSH) Protocol, and Simple Network
Management Protocol Version 3 (SNMPv3) implementing Advanced Encryption Standard
(AES), hardware-enforced zoning and per-VSAN role-based access control (RBAC)
 Support for virtual SAN (VSAN) technology and Inter-VSAN Routing (IVR)
 Network services such as access control lists (ACLs) and quality of service (QoS)
 Smart zoning
 Power-on self-test (POST) and diagnostics
 Switched Port Analyzer (SPAN) and Remote Switched Port Analyzer (RSPAN)

1-8 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• When talking about per-port, slot, or fabric module speeds, you need to
define which speed you are referring to. Are you referring to clocking
rate, encoded rate, or actual data throughput?
• Encoded rates, Fibre Channel or Ethernet, are a function of interface
clock speed and data encoding.
• Data throughput takes into account protocol specific, additional
overhead such as idles and headers in Fibre Channel leaving just the
payload to count.
Data
Clocking Encoding Encoded Rate Throughput
Protocol
(Gbps) Type
Gbps MBps MBps*
8-Gb Fibre
8.500 8b/10b 6.8 850 788
Channel
16-Gb Fibre
14.025 64b/66b 13.6 1700 1581
Channel
* MB/sec here means mega-bytes/second
(1,048,576 bytes/second)
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-10

Link rate differs from actual data rate or throughput as follows:

 1-, 2-, 4-, and 8-Gbps Fibre Channel encoding adds 25 percent overhead. In other words,
for every 8 bits of data, 10 bits are actually transmitted due to the 8B/10B encoding type.
 16 Gbps Fibre Channel encoding adds 3.125 percent overhead. In other words, for every 64
bits of data, 66 bits are actually transmitted due to the 64B/66B encoding type.

Since Fibre Channel data rates double every time, the speed (clock rate) of 16 Gbps was
reduced due to the lower overhead.
If you do the math, the derived data throughput for 8-Gb Fibre Channel is 8.5 Gbps of clocking
speed with 8b/10b encoding scheme, approximately 25 percent overhead = 8.5 –(8.5* 25%)
=6.8 Gbps or 850 MBps.
Thus the derived data throughput for 16-Gb Fibre Channel is 14.025 Gbps of clocking speed
with 64/66b encoding scheme, approximately 3.125 percent overhead = 14.025 – (14.025
*3.125% )=13.587 Gbps or 1700 MBps.
The math that is involved in calculating the encoded data rate of Fibre Channel involves two
variables, which are clocking speed and encoding schema. Since 16-Gb Fibre Channel uses
64b/66b encoding, there is an approximate overhead of approximately 3 percent. For 16Gb
Fiber Channel, a 3 percent deduction leaves 13.6 Gbps.
Throughput takes into account protocol specific, additional overhead such as idles and headers
in Fibre Channel leaving just the payload to count.
8-Gb Fiber Channel = 788MB/s Throughput (Payload).

Note Since MB/sec here means (1,048,576 bytes/second), 788MB/sec actually equals 826
millions of bytes /sec and 1581MB/sec actually equals 1581 millions of bytes/sec.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-9
• Ethernet interfaces are different because apparent bandwidth equals
actual data bandwidth.

Data
Clocking Encoding Encoded Rate
Protocol Throughput
(Gbps) type
Gbps MBps MB/sec*
8-Gb FC 8.500 8b/10b 6.8 850 788
10-Gb FC 10.51875 64b/66b 9.7 1250 1191
10 GE FCoE 10.3125 64b/66b 9.7 1250 1146
16-Gb FC 14.025 64b/66b 13.6 1600 1581
32-Gb FC 28.050 64b/66b 27.2 3200 3162
40 GE FCoE 41.250 64b/66b 40.0 4800 4584

* MB/sec here means megabytes per

second (1,048,576 bytes per second)

Fibre Channel Protocol (FCP) is a high-speed SAN protocol that can support standard speeds
like 1-, 2-, 4-, 8-, and 16-Gbps. FCP can also support 10-Gbps speed. Consider the difference
between 8-Gbps and 10-Gbps Fibre Channel interfaces and the benefits of choosing one over
the other. One benefit of having a 10-Gbps interface is more bandwidth. The 10-Gbps Fibre
Channel interface provides 50 percent more throughput when compared to 8-Gbps Fibre
Channel interfaces. The reason is the clocking speed and encoding schema that is defined for
10-Gbps. As per the 8-Gbps Fibre Channel standard, the schema uses 8.5-Gbps of clocking
speed with an 8b/10b encoding scheme (approximately 25 percent overhead). The 10-Gbps
Fibre Channel standard uses 10.51875 Gbps of clocking speed with 64b/66b encoding scheme
(approximate overhead of 3 percent). Due to this formulation, the actual data rate of 8-Gbps
Fibre Channel interface will provide 6.8-Gbps of actual throughput when compared to 10-Gbps
Fibre Channel interface, which provides 10.2-Gbps.
Consider this statement from Dave Alexander’s blog
(http://www.unifiedcomputingblog.com/2011/01/31/update-on-the-8gb-fc-vs-10gb-fcoe-
discussion/): “...think of how hard drive manufacturers prefer to define a megabyte (1 million
bytes) versus how the rest of the world define[d] a megabyte (2^20 bytes or 1,048,576 bytes).”

1-10 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Each fabric module provides
256 Gbps of Fibre Channel
front-end bandwidth.
• Three fabric modules provide
768 Gbps of Fibre Channel
front-end bandwidth.
• With six fabric modules, the
maximum per-slot bandwidth is
1.536 Tbps Fibre Channel front-
end bandwidth. (2x768 Gbps)

The MDS 9710 Multilayer Director supports up to six crossbar fabric modules. The crossbar
fabric module (DS-X9710-FAB1) is designed specifically for the MDS 9710 Multilayer
Director. The crossbar fabric modules are installed vertically in slots 21 through 26, numbered
from left to right, at the back of the chassis behind the fan trays.
A minimum of three crossbar fabric modules are required to deliver full line rate and bandwidth
for the switch. A fourth crossbar fabric module is required for N+1 protection.
Each crossbar fabric module connects to eight switching modules and two supervisor modules.
In addition, each crossbar fabric module supports four 55-Gbps fabric ports that are connected
to each switching module and one 55-Gbps fabric port that is connected to each supervisor
module.
For more information, visit
http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/hw/9710/overview.html.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-11
• The line rate on a 48-port 16-Gb Fibre Channel module needs only three
fabric cards.
1
Host Ports

2
48-Port 16-Gb
Fibre Channel Module
3

Number Front Panel Fibre FCoE Front Panel Full Line N+1 Fabric
of Fabric Channel Bandwidth Fabric Bandwidth Rate Card
Cards per Slot per Slot Protection 4
1 256 Gbps 220 Gbps No No

2 512 Gbps 440 Gbps No No

3 768 Gbps 660 Gbps Yes No

4 768 Gbps 880 Gbps Yes Yes

The figure describes fabric module bandwidth scalability.

Host Ports
2

Fibre Channel,
FCoE Modules 3

Number of Apparent Fibre Actual Fibre FCoE Front Panel

Fabric Channel Channel Data and Fabric 4
Cards Bandwidth per Slot Rate per Slot Bandwidth per Slot
1 256 Gbps 217.6 Gbps 220 Gbps

2 512 Gbps 435.2 Gbps 440 Gbps 5

3 768 Gbps 652.8 Gbps 660 Gbps

4 1024 Gbps 870.4 Gbps 880 Gbps

6
5 1280 Gbps 1088.0 Gbps 1100 Gbps

6 1536 Gbps 1305.6 Gbps 1320 Gbps

The figure describes how architecture is designed for growth.

1-12 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Fabric modules are located
behind the fan trays.
• Fabric modules are numbered 1
to 6 from left to right when facing 1 2 3 4 5 6
the rear of the chassis.
• When the system is running,
only remove one fan tray at a
time to access the appropriate
fabric modules:
- Fan tray 1: Fabric modules 1–2
- Fan tray 2: Fabric modules 3–4
- Fan tray 3: Fabric modules 5–6
• Fabric modules may be installed
in any slot. A best practice is to
install one module behind each
fan tray. Fabric Module LED is
visible thru Fan Tray.

The figure describes MDS 9710 Multilayer Director fabric module detail.

• MDS 9710 has front-to-

Hot Air
back airflow. Back
Out
• Air enters through
perforations in line
cards, supervisor
modules, and power Front
supplies.
• Air exits through
perforations in fan
trays.
• Blank panels must be
installed in an empty
line card or in PSU
slots to ensure proper
airflow.
• The minimum airflow Cold Air In
clearance requirement
is 7 inches each side in
the front and back.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-16

The figure discusses MDS 9710 Multilayer Director airflow planning.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-13
Cisco 16-Gbps Line Card
Performance 48 x 16-Gbps ports, 768 Gbps Fibre Channel
Port Speeds 2-, 4-, 8-, 10-, and 16-Gbps Fibre Channel
Optics (SFP+) 2-, 4-, and 8-Gb Fibre Channel, 4-, 8-, and 16-Gb Fibre
Channel, 10-Gb Fibre Channel, 10 Gigabit Ethernet (Fibre
Channel with a 10 Gigabit Ethernet clock)
Port Types Fabric (F) port, fabric loop (FL) port, expansion (E) port,
trunking expansion (TE) port, SPAN destination (SD) port,
SPAN tunnel (ST) port
Port Groups Twelve 4-port port groups
Intelligent Capabilities VSAN, IVR, Fibre Channel Redirect
Buffer-to-Buffer Credits Up to 500 per port, 4095 with an Enterprise License (510
km at 16 Gb)

The figure describes the MDS 9710 Multilayer Director 48-port 16-Gb Fibre Channel line card.

1-14 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 2-, 4-, 8-, 10-, and 16-Gbps Fibre Channel module for MDS 9710
• 4-port port groups
• Up to 384 line-rate 16-Gbps Fibre Channel ports per chassis

MDS 9710 Multilayer Director 48-Port 16-Gbps Fibre Channel Switching Module: DS-
X9448-768K9
The MDS 9710 Multilayer Director 48-port 16-Gbps Fibre Channel switching module is
designed especially for the MDS 9700 Series. The module delivers line rate 16-Gbps Fibre
Channel performance to enable scalability in virtualized data centers. The MDS 9710
Multilayer Director can have up to 384 line-rate 16-Gbps Fibre Channel ports per chassis. The
MDS 9710 Multilayer Director supports up to eight MDS 9710 Multilayer Director 48-port 16-
Gbps Fibre Channel switching modules. These modules are hot-swappable and compatible with
2-, 4-, 8-, 10-, and 16-Gbps interfaces. The modules support hot-swappable enhanced small
form-factor pluggable (SFP+) transceivers.
The Fibre Channel switching module has twelve 4-port port groups. Each port group is capable
of a speed of 64 Gbps in each direction simultaneously. The ports of this switching module
support expansion port (E Port), fabric port (F Port), fabric loop port (FL Port), SPAN
destination port (SD Port), and trunking expansion port (TE Port) modes.
Individual ports can be configured with Cisco 16-Gbps, 8-Gbps, or 10-Gbps shortwave or long-
wave SFP+ transceivers. Each port supports 500 buffer credits with no additional licensing
required. With the Cisco Enterprise Package, up to 4095 buffer credits can be allocated to an
individual port.
The Fibre Channel switching module also provides Cisco VMpath technology that enables
advanced virtual machine(VM)-aware SAN provisioning and monitoring for virtualized data
centers. With Cisco VMpath, you can monitor, manage, and control SAN resource allocation
and performance on a per-VM basis and map out paths from the server to storage.
The Fibre Channel switching module is hot-swappable and has hot-swappable SFP+
transceivers. The module includes online diagnostics, stateful process restart, and nondisruptive
supervisor failover. In addition, the Fibre Channel switching module has any module, any port
configuration for port channels, fabric-based multipathing, per-VSAN fabric services, and port
tracking. The module also supports Virtual Routing Redundancy Protocol (VRRP) for
management.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-15
The major capabilities of the switching module include the following:
 Configuration file management
 Nondisruptive software upgrades for Fibre Channel interfaces
 Call Home
 Power-management LEDs
 Port beaconing
 System LED
 SNMP traps for alerts
 Network boot

1-16 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• New form factor 3000 W power
supply module
• Autosensing voltage detection
• Both AC and DC power supplies
are available
• You can mix AC and DC power
supplies in the same chassis

MDS

The MDS 9710 Multilayer Director provides outstanding availability and reliability. The MDS
9710 Multilayer Director is the first director-class switch in the industry to enable redundancy
on all major components, including the fabric card.
This switch provides grid redundancy on power supply and 1+1 redundant supervisors. Users
can add an additional fabric card to enable N+1 fabric redundancy.
The MDS 9710 Multilayer Director supports the following types of power supplies:
 3000-W AC power supply (AC input and DC output)
 3000-W DC power supply (DC input and DC output)

You can mix and match AC and DC power supplies in a single chassis.
The MDS 9710 Multilayer Director supports up to eight hot-swappable 3000-W AC power
supplies (AC input) (DS-CAC97-3KW).
When connected to 220 VAC, the DS-CAC97-3KW power supplies for the MDS 9710
Multilayer Director provide output power of 3000 W to power the modules and fans. When
connected to a 110 VAC power system, the power supply provides approximately 1400 W. In
this case, and if the power supplies are used in redundant rather than combined mode, the
power supplies might not provide adequate power. The ability to supply adequate power
depends on the number of modules that are loaded in the chassis.
Each power supply module monitors its output voltage and provides the status to the
supervisor. In addition, the power supply modules provide information about local fans, power,
shutdown control, and EEPROM to the supervisor.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-17
• Maximum MDS 9710 Multilayer
Director configuration requires
three power supplies to operate. Grid A Grid B
• Six power supplies provide grid
redundancy, three per power
grid. Any to Any
• Power supplies can connect to
any grid externally.

Power Power
Distribution Distribution
Unit A Unit B

The MDS 9710 Multilayer Director requires three power supplies for fully loaded chassis.
More than four power supplies will provide redundancy and reserve power in case of failure.

1-18 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Each fan tray has four variable
speed fans.
• Each fan tray has redundant
power and control paths.
• All fans in a fan tray run at the
same speed. Fans in different
trays can operate at different
speeds as needed. 23.54”
• Under normal operating
temperatures, a single fan tray
can be removed for an extended
period for service.

4.09” 5.15”
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-21

The MDS 9710 Multilayer Director has three fan trays, each with four fans. The fan trays are
installed vertically at the back of the chassis. Each fan tray can be removed temporarily while
the other two fan trays continue to move air through the chassis.

Note A fan tray cannot be left out of chassis for long periods of time.

Fan trays cover the fabric modules in the back of the chassis. Fan tray 1 must be removed to
access fabric modules 1 and 2. Fan tray 2 must be removed to access fabric modules 3 and 4.
Fan tray 3 must be removed to access fabric modules 5 and 6.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-19
Cisco MDS 9500 Series Director-Class Switches
and Components
This topic describes Cisco MDS 9500 Series director-class switches and components.

• Fully redundant • Industry-leading port density

- No single point of failure - Highly scalable with up to 528
- Greater than 99.999 percent ports per chassis and 1584 ports
availability per rack
- Dual supervisors, power supplies, - Total system-switching capacity
crossbars, clocks, and fans up to 8.4 Tbps
- Hot-swappable line cards and - Up to 256 Gbps per slot
modules
- Nondisruptive code upgrades

14 RU

7 RU

MDS 9506 MDS 9513

• 6 chassis slots • 13 chassis slots
• 2 supervisors • 2 supervisors
• 4 line cards • 11 line cards
• 192 Fibre Channel ports • 528 Fibre Channel ports
maximum maximum
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-23

The Cisco MDS 9500 Series Multilayer Directors elevate the standard for director-class
switches. These switches are enterprise-class, award-winning Multilayer Director Switches.
The switches provide industry-leading availability, multiprotocol support, advanced scalability,
security, nonblocking fabrics that are 10 Gbps ready, and a platform for storage management.
These features allow the MDS 9500 Series Multilayer Directors to deploy high-performance
SANs with a lower total cost of ownership (TCO).
Layering a rich set of intelligent features and hardware-based services onto a high-performance,
protocol-agnostic switch fabric, MDS 9500 Series Multilayer Directors address the stringent
requirements of large data center storage environments.
MDS 9500 Series switch chassis are available in two sizes, which include Cisco MDS 9513 (14
rack units [RU]) and Cisco MDS 9506 (7 RU).

Cisco MDS 9506 Multilayer Director Chassis

The MDS 9506 Multilayer Director supports the same director-class features as the Cisco MDS
9509 Multilayer Director, but with a more compact six-slot (7-RU) chassis design because the
power supplies are located at the rear. The switch has slots for two Cisco MDS 9500 Series
Supervisor Modules and four switching or service modules. Power supplies are installed in the
back for easy removal, with the power entry modules (PEMs) in the front of the chassis for
easy access.
Up to six MDS 9506 Multilayer Director chassis can be installed in a standard 42-RU rack.
Cable management is facilitated by the single side position of both interface and power
terminations.

1-20 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco MDS 9513 Multilayer Director Chassis
The MDS 9513 Multilayer Director has a 13-slot chassis with redundant MDS 9500 Series
Supervisor-2 Modules, up to 11 switching modules, redundant 6-kW power supplies, a
removable fan module at the front, and additional removable fan modules at the rear for the
fabric modules. Slots 7 and 8 are reserved for redundant MDS 9500 Series Supervisor-2
Modules, which provide control, switching, and local and remote management.
The MDS 9513 Multilayer Director supports an industry-leading port density per system that is
expandable up to 528 Fibre Channel ports in a single chassis. There are two removable system-
clock modules at the rear for added high availability. Dual redundant 6-kW power supplies are
located at the rear of the chassis. The MDS 9513 Multilayer Director has a revised airflow
system at the rear of the chassis. The air flows in at the bottom and out at the top.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-21
• Dual supervisors
- Active and standby
- Hot-swappable
- Stateful standby stays
synchronous with all major
management and control
protocols of the active supervisor
• Nondisruptive upgrades
- Load and activate new software
without disrupting traffic
- Standby supervisor maintains the
previous version of code while the
active supervisor is updated

MDS 9500 Series Multilayer Directors support two MDS 9500 Series Supervisor Modules in
the chassis for redundancy. Each MDS 9500 Series Supervisor Module consists of a control
engine and a crossbar fabric. The control engine is the central processor responsible for the
management of the overall system. In addition, the control engine participates in all of the
networking control protocols, including all Fibre Channel services. In a redundant system, two
control engines operate in active/standby mode.
The control engine that is in standby mode is actually in a stateful-standby mode that syncs
with all major management and control protocols that the active control engine maintains.
While the standby control engine is not actively managing the switch, the engine continually
receives information from the active control engine. This architecture allows the state of the
switch to be maintained between the two control engines. If the active control engine fails, the
secondary control engine seamlessly resumes function.
The crossbar fabric is the switching engine of the system. The crossbar fabric provides a high-
speed matrix of switching paths between all ports within the system. A crossbar fabric is
embedded within each MDS 9500 Series Supervisor Module. The two crossbar fabrics operate
in a load-shared active/active mode. Each crossbar fabric has a total switching capacity of 720
Gbps and serves 80 Gbps of bandwidth to each slot on MDS 9506 Multilayer Directors. Since
each switching module of the MDS 9506 Multilayer Directors does not consume more than 80
Gbps of bandwidth to the crossbar, the system operates at full performance even with one MDS
9500 Series Supervisor Module. In a fully populated MDS 9500 Series switch, the system does
not experience any disruption or loss of performance with the removal or failure of one MDS
9500 Series Supervisor Module.
The supervisor module is a hot-swappable module. In a dual MDS 9500 Series Supervisor
Module system, this functionality allows the module to be removed and replaced without
causing disruption to the rest of the system.

1-22 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• High-performance integrated crossbar
- Active when installed in MDS 9506 chassis
- Bypassed when installed in MDS 9513 chassis
- Supports up to 48 Gbps of front-panel bandwidth per slot
• MDS 9513 requires Supervisor-2 or -2A
Front Panel Interfaces
• FCoE support requires Supervisor-2A
1. Status LEDs
2. Reset Button
3. Console Port
4. Management Eth Port 10/100/1000
5. Serial Port
6. CompactFlash LED
7. Flash Eject Button
8. Compact Flash Slot
9. Two USB Ports
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-25

Cisco MDS 9500 Series Supervisor-2 and Supervisor-2A Module Features

The MDS 9500 Series offers redundant, hot-swappable Supervisor-2 Modules. Supervisor-2
Modules can be used in the MDS 9506 Multilayer Directors in slots 5 and 6. Supervisor-2
Modules must be used in slots 7 and 8 of the MDS 9513 Multilayer Director.
Supervisor-2 Modules provide an integrated crossbar switching fabric to connect all of the
switching modules when used in MDS 9506 Multilayer Directors. Single fabric configurations
provide 720-Gbps full duplex speed with 80-Gbps full duplex bandwidth per switching module.
Dual fabric configurations provide 1.4-Tbps speed with 160-Gbps full duplex bandwidth per
switching module.
This integrated crossbar switching fabric is disabled when a Supervisor-2 Module is installed in
an MDS 9513 Multilayer Director. The MDS 9513 Multilayer Director supports two external
crossbar modules that are located at the rear of the chassis and manage this function.
MDS 9500 Series Supervisor-2A is an upgraded version of MDS 9500 Series Supervisor-2 with
2-GB memory and support for FCoE.
All frames pass directly from line card ASICs across the crossbar and out to their destination
interfaces. Frame flow is not regulated by the MDS 9500 Series Supervisor. Cisco MDS 9500
Series Supervisor-2 uses a PowerPC management processor to provide the following Fibre
Channel services to connected devices:
 Fabric Shortest Path First (FSPF)
 Zoning
 Name Server
 Fabric login (FLOGI) server
 Security
 VSANs
 IVR

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-23
When used in an MDS 9506 Multilayer Director, the integral crossbar is used. When used in
the MDS 9513 Multilayer Director, the integral crossbar is bypassed and the crossbar fabric
modules are used instead.
MDS 9500 Series Supervisor-2 and MDS 9500 Series Supervisor-2A support 1024 destination
indexes, providing up to 528 ports in the MDS 9513 Multilayer Director when there are no
generation-1 modules that are installed in the chassis. If any generation-1 modules are installed
in the MDS 9513 Multilayer Director, then only 252 ports can be used.

1-24 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Feature Supervisor-2 Supervisor-2A
Memory 1G 2G
Support for existing Fibre Channel modules Yes Yes
Support for high-density line rate 8-Gb modules Yes Yes
Support for FCoE modules No Yes

Feature Supervisor-2A
Nondisruptive upgrade from Supervisor-2 Yes
Mix of Supervisor-2 and Supervisor-2A in same Supported during upgrade only
chassis
Shipping by default in chassis Yes, based on per-Optical
Services Module (OSM) basis

Supervisor-2A
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-26

The Cisco MDS 9500 Series Supervisor-2A Module can supply up to 1.4 Tbps of nonblocking
performance to Cisco MDS 9500 Series Multilayer Directors. Its robust switching performance
enables the MDS 9500 Series to provide 1-, 2-, 4-, 8-, and 10-Gbps Fibre Channel port
densities. When installed in an MDS 9513 Multilayer Director chassis, the Cisco MDS 9500
Series Supervisor-2A Module works with the two crossbar modules to provide 2.2 Tbps of fully
redundant system bandwidth, helping ensure high scalability in any SAN environment.
The MDS 9500 Series Supervisor-2A Module delivers the latest advanced switching
technology. The module uses proven Cisco NX-OS Software to power a new generation of
scalable and intelligent multilayer switching solutions for SANs.
The MDS 9500 Multilayer Director Supervisor-2A Module does the following:
 Helps enable converged, high-performance, intelligent, resilient, scalable, and secure
multilayer SAN switching solutions
 Supports deployment of FCoE in the MDS 9500 Series Multilayer Director chassis
 Is designed to integrate multiprotocol switching and routing, intelligent SAN services, and
storage applications onto highly scalable SAN switching platforms
 Is supported in the MDS 9506 and 9513 Multilayer Director chassis

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-25
• Redundant crossbar fabric
- Active/active operation balances the load across
both crossbars.
- Rapid failover in case of failure ensures no loss
of frames.
• High bandwidth nonblocking architecture
- Fabric 2 provides up to 96-Gbps per slot.
- Fabric 3 provides up to 256-Gbps per slot.
- A single crossbar fabric still provides sufficient
bandwidth for all line cards.
• High-performance centralized architecture
- Ensures consistent latency across the switch.
- Supports up to 1024 indexes (destination
interfaces).
- Enhanced high-performance arbiter schedules
frames at over 1 billion per second.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-27

The figure describes MDS 9513 Multilayer Director fabric modules.

• Increases backplane bandwidth to support 8-Gbps line rate ports

(advanced 8-Gb modules).
- The 256-Gbps bandwidth capability of the 8-Gbps Advanced Fibre Channel
modules requires the Fabric 3 module in the Cisco MDS 9513 Multilayer
Director.
• There is a nondisruptive upgrade from Fabric 2 to Fabric 3.
• Fabric 3 is supported in Cisco NX-OS 5.2(1) or higher releases only.

The figure describes the MDS 9513 Multilayer Director Fabric 3 Module.

1-26 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 32-port 8-Gbps advanced Fibre Channel switching module
- No oversubscription ratio, 8-Gbps full rate simultaneously
- Investment protection because the module is compatible with every MDS
9500 Series switch that is shipped
- For high-end storage systems, there is ISL connectivity
• 48-port 8-Gbps advanced Fibre Channel switching module
- 256 Gbps of Fibre Channel bandwidth, six ports per port group
- Standard, high performance, and highly virtualized servers
• Cisco TrustSec Fibre Channel link encryption
- Provides secure transmission of data between switches and data centers

32-Port 1/2/4/8/10-Gbps Advanced 48-Port 1/2/4/8/10-Gbps Advanced

Fibre Channel Switching Module Fibre Channel Switching Module
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-29

MDS 9513 32-Port 8-Gbps Advanced Fibre Channel Switching Module: DS-X9232-
256K9
This module is more suitable for high-end storage systems as well as for Inter-Switch Link
(ISL) connectivity. This module delivers 256 Gbps of front panel bandwidth with a total of
thirty-two 8-Gbps interface connectivity to the back-end storage systems. There are eight port
groups with four ports in each group. This module has no oversubscription ratio and all 32 ports
can run at 8-Gbps full speed simultaneously.

MDS 9513 48-Port 8-Gbps Advanced Fibre Channel Switching Module: DS-X9248-
256K9
With the 8-Gbps Fibre Channel bandwidth option, this module is more suitable for port density
and high-speed performance. The front panel delivers 256 Gbps of Fibre Channel bandwidth
with 48 ports, which are divided in eight groups of six ports per group. The total allocated
bandwidth per port group is 32-Gbps with maximum speed of 8-Gbps per port.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-27
• 32 ports at 8-Gb Fibre Channel full • 48 ports at 4-Gb Fibre Channel full
rate: rate:
- No oversubscription at 8 Gbps - 1.5:1 oversubscribed at 8 Gbps
- 4-port port groups - 6-port port groups

• 24 ports at 10-Gb Fibre Channel full • 24 ports at 10-Gb Fibre Channel full
rate: rate:
- Ports 2, 3, 4, 5, 6, and 8 at 10 Gbps - Ports 4, 5, 6, 7, 8, and 10 at 10 Gbps
- Ports 1 and 7 disabled - Ports 1, 2, 3, 9, 11, and 12 disabled
(three ports per port group)

32-port 8-Gb Advanced Fibre Channel Switching Module 48-port 8-Gb Advanced Fibre Channel Switching Module

All line card modules have their own port group settings. Depending on the port group
configuration, you can configure a port for regular Fibre Channel speed or 10-Gbps Fibre
Channel speed.

Note Not all ports can be configured for 10-Gbps Fibre Channel speed.

The figure shows ports from individual port groups that can be configured as 10-Gbps Fibre
Channel speed. The interfaces that can be configured out of the port groups are highlighted in
yellow on the figure. The interfaces that will be disabled by the switch are marked by a red “X”
on the figure. The 10-Gb Fibre Channel enablement works at the ASIC level. One ASIC
controls two port groups simultaneously as follows:
 32-port 8-Gbps module, eight port groups with four ports per port group.
— The 10-Gbps mode configuration for a group of eight ports is six ports at 10-Gbps
and other two ports disabled.
 48-port 8-Gbps module , eight port groups with six ports per port group, with up to four 8-
Gbps dedicated ports per port group.
— The 10-Gbps mode configuration for a group of 12 ports is six ports at 10-Gbps and
the other six ports disabled.

1-28 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 8-port 10-Gbps FCoE module
- Multihop FCoE module for the core
- Requires Supervisor-2A module
• Supports deployment of FCoE in the MDS 9500 Series chassis
• Nondisruptive upgrade from Supervisor-2
• 2-Gb memory
FCoE Fibre Channel

Fibre Channel Storage

FCoE Storage
Core: MDS
9500/9700 with
FCoE Module

FCoE

Aggregation:
Pod N Nexus 7000

Unified I/O
Access
Nexus 5000

Unified I/O

Server Rack 1 Server Rack N

MDS 9513 FCoE Module

The MDS 9513 FCoE Module provides support for multihop FCoE in MDS 9500 Series
chassis.
Developed for MDS 9500 Series Fibre Channel directors, the MDS 9513 10-Gigabit Ethernet
8-port FCoE Module helps consolidate the network in the core to reduce complexity. This
module offers enterprise-class systems and topologies for FCoE deployments. By providing a
converged fabric, the module helps the core Fibre Channel capacity connect to the FCoE access
layer while preserving storage network services. This module is also used by service providers
to deploy as part of their managed network services.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-29
• FCoE does not require a license on the Cisco MDS 9500 Series.
• You can only enable FCoE on an FCoE module.
• You cannot enable FCoE on VLAN 1 (default) or the native VLAN.
• Cisco MDS 9500 supports only Generation-2 converged network
adapters.
• Cisco MDS 9500 does not support LAN traffic and only accepts and
processes FCoE frames.
• The QoS policy must be the same on all Cisco FCoE switches in the
network.
• You must install a Supervisor-2A module to configure FCoE.

The figure describes MDS 9500 Series FCoE guidelines and limitations.

1-30 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco MDS 9200 Series Multiservice Switch and
Service Modules
This topic describes the Cisco MDS 9200 Series Multiservice Switch and Service Modules.

• SAN services for Cisco MDS and Cisco Nexus product lines
- Optimized for performance, power, and space
• 40 line-rate 16-Gbps Fibre Channel ports and 8 10GE FCoE ports
• Front-to-back air flow
• 3 x 300 W power supplies, 2 + 1 for hardware failure
- Remote SAN extension
• Two 1/10-Gbps FCIP and iSCSI ports for high-performance FCIP
• 6 FCIP tunnels per GbE port
- Intelligent fabric services, such as the following:
• Cisco MDS 9000 IOA and Cisco DMM
- Flexibility for future growth and capacity expansion

The Cisco MDS 9250i Multiservice Fabric Switch is the next generation of the highly flexible,
industry-leading, proven Cisco MDS 9200 Series Multiservice Switches. This switch is an
optimized platform for deploying high-performance SAN extension solutions, distributed
intelligent fabric services, and cost-effective multiprotocol connectivity for both open systems
and mainframe environments. With a compact form factor and advanced capabilities normally
available only on director-class switches, this switch is an ideal solution for departmental and
remote branch-office SANs and in large-scale SANs with the MDS 9710 Multilayer Director.
The MDS 9250i Multiservice Fabric Switch offers up to forty 16-Gbps Fibre Channel ports,
two 1/10 Gigabit Ethernet IP storage services ports (10Gbps support initially), and eight 10
Gigabit Ethernet FCoE ports in a fixed 2-RU form factor. The MDS 9250i Multiservice Fabric
Switch connects to existing native Fibre Channel networks, protecting current investments in
storage networks. The Cisco SAN Extension over IP application package license is enabled as
standard on the two fixed 1/10 Gigabit Ethernet IP storage services ports, enabling features
such as FCIP and compression on the switch without the need for additional licenses. Also,
using the eight 10 Gigabit Ethernet FCoE ports, the MDS 9250i Multiservice Fabric Switch
platform attaches to directly connected FCoE and Fibre Channel storage devices and supports
multitiered unified network fabric connectivity directly over FCoE.

Main Features and Benefits

The MDS 9250i Multiservice Fabric Switch provides unique multiservice and multiprotocol
functions in a compact 2-RU form factor:
 The MDS 9250i Multiservice Fabric Switch scales up to 40 ports of 16-Gbps Fibre Channel
in a fixed configuration switch. The base configuration comes with 20 ports of 16-Gbps
Fibre Channel enabled for high-performance SAN connectivity. The switch can be

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-31
upgraded onsite to enable an additional 20 ports of 16-Gbps Fibre Channel by adding the
MDS 9250i Multiservice Fabric Switch On-Demand Port Activation license. Additionally,
the MDS 9250i Multiservice Fabric Switch cost-effectively scales up for IBM Fibre
Connection (FICON) mainframe environments. Also, two ports of 10 Gigabit Ethernet
provide FCIP and iSCSI storage services, and eight ports of 10 Gigabit Ethernet for FCoE
connectivity.
 The switch provides an intelligent application services engine. The MDS 9250i
Multiservice Fabric Switch includes as standard a single application services engine. The
engine enables the included Cisco SAN Extension over IP software solution package to run
on the two fixed 1/10 Gigabit Ethernet storage services ports. The Cisco SAN Extension
over IP package provides an integrated, cost-effective, and reliable business-continuance
solution. The solution uses IP infrastructure by offering FCIP for remote SAN extension,
along with various advanced features to optimize the performance and manageability of
FCIP links.
 Hardware-based virtual fabric isolation with VSANs and Fibre Channel routing with IVR.
VSANs and IVR enable deployment of large-scale multisite and heterogeneous SAN
topologies. Integration into port-level hardware allows any port in a system or in a fabric to
be partitioned into any VSAN. Included in the optional Cisco MDS 9000 Enterprise
advanced software package, IVR provides line-rate routing between any of the ports in a
system or in a fabric without the need for external routing appliances.
 Remote SAN extension with high-performance FCIP:
— Simplifies data protection and business continuance strategies by enabling backup,
remote replication, and other disaster-recovery services over WAN distances using
open-standards FCIP tunneling.
— Optimizes utilization of WAN resources for backup and replication by enabling
hardware-based compression, hardware-based encryption, FCIP Write Acceleration,
and FCIP tape read and write acceleration. Up to 16 virtual ISL connections are
provided on the two 10 Gigabit Ethernet ports through tunneling.
— Preserves Cisco MDS 9000 Series enhanced capabilities, including VSANs, IVR,
advanced traffic management, and network security across remote connections.
 Cost-effective iSCSI connectivity to Ethernet-attached servers:
— Extends the benefits of Fibre Channel SAN-based storage to Ethernet-attached
servers at a lower cost than is possible using Fibre Channel interconnect alone.
— Increases storage utilization and availability through consolidation of IP and Fibre
Channel block storage.
— Through transparent operation, preserves the capability of existing storage
management applications.
 Advanced FICON services: The MDS 9250i Multiservice Fabric Switch supports FICON
environments, including cascaded FICON fabrics, VSAN-enabled intermix of mainframe
and open systems environments, and NPIV for mainframe Linux partitions. IBM Control
Unit Port (CUP) support enables in-band management of Cisco MDS 9200 Series switches
from the mainframe management console. FICON tape acceleration reduces latency effects
for FICON channel extension over FCIP for FICON tape read and write operations to
mainframe physical or virtual tape. This feature is sometimes referred to as tape pipelining.
The MDS 9250i Multiservice Fabric Switch also supports the IBM Extended Remote Copy
(XRC) Acceleration feature. That feature enables acceleration of dynamic updates for IBM
z/OS Global Mirror, formerly known as XRC.

1-32 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Cisco Data Mobility Manager (DMM) as a distributed fabric service: Cisco DMM is a
fabric-based data migration solution that transfers block data nondisruptively across
heterogeneous storage volumes and across distances, whether the host is online or offline.
 Platform for intelligent fabric applications: The MDS 9250i Multiservice Fabric Switch
provides an open platform that delivers the intelligence and advanced features that are
required to make multilayer intelligent SANs a reality. The switch includes hardware-
enabled innovations to host or accelerate applications for data migration, storage backup,
and data replication. Hosting or accelerating these applications in the network can
dramatically improve scalability, availability, security, and manageability of the storage
environment, resulting in increased utility and lower TCO.
 ISSU for Fibre Channel interfaces: MDS 9250i Multiservice Fabric Switch promotes high
serviceability by allowing MDS 9000 NX-OS Software to be upgraded while the Fibre
Channel ports are carrying traffic.
 Intelligent network services: MDS 9250i Multiservice Fabric Switch uses VSAN
technology for hardware-enforced, isolated environments within a single physical fabric,
ACLs for hardware-based intelligent frame processing, and advanced traffic management
features such as fabric-wide QoS to facilitate migration from SAN islands to enterprise-
wide storage networks.
 High-performance ISLs: MDS 9250i Multiservice Fabric Switch supports up to 16 Fibre
Channel ISLs in a single port channel. Links can span any port on any module in a chassis
for added scalability and resilience. Up to 256 buffer-to-buffer credits can be assigned to a
single Fibre Channel port to extend storage networks over long distances.
 Comprehensive network security framework: The MDS 9250i Multiservice Fabric Switch
supports RADIUS and TACACS+, FC-SP, SFTP, SSH Protocol, SNMPv3 implementing
AES, VSANs, hardware-enforced zoning, ACLs, and per-VSAN RBAC. Additionally, the
10 Gigabit Ethernet ports offer IP Security (IPsec) authentication, data integrity, and
hardware-assisted data encryption for FCIP and iSCSI.
 IP version 6 (IPv6)-capable: The MDS 9250i Multiservice Fabric Switch supports IPv6 as
mandated by the U.S. Department of Defense, Japan, and China. IPv6 support is provided
for FCIP, iSCSI, and management traffic routed in-band and out of band.
 FIPS compliance: The MDS 9250i Multiservice Fabric Switch is FIPS 140-2 compliant as
mandated by the U.S. federal government.
 Sophisticated diagnostics: The MDS 9250i Multiservice Fabric Switch provides intelligent
diagnostics, protocol decoding, and network analysis tools as well as integrated Cisco Call
Home capability for added reliability, faster problem resolution, and reduced service costs.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-33
Three Power Supply Units: Chassis size:
• Two connected with Mother 17.30”x3.48”x21.44” (W x H x D)
board at the bottom and one at
the upper layer. Rear
Panel
• Two Fan Tray in the middle with
2RU height.

Front Panel
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-35

The MDS 9250i Multiservice Fabric Switch provides the following types of ports:
 Console port: An RS-232 port that you can use to create a local management connection
 MGMT 10/100/1000 Ethernet port: An Ethernet port that you can use to access and
manage the switch by IP address, such as through Cisco DCNM
 Fibre Channel ports: Fibre Channel ports that can be used to connect to the SAN or for
in-band management
 FCoE: FCoE ports that you can use for FCoE connectivity
 IP storage ports: Gigabit Ethernet ports that can be used for IP storage services such as
FCIP or iSCSI
 USB drive: USB port on the front panel for code loads, configuration file backups, log
dumps, and report capture

1-34 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Not Requiring App Services Engine Requiring App Services Engine*
Cisco Prime DCNM Base Included Cisco MDS 9000 SAN Included
Version Extension over IP
Cisco Prime DCNM Optional Cisco MDS 9000 IOA Optional
Licensed Version Services
Cisco MDS 9000 Optional Cisco DMM Optional
Enterprise
Cisco MDS 9000 Optional
Mainframe

*MDS 9250i includes one

Application Services Engine.

Advanced Software Packages

The MDS 9250i Multiservice Fabric Switch can be further enhanced through additional
optional licensed software packages that offer advanced intelligence and functions. The figure
summarizes the packages. Currently available software packages include the following:
 Cisco MDS 9000 Enterprise Package: This package includes a set of traffic engineering
and advanced security features, such as extended-distance buffer-to-buffer (B2B) credits,
IVR, QoS, switch-to-switch and host-to-switch authentication, logical unit numbering
(LUN) zoning, and read-only zones, that are recommended for enterprise SANs.
 Cisco Prime DCNM License: This license is the licensed version of Cisco Prime DCNM
that provides server federation, historical performance monitoring for network traffic
hotspot analysis, centralized management services, and advanced application integration.
 Cisco MDS 9000 DMM Package: This package enables Cisco DMM to perform fabric-
based data migration that transfers block data nondisruptively across heterogeneous storage
volumes and across distances, whether the host is online or offline.
 Cisco MDS 9000 IOA Services Package: The MDS 9250i Multiservice Fabric Switch
supports Cisco MDS 9000 Input/Output Accelerator (IOA) services, an advanced software
package that can significantly improve application performance when storage traffic is
extended across long distances. When Fibre Channel and FCIP Write Acceleration is
enabled, WAN throughput is optimized through reduced latency for command
acknowledgments.
 Cisco MDS 9000 Mainframe Package: This package is a comprehensive collection of
features that are required for using the MDS 9500 Series and MDS 9200 Series switches in
mainframe storage networks, including FICON protocol, FICON tape acceleration (read
and write), IBM CUP management, switch cascading, fabric binding, and intermixing.
 Cisco MDS 9000 XRC Acceleration Package: With the SAN Extension over IP and
Mainframe packages, this package provides acceleration (channel extension) over IP for the
IBM z/OS Global Mirror replication solution, reducing the effects of latency at distances of
up to 20,000 km.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-35
• Expansion slot
• 18 Fibre Channel ports at 4 Gbps
• 4 Gigabit Ethernet ports for FCIP and iSCSI
• 1.3 GHz PowerPC 8548 processor
• Dual removable power supply and removable fan tray
• 1 GB compact flash and 1 GB of memory
• Quick Configuration Wizard
• FCIP hardware compression using deflate data format (RFC 1951)
• FCIP hardware encryption and acceleration
• Supports Cisco MDS 9000 IOA
• Supports Cisco SANTap and Cisco DMM

The Cisco MDS 9222i Multiservice Modular Switch delivers state-of-the-art multiprotocol and
distributed multiservice convergence. This switch offers high-performance SAN extension and
disaster recovery solutions, intelligent fabric services, and cost-effective multiprotocol
connectivity. With a compact form factor, the modularity of the expansion slot, and advanced
capabilities normally available only on director-class switches, the MDS 9222i Multiservice
Modular Switch is an ideal solution for departmental and remote branch office SANs. The
switch provides the features present in a director but at a lower cost of entry.
Product highlights include the following:
 High-density Fibre Channel switch, scales up to 66 Fibre Channel ports
 Integrated hardware-based virtual fabric isolation with VSANs and Fibre Channel routing
with IVR
 Remote SAN extension with high-performance FCIP
 Long distance over Fibre Channel with extended BB credits
 Multiprotocol and mainframe support (Fibre Channel, FCIP, iSCSI, and FICON)
 IPv6-capable
 Platform for intelligent fabric applications such as Cisco SANTap and Cisco DMM
 Cisco IOS ISSU
 Comprehensive network security framework
 Provides hosting, assisting, and acceleration of storage applications such as volume
management, data migration, and data protection

1-36 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 16x Gigabit Ethernet ports for
FCIP WAN connectivity
- Up to three FCIP tunnels per port
• Four independent service engines
supporting the following:
- FCIP
• FCIP hardware compression
• FCIP hardware encryption
• FCIP write acceleration
• FCIP tape read/write acceleration
with port channel support
• FICON over IP
• Cisco MDS 9000 XRC Cisco MDS 9000 16-Port
Acceleration over IP Storage Services Node
- IOA
• Cisco MDS 9000 IOA Fibre
Channel write acceleration
• Cisco MDS 9000 IOA Fibre
Channel tape read/write
acceleration
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-38

The Cisco MDS 9000 16-Port Storage Services Node (SSN-16) hosts four independent service
engines. Each engine can be activated individually and incrementally to scale as business
requirements change, or the engines can be configured to run separate applications.
Based on the single service engine in its predecessor, the Cisco MDS 9000 18/4-port
Multiservice Module (MSM), this four-to-one consolidation delivers dramatic hardware
savings and frees valuable slots in the Cisco MDS 9500 Multilayer Director chassis.
The MDS 9000 SSN-16 supports the following applications:
 Remote SAN extension with high-performance FCIP
 Metropolitan-area network (MAN) link optimization with Cisco MDS 9000 IOA
 IBM System z Channel Extension through FICON over IP, including tape acceleration
 Cisco Extended Remote Copy (XRC) Acceleration over IP for IBM z/OS Global Mirror

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-37
• MDS 9000 SSN-16
- Four independent services engines
• Four separate concurrent applications FCIP
MSM-18/4
• Four times the performance and throughput
• Saves valuable slots in the MDS 9500 chassis
- Transparently delivers services to any port in the fabric MSM-18/4
IOA
• No need for host or target to be directly attached
• No SAN reconfiguration, no rewiring
• Highly available with multimodule clustering, balancing MSM-18/4 FCIP
- Fibre Channel speed-agnostic
• Any 4-Gbps, 8-Gbps, or 10-Gbps Fibre Channel port
can use services XRC
MSM-18/4
• 16 Gigabit Ethernet front panel ports for FCIP
FCIP
IOA
FCIP
XRC

MDS 9000 SSN-16)

The MDS 9000 SSN-16 basically takes the services engine that is the “/4” part of its
predecessor, the MDS 9000 MSM-18/4, and puts four of the services engines onto a single
module. This solution allows you to run four concurrent applications, which provides four
times the performance and throughput when compared to the MDS 9000 MSM-18/4. This
solution also helps you save slots in the 9500 chassis where previously you may have had to
deploy multiple MDS 9000 MSM-18/4 modules.
More importantly, the MDS 9000 SSN-16 provides centralized Fibre Channel services from
any Fibre Channel port in the fabric, regardless of line rate, whether 1-, 2-, 4-, or 8-Gbps.

1-38 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 18x 1/2/4-Gbps Fibre Channel Ports
• 4x Gigabit Ethernet ports for FCIP
WAN connectivity
- Up to three FCIP tunnels per port
• Single services engine supporting:
- FCIP
• FCIP hardware compression
• FCIP hardware encryption
• FCIP write acceleration
• FCIP tape read/write acceleration with
PortChannel support
- IOA
• IOA Fibre Channel write acceleration
• IOA Fibre Channel tape read/write
acceleration
- iSCSI gateway
- DMM
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-40

The figure describes the Cisco MDS 9000 MSM-18/4.

MAN/WAN

Cisco IOA Cisco IOA

Unified Solution Flexible Resilient Transport-Agnostic

Single solution for • No rewiring Port channels for • Any transport
both disk and tape • Extend Cisco MDS tape I/O acceleration protocol and any
I/O acceleration 9000 IOA to any interface
device in the SAN • Fibre Channel and
Gigabit Ethernet

Cisco MDS 9000 IOA is a unified solution for accelerating SCSI or open systems I/O. Cisco
MDS 9000 IOA is not compatible with FICON. There are acceleration solutions for FICON,
but those solutions are not part of Cisco MDS 9000 IOA.
Like other services-oriented SAN solutions, you can insert Cisco MDS 9000 IOA into the
network very easily, with no rewiring or recabling. The insertion is simply done by
administrative configuration using a CLI or Cisco DCNM SAN-Client. You can extend
acceleration services to any device in the fabric. The previous Gigabit Ethernet line card
modules 14/2 and 18/4 tied the services to the Gigabit Ethernet port on that line card. Cisco
MDS 9000 IOA provides these services for any Fibre Channel port in the fabric, regardless of
link rate. Now, you can have the acceleration on the MDS 9000 IOA node and use port
channels across your fabric to combine multiple Gigabit Ethernet ports. Combining the ports
means that if one port fails, the entire tape job does not reset. Because Cisco MDS 9000 IOA is
transport agnostic, Cisco MDS 9000 IOA works equally well for FCIP or for native Fibre
Channel extensions as, for example, over dense wavelength-division multiplexing (DWDM).

1-40 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Encryption of data in flight
- Preserves integrity and confidentiality of Fibre Channel traffic
- Integrated, high-performance functionality
- No change to existing SAN, enables functionality only on edge switches
- Enabled with an Enterprise license
- Supported on 8-Gb and 16-Gb* modules

Fibre Channel Data Integrity and Encryption

FC HDR Payload FC HDR Payload

FC HDR Encrypted Payload

FC HDR = Fibre Channel Header

16-Gb* modules soon

Cisco TrustSec Fibre Channel Link Encryption provides encryption for Fibre Channel frames
in transit through 8-Gbps ISLs.
This feature is integrated into the line cards for all of the 8-Gbps modules, so the 48-port and
32-port advanced switching modules support this capability.
There is no change to the existing SAN. You enable the functionality for any links. This feature
requires the Enterprise license.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-41
Cisco MDS 9100 Series Fabric Switches
This topic describes Cisco MDS 9100 Series Fabric Switches.

• 48 line-rate 8-Gbps Fibre • Two hot-swappable power

Channel ports supplies
• 384 Gbps aggregate bandwidth • Four fans plus one fan per
• 128 buffer-to-buffer credits per power supply
4-port group • 1 RU high, less than 20 inches
• 1-, 2-, 4-, and 8-Gbps Fibre deep
Channel 16-, 32-, or 48-port • Nondisruptive software
base configuration upgrades
• 8-port incremental licensing • Cisco NX-OS 5.x feature set
• Comprehensive security • Up to 16 ports per port channel
framework • Up to 16 VSANs per switch
• Full MIB and SMI-S support • Cisco NPV and NPIV support
with FlexAttach

The Cisco MDS 9148 Multilayer Fabric Switch provides an affordable, highly capable, and
scalable storage networking solution for small, midsize, and large enterprise customers.
The switch offers outstanding value by providing flexibility, high availability, security, and
ease of use at an affordable price in a compact 1-RU form factor. With the flexibility to expand
from 16 to 48 ports in eight-port increments, the MDS 9148 Multilayer Fabric Switch offers the
densities that are required to scale from entry-level departmental switch to top-of-the-rack
switch to edge connectivity in enterprise SANs. The MDS 9148 Multilayer Fabric Switch
offers nonblocking architecture, with all 48 2-, 4-, and 8-Gbps ports operating at line rate
concurrently.
The MDS 9148 Multilayer Fabric Switch supports quick configuration, zero-touch immediately
active (plug-and-play) features, and task wizards that allow the switch to be deployed quickly
and easily in networks of any size. Powered by Cisco NX-OS Software, the switch includes
advanced storage networking features and functions. The switch is compatible with Cisco MDS
9500 Series Multilayer Directors and Cisco MDS 9200 Series Multilayer Fabric Switches,
providing transparent, end-to-end service delivery in core-edge deployments.
The MDS 9148 Multilayer Fabric Switch offers intelligent storage networking capabilities such
as VSANs, IVR, port channels, QoS, and security for cost-effective design, deployment, and
management of departmental and enterprise SANs.
Server consolidation as a result of server virtualization technologies has increased the number
of fabric logins that are needed on a SAN. The MDS 9148 Multilayer Fabric Switch uses Cisco
NX-OS Software intelligence to provide storage access for virtual machine server
environments. With full NPIV support and Cisco N-Port Virtualizer (NPV) technology, the
MDS 9148 Multilayer Fabric Switch is built for increased scalability. Using Cisco VSAN
technology, the MDS 9148 Multilayer Fabric Switch is the only entry-level SAN switch that
can extend server virtualization. Cisco NPV and Cisco FlexAttach technology on the MDS
9148 Multilayer Fabric Switch enable transparent server deployment and movement without
the need to reconfigure the SAN.

1-42 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Grow as you go.
- 16-port base
- Additional Licensed in 8-port increments

48 x 8-Gbps Fibre Channel Ports High Availability

1 RU Form Factor Redundant Power and Fans

The MDS 9148 Multilayer Fabric Switch is a 1-RU Fibre Channel switch with 48 ports, all
supporting full-line-rate 8-Gigabit operation.
To support high levels of business resilience, the MDS 9148 Multilayer Fabric Switch is built
to enterprise-class specifications with redundant hot swappable power supplies and replaceable
fan trays.
A significant difference between the MDS 9148 Multilayer Fabric Switch and its predecessors
is its full support for the complete set of enterprise SAN features that are found on MDS 9200
and 9500 Series switches.
The MDS 9148 Multilayer Fabric Switch offers a flexible capability to expand as needed, with
an initial deployment of 16, 32, or 48 ports. The Cisco on-demand port licensing model allows
customers to activate ports in sets of eight.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-43
• Grow as you go.
- 16-port base
- Additional Licensed in 8-port increments

48 x 8-Gbps Fibre Channel Ports High Availability

1 RU Form Factor Redundant Power and Fans

Email
MDS 9148
Departmental Shared Storage
or Entry-Level Common Backup
SAN Simplified Management
Database

The MDS 9148 Multilayer Fabric Switch is a flexible option for entry-level departmental or
midmarket SANs. For these opportunities, features such as shared storage, a common backup
capability, and simplified storage management are key benefits.
If you are migrating to a virtualized server environment, the MDS 9148 will be a key
component of your virtualization strategy, supporting virtual machine isolation and mobility.
As your needs grow, the MDS 9148 can become part of a larger enterprise core/edge SAN
architecture. Using MDS 9500 directors in the core and moving the MDS 9148 towards the
edge provides investment protection.

1-44 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Grow as you go.
- 16-port base
- Additional Licensed in 8-port increments

48 x 8-Gbps Fibre Channel Ports High Availability

1 RU Form Factor Redundant Power and Fans

NPIV
Tier 1 Scalability for VM
Virtualization Tier 2
environments
Optimized VM-granular security
Tier 3
SAN and QoS

MDS 9148
VM mobility

If you are migrating to a virtualized server environment, the MDS 9148 will be a key
component of your virtualization strategy, supporting virtual machine isolation and mobility.

• Grow as you go.

- 16-port base
- Additional Licensed in 8-port increments

48 x 8-Gbps Fibre Channel Ports High Availability

1 RU Form Factor Redundant Power and Fans

MDS 9148
Scalability
Enterprise
Core-Edge Security
SAN Compliance
Consolidation

As your needs grow, the MDS 9148 Multilayer Fabric Switch can become part of a larger
enterprise core/edge SAN architecture. Using MDS 9500 Series directors in the core and
moving the MDS 9148 Multilayer Fabric Switch towards the edge provides investment
protection.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-45
• Grow as you go.
- 16-port base
- Additional Licensed in 8-port increments

48 x 8-Gbps Fibre Channel Ports High Availability

1 RU Form Factor Redundant Power and Fans

MDS 9148 MDS 9148

Business Business continuity
Continuity MAN
Compliance
Metro-SAN Simplified management

If you need to extend your SAN connectivity across data centers to enable remote backup or
data replication, this switch still has a role to play with full support for features like IVR that
make SAN extension more reliable.

1-46 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Enables “green” SAN
deployments and provides the
ideal switch for the Rack Deployment
environmentally and energy-
conscious customer. 1RU

- Performance-optimized Server
• 768 Gbps of aggregate
Server
bandwidth in 1 RU
- Space-optimized Server

• 48 8-Gbps line-rate ports in Server

1RU
- Low power consumption Server

• Low power consumption for 48 DiskArray

Disk Array
ports
Disk Array
• High performance with low
power in 1 RU

The MDS 9148 Multilayer Fabric Switch enables “green” SAN deployments by having high
performance while consuming low power for its class of product.
As shown in the figure, the MDS 9148 Multilayer Fabric Switch is only 1 RU high and is
packed with 768 Gbps of aggregate bandwidth. These features mean that the switch is
optimized to give high performance.
The switch is also space-optimized by having 48 ports that are packed in a small footprint. This
fabric switch has 48 ports in 1 RU and is less than 2 feet deep.
The MDS 9148 Multilayer Fabric Switch consumes a small amount of power for an 8-G switch
having 48 ports or more. The switch offers the highest port density and lowest power
consumption available in the industry in a compact 1-RU chassis form factor.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-47
Summary
This topic summarizes the key points that were discussed in this lesson.

• The Cisco MDS 9000 Series of components includes director-class

switches, fabric switches, chassis switches, line-card modules, service
modules, supervisor modules, and fabric modules.
• Cisco MDS 9700 Series director-class switches support unique
supervisor, fabric, and switch modules.
• Cisco MDS 9500 Series director-class switches supports 1-, 2-, 4-,
8-, 10-, and 16-Gbps modules.
• Cisco MDS 9200 Cisco Fabric Switches include the MDS 9222i
Multiservice Modular Switch and the MDS 9250i Multiservice Fabric
Switch.
• The Cisco MDS 9100 Fabric Switches include the MDS 9148 Multilayer
Fabric Switch.

1-48 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 2

Implementing Integrated
Management
Overview
This lesson describes and explains the configuration and use of the Cisco Nexus Operating
System (NX-OS) on the Cisco MDS 9000 Series and Cisco Nexus product lines. The lesson
also describes the features and applications of Cisco Prime Data Center Network Manager
(DCNM) for SAN (DCNM-SAN). This lesson closes with a description of Cisco Performance
Manager and Cisco Traffic Analyzer.

Objectives
Upon completing this lesson, you will be able to describe Cisco NX-OS management services
provided by the Cisco MDS 9000 Series switches and identify access to the CLI through the
console and Mgmt0 interfaces. This ability includes being able to meet these objectives:
 Describe the Cisco NX-OS
 Describe the Cisco NX-OS CLI command set and its structure
 Describe Cisco Prime DCNM and its management scope and management functions
 Describe the various components of Cisco Prime DCNM management tools
 Describe licensing for Cisco Prime DCNM
 Describe Cisco MDS Device Manager
 Describe Cisco Prime DCNM Federated Server
 Explain virtual machine (VM) awareness in Cisco Prime DCNM
Cisco NX-OS
This topic describes the Cisco NX-OS.

• Modular software architecture

• In-service software upgrades
• Protected and restartable
processes
• Process survivability
• Role-based administration
• Programmable XML API

The Cisco NX-OS was built for data center environments, with features that include the
following:
 A modular software architecture
 In-service software upgrades (ISSU)
 Protected and restartable processes
 Process survivability
 Role-based administration
 A programmable application programming interface (API) using XML
Features and functionality up to Cisco NX-OS Release 6.2(3) are discussed in this course.

1-52 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Cisco NX-OS was designed as an operating system for the data center.
- Targeted primarily for the data center, but also applicable to other Ethernet
environments
- Integrates multiple technologies

SAN Layer 2
Future Layer 3
Protocols Protocols
Protocols Protocols

Cisco
NX-OS

- Initially runs on the Cisco Nexus and MDS hardware platforms

- Designed for portability to other platforms
- Built with distributed forwarding and hardware abstraction layer
- Underlying operating system based on an open platform

The figure describes the Cisco NX-OS design goals. The goal of Cisco NX-OS is to divorce
protocols from the hypervisor and supervisor. This ability makes future implementation of new
protocols easier and more efficient.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-53
• Services and features developed as modules
• Easily extensible for future features

Layer 2 Protocols Layer 3 Protocols Storage Protocols Other Services

SNMP, XML, CLI Management

and Services
VLAN mgr UDLD
Sysmgr, PSS, and MTS
OSPF GLBP
STP CDP BGP HSRP VSANs
Zoning
IGMP SNP 802.1X EIGRP VRRP
FCIP
LACP CTS PIM SNMP FSPF
OTV … IVR

Future Services
Protocol Stack (IPv4, IPv6, L2)

Interface Management
Independent, memory-protected restartable processes:
Chassis Management
• Stateful restart with PSS
• Chip Nonstop
Stateful restart with Cisco and Driver Infrastructure
Forwarding and graceful restart
• Stateless restart
• Multiple service instances Kernel

The Cisco NX-OS kernel performs the hardware and control functions, and builds separate
processes for Layer 2 and Layer 3 protocols, including Fibre Channel over Ethernet (FCoE).
Cisco NX-OS is a next-generation operating system that brings three fundamental technologies
into a single platform:
 Layer 2 classical and unified I/O switching
 Layer 3 multiprotocol routing
 Other protocols can be added in the future

The design philosophy of Cisco NX-OS includes the following:

 Invest in sophisticated software infrastructure so that multiple features can leverage that
infrastructure
 Manage software complexity that is growing all the time
 Focus on serviceability
 Provide comprehensive management that extends well beyond a CLI using a wizard-based
GUI
 Modularity is paramount

Cisco NX-OS can restart processes. Restartability means that a process can restart
automatically when necessary. Protocols, table managers, and different subsystems all run as
distinct memory-protected restartable processes. The process is started only when the feature is
configured. Service restartability using persistent storage service (PSS) registers states to PSS
and recovers states from PSS upon restart. Stateful restart with Cisco Nonstop Forwarding
(using graceful restart) recovers states that are based on information from other services or
networks, mainly routing protocols. Stateless restart causes a fresh start without traces of
former instances.

1-54 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• A multithreaded and modular • Complete separation of control
control plane enables control plane and data plane.
plane performance to scale with - No interruption of forwarding for
symmetric multiprocessing and control plane events
multicore CPUs. - Also known as Cisco Nonstop
- Faster route convergence Forwarding
- Lower mean time to recovery - Fully distributed, hardware-based
switching
• Modular code with real-time
preemptive scheduling enables
Control Plane time-critical functions to run. For
example, Fabric Shortest Path
First (FSPF) Hello messages
run while the system is
UPC Data Plane UPC operating at 100 percent CPU
load.

The control and data planes of the switch are separated for greater availability. All switching is
done in hardware and is separate from the supervisor control plane. Because of this separation,
traffic still forwards even in the event of a control plane failure.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-55
Cisco NX-OS CLI Command Set and Structure
This topic describes the Cisco NX-OS CLI command set and its structure.

• The CLI can be accessed through multiple connections and protocols.

- Direct console with serial link: VT100
- Terminal Telnet: TCP/IP over Ethernet or Fibre Channel
- Secure Shell access: SSH (encrypted), TCP/IP over Ethernet or Fibre
Channel
- In-band management: IPFC over a Fibre Channel link

Web/SNMP
(DCNM SAN-Client, Unified
Web Client)

Out-of-Band
Management Fibre Channel
(Ethernet Port) Port Fibre
Channel
IP Network Fabric
Console
Port In-Band
Management
(IPFC)

Telnet or SSH VT100 Terminal

Many connection options and protocols are available to manage the Cisco MDS 9000 Series
Switches with the CLI. The initial configuration must be done with VT100 console access.
VT100 console access can be achieved via direct connection or serial link connection, such as a
modem. After the initial configuration is complete, you can access the switch using either
Secure Shell (SSH) or Telnet. The SSH Protocol provides a secure, encrypted means of access.
Terminal Telnet access involves a TCP/IP out-of-band (OOB) connection through the 10/100
MB Ethernet port or an in-band connection by way of IP over Fibre Channel.
You can access the Cisco MDS 9000 Series of switches for configuration, status, or
management through the console port. You can initiate a Telnet session through the OOB
Ethernet management port or through the in-band IP over Fibre Channel management feature.
The console port is an asynchronous port with a default configuration of 9600 b/s, 8 data bits,
no parity, and 1 stop bit. This port is the only means of accessing the switch after the initial
power-up until an IP address is configured for the management port.
After an IP address is configured, you can use Telnet to connect to the switch through the
Mgmt0 interface on the supervisor card.
In-band IP over Fibre Channel (IPFC) is used to manage remote switches through the local
Mgmt0 interface.

1-56 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Features of the CLI include the following:
- Structured hierarchy, which is easier to remember
• Style consistent with Cisco IOS Software
• Commands can be abbreviated
- Help facility
• Context-sensitive help (?)
• Command completion (Tab)
• Command history buffer (using ↕ and ↔ keys)
• Console error messages
- Command scheduler with support for running shell scripts
- Support for command variables and aliases
- Configuration changes must be explicitly saved before reboot
• copy running-config startup-config (abbreviated to copy run start)

The CLI enables you to configure every feature of the switch. More than 1700 combinations of
commands are available and are structurally consistent with the style of the Cisco IOS Software
CLI.
The CLI help facility provides the following assistance:
 Context-sensitive help: Provides a list of commands and associated arguments. Type a
question mark (?) at any time, or type part of a command and type a question mark.
 Command completion: The Tab key completes the keyword that you start to type.
 Console error message: Identifies problems with any switch commands that are
incorrectly entered, so that the commands can be corrected or modified.
 Command history buffer: Allows recalling of long or complex commands or entries for
re-entry, renewing, or correction.
 Cisco MDS command scheduler: Provides a UNIX cron-like facility that allows you to
schedule a job at a particular time or periodically in Cisco NX-OS.

Configuration changes must be explicitly saved, and configuration commands are serialized for
execution across multiple Simple Network Management Protocol (SNMP) sessions. To save
the configuration, enter the copy running-config startup-config command from the config
mode prompt to save the new configuration in nonvolatile storage. After this command is
issued, the running and startup copies of the configuration are identical.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-57
• System-level commands are executed from the CLI.
- The system show command may be executed from any level of CLI.
switch# show interface fc 3/1 capabilities
fc3/1
Min Speed is 2 Gbps
Max Speed is 16 Gbps
FC-PH Version (high, low) (0,6)
Receive data field size (max/min) (2112/256) bytes
Transmit data field size (max/min) (2112/128) bytes
Classes of Service supported are Class 2, Class 3, Class F
Class 2 sequential delivery supported

• Module-specific commands are executed from a specific module.

- Module commands are accessed after switching to a module.
switch# attach module 3
Attaching to module 3 ...
To exit type 'exit', to abort type '$.'
Wind River Linux glibc_small (standard) 3.0
module-3#

CLI commands can be system-specific and module-specific. To use module-specific

commands, use the command attach module x where x is the number of the module.

1-58 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• EXEC mode
- Show system information and run debug.
- Copy and delete files, and get directory listing for bootflash.
• Configuration mode
- Configure features that affect the switch as a whole.
• Configuration submode
- Configure switch subparameters.

Switch Prompt (switch#)

EXEC
Mode

exit
show copy dir debug config terminal
Config
flogi fcns bootflash: slot0: fspf interface fcdomain zoneset Mode

end
exit
Config
database database port-channel fc fcip iscsi mgmt 0 Submodes

exit
switchport shut no shut ip address

Cisco MDS 9000 Series switches have three command mode levels:
 User EXEC mode
 Configuration mode
 Configuration submodes

The commands available to you depend on the mode that you are in. To obtain a list of
available commands, type a question mark (?) at the system prompt.
From the EXEC mode, you can perform basic tests and display system information, including
operations other than configuration, such as show and debug. Show commands display system
configuration and information. Debug commands enable printing of debug messages for
various system components. Show commands may be executed from any command mode
level.
Use the config or config terminal commands from EXEC mode to enter configuration mode.
Configuration mode has a set of configuration commands that can be entered after a config
terminal command to set up the switch.
CLI commands are organized hierarchically, with commands that perform similar functions
that are grouped under the same level. For example, all commands that display information
about the system, configuration, or hardware are grouped under the show command. All
commands that allow you to configure the switch are grouped under the config terminal
command, which includes switch subparameters at the configuration submode level.
To execute a command, enter the command by starting at the top level of the hierarchy. For
example, to configure a Fibre Channel interface, use the config terminal command. After you
are in configuration mode, issue the interface command. When you are in the interface
submode, you can query the available commands for the interface submode.

GW 10.0.7.254
NMS
mgmt0 10.0.7.5

# config
(config)# interface mgmt 0
(config-if)# ip address 10.0.7.5 255.255.255.0
(config-if)# no shutdown
(config-if)# exit Console
(config)# ip default-gateway 10.0.7.254
(config)# end Ctrl-Z performs
# the same function

Management Interface Configuration

The initial setup utility prompts for the management interface (mgmt0) IP address and mask,
plus the default gateway for the management subnet. Experienced administrators can forego
running the initial setup if they know the command syntax to create the settings.
The figure displays an example of manually configuring the mgmt0 IP address and default
gateway:
Step 1 Enter configuration mode. You can abbreviate the command syntax as # con.
MDS# config terminal
Step 2 Enter the interface configuration mode on the specified interface (mgmt0).
MDS(config)# interface mgmt 0
Step 3 Enter the IP address with a 24-bit subnet mask and activate the port.
MDS(config)# ip address 10.0.7.5 255.255.255.0
MDS(config-if)# no shutdown
Step 4 Return to configuration mode. This step is optional because any config-mode
command may be invoked without exiting interface mode.
MDS(config-if)# exit
Step 5 Configure the default gateway address and return to EXEC mode.
MDS(config)# ip default-gateway 10.0.7.254
MDS(config)# end

1-60 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Command aliases replace complex command strings with an alias
name.
- Command aliases persist across reboots.
- Commands that are being aliased must be typed in full, without abbreviation.
- Command aliases always take precedence over CLI keywords.

switch(config)# cli alias name wr copy running-config startup-config

switch(config)# exit
switch# wr
[########################################] 100%
Copy complete, now saving to disk (please wait)...
switch#

Some commands are quite long or used frequently. An example of such a command is copy
running-config startup-config command, which can sometimes be shortened to wr.
Shortening commands is sometimes useful to group several commands and subcommands
together. Command grouping can be done using command aliases.
Command aliases are saved in NVRAM, so the aliases can persist across reboots.
When creating an alias, you must type the individual commands in full, without abbreviation.
If you define an alias, the alias takes precedence over CLI keywords starting with the same
letters, so be careful when using abbreviations.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-61
• The command scheduler schedules configuration and maintenance jobs
in any Cisco MDS 9000 Series switch.
- Schedule jobs on a one-time basis or periodically.
• One-time mode: The job is executed once at a predefined time.
• Periodic mode: The job is executed daily, weekly, monthly,
or delta (configurable).
- The Cisco MDS 9000 Series switch date and time must be accurately
configured.
- Scheduled jobs can fail if an error is encountered, for example, if a license has
expired or if a feature is disabled.
- All jobs are executed noninteractively.

The Cisco MDS NX-OS provides a UNIX cron-like facility that is called the command
scheduler, to which the following principles apply:
 Jobs can be defined by listing several commands that should be executed in sequence.
 Jobs can be scheduled to run at the same time every day, week, or month, or at a
configurable frequency (delta).
 All jobs are executed noninteractively, that is, without administrator response.

A job can fail if a command that is issued is disabled or no longer supported, because a license
might have expired. The job fails at the point of error, and all subsequent commands are
ignored.

1-62 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Command Function
# copy run start Save active configuration in NVRAM
# dir bootflash: List files stored on bootflash
# erase bootflash:temp Erase file stored on bootflash
# copy slot0:tmp bootflash:temp.txt Copy file and change the name
# debug flogi Monitor all fabric login (FLOGI) operations
# no debug all Switch off debugging
# show tech-support Gather switch information for support
# show tech-support > tempfile Save output in volatile:tempfile
# gzip volatile:tempfile Compress tempfile
# copy volatile:tempfile slot0:temp Copy file to external flash card
# tac-pac Run show tech-support, zip output, copy to volatile:
# config t Enter configuration mode to change settings
(config)# int fc x/y Configure a specific interface
(config-if)# switchport speed 1000 Configure as 1-Gbps port
(config-if)# where Display CLI tree context
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-16

The top part of the table that is shown in the figure lists useful commands that can be entered in
EXEC mode. Changes to the configuration can be made only by entering configuration mode
first and then entering the appropriate commands.
More information can be found by referring to the Cisco MDS Command Reference Guide.

Command Function
# show environment power Check power ratings
# show interface Summary of all interfaces
# show interface fc x/y View detailed information about an interface
# show module View detailed status about all modules
# show hardware View detailed hardware status
# show version View current software versions
# show license usage List installed licenses and status
# show running-config View active switch settings
# show VSAN List all created VSANs
# show VSAN membership List interfaces by VSAN
# show zoneset active Show all active zones and zone sets
# show flogi database List all devices logged in to MDS
# show fcns database List all name server entries
# show cli history List your recent command sequence

Because show commands are too extensive to list comprehensively, the figure lists only some
of the most commonly used. You can find more information in the Cisco MDS Command
Reference Guide.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-63
switch# show redundancy status switch# show environment
Redundancy mode Power Supply:
--------------- Voltage: 50 Volts
administrative: HA Power Actual Total
operational: HA Supply Model Output Capacity Status
(Watts)(Watts)
This supervisor (sup-5) ------- ------------ ------- -------- -----
----------------------- 1 DS-CAC97-3KW 460 W 3000 W Ok
Redundancy state: Active 2 DS-CAC97-3KW 0 W 0 W Shutdown
Supervisor state: Active 3 DS-CAC97-3KW 0 W 0 W Shutdown
Internal state: Active with HA standby 4 DS-CAC97-3KW 0 W 0 W Shutdown
5 DS-CAC97-3KW 464 W 3000 W Ok
Other supervisor (sup-6) 6 DS-CAC97-3KW 0 W 0 W Shutdown
------------------------
Redundancy state: Standby Actual Power
Module Mod Draw Allocated Status
Supervisor state: HA standby (Watts)(Watts)
Internal state: HA standby ----- ----------- ------- -------- ---------
3 DS-X9448-768K9 262 W 650 W Powered-Up
System start time: Tue Feb 5 01:11:54 2013 5 DS-X97-SF1-K9 N/A 190 W Powered-Up
6 DS-X97-SF1-K9 N/A 190 W Powered-Up
System uptime: 3 days, 3 hours, 5 minutes, 8 Xb1 xbar N/A 150 W Absent
seconds Xb2 xbar N/A 150 W Absent
Kernel uptime: 3 days, 3 hours, 7 minutes, 45 Xb3 xbar N/A 150 W Absent
seconds Xb4 DS-X9710-FAB1 60 W 150 W Powered-Up
Active supervisor uptime: 3 days, 3 hours, 5 Xb5 DS-X9710-FAB1 60 W 150 W Powered-Up
minutes, 8 seconds Xb6 DS-X9710-FAB1 61 W 150 W Powered-Up
fan1 DS-C9710-FAN 60 W 600 W Powered-Up
fan2 DS-C9710-FAN 67 W 600 W Powered-Up
fan3 DS-C9710-FAN 60 W 600 W Powered-Up

The figure provides examples of show redundancy status and show environment commands
on the Cisco MDS 9710 Multilayer Director.

1-64 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco Prime DCNM
This topic describes Cisco Prime DCNM management scope and management functions.

Unprecedented visibility for converged data centers

• Features • Benefits
- Converged management of data center SAN - Reduces TCO of Cisco data center solutions
and LAN with a single point of management for SAN
and LAN
• Data center dashboards and topology
views - Simplifies operational management of
virtualized data centers
• Proactive monitoring and issue diagnostics
- Simplifies deployment of innovative Cisco NX-
• Performance and capacity monitoring OS features
- Cisco VMpath (VM-network-storage) analytics - Enables easy integration into customer
to simplify troubleshooting operations support systems (OSS) and
- Unified fabric automation (template-based business support systems (BSS) via APIs and
provisioning) dashboard portals.

Cisco Prime DCNM is a GUI management tool for managing LAN and SAN network devices.

Features of Cisco Prime DCNM

Cisco Prime DCNM provides the following assistance in operational management of
virtualized data centers:
 Data center-oriented operational dashboards for SAN and LAN health and performance.
 Proactive monitoring of SAN and LAN health and events and alerts that can be forwarded
to customers.
 Performance monitoring and thresholding.
 Long-term trending and capacity planning.
 Cisco Prime DCNM features an innovative new technology, Cisco VMpath, that provides
unprecedented visibility into the network path starting from a virtual machine (VM)
through the compute and network (SAN) layers and all the way to storage ports and
network services. Cisco VMpath, coupled with end-to-end performance views, enables
quick troubleshooting of performance bottlenecks and network issues.
 Cisco Prime DCNM streamlines the provisioning of the unified fabric with template-based
provisioning for all key Cisco NX-OS features, including multihop FCoE.
 Cisco Prime DCNM provides comprehensive coverage for the provisioning and monitoring
of data center networks that are built using Cisco Nexus, Cisco Unified Computing System
(UCS), and Cisco MDS 9000 Series products. Cisco DCNM is the solution that is
recommended by Cisco for managing mission-critical data centers.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-65
Benefits of Cisco Prime DCNM
 By providing a single pane of glass visibility across both SAN and LAN environments,
Cisco Prime DCNM lowers the total cost of operating (TCO) Cisco data centers.
 Features such as operational dashboards, performance views, and Cisco VMpath enable
proactive monitoring and diagnostics. These features simplify operational management of
virtualized data centers.
 Cisco Prime DCNM provides coverage for deploying a wide variety of Cisco NX-OS
innovations, using easy-to-use wizards and templates (for scale, consistency, and
programmability).

All these features can be accessed using very intuitive web-based dashboards or by using a
robust set of web services APIs that can integrate with enterprise management systems.

1-66 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• One converged product
- SAN and LAN health and performance dashboards
- Can be licensed for SAN and/or LAN
- Common operations (discovery, topology)
- Common database
- Single installer, RBAC Cisco Prime DCNM
- Consistent licensing model (licenses on server)
FM Cisco DCNM
DB
Unified Web Client
LA
DCNM for LAN
Fabric Manager DB
FMS
DB LAN and SAN
DCNM for SAN
2002 2008 2011+
Evolution
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-21

Cisco Prime DCNM is a converged management platform that combines the best features of
Cisco Fabric Manager and Cisco DCNM.
IT departments today are challenged to look beyond traditional silos of networking and storage
to manage the converged, virtualized data center. Meeting this challenge calls for unification of
the management plane to enable holistic management of the data center infrastructure.
Recognizing the need to support this convergence in management, Cisco has evolved and
merged management solutions Cisco Fabric Manager and Cisco DCNM for SAN and Cisco
DCNM for LAN, into one unified product called Cisco Prime DCNM.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-67
• Performs discovery and management across the following product
families running Cisco NX-OS Software
Cisco UCS Cisco MDS Cisco Nexus Series
Series Switches Series Switches Switches
• Cisco UCS 6200 • Cisco Nexus 1000V
Series Fabric • MDS 9700 • Cisco Nexus 1010
Interconnect • MDS 9500 • Cisco Nexus 2000 FEX
• MDS 9200 • Cisco Nexus 3000
Cisco Catalyst
• MDS 9100 • Cisco Nexus 4000
Series Switches
 Catalyst 6500 • Cisco Nexus 5000
 Cisco FWSM • Cisco Nexus 6000
• Cisco Nexus 7000

Cisco Cisco
Nexus 2000 Nexus 5000
Cisco
Nexus 1000V
Catalyst 6500
MDS 9500/9700 Cisco Nexus
Cisco Cisco
Nexus 3000 Nexus 4000 7000

Cisco Prime DCNM combines the discovery and management of Ethernet and storage networks
into a single dashboard. This combination helps network and storage administrators manage
and troubleshoot health and performance across the following product families that run Cisco
NX-OS Software:
 Cisco MDS 9700 and 9500 Series Multilayer Directors, and Cisco MDS 9200 and 9100
Series Multilayer Switches
 Cisco Nexus 1000V Series Switches and Cisco Nexus 1010 Virtual Services Appliances
 Cisco Nexus 2000 Series Fabric Extenders
 Cisco Nexus 3000, 4000, 5000, 6000, and 7000 Series Switches

Cisco Prime DCNM provides limited support for the following Cisco platforms:
 Cisco Catalyst 6500 Series Switches
 Cisco Catalyst 6500 Series Firewall Services Module (FWSM)
 Cisco UCS 6200 Fabric Interconnect

1-68 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Configuration/Provisioning
• Single Fabric Discovery
• Topology…

Heterogeneous Storage Array Discovery

Scale-out federation architecture
SAN host path redundancy analysis
Advanced Automatic fabric failover
Edition VMware vCenter plug-in
License that Multiple fabric management
unlocks advanced Centralized management server with discovery
features of Cisco Continuous health and event monitoring
DCNM Historical performance monitoring and reporting
Event forwarding
Cisco DCNM proxy services
Configuration backup, archive, and compare
Roaming user profiles
Cisco VMpath analytics

Cisco Prime DCNM

Standalone
(converged) Essentials Edition (Free)
Fibre Channel/FCoE/FICON/iSCSI Topology
One package
Edition (Free) View
Features that come Fabric, device, and summary views
at no cost as part of Port, switch, and fabric-level configuration
the product install Advanced
Event and security management
(unlocked) Edition
Configuration analysis tools
(Licensed)
Network diagnostic and troubleshooting tools
Real-time performance monitoring
One command multiswitch CLI access
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-23

The standalone editions of Cisco Prime DCNM-LAN and Cisco Prime DCNM-SAN are
offered free. The advanced editions are licensed products.
Cisco Prime DCNM can be licensed to manage a combination of SAN and LAN environments.
Administrators still maintain control and segmentation through role-based access control
(RBAC), now with single-pane visibility across the network and storage access infrastructure.
Depending on licensing, different features for management of SAN and LAN infrastructure are
available.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-69
• For managing the entire fabric
- Switch-embedded Java application
- Fabric-based discovery and topology
mapping
- Embedded Cisco Device Manager
- Robust event monitoring and alerts
- Configuration wizards to simplify
deployment:
• Zoning
• Inter-VSAN routing
• ISL aggregation (port channel)
• FCIP
• Fibre Connection (FICON)
- Diagnostic tools to simplify fault prevention
and mitigation
• Zone merge analysis
• End-to-end connectivity analysis
• Switch health analysis checks
• Fabric configuration analysis
• Fibre Channel ping and traceroute
- SNMPv3 for secure communication

Cisco Prime DCNM is an SNMP-based device-management application. The application has a

Java web-based GUI to view and configure multiple Cisco MDS 9000 Series directors and
fabric switches, and Cisco Nexus switches. The software is downloaded to the end users,
management, and workstations. Secure SNMP version 3 (SNMPv3) communications are used
to get and set switch parameters.
The Cisco Prime DCNM-SAN Client provides three management views and a Performance
Manager traffic analysis interface.
 Fabric View displays a map of your network fabric, including Cisco MDS 9000 Series
switches, hosts, and storage devices.
 Device View displays a graphic representation of the switch configuration and provides
access to statistics and configuration information for a single switch.
 Summary View displays a summary of xE Ports (Inter-Switch Links [ISLs]), Fx Ports
(fabric ports), and Nx Ports (attached hosts and storage) on a single switch.
 Performance Manager provides detailed traffic throughput reporting. This data is
compiled in various graphs and charts, which can be viewed with any web browser.

Cisco DCNM-SAN Client discovers network devices and creates a topology map with virtual
storage area network (VSAN) and zone visualization. VSAN or zone and switch trees are also
available to simplify configuration. Immediately after the fabric view is opened, the discovery
process begins. The Cisco DCNM-SAN Client can draw a fabric topology in a user-
customizable map. The client creates the map using information that is gathered from a seed
Cisco MDS 9000 Series switch. The information that is gathered includes name server
registrations and Fibre Channel Generic Services 3 (FC-GS-3) fabric configuration server
information. Because of the source of this information, any third-party devices, such as other
fabric switches that support Fibre Channel Generic Services (FC-GS) and FC-GS-3 standards,
are discovered and displayed on the topology map. Vendor Organizationally Unique Identifier
(OUI) values are translated to derive the manufacturer of third-party devices.

1-70 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Connectivity can be browsed
with the menu tree.
• Links and switches are
highlighted on the map.
• You can double-click the menu
or map objects to see attributes.
• Map refresh and purge removes
down elements.
• Nodes and links can have the
following status:
- Down: red X
- Unmanageable: red slash (/)
- Alarm: orange dashed X

The menu tree is used to view fabric connectivity where links and switches are highlighted on
the topology map. Object attributes can be seen by double-clicking the associated menu or map
object. On the topology map, ISL colors change based on utilization. Black is normal, orange is
greater than the low-utilization threshold, and red is greater than the high-utilization threshold.
Slashes (\) and Xs on the map are used as follows:
 Orange, dashed X: A switch that has an alert warning, for example, when a port channel
member is down
 Red X: A switch that is unmanageable and nonfunctional, such as with a link down
condition
 Red \: A switch that is unmanageable but functional

Down elements can be removed from the map with a map refresh or purge.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-71
• A fabric pane can be filtered by
group.
- Hosts and storage devices
• Initiators and targets
• Enclosures created
• Distributed device aliases
provide friendly names
- Storage shows LUNs reported
• LUN0 inquiry

The figure describes the connectivity of hosts and storage.

1-72 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco Prime DCNM Components
This topic describes the various components of the Cisco Prime DCNM management tools.

Unified Web Client

Discover Discover

Cisco DCNM-SAN Java Client Device Cisco DCNM-LAN Java Client

Manager
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-28

The Cisco Prime DCNM framework consists of various components including Unified Web
Client, LAN Java Client, SAN Java Client, and Device Manager. Each of these components
will be discussed.
The Java Remote Method Invocation (RMI) API performs the object-oriented equivalent of
Remote Procedure Calls (RPCs), with support for direct transfer of serialized Java objects and
distributed garbage collection. The Cisco Prime DCNM-LAN Client and Cisco Prime DCNM-
SAN Client use Java RMI for communications with the Cisco Prime DCNM servers. SNMP is
used initially for communications between the Cisco Prime DCNM server and switches. SSH
and XML are also used extensively. Authentication can be provided locally or through
RADIUS, TACACS+, or Lightweight Directory Access Protocol (LDAP). External database
access is through JDBC, a Java-based data access technology.
Cisco Prime DCNM Release 6.2(x) supports the following databases:
 Oracle 10g and Oracle11g Express (XE), Standard, and Enterprise Editions, and Oracle 11g
Real Application Cluster (RAC)
 PostgreSQL 8.3

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-73
• LAN and SAN inventory, health,
and performance:
- Unified discovery
ISL, Host, Target
- Dashboard views include Fabric Health Inventory Performance
summary, switches, hosts, and
storage enclosures
- Topology and path analytics
- Inventory and performance views
- Configuration archive and restore
- SAN inventory, health and
performance reports
- Based on HTML and Flash

ISL Monitoring Top Talkers Fabric Topology

With the web client you can monitor switch events, performance, and inventory from a remote
location using a web browser:
 Performance Manager summary reports: Performance Manager summary reports
provide a high-level view of your network performance. These reports list the average and
peak throughput and provide hot-links to additional performance graphs and tables with
additional statistics. Both tabular and graphical reports are available for all interconnections
that are monitored by Performance Manager.
 Performance Manager drill-down reports: Performance Manager can analyze daily,
weekly, monthly, and yearly trends. You also can view the results for specific time
intervals using the interactive zooming functionality. These reports are only available if
you create a collection using Performance Manager and start the collector.
 Zero maintenance database for statistics storage: No maintenance is required to
maintain Performance Manager’s round-robin database, because its size does not increase
over time. At prescribed intervals, the oldest samples are averaged (rolled-up) and saved. A
full two days of raw samples are saved for maximum resolution. Gradually, the resolution
is reduced as groups of the oldest samples are rolled up together.

1-74 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Search server by name, world
wide name (WWN), and alias
• Server details
• Server to storage path
• Server traffic statistics
• Server port events
• VM-centric view per VMware
vSphere

The figure describes the dashboard of the Unified Web Client.

• Type in one place to apply command to many devices.

• Can be used to perform an inquiry across multiple devices using show
commands.
• Can be used to perform a configuration command across
multiple devices simultaneously.

The figure shows CLI access to many devices (multiple switches) using a single command.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-75
• Similar to Cisco Fabric Manager
• Per-fabric discovery
• Uses SNMP (v2c or v3) and
SSH
• Per-VSAN discovery
• Cisco DCNM discovers the
entire fabric
• Cisco DCNM-SAN client
modifies the switch configuration
• Synchronization by Cisco
DCNM-SAN Client
- Relies on traps sent by the switch SAN Java
Client
- Incremental per-fabric
resynchronization every 300
seconds
- Full rediscovery every 5 hours

The figure describes SAN discovery.

1-76 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• SAN Fabric Management (Fibre Channel, FCoE, FICON, iSCSI, FCIP)
• VSAN and zone resources highlight
• Protocol-specific link display
• Per-link attributes display
• Wizard-based configuration
• Topology views and path display
• Troubleshooting tools
• Events and performance

The Cisco Prime DCNM-SAN Client is a Java and SNMP-based network fabric and device
management tool with a GUI that displays real-time views of your network fabric. These views
include Cisco Nexus 5000 Series switches, Cisco MDS 9000 Series switches, and third-party
switches, hosts, and storage devices. The Cisco Prime DCNM-SAN Client provides Fibre
Channel troubleshooting tools, and complete configuration and status monitoring capabilities
for MDS 9000 Series switches and Cisco Nexus 5000 Series Switches. You can use these
health and configuration analysis tools on the MDS 9000 Series switches or Cisco Nexus 5000
Series switches to perform Fibre Channel ping and traceroute.
On the topology map, ISL colors change based on utilization. Black is normal, orange is greater
than the low-utilization threshold, and red is greater than the high-utilization threshold.
Slashes (\) and Xs on the map are used as follows:
 Orange, dashed X: A switch that has an alert warning, for example, when a port channel
member is down
 Red X: A switch that is unmanageable and nonfunctional, such as with a link down
condition
 Red \: A switch that is unmanageable but functional

Down elements can be removed from the map with a map refresh or purge.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-77
Licensing Cisco Prime DCNM
This topic describes the licensing requirements for Cisco Prime DCNM.

Type of License Applies to:

FC and FCoE
MDS 9100
Licensing is Cisco Prime SAN MDS 9200
MDS 9500
determined by DCNM Advanced MDS 9700
the following: Edition Cisco Nexus 5000
Cisco Nexus 6001

Essentials Edition Free

• Switch SAN Client Cisco Nexus 6004
Cisco Nexus 7000
platform Cisco Nexus 7700

• Number of SAN and LAN FC, FCoE, L2 and

units Advanced L3
Cisco Nexus 5000
• Modular vs. LAN and
Edition Cisco Nexus 6001
fixed SAN Cisco Nexus 6004
Cisco Nexus 7700
• LAN vs. SAN
• Essentials vs. L2 and L3
Advanced LAN Advanced Cisco Nexus 3000
Cisco Nexus 5000
LAN Client Edition Cisco Nexus 6001
Cisco Nexus 6004
Cisco Nexus 7000
Cisco Nexus 7700

Cisco Prime DCNM can be licensed for SAN and LAN environments separately or together.
Most Cisco Prime DCNM features used for provisioning and discovery are available for free
with the base image (the Essentials edition). Advanced features require a license to unlock the
features. Cisco Prime DCNM is priced by switch platform and licensed per number of switches
on the platform. In addition, a yearly service contract is required to upgrade from one major
release to another.

1-78 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Features Cisco DCNM Essential Cisco DCNM Advanced
Edition (Free) Edition (Licensed)
Fibre Channel/FCoE/FICON/iSCSI topology view Yes Yes
Fabric, device, and summary views Yes Yes
Port, switch, and fabric-level configuration Yes Yes
MDS event and security management Yes Yes
Configuration wizards Yes Yes
MDS configuration analysis tool Yes Yes
Network diagnostic and troubleshooting tools Yes Yes
Real-time performance monitoring Yes Yes
Multiple fabric management (concurrently) Yes
Centralized management server with discovery Yes
Web client for operation view Yes
Continuous health and event monitoring Yes
Historical performance monitoring and reporting Yes
Event forwarding Yes
Cisco DCNM proxy services Yes
Cisco fabric analyzer integration Yes
Roaming user profiles Yes
Cisco VMpath analytics Yes
Dashboards (summary, host, switch, network) Yes
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-36

The Cisco Prime DCNM-SAN Client edition of Cisco Prime DCNM is offered free. The
advanced edition is a licensed product.
Cisco Prime DCNM can be licensed to manage a combination of SAN and LAN environments.
Administrators still maintain control and segmentation through RBAC, now with single-pane
visibility across the network and storage access infrastructure.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-79
• Cisco Prime DCNM licenses are now server-based instead of device-
based.
- Previously, licenses were deployed on the switch (embedded).
- Switch licenses are now primarily deployed on the Cisco Prime DCNM server.
• This model enables pooling of licensing in one central location for ease
of management and portability.
• Existing Cisco Fabric Manager licenses remain applicable to help
ensure transparent upgrades.

Licenses are now server-based instead of device-based. Licenses were deployed on the switch
(embedded). Switch licenses are now primarily deployed on the Cisco Prime DCNM server.
This model enables pooling of licensing in one central location for ease of management and
portability. Existing Cisco Fabric Manager licenses remain applicable to help ensure
transparent upgrades.

1-80 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco MDS Device Manager
This topic describes Cisco MDS Device Manager.

• Detailed representation of one Menus and

physical switch at a time Toolbar

• Monitors performance statistics Ability to Sort

in real time Interfaces by
VSAN
• Configures all features except
zoning Line Card and
Interface Status
• Single and multiple port
configuration
Supervisor and
Switch Status

Summary
View

Device Manager provides a graphical representation of one Cisco MDS 9000 Series switch
chassis, including the installed switching modules, the supervisor modules, the status of each
port within each module, the power supplies, and the fan assemblies. Device Manager provides
two views, which are Device View and Summary View.
Use the Summary View to monitor interfaces on the switch.
Use device view to perform switch-level configurations including the following configurations:
 Configuring virtual Fibre Channel interfaces
 Configuring FCoE features
 Configuring zones for multiple VSANs
 Managing ports, port channels, and trunking
 Managing SNMPv3 security access to switches
 Managing CLI security access to the switch
 Managing alarms, events, and notifications
 Saving and copying configuration files and software images
 Viewing hardware configuration
 Viewing chassis, module, port status, and statistics

Device Manager offers an alternative to the CLI for switch configuration. Parameters on single
or multiple switches can be updated rapidly. Fabric View displays the values for parameters for
multiple switches side by side. This feature allows you to copy and paste values from one MDS
9000 Series switch to many others very rapidly. Alternatively, you can configure values on a
single switch through the Device View. In that view, you can open configuration and status
© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-81
dialog boxes by clicking various parts of the Device View, such as switching modules and
ports.
Some of the functions that you can configure with the Device Manager are VSANs, port
channels, Remote Monitoring (RMON) alerts, general switch event filters, SNMP users and
roles, and Fibre Channel services.
Switch status and statistics are displayed in real time by the Cisco Device Manager in all its
views. The views are refreshed at a user-specified interval, which can be as frequent as every
10 seconds. Because of the frequency of refresh that is available, you can monitor switch
health, performance, and error statistics in real time. Historical statistics can be viewed when
the monitor option is deployed.
The device view tab graphically depicts the switch enclosure. The status of the switch cards,
ports, fans, and power supplies can be determined at a glance. Device Manager also provides an
intuitive way to drill down for configuration and to examine specific statistics, as well as view
all the log information.
You can use the License Manager to view license information. This information includes items
such as the name of the feature package, types of licenses that are installed, number of licenses
that are used, and expiration date. You can also use License Manager to install or remove
license key files.

1-82 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco Prime DCNM Federated Server
This topic describes the purpose of the Cisco Prime DCNM federated server.

• Provides a group of two or more Cisco DCNM-SAN servers (maximum

of 10) federating workload management to provide higher levels of
scalability, availability, and reliability.
• Provides the storage administrator with the ability to load balance Cisco
DCNM-SAN server workloads by moving fabrics across the federation.
• Can manage an aggregate of more than 75,000 end devices.
Cisco DCNM Unified Web Client
Cisco DCNM-SAN GUI Client

Cisco DCNM-SAN Cisco DCNM-SAN Cisco DCNM-SAN

Server I Server II Server III

FC FC FC FC
Central
Database

Fabric-1 Fabric-2 Fabric-3 Fabric-4

Cisco DCNM-SAN federated server architecture is an architecture that lets you federate
multiple Cisco DCNM-SAN servers.
Cisco has improved the scalability of an individual instance of Cisco DCNM-SAN server.
Previously, the maximum was 10,000 devices with Cisco NX-OS Release 4.1. Now the
maximum is 15,000 devices per instance of Cisco Fabric Manager Server (FMS).
Cisco DCNM-SAN federated services can now federate up to 10 instances of Cisco DCNM-
SAN server, and each of them manages its own set of devices independently.
For reporting purposes, you can pull the data together and generate aggregated reports from all
of devices. You can also move fabrics across the federation. If one fabric is getting close to the
maximum in one place, you can move the fabric to another place.
Cisco has tested the process of moving fabrics across the federation with up to 75,000 end
devices, and will continue to test larger fabrics.
This feature will become very important because now, with FCoE, every server is implicitly
SAN enabled. Therefore, the ability to move fabrics will become a critical and required feature
in the future.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-83
• The Cisco Prime DCNM control panel lists all discovered fabrics in the
federation.
• You can discover and manage fabrics on any Cisco Prime DCNM server
in the federation.
• Each fabric is associated with its corresponding Cisco Prime DCNM
server.
• You can open fabrics belonging to any Cisco Prime DCNM server in the
federation.

As Ethernet networks are becoming more relevant to storage networking, Fabric Manager
needs to extend the management scope from SAN to incorporate LANs that are relevant to
storage networking paths carved out over Ethernet networks.
FCoE provides a method of transporting Fibre Channel traffic over a physical Ethernet
connection by encapsulating Fibre Channel frames in Ethernet packets.
In Cisco Nexus 5000 Series switches, FCoE is supported on all 10-Gigabit Ethernet interfaces.
Each FCoE interface in the Cisco storage network is configured with a MAC address, a VLAN
ID, and a Cisco Discovery Protocol neighbor. As such, visibility and manageability of these
storage networking paths over Ethernet is a key scalability requirement. Customers looking to
capitalize on the full business benefits of the Cisco Data Center Business Advantage strategy
find such scalability necessary.

1-84 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Users are able to discover switches and Ethernet ISLs through the Cisco
Discovery Protocol.
• Unified discovery discovers all Cisco MDS 9000 Series, Cisco Nexus,
and Cisco Catalyst switches, as well as interconnects (Ethernet, Fibre
Channel, and FCoE).

There have also been enhancements in the area of unified discovery.

In a SAN fabric, discovering storage devices and SAN devices is the only important task. Now,
however, the discovery and mapping of the topology of all MDS 9000 Series and storage
devices, Cisco Nexus Series devices, and Catalyst switches as well as the interconnections
between all these devices is possible. The discovery is done using Cisco Discovery Protocol.
This description is just an example of what the dialogue looks like. This feature becomes more
critical when there are many FCoE devices connected.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-85
• Shared external database
• Suitable for remote high
availability, which requires
reliance on database replication
services
• Explicit assignment of a fabric to
a server
• Manual failover
• Cisco Prime DCNM Release
6.2(1) supports nondisruptive
automated failover for Cisco
DCNM for SAN management
• Secondary server installation
- Select the option to add server to
federation

This figure describes Cisco Prime DCNM-SAN server federation.

1-86 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco Prime DCNM VM Awareness
This topic describes VM awareness in the Cisco Prime DCNM Software.

I/O Stats per Server

VM-Level Drill Down

Service Profile to Map Server

to Physical Blade Server

VM-Aware Path Management

Cisco Prime DCNM extends network visibility to the virtual infrastructure by mapping the
entire path from the VM and switch to the physical storage and Ethernet networks. The Cisco
VMpath feature that is part of the host dashboard provides views to help troubleshoot the
performance of a VM and virtual host while also reporting on the health and performance of the
network and storage paths. The VM-aware dashboard displays performance charts, path
attributes, topology, path alerts, and information about the utilization of VMs and virtual hosts.
The increased visibility into virtualized infrastructure helps IT administrators locate
performance anomalies that may be causing service degradation and eliminate virtual
computing and networking as a root cause of the problem.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-87
• Navigate to VM > Physical Server > Embedded Switch > Network
Switch(s) > ISLs > Target Port.

The figure describes end-to-end Cisco VMpath visualization.

• Increased VM-awareness in Cisco Prime DCNM

- VMware discovery and SAN and LAN correlation of HBAs and NICs
- Association of VM datastores to array ports
- Association of VMs to fabric paths
- One-click access to VMware stats on disk latency, CPU, memory, flow traffic,
and errors
- Host and storage path troubleshooting
• All paths, shortest path, Fibre Channel ping, traceroute
• NPV, IVR, IVR-region-aware tools
- Better on-demand discovery of non-ESX hosts
- Web host dashboard pages that provide a host-centric view of SAN and LAN

The figure describes host path discovery and monitoring.

1-88 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• Cisco NX-OS is a highly available operating system that encompasses the

best features of Cisco MDS SAN-OS and Cisco IOS to provide a unified
management platform.
• The Cisco NX-OS CLI is used to configure and monitor information on the
Cisco MDS 9000 Series switches.
• The CLI has several modes, which include EXEC, global configuration, and
subconfiguration.
• Cisco Prime DCNM has evolved from Cisco Fabric Manager through
individual Cisco DCNM-SAN and Cisco DCNM-LAN clients to a Unified Web
Client with LAN and SAN sharing a common database.
• Cisco DCNM-SAN is an SNMP device management application for viewing
connectivity and configuring multiple Cisco MDS 9000 Series director and
fabric switches.
• Cisco Device Manager offers a GUI-based alternative to the CLI for switch
configuration.
• Cisco Prime DCNM server federation is an architecture that lets you
federate multiple Cisco Prime DCNM servers.
• Cisco Prime DCNM includes VM awareness and end-to-end VM path
discovery.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—1-49

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-89
1-90 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Summary
This topic summarizes the key points that were discussed in this module.

• Cisco MDS 9000 Series switches range from fabric switches to

multiservice switches to director-level switches with ports supporting 1-,
2-, 4-, 8-, and 16-Gb Fibre Channel speeds.
• The Cisco Nexus Operating System is a highly available operating
system that provides a unified CLI configuration platform while Cisco
Prime Data Center Network Manager provides a common GUI-based
management platform across the Cisco MDS and Cisco Nexus Series of
devices.

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-91
Module Self-Check
Use the questions here to review what you learned in this module. The correct answers and
solutions are found in the Module Self-Check Answer Key.
Q1) What is the number of fabric cards in the Cisco MDS 9710 Multilayer Director needed
to provide 768 Gbps per slot? (Source: Introducing Cisco MDS 9000 Series Switches)
A) 8
B) 3
C) 4
D) 6
Q2) Which two Cisco MDS 9000 Series switch line cards have service capability? (Choose
two.) (Source: Introducing Cisco MDS 9000 Series Switches)
A) 16-port SSN
B) 18/4-port MSM
C) 32-port 8-Gb Advanced Fibre Channel Module
D) Supervisor-2A
E) 48-port 16-Gb Fibre Channel module
Q3) Which option supports FCIP? (Source: Introducing Cisco MDS 9000 Series Switches)
A) Cisco 48-port 8-Gb/s Advanced Module
B) Cisco MDS 9148
C) Cisco MDS 9222i
D) Cisco 48-port 16-Gb/s FC Module
Q4) How many port groups are there on a 32-port 8-Gb/s Advanced line card? (Source:
Introducing Cisco MDS 9000 Series Switches)
A) 4
B) 6
C) 8
D) 10
Q5) How many port groups are there on a 48-port 8-Gb/s Advanced line card? (Source:
Introducing Cisco MDS 9000 Series Switches)
A) 2
B) 4
C) 6
D) 8

© 2013 Cisco Systems, Inc. Cisco MDS 9000 Series Switch Platforms 1-93
Q6) Which Cisco MDS 9000 Series tool allows you to monitor switch events, performance,
inventory, and fabric-wide, long-term behavior reporting from a remote location using
a web browser? (Source: Implementing Integrated Management)
A) Cisco Prime DCNM Device Manager
B) Cisco Fabric Manager
C) Cisco Prime DCNM Unified Web Client
D) Cisco Prime DCNM-SAN Client
Q7) What is the maximum number of Fibre Channel ports supported on the Cisco MDS
9710 Multilayer Director? (Source: Introducing Cisco MDS 9000 Series Switches)
A) 512
B) 256
C) 528
D) 384
Q8) How many FCoE ports are supported on the Cisco MDS 9250i Multiservice Fabric
Switch? (Source: Introducing Cisco MDS 9000 Series Switches)
A) 10
B) 20
C) 8
D) 40
Q9) What is the maximum number of full rate 10 Gbps Fibre Channel ports supported on
the 8-Gbps Advanced Modules? (Source: Introducing Cisco MDS 9000 Series
Switches)
A) 6
B) 8
C) 32
D) 24

1-94 Configuring Cisco MDS 9000 Series Switches (DCMDS) v1.0 © 2013 Cisco Systems, Inc.
Self-Check Answer Key
Q1) B
Q2) A, B
Q3) C
Q4) D
Q5) C
Q6) C
Q7) D
Q8) D
Q9) D

System Installation and Initial

Configuration
Overview
In this module, you will explore system installation and perform initial configuration of Cisco
Multilayer Director Switches (MDS).

Module Objectives
Upon completing this module, you will be able to install and configure MDS 9000 Series
switches. You will be able to meet these objectives:
 Describe the boot sequence and how to perform the initial switch configuration using the
CLI, and demonstrate common uses of the CLI
 Describe each of the Cisco NX-OS license packages, what each package provides, and the
process of upgrading or downgrading switch software
2-2 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 1

Performing the Initial Switch

Configuration
Overview
When a Cisco Multilayer Director Switch (MDS) is restarted, the switch goes through a boot
sequence to load the kickstart and system files from the internal bootflash. This lesson
describes the boot sequence and then explains how to perform the initial switch configuration
using the CLI. This lesson also demonstrates common uses of the CLI.

Objectives
Upon completing this lesson, you will be able to perform the initial switch configuration
process and install Cisco Prime Data Center Network Manager (DCNM). This ability includes
being able to meet these objectives:
 Describe the boot sequence and purpose of each of the system memory areas
 Describe the steps necessary to complete the initial setup routine on a newly installed
switch
 Describe the essential CLI show commands
 Explain the setup of the classroom lab environment
Switch Boot Sequence
This topic describes the switch boot sequence and purpose of each of the system memory areas.

• The bootflash contains the kickstart and system images.

• All configuration changes made by CLI, Cisco Prime DCNM-SAN Client,
or Cisco Device Manager are instantly active and held in the running-
config.
• The copy run start command saves the running configuration to the
startup configuration in NVRAM.
• The startup configuration is loaded when the switch is rebooted.
• Temporary files can be stored in the volatile system area.

RAM Memory NVRAM Memory Flash Memory

System NVRAM Bootflash
(internal flash)
• Cisco NX-OS • Boot parameters • Kickstart image
• Linux system space • (Kickstart + system) • System image
copy run • Startup-config
start

Volatile Log Slot 0/USB ports

Temporary file space log file (external flash)
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-4

The Cisco MDS 9000 Series contains an internal bootflash that is used for holding the current
bootable images, which are the kickstart and system images. License files are also stored in the
bootflash. The bootflash can also be used for storing any file, including copies of the startup
configuration. In addition, Cisco MDS 9500 and 9700 Series supervisors have an external
bootflash memory slot or USB slots, which can be used for transferring image files between
switches.
The Linux operating system uses the system RAM memory, and a volatile file system is used
for storing temporary files. Any changes that are made to the switch operating parameters or
configuration are instantly active and held in the running configuration in RAM.
All data that is stored in RAM is lost when the Cisco MDS 9000 Series switch is rebooted, so
an area of NVRAM is used for storage of critical data. The most critical of these factors is the
running configuration for the switch. The running configuration should be saved to the startup
configuration in NVRAM with the CLI copy run start command. The configuration is then
preserved during the switch reboot.
During the switch boot process, the switch must know where to find the kickstart and system
images, and what the images are called. Two boot parameters that point to these two files are
held in NVRAM.

2-4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Both the kickstart and system images must be present for a successful
boot.
• Boot parameters point to the location of kickstart and system images.
• The boot process fails if parameters are wrong or images are missing.
• The install command simplifies the process and checks for errors.

System RAM
System Cisco NX-OS
 Loads Cisco NX-OS Linux system space
 Checks file systems running-config
 Loads startup-config
 switch # prompt
Kickstart
NVRAM
 Loads Linux kernel and drivers Startup configuration
 Gets system boot parameters Boot parameters
 Verifies system image and loads #boot system bootflash:system6x img
Loader  switch (boot) # prompt #boot kickstart bootflash:kickstart6x.img

 Gets kickstart boot parameters

 Verifies kickstart image and loads Bootflash:
 Loader > prompt (internal flash)
BIOS system6x.img
System image
 Runs POST Kickstart image kickstart6x.img
 Runs Loader
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-5

The BIOS on the supervisor module first runs power-on self-test (POST) diagnostics and then
runs the loader bootstrap function.
The boot parameters are held in NVRAM and point to the location and name of both the
kickstart and system images. The loader obtains the location of the kickstart file, usually on
bootflash, and verifies the kickstart image before loading the image.
The kickstart loads the Linux kernel and device drivers and then loads the system image.
Again, the boot parameters in NVRAM should point to the location and name of the system
image, usually on bootflash. The kickstart then verifies the system image and loads the system
image.
Finally, the system image loads the Cisco Nexus Operating System (NX-OS), checks the file
systems, and proceeds to load the startup configuration, containing the switch configuration,
from NVRAM.
If the boot parameters are missing or have an incorrect name or location, then the boot process
fails at the last stage. If this failure happens, the administrator must recover from the error and
reload the switch. The install all command is a script that greatly simplifies the boot procedure
and checks for errors and the upgrade impact before proceeding.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-5
Completing the Initial Setup Routine
This topic describes the steps necessary to use the CLI to complete the initial switch
configuration process.

• Verify that the switch is powered

Cisco MDS 9000
off. Switch
• Cable the VT100 to the switch
console port.
• For terminal setup, use the
following settings:
- 9600 bps
- 8 data bits Console
- No parity Port
- 1 stop bit
- No flow control
• Power the switch on.
VT100
• The switch boots automatically. Terminal
• The switch prompt appears on
the console screen.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-7

The initial setup routine must be performed to allow administrative users to access the switch
for out-of-band (OOB) management. This topic explains the steps that are required to perform
this task.
The Initial Configuration dialog box is run automatically upon reboot when the switch first
comes from the factory. All Cisco MDS 9000 Series switches have the network administrator
as a default user (admin) and the first requirement of the configuration is to set the password for
the admin user. This password needs to have a minimum of eight characters. The default user
must always be admin.
The console needs a rollover RJ-45 cable. For example, there is a switch on the supervisor
module of the Cisco MDS 9500 Series switches that, if placed in the out position, allows the
use of a straight-through cable. The switch is shipped in the in position and is located behind
the LEDs.

2-6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Three mechanisms launch the initial setup utility:
- Automatic from factory.
- After the write erase and reload command sequence.
- Manually, using the setup command:
• The text menu prompts the administrator for required information.
• Press Enter to accept defaults.

write erase
Start device
reload

Set admin
password

Enter
Enter setup Configure the Ctrl-C
setup
command Yes device
script?

No or No or
Ctrl-C Edit Ctrl-C Save Yes
Display config? config?
switch#
prompt Save and
Yes No or
Ctrl-C apply config

The Cisco NX-OS setup utility is an interactive CLI mode that guides you through a basic
configuration (called a startup configuration) of the system. The setup utility allows you to
configure enough connectivity for system management, and to build an initial configuration file
using the System Configuration dialog.
The setup utility is used mainly for configuring the system initially, when no configuration
exists, although the utility can be used at any time for basic device configuration. Any
configured values are kept when you skip steps in the script. For example, if there is already a
configured mgmt0 interface address, the setup utility does not change that configuration if you
skip that step. However, if there is a default value for the step, the setup utility changes to the
configuration using the default and not the configured value.

Note Be sure to configure the IP version 4 (IPv4) route, the default network IPv4 address, and the
default gateway IPv4 address to enable Simple Network Management Protocol (SNMP)
access.

Do you want to enforce secure password standard (yes/no) [y]: y

Enter the password for "admin":1234QWer Strong passwords

Confirm the password for "admin":1234QWer are enforced

--- Basic System Configuration Dialog ---

This setup utility will guide you through the basic configuration of the system.
Setup configures only enough connectivity for management of the system.

*Note: setup is mainly used for configuring the system initially, when no
configuration is present. So setup always assumes system defaults and not the
current system configuration values.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the
remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): y

Create another login account (yes/no) [n]: <Enter>

*The actual script may vary depending on switch model number and Cisco NX-OS version.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-9

The setup utility has the following prerequisites:

 Have a strong password strategy for the network environment.
 Connect the console port on the supervisor module to the network. If dual supervisors are
present, connect the console ports on both supervisor modules.
 Connect the Ethernet management port on the supervisor module to the network. If dual
supervisors are present, connect the Ethernet management port on both supervisor modules.

Once the initial configuration has been created, the switch will be accessible through Cisco
Prime DCNM.
You can enable a secure password standard using the password strength-check command. A
secure password should contain characters from at least three of the following classes:
 Lowercase letters
 Uppercase letters
 Digits
 Special characters

Passwords are case sensitive.

2-8 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configure read-only SNMP community string (yes/no) [n]: <Enter>

Configure read-write SNMP community string (yes/no) [n]: <Enter>

Enter the switch name: PxMDSy (where x is your pod number and y is the switch
number; for example: P10MDS1 or P9MDS1)

Continue with Out-of-band (mgmt0) management config? (yes/no) [y]: <Enter>

Mgmt0 IPv4 address : 10.0.x.N

Mgmt0 IPv4 netmask : 255.255.255.0
Configure the default gateway? (yes/no) [y]: <Enter> mgmt0 Configuration
IPv4 address of the default gateway : 10.0.x.254

Configure advanced IP options? (yes/no) [n]: <Enter>

…
Configure congestion/no_credit drop for fc interfaces? (yes/no)[y]:<Enter>
Enter type of drop to configure congestion/no_credit drop? (con/no) [c]:<Enter>
Enter milliseconds in multiple of 10 for congestion-drop in range [100 - 500],
or [d/default] for default:<d>
Enter mode for congestion/no_credit drop[E/F]:<Enter> Slow Drain Device
Detection and Congestion
Avoidance

At this point, the name of your switch is entered along with the IP address and subnet mask of
the OOB Ethernet management port interface. Without this information, management access to
the switch through the OOB Ethernet port would not be possible.
When there are options to choose with each dialog, you have two choices. Pressing Enter will
accept the choice indicated between the square brackets (for example, [n]), or you can select the
alternative. In the example, “n” (for “no”) was entered at the “Enable IP routing?,” “Configure
static route?,” and “Configure the default network?” prompts where [y] was the current
selection and where these items were not desired in the configuration. However, “Configure the
default gateway?” was desired, so pressing Enter enabled the user to enter an IP address on the
next dialog line. No other options in the example dialog script were changed.
The option Slow Drain Device Detection and Congestion Avoidance is automatically enabled.
Configuration of the congestion/no_credit settings can be modified in the initial setup. For now,
choose the defaults for initial setup.
The default stuck frame timeout value is 500 ms. Cisco recommends that you retain the default
configuration for Inter-Switch Links (ISLs) and configure a value not exceeding 500 ms (100 to
200 ms) for fabric (F) ports. The default is 500 ms.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-9
Enable the ssh service? (yes/no) [y]: <No>
Enable the telnet service? (yes/no) [n]: <Yes> Recommended:
Enable the http-server? (yes/no) [y]: <Enter> Disable Telnet (the instructor may
Configure clock? (yes/no) [n]: <Enter> want Telnet enabled), use SSH,
Configure timezone? (yes/no) [n]: <Enter> and configure NTP service.
Configure summertime? (yes/no) [n]: <Enter>
Configure the ntp server? (yes/no) [n]: <Enter>

Configure default switchport interface state (shut/noshut) [shut]: <Enter>

Configure default switchport trunk mode (on/off/auto) [on]: <Enter>

Configure default switchport port mode F (yes/no) [n]: <Enter>

Recommended
Configure default zone policy (permit/deny) [deny]: <Enter> defaults for locking
down the fabric
Enable full zoneset distribution? (yes/no) [n]: <Enter>

Configure default zone mode (basic/enhanced) [basic]: <Enter>

A Network Time Protocol (NTP) server provides a precise time source (radio clock or atomic
clock) to synchronize the system clocks of network devices. NTP is transported over UDP/IP.
All NTP communications use Coordinated Universal Time (UTC). An NTP server receives its
time from a reference time source, such as a radio clock or atomic clock, attached to the time.
NTP distributes this time across the network. Using NTP is optional but recommended.
Telnet services are enabled to remotely log into the switch. The Domain Name System (DNS)
client on the switch communicates with the DNS server to perform the IP address-to-name
mapping. Setting up the DNS server is optional but recommended.
The final section of the initial configuration procedure sets the default states of interfaces, the
default modes for interfaces, and the designated default zone policy.

2-10 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The system prints a summary of the configuration for your review.
The following configuration will be applied:
password strength-check
switchname P7MDS1
interface mgmt0
ip address 10.0.7.5 255.255.255.0
no shutdown
ip default-gateway 10.0.7.254
ssh key rsa 1024 force
feature ssh
system timeout congestion-drop default mode F
feature http-server
system default switchport shutdown
system default switchport trunk mode on
no system default zone default-zone permit
no system default zone distribute full
no system default zone mode enhanced

Would you like to edit the configuration? (yes/no) [n]: <Enter>

Use this configuration and save it? (yes/no) [y]: <Enter>

[########################################] 100%
Copy complete, now saving to disk (please wait)...

The system prints a summary of the configuration for your review. The configuration that is
printed will be exactly what you entered. Compare the configuration once more with the
information you obtained in the initial setup requirements to verify that you made no typing
errors. If everything was entered correctly, then you do not need to edit.
The system will ask if you would like to edit the configuration that just printed out. Any
configuration changes made to a switch are immediately enforced but are not saved. If no edits
are needed, then you will be asked if you want to use this configuration and save the
configuration as well. Because [y] (“yes”) is the default selection, pressing Enter will activate
this function, and the configuration becomes part of the running-config and is copied to the
startup-config.
This step also ensures that the kickstart and system boot images are automatically configured.
Therefore, you do not have to run a copy command after this process. A power loss will restart
the switch using the startup-config, which has everything saved that has been configured to
nondefault values. If you do not save the configuration at this point, none of your changes will
be updated the next time that the switch is rebooted.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-11
CLI Show Commands
This topic describes CLI essentials and provides examples of common CLI show commands.

switch# show module

Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ --------
3 48 2/4/8/10/16 Gbps Advanced FC Module DS-X9448-768K9 ok
5 0 Supervisor module-1 DS-X97-SF1-K9 active *
6 0 Supervisor module-1 DS-X97-SF1-K9 ha-standby

Mod Sw Hw
--- -------------- ------
3 6.2(0.302) 0.305
5 6.2(0.302) 0.302
6 6.2(0.302) 0.302
..
..
Xbar Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ --------
4 0 Fabric Module 1 DS-X9710-FAB1 ok
5 0 Fabric Module 1 DS-X9710-FAB1 ok
6 0 Fabric Module 1 DS-X9710-FAB1 ok
…

The Cisco NX-OS has a design similar to Cisco IOS with several enhancements:
 The show commands are executed identically from both EXEC mode and configuration
mode.
 The show commands have parser help available regardless of the interface mode that you
are in.

2-12 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
P10-MDS-1# show hardware
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2009, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
BIOS: version 1.0.19
loader: version N/A
kickstart: version 4.2(3)
system: version 4.2(3)
BIOS compile time: 02/01/10
kickstart image file is: bootflash:/m9200-s2ek9-kickstart-mz.4.2.3.bin
kickstart compile time: 10/26/2009 0:00:00 [12/05/2009 05:34:39]
system image file is: bootflash:/m9200-s2ek9-mz.4.2.3.bin
system compile time: 10/26/2009 0:00:00 [12/05/2009 06:14:21]

Hardware
cisco MDS 9222i ("4x1GE IPS, 18x1/2/4Gbps FC/Sup2")
Motorola, e500v2 with 1036308 kB of memory.
Processor Board ID JAE14190GGT
Device name: P10-MDS-1
bootflash: 1023120 kB
Kernel uptime is 0 day(s), 2 hour(s), 22 minute(s), 41 second(s)
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-15

The figure displays output from the show hardware command.

P10-MDS-1(config)# int fc 1/1

P10-MDS-1(config-if)# ?
channel-group Add to/remove from a port-channel
end Go to exec mode
errdisable Error Disable the port
exit Exit from command interpreter
fcdomain Configure fcdomain parameters
fspf Configure FSPF parameters
link-state-trap Enable/disable link state change traps
no Negate a command or set its defaults
out-of-service Put an interface out of service.
pop Pop mode from stack or restore from name
port-license Enable port activation license
push Push current mode to stack or save it under name
rspan-tunnel Configure remote span tunnel interface
shutdown Enable/disable an interface
switchport Configure switchport parameters
where Shows the cli context you are in
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-16

When using the CLI parser help, the Tab key displays a brief list of all available options at the
current branch. The question mark (?) key displays full parser help strings.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-13
switch# show run | ?
cut Print selected parts of lines.
egrep Egrep - print lines matching a pattern
grep Grep - print lines matching a pattern
head Display first lines
last Display last lines
less Filter for paging
no-more Turn-off pagination for command output
sed Stream Editor
sort Stream Sorter
tr Translate, squeeze, and/or delete characters
uniq Discard all but one of successive identical lines
vsh The shell than understands cli command
wc Count words, lines, characters
begin Begin with the line that matches
count Count number of lines
end End with the line that matches
exclude Exclude lines that match
include Include lines that match

Several advanced pipe (|) options are available for CLI output:
 egrep
 less
 no-more
 wc

Multiple levels of pipe are also available.

2-14 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# sho run | egrep ?
WORD Search for the expression
count Print a total count of matching lines only
ignore-case Ignore case difference when comparing strings
invert-match Print only lines that contain no matches for <expr>
line-exp Print only lines where the match is a whole line
line-number Print each match preceded by its line number
next Print <num> lines of context after every matching line
prev Print <num> lines of context before every matching line
word-exp Print only lines where the match is a complete word

Examples of egrep command output options are shown in the figure.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-15
• Saving the configuration in NVRAM
P7MDS1# copy running-config startup-config
[########################################] 100%

• Saving the configuration on a TFTP server

P7MDS1# copy run tftp://10.0.7.1/P7MDS1_running.cfg
Trying to connect to tftp server......

• Restoring the configuration from a TFTP server

P7MDS1# copy tftp://10.0.7.1/P7MDS71_running.cfg run
Trying to connect to tftp server......

Configuration files contain the Cisco NX-OS Software commands that are used to configure the
features on a Cisco NX-OS device. There are two types of software configuration files:
 Running-configuration contains the current configuration and changes in memory.
 Startup-configuration contains the saved configuration features in NVRAM.

To change the running configuration, use the configure terminal command to enter global
configuration mode.
To save those changes to the startup configuration, use the copy running-configuration
startup-configuration command.
A copy of the running configuration can also be saved to a remote server as a backup or to use
when configuring other Cisco NX-OS devices.

2-16 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lab Environment
This topic describes the setup of the classroom lab environment.

To other pods To other pods

Shared MDS 9710s

MDS 9710 MDS 9710

JBOD1
MDS9710-A MDS9710-B

P1 P2

1/1 1/2 1/6 1/6 1/1 1/2

1/7 1/7
1/8 1/8
1/9 1/9
MDS_1
MDS1 MDS2 1/5
1/5 1/14 1/14
10.0.x.5
10.0.x.5 10.0.x.3
1/15 1/15
1/10 G1/2 G1/1 G1/1 G1/2 1/10
P1 P2

1/5 1/5

P2 1/1 1/1 P1
MDS3 MDS4
HOST 1 10.0.x.13 10.0.x.14 HOST 2

E0 10.0.x.1 E0 10.0.x.2
E1 10.1.x.2 E1 10.1.x.6

The figure provides a detailed reference of the lab environment. Use this reference to
familiarize yourself with the layout of components in the lab environment before starting the
labs.
 Cisco MDS Switches 1 and 2 are connected with three 4-Gbps ISLs, one in each port
group, in slot 1. Hosts 1 and 2 contain Fibre Channel host bus adapters (HBAs) that provide
host connectivity to each Cisco MDS switch. Host 1 and Host 2 are preinstalled with Cisco
Device Manager and Cisco Prime DCNM for management of the pod SAN environment.
Labs will instruct students to uninstall and reinstall these programs.
 Cisco MDS Switches 3 and 4 can be put into Cisco N-Port Virtualizer (NPV) mode to
demonstrate the NPV and N-Port ID Virtualization (NPIV) features.
 Two or more dual-ported Fibre Channel disk drives are housed in Just a Bunch of Disks
(JBOD) and dual attached to Cisco MDS Switches 1 and 2.
 Two Gigabit Ethernet ports (G1/1 and G1/2) on MDS1 are connected to their
corresponding ports on MDS2 to support Fibre Channel over IP (FCIP) connectivity and
Internet Small Computer Systems Interface (iSCSI).
 MDS1 interfaces fc 1/1 and fc 1/2 are connected to shared MDS9710-A in each pod.
 MDS2 interfaces fc 1/1 and fc 1/2 are connected to shared MDS9710-B in each pod.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-17
Summary
This topic summarizes the key points that were discussed in this lesson.

• The Cisco MDS 9000 Series Switches contain an internal bootflash that
is used for holding the current bootable images, which are the kickstart
and system images.
• The initial setup asks you to define the default gateway IP address, the
system defaults for Cisco MDS switch interfaces, the switch port trunk
mode, and the zoning policy. Initial configuration requirements include
the administrator username and password, switch name, and
management Ethernet port. Before a switch can be managed remotely,
you must first configure its IP parameters, which are the IP address and
subnet mask.
• The CLI has several modes, which are EXEC, global configuration, and
subconfiguration.
• The lab pod contains two Cisco MDS 9222i Multiservice Modular
Switches, two Cisco MDS 9148 Multilayer Fabric Switches, and two
hosts that are used to manage the switches.

2-18 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 2

Installing and Licensing Cisco

NX-OS Software
Overview
Cisco MDS 9000 Series switches have two types of license models, which are feature-based
licensing and module-based licensing. This lesson describes each of the Cisco NX-OS license
packages, what each package provides, and the process of upgrading or downgrading switch
software.

Objectives
Upon completing this lesson, you will be able to explain the installation and licensing of Cisco
NX-OS Software. This ability includes being able to meet these objectives:
 Describe software licensing practices for Cisco MDS 9000 Series switches
 Describe the Cisco NX-OS image installation and upgrade process
 Explain the Cisco NX-OS version downgrade procedure
Software Licensing
This topic describes the software licensing practices for Cisco MDS 9000 Series switches.

• Feature-based licensing is • Module-based licensing is

applied per chassis. applied per module and engine.
- MDS 9000 Series Enterprise - MDS 9000 Series SAN Extension
Package over IP
- MDS 9000 Series Mainframe - Cisco DMM
Package - MDS 9000 Series IOA
- On-Demand Port Activation
Licenses

Any feature not included in a license package is bundled with the MDS 9000 Series switches
and is provided at no extra charge to you.
The licensing model that is defined for the Cisco MDS product line has two options:
 Feature-based licenses allow features that are applicable to the entire switch. The cost
varies based on per-switch usage.
 Module-based licenses allow features that require additional hardware modules. The cost
varies based on per-module usage. An example is the Cisco MDS 9000 SSN-16 module
using the Fibre Channel over IP (FCIP) feature. Each module requires its own separate
license. If you replace a module that requires a license with a module of the same type
(such as replacing a Storage Services Node [SSN] with another SSN), then the existing
license will support the new module.

Note The Cisco MDS 9222i Multiservice Fabric Switch enables SAN extension features on the
four fixed IP services ports only. If you install a module with IP ports in the empty slot on the
MDS 9222i Multiservice Fabric Switch, a separate SAN extension over IP license is required
to enable related features, such as FCIP, on the IP ports of the additional module.

2-20 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Mainframe (FICON) XRC Acceleration SAN Extension over IP
• FICON Protocol • XRC over FCIP • FCIP Protocol
• FICON CUP • XRC Extension • SAN Routing (IVR) for FCIP Storage Services Enabler
• Fabric Binding • Hardware-Based Compression • FAIS Partner
• Switch Cascading • FCIP Write and Tape Acceleration • SANTap Applications
• Intermixing • SAN Extension Tuner • NASB

I/O Accelerator
• Write Acceleration
• Tape Acceleration
• Deflate Compression

FC FC

FC Enterprise Package FC

• Advanced SAN Security

• Traffic Management (QoS)
• SAN Routing (IVR)
Prime DCNM Advanced
• Multiple Physical Fabrics Data Mobility Manager
• Historical Performance • Heterogeneous Data Mobility
Monitoring • Rate-Adjusted Online Migration
• Web-Based Dashboard

License Packages for Cisco NX-OS Release 6.2(3)

Standard Package
Most MDS 9000 Series software features are included in the base configuration of the switch,
which is the standard package. However, some features are logically grouped into add-on
packages that must be licensed separately, such as the Cisco MDS 9000 Series Enterprise
Package, Cisco MDS 9000 Series SAN Extension over IP Package, Cisco MDS 9000 Series
Mainframe Package, Cisco MDS 9000 Series Data Mobility Manager (DMM) Package, and
Cisco MDS 9000 I/O Accelerator (IOA) Package. On-Demand Port Activation Licenses are
also available for the Cisco MDS Blade Switch Series and 8-Gbps Cisco MDS 9148 Multilayer
Fabric Switches.
MDS 9000 Series Enterprise Package
The standard software package that is bundled at no charge with the MDS 9000 Series switches
includes the base set of features that Cisco believes is required by most customers for building
a SAN. The MDS 9000 Series also has a set of advanced features that are recommended for all
enterprise SANs. These features are bundled together in the MDS 9000 Series Enterprise
Package. Refer to the MDS 9000 Series Enterprise Package fact sheet for more information.
MDS 9000 Series SAN Extension Over IP Package
The MDS 9000 Series SAN Extension over IP Package allows the customer to use FCIP to
extend SANs over wide distances on IP networks using the MDS 9000 Series IP storage
services. Refer to the MDS 9000 Series SAN Extension over IP Package fact sheet for more
information.
MDS 9000 Series Mainframe Package
The MDS 9000 Series Mainframe Package uses the Fibre Connection (FICON) protocol and
allows Control Unit Port (CUP) management for in-band management from IBM S/390 and
z/900 processors. FICON virtual SAN (VSAN) support is provided to help ensure true
hardware-based separation of FICON and open systems. Switch cascading, fabric binding, and
© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-21
intermixing are also included in this package. Refer to the MDS 9000 Series Mainframe
Package fact sheet for more information.
MDS 9000 Series DMM Package
The MDS 9000 Series DMM Package enables data migration between heterogeneous disk
arrays without introducing a virtualization layer or rewiring or reconfiguring SANs. Cisco
DMM allows concurrent migration between multiple logical unit numbers (LUNs) of unequal
size. Rate-adjusted migration, data verification, dual Fibre Channel fabric support, and
management using Cisco Data Center Network Manager (DCNM) for SAN provide a complete
solution that greatly simplifies and eliminates most downtime that is associated with data
migration. Refer to the MDS 9000 Series DMM Package fact sheet for more information. The
MDS 9000 Series DMM Package is for use only with MDS 9000 Series switches.
MDS 9000 Series IOA Package
The MDS 9000 Series IOA Package activates MDS 9000 Series IOA on the MDS 9222i
Multiservice Modular Switch, the MDS 9250i Multiservice Fabric Switch, and the SSN-16
module. The MDS 9000 Series IOA Package is licensed per service engine and is tied to the
chassis. The number of licenses that are required is equal to the number of service engines on
which the intelligent fabric application is used. The SSN-16 requires a separate license for each
engine on which you want to run MDS 9000 Series IOA. Each SSN-16 engine that you
configure for MDS 9000 Series IOA checks out a license from the pool that is managed at the
chassis level. SSN-16 MDS 9000 Series IOA licenses are available as single licenses.
On-Demand Port Activation License
On-demand ports allow customers to benefit from Cisco NX-OS Software features while
initially purchasing only a few activated ports on the MDS 9148 Multilayer Fabric Switch and
the Cisco MDS 8-Gb Fabric Switch for HP c-Class Blade System. As needed, customers can
expand switch connectivity by licensing additional ports.
Cisco XRC Acceleration License
The Cisco Extended Remote Copy (XRC) Acceleration License activates FICON XRC
acceleration on the MDS 9222i Multiservice Modular Switch and MDS 9250i Multiservice
Fabric Switch. One license per chassis is required. You must install the MDS 9000 Series
Mainframe Package and the MDS 9000 Series SAN Extension over IP Package before you
install the Cisco XRC Acceleration License. The MDS 9000 Series Mainframe Package enables
underlying FICON support, and the FCIP license or licenses enable underlying FCIP support.

2-22 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Standard license package is free with • Standard package is free
the MDS 9000 Series switch - Fibre Channel and iSCSI
• Additional license packages - iSCSI server load balancing
- MDS 9000 Series Enterprise Package - VSANs and zoning
- MDS 9000 Series SAN Extension over - Port channels
IP Package
- Fibre Channel Congestion Control and
- MDS 9000 Series Mainframe Package virtual output queuing
- MDS 9000 Series DMM Package - Diagnostics (Switch Port Analyzer
- On-Demand Port Activation License [SPAN], Remote SPAN [RSPAN], FC-
Analyzer)
- MDS 9000 IOA Package
- Fabric Manager and Device Manager
- Cisco XRC Acceleration License
- SNMPv3, SSH, SSL, SFTP
• Nondisruptive installation
- SMI-S 1.10 and FDMI compliance
• Features evaluated free for 120 days - RBAC
- Exceptions are port licensing and Cisco - RADIUS and TACACS+, MS CHAP
MDS 9000 IOA
- RMON, syslog, Call Home
• License installation saves a
permanent copy of licenses to the - Brocade and McData native interop
chassis modes
• A license is not required for MDS - NPIV
9000 8-port 10-Gb/s FCoE module - Command scheduler
- IPv6 (management and IP services)
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-6

The Cisco MDS 9000 Series NX-OS is the underlying system software that powers the
Cisco MDS 9000 Series Multilayer Switches. Cisco NX-OS is designed for SANs following in
the tradition of Cisco IOS Software to create a strategic SAN platform of superior reliability,
performance, scalability, and features.
In addition to providing all the features that the market expects of a storage network switch,
Cisco NX-OS provides many unique features that help the Cisco MDS 9000 Series to deliver
low total cost of ownership (TCO) and a quick return on investment (ROI).

Common Software Across All Platforms

Cisco NX-OS runs on all Cisco MDS 9000 Series switches, from multilayer fabric switches to
multilayer directors. Using the same base system software across the entire product line enables
Cisco to provide an extensive, consistent, and compatible feature set on the Cisco MDS 9000
Series.
Most Cisco MDS 9000 Series Software features are included in the base switch configuration.
The standard software package includes the base set of features that Cisco believes are required
by most customers for building a SAN. However, some features are logically grouped into add-
on packages that must be licensed separately.

Software Licensing
Licenses are available for all switches in the Cisco MDS 9000 Series. Licensing allows you to
access specified premium features on the switch after you install the appropriate license for that
feature.
The licensing feature maintains the following high-availability standards for all switches in the
Cisco MDS 9000 Series:
 Installing any license in any switch is a nondisruptive process.
 Installing a license automatically saves a copy of permanent licenses to the chassis in all
switches.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-23
 Enabling a license feature without a license key starts a counter on the grace period. The
grace period allows 120 days to install the appropriate license keys or disable the use of
that feature. If, at the end of the 120-day grace period, the switch does not have a valid
license key for the feature, the feature is automatically disabled by the switch.

Directors in the Cisco MDS 9500 Series have the following additional high-availability
features:
 The license software runs on both supervisor modules and provides failover protection.
 The license key file is mirrored on both supervisor modules. Even if both supervisor
modules fail, the license file continues to function from the version that is available on the
chassis.

2-24 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• MDS 9000 Series Enterprise Package
• Feature-based license applied
- Enhanced security features
to the MDS chassis: • VSAN-based access control
- Enhanced security • FC-SP authentication
• DH-CHAP authentication locally, or
- Traffic engineering remotely through RADIUS or TACACS+
• Fibre Channel port security and fabric
- IVR binding
- Extended buffer-to-buffer credits • IPsec protocol for iSCSI and FCIP
(BB_Credits) • Cisco TrustSec Fibre Channel link
encryption
- QoS • IKE digital certificates
- Advanced traffic engineering
- Switch-to-switch authentication • Fibre Channel QoS
- Host-to-switch authentication • Fibre Channel write acceleration
• Zone-based traffic prioritizing
• Zone-based Fibre Channel QoS
• Extended BB_Credits
• SCSI flow statistics
• SAN device virtualization
• Extended BB_Credits
- Enhanced VSAN functionality
• IVR-NAT over Fibre Channel
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-7

The MDS 9000 Series Enterprise Package includes the following enhanced network security
features:
 Switch-switch and host-switch authentication: Fibre Channel Security Protocol (FC-SP)
capabilities in Cisco MDS 9000 NX-OS provide switch-switch and host-switch
authentication. This feature helps eliminate disruptions that may occur because of
unauthorized devices connecting to a large enterprise fabric.
 Diffie-Hellman Challenge Handshake Authentication Protocol (DH-CHAP): This
protocol is used to perform authentication locally in the Cisco MDS 9000 Series switch or
remotely through RADIUS or TACACS+. If authentication fails, a switch or host cannot
join the fabric.
 Port security: This feature locks down the mapping of an entity to a switch port. The
entity can be a host, target, or switch, and is identified by its world wide name (WWN).
This feature helps ensure that SAN security is not compromised by connection of
unauthorized devices to a switch port.
 VSAN-based access control: This feature allows customers to define roles in which the
scope of the roles is limited to certain VSANs. For example, a network administrator role
can be set up to allow configuration of all platform-specific capabilities. A VSAN-
administrator role can be set up to allow configuration and management of only specific
VSANs. VSAN-based access control reduces SAN disruptions by localizing the effects of
user errors to the VSANs for which the user has administrative privileges.
 IP Security (IPsec): IPsec is available for FCIP and Internet Small Computer Systems
Interface (iSCSI) over Gigabit Ethernet ports on the MDS 9222i Multiservice Modular
Switch and MDS 9250i Multiservice Fabric Switch. The proven IETF-standard IPsec
capabilities offer secure authentication, data encryption for privacy, and data integrity.
Internet Key Exchange version 1 (IKEv1) and IKE version 2 (IKEv2) protocols are used to
dynamically set up the security associations for IPsec using preshared keys for remote-side
authentication.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-25
 Digital certificates: Digital certificates are issued by a trusted third party and are used as
electronic passports to prove the identity of certificate owners. After the identity of the
owner is verified by the trusted third party, the certificate uses the public encryption key of
the owner to protect identity data that is contained in the certificate. On the Cisco MDS
9000 Series platform, digital certificates apply to IKE as well as to Secure Shell (SSH).
 Fabric binding for open systems: Fabric binding helps ensure that Inter-Switch Links
(ISLs) are enabled only between switches that have been authorized in the fabric binding
configuration. This feature helps prevent unauthorized switches from joining the fabric or
disrupting current fabric operations.
 Cisco TrustSec Fibre Channel Link Encryption: Cisco TrustSec Fibre Channel Link
Encryption helps ensure data integrity and privacy. Cisco TrustSec Fibre Channel Link
Encryption is an extension of the FC-SP feature and uses the existing FC-SP architecture.
Fibre Channel data between expansion ports (E Ports) of 8-Gbps and 16-Gbps modules can
be encrypted. The encryption algorithm is 128-bit Advanced Encryption Standard (AES),
and enables either AES Galois/Counter Mode (AES-GCM) or AES Galois Message
Authentication Code (AES-GMAC) for an interface. AES-GCM mode provides encryption
and authentication of the frames, and AES-GMAC provides only the authentication of the
frames that are being passed between the two E Ports. Encryption is performed at line rate
by encapsulating frames at egress, with encryption using GCM and AES 128-bit
encryption. At ingress, frames are decrypted and authenticated for integrity. There are two
primary use cases:
— Customers connecting outside the data center over native Fibre Channel (for
example, using dark fiber, Coarse Wavelength-Division Multiplexing [CWDM] or
Dense Wavelength-Division Multiplexing [DWDM])
— Encryption within the data center

2-26 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• MDS 9000 Series SAN Extension
over IP Package for MDS 9250i
Multiservice Fabric Switch, MDS
9222i Multiservice Modular
Switch, and MDS 9000 SSN-16.
- Integrated support for FCIP MDS 9222i Multiservice
Modular Switch
- TCP optimization, traffic shaping
- FCIP compression
- IVR over FCIP
- IVR NAT over FCIP
- FCIP write acceleration MDS 9250i Multiservice
- FCIP tape acceleration Fabric Switch
- Seamless integration of Unified I/O
over FCIP
- SAN extension tuner
• The license is integrated into the
MDS 9250i Multiservice Fabric MDS 9000 SSN-16
Switch and MDS 9222i
Multiservice Modular Switch.

The MDS 9000 Series SAN Extension over IP Package includes the following features:
 Integrated support for FCIP: FCIP can be used to connect Fibre Channel SANs across
long distances using IP networks. Each Cisco MDS 9000 Series Gigabit Ethernet port can
manage up to three FCIP tunnels. Without the MDS 9000 SAN Extension over IP Package,
these capabilities would require multiple systems from different vendors.
 Complete integration of the Cisco FCIP implementation with value-added features on
the Cisco MDS 9000 Series switches: The Cisco VSAN function is supported across FCIP
links between SANs. FCIP can be used with the MDS 9000 Series Enterprise Package
features such as quality of service (QoS) over a WAN. Use of Virtual Routing Redundancy
Protocol (VRRP) increases IP network availability for FCIP connections by allowing the
failover of connections from one Gigabit Ethernet port to another. Load balancing using
port channels can also be performed over FCIP links.
 Optimization of the Cisco MDS 9000 NX-OS Software implementation of FCIP: The
Cisco NX-OS Software implementation of FCIP on the Cisco MDS 9000 Series products is
optimized for wire performance through enhancements that address out-of-order delivery
problems, support jumbo frames, provide traffic shaping, and perform TCP optimization.
 FCIP compression: FCIP compression in Cisco MDS 9000 NX-OS increases the effective
WAN bandwidth without costly infrastructure upgrades. By integrating data compression
in the MDS 9222i Multiservice Modular Switch, MDS 9250i Multiservice Fabric Switch,
and 16-Port Storage Services Node (SSN-16) modules, more efficient FCIP-based
business-continuity and disaster-recovery solutions can be implemented without the need to
add and manage a separate device. Gigabit Ethernet ports on the Cisco MDS 9000 Series
products can achieve up to a 43:1 compression ratio, with typical ratios of 4:1 over a wide
variety of data sources.
 Inter-VSAN Routing (IVR) for FCIP: IVR allows selective transfer of data traffic
between specific initiators and targets on different VSANs without the need to merge
VSANs into a single logical fabric. IVR can be used with FCIP to increase the resiliency of
SAN extension over IP networks and create more efficient business-continuity and disaster-

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-27
recovery solutions. IVR for FCIP is included in the MDS 9000 Series SAN Extension over
IP Package. To use IVR for Fibre Channel, the MDS 9000 Series Enterprise Package is
required.
 FCIP Write Acceleration: FCIP Write Acceleration significantly improves application
performance when storage traffic is routed over WANs using FCIP. When FCIP Write
Acceleration is enabled, WAN throughput is increased, and write I/O latency is decreased
by reducing the effects of WAN latency.
 FCIP Tape Acceleration: Centralizing tape backup and archive operations provides
significant cost saving by allowing expensive robotic tape libraries and high-speed drives to
be shared. This sharing poses a challenge for remote backup media servers that need to
transfer data across a WAN. High-performance streaming tape drives require a continuous
flow of data to avoid write data underruns, which dramatically reduce write throughput.
Without FCIP Tape Acceleration, the effective WAN throughput for remote tape backup
decreases exponentially as the WAN latency increases. Cisco MDS 9000 NX-OS FCIP
Tape Acceleration helps achieve near-full throughput over WAN links for remote tape
backup operations.
 Seamless integration of Unified I/O with FCIP: This feature provides support for FCIP
for Fibre Channel traffic originating from FCoE access switches. This support provides
seamless integration of FCoE traffic with the existing FCIP installed base.
 SAN Extension Tuner: To help optimize FCIP performance, the SAN Extension Tuner
generates SCSI I/O commands that are directed to a specific virtual target. The tuner
reports the number of I/O operations per second and I/O latency results, which helps
determine the number of concurrent I/O operations that are needed to increase FCIP
throughput.
This package is licensed on a per-engine, per-module basis. The number of licenses that a
customer needs to purchase is equal to the number of engines to be enabled for the feature on
the MDS 9000 SSN-16. The Cisco MDS 9000 Series SAN Extension license features are
enabled by default on the embedded ports on the MDS 9222i Multiservice Modular Switch and
MDS 9250i Multiservice Fabric Switch chassis. No additional license is required to use FCIP
and FCIP compression on these ports.

2-28 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Mainframe package (FICON)
- FICON protocol and CUP management (MAINFRAME_PKG)
- FICON VSAN and intermixing
- Switch cascading
- Fabric binding for FICON
- IBM TotalStorage Virtual Tape Server
- IBM TotalStorage XRC application
- FICON tape acceleration
- FICON for the Cisco MDS 9100 Series switches
- Persistent FCIDs for FICON
- Configuration locking for FICON
- Port swap, block, prohibit
- FICON qualification
• XRC Acceleration
- Activates FICON XRC acceleration on the Cisco MDS 9222i Multiservice Modular
Switch and on the MSM-18/4 line card module.
- Requires MAINFRAME_PKG and SAN_EXTN_OVER_FCIP.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-9

The Cisco NX-OS Mainframe Package consists of the following:

 IBM Fibre Connection (FICON) protocol and Control Unit Port (CUP) management
 FICON virtual storage area network (VSAN) and intermixing
 Switch cascading
 Fabric binding for FICON
 IBM TotalStorage Virtual Tape Server (VTS)
 IBM TotalStorage Extended Remote Copy (XRC) application
 FICON Tape Acceleration
 Persistent Fibre Channel ID (FCID) for FICON
 Configuration locking for FICON
 Port-swap, block, prohibit
 FICON qualification

XRC_ACCL activates FICON XRC acceleration on the Cisco MDS 9222i Multiservice
Modular Switch and MDS9250i Multiservice Fabric Switch and on the MSM-18/4 in the Cisco
MDS 9500 Series Directors.
You must install the mainframe package and the MDS 9000 Series SAN Extension over FCIP
package before you install the XRC Acceleration License. The mainframe package enables the
underlying FICON support, and the FCIP license or licenses enable the underlying FCIP
support.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-29
• PORT_ACTIVATION_PKG
• MDS 9148 Multilayer Fabric Switch
- The first 16 ports are licensed by default.
- There are additional ports in 8-port increments.
- Allows users to expand SAN connectivity as needed by enabling users to
purchase and install additional port licenses.
- By default, all ports are eligible for license activation.

The On-Demand Port Activation License allows users to expand their SAN connectivity as
needed by enabling users to purchase and install additional port licenses. By default, all ports
are eligible for license activation.
On Cisco MDS 9100 Series Multilayer Fabric Switches, licenses are allocated sequentially.
However, you can move or reassign licenses to any eligible port on the switch. The first group
of ports is licensed by default. You are not required to perform any tasks beyond the default
configuration unless you wish to immediately activate additional ports, make ports ineligible,
or move port licenses.
The On-Demand Port Activation License activates ports, in 16-port increments, on the MDS
9148 Multilayer Fabric Switch, which has 48 ports. The first 16 ports are licensed by default.

2-30 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The MDS 9000 DMM feature runs on the MSM 18/4 module, MDS 9222i
Multiservice Modular Switch, and MDS 9250i Multiservice Fabric Switch.
- Llicense activates Cisco DMM
- Provides online migration of heterogeneous arrays
- Provides simultaneous migration of multiple LUNs
- Unequal size LUN migration
- Rate-adjusted migration
- Verification of migrated data
- Dual fabric support Existing
host
Storage
- Two types of licenses:
• Permanent
New
• 180-day license Storage

- Moving FC to FCoE or back is supported

The data migration task is typically both complex and manual because of the myriad operating
systems, file systems, application servers, volume management systems, physical devices, and
networks involved. IT departments face the following challenges in migrating data:
 The downtime that is incurred
 The need to add data migration software to servers
 The potential for data loss and corruption
 Additional errors from the complexities of heterogeneous environments
 Simply running out of time before the job is done

The practice of conventional planned-outage periods for server maintenance and changes is
nearly obsolete as businesses increasingly require continuous system operation. A special
outage period solely for upgrading storage is often difficult for IT administrators to schedule.
A solution that addresses all these data migration concerns will greatly diminish the challenges
that are currently experienced in migrating data. Cisco DMM is such a solution. Cisco DMM is
a fabric-based data migration solution that transfers block data nondisruptively across
heterogeneous storage volumes and across distances, whether the host is online or offline. With
no host components to deploy, the time-consuming, costly, and sensitive task of installing and
licensing hardware and software on servers is avoided. In addition, the host CPU and
bandwidth cycles are available to the application in their entirety, so applications do not
experience any deterioration in performance.
This innovative Cisco network-integrated solution has numerous advantages over competitive
solutions available today:
 Transparent online data migration across heterogeneous storage arrays. Neither the host
server nor the storage array needs reconfiguration when Cisco DMM is introduced into the
user environment, and no zoning configuration is required. The storage administrator can
complete the migration task without needing to inform the server, system, or database
administrator that a migration is being planned.
© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-31
 Cisco DMM can move data over long distances when the copy operation must be
completed asynchronously with write I/O operations.
 Cisco DMM can securely erase the data from the existing storage so that this step can be
completed before the array leaves the customer data center.
 Centralized management with the GUI and a wizard for easy configuration. This feature
enables you to do the following:
— Pace the data migration job.
— Schedule the start and cutover times.
— View the effect on the SAN of the extra traffic that is generated by the data
movement.
 A configuration wizard simplifies setup and use, and a CLI allows advanced users to
complete their migration tasks using scripts.
 Perform per-server and per-storage array configuration.
 Perform large-scale, high-performance migration with rate-limiting options.
 No rewiring is required in most cases.
 No host agents are required, minimizing the effect on the CPU and mitigating software
image management concerns.
 Support is provided for active/active or active/passive storage access across dual redundant
fabrics.

2-32 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Provides centralized MDS 9000 • IOA_9222i_PKG, and
IOA_9250i_PKG
IOA features
- Activates on the single engine MDS
- Fibre Channel Write Acceleration 9222i Multiservice Modular Switch and
MDS 9250i Multiservice Fabric Switch.
- Fibre Channel Tape Acceleration
• IOA_SSN16_PKG
- FCIP Write Acceleration - Activates MDS 9000 IOA for an MDS
- FCIP Tape Acceleration 9000 SSN-16 engine.
- The MDS 9000 SSN-16 requires a
- Extend MDS 9000 IOA to any separate license for each engine on
device in the SAN which you want to run MDS 9000 IOA.
- Transparent deployment, which - Each MDS 9000 SSN-16 engine that
you configure for MDS 9000 IOA checks
means no reconfiguration or out a license from the pool that is
wiring managed at the chassis level.

- Transport and protocol agnostic, - SSN-16 MDS 9000 IOA licenses are
available singly (the usual model) or in a
so package works with Fibre four-pack. If you install a four-pack on
Channel, Gigabit Ethernet, and your chassis, the MDS 9000 IOA four-
FCIP pack license appears the same as four
single MDS 9000 IOA licenses.
• IOA_18/4_PKG

The MDS 9000 IOA feature provides Small Computer Systems Interface (SCSI) acceleration in
a SAN where the sites are interconnected over long distances using Fibre Channel or FCIP
ISLs.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-33
• Feature-rich standard package (no extra charge)
- Simple bundles for advanced features that provide significant value
- All upgrades included in support pricing
• High availability
- Nondisruptive installation
- 120-day grace period for enforcement (except port license and MDS 9000
IOA)
• Ease of use
- Automated license key installation
- Centralized license management console

License management is a notorious problem. License usability can be a problem with existing
products, and customers have concerns about compromising availability with disruptive
software installations for licensed features.
Cisco license packages require a simple installation of an electronic license. No software
installation or upgrade is required. Licenses can also be installed on the switch in the factory.
MDS switches store license keys on the chassis serial PROM (SPROM) so license keys are
never lost, even during a switch software reinstall.
Cisco Prime DCNM includes a centralized license management console that provides a single
interface for managing licenses across all MDS switches in the fabric. This ability reduces
management overhead and prevents problems due to improperly maintained licensing. If an
administrative error does occur with licensing, the switch provides a grace period before the
unlicensed features are disabled. This grace period provides sufficient time to correct the
licensing issue.
Most licensed features can be evaluated for up to 120 days before a license is required.

2-34 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Obtain the Cisco serial number for your switch using the show license
host-id command.
• Obtain your claim certificate or proof of purchase document.
• Locate the website URL address from the claim certificate or proof of
purchase. (www.cisco.com/go/licenses)
• Go to the website and enter the switch serial number and PAK.
• The digitally signed license key will be sent to you by email.
Web Browser
Proof of Purchase
Website URL URL Address

PAK PAK License Key File

Through Email
Switch Serial Number (Switch ID)

Cisco MDS Switch

Device Serial Number
(Device ID)

Use the following steps to obtain a serial number by accessing the license key file:
Step 1 Obtain the serial number for your switch using the show license host-id command.
Step 2 Obtain your claim certificate or proof of purchase document.
Step 3 Locate the website URL address from the claim certificate or proof of purchase.
Step 4 Go to the website and enter the switch serial number and Product Authorization Key
(PAK).
Step 5 The digitally signed license key will be sent to you by email.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-35
• From the CLI, use the install license command.
• Use the show license command to verify status.
switch# install license bootflash:license_file.lic
Installing license ..done
switch# show license
AMS1-9222i-ALL-FOX1229H7U7.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT FM_SERVER_PKG cisco 1.0 permanent uncounted \
VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>M9200-ALL-LICENSES-INTRL</SKU> \
HOSTID=VDH=FOX1229H7U7 \
NOTICE="<LicFileID>20090107135901713</LicFileID><LicLineID>1</LicLineID> \ <PAK></PAK>"
SIGN=243547A4AD2E
INCREMENT SAN_EXTN_OVER_IP cisco 1.0 permanent 1 \
VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>M9200-ALL-LICENSES-INTRL</SKU> \
HOSTID=VDH=FOX1229H7U7 \
NOTICE="<LicFileID>20090107135901713</LicFileID><LicLineID>2</LicLineID> \ <PAK></PAK>"
SIGN=2E3916D62B9C
INCREMENT STORAGE_SERVICES_ENABLER_PKG cisco 1.0 permanent 1 \
VENDOR_STRING=<LIC_SOURCE>MDS_SWIFT</LIC_SOURCE><SKU>M9200-ALL-LICENSES-INTRL</SKU> \
HOSTID=VDH=FOX1229H7U7 \
NOTICE="<LicFileID>20090107135901713</LicFileID><LicLineID>3</LicLineID>
--More--

Once you have obtained the necessary information, you can install the license using the
following steps:
Step 1 From the CLI, use the install license command.
Step 2 Use the show license command to verify status.

2-36 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco NX-OS Software Installation and Upgrade
This topic describes the Cisco NX-OS image installation and upgrade process.

• Software images
- The kickstart and system image files must be accessible from the Cisco MDS
9000 Series switch prompt.
• Image version
- Each image file has a version.
- Kickstart and system image versions must be compatible.
• Flash disks on the switch
- The bootflash is internal to the supervisor.
- External CompactFlash disks are inserted in the MDS 9500 slot0 device.
• Supervisor modules
- Supervisor-2A and Supervisor-1 have specific image requirements:
• MDS 9500 Sup-2A image filename contains -sf2ek9.
• MDS 9710 Sup-1 image filename contains -sf3ek9.
- The software install process is disruptive on systems with a single supervisor
module.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—2-17

This figure describes details for four options:

 Software images
 Image version
 Flash disks on the switch
 Supervisor modules

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-37
• Customer service
- Before performing an upgrade, contact your customer service representative
to review your software upgrade requirements.
• Scheduling
- Schedule the upgrade when the fabric is stable. Ensure that no configuration
changes will occur during the upgrade.
• Space
- Verify that sufficient space is available in the location where you are copying
the images.
• Hardware
- Avoid power interruption during any installation procedure. Power interruption
can corrupt the software images.
• Connectivity
- Retrieve images from remote servers.
- Configure the IP address for interface mgmt0.
- Ensure that the switch has a route to the remote server.

This figure details the essential prerequisites for performing upgrades:

 Customer service
 Scheduling
 Space
 Hardware
 Connectivity

2-38 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Nondisruptive
- Automated, one step using the install all command
- Manual, step-by-step process
• Exceptions
- Kickstart image change, single supervisor
- Incompatible system software images
- Single- or dual-supervisor system

This slide details three CLI mechanisms for switch software upgrades:
 Nondisruptive
 Disruptive
 Exceptions

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-39
• Launch a console session to both supervisors.
• Create a backup of your existing configuration file with the command
copy run tftp://A.B.C.D/file.
• From the active supervisor, perform an upgrade using the install all
command.
• View upgraded supervisor module using the show module command.
• Save your running configuration using the copy running-config
startup-config command.

Switch Before Issuing the Switch After the install all

install all Command Command Completes

Slot 5 Slot 6 Slot 5 Slot 6

install all Active Standby Standby Active
Supervisor Supervisor Supervisor Supervisor
5.2(6) to 6.2(3) Module Module Module Module
5.2(6) 5.2(6) 6.2(3) 6.2(3)

The Cisco MDS 9513 Multilayer Director has supervisors in slots 7 and 8. The MDS 9710
Multilayer Director has supervisors in slots 5 and 6.
These steps detail the method for using the install all command.
Step 1 Launch a console session to both supervisors.
Step 2 Create a backup of your existing configuration file with the command copy run
tftp://A.B.C.D/file.
Step 3 From the active supervisor, perform an upgrade using the install all command.
Step 4 View the upgraded supervisor module using the show module command.
Step 5 Save your running configuration using copy running-config startup-config.

2-40 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The install all kickstart bootflash:file system bootflash:file command

switch# install all kickstart bootflash:m9200-s2ek9-kickstart-mz.6.2.3.bin system m9200-

s2ek9-mz.6.2.3.bin
Verifying image bootflash:/m9200-s2ek9-kickstart-mz.5.2.1.bin for boot variable "kickstart".
[####################] 100% -- SUCCESS
Verifying image bootflash:/m9200-s2ek9-mz.6.2.3.bin for boot variable "system".
[####################] 100% -- SUCCESS
Verifying image type.
[####################] 100% -- SUCCESS

Extracting "system" version from image bootflash:/m9200-s2ek9-mz.6.2.3.bin.

[####################] 100% -- SUCCESS

Extracting "kickstart" version from image bootflash:/m9200-s2ek9-kickstart-mz.6.2.3.bin.

[####################] 100% -- SUCCESS

Extracting "bios" version from image bootflash:/m9200-s2ek9-mz.6.2.3.bin.

[####################] 100% -- SUCCESS

Performing Compact Flash and TCAM sanity test.

[####################] 100% -- SUCCESS

Notifying services about system upgrade.

[####################] 100% -- SUCCESS

This figure and the two that follow detail the upgrade process using the CLI.

• Compatibility check and image upgrade table

Compatibility check is done:
Module bootable Impact Install-type Reason
------ -------- -------------- ------------ ------
1 yes non-disruptive none

Other miscellaneous information for installation:

Module info
------ ----------------------------------
1 FC ports 1-18 are hitless, GigE 1-4 are hitful, and Intelligent Applications running
are hitful

Images will be upgraded according to following table:

Module Image Running-Version(pri:alt) New-Version Upg-Required
------ ---------- ---------------------------------------- --------------------
1 system 5.2(6) 6.2(3) no
1 kickstart 5.2(6) 6.2(3) no
1 bios v1.0.19(02/01/10): v1.0.19(02/01/10) no

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-41
• Continuing the installation
Do you want to continue with the installation (y/n)? [n] y

Install is in progress, please wait.

Setting boot variables.
[####################] 100% -- SUCCESS

Performing configuration copy.

[####################] 100% -- SUCCESS

Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.

Warning: please do not remove or power off the module at this time.
[####################] 100% -- SUCCESS

Install has been successful.

P7MDS1#

2-42 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Version Downgrade Procedure
This topic describes the Cisco NX-OS version downgrade procedure.

• When downgrading Cisco NX-OS Software, consider the following:

- New features need to be disabled for a nondisruptive upgrade.
- Features in the running configuration that are not supported by the image
version being installed can put the switch in an inconsistent state.
- The install all compatibility check shows that the impact is disruptive. The
disruptive impact is why the current running configuration is not supported by
the new image.

Compatibility check is done:

Module bootable Impact Install-type Reason
------ -------- -------------- ------------ -------
2 yes disruptive reset Current running-config is not
supported by new image
3 yes disruptive reset Current running-config is not
supported by new image
5 yes disruptive reset Current running-config is not
supported by new image
6 yes disruptive reset Current running-config is not
supported by new image

This figure lists the factors to consider before proceeding with a Cisco NX-OS version
downgrade.

• The show incompatibility system bootflash: image-filename

command displays the enabled features that are not compatible with the
image that is being installed.
switch# show incompatibility system bootflash:m9200-ek9-mz.1.3.4b.bin
The following configurations on active are incompatible with the system image
1) Service :port-channel , Capability :CAP_FEATURE_AUTO_CREATED_PORT_CHANNEL
Description :active mode port channels, auto create enabled ports or auto created
port-channels are present
Capability requirement :STRICT
2) Service :cfs , Capability :CAP_FEATURE_CFS_ENABLED_VSAN
Description :CFS - Distribution is enabled for VSAN
Capability requirement :STRICT
3) Service :cfs , Capability :CAP_FEATURE_CFS_ENABLED_SYSLOGD
Description :CFS - Distribution is enabled for SYSLOGD
Capability requirement :STRICT
4) Service :cfs , Capability :CAP_FEATURE_CFS_ENABLED_ROLE
Description :CFS - Distribution is enabled for ROLE
Capability requirement :STRICT
5) Service :cfs , Capability :CAP_FEATURE_CFS_ENABLED_CALLHOME
Description :CFS - Distribution

This figure shows the CLI steps to complete the process.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-43
• FC-SP will be supported on the Cisco MDS 9710 Multilayer Director.
• The Generic Online Diagnostics (GOLD) system provides diagnostics for
the Cisco MDS 9710 Multilayer Director instead of the Online Health
Management System (OHMS). In Cisco NX-OS Release 6.2(3), the
GOLD system does not provide any support for automation of corrective
actions such as rebooting modules based on error thresholds.
• Cisco NX-OS Release 6.2(3) does not support the following hardware:
- Cisco MDS 9134 Multilayer Fabric Switch
- Cisco MDS 9124 Multilayer Fabric Switch
- Cisco MDS 4-Gbps Fabric Switch for HP c-Class BladeSystem
- Cisco MDS 4-Gbps Fabric Switch for IBM BladeCenter

The figure describes things to note about Cisco NX-OS Release 6.2(3).

2-44 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• The licensing model that is defined for the Cisco MDS product line has
two options, which are feature-based licenses for director-class switches
and module-based licenses for modular switches.
• A nondisruptive procedure to install Cisco NX-OS Software is the
automated, one-step process using the install all command.
• When performing a software downgrade, you may need to disable new
features to allow for a nondisruptive downgrade.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-45
2-46 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Summary
This topic summarizes the key points that were discussed in this module.

• Initial configuration requirements while setting up the switch include the

administrator username and password, switch name, management
Ethernet port, default gateway IP address, and system defaults for Fibre
Channel switch interfaces, switch port trunk mode, and zoning policy.
Before a switch can be managed remotely, you must first configure its IP
parameters.
• The base license that is provided with every Cisco MDS 9000 Series
switch includes the functionality to create and manage VSANs. You have
access to many features for security, troubleshooting, and management
through the CLI, the Cisco MDS 9000 Device Manager, and the Cisco
DCNM-SAN Client.

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-47
2-48 Configuring Cisco MDS 9000 Series Switch (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Self-Check
Use the questions here to review what you learned in this module. The correct answers and
solutions are found in the Module Self-Check Answer Key.
Q1) In Cisco MDS NX-OS Release 6.2.1, what is the default state of Fibre Channel switch
interfaces and the default switch port trunk mode? (Source: Performing the Initial
Switch Configuration)
A) shut, off
B) no shut, off
C) shut, on
D) no shut, on
Q2) Which set of commands configures the management interface for Cisco MDS 9000
Series switches? (Source: Performing the Initial Switch Configuration)
A) switch(config)# ip address 1.1.1.21 255.255.255.0
switch(config)# no shutdown
switch(config)# exit
switch(config)# ip default-gateway 1.1.1.1
B) switch(config)# interface mgmt0
switch(config)# ip address 1.1.1.21 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# exit
switch(config)# ip default-gateway 1.1.1.1
C) switch(config)# interface mgmt0
switch(config-if)# ip address 1.1.1.21 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# exit
switch(config)# default-gateway 1.1.1.1
D) switch(config)# interface mgmt0
switch(config-if)# ip address 1.1.1.21 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# exit
switch(config)# gateway 1.1.1.1
Q3) Which option is the switch boot sequence? (Source: Installing and Licensing Cisco
NX-OS Software)
A) BIOS, loader, kickstart, system
B) loader, kickstart, system, BIOS
C) BIOS, kickstart, loader, system
D) kickstart, loader, BIOS, system

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-49
Q4) Which command installs a license on a Cisco MDS 9000 Series switch? (Source:
Installing and Licensing Cisco NX-OS Software)
A) switch# install bootflash:license_file.lic
B) switch(config)# install bootflash:license_file.cfg
C) switch# install license bootflash:license_file.lic
D) switch(config)# install license
bootflash:license_file.cfg
Q5) Which option does not have a license grace period? (Source: Installing and Licensing
Cisco NX-OS Software)
A) Cisco IVR
B) QoS
C) SAN Extension
D) FICON
E) port activation
Q6) Where is the startup configuration file saved? (Source: Installing and Licensing Cisco
NX-OS Software)
A) NVRAM
B) RAM
C) flash memory
D) log directory
Q7) When should you use this command: switch# show incompatibility system
bootflash:file-name? (Source: Installing and Licensing Cisco NX-OS Software)
A) upgrading switch software
B) downgrading switch software
C) upgrading BIOS
D) downgrading BIOS
Q8) Which feature requires the Enterprise license? (Source: Installing and Licensing Cisco
NX-OS Software)
A) iSCSI
B) Cisco Fibre Channel QoS
C) Cisco PortChannels
D) native interoperability modes

2-50 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Self-Check Answer Key
Q1) C
Q2) B
Q3) A
Q4) C
Q5) E
Q6) A
Q7) B
Q8) B

© 2013 Cisco Systems, Inc. System Installation and Initial Configuration 2-51
2-52 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module 3

Building a SAN Fabric

Overview
In this module, you will explore building a SAN fabric in the Cisco MDS 9000 Series switch
platforms.

Module Objectives
Upon completing this module, you will be able to describe the switch platforms for the MDS
9000 Series switches. You will be able to meet these objectives:
 Explain the use of fabric login (FLOGI) and Fibre Channel Name Server (FCNS) databases
 Describe how to configure Fibre Channel interfaces
 Describe the purpose and use of port channel functionality on the MDS 9000 Series
Multilayer Switches
 Explain Cisco N-Port Virtualizer (NPV) and N-Port ID Virtualization (NPIV), a feature
that allows a single Fibre Channel switch port to both manage multiple logins and assign
multiple Fibre Channel IDs (FCIDs)
 Describe the process of configuring virtual SANs (VSANs)
 Describe the purpose of Fibre Channel domains, their behavior in a VSAN environment,
and how to prevent fabric isolation.
 Explain the differences between a Fibre Channel alias and a distributed device alias
 Describe the purpose and use of zoning within Fibre Channel SANs
3-2 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 1

Using FLOGI and FCNS

Databases
Overview
This lesson describes the fabric login (FLOGI) and Fibre Channel Name Server (FCNS)
databases. Every Cisco MDS 9000 Series switch contains an FLOGI database and FCNS
database. The FLOGI database logs every end device that has successfully performed a fabric
login and obtained a Fibre Channel ID (FCID) from the switch. Unlike the FLOGI database
that is specific to a single switch, the FCNS database is distributed to every switch within the
same fabric virtual SAN (VSAN). The FCNS database contains a record of every device within
each VSAN that has successfully logged in and is online.

Objectives
Upon completing this lesson, you will be able to explain the use of FLOGI and FCNS
databases. This ability includes being able to meet these objectives:
 Describe the device login sequence
 Describe the use of the FLOGI and FCNS databases for device registration
 Describe the function of FCID persistence
Fabric Login Sequence
This topic describes the fabric login sequence.

• After the Fibre Channel link comes up, the host Fibre Channel driver
sends a FLOGI frame to the switch with its pWWN in the payload.
• The switch responds with a unique device FCID.
- The domain ID is a unique ID for each switch within a VSAN.
- The area ID represents one or more device ports connected to the same
switch port.
- The port ID is usually 0 or contains a unique AL-PA for each FC-AL device
connected to the same FL Port.
• Cisco MDS switches combine area and port to provide a unique 16-bit
address for each device.
FLOGI 1 1 FLOGI
pWWN1 pWWN2
FLOGI FLOGI FC
Database Database
FC Link ISL FC Link
FC
HBA

Host ACC Storage

2 ACC 2
0x010500 0x020300

Every Fibre Channel end device is allocated a fixed port world wide name (pWWN) by the
manufacturer of that device. The pWWN is used to uniquely identify that device within the
SAN. This process is similar to MAC addresses on Ethernet network interface cards (NICs).
After the Fibre Channel link comes up, the host Fibre Channel driver sends a FLOGI frame to
the switch with its unique pWWN in the payload.
The MDS 9000 Series switch responds with a unique device FCID:
 Domain ID: This ID is a unique ID for each switch within a VSAN. The domain ID is an
8-bit field, but some upper addresses are reserved for Fibre Channel services. The
maximum number of Fibre Channel switches that can be supported in a fabric or VSAN is
239. However, most vendors will only support 40.
 Area ID: This ID represents one or more device ports that are connected to the same
switch port. The area ID is an 8-bit field. That size field would normally limit the number
of ports per Fibre Channel switch to 256. However, MDS 9000 Series switches combine
area and port IDs to provide a unique 16-bit address for each device.
 Port ID: This ID usually has a value of zero (0), or contains a unique arbitrated loop
physical address (AL-PA) for each Fibre Channel Arbitrated Loop (FC-AL) device that is
connected to the same fabric loop (FL) port. This ID is an 8-bit field so theoretically 256
arbitrated loop devices could be supported, however limitations in the encoding limits the
devices to 126 per switch port.

The FCID is used by the Fabric Shortest Path First (FSPF) protocol. The ID is used for routing
between Fibre Channel switches within a fabric or VSAN using the switch domain ID found in
the first 8 bits of the FCID that is in the Fibre Channel frame header.

3-4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Some legacy HBAs do not discover targets that have FCIDs with the
same domain and area.
- Cisco NX-OS maintains a list of HBAs with this behavior.
- HBAs are tracked by the OUI in the pWWN.
- These HBAs are allocated an entire area with the port bits set to 00 in their
assigned FCID, which is eb 01 00.
- Other HBAs may share the same area bits with distinct port bits. An example
would be eb 01 02.
- FCIDs are persistent in either case.

21:00:00:e0:8b:05:40:29 21:00:00:0c:50:9e:8b:78

HBA

Domain Area Port Domain Area Port

eb 00 01 eb 00 e8
FCID FCID
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-5

Some existing Fibre Channel host bus adapters (HBAs) do not discover targets that have an
FCID with the same domain and area.
 Cisco Nexus Operating System (NX-OS) maintains a list of HBAs with this behavior.
 HBAs are tracked by the Organizationally Unique Identifier (OUI) in the pWWN.
 These HBAs are allocated an entire area with the port bits set to 00 in their assigned FCID,
which is eb 01 00.
 Other HBAs may share the same area bits with distinct port bits. An example would be eb
01 02.

FCIDs are persistent in either case.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-5

switch# show fcid-allocation
area
Fcid area allocation company pWWN 21:00:00:e0:8b:05:40:29
id info:
HBA

00:50:2E
00:50:8B Domain Area Port

00:60:B0 eb 02 00
00:A0:B8 FCID

00:D0:B2
00:E0:69
00:E0:8B
…

The show fcid-allocation area command displays a list of restricted OUIs.

• The FLOGI database lists all devices that are currently online and that
have successfully logged in to the (local) switch.

switch# show flogi database

----------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
----------------------------------------------------------------------
fc1/5 10 0x6f0100 10:00:00:06:2b:08:f2:1d 20:00:00:06:2b:08:f2:1d
fc1/6 10 0x6f029b 22:00:00:00:87:6e:45:3c 20:00:00:00:87:6e:45:3c
fc1/6 10 0x6f02b3 22:00:00:00:87:6e:ad:38 20:00:00:00:87:6e:ad:38

Total number of flogi = 3.

The FLOGI database lists all devices that are currently online and that have successfully logged
in to the local switch.

3-6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Device Registration
This topic explains how to register devices using the FLOGI and FCNS databases.

1. After FLOGI, each device registers with the FCNS.

2. The FCNS database is distributed to all switches within the same VSAN so
that every switch has an identical copy.
3. Host ports query the name server to discover storage ports which the host
ports are allowed to see. Zoning is used to group devices together.
4. Each host sends a PLOGI to exchange Fibre Channel parameters (for
example, number of credits).
5. Finally the host (SCSI Initiator) sends a PRLI to storage (SCSI Target) to
exchange SCSI parameters and establish a channel of communication.

FLOGI 4 5 FLOGI
pWWN1 pWWN2
FLOGI FLOGI
Database Database FC
FC

FC Link ISL FC Link

FC
HBA
FCNS FCNS
Database 2 Database
Host Storage

3 FC

The figure shows the device registration steps:

1. After FLOGI is complete, each device registers with the FCNS.

2. The FCNS database is distributed to all switches within the same VSAN so that every
switch has an identical copy.
3. Host ports query the name server to discover storage ports which the host ports are allowed
to see. Zoning is used to group devices.
4. Each host sends a port login (PLOGI) to exchange Fibre Channel parameters (for example,
number of credits).

5. Finally, the host (Small Computer Systems Interface [SCSI] Initiator) sends a process login
(PRLI) to the storage (SCSI target) to exchange SCSI parameters and establish a channel of
communication.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-7

• The FCNS database lists all devices that are currently online and that
have successfully logged in to all switches within each VSAN.

switch# show fcns database

VSAN 10:
-------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
-------------------------------------------------------------------------
0x0b0000 N 10:00:00:06:2b:08:f2:1d (LSI) ipfc scsi-fcp:init
0x0b019b NL 22:00:00:00:87:6e:45:3c scsi-fcp:target
0x0b01b3 NL 22:00:00:00:87:6e:ad:38 scsi-fcp:target
0xef0000 N 10:00:00:06:2b:08:f9:54 (LSI) ipfc scsi-fcp:both
Total number of entries = 4

VSAN 20:
-------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
-------------------------------------------------------------------------
0x160000 N 10:00:00:06:2b:08:f9:55 (LSI) ipfc scsi-fcp:init
0x16019b NL 21:00:00:00:87:6e:45:3c scsi-fcp:target
0x1601b3 NL 21:00:00:00:87:6e:ad:38 scsi-fcp:target
0xef0000 N 10:00:00:06:2b:08:f2:1c (LSI) ipfc scsi-fcp:both
Total number of entries = 4

The FCNS database lists all devices that are currently online and that have successfully logged
in to all switches within each VSAN.

• The FCNS database contains a list of all devices that are online and that
have registered with the switch.

The FCNS database contains a list of all devices that are online and that have registered with
the switch.

3-8 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
fc

FLOGI

Name Server
Registration

PLOGI
PRLI

[Malformed Packet: FC ELS]

A Fibre Channel analyzer can be used to receive Fibre Channel frames from an MDS Series
switch. The analyzer encapsulate the frames into Ethernet frames, so that the frames can be
captured by a packet analyzer like Wireshark. The figure shows that the analyzer has captured
the sequence of frames issued as a result of a device logging into the fabric. The frames
displayed range from the FLOGI from a device to the switch, the process login (PLOGI)
containing the name server registration, another PLOGI for the device to log in to another
device, and the PLOGI to check the upper layer protocol compatibility.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-9

FCID Persistence
This topic describes the function of FCID persistence.

• FCIDs of storage ports should not change in such cases.

Class I H/W Path Driver S/W State H/W Type Description

---------------------------------------------------------------------------------
fc 0 0/1/2/0 td CLAIMED INTERFACE HP Mass Storage Adapter
fcp 0 0/1/2/0.1 fcp CLAIMED INTERFACE FCP Domain
ext_bus 3 0/1/2/0.1.19.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
Target 6 0/1/2/0.1.19.0.0.0 tgt CLAIMED DEVICE
disk 3 0/1/2/0.1.19.0.0.0.0 sdisk CLAIMED DEVICE HP OPEN-8
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 10 0/1/2/0.1.19.0.0.0.7 sdisk CLAIMED DEVICE HP OPEN-8
/dev/dsk/c4t0d7 /dev/rdsk/c4t0d7
target 7 0/1/2/0.1.19.0.0.1 tgt CLAIMED DEVICE
disk 18 0/1/2/0.1.19.0.0.1.7 sdisk CLAIMED DEVICE HP OPEN-9
/dev/dsk/c4t1d7 /dev/rdsk/c4t1d7

The FCID persistence feature ensures that a device always receives the same FCID when the
device performs a FLOGI. This action occurs regardless of which interface the device is
connected to on the same switch. FCID persistence is enabled by default on all MDS 9000
Series switches.

3-10 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• FCID persistence prevents FCIDs from changing:
- The same FCID is assigned after switch reboots.
- The same FCID is assigned when a device is moved to another port within the
same switch and VSAN.
• Domain IDs must be statically assigned.

D_ID 0x53 FCID

FC
FC 0x530200
FCID
X
0x530100
HBA

FCID persistence prevents FCIDs from changing:

 The same FCID is assigned after switch reboots.
 The same FCID is assigned when a device is moved to another port within the same switch
and VSAN.

Note Domain IDs must be statically assigned.

FCID persistence also improves Inter-VSAN Routing (IVR) management by providing the
following features:
 FCID persistence allows you to control and assign a specific virtual domain to use in a
native VSAN.
 FCID persistence allows you to control and assign a specific virtual FCID for a device.

The benefits of persistent FCIDs for IVR include the following:

 Host devices always see the same FCID for targets.
 FCIDs help you plan your SAN layout better by assigning virtual domains for the IVR to
use.
 FCIDs can make SAN monitoring and management easier. When you see the same domain
or FCID consistently assigned, you can readily determine the native VSAN or device to
which the FCID refers.

Before configuring persistent FCIDs, note that you can configure two types of database entries
for persistent IVR FCIDs:
 Virtual domain entries, which contain the virtual domain that should be used to represent a
native VSAN in a specific VSAN (current VSAN). Virtual domain entries contain the
following information:
— Native autonomous fabric ID (AFID)
© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-11
— Native VSAN
— Current AFID
— Current VSAN
— Virtual domain used for the native AFID and VSAN in current AFID and VSAN
 Virtual FCID entries, which contain the virtual FCID that should be used to represent a
device in a specific VSAN (current VSAN). Virtual FCID entries contain the following
information:
— pWWN
— Current AFID
— Current VSAN
— Virtual FCID used to represent a device for the given pWWN in the current AFID
and VSAN
If you use persistent FCIDs for IVR, use persistent FCIDs for all the devices in the IVR zone
set. The use of persistent FCIDs is not recommended for some of the IVR devices while using
automatic allocation for other devices.
IVR Network Address Translation (NAT) must be enabled to use IVR-persistent FCIDs.
In an IVR NAT configuration, if one VSAN in the IVR topology is configured with static
domain IDs, then the IVR domains that can be exported to that VSAN must also be assigned
static domains.

3-12 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Persistent FCIDs can also be statically assigned.
- Static assignment provides greater administrative control over FCID
assignment.
- Area and port octets can be manually configured.
- The configured domain must match the VSAN domain.
- This type of assignment is useful when migrating devices from other switches.
• Persistent FCIDs (dynamic) can be administratively purged, but statically
assigned FCIDs must be expressly deleted.

FC
D_ID 0x25 Third-Party Switch
D_ID 0x25

X
FCID
0x250200

Persistent FCIDs can be statically assigned for the following reasons:

 Static assignment provides greater administrative control over FCID assignment.
 Area and port octets can be manually configured.
 The configured domain must match the VSAN domain.
 This type of assignment is useful when migrating devices from other switches

Persistent FCIDs (dynamic) can be administratively purged, but statically assigned FCIDs must
be expressly deleted.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-13

Determine the pWWN of the HBA:
switch# show flogi database

Assign static FCIDs:

switch# conf
switch(config)# fcdomain fcid database
switch(config-fcid-db)# vsan 3 wwn
50:05:08:b2:00:71:c8:c2 fcid 0x6fee00

Purge persistent FCIDs:

switch# purge fcdomain fcid vsan 25

The figure illustrates the steps to configure and purge a persistent FCID. A device with the
world wide name (WWN) 50:05:08:b2:00:71:c8:c2 is configured with the FCID 0x070128 in
VSAN 3. All dynamic and unused FCIDs are purged from VSAN 25.

3-14 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• The FLOGI database contains a list of all devices that are online and
logged into a single switch.
• The FCNS database contains a list of all devices that are online and
logged into all switches within each VSAN and the FCNS database is
distributed to all switches within a VSAN.
• FCID persistence ensures that a device will always receive the same
FCID when the device logs in, regardless of which interface the device is
connected to on the same switch.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-15

3-16 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 2

Configuring Interfaces
Overview
This lesson describes how to configure Fibre Channel interfaces on Cisco MDS Multiservice
Fabric Switches using the CLI or Cisco Device Manager and Cisco Prime Data Center Network
Manager for SAN Client (DCNM-SAN Client). The lesson also describes the various Fibre
Channel port types available in a SAN and how the port types are used.

Objectives
Upon completing this lesson, you will be able to configure Fibre Channel interfaces. This
ability includes being able to meet these objectives:
 Describe the basic interface configuration that is required on the switch
 Explain how to configure dedicated or shared bandwidth on each of the interfaces within
port groups on the switching modules
 Explain consequences of and recommendations for slow drain detection
 Explain interface BB_Credits
 Explain how to configure the trunking feature of Cisco MDS 9000 Series switches
Configuring Fibre Channel Interfaces
This topic explains Fiber Channel interface configuration.

Private
NL AL Hub NL
Blade Server
Blade n
TL EISL FL
EISL
TNP TF TE TE

Blade 2 TNP TF TE TE
Blade 1 TNP-TF Port TE TE Port
Channel Channel SD
VSAN EISL
Trunking
Analyzer
TE
NP F
EVFP
FC TF TN FC

N F HBA

E E
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-4

Cisco MDS 9000 Series Multilayer Switches support all of the standard Fibre Channel ports as
defined in the ANSI T11 specifications. These switches also support trunking ports. This
feature is also known as virtual SAN (VSAN) trunking, which is a feature specific to switches
in the MDS 9000 Series. Trunking enables interconnect ports to transmit and receive frames in
more than one VSAN over the same physical link. Trunking is supported on expansion (E)
ports and fabric (F) ports.
Supported interfaces include the following:
 F Port: In F Port mode, an interface functions as a fabric port. This port may be connected
to a peripheral device (host or disk) operating as a node (N) port. An F Port can be attached
to only one N Port. F Ports support class 2 and class 3 service.
 FL Port: In fabric loop (FL) port mode, an interface functions as an FL Port. This port may
be connected to one or more node loop (NL) ports, including fabric loop (FL) ports in other
switches, to form a public arbitrated loop. If more than one FL Port is detected on the
arbitrated loop during initialization, only one FL Port becomes operational and the other FL
Ports enter nonparticipating mode. FL Ports support class 2 and class 3 service.
 N_Port: A node (N) port is a port that is assigned to a Fiber Channel device capable of
communicating switched Fabric protocol to an F Port on a switch.
 NP Ports: A proxy N (NP) port is a port on a device that is in Cisco N-Port Virtualizer
(NPV) mode and connected to the core switch via an F Port. NP Ports function like N Ports
except that in addition to providing N Port operations, these ports also function as proxies
for multiple, physical N Ports.
 TL Port: In translative loop (TL) port mode, an interface functions as a translative loop
port. The port may be connected to one or more private loop devices (NL Ports). TL Ports

3-18 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
have similar properties as FL Ports. TL Ports enable communication between a private loop
device and one of the following devices:
— A device that is attached to any switch on the fabric
— A device on a public loop anywhere in the fabric
— A device on a different private loop anywhere in the fabric
— A device on the same private loop
TL Ports support class 2 and class 3 services. Private loop devices refer to legacy devices that
reside on arbitrated loops. These devices are not aware of a switch fabric because the devices
only communicate with devices on the same physical loop.
 E Port: In E Port mode, an interface functions as a fabric expansion port. This port may be
connected to another E Port to create a Cisco Inter-Switch Link (ISL) between two
switches. E Ports carry frames between switches for configuration and fabric management.
These ports serve as a conduit between switches for frames that are destined to remote N
Ports and NL Ports. E Ports support class 2, class 3, and class F service.
 TE Port: In trunking expansion (TE) port mode, an interface functions as a trunking
expansion port. The port may be connected to another TE Port to create an Enhanced Inter-
Switch Link (EISL) between two switches. TE Ports are specific to the family of MDS
9000 Series switches. These ports expand the functionality of E Ports to support the
following:
— VSAN trunking
— Transport quality of service (QoS) parameters
— Fibre Channel Traceroute (fctrace) feature
In TE Port mode, all frames are transmitted in EISL frame format, which contains VSAN
information. Interconnected switches use the VSAN ID to multiplex traffic from one or more
VSANs across the same physical link. This feature is referred to as trunking in the family of
MDS 9000 Series switches. TE Ports support class 2, class 3, and class F service.
 TF Port: In trunking fabric (TF) port mode, an interface functions as a trunking fabric port.
The port may be connected to another trunked N (TN) port or trunked NP (TNP) port to
create a link between a core switch and an NPV switch or a host bus adapter (HBA) to
carry tagged frames. TF Ports are specific to the family of MDS 9000 Series Multiservice
Fabric Switches. These ports expand the functionality of F Ports to support VSAN
trunking. In TF Port mode, all frames are transmitted in EISL frame format, which contains
VSAN information. Interconnected switches use the VSAN ID to multiplex traffic from
one or more VSANs across the same physical link.
 TNP Port: In TNP Port mode, an interface functions as a TE Port. The port may be
connected to a TF Port to create a link to a core N-Port ID Virtualization (NPIV) switch
from an NPV switch to carry tagged frames.
 SD Port: In a Switched Port Analyzer (SPAN) destination (SD) port, an interface functions
as a SPAN. The SPAN feature is specific to switches in the family of MDS 9000 Series
switches. The port monitors network traffic that passes through a Fibre Channel interface.
This monitoring is done using a standard Fibre Channel analyzer (or a similar switch probe)
that is attached to an SD Port. SD Ports do not receive frames. These ports only transmit a
copy of the source traffic. The SPAN feature is nonintrusive and does not affect switching
of network traffic for any SPAN source ports.
 ST Port: In the SPAN tunnel (ST) port mode, an interface functions as an entry point port
in the source switch for the Remote SPAN (RSPAN) Fibre Channel tunnel. The ST Port
mode and the RSPAN feature are specific to switches in the MDS 9000 Series. When
© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-19
configured in ST Port mode, the interface cannot be attached to any device. Therefore, the
port cannot be used for normal Fibre Channel traffic.
 Fx Port: Interfaces that are configured as Fx Ports can operate in either F Port or FL Port
mode. The Fx Port mode is determined during interface initialization depending on the
attached N Port or NL Port. This administrative configuration disallows interfaces to
operate in any other mode—for example, preventing an interface to connect to another
switch.
 Auto Mode: Interfaces that are configured in auto mode can operate in F Port, FL Port, E
Port, TE Port, or TF Port mode. The port mode is determined during interface initialization.
For example, if the interface is connected to a node (host or disk), the port operates in F
Port or FL Port mode depending on the N Port or NL Port mode. If the interface is attached
to a third-party switch, the port operates in E Port mode. If the interface is attached to
another switch in the MDS 9000 Series, the interface may become operational in TE Port
mode. TL Ports and SD Ports are not determined during initialization and are
administratively configured.

3-20 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Each interface has an associated administrative and operational state.

Administrative State Description

Up Use no shut command to enable interface.
Down Use shut command to disable interface.

Operational State Description

Up Interface is transmitting and receiving traffic as desired.
When the interface is administratively up, the link layer
state is up, and interface initialization must be
completed.
Down Interface cannot transmit or receive data.
Trunking Interface is operational in TE or TF mode.

Configuration allows for administratively enabling or disabling an interface. Any interface that
is administratively disabled will not function operationally, meaning that the interface never
comes to the “up” state.
The shutdown command in the interface configuration submode disables a port. Preceding the
shutdown with the no shutdown option in the command enables the port. In other words, the
command makes the port administratively active. More examples are given in the figure
showing actual configuration.
The operational state can be down even though the data link layer is up. This state is possible
because the operational state of an interface depends on the protocol that is running on the
interface. For example, a TE Port might be running Fabric Shortest Path First (FSPF). If you
receive the FSPF status for the interface, the interface could be in the initial (INIT) state. This
status means that the interface will not be in the full operational state until the FSPF protocol
successfully completes the initialization sequence.
Traffic flows through an interface only when the interface is both administratively and
operationally functional.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-21

• In Cisco Device Manager, do the
following:
- Right-click the interface.
- Choose Enable or Disable.

• From the CLI interface

configuration mode:
switch# conf
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# interface fc1/5
switch(config-if)# no shutdown
switch(config-if)# shutdown

The shutdown command administratively disables the interface. The no shutdown command
administratively enables, or activates, the interface.
Complete these steps using Cisco Device Manager:
Step 1 Right-click the port.
Step 2 Choose Enable or Disable from the menu.
Interfaces on a port are shut down by default unless the initial configuration is modified. The
Cisco Nexus Operating System (NX-OS) Software implicitly performs a graceful shutdown in
response to either of these scenarios when an interface is operating in the E Port mode:
 When an interface is shut down administratively
 If the Cisco NX-OS Software application executes a port shutdown as part of its function

A graceful shutdown ensures that no frames are lost when an interface is shutting down. When
a shutdown is triggered, the switches that are connected to the shutdown link coordinate with
each other. This coordination ensures that all frames in the ports are safely sent through the link
before shutting down. This enhancement reduces the chance of frame loss.

3-22 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Display the interface switch port configuration options:

Interface Type
switch# conf fc = Fibre Channel
switch(config)# interface fc2/1 2 = Module number
/1 = Port number
switch(config-if)# switchport ?
beacon Disable/enable the beacon for an interface
description Enter description of maximum 254 characters
encap Configure encapsulation for the port
fcbbscn Disable/enable buffer-to-buffer state change
notification
fcrxbbcredit Configure receive BB_credit for the port
fcrxbufsize Configure receive data field size for the port
fill-pattern Configure the fill pattern for an interface
ignore Enter parameter to be ignored
mode Enter the port mode
owner Specify the owner of an interface
rate-mode Configure the rate mode for an interface
speed Enter the port speed
trunk Configure trunking parameters on an interface

There is a special configuration submode for interface configuration. This submode is entered
with the interface command.
The switchport ? command from the interface configuration submode provides a listing of all
the options that are available for the switch port configuration of the interface.
switch(config)# interface fc2/1
switch(config-if)# switchport ?
beacon Disable/enable the beacon for an interface
description Enter description of maximum 80 characters
encap Configure encapsulation for the port
fcbbscn Disable/enable buffer-to-buffer state change notification
fcrxbbcredit Configure receive BB_credit for the port
fcrxbufsize Configure receive data field size for the port
ignore Enter parameter to be ignored
mode Enter the port mode
owner Specify the owner of an interface
rate-mode Configure the rate mode for an interface
speed Enter the port speed
trunk Configure trunking parameters on an interface

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-23

• Use commands to configure interface fc1/5, mode Fx, with speed of 2
Gbps.
switch# conf
switch(config)# interface fc1/5
switch(config-if)# switchport mode fx
switch(config-if)# switchport speed 2000
switch(config-if)# no shutdown
switch(config-if)# end

switch# show interface fc1/5 brief

-------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------
fc1/5 1 FX -- up swl F 2 --

The switchport mode fx command configures the interface for F Port or FL Port operation. To
configure interface fc1/5 as an F Port, with a speed of 2 Gbps, use the following commands:
conf
interface fc1/5
switchport mode fx
switchport speed 2000
no shutdown

To verify your configuration, use the show interface command.

Note The trunk mode is ignored for any port that is not configured as an E Port.

3-24 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• From the Physical Attribute navigate to FC Interfaces > Physical:

Select the General tab for each interface, then

change port types, speed, and VSAN assignment.

Interface fc1/5
Mode F
VSAN 1
2 Gbps
Admin up

From the Physical Attribute pane, navigate to FC Interfaces > Physical. In the Fibre Channel
interfaces table, you can configure the following:
 Port mode (F, FL, FX, E, TL, SD, ST, Auto)
 Port VSAN
 Description
 Speed (1 Gbps, 2 Gbps, Auto)
 Administrative state (up or down)
 Rate mode (dedicated or shared)
 Service status (in or out)

To specify the trunk mode, choose the Trunk Config tab.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-25

• Right-click the interface.

Complete the following steps to configure port mode in Cisco Device Manager:
Step 1 Launch Cisco Device Manager with switch credentials.
Step 2 Right-click the interface in the Device tab.
Step 3 Select Configure.
Step 4 Configure the PortVSAN, Mode, Speed, Rate Mode, and Status up.
Step 5 Select the Trunk Config tab and choose trunk admin mode to configure trunking
mode.
Step 6 Enter the allowed VSANs and click Apply. Repeat this step for the other side of the
ISL.

3-26 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# conf
Enter configuration commands, one per line. End with CNTL/Z.
switch(config)# interface fc1/5
switch(config-if)# switchport owner Cisco_Lab
switch(config-if)# no switchport owner

switch# show run int fc 1/5

…….
interface fc1/5
switchport owner Cisco_Lab
no shutdown

switch# show port internal info interface fc 1/5

fc1/5 - if_index: 0x01004000, phy_port_index: 0x4
local_index: 0x4
Admin Config - state(up), mode(FX), speed(auto), trunk(on)
beacon(off), snmp trap(on), tem(false)
rx bb_credit(default), rx bb_credit multiplier(default)
rx bb_credit performance buffers(default)
bb scn config(on)
ignore flags (ignore:none), service state(in service)
rxbufsize(2112), encap(default), user_cfg_flag(0x1)
description()
port owner(Cisco_Lab)
…………….
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-11

The port owner is a descriptive string. The string can be configured on an interface to inform
other administrators about who owns the port and the purpose for which the port is being used.
The port owner feature is available for all ports, regardless of the operational mode of the port.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-27

• Port guard will
automatically error
disable an interface
with a flapping link.

switch# conf
switch(config)# interface fc 1/1
switch(config-if)# errdis detect cause ?
link-down
Bit-errors
switch(config-if)# errdisable detect cause bit-errors
num-times 5 duration 120

The port guard feature works in environments where the system and application environments
do not adapt quickly and efficiently to a port going down and back up, or to a port rapidly
cycling up and down. The SAN administrator can use the port guard feature to prevent this
issue from occurring in environments that are vulnerable to these problems. Using this feature,
the port can be configured to stay down after the first failure or after a specified number of
failures within a certain time period. This ability permits the SAN administrator to intervene
and control the recovery, avoiding any problems that are caused by the cycling.
You can also restrict the number of error reports and bring a malfunctioning port to the down
state dynamically by using the port guard feature. You can configure the port to go into an
error-disabled state for specific types of failures.
Any of these events can cause link failure:
 Encapsulating Security Payload (ESP) Cisco TrustSec violation
 Bit errors
 Signal loss
 Sync loss
 Link reset
 Credit loss

Additional causes may include the following:

 Not operational
 Too many interrupts
 Cable disconnected
 Hardware recoverable errors
 The connected device rebooted (F Ports only)
 The connected line card rebooted (ISL only)
3-28 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The port monitor monitors traffic flow through an interface and reports
any link errors, frame errors, discards, and so on.

Right-click a port and

select Monitor.

The port monitor feature helps to monitor the performance and status of ports and generates
alerts when problems occur. You can configure the thresholds for various counters, and trigger
an event when the values cross the threshold settings.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-29

Port Group

• Predefined set of ports

• Share same backplane
bandwidth

switch# conf
switch(config)# port-group-monitor enable
switch(config)# port-group-monitor name policyname
switch(config-port-group-monitor)# counter rx-performance poll-
interval secods delta rising-threshold percentage1 falling-threshold
percentage2

Each line card or module has a predefined set of ports, called port groups, that share the same
backplane bandwidth. While oversubscription is a feature, the port group monitor feature helps
to monitor the spine bandwidth utilization. An alarm syslog is generated so that you can
provision the ports across port groups evenly to better manage the oversubscription.
When the port group monitor feature is enabled and there is a policy consisting of a polling
interval in seconds, and the raising and falling thresholds in percentage are specified, port
group monitor generates a syslog. If port group traffic goes above the specified percentage of
the maximum supported bandwidth for that port group (for receive [Rx] and transmit [Tx]
traffic), another syslog is created if the value falls below the specified threshold.

3-30 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configuring Bandwidth
This topic explains how to configure dedicated or shared bandwidth on each of the interfaces
within port groups on the switching modules.

4-port Port
Group

E E Port Group

FX E E
FC

HBA

switch(config)# interface fc1/20

switch(config-if)# switchport rate-mode dedicated
switch(config-if)# switchport speed 8000
switch(config)# interface fc1/21-24
switch(config-if)# switchport rate-mode dedicated
Switch(config-if)# switchport speed 16000

The Cisco MDS 9700 48-Port 16-Gbps Fibre Channel switching module has 48 full-rate mode
ports. This feature means that each line card has 12 port groups of four ports each. Each port
group has 64-Gbps bandwidth.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-31

• Use the show port-resources command.
switch# show port-resources module 1
Module 1
Available dedicated buffers for global buffer #0 [port-group 1] are 2150
Available dedicated buffers for global buffer #1 [port-group 2] are 2150
Available dedicated buffers for global buffer #2 [port-group 3] are 2150
Available dedicated buffers for global buffer #3 [port-group 4] are 2618
Available dedicated buffers for global buffer #4 [port-group 5] are 2150
…
Port-Group 3
fc1/20 Fx port 32 8.0 dedicated
…
Port-Group 4
Total bandwidth is 64.0 Gbps
Allocated dedicated bandwidth is 64.0 Gbps
--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode
Buffers (Gbps)
--------------------------------------------------------------------
fc1/21 500 16.0 dedicated
fc1/22 E port 500 16.0 dedicated
fc1/23 500 16.0 dedicated
fc1/24 500 16.0 dedicated
--- Output truncated ---
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-17

Verify the configuration of the first port. Note the shared and allocated dedicated figures in the
output. E Ports typically require more buffer credits (for distance) than Fx Ports.
Go to this link for more information about buffer credits:
http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/6_2/configuration/guides/i
nterfaces/nx-os/buffers.html

• Right-click a module and

choose Show Port
Resources from the menu.

• Right-click an interface. Choose

Configure from the menu. Click
the dedicated radio button.

The figure describes how to show port resources.

3-32 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• 32-port 8-Gbps Adv FC Module • 48-port 8-Gbps Adv FC Module
- 32-ports full rate at 8-Gbps FC - 48-ports 1.5:1 oversubscribed at
- 4-port port groups 8-Gbps FC
Port Group - 6-port port groups
Port Group

- 24 ports full rate at 10-Gbps FC - 24 ports at 10-Gbps FC full rate

2 Port Groups, 1 ASIC

Example shows
outlined ports
configured as
10-Gbps FC

Note: Not all ports can be configured as 10-Gbps Fibre Channel speed. The 10-Gb Fibre Channel
enablement works at asic level and one asic controls two port groups simultaneously.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-19

All line card modules have their own port group settings. Depending on the port group
configuration, you can configure that port for regular Fibre Channel speed or 10-Gbps Fibre
Channel speed. Note that not all ports can be configured as 10-Gbps Fibre Channel speed.
The figure shows ports out of individual port groups that can be configured as 10-Gbps Fibre
Channel speed. The interfaces that can be configured out of the port groups are indicated by the
yellow border. Interfaces that will be disabled by the switch are marked with a red X. Note that
two port groups are the same and that those groups will continue for the rest of the port groups.

Port Group 1
1. The 10-Gbps Fibre Channel enablement works at the ASIC level and one ASIC controls
two port groups simultaneously.
2. In the 32-port 8-Gbps module, there are eight port groups with four ports per port group.
3. The 10-Gbps mode configuration for a group of eight ports is six ports at 10-Gbps with the
other two ports disabled.

Port Group 2
1. For the 48-port 8-Gbps module, there are eight port groups with six ports per port group,
with up to four 8-Gbps dedicated ports per port group.
2. The 10-Gbps mode configuration for a group of 12 ports is six ports at 10-Gbps with the
other six ports disabled.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-33

Port Group

E FX E

FX FX E

switch(config)# interface fc1/1

switch(config-if)# switchport rate-mode dedicated
switch(config-if)# switchport speed 8000
switch(config)# interface fc1/2-4
switch(config-if)# switchport rate-mode dedicated
Switch(config-if)# switchport speed 4000
switch(config)# interface fc1/5-6
switch(config-if)# switchport rate-mode shared
switch(config-if)# switchport speed 8000

Configure the correct port to be a dedicated 10-Gbps port to configure port bandwidth
reservation.

Port Group

E FX E

FX FX E

switch# show port-resources module 1

Module 1
1.33:1 oversubscription
…
Port-Group 1
Total bandwidth is 32.0 Gbps
Allocated dedicated bandwidth is 20.0 Gbps
--------------------------------------------------------------------
Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode
Buffers (Gbps)
--------------------------------------------------------------------
fc1/1 250 8.0 dedicated
fc1/2 32 4.0 dedicated
fc1/3 32 4.0 dedicated
fc1/4 32 4.0 dedicated
fc1/5 250 8.0 shared
fc1/6 250 8.0 shared

In the figure, the ports in Port-Group 1 are configured as the following:

 One port with a dedicated rate mode and 8-Gbps speed
 Plus three ports with a dedicated rate mode and 4-Gbps speed
 Plus two ports with a shared rate mode and 8-Gbps speed (1.33:1 oversubscription)

3-34 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Port Group 1.5:1 oversubscription
--------------------------------------------
fc1/7 32 8.0 shared
FX FX FX fc1/8 32 8.0 shared
fc1/9 32 8.0 shared
FX FX FX fc1/10 32 8.0 shared
fc1/11 32 8.0 shared
fc1/12 32 8.0 shared

----------------------------------------------
fc1/13 - 8.0 Disabled
fc1/14 - 8.0 Disabled
fc1/15 - 8.0 Disabled
fc1/16 250 10.0 Dedicated
fc1/17 250 10.0 Dedicated
fc1/18 250 10.0 Dedicated
fc1/19 250 10.0 Dedicated
fc1/20 250 10.0 Dedicated
fc1/21 250 10.0 Dedicated
fc1/22 - 8.0 Out-of-Service
fc1/23 - 8.0 Out-of-Service
fc1/24 - 8.0 Out-of-Service

In the first example that is shown in the figure, there are six ports with a shared rate mode and
8-Gbps speed (1.5:1 oversubscription) (default). In the second example, specific ports can be
configured at the 10Gb speed.

Dedicated Maximum Allowed Ports in Shared

Bandwidth per Ports that Can Mode
Port Come Up
48-port 8-Gbps 10 Gbps 24 Ports All the remaining
Advanced Fibre 8 Gbps 32 Ports ports are 8 Gbps
Channel switching shared.
module 4 Gbps 48 Ports
2 Gbps 48 Ports
1 Gbps 48 Ports
32-port 8-Gbps 10 Gbps 24 Ports All the remaining
Advanced Fibre 8 Gbps 32 Ports ports are 8 Gbps
Channel switching shared.
module 4 Gbps 32 Ports
2 Gbps 32 Ports

The figure provides a summary of the Cisco MDS 9500 Series Multilayer Director 8-Gb
Advanced Modules.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-35

Slow Drain Detection
This topic explains consequences of and recommendations for slow drain detection.

• Edge devices
- An edge device can be slow to respond for a variety of reasons:
• Server performance problems: application or operating system
• HBA problems: Driver or physical failure speed mismatches, meaning there
is one fast device and one slow device
• Nongraceful virtual machine exit on a virtualized server, resulting in packets
held in HBA buffers
• Storage subsystem performance problems, including overload
- Poorly performing tape drives
• ISLs
- Lack of B2B credits for the distance that the ISL is traversing
- The existence of slow drain at edge devices

The figure describes reasons for slow drain.

3-36 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Devices, applications, connections issues and so on may cause network
devices to react by delaying the release of B2B credits.
1. Red host issues a large 5 MB read to Red disk.
2. Host H1 is not able to process all the data packets.
3. H1 delays issuing R_RDY to port F1.
4. T1 continues sending packets to F2.
5. Because F1 is unable to send packets to H1 fast enough…
6. …eventually buffers start to become full at E1 and E2.

Switch 1 Switch 2

6
H2
E1 Single E2
ISL
5MB

2 F2 Read 1
H1
NO R_Rdy
sent
NO B2B
Credits
T1
F1
NO B2B
NO R_Rdy
5 Credits 4
3 sent

Consider the situation that is shown in the figure. Host H1 issues a large 5-MB read to Target
T1. A Small Computer Systems Interface (SCSI) packet is 2148 bytes with 36 bytes of start-of-
frame (SOF), Fibre Channel header, end-of-frame (EOF), cyclic redundancy check (CRC), and
a payload of 2112 bytes. Given these details, the response from the target will be approximately
2500 SCSI data packets.
Host H1 cannot process all the data packets and begins to delay issuing a receiver ready
(R_RDY) to the port F1 on Switch 1, creating a slow drain situation. On Switch 2, Target T1
continues to send packets to port F2 filling its virtual output queue (VOQ). However, as F1 is
unable to send packets to H1 fast enough, the packet buffers start to become full, which
cascades upstream to Port E1 and eventually E2. Now the congestion has spread in the network
such that the only link between the two switches cannot send data and the single slow device
(H1) has adversely affected the entire network performance. Given the seriousness of this
situation, it is extremely important to monitor the network for slow drain devices and detect
their presence at the earliest possible time so you can take preventive measures.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-37

• Do the following to alleviate congestion brought on by slow drain:
- Ports must be identified where slow drain is happening.
• Slow port: A port that receives Tx credits slowly
• Stuck port: A port that is continuously out of Tx credits
Just get out
- Take action to relieve congestion. of my way!

Congestion

E1 Single E2
ISL

F2
NO R_Rdy NO B2B
F1 sent Credits
NO B2B
Credits

The figure describes slow drain identification and action.

Port-monitor process with over 15 different counters monitored

• Level 1: Latency
- Check for Credit Not Available
- Lack of B2B Credits for the Distance the ISL is Traversing
- Check ISLs for Lack of Transmit Credits
- Check for Frame Queuing on Ingress Ports
- Check for Arbitration Timeouts
- Monitoring Tx and Rx B2B Transitions to Zero
• Level 2: Retransmission
- Check for Transmit Frame Drops
- Check for Frame Transmit Timeout History on Egress Port
• Level 3: Extreme Delay
- Check for Link Failure with "LR Rcvd B2B" Message
- Check for Credit Loss on an Egress Port

The figure describes how MDS switches monitor for slow drain ports.

3-38 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# show port-config internal link-events
switch# show process creditmon statistics
switch# show process creditmon credit-loss-events
switch# show hardware internal fcmac port <port_number> link-
event | include LINK_LR_TX
switch# show hardware internal statistics module <x> pktflow
dropped
switch# show hardware internal packet-flow drop
switch# show hardware internal packet-dropped-reason
switch# show logging onboard module 13 error-stats
switch# show system internal snmp credit-not-available
switch# show interface | include "fc|Belong|low priority" |
exclude "description" | exclude "Peer" | include "trunking"
next 2

The figure describes port monitoring.

• Error disable or flap the port through port monitor port guard
• Dropping packets when the congestion-drop threshold is reached
switch(config)# system timeout congestion-drop <value> mode (F)/(E)
switch(config)# system timeout congestion-drop default mode (F)/(E)

• Setting the no-credit-drop timeout value

switch(config)# system timeout no-credit-drop <value> mode F
switch(config)# system timeout no-credit-drop default mode F

• Results in automatic credit loss recovery

The figure describes actions that can be used to relieve congestion.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-39

Explain Interface BB_Credits
This topic explains interface buffer-to-buffer credits (BB_Credits) on Fibre Channel interfaces.
Buffer credits are an inherent part of the Fibre Channel standards. Buffer credits can limit the
overall throughput on a link if insufficient BB_Credits are available to support the link
distance.

• At login, adjacent ports tell each other how many buffers they have
(BB_Credit).

Next
port has
8 buffers
3
2
1

I have 4 I have 8
buffers buffers

I have 8 I have 8
Buffers
buffers buffers

The base credit management method works as follows:

When the transmitting port sends a port login request, the receiver responds with an accept
(ACC) frame that includes information on the size and number of frame buffers the receiver has
(BB_Credit). The transmitting port stores the BB_Credit value in a table.
The transmitting port also stores another value that is called BB_Credit_CNT, which represents
the number of “used” buffer credits. BB_Credit_CNT is set to zero after the ports complete the
login process.
Each time the transmitting port sends a frame, the BB_Credit_CNT increments.
Upon receiving the frame, the receiver processes the frame and moves the frame to upper-layer
protocol (ULP) buffer space. The receiving port then sends an R_RDY acknowledgement
signal back to the transmitting port, informing the port that a buffer is available.
When the transmitting port receives the R_RDY signal, the port decrements its
BB_Credit_CNT.
To prevent overrunning the buffers of the receiving port, the transmitting port can never allow
BB_Credit_CNT (the count of frames that have not yet been acknowledged) to exceed
BB_Credit (the total number of buffers in the receiving port). In other words, if the port cannot
confirm that the receiving port has a free buffer, the port does not send any more frames.

3-40 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• During data flow, the following occurs
- Upon receiving (1) a B2B credit signal from the FC switch, the server will
transmit (2) a frame to the switch.
- The switch will hold the frame in its buffer (3) until the switch receives (4) a
B2B credit signal from the destination device. After a credit is received, the
switch then transmits (5) the frame to its destination and will release (6/1) a
B2B credit to the server so the server can send another frame.
- Tx sends only when BB_Credit_CNT < BB_Credit.

Next port
has 1 free
buffer
BB_Credit: 8
BB_Credit_CNT: 0 7
4
3
2
1 Transmit Transmit
Frame Frame
3
2 5 PLOGI
DATAFrame

Tx R_RDY Receive Receive Rx

B2B Credit Buffers B2B Credit

1 4
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-33

The figure describes data flow.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-41

BB_Credit Buffer Allocation 8-Gb Module 16-Gb Module
Dedicated Shared Dedicated
ISL (E) (Fx) (Fx) ISL (E) (Fx)
Default 48 port 250 32 32 500 32
32 port 500 32 32 N/A N/A
Maximum user configuration 500 500 32 500 500
Maximum with Enterprise 4095 4095 N/A 4095 4095
license

Max per port group with license 4095 4095 4095

Max user-configurable 32x500 48x500 48x500

Default dedicated (Fx or E) 32x500 48x250 48x500

Default shared (Fx) (32x32) (48x 32) N/A

32-Port 1/2/4/8/10G 48-Port 1/2/4/8/10G 48-Port 2/4/8/10/16G

8-Gb and 16-Gb Generation BB_Credit Allocation

By default, Fx Ports are allocated 32 BB_Credits and E and TE Ports are allocated 250 credits
(8Gb) and 500 credits (16Gb). Any port can configure between 2 and 500 BB_Credits per port.
Ports that are configured in dedicated mode may use licensed, extended BB_Credits, up to 4095
credits per port. The port group limit is 4095, so if one port uses 4095, the other ports do not
have credits. This limit is the highest BB_Credits per port in the industry.
The following are the requirements to maximize link bandwidth and to ensure that frames are
not delayed while waiting for credits:
 One BB_Credit is required per 2 km at 1 Gbps with a 2-KB frame payload.
 One BB_Credit is required per km at 2 Gbps with a 2-KB frame payload.
 Two BB_Credits are required per km at 4 Gbps with a 2-KB frame payload.
 Four BB_Credits are required per km at 8 Gbps with a 2-KB frame payload.
 Eight BB_Credits are required per km at 16Gbps with a 2-KB frame payload.

Extended Credits
Full line-rate Fibre Channel ports provide at least 255 buffer credits as the standard. Adding
credits lengthens distances for Fibre Channel SAN extension. Extended credits, up to 4095
buffer credits, can be allocated to ports as needed to greatly extend the distance of Fibre
Channel SANs.

3-42 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Shared mode ports have 32 to 255 BB_Credits (depending on module).
• Dedicated mode ports have up to 500 BB_Credits.
• A license may be required to optimize bandwidth over distances.
switch(config-if)# switchport fcrxbbcredit ?
<1-500> Enter receive BB_credit
default Default receive BB_credit
performance-buffers Configure performance buffers for receive
BB_credit
switch(config-if)# switchport fcrxbbcredit 56

• To configure extended BB_Credits (with license), use the following:

switch(config)# feature fcrxbbcredit extended
switch(config)# interface fc1/1
switch(config-if)# switchport fcrxbbcredit extended 1500
switch# show interface 1/1
fc1/1 is up
…..
Transmit B2B Credit is 500
Receive B2B Credit is 1500
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-35

The figure shows how to configure interface BB_Credits and receive buffers.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-43

• The following configurations should be performed with extreme care. In
most cases, the default values should be sufficient.
- Configuration of maximum receive buffer size on an interface:
• A default of 2112 bytes will receive payload sizes up to 2112 bytes.
• Modification may be required for interoperability with legacy devices.
- Configuration of Fibre Channel timers:
• Values for E_D_TOV and R_A_TOV are used in Exchange Link Protocol
and FLOGI and can be modified on a global basis. (The values must be the
same on all switches within a VSAN.)
• Any modification does not affect the active ports that are in the up state, and
the changes take effect only for subsequent logins (ELP or FLOGI).

Configurations of error detect timeout value (E_D_TOV) and resource allocation timeout
values (R_A_TOV) are on a per-switch basis.
These configurations should be made before connecting to an existing fabric, because the
values are exchanged during the fabric login (FLOGI) process.
See this link for more information about buffer credits:
http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/6_2/configuration/guides/i
nterfaces/nx-os/buffers.html

3-44 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configuring Trunking
This topic explains how to configure the trunking feature of Cisco MDS 9000 Series switches.

• Trunking allows interfaces to carry frames that belong to different

VSANs. Trunk mode is on by default.

EISL or ISL
(TE) (E)
1 2

Administrative Trunk Administrative Trunk Operational Trunk Mode

Mode (1) Mode (2)
On On TE Port
On Off E Port
On Auto TE Port
Auto Auto E Port

Trunking mode is specific to Cisco MDS 9000 Series switches. If equipment that is attached to
an ISL E Port does not support trunk mode, the parameter is ignored. Therefore, equipment of
other vendors will not operate with the Cisco MDS 9000 Series switches in the trunking mode.
The equipment operates as a regular E Port.
Trunking mode operates with one Cisco MDS 9000 Series switch that is set to trunking mode
“auto” and the other set to trunking mode “on.” However, it is recommended to set both ends to
“on” if you use the link in a port channel, which you will generally do. Note that the port
channel demands only that all interfaces on one end of the channel have the same
configuration.
In TE mode, an allowed list of VSANs can be configured for the interface. By default, all
VSAN traffic is allowed.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-45

• Configure a TE port for interface fc1/2, mode E, speed of 2-Gbps,
trunking:
switch# conf
Trunk mode is usually
switch(config)# interface fc1/2
on, because that mode
switch(config-if)# switchport rate-mode dedicated is set during the
switch(config-if)# switchport mode e original switch setup.
switch(config-if)# switchport trunk mode on
switch(config-if)# switchport speed 2000
switch(config-if)# no shutdown
switch(config-if)# end
switch # show trunk protocol
Trunk protocol is enabled.
switch# show interface fc1/2 brief
-------------------------------------------------------------
Interface Vsan Admin Admin Status FCOT Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------
fc1/2 1 E on up swl TE 2 --
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-39

The switchport mode e command configures the interface for E Port operation. Enter these
commands to configure interface fc1/2 as an E Port, with a speed of 2 Gbps and trunking
enabled:
conf
interface fc1/2
switchport rate-mode dedicated
switchport mode e
switchport trunk mode on (default)
switchport speed 2000
no shutdown

To verify your configuration use variations of the following commands.

show interface
show trunk protocol,
show interface trunk vsan #

3-46 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The Trunk Allowed VSAN List is used to selectively block or allow
frames through the interface. All frames are allowed by default.

switch(config-if)# switchport trunk ?

allowed Configure allowed list for interface(s)
mode Configure trunking mode
switch(config-if)# switchport trunk allowed ?
vsan Configure allowed list for interface(s)

switch(config-if)# switchport trunk allowed vsan 1

switch(config-if)# switchport trunk allowed vsan add 20
updated trunking membership
Without the add
switch(config-if)# switchport trunk mode ? option, any existing
auto Trunk mode auto configuration is
off Trunk mode off overwritten
on Trunk mode on

switch(config-if)# switchport trunk mode on

The switchport trunk ? command returns two options, which are the allowed VSAN list and
the trunk mode configuration. In a LAN environment, these options are the same as manual
pruning.
Remember that a TE Port allows traffic only for active allowed VSANs. If you do not configure
the VSAN allowed list correctly, certain VSAN traffic is not able to traverse the link. The
trunking mode should be set to “on” for TE Ports and “off” for E Ports.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-47

Summary
This topic summarizes the key points that were discussed in this lesson.

• The switch interfaces support various port types, depending on the

device type that is connecting.
• An understanding of port groups and bandwidth sharing may improve
the use of certain modules.
• Interface configuration for slow drain can improve the traffic flow to
uninvolved devices.
• Depending on the series of switch or module, different BB_Credits are
supported.
• The trunking feature of the Cisco MDS 9000 Series is used to carry
multiple VSAN traffic over the same physical connection.

References
For additional information, refer to these resources:
 See this link for more information about buffer credits:
http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/5_2/configuration/guid
es/int/nx-os/buffers.html#wp1108272

3-48 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 3

Configuring Port Channels

Overview
This lesson describes how to configure and manage port channels with trunking expansion (TE)
ports and trunking fabric (TF) ports.

Objectives
Upon completing this lesson, you will be able to configure port channels. This ability includes
being able to meet these objectives:
 Describe the purpose and use of port channel functionality on the Cisco MDS 9000 Series
Multilayer Switches
 Explain how to configure port channels with expansion (E) and TE Ports
Port Channel Overview
This topic defines port channels and their function over Inter-Switch Links (ISLs).

• A port channel provides the following functionality:

- Increases the aggregate bandwidth
- Balances loads across multiple links and maintains optimum bandwidth
utilization
- Provides fault tolerance on an ISL
- Can include up to 16 ISLs in a port channel
- Up to 256 port channels per switch

Single port channel between Multiple port channels between

two MDS switches two MDS switches

The Cisco MDS 9000 Series PortChannel is a proprietary feature that can provide an
aggregation of multiple physical interfaces into one logical interface. Port channels provide
higher aggregated bandwidth, load balancing, and link redundancy. Port channels can connect
to interfaces across switching modules, so a failure of a switching module cannot bring down
the port channel link.
A port channel has the following features:
 Provides a point-to-point connection over ISL E Ports or Enhanced ISL (EISL) TE Ports.
Multiple links can be combined in a port channel.
 Increases the aggregate bandwidth on an ISL by distributing traffic among all functional
links in the channel.
 Load-balances across multiple links and maintains optimum bandwidth utilization. Load
balancing is based on the source ID (SID), destination ID (DID), and originator exchange
ID (OXID).
 Provides high availability on an ISL. If one link fails, traffic that was previously carried on
this link is switched to the remaining links. If a link goes down in a port channel, the upper
protocol is not aware of the link going down. To the upper protocol, the link is still there,
although the bandwidth is diminished. The routing tables are not affected by link failure.
Port channels can contain up to 16 physical links and can span multiple modules for added
high availability.

Cisco MDS 9000 Series switches provide support for port channels with 16 ISLs per port
channel with a maximum of 256 port channels.

3-50 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Configure the port channel across switching modules for redundancy.
• Port channels require point-to-point connections between the same pair
of switches.
Valid Invalid
1
2
1 3
1
A B A 4 B

1
2
2 3 2
4
A B A B

1
2
1 3 1
4
A A

Before configuring a port channel, you should consider the following physical guidelines:
 Configure the port channel across switching modules to provide redundancy on switching-
module reboots or upgrades. (This action is a best practice.)
 Ensure that one port channel is not connected to different sets of switches. Port channels
require point-to-point connections between the same set of switches.

Error Detection
If you invalidly connect port channels, you may receive an error message. If you receive an
error message, the port channel physical links are disabled after an error has been detected. A
port channel error is detected when the following requirements are not met:
1. Each switch on either side of a port channel must have a port channel with the same
number of interfaces.
2. Each port channel interface must be connected to a similar (same number of physical links)
interface on the other side. A 4-port port channel interface on one switch must be connected
to a 4-port port channel interface on a second switch. If a link goes down, the configuration
is still a valid.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-51

There are two load-balancing mechanisms:
Flow-based: Frames between Exchange-based: Frames in the
source and destination follow the same exchange follow the same
same links. link (default).

Link 1 Link 1
SID1, DID1 SID1, DID1, OX1

SID1, DID1 SID1, DID1, OX1

Link 2 Link 2
SID2, DID2 SID1, DID1, OX2

SID2, DID2 SID1, DID1, OX2

Two mechanisms support the load-balancing functionality:

 Flow-based: All frames between source and destination follow the same links for a given
flow. This mechanism means that whichever link is selected for the first exchange of the
flow is used for all subsequent exchanges.
 Exchange-based: The first frame in an exchange chooses a link, and subsequent frames in
the exchange follow the same link. However, subsequent exchanges can use a different
link. This process provides more granular load balancing while preserving the order of
frames for each exchange.

The left side of the figure illustrates how source ID 1 (SID1) and destination ID1 (DID1)-based
load balancing works. When the first frame in a flow is received on an interface for forwarding,
Link 1 is selected. Each subsequent frame in that flow is sent over the same link. None of the
frames in SID1 and DID1 use Link 2.
The right side of the figure illustrates how exchange-based load balancing works. When the
first frame in an exchange is received for forwarding, Link 1 is chosen by a hash algorithm. All
remaining frames in that particular exchange are sent on the same link. For exchange 1, no
frame uses Link 2. For the next exchange, Link 2 is chosen by the hash algorithm. Now all
frames in exchange 2 use Link 2.

3-52 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• On mode
- Member ports only operate as
part of a port channel.
- Explicitly enable or disable ports
at the end if you add or remove
ports from a port channel.
- Requires manual verification for
connection.
• Active mode (best practice)
- Member ports initiate port
channel protocol negotiation with
peer.
- Supports autorecovery without
disabling or enabling member
ports.
- Provides automatic verification of
connection.
- F port channels only support
active mode.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-7

You can configure each port channel with a channel group mode parameter to determine the
port channel protocol behavior for all member ports in the channel group. The possible values
for a channel group mode are as follows:
 On mode (default): The member ports only operate as part of a port channel or remain
inactive. In this mode, the port channel protocol is not initiated. However, if a port channel
protocol frame is received from a peer port, the software indicates its nonnegotiable status.
This mode is backward compatible with the existing implementation of port channels in
releases before Release 2.0(1b), where the channel group mode is implicitly assumed to be
on. In Cisco MDS SAN-OS Releases 1.3 and earlier, the only available port channel mode
was the on mode. Port channels configured in the on mode require you to explicitly enable
and disable the port channel member ports at either end if you add or remove ports from the
port channel configuration. You must physically verify that the local and remote ports are
connected to each other.
 Active mode: The member ports initiate port channel protocol negotiation with the peer
port regardless of the channel group mode of the peer port. If the peer port, while
configured in a channel group, does not support the port channel protocol, or responds with
a nonnegotiable status, the peer will default to the on mode behavior. The active port
channel mode allows automatic recovery without explicitly enabling and disabling the port
channel member ports at either end.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-53

Port Channel Configuration for E and TE Ports
This topic describes the commands that are used for creating a port channel.

• The following shows use of the configuration mode interface port-

channel number command:

switch(config)# interface port-channel 1 switch(config)# interface port-channel 1

switch(config)# end switch(config)# end
switch# show port-channel database switch# show port-channel database
port-channel 1 port-channel 1
Administrative channel mode is on Administrative channel mode is on
Operational channel mode is on Operational channel mode is on
Last membership update succeeded Last membership update succeeded
No port No port

fc1/1 fc1/7 FC

MDS A MDS B
fc2/8 fc3/5

This figure describes a basic ISL configuration on peer switches. Optionally, you can create the
port channel interface by first using the interface port-channel command, and then, in a
separate task, add the Fibre Channel interfaces. Port channels are created with default values.
You can change the default configuration just like you can with any other physical interface.
To create a port channel, use the interface port-channel command in configuration mode:
switch(config)# interface port-channel 1
The valid range for a port channel interface ID is 1 to 256. The configuration process must be
repeated on both switches.

3-54 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• The following shows use of the interface configuration channel-group
command:
switch(config)# interface fc1/1 switch(config)# interface fc1/7
switch(config-if)# channel-group 1 switch(config-if)# channel-group 1
switch(config-if)# no shutdown switch(config-if)# no shutdown

switch(config-if)# interface fc2/8 switch(config-if)# interface fc3/5

switch(config-if)# channel-group 1 switch(config-if)# channel-group 1
switch(config-if)# no shutdown switch(config-if)# no shutdown

channel-group 1 FC

FC
fc1/1 fc1/7

MDS A MDS B
fc2/8 fc3/5

To add Fibre Channel interfaces to a port channel, use the channel-group command in
interface mode. If the port channel exists, the interfaces are added. If the port channel does not
exist, then the port channel is created.
You can add a physical interface (or a range of interfaces) to either an existing port channel or a
nonexistent port channel. If the port channel does not exist, the port channel is automatically
created. The compatible parameters on the configuration are mapped to the port channel.
To add interfaces to a port channel, use the channel-group command in interface configuration
mode. The following command sequence adds fc1/1 and fc2/8 port channel 1 on MDS A:
MDSA(config)# interface fc1/1
MDSA (config-if)# channel-group 1
MDSA (config-if)# interface fc2/8
MDSA (config-if)# channel-group 1
The configuration process must be repeated on MDS B for interfaces fc1/7 and fc3/5.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-55

• Using the channel-group command against a range of interfaces is
potentially disruptive.
• Interfaces are temporarily disabled and enabled during the creation of
the port channel.

switch(config)# interface fc1/1,fc2/8

switch(config-if)# channel-group 1
switch(config-if)# no shutdown

switch(config)# interface fc1/7,fc3/5

switch(config-if)# channel-group 1
switch(config-if)# no shutdown

FC
channel-group 1
FC
fc1/1 fc1/7

MDS A MDS B
fc2/8 fc3/5

Using the channel-group command against a range of interfaces or with multiple interfaces
that are selected with the interface command is potentially disruptive. This disruption occurs if
there is not an additional ISL between the switches that is not selected for inclusion with the
channel-group command. Interfaces are temporarily disabled and enabled while being added
to the port channel.

3-56 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Adding port channel links might be disruptive:
- Corresponding ports are shut down.
- If possible, add ports before they are enabled.
• Disabling port channel links with no frame loss:
- Links automatically quiesced upon shutdown command.
- Cisco NX-OS provides graceful ISL shutdown.

Whenever links are administratively added to or deleted from a port channel, the corresponding
interfaces are shut down. To avoid traffic disruption when adding links to port channels, the
interfaces should be added before the port channel interfaces are enabled.
Administratively disabling port channel links is possible with no frame loss.
The software implicitly performs a graceful shutdown in response to disabling interfaces that
are operating in E Port mode. A graceful shutdown ensures that no frames are lost when the
interface is shutting down. When a shutdown is triggered, either by an administrator or by
software, the switches that are connected to the link coordinate to ensure that all frames in the
ports are safely sent through the link before shutting down. This enhancement reduces the
chance of frame loss.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-57

• Use the following command to display port channel compatibility
parameters from the CLI:

switch# show port-channel compatibility-parameters

Parameters that have to be consistent across all members in a port-channel.
1. physical port layer
Members must have the same interface type, such as Fibre Channel, ethernet or
FCIP.

2. port mode
Members must have the same port mode configured, either E or AUTO. If they are
configured in AUTO port mode, they have to negotiate E mode when they come up.
If a member negotiates a different mode, it will be suspended.

3. trunk mode
Members must have the same trunk mode configured. If they are configured in AUTO
trunking mode, they have to negotiate the same trunking mode when they come up.
If a member negotiates a different mode, it will be suspended.
. . .

Use the compatibility-parameters command in EXEC mode:

switch# show port-channel compatibility-parameters
Parameters that must be consistent across all members in a port channel are listed.

3-58 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Parameter Description

Physical port layer Members must have the same interface type, such as Fibre Channel, Ethernet, or
FCIP.

portmode Members must have the same port mode configured, either E or AUTO. If
members are configured in AUTO port mode, the members must negotiate E mode
when the members come up. If a member negotiates a different mode, that
member is suspended.

Trunk mode Members must have the same trunk mode configured. If members are configured
in AUTO trunking mode, members must negotiate the same trunking mode when
the member comes up. If a member negotiates a different mode, the member is
suspended.

Speed Members must have the same speed configured. If members are configured in
AUTO speed, members must negotiate the same speed when the member comes
up. If a member negotiates a different speed, the member is suspended.

MTU Members must have the same maximum transmission unit (MTU) configured. This
requirement applies only to Ethernet port channels.

Ethernet port This parameter applies only to Ethernet port channel. Each Ethernet port channel
index can have only two Ethernet ports. The ports must be in the same slot, their port
indices must be adjacent, and the lower number must be odd, for example, Gigabit
Ethernet 8/5–6.

portVSAN Members must have the same port VSAN.

IP address Members must not have an IP address configured. This restriction applies only to
Ethernet port channels.

Port-security Members must all be permitted by the activated port-security bindings and fabric-
active bindings bindings in all the allowed VSANs.

FC receive buffer Members must have the same Fibre Channel receive buffer size. If the configured
size receive buffer size is not compatible with the port capability, the port is error
disabled.

IP ACLs Members must not have IP access control lists (ACLs) configured individually on
them. This restriction applies only to Ethernet port channels.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-59

• Click the Port Channel Wizard icon, and then choose Select Switch Pair.

Use the following procedure to create a port channel from the Cisco Prime Data Center
Network Manager for SAN Client (DCNM-SAN Client):
Step 1 Click the Port Channel Wizard icon in the Cisco DCNM-SAN Client toolbar.
Step 2 To add a link to an existing port channel, right-click an ISL on the Cisco DCNM-
SAN Client map and select Add to Port Channel from the pop-up menu. The Port
Channel Wizard is displayed.
Step 3 The Port Channel Wizard in Cisco DCNM-SAN Client steps you through three
dialog boxes that request information that is required to complete the port channel
configuration:
 Select a switch pair: Highlight the switch pair that the port channel will
connect. In the figure, only one pair is displayed. Click Next.
 Select ISLs: Highlight the ISLs to add to the port channel. Click Next.
 Create the port channel: To complete the process, the last dialog box allows
you to specify the trunk mode options (nontrunk, trunk, or auto). If you select
the trunk option, the VSAN List window lets you prune the VSANs that are
allowed over the port channel.
Step 4 Click Finish.
Optionally, a faster method is to launch the Port Channel Wizard from the fabric map. Right-
click one of the Fibre Channel over IP (FCIP) links and select Create Port Channel. The
wizard launches immediately into the second dialog box (2 of 3: Select ISLs Displaying the
FCIP Link That Is Already Selected) and allows you to select additional links. The wizard then
continues as described.

Note These steps must be performed after you have already brought up the EISL links.

3-60 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Verify that ISLs are selected.

Choose trunk mode.

Click Yes to continue.

The figure displays Steps 3 and 4 of the Port Channel Wizard:

Upon completing Step 4, a dialog box appears warning you that converting the ISLs to a port
channel will be disruptive. The port channel interfaces are reset during the creation process,
causing a momentary disruption of service.
Step 5 Click Yes to continue and complete port channel creation. Be aware that this process
can be disruptive if ports are brought up or down during this operation.
When the wizard closes, use Cisco DCNM-SAN Client and the CLI to verify that the port
channel is operating.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-61

switch# show port-channel database
port-channel 1
Administrative channel mode is active
Operational channel mode is active
Last membership update succeeded
First operational port is fc1/10
1 port in total, 1 port up
Ports: fc1/10 [up] *

switch# show port-channel summary

------------------------------------------------------------------------
Interface Total Ports Oper Ports First Oper Port
------------------------------------------------------------------------
port-channel 1 1 1 fc1/10

The show port-channel summary command displays a summary of the port channels that are
configured on the switch. The one-line summary provides the following information:
 Administrative state
 Operational state
 Number of attached and active interfaces (up)

The first operational port, which is the primary operational interface that is selected in the port
channel to carry control-plane traffic. The first operational port is identified by the asterisk (*).

3-62 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• Port channels are an aggregation of multiple physical interfaces into one

logical interface. Port channels provide higher aggregated bandwidth,
load balancing, and link redundancy.
• E and TE port channels are supported between two switches.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-63

3-64 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 4

Configuring Cisco NPV and

NPIV
Overview
Fibre Channel standards limit the number of switch domain IDs to a maximum of 239 per
fabric or virtual SAN (VSAN), but only about 40 are supported today. This limitation restricts
the number of blade switches inside the blade chassis that can be deployed in a fabric or
VSAN. Cisco N-Port Virtualizer (NPV) was introduced to overcome this restriction. This
lesson explains Cisco NPV and N-Port ID Virtualization (NPIV), a feature that allows a single
Fibre Channel switch port to both manage multiple logins and assign multiple Fibre Channel
IDs (FCIDs).

Objectives
Upon completing this lesson, you will be able to configure both Cisco NPV and NPIV. This
ability includes being able to meet these objectives:
 Describe the benefits of Cisco NPV
 Describe how NPIV provides a means to assign multiple FCIDs to a single node (N) Port
 Describe how to configure F and TF port channels
Cisco NPV
This topic describes the benefits of Cisco NPV.

FC FC
• Scalability
FC FC

- Each blade switch uses a single

domain ID.
- The theoretical maximum number
of domain IDs is 239 per VSAN.
- The supported number of domains
is quite smaller (and depends on
OSM).
• EMC: 40 domains
• Cisco tested: 100
• Other OSMs do not specify
• Manageability
- There are more switches to
manage.
- There is shared management
of blade switches between
storage and server
administrators

The Fibre Channel standards that are defined by the ANSI T11 committee allow for up to 239
Fibre Channel domains per fabric or VSAN. However, original storage manufacturers (OSMs)
have only qualified up to 40 domains per fabric or VSAN.
Each Fibre Channel switch is identified by a single domain ID, thus there can be no more than
40 switches that are connected together.
Blade switches and top-of-rack (TOR) access layer switches consume domain IDs, which limits
the number that can be deployed in data centers.

3-66 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Cisco MDS switch mode
- All Fibre Channel services are provided:
• Supported services include FLOGI, name server, zoning, domain server, FSPF,
and management.
• FSPF, zoning, and name server databases are distributed among connected
switches.
- Local switching is enabled.
- ISL between switches becomes a path within the FSPF routing table.
- Up to 16 ISLs may belong to a port channel.
- Each switch consumes a domain ID.
• Cisco NPV mode for MDS 9100 Series switches
- Most Fibre Channel services are switched off.
- This mode provides no ISLs, trunking, port channels, or QoS.
- The Cisco NPV-enabled switch becomes a multiplexor for connected devices.
- The Cisco NPV switch does not use a domain ID.
• The switch is not subject to domain ID limitation.
- There is a smaller number of switches to manage.
- This mode eliminates the need for server administrators to manage the SAN.

Cisco MDS 9100 Series Multilayer Fabric Switches and blade switches support two operating
modes, which are MDS switch mode and Cisco NPV mode.
MDS switch mode provides the following features:
 All Fibre Channel services are provided.
— Supported services include fabric login (FLOGI), name server, zoning, domain
server, Fabric Shortest Path First (FSPF), and management.
— FSPF, zoning, and name server databases are distributed among connected switches.
 Local switching is enabled.
 Inter-Switch Link (ISL) between switches becomes a path within the FSPF routing table.
 Up to 16 ISLs may belong to a port channel.
 Each switch consumes a domain ID.

Cisco NPV mode provides the following features:

 Most Fibre Channel services are switched off.
 This mode provides no ISLs, trunking, port channels, or quality of service (QoS).
 The Cisco NPV-enabled switch becomes a multiplexor for connected devices.
 The Cisco NPV switch does not use a domain ID, which means the switch is not subject to
domain ID limitation.
 There is a smaller number of switches to manage.
 This mode eliminates the need for server administrators to manage the SAN.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-67

• Cisco NPV edge switches
- Need to enable switch in Cisco NPV Servers
mode.
- Changing to or from Cisco NPV mode is
disruptive:
• Causes switch reboots.
MDS
• Configuration is not kept. 9100
Blade
Switches
- Supports only F, SD, and NP modes.
- Supports 16 VSANs.
- Local switching is not supported.
• Switching is done at the core.
• Cisco NPV core switches NPIV-enabled MDS 9200
- Must enable the NPIV feature. MDS 9500
- Supports up to 105 Cisco NPV edge
switches, 500 FLOGIs per line card,
2500 total FLOGIs.
FC FC
• Cisco NPV-enabled switches are
standards-based and interoperable
with other third-party switches in the
SAN.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-6

Cisco NPV mode is disruptive. During reboot, the original switch configuration is lost and the
switch comes back up in Cisco NPV mode as a multiplexor of connected devices.
Cisco NPV edge switches are essentially transparent to the fabric and most Fibre Channel
services of the switches are disabled. The switches do not have a domain ID and therefore do
not appear in the FSPF routing table.
Cisco NPV edge switches pass traffic between end devices and Cisco NPV core switches that
provide Fibre Channel services for the connected devices.
The Cisco NPV core switch must enable the NPIV feature to support FLOGI from each device
that is connected through the Cisco NPV edge switch.
Cisco NPV-enabled switches are standards-based and interoperable with third-party switches in
the SAN. Cisco supports four interoperation modes to accommodate third-party switches:
 Interop mode 1: The ANSI T11 FC-SW-2 standard enables Cisco MDS 9000 Series
switches to interoperate with third-party switches that have been configured for
interoperability. Interop mode 1 allows Cisco MDS 9000 Series switches to communicate
over a standard set of protocols with these switches.
 Interop mode 2: This mode allows seamless integration with older Brocade switches that
use a restrictive process ID (PID) format (PID=0) that permits only 16 devices per domain
(Brocade models 2100, 2400, 2800, and 3800 Series). Core PID format 0 (CORE PID=0) is
common in Brocade fabrics that do not include a model 200e, 3900, 4100, 4900, 12000, or
24000.
 Interop mode 3: This mode allows seamless integration with specific Brocade switches
(3900, 12000, and 24000) running CORE PID=1. This format requires other lower-end
models to upgrade, which is a disruptive operation.
 Interop mode 4: This mode allows seamless integration with McData, available in SAN-
OS 3.0. This mode requires manual configuration of the switch world wide name (sWWN)
according to the IEEE specification before changing the VSAN interop attribute. The
allowed domain ID range is 1 to 31.

3-68 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Cisco NPV edge switches
- Cisco MDS 9148 Multilayer
Fabric Switch
MDS 9148 Multilayer
- Cisco Nexus 4000 Series Blade Fabric Switch
Switch
- Cisco Nexus 5000 and 5500
Series Switches
- Cisco UCS 6200 Series Fabric
Interconnects
• Cisco NPV core switches
(support NPIV)
- Cisco MDS 9710 and 9500 Series
Multilayer Directors
MDS 9506, 9513, 9710 Multilayer Directors
- Cisco MDS 9222i Multiservice
Modular Switch and Cisco MDS
9250i Multiservice Fabric Switch
MDS 9250i MDS 9222i
- Third-party switches Multiservice Multiservice
Fabric Switch Modular Switch
• Need to support NPIV
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-7

Cisco NPV is supported on the following Cisco NPV edge switches:

 Cisco MDS 9148 Multilayer Fabric Switch
 Cisco Nexus 4000 Series Blade Switch
 Cisco Nexus 5000 and 5500 Series Switches
 Cisco UCS 6200 Series Fabric Interconnect (Cisco NPV is the default.)

NPIV is supported on the following core switches:

 Cisco MDS 9710 Series Multilayer Director
 Cisco MDS 9500 Series Multilayer Director
 Cisco MDS 9250i Multiservice Fabric Switch
 Cisco MDS 9222i Multiservice Modular Switch

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-69

Cisco NPV edge switch:
switch# conf
switch(config)# npv enable (followed by reboot)
switch(config)# interface fc 1/1
switch(config-if)# switchport mode NP
switch(config-if)# no shutdown
switch(config-if)# vsan database
switch(config-vsan-db)# vsan 8 interface fc 1/1
switch(config-vsan-db)# interface fc 1/2-6
switch(config-if)# switchport mode F
switch(config-if)# no shutdown
switch(config-if)# vsan database
switch(config-vsan-db)# vsan 12 interface fc 1/2-6
switch(config-vsan-db)#

N Port F Port NP Port F Port

fc1/2 fc1/1 fc2/1
Cisco NPV Edge Switch

Cisco NPV Core Switch

3-70 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
N-Port ID Virtualization
This topic describes how NPIV allows assignment of multiple FCIDs to a single node (N) port.

• NPIV provides a way to assign multiple FCIDs to a single N Port. NPIV is

required on the Cisco NPV core switch to accept multiple logins from end
devices connected to the Cisco NPV edge switch.
• NPIV also allows multiple applications to share the same HBA port. The use of
different pWWNs allows access control, zoning, and port security to be
implemented at the application level.
• Usage also applies to virtual server applications such as VMware ESX,
Microsoft Hyper-V, and Linux Xen Server.

Email Email I/O

N_Port_ID 1 HBA F_Port
Fibre Channel Link
Web Web I/O
N_Port_ID 2

File Services File Services I/O Each VM will log into the switch
N_Port_ID 3 with its virtual pWWN and each VM
will receive a unique FCID. MDS Switch
Virtual Servers

A single N Port on the switch can only support FLOGI from a single N Port device. The N Port
sends a FLOGI and receives its unique FCID from the switch.
NPIV provides a way to assign multiple FCIDs to a single N Port and therefore NPIV allows
multiple applications to share the same host bus adapter (HBA) port.
The use of different pWWNs allows access control, zoning, and port security to be
implemented at the application level.
Usage applies to virtual server applications such as VMware ESX, Microsoft Hyper-V, and
Linux Xen Server.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-71

• The Cisco NPV-enabled switch MDS #1
NPIV-enabled
MDS #2
NPIV-enabled
acts as an NPIV proxy.
• The NPIV proxy captures all
FLOGI frames from the HBA and
external interfaces, and resends
the frames to the switch as FDISC F Port F Port
frames. FDISC FDISC
NP Port NP Port
• Hosts are pinned to external Cisco NPV
interfaces. Edge
F Port F Port F Port Switch
• Supports NPIV over server
interfaces.
• Relies on NPIV on external
interfaces. HBA #1 HBA #2 HBA #3
(flogi)
• Retries failed login requests from (Flogi) (flogi)

one external interface on a

different interface.
• Manages events by generating
proxy logos.

The Cisco NPV edge switch acts as a proxy for the end devices. The NPIV proxy captures all
FLOGI frames from the HBA and external interfaces and resends them to the switch as fabric
discovery (FDISC) frames. To simplify Cisco NPV switch operation, hosts are pinned to their
respective external interfaces or uplinks that, in turn, connect to the Cisco NPV core switch.

NPIV on core switch:

switch# conf
switch(config)# feature npiv

switch(config)# interface fc 2/1

switch(config-if)# switchport mode F
switch(config-if)# no shutdown

switch(config-if)# vsan database

switch(config-vsan-db)# vsan 8 interface fc 2/1

N Port F Port NP Port F Port

fc1/2 fc1/1 fc2/1
Cisco NPV Edge Switch

NPIV on Core Switch

3-72 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
F and TF Port Channel Configurations
This topic describes F and TF port channels, their restrictions, and the commands that are used
for configuration.

NP Port F Port

Cisco NPV Edge Switch

TNP Port TF Port

Trunking Enabled
Cisco NPV Edge Switch

An F port channel is a logical interface that combines a set of fabric ports (F Ports) that are
connected to the same Fibre Channel node. The F port channel operates as one link between the
F Port and the NP Port. An F port channel supports the same bandwidth utilization and
availability as E port channels. The F port channel is used mainly between a Cisco MDS 9000
Series core and Cisco NPV switch, to provide optimal bandwidth utilization and transparent
failover between the uplinks of a VSAN.
A TF port channel is a trunking F Port on a Cisco MDS 9500 Series switch carrying multiple
VSAN traffic to and from an NP Port on a Cisco MDS switch in Cisco NPV mode.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-73

On the Cisco NPV Core Switch, do the following:
1. Enable features.
switch(config)# feature fport-channel-trunk
switch(config)# feature npiv

2. Configure port channel.

switch(config)# interface port-channel 1
switch(config-if)# switchport mode F
switch(config-if)# channel mode active
switch(config-if)# switchport trunk mode off
switch(config-if)# switchport rate-mode shared

3. Configure interfaces.
switch(config)# interface fc 2/1-2
switch(config-if)# shut
switch(config-if)# switchport mode F
switch(config-if)# switchport trunk mode off
switch(config-if)# switchport speed 4000
switch(config-if)# switchport rate-mode shared
switch(config-if)# channel-group 1
switch(config-if)# no shut

The figure shows the commands that are used to configure an F port channel in shared mode. A
port in shared mode shares the remaining bandwidth in the port group with other shared mode
ports, after ports in dedicated mode have received their allocation.
Also shown are the commands to bring up the link (not supported on Cisco MDS 91x4
switches) between F Ports on the NPIV core switch and the NP Ports on the Cisco NPV
switches.
Port channel configuration requires the following steps:
Step 1 Enable the F Port trunking and channeling protocol on the MDS 9000 Series core
switch. Enable NPIV on the MDS 9000 Series core switch.
Step 2 Create the port channel on the MDS 9000 Series core switch.
Step 3 Configure the port channel member interfaces on the MDS 9000 Series core switch.

3-74 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
On the Cisco NPV Edge Switch, do the following:
1. Enable the Cisco NPV feature.
switch(config)# NPV enable

2. Configure the port channel.

switch(config)# interface port-channel 1
switch(config-if)# switchport mode NP
switch(config-if)# switchport rate-mode shared

3. Configure the uplink interface.

switch(config)# interface fc 2/1-2
switch(config-if)# shut
switch(config-if)# switchport mode NP
switch(config-if)# switchport trunk mode off
switch(config-if)# switchport speed 4000
switch(config-if)# switchport rate-mode shared
switch(config-if)# channel-group 1
switch(config-if)# no shut

Use the following steps to create the port channel on the Cisco NPV switch:
Step 1 Configure the port channel member interfaces on the Cisco NPV switch.
Step 2 Set the administrative state of all the port channel member interfaces on both the
NPIV core switch and the NPV switch to on.
Step 3 When configuring a TF port channel, set the trunk mode to on and the rate mode to
dedicated.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-75

Summary
This topic summarizes the key points that were discussed in this lesson.

• Cisco NPV allows the edge switch to appear as a host to the upstream
core switch, which eliminates the need to configure a domain ID at the
edge.
• NPIV supports multiple FCIDs being assigned to a single N Port.
• F and TF interfaces can become members of a port channel.

3-76 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 5

Configuring VSANs
Overview
This lesson explains how virtual SANs (VSANs) can be used in a physical SAN environment to
reduce underutilized SAN components through SAN consolidation. This lesson also explains
the purpose of the default VSAN and how to configure VSANs.

Objectives
Upon completing this lesson, you will be able to describe the process of configuring VSANs.
This ability includes being able to meet these objectives:
 Describe how VSANs provide logical separation of fabric services in a unified physical
infrastructure
 Describe the process of creating VSANs and assigning interface members
 Describe the process for displaying the attributes of configured VSANs
 Present and describe the recommended practices for creating and managing VSANs
VSAN Overview
This topic describes how VSANs provide logical separation of fabric services in a unified
physical infrastructure.

• Virtual SANs address common Cisco MDS 9000

limitations of SAN deployments: Series Switches with VSAN Service

- VSANs are virtual Fibre Channel

fabrics.
- VSANs share MDS switch resources
while maintaining isolation between FC
FC

virtual fabrics. FC

- Traditional SAN islands are virtualized

onto a common SAN infrastructure:
Independent physical SAN islands are
• Higher utilization of switch
resources virtualized onto a common SAN infrastructure.

• Reduced space, power, and cooling

- Similar to VDC on Cisco Nexus 7000
Series switches
- Fabric services are per-VSAN.
- Statistics gathered are FC

per-VSAN.
FC

Today, many SAN environments consist of numerous islands of connectivity. Commonly

deployed SAN islands are physically isolated environments that consist of one or more
interconnected switches, where each island is typically dedicated to a single or to multiple
related applications.
A SAN island might be independently managed by a separate administration team, because
strict isolation from faults is achieved through physical network deployment separation.
However, because this physical isolation restricts access by other networks and users, the
sharing of critical storage assets and the economic savings of storage consolidation are limited.
Cisco developed the VSAN functionality feature to leverage the advantages of isolated SAN
fabrics with capabilities that address the limitations of isolated SAN islands.
VSANs provide a method for allocating ports within a physical fabric to create virtual fabrics.
Independent physical SAN islands are virtualized onto a common SAN infrastructure.
Separate fabric services are available on each VSAN, because the fabric is a virtual fabric, as
are statistics, which are gathered on a per-VSAN basis. Each CPU process is common to all
VSANs. For example, only one instance of the name server service runs on each switch.
However, the process uses separate databases for each VSAN.

3-78 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• There are up to 256 VSANs per physical switch:
- There are up to 239 switches per VSAN, and
typically less than 40. VSAN 10 VSAN 20 VSAN 30 VSAN 1
(Default)
- Each virtual switch has a unique domain ID.
• Fibre Channel frames are uniquely identified:
- The frames are tagged with a VSAN_ID header
on the ingress port.
- The VSAN_ID header is removed on egress port MDS 9000 Series
before delivery to a Fibre Channel end device. Switch Chassis
- Headers are maintained across TE ports
between switches.
- Each frame may be prioritized through the Fibre
Channel QoS feature.
• IVR allows storage devices to be shared
between VSANs.
• VSANs offer ease of management:
- It is easy to create VSANs and allocate Fibre
Channel ports to them. Fabric 10 Fabric 20 Fabric 30 Fabric 1
- A Fibre Channel port can only belong to a single Domain ID Domain ID Domain ID Domain ID
VSAN or fabric. 0x61 0x94 0x33 0x12
44 Ports 24 Ports 12 Ports 8 Ports
(Logically within the MDS 9000 SeriesSwitch Chassis)

VSANs help achieve traffic isolation in the fabric by adding control over each incoming and
outgoing port. There can be up to 4000 VSANs in the physical fabric and a maximum 239
switches per VSAN. This ability effectively helps with network scalability, because the fabric
is no longer limited by 239 domain IDs, and because the domain IDs can be reused within each
VSAN. These figures are theoretical maximums, but 80 VSANs per switch and 60 switches per
VSAN are the recommended limitations for Cisco Nexus Operating System (NX-OS) Software
Release 5.0.
The default VSAN number is VSAN 1. The maximum number of VSANs per switch is 256.
This number includes default VSAN 1 and isolated VSANs 4079 and 4094, so effectively an
administrator can create 253 VSANs.
To uniquely identify each frame in the fabric, the frame is labeled with a VSAN identification
(VSAN ID) tag on the ingress port. The VSAN ID is stripped away across expansion (E) ports
because those ports only support a single VSAN, however across trunking expansion (TE)
ports, the VSAN ID is still maintained. By carrying SAN and frame priority in the header, Fibre
Channel quality of service (QoS) can be properly applied. The VSAN ID is always stripped
away at the other edge of the fabric. If an E Port is capable of carrying multiple VSANs, that
port then becomes a TE Port.
VSANs also facilitate the reuse of address space by creating independent virtual SANs, thus
increasing the available number of addresses and improving switch granularity. Without a
VSAN, an administrator needs to purchase separate switches and links for separate SANs. The
system granularity is at the switch level, not at the port level.
VSANs are easy to manage. To move or change users, you need to change only the
configuration of the SAN, not its physical structure. To move devices between VSANs, you
simply change the configuration at the port level. No physical moves are required.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-79

• A total of 256 VSANs may be created per physical switch.
• VSAN 1 is the default VSAN.
- This VSAN is a reserved, predefined VSAN that is automatically configured by the
switch.
• All ports originally belong to VSAN 1.
• VSAN 1 is always present and cannot be deleted.
• VSANs 2 through 3839 are user VSANs
- A maximum of 253 other VSANs per physical switch can be created in this range.
• VSANs 3840 to 4078 are reserved.
• VSAN 4079 – EVFP isolated VSAN
• VSANs 4080 to 4093
- Used for vendor-specific VSANs
• VSAN 4094 is an isolated VSAN.
- This VSAN is a reserved, special VSAN that is automatically created by the switch.
• This VSAN is used to isolate ports whose port VSAN has been deleted.
• This VSAN is not propagated across switches.
• VSAN 4094 is always present and cannot be deleted.

VSAN 1 is the default VSAN that is automatically created by the MDS switch. VSAN 1 is
always present and cannot be deleted. All Fibre Channel ports originally belong to VSAN 1
until the ports are moved to a different VSAN.
The administrator can create VSANs 2 through 3839 to create logical SAN fabrics for different
applications or departments within the organization. Fibre Channel ports are then moved to
their appropriate VSAN so that the ports are logically isolated from ports that belong to
different VSANs.
If a VSAN is deleted, then the ports that belonged to that VSAN are moved to the isolated
VSAN. There the ports are logically isolated from one another and held until the administrator
moves the ports to another VSAN.

3-80 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Each VSAN has its own Domain 100
Domain 200
Domain 105
Domain 223
Domain 126 Domain 153
Domain 173
principal switch and domain ID
allocation policy.
• Principal switches for different
VSANs can reside on different
physical switches. Domain 112
Domain 110
Domain 171
• Each switch has a separate Domain 153

domain ID for each active

VSAN.
• The same domain IDs can exist Domain 156 Domain 113
Domain 180
Domain 102
in different VSANs.
• The domain ID and FCID
allocation policy is static or
dynamic.
• All ports are originally in VSAN Domain 104 Domain 170 Domain 201
Domain 204 Domain 157 Domain 215 Domain 162
1.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-7

Each switch maintains a separate domain ID for each active VSAN and each VSAN has its own
principal switch and domain ID allocation policy, either static or dynamic. Static is the
preferred policy. Principal switches for different VSANs do not necessarily need to reside on
the same physical switch. The administrator can nominate which switch will be the principal
switch or have that choice automatically elected.
Each VSAN can also have a separate Fibre Channel ID (FCID) allocation policy, either static
or dynamic, but static is preferred.
All ports are originally configured in default VSAN 1 and then moved by the administrator to
their appropriate VSAN.
As shown in the figure, each switch that has end ports in a particular VSAN has a domain ID
that is assigned to that particular VSAN. Core switches that trunk these VSANs also have
assigned domain IDs in these VSANs, so that frames can be routed between switches.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-81

• The VSAN feature consists of FC

three primary functions: Fibre

- Hardware-based isolation of traffic Channel
Services for
- No special drivers or configuration Blue VSAN
VSAN tagged header is
required for end nodes removed at egress port
- Traffic is tagged at a Fibre Channel Fibre
ingress port and carried across Cisco MDS 9000 Series Channel
EISL with VSAN Services Services for
Red VSAN
• Independent Fibre Channel
fabric services for each VSAN: Trunking E Port
EISL trunk carries tagged (TE Port)
- Name server traffic from multiple VSANs.
- Zone server Trunking E Port
- Management server (TE Port)

- Principal switch selection VSAN tagged header

(VSAN_ID) is added at Fibre
- FSPF routing protocol ingress point, indicating Channel
membership of VSAN. Services for
- Services managed and configured Blue VSAN
independently No special support is
- Faults in one VSAN do not impact required by end nodes. Fibre
another VSAN Channel
Services for
Red VSAN

The VSAN feature provides two primary functions, which are hardware-based isolation of
traffic and independent fabric services for each VSAN.
Hardware-based isolation of tagged traffic belonging to different VSANs requires no special
drivers or configuration at the end nodes, such as hosts, disks, and so on. Traffic is tagged at the
Fibre Channel ingress port (Fx Port) and carried across Enhanced Inter-Switch Links (EISLs)
between Cisco MDS 9000 Series switches. Because VSANs use explicit frame tagging, the
VSANs can be extended over the metropolitan-area network (MAN) or WAN if trunking is
supported. The Cisco MDS 9000 Series IP Storage Services Module can add tags to be
transported in Fibre Channel over IP (FCIP) for greater distances between data centers.
Fibre Channel, and therefore VSANs, can easily be carried across dark fiber. However, VSANs
add 8 bytes of header, which might be a concern for existing channel extenders. The channel
extenders might consider the frame invalid and drop the frame.
Dense wavelength-division multiplexing (DWDM) switches may also count frames as invalid,
but might pass the frames anyway. Qualification is still ongoing within Cisco to validate
various extension methods.

3-82 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Domain 100 Domain 105 Domain 126
• IP connectivity between Domain 200 Domain 223
Domain 1 Domain 2 Domain 3
switches over Fibre Channel: 10.20.1.6/24 10.20.1.5/24 10.20.1.4/24

- RFC 2625 (IP over Fibre Management

Station on
Channel) Ethernet
• Configured on a per-VSAN Network

basis.
• IP address is mapped to each Domain 110
virtual switch within the VSAN. Domain 153
Domain 5
• Switch selectively routes 10.20.1.1/24

between subnets (VSANs) and

OOB management.
• An in-band management VSAN
configuration is recommended
to back up the OOB
management access.
Cisco Prime DCNM Configuration Screen
for IP-VSAN Interfaces
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-9

In addition to an out-of-band (OOB) management network, an in-band IP overlay management

network can be configured on a per-VSAN basis. An IP subnet can be mapped to each VSAN,
which allows IP connectivity between switches over Fibre Channel for management purposes,
using RFC 2625 (IP over Fibre Channel [IPFC]).
Each switch can selectively route between subnets (VSANs) and the OOB Ethernet network.
This setup is recommended to configure a management VSAN as a backup management
network in case of failure on OOB management access.
The figure shows a Cisco Prime Data Center Network Manager (DCNM) configuration screen
for IP-VSAN interfaces. VSAN 20 (the green VSAN) with the IP address and mask of
10.20.1.1/24 has been configured as the management VSAN.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-83

• Hierarchical relationship
- Physical ports are assigned to VSANs. Relationship of VSANs to Zones
- Independent zones are configured per
VSAN. Physical Topology
• VSANs VSAN 2
Host1
- Virtualize the physical infrastructure. ZoneA
- Logical isolation of Fibre Channel fabric Disk2
Tape1
services.
Host2 ZoneC
- Change only when VSANs are added or
ports are moved to a different VSAN. ZoneB Disk1
Disk4
• Zones
- Subdivide the VSAN infrastructure. VSAN 7
- Maintain connectivity between devices. Host4
- Maintain isolation between hosts.
ZoneD
- Can change frequently (for example, Host3
backup). ZoneE Disk5
Disk3
• Ports are added and removed
nondisruptively to and from VSANs.

VSANs and fabric zoning are complementary. For the hierarchical relationship, you first assign
physical ports to VSANs, and then you configure independent zones for each VSAN. VSANs
divide the physical infrastructure, provide traffic statistics, and are changed only when ports are
needed for each VSAN fabric. VSANs also allow ports to be added or removed
nondisruptively.
Zones divide the VSAN infrastructure while providing added security and allowing the sharing
of device ports. Zones can be changed frequently. An example situation is backup applications
where a host might not communicate to a server during the day but needs to connect to the
server at night for backup storage.
Zones are similar to access control lists (ACLs) in that zones restrict connectivity between
devices and provide security through segmentation. Zoning is covered in more detail elsewhere
in the course.

3-84 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Without DPVM • With DPVM
- VSAN membership is based - VSAN membership is based on the
on the physical switch port. pWWN of the server or storage
- Configuration is tied to a single device.
switch. - Reconfiguration is fabricwide and
- Reconfiguration is required when a distributed using Cisco Fabric
host or storage device moves to Services.
another switch. - No reconfiguration is required when a
- The switch port belongs to a VSAN. host or storage device is moved to
another switch.
- The device port belongs to a VSAN.
SW1 SW2 SW1 SW2

pWWN1 pWWN1
Move can require Move without
FC reconfiguration FC FC reconfiguration FC
HBA
on SW2 HBA
HBA HBA

VSAN membership can be defined, either based on the physical switch port, or based on the
port world wide name (pWWN) of hosts and storage devices.
With Dynamic Port VSAN membership (DPVM), hosts and targets can be moved from one
port to any other switch port anywhere in the MDS fabric. This move does not require manual
reconfiguration of the port VSANs.
Once DPVM is enabled on the switches, the DPVM database keeps a record of the pWWN of
the device and associated VSAN membership. If the device moves to another interface, the
DPVM database record will be used to determine the VSAN membership of the device.
As a result, the device can be moved to other ports belonging to other VSANs. The device will
remain in its original VSAN due to the pWWN mapping.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-85

Creating VSANs
This topic describes the process of creating VSANs and assigning interface members.

• Create a VSAN and specify its name:

switch(config)# vsan database
switch(config-vsan-db)# vsan 2
switch(config-vsan-db)# vsan 2 name HR_VSAN
switch(config-vsan-db)# vsan 3
switch(config-vsan-db)# vsan 3 name DEV_VSAN

• Assign VSAN interface membership:

switch(config-vsan-db)# vsan 2 interface fc1/10-15, fc2/3
switch(config-vsan-db)# vsan 3 interface iscsi2/1

The VSAN first needs to be created before you can configure any application-specific
parameters for a VSAN. To create and configure VSANs, follow these steps:
Step 1 Enter the configuration mode with the config command.
Step 2 Configure the VSAN database with the vsan database command.
Step 3 Create a VSAN number with the vsan x command, where x is a VSAN number that
is not currently assigned.
Step 4 Create a name for the new VSAN with the vsan x name abc command, where abc is
the actual VSAN name you will use.
The VSAN name is a text string that identifies the VSAN for management purposes. The name
can be from 1 to 32 characters long, but the name must be unique across all VSANs. By
default, the VSAN name is a concatenation of VSAN and a four-digit string representing the
VSAN ID. For example, the default name for VSAN 3 is VSAN0003. The response to the vsan
x name command should be updated as vsan x.
To assign VSAN membership, follow these steps:
Step 1 Enter the configuration mode with the config command.
Step 2 Configure the VSAN database with the vsan database command.
Step 3 Create a VSAN number with the vsan x command, where x is a VSAN number that
is not currently assigned.
Step 4 Assign a specific Fibre Channel interface for membership to the specified VSAN.
Use the vsan x interface fcy/z command, where y/z is the slot and port number of
the associated Fibre Channel interface.

3-86 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
While VLAN assignment is performed with an interface command, VSAN assignment is
performed using the VSAN database.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-87

• Delete a VSAN:
switch(config)# vsan database
switch(config-vsan-db)# no vsan 2

• Suspend or activate a VSAN:

switch(config-vsan-db)# vsan 2 suspend
switch(config-vsan-db)# no vsan 2 suspend

• Configure VSAN load-balancing parameters:

switch(config-vsan-db)# vsan 2 loadbalancing src-dst-id
switch(config-vsan-db)# vsan 3 loadbalancing src-dst-ox-id

When an active VSAN is deleted, all of its attributes are removed from the running
configuration. System software maintains VSAN-related information. VSAN Manager
maintains VSAN attributes and port membership details.
This feature is affected when you delete a VSAN from the configuration. When a VSAN is
deleted, all the ports in that VSAN are made inactive, and the ports are moved to the isolated
VSAN. If the same VSAN is re-created, the ports do not automatically get assigned to that
VSAN.
You must explicitly reconfigure the port VSAN membership. VSAN-based run time (name
server), zoning, and configuration (static routes) information is removed when the VSAN is
deleted. Configured VSAN interface information is removed when the VSAN is deleted.
To delete a VSAN and its various attributes, follow these steps:
Step 1 Enter the configuration mode with the config command.
Step 2 Configure the VSAN database with the vsan database command.
Step 3 Go into the VSAN configuration mode by selecting an existing VSAN number with
the vsan x command, where x is a VSAN number that exists.
Step 4 Delete the desired VSAN by using the no vsan x command.
To exit from the configuration mode and return to the executive mode, use the end command.

Configuring VSAN Load-Balancing Parameters and the VSAN State

To continue configuring a VSAN by setting the load-balancing parameters and VSAN state,
follow these steps:
Step 1 Enable load balancing with a path selection process other than a default to the
selected VSAN number (x) by using the vsan x loadbalancing src-dst-id command.
This command sets the path selection to source ID (SID) and destination ID (DID).
You can negate the previous command and revert to the default path selection for
load balancing with the no vsan x loadbalancing src-dst-id command.

3-88 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Step 2 You can change the path selection setting directly to the default with the vsan x
loadbalancing src-dst-ox-id command. The default path selection is SID/
DID/originator exchange ID (OXID).
Step 3 To suspend the administrative state of a VSAN, use the vsan x suspend command.

Note The administrative state of a VSAN can be either active (default) or suspended. The active
state of a VSAN indicates that the VSAN is configured and enabled. By enabling a VSAN,
you activate the services for that VSAN. The suspended state of a VSAN indicates that the
VSAN is configured but not enabled. If a port is configured in this VSAN, the port is disabled.
You can use the suspended state to deactivate a VSAN without losing the configuration of
the VSAN, but note that all ports in a suspended VSAN are disabled. By suspending a
VSAN, you can preconfigure all the VSAN parameters for the entire fabric and activate the
VSAN when ready.

Step 4 You can negate the previous command and reactivate a VSAN with the no vsan x
suspend command.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-89

• Do the following to create a VSAN on multiple switches:

Use the “Create VSAN” icon on the toolbar.

In Cisco Prime DCNM-SAN Client, select the Create VSAN icon from the toolbar. The Create
VSAN dialog box allows you to configure the following:
 Select one or more switches where the VSAN will be created.
 Specify the VSAN ID. (The valid range is from 2 to 4093.)
 Select the load-balancing scheme.
 Select the interop mode for the VSAN. Cisco MDS 9000 Series switches support several
interop modes that allow MDS switches to connect to other third-party switches
seamlessly.
 Specify the administrative state (active or suspended).
 Choose whether to specify static domain IDs for this VSAN (optional).
 Choose if this VSAN will be exclusively used for fiber connectivity (FICON) protocol.

3-90 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Do the following to achieve port VSAN assignment:

In the Physical Attributes pane, select Interfaces > FC Physical.

You can also assign VSANs in the Physical Attributes pane by selecting the FC Physical
folder.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-91

Displaying VSAN Information
This topic describes how to display VSAN information.

• Verify VSAN configuration.

switch# show vsan
vsan 1 information
name:VSAN0001 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up

vsan 10 information
name:VSAN0010 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up

vsan 20 information
name:VSAN0020 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:up

vsan 4079:evfp_isolated_vsan

vsan 4094:isolated_vsan
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-18

To verify the current VSAN configuration and status, use the show vsan command from
executive mode. Use this command to display the following:
 VSANs created
 VSAN name
 Administrative state (active or suspended)
 Interoperability setting (default, 1, 2, 3)
 Load-balancing scheme (SID/DID/OXID with a default of SID/DID)
 Operational state (up or down)
To report the status for a specific VSAN, use the show vsan x command, where x is the number
of the VSAN.

3-92 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Verify VSAN membership.
switch# show vsan membership
vsan 1 interfaces:
fc1/1 fc1/2 fc1/3 fc1/4
fc1/7 fc1/8 fc1/9 fc1/10
fc1/11 fc1/12 fc1/13 fc1/14
fc1/15 fc1/16 fc1/17 fc1/18

vsan 10 interfaces:
fc1/5 fc1/6

vsan 20 interfaces:

vsan 4079(evfp_isolated_vsan) interfaces:

vsan 4094(isolated_vsan) interfaces:

To verify port VSAN membership, use the show vsan membership command from executive
mode. The report displays port VSAN assignments, including the isolated VSAN (4094).
To report the membership for a specific VSAN, add the VSAN number to the show vsan x
membership command.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-93

VSAN Recommended Practices
This topic lists basic VSAN recommended practices.

• Avoid using VSAN 1 for production network traffic.

• Create VSANs to carry production traffic.
• Use VSANs to isolate devices:
- By department
- By application
- For test and development environments
• Continue to use zones inside of each VSAN.
• Use IVR to share devices (for example, tape libraries).

VSANs provide a method of isolating devices that are physically connected to the same storage
network, but are logically considered to be part of different SAN fabrics and need to be
logically isolated from one another. VSANs provide practical isolation of devices that are
physically connected to the same fabric. VSANs reduce the size of a Fibre Channel distributed
database and enable more scalable and secure fabrics.
Follow these guidelines when implementing VSANs:
 Avoid using VSAN 1 (the default VSAN) for production network traffic.
 Create at least one VSAN to carry your network traffic.
 Isolate devices in VSANs whenever practical. Isolation by department and by application
are two common practices. Isolating test and development environments from production
environments is also recommended.
 Continue to use zones inside of each VSAN.
 User Inter-VSAN Routing (IVR) to share devices across VSANs. An example would be
sharing a tape library across backup media servers that are each located in different
VSANs. IVR is discussed in detail elsewhere in the course.

3-94 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• VSANs provide a method for allocating ports within a physical fabric to

create virtual fabrics. Independent physical SAN islands are virtualized
onto a common SAN infrastructure.
• You configure VSANs by setting the following attributes: VSAN_ID,
VSAN name, load balancing, VSAN state, and VSAN membership.
• Display VSAN information using the show vsan and show vsan
membership commands.
• VSANs 1, 4079, and 4094 are the default and isolated VSANs. These
VSANs are configured on the switch by default. When a VSAN is
deleted, member ports are moved to VSAN 4094.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-95

3-96 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 6

Managing Domains
Overview
This lesson explains the purpose of Fibre Channel domains, their behavior in a virtual SAN
(VSAN) environment, and how to prevent fabric isolation. The lesson describes how the Cisco
MDS 9000 Series Multilayer Switches allocate Fibre Channel IDs (FCIDs) and describes how
to configure static domain IDs and FCIDs.

Objectives
After completing this lesson, you will be able to explain the purpose of Fibre Channel domains,
their behavior in a VSAN environment, and how to prevent fabric isolation. This ability
includes being able to meet these objectives:
 Describe how to configure domain IDs within a VSAN
 Configure the principal switch priority
 Describe the steps to manually configure domain ID settings
 Describe how to view configured and run-time fcdomain information from the CLI and
Cisco Device Manager
 List and describe the configuration settings that determine if switches achieve adjacency or
become isolated during a fabric merge event
 Describe the effects of configuring the auto-reconfigure, rcf-reject, and fcdomain
command options for fabric merge
Domain Configuration Process
During fabric reconfiguration, a principal switch is nominated to assign a unique Fibre Channel
domain ID to every switch in the fabric or VSAN. The domain ID is used when routing frames
from one switch domain to another within the fabric. This topic explains the domain
configuration process and principal switch selection.

• Fabric reconfiguration
- Starts with a disruptive restart or nondisruptive restart
- Invokes a resynchronization of all switches in the fabric VSAN
- Ensures that switches simultaneously select a new principal switch
• Principal switch selection
- Guarantees the selection of a unique principal switch across the fabric VSAN
- The principal switch manages the assignment of domain IDs to the other
switches in the fabric VSAN
• Domain ID distribution
- Guarantees that each switch in the fabric VSAN obtains a unique domain ID
• FCID allocation
- Guarantees a unique FCID assignment to each device that is attached to the
corresponding switch in the fabric VSAN

The Fibre Channel domain (fcdomain) feature performs principal switch selection, domain ID
distribution, FCID allocation, and fabric reconfiguration functions as described in the FC-SW-2
standards. The domains are configured on a per-VSAN basis, and if you do not configure a
domain ID, the local switches use a random ID.
To successfully configure domain parameters and prevent fabric segmentation, you need to
understand the anticipated behavior of the fcdomain feature phases. There are four phases that
are associated with the fcdomain feature process:
 Fabric reconfiguration: This phase guarantees a resynchronization of all switches in the
fabric to ensure that the switches simultaneously restart a new principal switch select phase.
 Principal switch selection: This phase guarantees the selection of a unique principal switch
across the fabric. The role of the principal switch is to manage domain ID assignments
using the locally configured allowed domain list.
 Domain ID distribution: This phase guarantees that each switch in the fabric obtains a
unique domain ID.
 FCID allocation: This phase guarantees a unique FCID assignment to each device that is
attached to the corresponding switch in the fabric.

Domain IDs must be unique across interconnected VSANs. To ensure unique domain IDs for
each switch across interconnected VSANs, consider minimizing the number of switches that
require a domain ID assignment for minimum traffic disruption. This process can be achieved

3-98 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
by using the Cisco N-Port Virtualizer (NPV) feature that puts the switches in a mode of
operation that does not require domain IDs. NPV is mentioned in detail elsewhere in the course.

• A new principal switch is elected, based upon these criteria:

- Highest runtime priority
• The lowest number is the highest priority. The default is 128.
- Lowest switch WWN

Principal
Local WWN VSAN 10
20:01:ab:ba:cd:dc:f4:00 Runtime Fabric Name
Configured Priority = 128 20:01:ab:ba:cd:dc:f4:00
• Highest Priority or Runtime Priority = 2
• Lowest Local WWN
Lowest number
is highest priority

Subordinate Local WWN Subordinate Local WWN

20:03:ab:ba:cd:dc:f4:00 20:02:ab:ba:cd:dc:f4:00
Configured Priority = 128 Configured Priority = 128
Runtime Priority = 128 Runtime Priority = 128

The principal switch selection phase guarantees the selection of a unique principal switch
across the fabric. The principal switch allocates domain IDs to subordinate switches.
The principal switch is elected, based upon these criteria:
 Highest run-time priority: This number is in the range of 1 to 254, but the default is 128.
 Lowest switch world wide name (WWN): If all switches in the fabric have the same run-
time priority, then the principal switch becomes the switch with the lowest WWN.

Note the difference between the configured priority and the run-time priority. When a fabric
reconfiguration takes place, the configured priority becomes the switch run-time priority. If the
principal switch is elected because the switch has the lowest WWN, the switch changes its run-
time priority to 2. This change ensures that the switch has a higher priority than the others.
When the fcdomain feature is disabled, the run-time fabric name is the same as the configured
fabric name.
When the fcdomain feature is enabled, the run-time fabric name is the same as the WWN of the
principal switch.
In this example, the configured fabric name is 20:01:ab:ba:cd:dc:f4:00.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-99

Switch 1 Principal Switch 2
Empty domain ID Priority 128 Priority 128
in switch 2. Domain ID 1 Domain ID 0
WWN 20:01:ab:ba:cd:dc:f4:00 WWN 20:02:ab:ba:cd:dc:f4:00

Switch 1 Switch 2 Principal

Domain ID on both.
Priority 128 Priority 99
Switch 2 has
Domain ID 1 Domain ID 2
higher priority.
WWN 20:01:ab:ba:cd:dc:f4:00 WWN 20:02:ab:ba:cd:dc:f4:00

Switch 1 Principal Switch 2

Priorities equal.
Switch 1 has Priority 128 Priority 128
lower WWN. Domain ID 1 Domain ID 2
WWN 20:01:ab:ba:cd:dc:f4:00 WWN 20:02:ab:ba:cd:dc:f4:00

The principal switch is used to issue domain IDs when a new switch is added to an existing
fabric. When two fabrics merge, the principal switch selection process determines which one of
the existing switches becomes the principal switch.
These rules characterize the election of the new principal switch:
 A switch that has already been allocated a domain ID list has priority over a switch that has
an empty domain ID list. The principal switch is the principal switch of the first fabric.
When working with a single-switch fabric, the switch does not contain a domain ID list.
 If both fabrics have a domain ID list, the priority between the two principal switches is
determined by configured switch priority. This parameter can be set by the user—the lower
the value, the higher the priority. However, when connecting a single-switch fabric to a
multiswitch fabric, the multiswitch fabric always retains its principal switch, regardless of
the principal switch priority setting on the single-switch fabric.

If the principal switch cannot be determined by either of the two previous criteria, then the
WWNs of the two switches determine the principal switch. The lower value has the higher
priority. Unfortunately, this switch is usually the oldest and slowest switch in the fabric.
Because of this fact, it is advisable to manually choose the principal switch by configuring a
lower run-time priority than the rest.

3-100 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• This behavior is default behavior unless static domain IDs are
configured.
• There is no guarantee that a domain ID will not change upon
reassignment.
VSAN 10
Principal Requested Domain ID Available Subordinate

Preferred domain ID = 0 1 Preferred domain ID = 7

Runtime domain ID = 2 2 Runtime domain ID = 7

Principal Subordinate
Requested Domain ID Unavailable

Preferred domain ID = 0 3 Preferred domain ID = 7

Runtime domain ID = 7 4 Runtime domain ID = 9

The configured domain ID can be preferred or static. By default, the configured domain ID is 0
(zero) and the configured type is preferred. If you do not configure a domain ID, the local
switch sends a random ID in its request. All switches within a fabric or VSAN should be
configured with the same domain ID type, preferred or static.
When a subordinate switch requests a domain, the local switch sends a configured domain ID
request to the principal switch. The principal switch then assigns the requested domain ID, if
that ID is available. Otherwise, the principal switch assigns another available domain ID.
In this example, domain ID distribution is exemplified in the figure as follows:
1. The subordinate switch sends a configured preferred domain ID request (7) to the principal
switch.
2. The requested domain ID (7) is assigned by the principal switch if that ID is available.
3. The subordinate switch sends a configured preferred domain ID request (7) to the principal
switch.

4. This time, the requested domain ID (7) is unavailable because that ID is already assigned to
the principal switch, so another available domain ID (9) is assigned from the pool.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-101

VSAN 10
Principal Subordinate
Requested Domain ID Available

Static domain ID = 2 1 Static domain ID = 12

Runtime domain ID = 2 2 Runtime domain ID = 12

Subordinate
switch# show fcdomain
allowed vsan 10 Requested Domain ID Conflict
Assigned or unallowed
domain IDs: 2,12. [User] 3 Static domain ID = 12
configured allowed domain Runtime domain ID = 12
IDs: 1-239 4

Principal 5 Subordinate
6
Logical ISL Isolated for that VSAN
Static domain ID = 2 Static domain ID = 12
Runtime domain ID = 2 Runtime domain ID = 12
Runtime priority = 2 7 Runtime priority = 128

1. When a static domain ID is configured, the subordinate switch will send a static domain ID
(12) request to the principal switch.
2. If that requested domain ID (12) is available, the ID will be assigned to the subordinate
switch and become its run-time domain ID (12).
3. If the subordinate switch sends a request for a domain ID (12) that is unavailable, because
the ID is already assigned, then the principal switch will still try to send an unassigned
domain ID to the subordinate switch.
4. When the subordinate switch receives this allocated domain ID, because the ID does not
match the locally configured domain ID, the subordinate switch will discard that ID.
5. The subordinate switch retains its static domain ID (12), which now becomes the run-time
domain ID (12).
6. However, to avoid a domain ID conflict, the logical EISL is isolated for that VSAN. The
VSAN will become segmented, and all local interfaces on the subordinate switch will
become isolated. Other VSANs using this EISL will not be isolated.
7. The subordinate switch will then assign itself the configured domain ID, which becomes
the run-time domain ID within its own segmented fabric.

3-102 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
VSAN 4
Principal Subordinate
Requested Domain ID Available

Static domain ID = 2 1 Static domain ID = 12

Runtime domain ID = 2 2 Runtime domain ID = 12

Subordinate
switch# show fcdomain
allowed vsan 4 Requested Domain ID Conflict
Assigned or unallowed
domain IDs: 2,12[User] 3 Static domain ID = 54
configured allowed Runtime domain ID = 54
domain IDs: 1-50 4

Principal 5 Subordinate
6
Logical ISL Isolated for that VSAN
Static domain ID = 2 Static domain ID = 54
Runtime domain ID = 2 Runtime domain ID = 54
Runtime priority = 2 7 Runtime priority = 128

The behavior for a subordinate switch changes based on the allowed domain ID lists,
configured domain ID, and domain ID that the principal switch has assigned to the requesting
switch:
 When the assigned and requested domain IDs are the same, and the domain ID is within the
allowed list, then the following occurs:
— The preferred and static options are not relevant.
— The assigned domain ID becomes the run-time domain ID.
 When the assigned and requested domain IDs are different, and the configured type is
static, then the following occurs:
— The assigned domain ID is discarded.
— All local interfaces are isolated.
— The local switch assigns itself the configured domain ID, which becomes the run-
time domain ID.
By default, the valid range for an assigned domain ID list is from 1 to 239. You can specify a
list of ranges to be in the allowed domain ID list and separate each range with a comma. The
principal switch assigns domain IDs that are available in the locally configured allowed domain
list.
If you configure an allowed list on one switch in the fabric, you should configure the same list
on all other switches in the fabric to ensure consistency.
An allowed domain ID list must satisfy the following conditions:
 If this switch is a principal switch, all currently assigned domain IDs must be in the
allowed list.
 If this switch is a subordinate switch, the local run-time domain ID must be in the allowed
list.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-103

 The locally configured domain ID of the switch must be in the allowed list.
 The intersection of the assigned domain IDs with other already configured domain ID lists
must not be empty.

In the example that is shown in the figure, domain ID distribution is exemplified as follows:
1. The subordinate switch sends a configured static domain ID request (12) to the principal
switch.
2. The requested domain ID (12) is available and is assigned, becoming the run-time domain
ID.
3. The subordinate switch sends a configured static domain ID request (54) to the principal
switch.
4. The requested domain ID (54) is unavailable because the ID is outside the allowed domain
address range. Another available domain ID is assigned from the pool.
5. The subordinate switch discards the assigned domain ID.
6. The VSAN is segmented and all local interfaces on the subordinate switch become isolated.
7. The subordinate switch assigns itself the configured static domain ID, which becomes the
run-time domain ID (54).

3-104 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
If domain IDs are not static, then FCIDs may change.
HOST 1
Principal Subordinate FC FCID
HBA 07.00.00
FC HOST 2 Domain ID and FCID Values
HBA
FCID Before Fabric Reconfiguration
07.01.00
Switch MDS-2
Configured Domain ID = 0 Preferred
Run-Time Domain ID = 7

Disruptive Restart - RCF

1 Resynchronization 2 Domain ID Distribution 3 FCID Allocation

Principal Subordinate HOST 1

FC FCID
03.00.00
HBA
FC HOST 2 Domain ID and FCID Values
HBA
FCID After Fabric Reconfiguration
03.01.00
Switch MDS-2
Configured Domain ID = 3 Static
Run-Time Domain ID = 3

Domain Restart
Fibre Channel domains can be started disruptively or nondisruptively:
 If you perform a disruptive restart, Reconfigure Fabric (RCF) frames are sent to other
switches in the fabric.
 If you perform a nondisruptive restart, Build Fabric (BF) frames are sent to other switches
in the fabric.

A static domain is specifically configured by the user and can be different from the run-time
domain. If the domain IDs are different, the run-time domain ID changes to take on the static
domain ID after the next restart.
If a VSAN is in interoperability mode, you cannot restart the fcdomain feature for that VSAN
disruptively.
You can apply most of the configurations to their corresponding run-time values. Each of the
following sections provides further details on how the fcdomain parameters are applied to the
run-time values.
The fcdomain restart command applies your changes to the run-time settings. Use the restart
disruptive option to apply most of the configurations to their corresponding run-time values.

Resolving Domain ID Overlaps

To resolve a domain ID overlap, note these considerations:
 Manually assign domain IDs.
 Disruptive restart is required when assigning preferred IDs and usually when assigning
static IDs.
 Nondisruptive restart is acceptable only when changing a preferred ID into a static ID
without actually changing the ID.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-105

Fabric reconfiguration is described in the figure, as follows:
 Resynchronization: All switches in the fabric simultaneously restart principal switch
selection.
 Domain ID distribution: Each switch receives a unique domain ID.
 FCID allocation: A unique FCID is assigned to each device that is attached to the switch.

3-106 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configuring the Principal Switch Priority
This topic describes how to configure the principal switch priority.

• The default configured priority is 128.

- The valid range is 1 to 254.:
• Priority 1 is the highest priority.
• 255 is accepted from other switches only and cannot be applied locally.
- Changes to priority are applied to run time when the fcdomain is restarted.

switch# conf

switch(config)# fcdomain priority 25 vsan 99

switch(config)# fcdomain restart disruptive vsan 99

Each VSAN has a principal switch. Configuring the principal switch is a per-VSAN function.
By default, the configured switch priority is 128. The valid range to set the priority is between 1
and 254. Priority 1 has the highest priority. Value 255 is accepted from other switches, but that
value cannot be locally configured.
Any new switch cannot automatically become the principal switch when the switch joins a
stable fabric. During the principal switch selection phase, the switch with the highest priority
becomes the principal switch. If two switches have the same configured priority, the switch
with the lower WWN becomes the principal switch.
The priority configuration is applied to run time when the fcdomain feature is restarted. This
configuration is applicable to both disruptive and nondisruptive restarts.
To configure the priority for the principal switch, follow these steps:
Step 1 Enter configuration mode.
switch # config
Step 2 Configure a priority of 25 for the local switch in VSAN 99.
switch(config)# fcdomain priority 25 VSAN 99

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-107

Configuring the Domain ID
This topic explains how to configure domain IDs.

• Static request:
switch(config)# fcdomain domain 3 static vsan 8

Recommended practice:
Static domain IDs should be used for most SAN environments.

• Preferred request:
switch(config)# fcdomain domain 5 preferred vsan 14

• Domain allowed lists:

switch(config)# fcdomain allowed 25-50 vsan 14

• Changes must be applied:

switch(config)# fcdomain restart disruptive vsan 8

Cisco MDS 9000 Series switches can be configured to make either static or preferred domain
ID requests:
 Static requests: These requests are the recommended best practice. These requests specify
the only domain ID that the requestor (subordinate switch) accepts from the principal
switch. If the principal switch is unable to satisfy the request, the switch passes a different
(available) domain ID back to the requestor. The requestor discards this domain ID, assigns
itself the statically configured domain ID, and isolates itself from the attached fabric.
 Preferred requests: These requests specify the domain ID that the requestor prefers, but
the requestor accepts any domain ID that is provided by the principal switch.
 With preferred requests, a domain-allowed list can be configured to restrict the range of
domain IDs that can be returned by the principal switch.

You must issue the fcdomain restart disruptive vsan vsan command to apply the configured
domain changes to the run-time environment.
Like most CLI commands, domain ID commands can be negated by preceding the command
with the no string, such as no fcdomain vsan vsan.

3-108 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Contiguous domain allocation:
- Disabled by default
- If you enable contiguous allocation, the following occurs:
• Takes immediate effect at run time
• Does not require fcdomain restart

switch(config)# fcdomain contiguous-allocation vsan 25-105

By default, contiguous domain assignment is disabled. When a subordinate switch asks the
principal switch for two or more domains, and the domains are not contiguous, the following
cases apply:
 Case 1: If the contiguous domain assignment is enabled on the principal switch, the
principal switch locates contiguous domains, and assigns the domains to the subordinate
switches. If contiguous domains are not available, the Cisco Nexus Operating System (NX-
OS) Software rejects this request.
 Case 2: If the contiguous domain assignment is disabled on the principal switch, the
principal switch assigns the available domains to the subordinate switch.

To enable contiguous domains in a specific VSAN (or a range of VSANs), issue the following
commands:
switch# config
switch(config)# fcdomain contiguous-allocation vsan 25-105

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-109

• The fcdomain feature can be restarted disruptively or nondisruptively:
- Disruptive restart is required to apply most configuration changes to runtime.
• Sends reconfigure fabric (RCF) frames to the other switches in the fabric.
• Required when assigning preferred IDs and usually when assigning static
IDs.
- Nondisruptive restart
• Sends build fabric (BF) frames to the other switches in the fabric.
• Acceptable only when changing a preferred ID into a static ID without
actually changing the ID or changing the principal switch.

switch(config)# fcdomain domain 3 preferred vsan 1

switch(config)# fcdomain restart disruptive vsan 1

switch(config)# fcdomain domain 3 static vsan 10

switch(config)# fcdomain restart disruptive vsan 10

Although the static option can be applied to the run-time configuration after a disruptive or
nondisruptive restart, the preferred option is applied to the run-time configuration only after a
disruptive restart.

3-110 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Choose VSAN > Domain Manager > Configuration.

Change domain configured type to Static.

Persistent FCIDs do not change.

Domain information can be configured from the Cisco Prime Data Center Network Manager
for SAN Client (DCNM-SAN Client). Configuration options are per-VSAN and include the
following settings:
 Enable: Enable or disable the fcdomain feature.
 ConfigDomain Domainid: Configure the domain ID.
 ConfigDomain Type: Set the domain ID allocation type to static or preferred.
 Priority: Configure the switch priority.
 Contiguous allocation: Select or remove the contiguous domain ID allocation feature.
 Autoreconfigure: Select or remove the autoreconfigure feature.
 FCID persistency: Configure the persistent ID behavior for fcdomain.
 FCID purge: Select to purge persistent FCID entries.
 Restart: Select disruptive or nondisruptive restarting of fcdomain.

Clicking Apply initiates the selected restart and applies configured settings to run time.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-111

Displaying the Domain Database
This topic describes the commands for showing information from the domain database. You
will learn to view configured and run-time fcdomain information from the CLI and Cisco
Device Manager.

switch# show fcdomain

VSAN 1
The local switch is a Subordinated Switch.

Local switch run time information:

State: Stable
Local switch WWN: 20:01:00:05:30:00:13:9f
Running fabric name: 20:01:00:05:30:00:13:9e
Running priority: 128
Current domain ID: 0x4a(74)

Local switch configuration information:

State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Configured priority: 128
Configured domain ID: 0x00(0) (preferred)

Principal switch run time information:

Running priority: 2

Issue the show fcdomain command with no arguments to display all VSANs. The VSANs
should be active or an error is generated.

3-112 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Another switch is the principal switch:
switch1# show fcdomain domain-list vsan 1
Number of domains: 2
Domain ID WWN
--------- --------------------------------
0x4a(74) 20:01:00:05:30:00:13:9f [Local]
0x4b(75) 20:01:00:05:30:00:13:9e [Principal]

The local switch is also the principal switch:

switch2# show fcdomain domain-list vsan 1
Number of domains: 1
Domain ID WWN
--------- -----------------------
0x4a(75) 20:01:00:05:30:00:13:9e [Local] [Principal]

To verify that each switch is able to see the other switches, use the command show fcdomain
domain-list vsan vsan. If the command does not include a specific VSAN number, the display
lists the output for all VSANs.
The output of the command lists the set of domain IDs and associated WWNs for each switch
within a VSAN. This list provides the WWN of the switches that own each domain ID. The list
also indicates if a switch is the principal switch of the switches in the fabric or VSAN to which
the switch belongs.

Two Switches in VSAN 1

The first example in the figure shows two switches in VSAN 1. This situation indicates that the
switch where the command was issued has built its adjacency in VSAN 1 with the other switch
in the same VSAN.

One Switch in VSAN 1

In the second example that is shown in the figure, only one switch is recognized. This situation
indicates that the switch where the command was issued has not established adjacency with the
neighboring switch in VSAN 1.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-113

Information on the fcdomain can be viewed from the Cisco Device Manager by choosing the
FC menu and then selecting Domain Manager.

3-114 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Merging Fabrics
This topic describes the configuration settings that determine if switches achieve adjacency or
become isolated during a fabric merge event.

Single Switch to Single Switch

• The principal switch is determined by the
administratively assigned priority.
• If no priority is assigned, the principal switch is
determined by the WWN.

Single Switch to Multiple Switches

• The existing switch fabric always retains its principal
switch, even if the new switch has a higher
administratively assigned principal switch priority.

Multiple Switches to Multiple Switches

• The principal switch is determined by the
administratively assigned priority.
• If no priority is assigned and the default value is used
by every switch, the principal switch is determined by
the WWN of the existing principal switches of the two
fabrics.

In a Fibre Channel network, the principal switch issues domain IDs when a new switch is added
to an existing fabric. However, when two fabrics merge, the principal switch selection process
determines which of the existing switches becomes the principal switch for the merged fabric.
These rules characterize the election of the new principal switch:
 Rule 1: A switch with a nonempty domain ID list has priority over a switch that has an
empty domain ID list, and the principal switch is the principal switch of the first fabric.
When working with a single-switch fabric, the fabric does not contain a domain ID list.
 Rule 2: If both fabrics have a domain ID list, the priority between the two principal
switches is determined by configured switch priority. You can set this parameter. The
lower the value, the higher the priority.
 Rule 3: If the principal switch cannot be determined by the two previous criteria, the
principal switch is then determined by the WWNs of the two switches. The lower value has
the higher priority.

When merging two fabrics, the administrator can expect the following behavior:
 When connecting a single-switch fabric to a multiswitch fabric, the multiswitch fabric
always retains its principal switch regardless of the principal switch priority setting on the
single-switch fabric.
 When powering up a new switch that is connected to an existing fabric with two or more
switches, the existing switch fabric always retains its principal switch. This rule applies
even if the new switch has a higher administratively assigned principal switch priority.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-115

 When powering up a new switch that is connected to a standalone switch, the new principal
switch is determined by the administratively assigned priority. If no priority is assigned and
the default priority is used in every switch, the principal switch is determined by the WWN.
This principle also applies to connecting to two single-switch fabrics.

When connecting a multiswitch fabric to another multiswitch fabric, the principal switch is
determined by the administratively assigned priority. If no priority is assigned and the default
value is used by every switch, the principal switch is determined by the WWN of the existing
principal switches of the two fabrics.

3-116 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Domain ID overlap causes isolation of the affected VSAN
• E Port isolation or VSAN isolation on TE ports
• Automatic reconfiguration prevents isolation by forcing a disruptive
restart:
- Disruptive restart is required if the auto-reconfigure option is enabled after
isolation occurs.
- Automatic reconfiguration must be enabled on all switches.
- Automatic reconfiguration is disabled by default.
Fabric 1 No Domain ID Overlap Fabric 2
Domain 110 Domain 74
Domain 153 Domain 28
Fabrics merge and switches establish adjacency.

Overlapping Domain IDs

Fabric 1 Fabric 2

Domain 110 Domain 110

Domain 153 Domain 28

If two switch fabrics with two or more switches are connected and both fabrics have switches
with the domain ID already assigned, the expansion ports (E Ports) that are used to connect the
two fabrics are isolated in that VSAN.
The auto-reconfigure option prevents isolation by forcing a disruptive restart of the VSAN.
However, a disruptive restart is still required if autoreconfiguration is enabled after isolation
occurs.
The auto-reconfigure option must be enabled on all switches. This option is disabled by
default.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-117

• E Port becomes isolated in native VSAN.
• TE Port remains up for trunking VSANs that do not have domain ID
conflict.

switch# show interface fc1/7

Fc1/7 is down (Isolation due to domain overlap)
Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
vsan is 10
Beacon is turned off
192 frames input, 3986 bytes, 0 discards
0 runts, 0 jabber, 0 too long, 0 too short

When a switch fails to establish adjacency with its neighboring switch, the failure might be the
result of a domain ID conflict. Domain IDs are configured on a per-VSAN basis, so it is
possible that a switch can see all switches in one VSAN, but be isolated in another.
The E Ports that are used to connect two fabrics can become isolated when both of the
following conditions occur:
 Condition 1: Both fabrics have switches with the same domain ID already assigned.
 Condition 2: The auto-reconfigure option is disabled. (This option is disabled by default.)

In this case, the show interface command returns the following error message:
fc2/14 is down (Isolation due to domain overlap)
Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
The following examples display domain overlap conditions for various ISL configurations. The
first command displays the status for three ISLs.
The following example shows that interface fc1/8 is isolated:
switch# show interface fc1/7-9 brief
--------------------------------------------------------------
Interface Vsan Admin Admin Status FCOT Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
--------------------------------------------------------------
fc1/7 99 E on trunking swl TE 2 --
fc1/8 99 E off isolated swl -- --
fc1/9 99 E on trunking swl TE 2 --

Example 1: Trunking E Port (TE Port) fc1/7. Interface fc1/7 is trunking. The following
example shows that VSAN 99 is isolated:

3-118 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# show interface fc1/7 trunk vsan
fc1/7 is trunking
Vsan 1 is up, FCID is 0xed0000
Vsan 99 is down (Isolation due to domain overlap)
Vsan 181 is up, FCID is 0x1a0200
switch# show interface fc1/7
fc1/7 is trunking
. . .
Trunk vsans (admin allowed and active) (1,99,181)
Trunk vsans (up) (1,181)
Trunk vsans (isolated) (99)
Trunk vsans (initializing) ()

Example 2: E Port fc1/8. Interface fc1/8 is an E Port (trunk mode off). The port is down due to
domain overlap in the native VSAN of the port (VSAN 99) in this example:
switch# show interface fc1/8 trunk vsan
fc1/8 is not trunking
switch# show interface fc1/8
fc1/8 is down (Isolation due to domain overlap)
. . .
Admin port mode is E, trunk mode is off
Port vsan is 99
. . .

Example 3: TE Port fc1/9. Interface fc1/9 is trunking. VSAN 99 is isolated, and only VSAN 99
is allowed across the interface in this example:
switch# show interface fc1/9 trunk vsan
fc1/9 is trunking
Vsan 99 is down (Isolation due to domain overlap)
switch# show interface fc1/9
fc1/9 is trunking
. . .
Trunk vsans (admin allowed and active) (99)
Trunk vsans (up) ()
Trunk vsans (isolated) (99)
Trunk vsans (initializing) ()

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-119

• To resolve a domain ID overlap:
- Manually assign domain IDs.
- Disruptive restart is required when assigning preferred IDs and usually when
assigning static IDs.
- Nondisruptive restart is acceptable only when changing a preferred ID into a
static ID without actually changing the ID.

switch(config)# fcdomain domain 3 static vsan 1

switch(config)# fcdomain restart disruptive vsan 1

To manually assign a domain ID, issue one of the following commands in configuration mode:
switch(config)# fcdomain domain domain-id static vsan x
switch(config)# fcdomain domain domain-id preferred vsan x
The static option tells the switch to request that particular domain ID. If the switch does not get
that particular address, the switch isolates itself from the fabric. With the preferred option, the
switch requests the specified domain ID. If that domain ID is unavailable, the switch accepts
another domain ID.
After configuring the domain ID, you must restart Cisco Domain Manager.
Although the static option can be applied to run time after a disruptive or nondisruptive restart,
the preferred option is applied to run time only after a disruptive restart.

3-120 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configuring Fabric Merge Options
This topic explains how to configure fabric merge options.

• When joining two stable fabrics with overlapping domain assignments,

the following cases apply:
- If the auto-reconfigure option is disabled (default) on either or both switches,
the links between the two switches become isolated.
- Beware, because if the auto-reconfigure option is enabled on both switches,
a disruptive reconfiguration phase is started

switch# show fcdomain

VSAN 1
. . .
Local switch configuration information:
State: Enabled
FCID persistence: Enabled
Auto-reconfiguration: Disabled

By default, the auto-reconfigure option is disabled. When you join two switches that belong to
two different stable fabrics that have overlapping domains, the following cases apply:
 Case 1: If the auto-reconfigure option is enabled on both switches, a disruptive
reconfiguration phase is started.
 Case 2: If the auto-reconfigure option is disabled on either or both switches, the links
between the two switches become isolated.

To display if the auto-reconfigure option is enabled, use the show fcdomain command. The
display shows the setting on the local switch for each VSAN that is configured.
To enable the auto-reconfigure option on a particular VSAN, use the fcdomain auto-
reconfigure vsan vsan command in configuration mode.
The auto-reconfigure option takes immediate effect at run time. You do not need to reissue the
fcdomain command.
If a domain is currently isolated because of domain overlap, and you later enable the auto-
reconfigure option on both switches, the fabric continues to be isolated. However, if you
enable the option on both switches before connecting the fabric, a disruptive RCF occurs. A
disruptive reconfiguration can affect data traffic. You can nondisruptively reconfigure the Fibre
Channel domain by manually changing the configured domains on the overlapping links and
eliminating the domain overlap.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-121

• Cisco MDS 9000 ISL interfaces can reject RCF frames:
switch(config-if)# fcdomain rcf-reject vsan 172

• Domain overlap can result if static domain IDs are assigned to

switches in the fabric:
switch# show fcdomain
VSAN 172
. . .
Interface Role RCF-reject
---------------- ------------- ------------
fc1/8 Downstream Enabled

mds1# Jan 19 08:37:28 mds1 %FCDOMAIN-2-EPORT_ISOLATED: Isolation of

interface fc1/8 (reason: invalid RCF request/RCF Reject received) -
VSAN 172.

The administration of domain IDs typically involves restarting the fabric or VSAN. Restarting
guarantees a resynchronization of all switches in the fabric to ensure that the switches
simultaneously restart a new principal switch selection phase. The principal switch guarantees
that each switch in the fabric obtains a unique domain ID. Fibre Channel domains can be
started disruptively or nondisruptively. With each disruptive restart, RCF frames are sent to
other switches in the fabric.
Cisco MDS 9000 Series switches can be configured to reject RCF frames. To determine if your
switch is configured to reject or accept RCFs, use the show fcdomain command. The output
shows the RCF rejection status for each ISL interface. The figure shows that interface fc1/8 has
RCF rejection enabled for VSAN 172. If an RCF for VSAN 172 is received from the fabric, the
following logging message is echoed:
Jan 19 08:37:28 mds1 %FCDOMAIN-2-EPORT_ISOLATED: Isolation of
interface fc1/8 (reason: invalid RCF request/RCF Reject
received) - VSAN 172.
To verify the trunking status of the interface, use the show interface fcx/y trunk vsan
command:
switch# show interface fc1/8 trunk vsan
fc1/8 is trunking
Vsan 1 is up, FCID is 0x640200
Vsan 171 is up, FCID is 0x640000
Vsan 172 is down (Isolation due to invalid fabric
reconfiguration)
Vsan 173 is up, FCID is 0x620100

3-122 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• In the SAN fabric, it is recommended that you configure static domain

IDs for stability.
• In a Fibre Channel network, the principal switch issues domain IDs when
a new switch is added to an existing fabric. However, when two fabrics
merge, the principal switch selection process determines which one of
the existing switches becomes the principal switch for the merged fabric.
• Domain IDs can be statically configured through either the CLI or the
Cisco Fabric Manager GUI.
• Use the show fcdomain command to display the domain database. Use
the fcdomain command or the Domain Manager menu of Cisco Device
Manager to configure domain settings.
• The fcdomain feature can be restarted disruptively or nondisruptively
using the fcdomain restart command or Cisco Fabric Manager.
Disruptive restart is required to apply most configuration changes to run
time.
• Fabric merge behavior can be configured by using the auto-reconfigure
option or by enabling RCF rejection.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-123

3-124 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 7

Configuring Distributed Device

Aliases
Overview
This lesson explains the differences between a Fibre Channel alias and a distributed device
alias. The lesson also covers the use and proper configuration of a distributed device alias.

Objectives
Upon completing this lesson, you will be able to configure distributed device aliases. This
ability includes being able to meet these objectives:
 Describe the function of distributed device aliases
 Describe the function of existing zone aliases
 Explain how to configure the distributed device alias database
 Explain how to configure distributed device aliases in a multiswitch fabric
 Explain how to verify the distributed device alias database configuration
Distributed Device Alias Overview
This topic describes the function of the distributed device alias service that simplifies SAN
configuration and management tasks by assigning a global alias to device port world wide
names (pWWNs).

• Distributed device alias simplifies SAN

configuration and management tasks: Cryptic WWNs
- Provides user-friendly CLI and Cisco DCNM-SAN WWN1 = 12:22:67:92:86:92:15:34
Client commands and outputs.
WWN2 = 02:12:35:86:93:08:64:43
- The fabricwide scope ensures no reconfiguration
when a device is moved across VSANs. FC

- Works in conjunction with many applications FC

(zoning, QoS, port security, and so on). FC

WWN1 WWN2
• Device alias features include the following:
- Independent of VSAN and zoning configurations.
- Permits importing of legacy Fibre Channel alias Global Device Aliases
definitions.
WWN1 = Server-Oracle-ERP
- Supports Cisco Fabric Services by default for
fabricwide synchronization. WWN2 = Array-OLTP

- Alias names are displayed in Cisco DCNM-SAN

Client and in the CLI output of show commands. FC
FC

- Offers support for 8192 device aliases fabricwide.

WWN1 WWN2
- Provides high availability support across switch
resets and switchovers.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-4

When a device pWWN must be specified to configure different applications such as zoning,
quality of service (QoS), and port security in a Cisco MDS 9000 Series Multilayer Switch, it is
important to assign the correct device name each time that the feature is configured. Failure to
do so may cause unexpected results. This problem can be avoided if a user-friendly name for a
pWWN is used instead, for all configuration commands, as required. These user-friendly names
are referred to as device aliases on MDS 9000 Series switches.
Device aliasing supports two modes, which are basic and enhanced.
 Basic: When a device alias runs in basic mode, the application immediately expands the
user-friendly name to pWWNs.
 Enhanced: When a device alias runs in enhanced mode, which is a best practice, all
applications accept the device alias configuration in the native format. The applications
store the device alias name in the configuration and distribute the name in the device alias
format instead of expanding to pWWNs. The applications track the device alias database
changes and take actions to enforce the changes.

The process can be automated using the force option. Use the no device-alias mode enhanced
force command to enable applications to automatically replace all device alias members with
the corresponding pWWNs. If a device alias member does not have a corresponding pWWN
mapping in the device alias database, the configuration will be removed.
Device aliases have the following features:

3-126 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Device alias information is independent of your virtual SAN (VSAN) configuration.
Aliases need to be defined only once and can then be used regardless of their VSAN.
 Device alias configuration and distribution are independent of the zone server and zone
server database.
 You can import existing zone alias configurations without losing data.
 The device alias application uses Cisco Fabric Services infrastructure to enable efficient
database management and distribution. Device aliases use coordinated distribution mode
and physical distribution scope. The device alias feature and Cisco Fabric Services
distribution are both enabled by default.
 When you configure zones, Inter-VSAN Routing (IVR) zones, or QoS features using
device aliases and you display the configuration, you automatically see that the device
aliases are displayed along with their respective pWWNs.
 Up to 8192 device aliases can be configured fabric-wide.
 Distributed device alias service is a highly available process. The device alias databases are
preserved across switch restarts and switchovers. The system can be restarted or switched
over during Cisco Fabric Services distributions or merges.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-127

• Device aliases map only to pWWNs.
• A given device alias can map to only one pWWN.
• A given pWWN can map to only one device alias.
• Device alias names must begin with a letter and include the following
restrictions:
- Names are restricted to 1 to 64 characters.
- Permissible characters include lowercase a to z, uppercase A to Z, numbers 0
to 9, and the characters hyphen (-), underscore (_), dollar sign ($), and caret
(^).

This_Is_A_$VerY$_Long-^bUt^-VaLid_Device_Alias_Name_9876543210$^

Device aliases have the following requirements:

 You can assign device aliases only to pWWNs.
 You should ensure that the mapping between the pWWN and the device alias to which the
pWWN is mapped has a one-to-one relationship. A pWWN can be mapped to only one
device alias, and vice versa.
 A device alias name must begin with a letter and is restricted to 1 to 64 characters.
Permissible characters include one or more of the following:
— Lowercase letters a to z
— Uppercase letters A to Z
— Numbers 0 to 9
— Hyphen (-)
— Underscore (_)
— Dollar sign ($)
— Caret (^)

3-128 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Existing Zone Alias Overview
This topic provides a comparison between Fibre Channel aliases and device aliases.

Fibre Channel Aliases Device Aliases

Aliases are limited to the specified You can define device aliases without
VSAN. specifying the VSAN number. You can also
use the same definition in one or more
VSANs without any restrictions.
Fibre Channel aliases are part of the Device aliases can be used for any feature
zoning configuration. Alias mapping that uses the pWWN.
cannot be used to configure other
features.
You can use any zone member type to Only pWWNs are supported, along with new
specify the end devices. device aliases like IP addresses.
Configuration is contained within the Device aliases are not restricted to zoning.
zone server database and is not Device alias configuration is available to the
available to other features. FCNS, zoning, Fibre Channel ping, fctrace,
and IVR applications.
Aliases are distributed by zoning Aliases are distributed by Cisco Fabric
activation. Services automatically.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-7

There are many functional differences between Fibre Channel aliases and device aliases.
Device aliases are restricted to pWWNs and IP addresses, but device aliases are not restricted
to the zone server database or by VSAN membership.
Distributed device alias service configurations are distributed using Cisco Fabric Services
automatically by default and available to other applications, such as Fibre Channel Name
Server (FCNS), Fibre Channel Ping (fcping), Fibre Channel Traceroute (fctrace), zoning, and
IVR. These applications enhance the utility of the configurations and make the use of the
aliases more compelling.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-129

• Importing legacy Fibre Channel alias definitions:
- Each Fibre Channel alias may have only one member of type pWWN.
- The Fibre Channel alias name and member may not be the same as an
existing device alias name or member.
• After importing, changes must be committed.

VSAN 10
Global Device
Legacy Fibre
Import Alias
Channel Alias
Database
Definitions

switch(config)# device-alias import fcalias vsan 5-10

switch(config)# device-alias commit

When an import operation is complete, the modified alias database is distributed to all other
switches in the physical fabric when you perform the commit operation. At that time, if you do
not want to distribute the configuration to other switches in the fabric, performing the abort
operation will discard the merge changes.

3-130 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Distributed Device Alias Database Configuration
This topic describes the procedure for configuring a distributed device alias database.

• Configure device aliases once all ISLs are up. Configuration locks the
fabric.
switch(config)# device-alias database
switch(config-device-alias-db)# device-alias name foo pwwn
21:02:03:4a:5c:36:7a:e8

- Commit the pending database.

switch(config)# device-alias commit

- Abort the pending changes.

switch(config)# device-alias abort

- Clear a locked session.

switch# clear device-alias session

The device alias feature uses two databases to accept and implement device alias configuration:
 Effective database: The effective database is the database that is currently used by the
fabric.
 Pending database: The pending database is the database where subsequent device alias
configuration changes are stored while awaiting a commit.

Begin configuration once all Inter-Switch Links (ISLs) are up. After adding or modifying the
device alias configuration, a commit or discard command must be performed, because the
fabric remains locked during this period.
Device alias distribution is enabled by default. Device alias distribution uses a coordinated
distribution mechanism to distribute the modifications to all switches in a fabric. When you
configure the first device alias task, the fabric is automatically locked for the device alias
feature. Once locked, the following applies:
 No other user can make any configuration changes to this feature.
 A copy of the effective database is obtained and used as the pending database. All
modifications occur in the pending database, which remains in effect until you commit the
modifications or discard (abort) the changes.
 If a device alias task is performed and the lock has not been released by either a commit or
discard command, an administrator can release the lock from any switch in the fabric. If
the administrator clears a lock, then any changes in the pending database are discarded.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-131

• Cisco DCNM-SAN Client supports creating, distributing, displaying, and
using device aliases.

Use the following procedure to configure and distribute device aliases using Cisco DCNM-
SAN Client:
Step 1 Choose the End Devices folder in the Physical Attributes pane.
Step 2 Enter the device alias names as desired in the Device Alias fields in the Information
pane.
Step 3 Click Apply Changes.
Step 4 To make device alias names appear as enclosure names in the topology map,
highlight the rows in the table, and click Alias Enclosure.
Device aliases can also be used in Cisco DCNM-SAN Client. For example, when you are
creating zones for a VSAN using Cisco DCNM-SAN Client, the configured device aliases
appear for all of the pWWNs that have been mapped to device aliases within that VSAN.

3-132 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Configuring Device Aliases in a Multiswitch Fabric
Device aliases are global and persistent across all VSANs in all MDS switches in the SAN. By
default, device alias distribution is enabled, so that all MDS switches contain the same
distributed device alias service database contents.

• The following are guidelines for successful device alias database merge:
- Device alias distribution is enabled by default.
- Device aliases with different names cannot be mapped to the same pWWN.
- Verify that the combined number of device aliases in both databases does not
exceed 8192.
- For example, if database N has 6000 device aliases and database M has
2192 device aliases, the merge operation fails.

switch(config)# device-alias database

1 switch(config-device-alias-db)# device-alias name
foo pwwn 21:02:03:4a:5c:36:7a:e8

Configuring a device alias locks the fabric.

3
M
N 2 switch(config)# device-alias commit
Commit the pending database. Device Alias Database
Distribution or Merge
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-13

By default, device alias distribution is enabled. The device alias feature uses the coordinated
distribution mechanism to distribute the modifications to all switches in a fabric.
A coordinated distribution is not allowed to start if locks are taken for the application anywhere
in the fabric. A coordinated distribution consists of three stages:
1. A fabric lock is acquired.
2. The configuration is distributed and committed.
3. The fabric lock is released.
Coordinated distribution for distributed device alias services is driven by Cisco Fabric Services.
The stages are executed by Cisco Fabric Services in response to an application request without
intervention from the application.
An application keeps the configuration synchronized in a fabric through Cisco Fabric Services.
Two fabrics might merge as a result of an ISL coming up between the fabrics. These two
fabrics could have two different sets of configuration information that need to be reconciled in
the event of a merge. Cisco Fabric Services provides notification each time an application peer
comes online. If two fabrics with M and N application peers merge, and if an application
triggers a merge action on every such notification, a link-up event results in M*N merges in the
fabric.
Cisco Fabric Services supports a protocol that reduces the number of merges that are required
to one by managing the complexity of the merge at the Cisco Fabric Services layer. This
protocol runs per application per scope. The protocol involves selecting one switch in a fabric

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-133

as the merge manager for that fabric. The other switches do not play any role in the merge
process.
During a merge, the merge managers in the two fabrics exchange their configuration databases
with each other. The application on one fabric merges the information, decides if the merge is
successful, and informs all switches in the combined fabric of the status of the merge.
When a successful merge occurs, the merged database is distributed to all switches in the
combined fabric, and the entire new fabric remains in a consistent state. You can recover from
a merge failure by starting a distribution from any of the switches in the new fabric. This
distribution restores all peers in the fabric to the same configuration database.

3-134 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Ensure that both fabrics are using the same mode.

Device Alias X Device Alias

Mode Basic Mode Advanced
Merge Failure

Device Alias Device Alias

Mode Advanced Merge Success Mode Advanced

If device alias distribution is turned on, the distribution is provided to the other switches in the
network whenever there is a change in the mode.

Merging Device Alias

If two fabrics are running different device alias modes and are joined, the device alias merge
will fail. There is no automatic conversion of one mode to the other mode during the merge
process.
At the application level, a merge takes place between the applications and the fabric. For
example, zone merge occurs when the expansion (E) port is up, and the IVR and dynamic port
VSAN membership (DPVM) merge occurs due to Cisco Fabric Services. This merge is
independent of the device alias merge.
If the application running on an enhanced fabric has a native device alias configuration, the
application will fail the merge if the other fabric is running in basic mode. This process occurs
regardless of whether the fabric can support enhanced mode. If the device alias merge fails
between two fabrics that are running different modes, the conflict can be resolved by selecting
one mode or the other.

Note If you choose basic mode, ensure that all applications running on the enhanced fabric
comply with the device alias merge.

The device alias merge will fail due to mode mismatch, but the application merge will succeed
if the fabric does not have any native device alias configurations.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-135

Distributed Device Alias Database Verification
This topic describes the command that is used to verify the distributed device alias database.

switch# show device-alias database

device-alias name DISK1-P1 pwwn 22:00:00:04:cf:70:4a:3b
device-alias name DISK1-P2 pwwn 22:00:00:04:cf:70:4b:f7
device-alias name DISK2-P1 pwwn 21:00:00:04:cf:70:4a:3b
device-alias name DISK2-P2 pwwn 21:00:00:04:cf:70:4b:f7
device-alias name HOST1-P1 pwwn 10:00:00:06:2b:08:e5:30
device-alias name HOST1-P2 pwwn 10:00:00:06:2b:08:e5:81
device-alias name HOST2-P1 pwwn 10:00:00:06:2b:08:e5:80
device-alias name HOST2-P2 pwwn 10:00:00:06:2b:08:e5:31

Total number of entries = 8

Use the following commands to verify the configuration of the device alias database:
 show device-alias database
 show flogi database
 show fcns database

3-136 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# show flogi database
-------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
-------------------------------------------------------------------------------
fc1/5 20 0x160000 10:00:00:06:2b:08:e5:80 20:00:00:06:2b:08:e5:80
[HOST2-P1]
fc1/6 20 0x16019b 21:00:00:04:cf:70:4b:f7 20:00:00:04:cf:70:4b:f7
[DISK2-P2]
fc1/6 20 0x1601b3 21:00:00:04:cf:70:4a:3b 20:00:00:04:cf:70:4a:3b
[DISK2-P1]

Total number of flogi = 3.

switch# show fcns database

VSAN 20:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x160000 N 10:00:00:06:2b:08:e5:80 (LSI) ipfc scsi-fcp:init
[HOST2-P1]
0x16019b NL 21:00:00:04:cf:70:4b:f7 (Seagate) scsi-fcp:target
[DISK2-P2]
0x1601b3 NL 21:00:00:04:cf:70:4a:3b (Seagate) scsi-fcp:target
[DISK2-P1]
0xea0000 N 10:00:00:06:2b:08:e5:31 (LSI) ipfc scsi-fcp:both
[HOST2-P2]
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-17

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-137

Summary
This topic summarizes the key points that were discussed in this lesson.

• Distributed device aliases are user-friendly names mapping to the

pWWN of a device. The aliases can be used in any configuration task
that requires a reference to a pWWN.
• Legacy zone aliases are user-friendly names mapping to a device, but
the aliases can be used only when configuring zoning.
• Configuration of device aliases is performed in the distributed device
alias database, where configuration changes go into a pending database
awaiting commitment across the fabric.
• When distributing the distributed device alias database in a multiswitch
fabric, it is important to verify the compatibility of the databases and the
mode of each switch.
• The distributed device alias database can be verified using one of the
various show device-alias commands.

3-138 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 8

Implementing Zoning
Overview
This lesson describes the differences between basic and enhanced zoning, how to configure
zoning as well as Smart Zoning, and how to manage zone sets. Also discussed are the causes of
zone merge conflicts and the tools available on the Cisco MDS 9000 Series Multilayer
Switches to identify and resolve conflicts in the SAN.

Objectives
Upon completing this lesson, you will be able to implement zoning. This ability includes being
able to meet these objectives:
 Describe the purpose and use of zoning within Fibre Channel SANs
 Describe the zone configuration process
 Explain how to verify the zone configuration
 Describe how to configure zone set distribution
 Explain how to merge zones and zone configurations without causing fabric disruption
 Explain how to recover from a zone merge failure
 Describe how to rename, clone, copy, back up, and restore a zone set
 Describe the enhanced zoning feature and configuration requirements
 Describe the process of committing configuration changes to the zone database in enhanced
mode
 Describe the advantages of the Smart Zoning feature for SANs with large numbers of
zones.
 Describe recommended zoning practices
Zoning Overview
Zoning is a security mechanism within Fibre Channel that is used to restrict communication
between devices with the same Fibre Channel fabric. Zoning segments a Fibre Channel fabric
into multiple partitions. Devices in one zone cannot learn of the existence of devices in other
zones. This topic explains how zone membership is used to uniquely identify a device or
devices that are to be included in a zone.

• Zones restrict communication between devices in the same Fibre Channel

fabric.
- Zones consist of one or more zone members.
- Zone sets consist of one or more zones.
• Devices not assigned to a zone belong to the default zone.
- The default zone is set to deny in order to isolate devices from one another.
• Single initiator zoning is recommended.
- Single initiator zoning stops hosts from trying to login to other hosts.

Zone Set
Zone A
FC
Default Zone FC
Zone B
FC FC
FC
FC FC
FC
FC FC
FC
FC

FC FC

FC FC
FC Zone C FC
FC FC
FC

With many different types of servers and storage devices on the network, the need for security
is critical. For example, if a host gained access to a disk being used by another host, potentially
with a different operating system, the data on this disk could become corrupted. To avoid any
compromise of critical data within the SAN, zoning allows the user to overlay a security map.
The map dictates which devices, namely hosts, can see which targets, reducing the risk of data
loss.
 A zone set consists of one or more zones with the following characteristics:
 A zone set can be activated or deactivated as a single entity across all switches in the fabric.
 Only one zone set can be activated at any time.
 A zone can be a member of more than one zone set.
 A zone consists of multiple zone members. Members in a zone can access each other, but
members in different zones cannot access each other.

Uses for Zoning

 Typical uses for zoning include the following:
 To separate initiators from their targets. Frequently each initiator port will belong in a
separate zone with its targets.
 To separate devices that use different operating systems. This practice is useful to protect
some operating systems from treating disks that are formatted by other operating systems
as blank disks and potentially taking over and overwriting their storage.
3-140 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 To separate devices that have no need to communicate with other devices in the fabric or
that have classified data.
 To separate devices into departmental, administrative, or other functional groupings.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-141

• Hierarchical relationship
- Physical ports are assigned to VSANs. Relationship of VSANs to Zones
- Independent zones are configured per Physical Topology
VSAN.
• VSANs VSAN 2 Active ZonesetA
- VSANs virtualize the physical switch ZoneA
infrastructure. Host1 Disk1
Disk3 Disk2
- VSANs provide logical isolation of Fibre
Channel Cisco Fabric Services. ZoneC
- VSANs change only when VSANs are Default ZoneB Host2
added or ports are moved to a different Zone Disk4
VSAN.
• Zones
VSAN 7 Active ZonesetD
- Zones subdivide the VSAN
infrastructure.
Host4
- Zones maintain connectivity between
devices. ZoneD
- Zones maintain isolation between hosts. ZoneA
Default Disk5
- Zones change frequently (for example, Zone
Host3
backup). Disk6

• Ports are added and removed

nondisruptively to and from VSANs.

Virtual SANs and fabric zoning are complementary. For the hierarchical relationship, you first
assign physical ports to virtual SANs (VSANs) and then you configure independent zones for
each VSAN.
VSANs divide the physical infrastructure, provide traffic statistics, and are only changed when
ports are needed for each VSAN fabric.
Zones divide the VSAN infrastructure, while providing added security and allowing the sharing
of device ports. Zones can be changed frequently, as with backup applications where a host
cannot communicate to a server during the day, but connects to the server at night for backup
storage.
In VSAN 2, three zones are defined, which are zone A, zone B, and zone C. Zone C overlaps
both zones A and B as permitted by Fibre Channel standards. In VSAN 7, two zones are
defined, which are zone A and zone D. No zone crosses the VSAN boundary. The zones are
completely contained within the VSAN. Zone A defined in VSAN 2 is different and separate
from zone A defined in VSAN 7. This process can be described hierarchically as follows:

VSAN 2
ZonesetA
 Zone A
— Host 1
— Disk 1
— Disk 2
— Disk 3
 Zone B
— Host 2
— Disk 4

3-142 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Zone C
— Host 2
— Disk 1
— Disk 2

VSAN 7
ZonesetD
 Zone A
— Host 3
— Disk 5
— Disk 6
 Zone D
— Host 4
— Disk 6

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-143

• MDS 9000 Series switches support the following: Device Alias
- Hardware-enforced zoning Fibre Channel Alias
FC
• Zoning is enforced on every frame in
hardware by the forwarding ASIC. IQN LUN
iSCSI
IP Address
- Fibre Channel standards
pWWN FCID
• FC-GS3, FC-GS-4, FC-SW2, FC-SW3, and
FC-MI fWWN Int fc1/1

- Zone merge analysis Device Alias

FC Alias
• In DCNM-SAN Client pWWN fWWN
FC
FCID Int fc1/2
• Prevents fabric merge failures due to zone HBA

database mismatch sWWN

- Zone member options

Zone Member Options

pWWN (attached Nx_Port) fWWN (switch port-based zoning) iSCSI IQN Node Name
FC alias (within a VSAN) Interface (fc1/2) iSCSI IP Address
Device alias (global within a SAN) sWWN and port
FCID
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS20 v2.0—3-6

Zoning is a mechanism to control access to devices with a Fibre Channel fabric. On Cisco MDS
9000 Series switches, zoning is enforced separately in each VSAN.
Even though the MDS 9000 Series supports both hard and soft zoning enforcement methods,
the MDS 9000 Series switches always enforce zoning in hardware at the ASIC level.

Soft Zoning
Soft zoning is enforced through selective query responses that are made to the Fibre Channel
Name Server (FCNS). Each end device (node [N] port or node loop [NL] port) discovers other
devices in the fabric by querying the name server. When a device logs in to the name server,
the name server returns the list of other devices that the querying device can accessed. If an Nx
Port does not know about the Fibre Channel IDs (FCIDs) of other devices outside its zone, the
port cannot access those devices.
In soft zoning, zoning restrictions are applied only during interaction between the name server
and the end device. If an end device somehow knows the FCID of a device outside its zone, the
end device can access that device.

Hard Zoning
Hard zoning is enforced on each data frame that is sent by an Nx Port by the forwarding ASIC.
As frames enter the switch, source-destination IDs are compared with permitted combinations
to allow the frame at wire speed.
Zoning can be based on port world wide name (pWWN) and fabric world wide name (fWWN),
FCID, interface, and logical unit number (LUN). A LUN is a group of Small Computer
Systems Interface (SCSI) blocks that are contained within one or more disk drives inside a
storage array. When presented to the operating system and volume manager, the LUN becomes
a disk volume that must be formatted before use.
Zone membership criteria can be based on many different SAN features including the following
options:
 pWWN: Defines the world wide name (WWN) of an attached device port.

3-144 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Fibre Channel alias: Defines the common name that is based on the pWWN, but is only
valid with a single VSAN.
 Device alias: Defines the common name that is based on the pWWN, but is global and
valid across all VSANs.
 FCID: MDS switches are unique in the industry because those switches assign FCIDs
dynamically so that the FCID is not tied to a physical switch port.
 Switch port zoning includes the following options:
 fWWN: Defines the WWN of a switch port on an MDS line card.
 Interface: The interface is the line card interface number. In other words, the interface
fc1/2 indicates slot 1 and second port from the left. (All Cisco MDS numbering starts at 1.)
 sWWN and port: The switch world wide name (sWWN) defines the WWN of the MDS
switch and port number within that switch.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-145

Configuring Zones and Zone Sets
This topic describes the commands that are used to configure zones and zone sets.

• Device aliases are kept when a device is moved to a different VSAN.

• Members can be added to a zone by using the following formats:
switch(config)# zone name Zone1 vsan 3
switch(config-zone)# member <type> <value>
switch(config-zone)# member pwwn 10:00:00:23:45:67:89:ab
switch(config-zone)# member fcid 0xce00d1
switch(config-zone)# member fcalias HRdisk1
switch(config-zone)# member device-alias host1-p1
switch(config-zone)# member ipaddress 10.15.0.0 255.255.0.0
switch(config-zone)# member interface fc2/1 swwn 20:00:00:05:30:00:4a:de
switch(config-zone)# member interface fc2/1 domain-id 25
switch(config-zone)# member domain-id 2 portnumber 23
switch(config-zone)# member fwwn 10:01:10:01:10:ab:cd:ef

Recommended practice:
Zone using device aliases.

For configuration purposes, a zone is made up of a zone name and members. The zone name is
an alphanumeric string that gives information about the zone contents. Zones can be configured
using the Cisco Data Center Network Manager (DCNM)-SAN Client Zone Member wizard or
the CLI.
To configure a zone from the CLI, use these commands:
switch# config
switch(config)# zone name Zone1 vsan 3
switch(config-zone) # member pwwn 10:00:00:23:45:67:89:ab
switch(config-zone) # member fwwn 10:01:10:01:10:ab:cd:ef
switch(config-zone) # member fcid 0xcd00d1
switch(config-zone) # member fcalias HRdisk1
switch(config-zone) # member domain-id 2 portnumber 23
switch(config-zone) # member ipaddress 10.15.0.0 255.255.0.0
Use a relevant display command like show interface or show flogi database to obtain the
required value in hexadecimal format.
Interface-based zoning works only with Cisco MDS 9000 Series switches. Include the sWWN
when zoning by interface to determine a local or remote switch. Use the show wwn switch
command to retrieve the sWWN. If you do not provide an sWWN, the software automatically
uses the local sWWN.

3-146 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Configure a zone set and add previously created zones.
switch# conf
switch(config)# zoneset name Zoneset1 vsan 3
switch(config-zoneset)# member Zone1
switch(config-zoneset)# member Zone2

• Configure a zone set and create and add new zones.

switch# conf
switch(config)# zoneset name Zoneset1 vsan 3
switch(config-zoneset)# zone name NewZone1
switch(config-zoneset-zone)# member device-alias ERPHost1
switch(config-zoneset-zone)# zone name NewZone2
switch(config-zoneset-zone)# member device-alias ERPHost2

Zones provide a mechanism for specifying access control, while zone sets are groupings of
zones to enforce access control in the fabric. Zone sets are configured using a zone set name
and the members of the zone set. Configured zone sets need to have the VSAN specified. The
alphanumeric name of the zone set is used for identification purposes and should carry meaning
within the fabric. The members of a zone set are zones, and the members are configured using
the names of the individual zones.
Zone sets, like zones, can be configured from the CLI or the Cisco DCNM-SAN Client. To
configure a zone set from the CLI, use these commands:
switch# config
switch(config)# zoneset name zoneset 1 vsan 3
switch(config-zoneset)# zone name NewZone1
switch(config-zoneset-zone)# member device-alias ERPHost1

Zone Set Guidelines

Before configuring a zone set, consider the following guidelines:
1. Each VSAN can have multiple zone sets, but only one zone set can be active at any given
time.
2. When you create a zone set, that zone set becomes a part of the full zone database.

3. When you activate a zone set, a copy of the zone set from the complete zone set is used to
enforce zoning and is called the active zone set. An active zone set cannot be modified. A
zone that is part of an active zone set is called an active zone. You can activate a zone set
using the zoneset activate name command.
4. The administrator can modify the complete zone set even if a zone set with the same name
is active. The changes do not take effect until the zone set is activated with the zoneset
activate name command.
5. When the activation is finished, the active zone set is automatically stored in persistent
configuration. This action enables the switch to preserve the active zone set information
© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-147
across switch resets. You do not have to issue the copy running-config startup-config
command to store the active zone set. However, you need to issue the copy running-config
startup-config command to explicitly store complete zone sets. The command is not
available across switch resets.
6. All other switches in the fabric receive the active zone set so the fabrics can enforce zoning
in their respective switches.
7. Hard and soft zoning are implemented using the active zone set. Modifications take effect
during zone set activation.

8. An FCID or Nx Port that is not part of the active zone set belongs to the default zone, and
the default zone information is not distributed to other switches.
9. If one zone set is active and you activate another zone set, the currently active zone set is
automatically deactivated. You do not need to explicitly deactivate the currently active
zone set before activating a new zone set.

3-148 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Activate a zone set.
switch(config)# zone activate name Zoneset1 vsan 3

• Deactivate a zone set.

switch(config)# no zoneset activate name Zoneset1 vsan 3

Zone Set A Zone Set B

Full Zone Set Zone 1 Zone 2

Zone 2 Zone 3

Zone 4

Zone Set A

Zone 1
Active Zone Set
Zone 2 X
Zone 4

The zoneset activate command activates the zone set named Zoneset1 in VSAN 3. The
activation will automatically deactivate any currently active zone set. The zone and zone set
configurations do not take effect until they are activated.
When using enhanced mode, you only need to commit. Enhanced mode will be discussed later.

• Logical domains:
- Edit full zone database.
- Deactivate zone set.
- Copy full zone database.

• Edit full zone database:

- Configure zoning.
- Activate and distribute zone sets.

Cisco DCNM-SAN Client provides an easy tool for all zone configuration tasks. Right-click
the VSAN folder in the Logical Domains pane to create and edit zone sets. The pop-up menu
displays several options, including the option Cisco DCNM-SAN Client Parameters.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-149

Zone Configuration Verification
This topic describes the commands that are used to verify the zone configuration on a local
switch.

switch# show zoneset vsan 3

zoneset name ZoneSet1 vsan 3
zone name Zone1 vsan 3
pwwn 21:00:00:e0:8b:03:18:24 [host1-p1]
pwwn 21:00:00:04:cf:d6:f3:bd [disk1-p1]
zone name Zone2 vsan 3
pwwn 21:01:00:e0:8b:22:29:66 [host2-p2]
pwwn 21:00:00:0c:50:9e:8b:36 [disk2-p1]
switch# show zoneset active vsan 3
zoneset name ZoneSet1 vsan 3
zone name Zone1 vsan 3
* fcid 0x420000 [pwwn 21:00:00:e0:8b:03:18:24] [host1-p1]
* fcid 0x4201e2 [pwwn 21:00:00:04:cf:d6:f3:bd] [disk1-p1]
zone name Zone2 vsan 3
* fcid 0x420200 [pwwn 21:01:00:e0:8b:22:29:66] [host2-p2]
* fcid 0x4201ef [pwwn 21:00:00:0c:50:9e:8b:36] [disk2-p1]

The current zoning configuration on the local switch can be displayed by using the show
zoneset command in EXEC mode.
To verify the current active zone set, use show zoneset active from EXEC mode. The asterisks
indicate that a device is visible (online). A missing asterisk may indicate an offline device or an
incorrectly configured zone, possibly a mistyped pWWN.

3-150 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Use show zone analysis commands.
- show zone analysis vsan 10
- show zone analysis active vsan 10
- show zone analysis zoneset zs1 vsan 10

switch# show zone analysis active vsan 10

Zoning database analysis vsan 10
Active zoneset: zoneset1
Activated at: 16:57:22 UTC Mar 04 2010
Activated by: Local [ CLI ]
Default zone policy: Deny
Number of devices zoned in vsan: 2/4 (Unzoned: 2)
Number of zone members resolved: 2/2 (Unresolved: 0)
Num zones: 1
Number of IVR zones: 0
Number of IPS zones: 0
Formatted size: 60 bytes / 2048 Kb

To better manage the zones and zone sets on your switch, display zone and zone set
information using the show zone analysis command. The following sample command outputs
are shown:
switch# show zone analysis vsan 10
Zoning database analysis vsan 10
Full zoning database
Last updated at: 16:57:06 UTC Mar 04 2010
Last updated by: Local [ CLI ]
Num zonesets: 1
Num zones: 1
Num aliases: 0
Num attribute groups: 0
Formatted size: 112 bytes / 2048 Kb
switch# show zone analysis active vsan 10
Zoning database analysis vsan 10
Active zonesets: zoneset1
Activated at: 16:57:22 UTC Mar 04 2010
Activated by: Local [ CLI ]
Default zone policy: Deny
Number of devices zoned in vsan: 2/4 (Unzoned: 2)
Number of zone members resolved: 2/2 (Unresolved: 0)
Num zones: 1
Number of IVR zones: 0
Number of IPS zones: 0
Formatted size: 60 bytes / 2048 Kb
switch# show zone analysis zoneset zoneset1 vsan 10
Zoning database analysis vsan 10
Zoneset analysis: zoneset1

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-151

Num zonesets: 1
Num zones: 1
Num aliases: 0
Num attribute groups: 0
Formatted size: 112 bytes / 2048 Kb

Zone sets, zones, and zone member information can be displayed by expanding the zone set
folder for the VSAN in question. Selecting a zone or zone set will highlight member devices in
the zone in the Cisco DCNM-SAN Client topology map. Zone configuration changes can be
made from the Zone > Edit Full Local Zone Database menu.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-153

Configuring Zone Set Distribution
This topic describes the methods that are used to distribute zone sets.

Full Zone Set • By default, only the active zone

set is distributed.
Zone Set A Zone Set B
• The full zone database is
Zone 1
Zone 2 resident on only a single switch.
Zone 2
Zone 3
Zone 4

Zone Set B Zone Set B

Active Zone Set Zone 2 Zone 2

Zone 3 Zone 3

By default, only the active zone set is distributed when the zone configuration is changed or
when two fabrics merge.
All MDS 9000 Series switches distribute active zone sets when new expansion (E) port links
come up or when a new zone set is activated in a VSAN. The zoneset distribute full vsan
command distributes the complete zone set along with the active zone set. Distribution takes
effect while sending merge requests to the adjacent switch or while activating a zone set.
All MDS 9000 Series switches distribute active zone sets when new E Port links come up or
when a new zone set is activated in a VSAN. Alternatively, you can use the zoneset distribute
full vsan configuration command to perform a one-time distribution of all inactive and active
zone sets. The zoneset distribute full vsan command distributes the complete zone set along
with the active zone set.
switch# zoneset distribute vsan 2
Zoneset distribution initiated. check zone status
This command only distributes the full zone set information. The command does not save the
information to the startup configuration. You must explicitly issue the copy running-config
startup-config command to save the full zone set information to the startup configuration.

3-154 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Using the zoneset command in configuration mode:
- The distribute full command enables full zone set propagation on activation.

switch(config)# zoneset distribute full vsan 3

switch(config)# zoneset activate name SetZ vsan 3

• Using the zoneset command in EXEC mode:

- The distribute command merges zone set databases. You use the zoneset
distribute command in EXEC mode.

switch# zoneset distribute vsan 3

Zoneset distribution initiated. check zone status

• A recommended practice is to manage zoning from a single switch.

You can distribute full zone sets using one of two methods:
 Executive mode level
 Configuration mode level

Enabling Full Zone Set Distribution

All switches in the MDS 9000 Series distribute active zone sets when new Inter-Switch Links
(ISLs) come up or when a new zone set is activated in a VSAN. The zone set distribution takes
effect while sending merge requests to the adjacent switch, or while activating a zone set.

One-Time Distribution
You can perform a one-time distribution of inactive, unmodified zone sets throughout the fabric
using the zoneset distribute vsan vsan-id command in EXEC mode.
Command example:
switch# zoneset distribute vsan 2
Zoneset distribution initiated. Check zone status

Note This command only distributes the full zone set information. The command does not save
the information to the startup configuration. You must explicitly issue the copy running start
command to save the full zone set information to the startup configuration.

The zoneset distribute vsan vsan-id command is supported in interoperability 2 and

interoperability 3 modes, but not in interoperability 1 mode.
The zoneset distribute vsan command (run in EXEC mode) does the following:
 Distributes the full zone set immediately.
 Does not distribute the full zone set information along with the active zone set during
activation, deactivation, or merge process.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-155

The zoneset distribute full vsan command (run in configuration mode) does the following:
 Does not distribute the full zone set immediately.
 Remembers to distribute the full zone set information along with the active zone set during
activation, deactivation, and merge processes.

3-156 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Merging Zones Without Disruption
This topic describes the procedure for merging zones without causing disruption to the switch.

• Zone servers interact to VSAN 200 VSAN 200

maintain consistent zoning Set200 Set200

Zone1 Zone1
information.  Host1  Host1
 Disk1  Disk11
• Change and merge protocols VSAN 201 VSAN 201
are used to synchronize the Set201 Set201
zoning database. Zone1 Zone1
 Host2  Host2
• Change protocol is used to  Disk2  Disk2

communicate changes between

zone servers. VSAN 200
VSAN 200 Traffic VSAN 200
• Merge protocol is used to merge Set200 Isolated Set200
databases to create a new Zone1 Zone1
 Host1  Host1
zoning database.  Disk1  Disk11

• An ISL between two switches VSAN 201 VSAN 201

becomes isolated if members of Set201 Set201

Zone1 VSAN 201 Zone1
a zone between the switches  Host2 Traffic OK  Host2
 Disk2  Disk2
are not identical.

The zone server is a distributed feature on all switches. The high-availability feature for zoning
permits stateful restartability and switchability. Activating a zone set will automatically replace
the currently active zone set. A copy of the active zone set is maintained to allow modifications
to the original zone set.

Note Always carefully compare the active zone set with the proposed changes to the zone set
before making changes.

Adding or deleting a member to a zone or adding or deleting a zone to a zone set can be
accomplished nondisruptively. This ability happens by modifying the stored copy of the active
zone set and then reactivating the zone set. There is no need to deactivate and then reactivate a
zone set, as doing so is disruptive.
When a zone set is activated and the fabric is configured for a full zone database update, then
the full database will be sent to all switches. Otherwise, only the active zone set will be sent to
all switches.

Note If you are in a homogeneous Cisco MDS fabric and you plan to manage zones from all
switches in the fabric, full zone database updates are recommended. More commonly, a
single switch will be used to administer zones, in which case only propagating the active
zone set is appropriate.

A zone merge occurs when an ISL is initialized. If the ISL is an Enhanced ISL (EISL), then
zone merge is done per VSAN. The merge process stops when the zoning database is
synchronized.

© 2013 Cisco Systems, Inc. Building a SAN Fabric 3-157

Each zone server interacts with other zone servers in the fabric to maintain consistent zoning
information across the fabric.
Change and merge protocols are used to synchronize the database among zone servers. The
change protocol is used to communicate any changes in the database. The merge protocol is
used whenever an ISL between two switches becomes operational. The merge protocol will
merge the two databases to create a new zoning database. If the members of a zone between
two switches are not identical, the ISL becomes isolated.

If a zone merge failure occurs, you can conduct a merge analysis by using Cisco DCNM-SAN
Client. To perform a Zone Merge Analysis from the Cisco DCNM-SAN Client, use the
following procedure:
Step 1 Choose Zone > Merge Analysis from the Cisco DCNM-SAN Client Zone menu.
The Zone Merge Analysis window is displayed.
Step 2 Choose the first switch to analyze from the Check Switch 1 drop-down list.
Step 3 Choose the second switch to analyze from the And Switch 2 drop-down list.
Step 4 Enter the VSAN ID where the zone set merge failure occurred in the For Active
Zoneset Merge Problems in VSAN ID field.
Step 5 Click the Analyze button to analyze the zone merge.
Step 6 Click the Clear button to clear the analysis data from the Zone Merge Analysis
window. If you click the Analyze button without clicking the Clear button, the new
Zone Merge Analysis data is displayed below the old data.
In this example, a Zone Merge Analysis was conducted between switch MDS-1 and switch
MDS-2 on VSAN 1. The result is that a VSAN 1 zone set (zoneset1) merge will succeed. The
success occurs because all of the necessary configuration criteria for a zone merge between the
two switches have been met for that VSAN.
If a zone merge fails, you can also resolve the merger from the CLI by issuing a show interface
command for the E Port to determine the isolated VSAN and by comparing active zone sets on
both switches (show zoneset active). If you prefer the database for one switch over the other,
issue a zone merge interface fc1/1 {import | export} [vsan x] command.
The import option of the command will overwrite the configuration of the local switch with the
configuration of the remote switch. The export command would overwrite the zoning
configuration of the remote switch with the zoning configuration of the local switch.

Recovering from Zone Merge Failures
This topic describes the procedure that is used to recover from a failure of a zone merge.

• When merging fabrics, TE and E Ports may become isolated when the
active zone set databases differ between the two switches or fabrics.
• You can recover from isolation using one of three options:
1. Import the active zone set database from the neighboring switch and replace
the current configured zone set. Then activate the corrected zone set.
2. Export the current database to the neighboring switch.
3. Manually resolve the conflict by editing the full zone set, activating the
corrected zone set, and then bringing up the link.

1
Switch 1 Switch 2
VSAN 200 VSAN 200
Set200 fc1/3 Set200
Zone1 X Zone1
3
 Host1  Host1 Host2
 Disk1  Disk11

The MDS 9000 Series switch provides a facility to correct a merge failure. The correction
occurs either by importing the database of an adjacent switch or by exporting its database to the
adjacent switch. This capability avoids the need to manually edit and fix the configuration at
either of the switches. Use this capability with caution, because this function affects all devices
within the configured zones.
If a zone merge fails, you can recover from isolation using one of three options:
1. Import the active zone set database of the neighboring switch and replace the current active
zone set.

2. Export the current database to the neighboring switch.

3. Manually resolve the conflict by editing the full zone set, activating the corrected zone set,
and then bringing up the link.

The following commands are used to resolve a failed zone merge by importing or exporting an
active zone set:
zoneset import interface interface-id vsan vsan-id
zoneset export vsan vsan-id

3-160 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Import the zone set from the adjacent switch connected through the
fc1/3 interface for VSAN 200.

1 switch# zoneset import interface fc1/3 vsan 200

switch# zoneset activate vsan 200

• Export the zone set to the adjacent switch connected through VSAN
200.
2 switch# zoneset export vsan 200

1
Switch 1 Switch 2
VSAN 200 VSAN 200
Set200 Set200
fc1/3
Zone1 X Zone1
 Host1  Host1
 Disk1  Disk11

An E Port is segmented (isolation due to zone merge failure) if the following conditions are
true:
 The active zone sets on the two switches differ from each other in terms of zone
membership (provided there are zones at either side with identical names).
 The active zone set on both switches contains a zone with the same name but with different
zone members.
To resolve the link isolation because of a failed zone merge, follow these steps using the CLI:
Step 1 Use the show interface command to confirm that the port is isolated because of a
zone merge failure:
switch# show interface fc1/3
Fc1/3 is down (Isolation due to zone merge failure)
Hardware is Fibre Channel, WWN is 20:4e:00:05:30:00:63:9e
vsan is 200
Beacon is turned off
40 frames input, 1056 bytes, 0 discards
0 runts, 0 jabber, 0 too long, 0 too short
0 input errors, 0 CRC, 3 invalid transmission words
0 address id, 0 delimiter
0 EOF abort, 0 fragmented, 0 unknown class
79 frames output, 1234 bytes, 16777216 discards
Received 23 OLS, 14 LRR, 13 NOS, 39 loop inits
Transmitted 50 OLS, 16 LRR, 21 NOS, 25 loop inits
Step 2 Verify the zoning information, using the following commands on each switch:
show zone vsan vsan-id
show zoneset vsan vsan-id

Step 3 You can use two different approaches to resolve a zone merge failure by overwriting
the zoning configuration of one switch with the configuration of the other switch.
This overwrite can be done with either of the following commands:
zoneset import interface interface-id vsan vsan-id
zoneset export vsan vsan-id
The import option of the command overwrites the active zone set of the local switch with the
zone set of the remote switch. The export option overwrites the active zone set of the remote
switch with the zone set of the local switch.

Note If the zoning databases between the two switches are overwritten, you cannot use the
import option. To work around this limitation, you can manually change the content of the
zone database on either of the switches. After that action, you can issue a shutdown or no
shutdown command sequence on the isolated port.

If the isolation is specific to one VSAN and not on an E Port, the correct way to issue the cycle
up or down is to remove the VSAN from the list of allowed VSANs on that trunk port and
reinsert the VSAN.

3-162 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Initiate the import or export of an active zone set from Cisco DCNM-SAN
Client by choosing Zone > Merge Fail Recovery.

Importing from one switch and exporting from another switch can lead to isolation again. You
can import active zone sets (do a merge fail recovery) if the cause of an ISL failure is a zone
merge failure. To import an active zone set, use the following procedure:
Step 1 From Cisco DCNM-SAN Client, choose Zone > Merge Fail Recovery. You will
see the Zone Merge Failure Recovery dialog box.
Step 2 Click the Import Zoneset radio button.
Step 3 From the drop-down list, choose the switch from which to import the zone set
information.
Step 4 From the drop-down list, choose the VSAN from which to import the zone set
information.
Step 5 Choose the interface to use for the import process.
Step 6 Click the OK button to import the active zone set, or click the Close button to close
the dialog box without importing the active zone set.

Exporting Active Zone Sets

You can export active zone sets (do a merge fail recovery) if the cause of an ISL failure is a
zone merge fail. To export an active zone set, use the following procedure:
Step 1 From Cisco DCNM-SAN Client, choose Zone > Merge Fail Recovery. You will
see the Zone Merge Failure Recovery dialog box.
Step 2 Click the Export Zoneset radio button.
Step 3 From the drop-down list, choose the switch that you want to receive the zone set
information.
Step 4 From the drop-down list, choose the VSAN to receive the zone set information.
Step 5 Choose the interface to use for the export process.

Step 6 Click the OK button to export the active zone set, or click the Close button to close
the dialog box without exporting the active zone set.

3-164 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Managing Zone Sets
This topic describes the procedure for backing up the zone configuration to a workstation.

• Use the Backup option from the File menu to create a backup of all
VSAN zones from Cisco DCNM-SAN Client.
> Edit Local Full Zone Database

You can back up the zone configuration to a workstation by using TFTP. This zone backup file
can then be used to restore the zone configuration on a switch. Restoring the zone configuration
overwrites any existing zone configuration on a switch.
To back up the full zone configuration using Cisco DCNM-SAN Client, use the following
procedure:
Step 1 From Cisco DCNM-SAN Client, choose Zone > Edit Local Full Zone Database,
or right-click a VSAN folder in the Logical tab and choose Edit Local Full Zone
Database from the pop-up menu.
Step 2 If you chose Zone > Edit Local Full Zone Database, then you see the Select
VSAN dialog box. Choose the VSAN.
Step 3 Click the OK button. You should see the Edit Local Full Zone Database window for
the VSAN you selected.
Step 4 Choose File > Backup to back up the existing zone configuration to a workstation
using TFTP.

• Choose Restore from the File menu to restore a backup of all VSAN
zones from Cisco DCNM-SAN Client.
> Edit Local Full Zone Database

To restore the full zone configuration using Cisco DCNM-SAN Client, use the following
procedure:
Step 1 From Cisco DCNM-SAN Client, choose Zone > Edit Local Full Zone Database,
or right-click a VSAN folder in the Logical tab and choose Edit Local Full Zone
Database from the pop-up menu.
Step 2 If you chose Zone > Edit Local Full Zone Database, then you see the Select
VSAN dialog box. Choose the VSAN.
Step 3 Click the OK button. You should see the Edit Local Full Zone Database window for
the VSAN you selected.
Step 4 Choose File > Restore to restore a saved zone configuration. You can optionally
edit this configuration before restoring the configuration to the switch.

3-166 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Use the CLI zoneset rename command to rename a zone set from
configuration mode.
- zoneset rename current-name new-name vsan vsan-id

switch(config)# zoneset rename ZoneSet1 Set200 vsan 200

• Rename zones and zone sets from Cisco DCNM-SAN Client.

> Edit Local Full Zone Database:

To rename a zone set, zone, or Fibre Channel alias, use these commands:
switch# config
switch(config)# zoneset rename oldname newname
switch(config)# zone rename oldname newname
switch(config)# fcalias rename oldname newname

• You can clone a zone, zone set, or zone attribute group. For example,
you can create a special zone set that is activated during backup
windows.
• Use the zoneset clone command to clone a zone set.

switch(config)# zoneset clone Set200 clone200 vsan 200

To clone a zone or zone set from the Edit Local Full Zone Database window, use the
following procedure:
Step 1 Choose the Zones or Zonesets folder, right-click the folder for the zone or zone set
that you want to clone, and choose Clone from the pop-up menu.
Step 2 Enter the name of the cloned zone or zone set. By default, the dialog box displays
the selected zone name. The new zone name prepends the original zone name with
Cloned (for example, ClonedZone1) and selects the read-only zone state to match
the cloned zone.
Step 3 Click the OK button to add the cloned zone to the zone database.
You can clone zone sets and zones if you want to create multiple zone set configurations. For
example, you can use this action if you need to create a special zone set that is activated during
backup windows.

3-168 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• You can copy an active zone set to the full zone set or to a remote
location (using FTP, SCP, SFTP, or TFTP).
• This ability can be helpful if the full zone set is lost.
- You cannot edit the active zone set directly.
- If a switch needs to be replaced and the full zone set is lost, you can copy the
active zone set to the full zone set using the following command:

switch# zone copy active-zoneset full-zoneset vsan 200

WARNING: This command may overwrite common zones in the full zoneset
Please enter yes to proceed.(y/n) [n]? y

Zone sets can be copied to allow for editing of the zone set without alteration of the active zone
set. Copy the active zone set to either the full zone set or to a remote location before making
edits. The full zone set must exist and be propagated for this method to succeed.

Note Use care when using this method to ensure that you do not overwrite existing common
zones in the full zone set.

This feature may be helpful if the full zone set is lost. Because you cannot edit the active zone
set directly, you cannot change zone configurations if the full zone set is lost. To fix this
problem, you can copy the active zone set to the full zone set using the zone copy command:
switch# zone copy active-zoneset full-zoneset vsan 200

• Use the Copy option from Cisco DCNM-SAN Client.
> Edit Local Full Zone Database

You can recover a database by copying the active zone database or the full zone database.
To copy a zone set, use the following procedure:
Step 1 From Cisco DCNM-SAN Client, choose a VSAN and right-click to select Edit
Local Full Zone Database.
Step 2 Highlight the zone set to be copied, right-click, and choose Copy.
Step 3 Enter the tag and click the Prepend or Append radio button.
Step 4 Click the OK button to create the copy.

3-170 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• In Step 1 of 5, enter the source switch and destination VSAN details.

To use the Zone Migration wizard to migrate a non-MDS database, use the following
procedure:
Step 1 From Cisco DCNM-SAN Client, choose Zone > Migrate Non-MDS Database.
You should see the Zone Migration wizard.
Step 2 Enter the IP address, login ID, and password of the Brocade or McData switch from
which you will be migrating the zone database.
Step 3 Choose the VSAN ID for the destination of the migrated zone database and the
switch platform from which the zone database will be migrated. The migration tool
requires IP connectivity from both the Cisco DCNM-SAN Client management
workstation and the Cisco MDS that will receive the zone database copy.
Step 4 Click the Next button.
Step 5 Translate port-based zones to the new Cisco MDS domain and port offset
information by choosing the drop-down menus in the To Domain and Port Offset
fields.
Step 6 Click the Next button.
Step 7 Use the Resolve Zone Members dialog to resolve zone membership as either fWWN
or pWWN. Click the Next button.
Step 8 Review the script that will be run on the Cisco MDS to migrate the zone database
information. Optionally, select a location to which to save the script as a text file, or
make any necessary edits directly to the displayed script.
Step 9 Click the Next button.
Step 10 Choose the Cisco MDS switch to which the zone database will migrate.
Step 11 Click Finish to apply the new zone configuration.

Enhanced Zoning
This topic describes how enhanced zones allow all configurations to be performed in a single
configuration session.

• Enhanced zoning provides the following additional features:

- VSAN scope
- Provides implicit full zone set distribution
- Distributes default zone setting
- Session locking
- Stages modifications until the modifications are explicitly committed or aborted
- Based on FC-GS-4 and FC-SW-3 standards
- Reduced database size
- Fabric-wide zone policy enforcement
- Enhanced error reporting

Enhanced zoning is compliant with the FC-GS-4 and FC-SW-3 standards specifications. These
standards support basic zoning as well as enhanced zoning functionalities.
With basic zoning, two or more administrators can make simultaneous configuration changes.
Upon activation, one administrator can overwrite the changes of another administrator. With
enhanced zoning, all configurations are performed within a single configuration session. When
a session begins, the switch locks the entire fabric to implement the change, ensuring
consistency within the fabric. In basic zoning, even with distribute full enabled, it is possible
that the full zone database is different between switches. In enhanced zoning, it is not possible
to change only the local full zoning database.
In basic zoning, if a zone is a member of multiple zone sets, an instance of the zone is created
in each zone set. With enhanced zoning, once a zone is defined, zone sets use references to the
zone as required. This process results in a reduced database payload size. The savings are more
pronounced with larger databases.
In basic zoning, the default zone policy of permit or deny is defined per switch. To ensure
proper fabric operation, all switches in the fabric must have the same default zone setting.
Enhanced zoning enforces consistent policies for the default zone and the default merge control
settings throughout the fabric. This fabric-wide policy enforcement reduces the potential for
zoning problems.

3-172 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• All switches in the VSAN must support enhanced zoning.
• Enable enhanced mode zoning on a VSAN from any switch.
switch(config)# zone mode enhanced vsan 20

- Fabric-wide lock is acquired on the specified VSAN.

- Active and full zoning databases are distributed.
- Zoning policies are distributed.
- Lock is released.
- All switches in the VSAN move to enhanced zoning.

• Verify that the operation was successful.

switch# show zone status vsan 20

By default, the enhanced zoning feature is disabled in all switches in the Cisco MDS 9000
Series. To enable enhanced zoning on a VSAN, you should perform the following steps:
Step 1 Verify that all switches in the fabric are capable of working in enhanced mode.
Step 2 If one or more switches are not capable of working in enhanced mode, then your
request to move to enhanced mode is rejected.

Note It is permissible to have one VSAN in basic mode and another VSAN in enhanced mode.

Use the zone mode enhanced vsan command to set the operation mode to enhanced zoning.
By doing so, you will automatically start a session, acquire a fabric-wide lock, distribute the
active and full zoning database using the enhanced zoning data structures, distribute zoning
policies, and then release the lock. All switches in the VSAN then move to the enhanced zoning
mode.
Use the show zone status command to verify that the operation was successful. After moving
from basic zoning to enhanced zoning, Cisco recommends that you save the running
configuration so that the setting will persist if the switch resets.
The Fibre Channel standards do not allow you to move back to basic zoning. However, MDS
9000 Series switches do allow this capability, which allows you to downgrade or upgrade to
other Cisco Nexus Operating System (NX-OS) releases.
To change to the basic zoning mode from the enhanced mode, use the following procedure:
Step 1 Verify that the active and full zone set do not contain any configuration that is
specific to the enhanced zoning mode (such as an attribute group). If such
configurations exist, delete them before proceeding. If you do not delete the existing
configuration, the Cisco NX-OS Software automatically removes them.
Step 2 Set the operation mode to basic zoning mode. By doing so, you will automatically
start a session, acquire a fabric-wide lock, distribute the zoning information using
the basic zoning data structure, apply the configuration changes, and release the lock

from all switches in the fabric. All switches in the fabric then move to basic zoning
mode.

3-174 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Cisco DCNM-SAN Client also supports enhanced zoning.
- Enable or disable enhanced zoning on a VSAN.
- Edit full zone database and commit changes.
- View enhanced zone and zone set status.

On the Enhanced tab, select enhanced zoning.

Cisco DCNM-SAN Client provides a graphical interface that allows you to perform enhanced
zoning operations, including the ability to do the following:
 Manage enhanced zones: Enable and disable enhanced zoning on a VSAN.
 Manage zone database: Edit the full zone database and commit changes.
 Monitor zones: View enhanced zone and zone set status.

To configure enhanced zoning from Cisco DCNM-SAN Client, perform one or both of the
following:
 Option 1: Choose a zone set in the Logical Domains pane, and then perform enhanced
zoning operations in the Enhanced pane.
 Option 2: Edit the full zone database as you would for basic zoning. When you activate a
new zone set, Cisco DCNM-SAN Client will automatically perform the required commit
operation.

• Use show commands for enhanced zoning.
switch# show zone|zoneset pending [vsan n]
switch# show zone|zoneset pending-diff [vsan n]
switch# show zone-attribute-group [vsan n]

• Additional zone status is displayed.

switch# show zone status vsan 10
VSAN: 10 default-zone: deny distribute: full Interop: default
mode: enhanced merge-control: allow
session: none
hard-zoning: enabled broadcast: enabled
Default zone:
qos: none broadcast: disabled ronly: disabled
Full Zoning Database :
DB size: 260 bytes
Zonesets:2 Zones:2 Aliases: 0 Attribute-groups: 1
Active Zoning Database :
DB size: 76 bytes
Name: zoneset1 Zonesets:1 Zones:1
Status: Set zoning mode complete at 10:26:08 UTC Mar 5 2010

To display the pending zone|zoneset information for the VSAN to be committed, use the
following command:
switch# show zone|zoneset pending vsan n
To display the difference between the pending and effective zone information for the specified
VSAN, use the following command:
switch# show zone pending-diff vsan n
The following is an example:
zone name testzone vsan 2
- member pwwn 21:00:00:20:37:4b:00:a2
+ member pwwn 21:00:00:20:37:60:43:0c
In enhanced zone mode, you can directly configure attributes using groups, adding the attribute
to the attribute-group object and attaching the object to a zone in a specific VSAN.
To display the pending zone information for the VSAN to be committed, use the following
command:
switch# show zone-attribute-group pending vsan n
The following is an example:
zone-attribute-group name testattgp vsan 2
read-only
Broadcast
qos priority high

3-176 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Modifying the Enhanced Zone Database
This topic describes the procedure for modifying the enhanced zone database.

• All modifications to the zone database must be committed before the

modifications take effect (including zone set activation).

switch(config)# zone commit vsan 20

Session Database: Effective Database:

Modifications Made on Commit Fabric-wide Zoning
a Copy of Database Database

• Use the force option to override a user lock.

- Requires appropriate permission (role).
- Must be done from the switch where the lock was acquired.
switch(config)# zone commit vsan 20 force

• Use this command to discard pending changes and release a lock.

switch(config)# no zone commit vsan n [force]

Modifications to the zone database are done within a session. A session is created at the time of
the first successful configuration command. On creation of a session, a copy of the zone
database is created. Any changes that are done within the session are performed on this copy of
the zoning database. These changes in the copy zoning database are not applied to the effective
zoning database until you commit the changes. When you have performed the required changes
to the enhanced zone configurations, apply the changes by issuing a commit command. Once
you have applied the changes, the session is closed. To commit changes to the zoning database
in a VSAN, use the zone commit vsan command.
The session is tied to the user that obtained the lock. If the fabric is locked by one user, and if
for some reason the lock has to be cleared by another user, use the force option. To use this
option, you must have permission. In other words, you must have the appropriate role to clear
the lock in this switch, and the action must be done from the switch from where the session was
originally created.
To discard pending changes and release the lock, use the no zone commit vsan command. If,
for some reason, the lock that is held by another user has to be cleared forcefully, then issue a
no zone commit vsan command with the force option. The following is an example:
switch(config)# no zone commit vsan 20 force

• Fabric merge considerations for enhanced zoning include the following:
- If fabric-wide merge control setting is set to restrict, then zone sets must be
identical or ISLs are isolated.
- If fabric-wide merge control setting is set to allow (which is the default), the
following rules apply:
Local Database Adjacent Database Merge Status Merge Results
Databases contain zone sets with the same name, Successful Union of local and
but different zones, aliases, and attribute groups adjacent databases
Databases contain zone, Fibre Channel alias, or Failed ISLs are located
zone attribute group object with same name, but
different members
Empty Contains data Successful Adjacent database
information populates
local database
Contains data Empty Successful Local database
information populates
adjacent database

When an ISL is brought up between two fabrics (VSANs), their zone sets will attempt to
merge. The merge behavior depends on the fabric-wide merge control setting, as follows:
 Restrict: If the two databases are not identical, the ISLs between the switches are isolated.
This setting can be used as a security precaution to ensure that fabrics do not
unintentionally merge. Note that basic zoning does not provide this fundamental security
mechanism, making this choice inherently less secure.
 Allow: The two databases are merged using the merge rules that are specified in the table
that is shown in the figure. This parameter is the default merge control setting.

In enhanced zoning mode, the active zone set does not have a name in interoperability mode 1.
Zone set names are present only for full zone sets.

The Merge Process

The zone merge process proceeds as follows:
1. The software compares the protocol versions. If the protocol versions differ, then the ISL is
isolated.
2. If the protocol versions are the same, then the zone policies are compared. If the zone
policies differ, then the ISL is isolated.
3. If the zone merge options are the same, then the comparison is implemented based on the
merge control setting:

— Restrict: If this setting is used, the active zone set and the full zone set must be
identical. Otherwise, the link is isolated.
— Allow: If this setting is used, then the merge rules are used to perform the merge
(see the table in the figure).
To configure a restricted merge control setting for a VSAN, use the following command:
switch(config)# zone merge-control restrict vsan n

3-178 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
To configure the default allow merge control setting for a VSAN, use the following command:
switch(config)# no zone merge-control restrict vsan n
After entering either of these commands, the changes must be committed to take effect.

Smart Zoning
This topic defines the advantages of the Smart Zoning feature for SANs with many zones.

• Zoning provides access control

security for Fibre Channel VSAN 2
devices. Disk2

Zone A Disk3
- Standard zoning model just has Host1 Disk1
“members.” Zone set 1
- Any member can talk to any other Disk4
Host2
member. Zone B
- Each pair consumes an ACL entry
in TCAM.
Zone A
- The result is n*(n-1) entries. Valid ACL in TCAM Created but unused
- The admin pays a price for Host1>Disk1 Disk1>Disk2
internal inefficiency.
Host1>Disk2 Disk1>Disk3
Host1>Disk3 Disk2>Disk3
Disk1>Host1 Disk2>Disk1
Disk2>Host1 Disk3>Disk1
Disk3>Host1 Disk3>DIsk2

The traditional zoning method allows each device in a zone to communicate with every other
device in the zone. The administrator is required to manage the individual zones according to
the zone configuration guidelines.
Any member can talk to any other member. This ability means every device must have an
access control list (ACL) entry in the ternary content addressable memory (TCAM) to talk to
all other devices in the zone. The result is that many entries are unused, which takes up switch
resources. This result is inefficient and could become problematic in great numbers.

3-180 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• One-to-one zoning One-to-One, Single Initiator Zoning
- 8 initiators, 4 targets, 32 zones FC FC FC FC FC FC FC FC
FC FC FC FC FC FC FC FC
- 64 ACL TCAMs (each zone 2
x1=2)
- A benefit is no unnecessary ACL
TCAMs FC FC FC FC

- Negatives are a high number of

zones, may reach limit, hard to
manage
One-to-Many, Single Initiator Zoning
• One-to-many zoning
FC FC FC FC FC FC FC FC
- 8 initiators, 1 target group, 8 FC FC FC FC FC FC FC FC
zones
- 160 ACL TCAMs (Each zone 5x4
=20 ) FC FC FC FC

- Benefits are low number of

zones, easy to manage
- A negative is a high number of With one-to-many zoning, large deployments reach
unnecessary ACL TCAMs maximum ACLs in TCAM early because of
unnecessary pairs, and therefore resort to 1:1 zoning.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS20 v2.0—3-44

Standard approaches to zoning configuration include one-to-many zoning and one-to-one

zoning.

One-to-Many Zoning
This type of zoning means that there are multiple initiators, multiple targets, or both grouped in
the same zone. This type of zoning is the quickest way to set up zoning, but has many negative
consequences. In a situation with multiple initiators in the same zone, a disruption with one
initiator port can cause a broadcast of State Change Notifications to all interfaces in the zone.
This situation can be very disruptive to other initiators in the zone. Another complication of
multiple initiators in the same zone is the access control of the same storage at the same time.
For these reasons, a traditional best practice is to perform single initiator zoning.
Multiple targets within the same zone are not as disruptive to each other, but do present an
abundance of unnecessary ACL entries in the TCAM.

One-to-One Zoning
Due to the above negatives of one-to-many zoning, administrators may choose one-to-one
zoning with only two ACL entries in the TCAM per zone. The alternative is single initiator (or
single-init, single-target) zones, but in large environments, the creation of all these separate
zones causes a big operational overhead.

• Smart Zoning reduces the number of ACL TCAMs.
- Specifically target-target and initiator-initiator
• Smart Zoning enables users to select the host, target, or both as the end
device type.
• Allows storage admins to create larger zones while still keeping premise
of single initiator and single target.
• Dramatic reduction in SAN administrative time for zoning.
• Utility to convert existing zone or zone set to Smart Zoning.

One-to-One Zoning One-to-Many Zoning Smart Zoning

Zones ACLs Zones ACLs Zones ACLs
32 64 8 160 8 64
Too High OK OK Too High OK OK
*Assume 8 Initiators , 4 Targets, Each zone n*(n-1)
Single Initiator Zoning

Smart Zoning combines the benefits of one-to-one and one-to-many zoning. Smart Zoning
offers the following:
 Simplicity of operational management with a single zone for all initiators and targets of an
application or cluster.
 No wasted switch resources as with two member zones.
 The benefit is particularly apparent in cloud data center and big data deployments. In those
environments you have a lot of initiators (ESX nodes in a vCluster) that access the same
targets (data stores). If you need to add a new ESX to your cluster, you simply add one
member to your zone and reactivate. This feature is integrated with Cisco Unified
Computing System (UCS).
 Smart Zoning supports zoning among more devices. Smart Zoning provides this support by
reducing the number of programmed zoning pairs. Pairs are created “smartly” by
considering device type information without increasing the size of the zone set.
 Smart Zoning enables user to select the host, target, or both as the end device type.
 Smart Zoning supports zoning by pWWN, FCID, FC alias, and Device alias.

Smart Zoning Scales Better

By reducing the number of zones that are required, Smart Zoning extends the amount of time
before extremely large fabrics need to worry about encountering software limits on the number
of zones in a fabric. This extension is especially valuable. New technologies such as converged
network adapters (CNAs) on the motherboard are reducing the cost of SAN-enabling servers to
nearly zero. This reduction has made SAN penetration of new servers reach nearly 100 percent.
Fabrics with fewer zones are much easier to maintain, because administrators can find a zone
that needs modification much more quickly. Because zone names can now correspond to
applications, clusters, and so on, the likelihood of administrative mistakes that could affect
application availability is reduced.

3-182 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Enable Smart Zoning.
switch#(config)# zone smart-zone enable vsan 10
Smart Zoning distribution initiated. Check zone status.
switch#(config)# zone convert smart-zone vsan 10
switch#(config)# zoneset activate name ZS_VSAN10 vsan 10
…
switch(config)# system default zone smart-zone enable

• Configure zone members.

switch(config)# zone name SmartZone vsan 10
switch(config-zone)#member pwwn 10:00:00:00:c9:2f:02:db init
switch(config-zone)#member pwwn 21:00:00:04:cf:db:3e:a7 target
switch(config-zone)#member pwwn 21:00:00:20:37:15:dc:02 target

Enable Smart Zoning

# conf t
(config)# zone smart-zone enable vsan 10
Smart Zoning distribution initiated. Check zone status.
(config)# zone convert smart-zone vsan 10
(config)# zoneset activate name ZS_VSAN10 vsan 10

Clear Smart Zoning

# conf t
(config)# clear zone smart-zoning vsan 10
WARNING: This command will clear Smart Zoning configs from the
specified
zone/zoneset/fcalias/vsan. Do you want to continue? (y/n) [n]
y
(config)# no zone smart-zoning enable vsan 10
WARNING: Disabling Smart Zoning feature may result in TCAM
space to exceed
and zone to switch to soft zoning in case the current active
database is more
than specified limits. Do you want to continue? (y/n) [n] y
Smart Zoning distribution initiated. check zone status
(config)#

Configure Zone Members

Once enabled, you add the keywords init, target, and both to the member entry. Smart Zoning
only programs TCAMs with entries that connect initiator and target.
You do not need to manually edit all zones. There is a conversion utility that uses the FCNS
data to add these keywords.
For example, both is used in replication scenarios where storage copy ports act as both init and
target when the ports replicate data. However, you could also use a two-member zone for that
function.

• Use the following commands to fetch the device-type information from
the name server and add that information to the member.
switch(config)# zone convert smart-zoning fcalias name alias-name vsan vsan no
switch(config)# zone convert Smart Zoning zone name zone name vsan vsan no
switch(config)# zone convert smart-zoning zoneset name zonesetname vsan vsan
no
switch(config)# zone convert smart-zoning vsan vsan no

• Use the following commands to configure the device types for zone
members.
switch(config-zoneset zone)# member device-alias name both
switch(config-zoneset-zone)# member pwwn number target
switch(config-zoneset-zone)# member fcid number

Converting Zones Automatically to Smart Zoning

This command fetches the device type information from the name server for the FC alias
members.
switch(config)# zone convert smart-zoning fcalias name alias-
name vsan vsan#
This command fetches the device type information from the name server for the zone members.
switch(config)# zone convert Smart Zoning zone name zone name
vsan vsan#
This command fetches the device type information from the name server for all the zones and
FC alias members in the specified zone set.
switch(config)# zone convert smart-zoning zoneset name
zonesetname vsan vsan#
This command fetches the device type information from the name server for all the zones and
FC alias members or all the zone sets present in the VSAN.
switch(config)# zone convert smart-zoning vsan vsan#
These commands are performed at zone, zone set, FC alias, and VSAN levels.

Commands to Configure Device Types for Zone Members

This command configures the device type for the device-alias member as both. For every
supported member-type, init, target, and both are supported.
switch(config-zoneset zone)# member device-alias name both
This command configures the device type for the pWWN member as target. For every
supported member-type, init, target, and both are supported.
switch(config-zoneset-zone)# member pwwn number target
This command configures the device type for the FCID member. There is no specific device
type that is configured. For every supported member-type, init, target, and both are supported.
switch(config-zoneset-zone)# member fcid number

3-184 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Note When there is no specific device type that is configured for a zone member, at the back-end,
zone entries that are generated are created as device type both.

New Zone Existing Zone

Smart Zoning requires all MDS 9000 Series switches in the fabric to use Cisco MDS 9000 NX-
OS Software Release 5.2(6) or later and have the Smart Zoning feature enabled.

Note Smart Zoning can be enabled at VSAN level , but can also be disabled at zone level.

3-186 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Recommended Zoning Practices
This topic describes the guidelines for implementing zones.

• Recommended single-initiator zoning:

- Each HBA should usually have its own zone.
- Use long-established practice, due to security concerns.
- Use to stop hosts from trying to discover other hosts.
• Device aliases should be used to simplify management.
• Use enhanced zoning or manage zones from a single switch.
• Enable Smart Zoning.
• Policy for the default zone should be set to deny.

Follow these guidelines for implementing zones:

 Zoning should always be implemented in a Fibre Channel fabric, if not from a security
perspective, then from the perspective of minimizing loss of data. It is recommended that as
many zones be used as there are host bus adapters (HBAs) that are communicating with
storage. For example, if there are two hosts each, with two HBAs, which are
communicating with three storage devices, it is recommended to use four zones. This type
of zoning is sometimes referred to as single-initiator zoning.
 Single-initiator zoning is a long-established practice. The practice was popularized partly
due to the additional level of security that is provided, but also as mitigation for firmware
bugs in early Fibre Channel devices. Although single-initiator zoning is still considered a
best practice, this practice is not as important as it was in the early years of Fibre Channel.
This practice may not be appropriate for all SAN environments.
 To simplify management, Fibre Channel aliases should be used wherever possible. It is
easier to identify devices with aliases than with WWNs. In general, you should assign
aliases to WWNs.
 Zone administration should generally be confined to a single Fibre Channel switch within a
given fabric. This confinement ensures that activating an incomplete zone set (which might
happen if the full zone set database is not consistent across Fibre Channel switches) cannot
happen.
 Enable Smart Zoning.
 Leave the default zone policy as deny so that devices cannot inadvertently access each
other when placed in the default zone.

Summary
This topic summarizes the key points that were discussed in this lesson.

• A zone consists of multiple zone members (devices). Zone members in

different zones cannot have access to each other. Zone members can
belong to multiple zones.
• Zones grouped together are called a zone set.
- Only one active zone set is allowed per VSAN.
- Zone sets and zones cannot span across multiple VSANs.
• Zones can be configured using device alias pWWN, F Port WWN, FCID,
Fibre Channel alias, domain ID, IP address, or interface. Device aliases
are recommended.
• Default zone membership includes all ports or WWNs that do not have a
specific zone membership. The default zone policy should be to deny all
traffic.
• By default, only the active zone set is distributed when a zone set is
activated. You can distribute full zone sets automatically or manually.

• A zone merge occurs when an ISL is initialized. If the ISL is an EISL, then
zone merge is done per VSAN.
• Zone sets can be imported and exported to resolve zone merge problems.
Zone sets can be backed up to an external server.
• Enhanced zoning ensures that all configurations are performed within a
single configuration session. Before you enable enhanced zoning in a
VSAN, verify that all switches in the VSAN are capable of enhanced mode.
• Modifications to the zone database are done within a session to ensure
consistency.
• The Smart Zoning feature brings advantages to SANs with large numbers of
zones.
• Recommended practices for zoning include the following:
- Single-initiator zoning
- Device aliases
- Enhanced zoning
- Default zone policy of deny
- Enable Smart Zoning

3-188 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
References
For additional information, refer to these resources:
 For information on standard zoning versus Smart Zoning, refer to
http://www.cisco.com/en/US/tech/tk1048/technologies_tech_note09186a0080c15b78.shtml
 For information on how to configure Smart Zone members, refer to
http://www.cisco.com/en/US/tech/tk1048/technologies_tech_note09186a0080c15b78.shtml

3-190 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Summary
This topic summarizes the key points that were discussed in this module.

• Every Cisco MDS 9000 Series Fibre Channel switch contains an FLOGI
database and FCNS database.
• There are many Fibre Channel port types available in a SAN. The switch
interfaces support various port types, depending on the device type that
is connecting.
• Port channels can be used in a physical switch to switch environment to
provide high availability in the multilayered SAN.
• Cisco NPV overcomes the restrictions set on the number of switch
domain IDs allowed per fabric or VSAN.
• VSANs can be used in a physical SAN environment to reduce
underutilized SAN components through SAN consolidation.
• Fibre Channel domains work in VSAN environments and prevent fabric
isolation.
• Distributed device aliases simplify SAN configuration and management
tasks.
• The Smart Zoning feature offers advantages in SANs with many zones.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—3-1

3-192 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Self-Check
Use the questions here to review what you learned in this module. The correct answers and
solutions are found in the Module Self-Check Answer Key.
Q1) What is the maximum number of VSANs per physical fabric? (Source: Configuring
VSANs)
A) 80
B) 256
C) 239
D) 4000
Q2) Which statement about VSAN configuration is correct? (Source: Configuring VSANs)
A) Use VSAN 1 for most production network traffic.
B) Suspend VSAN 1.
C) Isolate each host in a separate VSAN whenever practical.
D) Avoid modifying fabric timers and FSPF timers unless changes are required.
Q3) Choose the option that shows the correct order of the four phases that are associated
with the Fibre Channel domain process. (Source: Managing Domains)
A) fabric reconfiguration, FCID allocation, principal switch selection, domain ID
distribution
B) domain ID distribution, principal switch selection, FCID allocation, fabric
reconfiguration
C) principal switch selection, domain ID distribution, FCID allocation, fabric
reconfiguration
D) fabric reconfiguration, principal switch selection, domain ID distribution,
FCID allocation
Q4) Which type of Fibre Channel frame is sent to other switches following a disruptive
restart of a Fibre Channel domain? (Source: Managing Domains)
A) RCF
B) EFP
C) ESC
D) BF
Q5) Which command configures an interface for N-Port or NL-Port connectivity? (Source:
Configuring Interfaces)
A) (config-if)# switchport mode fx
B) (config-if)# switchport mode fe
C) (config-if)# trunk mode fx
D) (config-if)# switchport mode fl
Q6) Which port type receives a Fibre Channel ID that is assigned by a switch when logging
into the fabric? (Source: Managing Domains)
A) Fx Port
B) FL Port
C) Nx Port
D) TE Port

Q7) Which four statements about fabric zoning change and merge protocols are accurate?
(Choose four.) (Source: Implementing Zoning)
A) Change and merge protocols are used to synchronize the database among zone
servers.
B) If the members of a zone between two switches are identical, the ISL becomes
isolated.
C) Merge failure may be corrected by importing the database of an adjacent
switch.
D) The merge protocol is used whenever an ISL between two switches becomes
operational.
E) The merge protocol will merge the two databases to create a new zoning
database.
Q8) Which three characteristics are benefits of port channels? (Choose three.) (Source:
Configuring Port Channels)
A) load balancing
B) link redundancy
C) increased link transmission rate
D) support across all vendors
E) higher aggregated bandwidth
Q9) What is the maximum number of switches per VSAN? (Source: Configuring VSANs)
A) 40
B) 60
C) 128
D) 239
Q10) The default state of a VSAN is which of the following? (Source: Configuring VSANs)
A) administratively up
B) administratively down
C) suspended
D) active
Q11) What is the purpose of VSAN 4079? (Source: Configuring VSANs)
A) isolated VSAN
B) extended VSAN
C) Exchange Virtual Fabrics Protocol VSAN
D) standard VSAN
Q12) On the Cisco MDS switch, what is the run-time priority for the principal switch?
(Source: Managing Domains)
A) 1
B) 2
C) 128
D) 256
Q13) What is the recommended configuration for domain IDs? (Source: Managing Domains)
A) preferred
B) dynamic
C) static
D) allowed

3-194 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Q14) Which command would you use to perform a disruptive restart of VSAN 10? (Source:
Managing Domains)
A) switch# fcdomain restart disruptive vsan 10
B) switch# fcdomain disruptive restart vsan 10
C) switch(config)# fcdomain disruptive restart vsan 10
D) switch(config)# fcdomain restart disruptive vsan 10
Q15) Which command would you use to purge entries in the FCID database in VSAN 25?
(Source: Managing Domains)
A) switch# purge fcid vsan 25
B) switch(config)# purge fcid vsan 25
C) switch(config)# purge fcdomain vsan 25
D) switch# purge fcdomain fcid vsan 25
Q16) Which statement correctly describes the Distributed Device Alias Services features?
(Source: Configuring Distributed Device Aliases)
A) Device alias features are dependent on VSAN and zoning configurations.
B) Device alias features are independent of VSAN configurations.
C) Device alias features are independent of VSAN and zoning configurations.
D) Device alias features are dependent on zoning configurations.
Q17) You have been using the Fibre Channel alias feature and are migrating to using
distributed device aliases. Which set of commands would you use to import your
current Fibre Channel alias configuration in VSAN 10 to the distributed device alias
database? (Source: Configuring Distributed Device Aliases)
A) switch(config)# device-alias import fcalias vsan 10
switch(config)# device-alias commit
B) switch# device-alias import fcalias vsan 10
switch(config)# device-alias commit
C) switch(config)# device-alias import fcalias vsan 10
switch(config)# device-alias distribute
D) switch(config)# device-alias fcalias import vsan 10
switch(config)# device-alias commit
Q18) What is the maximum number of entries in the global distributed device alias database?
(Source: Configuring Distributed Device Aliases)
A) 2048
B) 4096
C) 8192
D) 16,384
Q19) Which set of commands would enable trunking on interface fc1/7? (Source:
Configuring Interfaces)
A) switch(config)# interface fc1/7
switch(config)# switchport trunk mode on
B) switch(config)# interface fc1/7
switch(config-if)# switchport trunk on
C) switch(config)# interface fc1/7
switch(config-if)# switchport trunk enable
D) switch(config)# interface fc1/7
switch(config-if)# switchport trunk mode on

Q20) Which feature allows HBA port sharing between server partitions or between virtual
machines? (Source: Configuring Interfaces)
A) NPV mode
B) NPIV
C) FL Ports
D) NL Ports
Q21) Which feature allows the fabric to extend beyond the recommended limit of domain
IDs? (Source: Configuring Interfaces)
A) NPV mode
B) NPIV
C) N Port proxy mode
D) VSANs
Q22) What is the default load-balancing method when using port channels? (Source:
Configuring Port Channels)
A) round robin
B) flow-based
C) source FCID
D) exchange-based
Q23) Which two subprotocols does the port channel protocol use? (Choose two.) (Source:
Configuring Port Channels)
A) autocreation protocol
B) creation protocol
C) PAgP
D) LACP
E) bringup protocol
Q24) Which option would you use to put an interface into a port channel using the port
channel protocol? (Source: Configuring Port Channels)
A) switch(config-if)# channel-group up
B) switch(config-if)# channel-group on
C) switch(config-if)# channel-group auto
D) switch(config-if)# channel-group enable
Q25) A TNP Port on an edge switch connects to the _____ on the core switch. (Source:
Configuring Interfaces)
A) F Port
B) E Port
C) TE Port
D) TF Port
Q26) Which command would you use to identify the first operational port in a port channel?
(Source: Configuring Port Channels)
A) switch# show port-channel
B) switch# show interface fc 1/7
C) switch# show port-channel database
D) switch# show database port-channel

3-196 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Q27) Which option is a valid example of zone configuration? (Source: Implementing
Zoning)
A) switch(config)# zone name Zone2 vsan 10
switch(config)# member fcid 58:02:01
switch(config)# member domain-id 2 portnumber 23
B) switch(config)# zone name Zone2 vsan 10
switch(config-zoning)# member fcid 0xce00d1
switch(config-zoning)# member domain-id 2 portnumber
23
C) switch(config)# zone name Zone2 vsan 10
switch(config-zone)# member fcid 0xce00d1
switch(config-zone)# member domain-id 2 portnumber 23
D) switch(config)# zone name Zone2 vsan 10
switch(config-zone)# member fcid 58:02:01
switch(config-zone)# member domain-id 2 portnumber 23
Q28) Which command is valid for activating a zone set in VSAN 10? (Source: Implementing
Zoning)
A) switch(config)# zone name Zoneset1 vsan 10 activate
B) switch# zone name Zoneset1 vsan 10 activate
C) switch(config-zoneset)# zone activate name Zoneset1
vsan 10
D) switch(config)# zone activate name Zoneset1 vsan 10

Module Self-Check Answer Key
Q1) B
Q2) D
Q3) C
Q4) A
Q5) A
Q6) C
Q7) A, C, D, E
Q8) A, B, E
Q9) D
Q10) D
Q11) C
Q12) B
Q13) C
Q14) D
Q15) D
Q16) C
Q17) A
Q18) C
Q19) D
Q20) B
Q21) A
Q22) D
Q23) A, E
Q24) C
Q25) D
Q26) C
Q27) C
Q28) D

Intelligent SAN Fabric Services

Overview
In this module, you will explore intelligent SAN fabric services available in the Cisco MDS
9000 Series switch platforms.

Module Objectives
Upon completing this module, you will be able to describe intelligent SAN fabric services. You
will be able to meet these objectives:
 Implement Cisco Data Mobility Manager
 Explain how to capture and monitor Fibre Channel Protocol data
4-2 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 1

Implementing Cisco MDS Data

Mobility Manager
Overview
Data migration is the process of copying data from an existing storage device to a new storage
device. This lesson discusses Cisco Data Mobility Manager (DMM) for the Cisco MDS 9000
Series switch family and its configuration process.

Objectives
Upon completing this lesson, you will be able to explain how to configure the Cisco DMM
feature. This ability includes being able to meet these objectives:
 Describe the purpose of Cisco DMM
 Describe Cisco DMM SAN topologies
 Describe how to install and configure Cisco DMM software and how to prepare for data
migration
 Describe how to use the Cisco DMM GUI for data migration
Cisco DMM Overview
This topic describes the purpose of the Cisco DMM. Cisco DMM for the Cisco MDS 9000
Series switches provides capabilities and features that simplify data migration and minimize
service disruptions.

• Cisco DMM is a fabric-based, online data

migration solution that transfers block data
nonintrusively across heterogeneous
storage volumes:
- Complete asynchronous write I/O operations
over long distance
- Rate control feature minimizes impact on
host
- No host agents required
- No rewiring required in most cases
- Online Data Migration moves data across
VSANs with no zoning required
- Support for active-active or active-passive
storage access across dual redundant
fabrics
- Integrates transparently into the existing Existing
environment New
Storage
- Supported on the following hardware: Storage
• MDS 9222i Multiservice Modular Switch,
MDS 9250i Multiservice Fabric Switch,
MSM 18/4

The data migration task today typically is complex and manual because of the myriad operating
systems, file systems, application servers, volume management systems, physical devices, and
networks involved. IT departments face challenges in migrating data. Those challenges include
the following:
 Downtime that is incurred
 The need to add data migration software to servers
 The potential for data loss and corruption
 Additional errors from the complexities of heterogeneous environments
 Running out of time before the job is done

4-4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Offers transparent online data migration across heterogeneous storage arrays, so that
neither the host server or storage array need reconfiguration when Cisco DMM is
introduced into the user environment. Additionally, no zoning configuration is required.
The storage administrator can complete the migration task without informing the server,
system, or database administrator that a migration is being planned.
 Cisco DMM moves data over long distances when the copy operation must be completed
asynchronously with write I/O operations.
 Cisco DMM securely erases data from the existing storage so that this step can be
completed before the array leaves the customer data center.
 This product offers centralized management with a GUI and wizard for easy configuration.
Cisco DMM includes capabilities to pace the data migration job, schedule the start and
cutover times, and view the effect on the SAN of the extra traffic that is generated by the
data movement. A configuration wizard simplifies setup and use, and a CLI allows
advanced users to complete their migration tasks using scripts.
 Cisco DMM provides per-server and per-storage array configuration.
 This product delivers large-scale, high-performance migration with rate limiting options.
 There is no rewiring that is required in most cases.
 No host agents are required, which minimizes the effect on the CPU and mitigating
software image management concerns.
 This product is supported on high-performance hardware Cisco MDS 9222i Multiservice
Modular Switch and Cisco MDS 9250i Multiservice Fabric Switch.
 Cisco DMM support active-active or active-passive storage access across dual redundant
fabrics.

• DM job: Defines a set of LUNs to be migrated Production
together.
• DM session: Executes migration and is created
for each LUN to be migrated.
• Production fabric: Carries traffic from server to
existing storage. New storage is not present.
• Migration fabric: Carries traffic between new Cisco
and existing storage. No server is present.
• Fibre Channel redirect: Used by Cisco DMM to VT DMM VI
redirect traffic to a Cisco DMM device.
• Virtual target: Proxy on Cisco DMM module
used by Fibre Channel redirect to redirect traffic
from the server to a virtual target on the MDS
9222i Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch. New Existing
• Virtual initiator: During data migration, the Storage Storage
MDS 9222i Multiservice Modular Switch or MDS
9250i Multiservice Fabric Switch use a virtual
initiator to forward redirected traffic to the Migration
existing storage port and to the new storage.

Cisco DMM performs data migration without any additional layer of virtualization. Cisco
DMM requires only the MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice
Fabric Switch configuration to enable the feature and SAN configuration to access the new
storage array. Cisco DMM can be enabled (when data needs to be migrated) and disabled (after
the migration is completed) without any major SAN or host reconfiguration.
Cisco DMM uses the following concepts and terminology:
 Existing storage: The storage that is currently used by the application server. The data that
are contained in the existing storage will be migrated to the new storage.
 New storage: The storage to which the data will be migrated.
 Logical unit number (LUN): A LUN is a reference to a unit of storage that you can
specify for migration. The LUN is only a unique number in the context of a storage port.
 Data migration session: A data migration session migrates the data from one LUN in the
existing storage to a LUN in the new storage.
 Data migration job: A data migration job defines a set of LUNs to be migrated together.
A data migration session is created for each LUN that is to be migrated. The data migration
job is the main unit of configuration and management. For example, the migration rate and
other attributes are configured for the data migration job. The data migration job (not
individual sessions) can be started or stopped.
 MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch:
An MDS switch that provides intelligent services. The Cisco DMM feature executes on
these switches.
 Peer MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric
Switch: In a dual-fabric topology, a data migration job runs on an MDS 9222i Multiservice
Modular Switch or MDS 9250i Multiservice Fabric Switch in each fabric. The two are
peers and communicate with each other to coordinate the data migration jobs.
 Fibre Channel redirect: Fibre Channel redirect allows on-demand insertion and removal
of MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch

4-6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
intelligent services with minimal disruption to existing traffic. No configuration changes
are required on the server or storage devices. Cisco DMM uses the Fibre Channel redirect
capability to redirect traffic to the SSM or MSM. This redirection is transparent to the host
and storage devices.
 Virtual target: A virtual target is a proxy target address for a storage port. During data
migration, the Fibre Channel redirect feature redirects traffic from the server to a virtual
target on the MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric
Switch.
 Virtual initiator: A virtual initiator is a proxy initiator address for a server host bus
adapter (HBA) port. During data migration, the MDS 9222i Multiservice Modular Switch
or MDS 9250i Multiservice Fabric Switch uses a virtual initiator to forward redirected
traffic to the existing storage port. The MDS 9222i Multiservice Modular Switch or MDS
9250i Multiservice Fabric Switch also uses the virtual initiator to forward data migration
traffic to the new storage.
 Control plane processor: The control plane processor is the main processor in the MDS
9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch. Cisco
DMM runs on the control plane processor.
 Data path processors: The data path processors are a set of resource processors in the
MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch. The
data path processor transfers blocks of data across the switch without impacting the control
plane processor. Cisco DMM uses a virtual initiator on the data path processor for
migrating data.
 Production fabric: The production fabric is a fabric where the host ports and existing
storage ports are connected. The server-to-storage traffic is carried in the production fabric.
The new storage port is not present in the production fabric.
 Migration fabric: The migration fabric is a fabric that is dedicated to data migration or
remote replication. The existing storage and the new storage are connected to the migration
fabric. No server port is present in the migration fabric.

Options Methods
• Server-based: • Method1:
- New SCSI writes are written to both
- LUNs accessed by the selected existing and new storage
server are available for migration
to new storage • Method2:
- Server SCSI Write I/Os are written
• Storage-based: only to existing storage and
- LUNs in a specified storage array changes are marked in the modified
region log before sending a
are available for migration to new response back to the server, then
storage migrated to new storage
• Online: • Method3:
- Existing storage is available to - Data from existing storage to new
server during migration storage is managed by MDS 9222i
Multiservice Modular Switch or
• Offline: MDS 9250i Multiservice Fabric
Switch and modified region log
- Servers must not initiate reads or changes are communicated to the
writes to the existing storage switches to migrate on subsequent
iterations

Cisco DMM supports the following features and capabilities:

 Server-based migration: In server-based migration, the focus is data migration for the
storage that is used by a particular server (or server HBA port). All LUNs accessed by the
selected server are available for migration to new storage.
 Storage-based migration: In storage-based migration, the focus is data migration for
storage that is exposed by a particular storage array (or storage array port). All LUNs in the
specified storage array are available for migration to new storage.
 Online data migration: Cisco DMM is designed to provide online data migration. The
existing storage is available to server applications while the MDS 9222i Multiservice
Modular Switch or MDS 9250i Multiservice Fabric Switch performs the data migration.
During migration, data reads from the server are directed to the existing storage. Cisco
DMM ensures that data writes are processed correctly. For example, if the write is to a
storage segment already migrated, the write is mirrored to the existing and new storage.
 Offline data migration: During offline data migration, servers must not initiate reads or
writes to the existing storage. Any server application using the existing storage must be
quiesced. Offline data migration is faster than online data migration and can be used for
noncritical data applications.

Cisco DMM supports the following methods:

 Method 1: For the section of existing storage LUN whose data is already migrated to a
new storage LUN, any new Small Computer Systems Interface (SCSI) write I/Os from the
server are written to both the existing and new storage LUN before sending a response back
to the server. Method 1 is typically used in local data migration.
 Method 2: SCSI write I/Os from the server to any section of existing storage LUN are
written only to the existing storage LUN. The write I/O changes to the existing storage
LUN are marked in the modified region log before sending a response back to the server.
These changes are then migrated to the new storage LUN in subsequent iterations. Method
2 is typically used in remote data center migration.

4-8 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Method 3: Method 3 is used for data migration in topologies where there is a dedicated
fabric or virtual SAN (VSAN) for the data copy independent of the production fabrics or
VSANs. The migration fabric connects the existing storage to the new storage in a remote
data center. The MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice
Fabric Switch in the production fabric is responsible for managing SCSI write I/Os from
the server to the existing storage LUNs. The write I/O changes to the existing storage LUN
are marked in the modified region log before sending a response back to the server. The
data copy from the existing storage LUN to the new storage LUN is managed by the MDS
9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch in the
migration fabric or VSAN. The modified region log changes are communicated to the MDS
9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch in the
migration fabric or VSAN to be applied to the new storage on subsequent copy iterations.

Premigration Postmigration
• Create a plan for the migration. • Finish data migration job.
• Configure new storage. • Verify data integrity between old
• Create the data migration job. and new storage LUNs.
Migration • Delete data migration job.
• Schedule jobs to start • Unmount server from old storage
automatically. LUNs.
• Manually start unscheduled jobs. • Mount server to new storage
• Monitor progress. LUNs.

Premigration Migration Postmigration

Finish (Method 2)

Plan Schedule Configure Create Start Job and Monitor Progress Schedule Verify Destroy
Job Job

A data migration job is typically composed of three major stages, which are premigration,
migration, and postmigration.
In the premigration stage, create a plan for the migration, configure the new storage, and create
the data migration job.
 Plan: Create a plan for the data migration and identify any external dependencies and
activities that need to be scheduled.
 Configure: Configure the new storage device and any other configurations.
 Create the data migration job: Create and configure the data migration job using the
Cisco DMM GUI. A job can be created with or without a schedule.

In the migration stage, jobs that have a configured schedule start automatically. Manually start
unscheduled data migration jobs.
 Start the migration: A data migration job will consist of one or more data migration
sessions. Each data migration session performs the migration of one LUN from the existing
storage to the new storage. During migration, the Cisco DMM feature ensures the integrity
of the storage data by intercepting all traffic from the server and storage ports that are
involved in the migration job. All server-initiated reads and writes are directed to the
existing storage.
 Monitor: During migration, you can monitor the progress of the Cisco DMM job.

In the postmigration stage, reconfigure the server to use the new storage. The configuration
steps vary depending on the server I/O.
 For Method 1 (two-SAN topology): The server write I/Os are mirrored to existing and
new storage in both SANs (typically for local migration).
— Schedule: Schedule a time and personnel to reconfigure the server to use the new
storage and remove references to the existing storage.

4-10 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
— Verify: Optionally, verify the data integrity between the existing and new storage
after the migration has completed. The existing storage must be offline during the
verification.
— Delete: Delete the data migration job.
 For Method 2 (two-SAN topology): The modified region log bitmap tracks server write
I/Os in both SANs (typically for remote data center migration).
— Schedule: Schedule a time and personnel to reconfigure the server to use the new
storage and remove references to the existing storage.
— Finish: To complete the method 2 data migration, click the Finish button to stop
access to the existing storage. The existing storage LUNs are offline for the servers.
Cisco DMM migrates the changed blocks from the existing storage LUNs to the new
storage LUNs for the last time.
— Verify: Optionally, verify the data integrity between existing and new storage after
the finish operation is completed. The existing storage LUNs will be offline during
the verification.
— Delete: Delete the data migration job.
 For Method 3 (three-SAN topology): The modified region log bitmap tracks server write
I/Os in all three SANs (typically when there is a dedicated fabric for migration).
— Schedule: Schedule a time and personnel to reconfigure the server to use the new
storage and remove references to the existing storage.
— Finish: To complete the method 3 data migration, click the Finish button to stop
access to the existing storage. The existing storage LUNs are offline for the servers.
Cisco DMM migrates the changed blocks from the existing storage LUNs to the new
storage LUNs for the last time.
— Verify: Optionally, verify the data integrity between existing and new storage after
the finish operation is completed. The existing storage LUNs will be offline during
the verification.
— Delete: Delete the data migration job.

1. Document the SAN topology for data migration.
2. Design the mapping of source LUNs to destination LUNs.
3. Develop a schedule for the migration job.
4. Ensure that all critical data on the existing storage has been backed
up.
5. Use Cisco MDS 9222i Multiservice Modular Switch switches or Cisco
MDS 9250i Multiservice Fabric Switch modules.
6. Configure existing and new storage.
7. Check the storage ASL status.
8. Configure enclosures.
9. Configure the SAN fabric.

Cisco DMM is designed to minimize the dependency on multiple organizations and to

minimize service disruption. However, data migration is still a complex activity and Cisco
recommends that you create a plan to ensure a smooth data migration.
The following steps should be used to create your data migration plan:
Step 1 Document the SAN topology for the data migration. Identify and obtain any
additional equipment and software licenses.
Step 2 Design the mapping of source LUNs to destination LUNs.
Step 3 Identify the LUNs that need to be migrated and the affected servers.
Step 4 Calculate the storage requirements of the new LUNs.
Step 5 Identify the LUNs on the new storage subsystem.
Step 6 Develop a schedule for the migration job.
Step 7 Identify any required equipment and resources. Availability of external resources,
such as a vendor service representative, may affect your schedule.
Step 8 Identify periods of low user activity and I/O activity to minimize disruption during
the migration. Cisco DMM provides features such as the ability to schedule the
migration during nonpeak hours, or to configure a slow migration rate.
Step 9 Identify any required premigration configuration changes.
Step 10 Plan for one short window in which service is not going to be available during the
server reconfiguration.
Step 11 Ensure that all critical data on the existing storage has a recent backup.

4-12 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Premigration Configuration
Configuring Cisco MDS 9222i Multiservice Modular Switch Switches and Cisco MDS 9000
18/4-Port MSM Modules
Executing on the Cisco MDS 9000 18/4-Port Multiservice Module (MSM), the Cisco DMM
feature supports data migration for storage LUNs that are exposed anywhere on the SAN fabric.
Provision an MDS 9000 18/4-Port MSM on a Cisco MDS 9000 Series switch in each fabric.
Configuring Existing and New Storage Devices
Complete the following configuration tasks for the storage devices. For new storage, connect
the new storage to the SAN. Create LUN maps and LUN masks. Configure access lists for the
new storage. For existing storage, verify that the LUNs are mapped.
Configuring VSANs and Zones
 VSANs: Ensure that the existing storage and new storage port pair in each fabric are
configured in the same VSAN. Also, ensure that for each existing storage port VSAN, there
is at least one new storage port, and that the server port has been configured in the same
VSAN.
 Zones: Optionally, configure zoning to add new storage. Cisco DMM does not enforce
zoning for the new storage. If you do not configure the zoning before migration, then you
must complete zoning after migration to allow the server to access the new storage.

Check the Storage ASL Status

The Cisco DMM feature contains the Array-Specific Library (ASL) which is a database of
information about specific storage array products. Cisco DMM uses the ASL to correlate the
LUN maps between multipath port pairs.
Use the Server LUN Discovery (SLD) tool to check the ASL status of LUNs on a storage array
port. If the LUNs all display “ASL = Yes,” then the Cisco DMM feature automatically
correlates the LUN maps.
The SLD tool is launched from the supervisor module CLI using the following command:
switch# sld module module-number vsan vsan-number server-pwwn
server pwwn target-pwwn target-pwwn discover
To display the ASL status for LUNs that are associated with a specified source and destination
port pair, run the following command:
switch# show sld module module-number vsan vsan-number server-
pwwn server-pwwn target-pwwn target-pwwn [detail]
The SLD tool can also be launched from the Cisco DMM GUI.
Configuring Enclosures
Ensure that the server and storage ports are included in enclosures before creating a migration
job.
If the server ports are not already included in existing enclosures, you will need to create those
enclosures for the server ports. If the server has multiple single-port HBAs, all of these ports
need to be included in one enclosure. Enclosures for existing and new storage ports are created
automatically.
Configuring the SAN Fabric
If the SAN is heterogeneous, you may need to install a new Cisco MDS 9000 Series switch or
switches, or adjust the SAN topology to meet Cisco DMM requirements. It is a requirement for

Cisco DMM that the existing and new storage devices are connected to MDS switches that
support the Fibre Channel redirect feature. The path from the Cisco DMM switch to the
storage-connected switch must be through a Cisco MDS fabric.

4-14 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Existing and new storage should be connected
to the same switch where MDS 9222i
Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch is installed. This step
alleviates excessive ISL traffic.
• Cisco DMM supports 16 simultaneous jobs on
each MDS 9222i Multiservice Modular Switch
or MDS 9250i Multiservice Fabric Switch
switch.
• The same initiator and target port pair should
not be added to more than one migration job
simultaneously.
• When using multipath ports, the server must
not send simultaneous I/O write requests to the Existing
New
same LUN from both multipath ports. Storage
Storage
• Cisco DMM is not compatible with LUN zoning,
IVR, or SAN device virtualization.
• Cisco DMM does not support migration to a
smaller destination LUN.
© 2013 Cisco and/or its affiliates. All rights reserved. DCMDS v2.0—4-9

The following deployment guidelines should be considered when planning and configuring data
migration using Cisco DMM:
 The same initiator and target port pair should not be added to more than one migration job
simultaneously.
 When using multipath ports, you must ensure that the server does not send simultaneous
I/O write requests to the same LUN from both multipath ports. The first I/O request must
be acknowledged as completed before initiating the second I/O request.
 Cisco DMM is not compatible with LUN zoning.
 Cisco DMM is not compatible with Inter-VSAN Routing (IVR). The server and storage
ports must be included in the same VSAN.
 Cisco DMM is not compatible with Cisco SAN device virtualization. The server and
storage ports cannot be virtual devices, or physical devices that are associated with a virtual
device.
 Cisco DMM does not support migration to a smaller destination LUN.

Cisco DMM SAN Topologies
This topic describes common SAN topologies and their implications for Cisco DMM.

• The host is connected to a Fibre Channel redirect switch.

Cisco DMM

Switch B
Existing Storage

Server Switch A Switch C

New Storage

While designed to support various SAN topologies, the Cisco DMM feature is also influenced
by topologies. Similarly, the SAN topology affects the location of the MDS 9222i Multiservice
Modular Switch or MDS 9250i Multiservice Fabric Switch. Cisco DMM supports
homogeneous and heterogeneous SANs, as well as single-fabric and dual-fabric SAN
topologies. Dual-fabric and single-fabric topologies support both single-path and multipath
configurations. In a single-path configuration, a migration job includes only the one path,
represented as an initiator and target port pair. In a multipath configuration, a migration job
must include all paths, represented as two initiator and target port pairs.

Fibre Channel Redirect

When a data migration job is in progress, all traffic that is sent between the server HBA port
and the existing storage is intercepted and forwarded to the MDS 9222i Multiservice Modular
Switch or MDS 9250i Multiservice Fabric Switch using the Fibre Channel redirect capability.
Fibre Channel redirect requirements for the SAN topology configuration include the following:
 Existing storage must be connected to a switch with the Fibre Channel redirect capability.
 Server HBA ports may be connected to a switch with or without the Fibre Channel redirect
capability.
 The switches with Fibre Channel redirect must be running Cisco Nexus Operating System
(NX-OS) Software Release 4.1(1b) or later.
 The server HBA port and the existing storage port must be zoned together. The default
zone policy must be set to deny.
 The MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch
can be located anywhere in the fabric, as long as the Fibre Channel Name Server (FCNS)
database in the Cisco MDS switch has the required information about the server HBA ports

4-16 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
and the existing storage ports. The Cisco MDS switch must be running Cisco NX-OS
Software Release 4.1(1b) or later.
The example shows the server-to-storage packet flow when a data migration job is in progress.
In the example, the MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice
Fabric Switch and the existing storage are connected to separate switches, so that the packet
flow can be easily described.
It is recommended that you connect the existing storage to the same switch as the MDS 9222i
Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch.
In the figure, the server HBA port is connected to switch A, and the existing storage is
connected to switch C. Both switches have Fibre Channel redirect capability. When the data
migration job is started, Fibre Channel redirect is configured on switch A to divert the server
traffic to switch B. Fibre Channel redirect is configured on switch C to redirect the storage
traffic to switch B.

• The host is not connected to a Fibre Channel redirect switch.

Cisco DMM

Switch B
Existing Storage

Server Switch A Switch C

New Storage

The figure shows that the server HBA port is connected to switch A, which does not have Fibre
Channel redirect. The existing storage is connected to switch C, which has Fibre Channel
redirect capability. The MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice
Fabric Switch is installed in the location of switch B. When the data migration job is started,
Fibre Channel redirect is configured on switch C to redirect the server and storage traffic to
switch B. This configuration introduces additional network latency and consumes additional
bandwidth because traffic from the server travels an extra network hop (A to C, C to B, and B
to C). The recommended configuration, placing the MDS 9222i Multiservice Modular Switch
or MDS 9250i Multiservice Fabric Switch in the location of switch C, avoids the increase in
network latency and bandwidth.

Cisco
Cisco DMMDMM Cisco
Cisco DMMDMM

FC FC

Existing Storage New Storage

A homogeneous SAN contains only Cisco MDS 9000 Series switches. Most topologies fit in
one of the following categories:
 Core-edge: Hosts at the edge of the network and storage at the core
 Edge-core: Hosts and storage at the edge of the network, and Inter-Switch Links (ISLs)
between the core switches
 Edge-core-edge: Hosts and storage are connected to opposite edges of the network and to
the core switches with ISLs

It is recommended for any of these topologies that the MDS 9222i Multiservice Modular
Switch or MDS 9250i Multiservice Fabric Switch is located in the switch that is closest to the
storage devices so that the Cisco DMM data migration introduces no additional network traffic.
In a homogeneous network, the MDS 9222i Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch can be located in the fabric where the existing storage is attached.
The new storage should be connected to the same switch as the existing storage. If there are
multiple fabrics, Cisco DMM needs to be installed in each fabric.

Application
Fabric A Server Fabric B

VSAN VSAN
Cisco 10 Cisco 20
DMMCisco DMM DMMCisco DMM
switch 1 switch 2

FC
FC

Cisco VSAN
Existing Storage DMMCisco DMM 15
switch 3
New Storage

Cisco DMM method 3 supports the dedicated migration fabric. This method is designed to
address the problem of migrating data from an array port that is connected to a dedicated SAN
that is different from the product SAN.
Many IT organizations require data migration to a remote data center. Some organizations
prefer to use a dedicated storage port (on the existing storage array) that is connected to a
separate physical fabric. This fabric is called the migration or replication fabric because the
fabric is used for data migration as well as continuous data replication services.
In Cisco DMM method 3, the MDS 9222i Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch in the migration SAN is responsible for executing the sessions in
the Cisco DMM job and copying the data from the existing storage to the new storage. The
MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch in the
production SANs are responsible for tracking the server writes to the existing storage. No
server writes are expected in the migration SAN.
The MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch logs
server writes in the production SAN in the fabric by maintaining a modified region log for each
LUN that is migrated. The MDS 9222i Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch in the migration SAN does not maintain any modified region log
for the LUN because no server writes to the existing storage LUN are expected in the migration
SAN. The MDS 9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch
in the migration SAN is responsible for retrieving the modified region logs for a LUN from
both the production SANs and for performing a union of the modified region logs to create a
superset of all modified blocks on the LUN, via paths from both production SANs. The MDS
9222i Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch then copies all
the modified regions from the existing storage LUN to the new storage LUN in the migration
SAN. This process is repeated until the administrator is ready to finish the Cisco DMM job and
perform a cutover. The finish operation on method 3 places all LUNs in the offline mode and
performs a final pass over the combined modified region log. This process synchronizes the
existing and new storage LUN in each session.

4-20 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
The three-fabric topology supports two production fabrics and one migration fabric. Each of the
fabrics has one VSAN per fabric, as shown earlier in this lesson.
The production fabric consists of the following:
 Two fabrics, fabric A and fabric B
 Two VSANs in each of the fabrics, VSAN 10 in fabric A, and VSAN 20 in fabric B
 Two Cisco DMM switches (where a Cisco DMM module may be an MDS 9222i
Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch) in each of the
fabrics
 Ports for the application server and the existing storage
 Application server port and storage port in the same VSAN for each fabric
 VSANs in both the fabrics can have different numbers

The migration fabric consists of the following:

 One fabric, fabric C
 One VSAN, VSAN 15
 One Cisco DMM, Cisco DMM switch 3
 Existing storage port and new storage port in the same VSAN

The migration fabric VSAN can have a different number from the production fabric VSAN.

• Single LUN Set, Active-Active • Multiple LUN Set, Active-Active
Array Array

H1 Server H2 H1 Server H2

Fabric 1 Fabric 2 Fabric 1 Fabric 2

ES1 NS1
ES1 NS2 ES2 NS2

FC FC FC FC

ES2 NS1 ES3 NS3

ES4 NS4
Existing Storage New Storage Existing Storage New Storage
(ES) (NS) (ES) (NS)

When creating a server-based migration job, all possible paths from the host to the LUNs being
migrated must be included. All writes to a migrated LUN need to be mirrored in the new
storage until the job is destroyed, so that no data writes are lost. All active ports on the existing
storage that expose the same set of LUNs to the server must be added to a single data migration
job.
In a multipath configuration, two or more active storage ports expose the same set of LUNs to
two HBA ports on the server (one initiator and target port pair for each path). Multipath
configurations are supported in dual-fabric topologies (one path through each fabric) and in a
single-fabric topologies (both paths through a single fabric).
In a single-path configuration, only one active storage port exposes the LUN set to the server.
The migration job includes one initiator and target port pair. (Cisco DMM does not support
multiple servers accessing the same LUN set.)

Single LUN Set, Active-Active Array

The server accesses three LUNs over fabric 1 using storage port ES1. The server accesses the
same LUNs over fabric 2 using storage port ES2.
Both storage ports (ES1 and ES2) must be included in the same data migration job, as both
ports are active and expose the same LUN set.
The following configuration would be used to create a data migration job:
Configuring a Data Migration Job for a Single LUN Set

Server Port Existing Server Port New Storage Port

H1 ES1 NS1

H2 ES2 NS2

4-22 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Multiple LUN Set, Active-Active Arrays

Server Port Existing Server Port New Storage Port

H1 ES1 NS1

H2 ES2 NS2

The server accesses three LUNs over fabric 1 using storage port ES1. The server accesses the
same LUNs over fabric 2 using storage port ES2. The server accesses three different LUNs
over fabric 1 using storage port ES3, and accesses the same LUNs over fabric 2 using storage
port ES4.
In this scenario, you would need to create two data migration jobs, as the server has access to
two LUN sets on two different storage ports. Each data migration job would need to include
two storage ports, as they are active-active multipathing ports.

Configuring a Data Migration Job for a Multiple LUN Set

This job includes three data migration sessions (for LUNs 1, 2, and 3).
The other migration job has the following configuration:
Configuring a Data Migration Job for Three Sessions

Server Port Existing Server Port New Storage Port

H1 ES3 NS3

H2 ES4 NS4

This job includes three data migration sessions (for LUNs 7, 8, and 9).

Cisco DMM Software installation and
Configuration
This topic describes how to install and configure Cisco DMM software.

• The Cisco DMM software package is included in the SSI image.

- Download the SSI image and copy to the internal bootflash.
- Install the SSI image.
- Enable the Cisco DMM feature.

switch# copy tftp://10.0.1.21/m9000-ek9-ssi-mz.6.2.1.bin bootflash:m9000-ek9-

ssi-mz.6.2.1.bin

switch# dir bootflash:

15651829 July 03 19:36:24 2013 m9000-ek9-ssi-mz.6.2.1.bin

20530688 July 3117:47:21 2013 m9200-s2ek9-kickstart-mz.6.2.1.bin

98807270 July 31 17:44:05 2013 m9200-s2ek9-mz.6.2.1.bin

switch# install all ssi bootflash:m9000-ek9-ssi-mz.6.2.1.bin

The Cisco DMM feature runs on the MDS 9222i Multiservice Modular Switch or MDS 9250i
Multiservice Fabric Switch. The Cisco DMM software package is included in the Storage
Service Image (SSI) image, which is delivered as part of Cisco NX-OS.

• Cisco DMM communicates with the supervisor over VSAN 1 using IP.
- Create the VSAN 1 interface and IP address.
mds2# conf
mds2(config)# interface vsan 1
mds2(config-if)# ip address 10.10.1.11 255.255.255.0
mds2(config-if)# no shut

mds2# conf
mds2(config)# interface vsan 1
mds2(config-if)# ip address 10.10.2.12 255.255.255.0
mds2(config-if)# no shut

Fabric A Fabric B
VSAN 1 IP IP Network VSAN 1 IP
10.10.1.11 Mgmt IP Mgmt IP 10.10.2.12
MDS2
172.22.43.94 172.22.43.95 MDS 9222i Multiservice Modular Switch
MDS1
MDS 9222i Multiservice Modular Switch FC
FC

Existing Storage New Storage

The Cisco DMM feature uses the CLI on the Cisco MDS 9000 Series switches (MDS 9222i
Multiservice Modular Switch and MDS 9250i Multiservice Fabric Switch).

Cisco DMM GUI (Cisco DCNM-SAN Client)

The GUI for the Cisco DMM is integrated into the Cisco Data Center Network Manager
(DCNM)-SAN Client. The Cisco DMM GUI communicates with the Cisco MDS 9222i
Multiservice Modular Switch or MDS 9250i Multiservice Fabric Switch using a Secure Shell
(SSH) protocol connection. Before you use Cisco DMM, you should enable SSH on the Cisco
DMM switch.

Configuring IP Connectivity for Cisco MDS Switches

Cisco DMM switches and the supervisor module in a Cisco MDS 9000 Series switch
communicate over VSAN 1 using IP. Cisco DMM switches communicate with their peer Cisco
DMM-capable switches using the management IP network, and each Cisco DMM-enabled
switch on the management IP network must have a unique subnetwork address that is
configured for VSAN 1.
The following tasks explain how to configure IP over Fibre Channel (IPFC) for a Cisco DMM-
capable switch:
 Create the VSAN 1 interface and configure an IP address for this interface.
 The IP subnet can use a private address space because these addresses will not be
advertised.
 The subnet number needs to be unique for each Cisco DMM-capable switch that is
connected to the same management IP subnetwork.

• Configure routing in MDS 1 and MDS 2.

mds1(config)# ip routing
mds1(config-if)# ip route 10.10.2.12 255.255.255.0 172.22.43.95

mds1(config)# exit

mds2(config)# ip routing

mds2(config-if)# ip route 10.10.1.11 255.255.255.0 172.22.43.94

mds2(config)# exit

VSAN 1 IP
Fabric A Fabric B VSAN 1 IP
10.10.1.11 Mgmt IP Mgmt IP 10.10.2.12
172.22.43.94
IP Network 172.22.43.95

MDS1 MDS2

Step 1 Enable IP version 4 (IPv4) routing.

Step 2 Configure static routes as required to access the peer Cisco DMM-capable switches.
This configuration enables the routing of packets between peer Cisco DMM-capable
switches.

4-26 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Select Physical Attributes > End Devices > Intelligent Features >
MSM.

switch# conf

switch(config)# ssm enable feature dmm module slot#

switch(config)# ssm enable feature dmm interface slot/port - port

The Cisco DMM feature can be enabled using the CLI or Cisco DCNM-SAN Client. The
example shows the procedure for both options.
To enable Cisco DMM on the Cisco MDS 9222i Multiservice Modular Switch, use the
following command:
switch(config)# ssm enable feature dmm module 1
To enable Cisco DCNM-SAN Client, and to provision ports to use the Cisco DMM service,
perform the following steps:
Step 1 Expand the End Devices folder and double-click the Intelligent Features folder in
the Physical Attributes pane. Choose the Cisco DMM switch on which to provision
Cisco DMM.
Step 2 Click the Create Row icon (third button in the toolbar). The Create SSM dialog box
opens.
Step 3 Choose the switch and module where you want to enable Cisco DMM.

Note The Use All Ports on Module check box is checked by default. Cisco DMM does not
support provisioning a subset of the ports on the card to use this service.

Step 4 From the feature drop-down list, choose Cisco DMM.

Step 5 Click the Create button to enable the Cisco DMM service.

Use of the Cisco DMM GUI for Data Migration
This topic describes how to prepare for a data migration job.

• Select Tools > Data MobilityManager > Storage Based

The Cisco DMM GUI provides wizards to configure and perform data migration jobs. The
Cisco DMM wizards can be launched from Cisco DCNM-SAN Client by choosing the Cisco
MDS Data Mobility Manager option from the Tools menu. There are separate options to launch
server-based data migration jobs and storage-based migration jobs.

There are five steps to configuring a Cisco DMM job using the Cisco DCNM-SAN Client
wizard.
1. Create job: This first step creates the job, identifying the existing and new storage that will
be used.
2. Select SSM: This second step identifies the module that will be used with Cisco DMM
capability.
3. Data path processor virtual initiator selection: This step identifies the virtual nodes that
are created during the Cisco DMM provisioning. Choose the preferred node and create or
activate the zone.
4. Zone Activation Status: This step verifies the activation of the zone.
5. Create Session: This step verifies the details for the session and creates the LUN map
correlations.

Summary
This topic summarizes the key points that were discussed in this lesson.

• The Cisco DMM is used to migrate data from existing storage resources
to new storage. To use the Cisco DMM feature, the SSI image must be
installed on the switch, and Cisco DMM must be provisioned for the
modules that will be using the feature.
• The location of the 18/4 MSM, Cisco MDS 9222i Multiservice Modular
Switch or MDS 9250i Multiservice Fabric Switch is influenced by the
SAN topology being used.
• The preconfiguration tasks required for configuring a Cisco DMM job,
such as enabling SSH, provisioning Cisco DMM, configuring the VSAN,
and enabling a zone set, should be performed first.
• Configuring and managing the Cisco DMM job can be performed
through the Cisco DCNM-SAN Client.

Monitoring Traffic Flow

Overview
This lesson is an overview of how to monitor all of the network traffic flow on the SAN fabric.

Objectives
Upon completing this lesson, you will be able to explain how to capture and monitor Fibre
Channel Protocol data.
This lesson explains how to use Wireshark and Switched Port Analyzer (SPAN) ports to
capture Fibre Channel Protocol traffic for analysis and how to use the built-in Cisco Fabric
Analyzer to monitor traffic flow to and from the supervisor. This ability includes being able to
meet these objectives:
 Describe how to configure a SPAN session
 Explain the capabilities and limitations of RSPAN sessions
 Explain the use of the Cisco Fabric Analyzer feature
 Describe how to configure Wireshark to capture and view protocol traces
SPAN Overview
This topic describes how to configure a SPAN session to capture traffic for an analyzer for
analysis purposes.

• SPAN is a nonintrusive, proactive monitoring tool for data analysis and

protocol debugging.
- SPAN replicates traffic to a designated SD port from up to four sources with no
effect on normal traffic.
- SPAN can be used with a protocol analyzer and software like Wireshark.

Analyzer

The Cisco SPAN utility provides a mechanism for performing detailed troubleshooting by
replicating selected traffic from multiple source interfaces for proactive monitoring and
analysis. SPAN is most useful when used with protocol analyzer software such as Wireshark
(formerly Ethereal).
The Cisco MDS 9000 Series switch debug commands only report control traffic. SPAN
sessions capture all traffic, including data in the protocol traces. Traffic through any Fibre
Channel interface can be replicated to a special port called the SPAN destination port (SD
Port). You can attach a Fibre Channel analyzer to the SD Port to monitor SPAN traffic.
You can create up to 16 independent SPAN sessions within the switch. Each session can have
up to four unique sources and one destination port. Filters can be applied to capture specific
ingress or egress traffic, or traffic from a particular virtual SAN (VSAN).

4-34 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Allowed source interfaces include F, FL, E, TE, TL, sup-fc0, FCIP, iSCSI,
and port channels
• VSAN as a source:
- Traffic on all interfaces included in a source VSAN is spanned only in the
ingress direction.
- When a VSAN is specified as a source, you cannot specify interfaces in that
VSAN as SPAN sources.
- If an interface in a VSAN is configured as a SPAN source, that VSAN cannot
be configured as a source.
- Interfaces are included as sources only when the port VSAN matches the
source VSAN.

The SPAN feature is available for the following interface types:

 Physical ports: These port types include fabric ports (F Ports), fabric loop ports (FL Ports),
expansion ports (E Ports), and translative loop ports (TL ports).
 Interface sup-fc0 (traffic to and from the supervisor): Fibre Channel traffic from the
supervisor module to the switch fabric is called ingress traffic. Ingress traffic is spanned
when sup-fc0 is chosen as an ingress source port. Fibre Channel traffic from the switch
fabric to the supervisor module is called egress traffic. Egress traffic is spanned when sup-
fc0 is chosen as an egress source port.
 Port channels: All ports in the port channel are included and spanned as sources. You
cannot specify individual ports in a port channel as SPAN sources.
 IP storage services module interfaces: Internet Small Computer System Interface (iSCSI)
and Fibre Channel over IP (FCIP) ports can be used as source interfaces.

Source port filter options include receive (Rx), transmit (Tx), and VSAN.
Traffic types are management, initiator, and target.
When configuring a VSAN as a SPAN source, note the following:
 When a VSAN source is specified, all physical ports, including trunking expansion ports
(TE Ports) and port channels that are assigned to that VSAN, become SPAN sources.
 A TE Port that is assigned to a different VSAN that is configured with the source VSAN in
the allowed VSAN list is excluded.
 Configuring source interfaces and source VSANs in the same SPAN session is
unsupported.

The following guidelines apply when configuring VSANs as a source:

 Traffic on all interfaces that are included in a source VSAN is spanned only in the ingress
direction.

 When a VSAN is specified as a source, you cannot perform interface-level SPAN
configuration on the interfaces that are included in the VSAN. Previously configured
SPAN-specific interface information is discarded.
 If an interface in a VSAN is configured as a SPAN source, you cannot configure that
VSAN as a source. You must first remove the existing SPAN configurations on such
interfaces before configuring a VSAN as a source.
 Interfaces are included as sources only when the port VSAN matches the source VSAN.

4-36 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Up to 16 SPAN sessions are allowed on a Cisco MDS 9000 Series
switch:
- A source can be shared by two sessions:
• Each session must be in a different direction.
- Frames are dropped if traffic exceeds SD port bandwidth.
- SD port characteristics include the following:
• Speed must be explicitly set and autodetection is unsupported.
• The SD port transmits traffic only and the port ignores BB credits.
• The SD port is not supported on SSMs.

switch(config)# span session 1

switch(config-span)# destination interface fc1/16
switch(config-span)# source interface fc1/5
switch(config-span)# source interface sup-fc0
switch(config-span)# no suspend

A SPAN session consists of a designated SD Port with up to four source interfaces. A given
VSAN can also serve as a SPAN session source. In this circumstance, traffic is replicated from
all interfaces that are assigned to the VSAN. When specifying a VSAN as a SPAN source, no
additional source ports are allowed. The SPAN configuration sequence requires that the
destination port is configured before specifying any SPAN sources. The minimum
configuration requirements for a SPAN session include the following:
 Specify a SPAN session number.
 Designate the SD Port.
 Configure source ports and optional filters. Source filters include Rx, Tx, VSAN, and
traffic type (management, initiator, or target).
 Activate the SPAN session. The session is activated by default.

The SD Port and at least one source port must be up and enabled for the SPAN session to
function. To temporarily deactivate or suspend a SPAN session, use the suspend command in
SPAN configuration submode. Issue the no suspend command to reactivate the session.

SD Port Characteristics
Any Fibre Channel port can be configured as an SD Port. SD Ports cannot be used for normal
data traffic because SD Ports do not behave as standard Fibre Channel interfaces. The
minimum SD Port configuration requires specifying the mode and speed. SD Ports ignore
buffer-to-buffer credits (BB_credits) because traffic is allowed in the egress direction only.
Speed autodetection is not supported. The port speed may be set to match the Analyzer setting.
The SD Port does not require a device or analyzer to be physically connected to become active.
Support for Enhanced Inter-Switch Link (EISL) encapsulation is optional, but is useful if
capturing ingress traffic or any traffic from a TE port.
Other SPAN session rules include the following:
 Two sessions can share a source. However, each session must be in a different direction,
that is, one ingress and one egress.

 SPAN frames are dropped if the sum of the bandwidth of the sources exceeds the speed of
the destination port. Frames are not buffered.
 Frames dropped by a source port are not spanned.
 The SD Port does not have a port VSAN.
 The port mode cannot be changed if the port is being used for a SPAN session.

4-38 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
switch# show span session
Session 1 (active)
Destination is fc1/16
No session filters configured
Ingress (rx) sources are
fc1/5, sup-fc0,
Egress (tx) sources are
fc1/5, sup-fc0,

Session 2 (admin suspended)

Destination is fc1/15
No session filters configured
Ingress (rx) sources are
vsans 161
No egress (tx) sources

To show a detailed display of SPAN session status, use the show span session command.
The figure shows two configured sessions, with SPAN session 2 suspended. The displayed
information includes the following:
 Session ID
 State (active or suspended)
 Destination interface
 Session filters
 Ingress and egress sources

RSPAN Overview
This topic explains the capabilities and limitations of Remote SPAN (RSPAN) sessions.

• RSPAN allows SPAN sessions to replicate traffic to a separate

destination switch.
• An RSPAN tunnel facilitates traffic replication across TE Ports using the
MPLS field in the EISL header.
• The ST Port on the source switch performs frame encapsulation.

ST RSPAN Tunnel SD

Source MDS Destination MDS

ISL

Host Storage
Analyzer

RSPAN allows you to remotely monitor traffic for one or more SPAN sources that are
distributed in one or more source switches in a Fibre Channel fabric. Like SPAN, the RSPAN
feature is nonintrusive. Traffic that is captured on the remote source switch is tunneled across
TE Ports using a Fibre Channel tunnel.
The Fibre Channel tunnel is a logical data path between the source switch, where the traffic is
captured, and the destination switch. The Fibre Channel tunnel originates at a designated SPAN
tunnel port (ST Port) on the source switch and terminates at the SD Port on the destination
switch. The configuration process involves binding the ST and SD Ports to the same Fibre
Channel tunnel, creating the RSPAN tunnel.

4-40 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Cisco Fabric Analyzer
This topic explains how to use the Cisco Fabric Analyzer to capture control traffic for analysis
purposes.

• Captures and decodes control traffic, not data traffic

• Consists of two separate components:
- Software on the Cisco MDS 9000 Series switch that supports two modes of
capture
- A GUI-based client that runs on Windows or Linux
switch(config)# fcanalyzer local
Capturing on inband interface
Frame 1 (164 bytes on wire, 164 bytes captured)
Arrival Time: Mar 8, 2010 12:35:43.740256000
Time delta from previous packet: 0.000000000 seconds
Time since reference or first frame: 0.000000000 seconds
Frame Number: 1
Packet Length: 164 bytes
Capture Length: 164 bytes
Ethernet II, Src: 00:00:00:00:00:0a, Dst: 00:00:00:00:ee:00
Destination: 00:00:00:00:ee:00 (00:00:00:00:ee:00)
Source: 00:00:00:00:00:0a (00:00:00:00:00:0a)
Type: Unknown (0xfcfc)
MDS Header(Unknown(0)/Unknown(0))
MDS Header
...0 0000 1000 0110 = Packet Len: 134
…………………………. Remainder removed

Fibre Channel protocol analyzers capture, decode, and analyze frames and ordered sets on a
link. With the Cisco Fabric Analyzer, you can capture Fibre Channel control traffic from a
switch and decode the traffic without having to disrupt any connectivity, and without having to
be local to the point of analysis. Unlike Port Analyzer Adapter (PAA), Cisco Fabric Analyzer is
a software application that runs on the Cisco MDS 9000 Series switch.
The Cisco Fabric Analyzer captures and decodes control traffic, not data traffic. Cisco Fabric
Analyzer is suitable for control path captures and is not intended for high-speed data path
captures. The Cisco Fabric Analyzer runs on the Cisco MDS 9000 Series switch and supports
two methods of capture:
 Text-based analyzer: This method supports local capture and decodes captured frames.
 Daemon: This method, which supports remote capture, is a GUI-based client. Daemon
runs on a host that supports libpcap (a packet capture library program), such as Windows or
Linux, and communicates with the remote capture daemon in a Cisco MDS 9000 Series
switch.

• Begin capturing the frames locally:
switch(config)# fcanalyzer local

• Display parameters:
switch# fcanalyzer local interface mgmt ?
<CR>
> Redirect it to a file
>> Redirect it to a file in append mode
brief Display only protocol summary
capture-filter Filter on packet-analyzer capture
display-filter Filter frames displayed
dump-pkt Hex(Ascii) dump the packet, troubleshoot
packet-analyzer
limit-captured-frames Maximum number of frames to be captured (default is 10)
limit-frame-size Capture only a subset of a frame
write Filename to save capture to
| Pipe command output to filter

You can configure the Cisco Fabric Analyzer by issuing the fcanalyzer local or fcanalyzer
remote commands in configuration mode:
 Use the local setting to enable local console display that cannot be saved to persistent
storage or synchronized to standby.
 Use the remote setting to enable remote capture.

The command setting can be saved to persistent storage with the copy command.
To use the Cisco Fabric Analyzer feature, traffic should be flowing to or from the supervisor
module.

4-42 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Send captured frames to a remote host:
switch(config)# fcanalyzer remote 10.0.16.1

• Enable active mode with the remote host; default is passive:

switch(config)# fcanalyzer remote 10.0.16.1 active

• Launch Wireshark in Windows to capture remote fcanalyzer:

The figure displays options for the fcanalyzer remote command:

 The command fcanalyzer remote ip-address active enables active mode (passive is the
default) with the remote host. Wireshark is assumed to be running when the capture is
performed. The switch tries to connect continuously unless a capture stop instruction is sent
from Wireshark.
 The command fcanalyzer remote ip-address active 5 enables active mode for a specified
port. In this example, it is port 5.

Wireshark Overview
This topic explains how to use and configure Wireshark with SPAN ports, to capture traffic for
analysis purposes.

• Wireshark is a free software protocol analyzer with the following

features:
- Runs on a PC or workstation running Windows or Linux
- Filter-based capture and display
- Minimal protocol statistics
- Can be used to capture iSCSI and FCIP traffic
- Colorized display
- Find and mark frames
- Find next or previous frame in exchange
- Customizable columns
- Download software and documentation:
• http://www.cisco.com/kobayashi/sw-center/sw-stornet.shtml
• http://www.wireshark.org

Wireshark (formerly Ethereal) is a free software protocol analyzer, or packet sniffer

application, which is used for network troubleshooting, analysis, software and protocol
development, and education. Wireshark has all of the standard features of a protocol analyzer.
Wireshark was created as a fork of Ethereal.
Wireshark software runs on a host, such as a PC or workstation running Windows or Linux.
Wireshark can be used to capture traffic from an iSCSI or FCIP port that is configured as a
source port for a SPAN session. The traffic is shown as iSCSI or FCIP encapsulating the Fibre
Channel traffic. The frame types are displayed as iSCSI or FCIP.
Wireshark provides functionality that is very much like tcpdump, but Wireshark has a GUI
front end and many more information sorting and filtering options. Wireshark allows the user to
see all traffic being passed over the network (usually an Ethernet network, but support is being
added for others) by putting the network card into promiscuous mode.
Wireshark is available under the general public license (GPL) or open-source license, and uses
the cross-platform GTK+ widget toolkit. Wireshark runs on UNIX and UNIX-like systems.
Those systems include:
 Linux
 Solaris
 FreeBSD
 NetBSD
 OpenBSD

4-44 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
 Mac OS X (although GTK+ only works with X11 on Mac OS X, so the user must run an X
server such as X11.app)
 Windows

Wireshark can recognize the structure of different network protocols. Wireshark can therefore
display encapsulation and single fields, and interpret their meaning. Wireshark uses pcap to
capture packets, so Wireshark can only capture on networks that pcap supports.
Wireshark includes the following rich feature set:
 Standard three-pane packet browser
 Multiplatform application that runs on Windows, Linux, OS X, Solaris, FreeBSD, NetBSD,
and many others
 Multi-interface and, along with a standard GUI, Wireshark includes TShark, a text-mode
analyzer that is useful for remote capture, analysis, and scripting
 The most powerful display filters in the industry
 VoIP analysis
 Live capture and offline analysis supported
 Ability to read and write many different capture file formats, including the following:
— tcpdump (libpcap)
— NAI Sniffer (compressed and uncompressed)
— Sniffer Pro
— NetXray
— Sun snoop and atmsnoop
— Shomiti/Finisar Surveyor
— AIX iptrace
— Microsoft Network Monitor
— Novell LANalyzer
— RADCOM WAN/LAN Analyzer
— HP-UX nettl
— i4btrace from the ISDN4BSD project
— Cisco Secure IDS iplog
— pppd log (pppdump-format)
— AG Group/WildPacket EtherPeek/TokenPeek/AiroPeek
— Visual Networks Visual UpTime
 Captured files that are compressed with gzip can be decompressed in real time
 Hundreds of protocols are supported, with more being added
 Coloring rules can be applied to the packet list, which simplifies analysis

To set up Wireshark on a PC with Windows, follow these steps:

Step 1 Download Wireshark and follow the instructions for installing the software.
Step 2 Confirm that the Cisco MDS 9000 Port Analyzer Adapter is installed and
configured.
Step 3 Configure SPAN or RSPAN on the Cisco MDS 9000 Series switch ports.
Step 4 Verify that the Fibre Channel port that is connected to the adapter is configured as
an SD Port by using the show interface command.
Step 5 Verify the correct destination and source of traffic by using the show span session
command.
To begin capturing frames, follow these steps:
Step 1 Launch the Wireshark network analyzer application.
Step 2 Navigate to Capture > Start to begin the capture.
Step 3 Select the correct network interface card (NIC).
Step 4 Choose Capture.

The Wireshark Filter window lets you filter the displayed protocol trace to display frames from
a specific source and destination, from a specific VSAN, or frames of a specific type.
You can use the logical “AND” and “OR” expressions. You can use “&&” in place of “AND.”
You can also use “||” in place of “OR.”
A list of specific filter expressions can be found at http://www.wireshark.org. Examples include
the following:
Use this filter to view packets in a specified VSAN:
mdshdr.vsan == 2 || brdwlk.vsan == 2
Use this filter to view Switch Fabric Internal Link Services (SW_ILS) frames:
fcswils
Use this filter to view Class F frames:
mdshdr.sof == SOFf || brdwlk.sof == SOFf
Use this filter to view Fabric Shortest Path First (FSPF) frames:
swils.opcode == JLO || swils.opcode == LSU || swils.opcode ==
LSA
Use this filter to view fabric login (FLOGI) frames:
fcels.opcode == FLOGI
Use this filter to view FLOGI frames in VSAN 2:
fcels.opcode == FLOGI && (mdshdr.vsan == 2 || brdwlk.vsan ==
2)
Use this filter to view all name server frames:
dNS

Summary
This topic summarizes the key points that were discussed in this lesson.

• The minimum configuration requirements for a SPAN session are SPAN

session number selection, SD Port designation, source port and optional
filter configuration, and SPAN session activation.
• RSPAN allows SPAN sessions to replicate traffic to a separate
destination switch.
• The fcanalyzer command allows you to capture, decode, and analyze
Fibre Channel control traffic only.
• When installed and properly configured, the Wireshark Filter window lets
you filter the displayed protocol trace to display frames from a specific
source and destination.

4-48 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Summary
This topic summarizes the key points that were discussed in this module.

• Cisco Data Mobility Migration provides a heterogeneous method of

moving data from one storage array to another on a LUN-by-LUN basis.
• Cisco MDS supports several tools for frame and throughput analysis.
SPAN allows an analyzer to connect nondisruptively to the fabric and
receive data and control frames for analysis. In addition, RSPAN allows
more flexibility to the use of an analyzer by supporting an analyzer
connection that is separated from the source of frames by an ISL. Cisco
Fabric Analyzer monitors control traffic.

4-50 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Self-Check
Use the questions here to review what you learned in this module. The correct answers and
solutions are found in the Module Self-Check Answer Key.
Q1) Which line card module supports the DMM feature? (Source: Implementing Cisco
MDS Data Mobility Manager)
A) 32-port 8-Gb/s Advanced Fibre Channel Module
B) 48-port 8-Gb/s Fibre Channel Advanced Module
C) Storage Services Node
D) MDS 9000 18/4-Port Multiservice Module
E) 48-port 16-Gb/s Fibre Channel Module
Q2) There are the three stages of data migration. Which three options are valid data
migration stages? (Choose three.) (Source: Implementing Cisco MDS Data Mobility
Manager)
A) migration
B) configure
C) schedule
D) postmigration
E) premigration
F) create job
G) plan
H) destroy job
Q3) There are three methods for using DMM. In DMM method 3, with a three-fabric
configuration, which of the DMM-capable modules is responsible for executing the
sessions in the DMM job? (Source: Implementing Cisco MDS Data Mobility Manager)
A) module 2 in production fabric B
B) module 1 in production fabric A
C) module 3 in migration fabric
D) module 3 in production fabric A
Q4) An analyzer encapsulates which protocol to facilitate traffic analysis using the existing
IP infrastructure? (Source: Monitoring Traffic Flow)
A) iSCSI
B) Fibre Channel
C) SCSI
D) TCP/IP
Q5) Which statement about the Wireshark remote capture protocol daemon is false?
(Source: Monitoring Traffic Flow)
A) The daemon is GUI-based.
B) The daemon requires a Fibre Channel connection to the switch.
C) The daemon allows the administrator to sort, filter, crop, and save traces to the
local workstation.
D) The daemon is used with SPAN or Cisco Fabric Analyzer.

Q6) Which command will capture and display control frames in the CLI window? (Source:
Monitoring Traffic Flow)
A) switch# fcanalyzer local interface mgmt. br
B) switch# fcanalyzer local interface mgmt. >
volatile:saveframes
C) switch# fcanalyzer local interface mgmt. write
saveframes
D) switch# fcanalyzer remote 10.0.16.1
Q7) What are the minimum configuration requirements for a SPAN session? (Source:
Monitoring Traffic Flow)
A) SPAN session number selection, ST Port designation, source port and optional
filter configuration, and SPAN session activation
B) SPAN session number selection, SD Port designation, source port and optional
filter configuration, SPAN session activation
C) SD Port designation, source port and optional filter configuration, and SPAN
session activation
D) ST Port designation, source port and optional filter configuration, and SPAN
session activation

4-52 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Self-Check Answer Key
Q1) D
Q2) A, D, E
Q3) C
Q4) B
Q5) B
Q6) A
Q7) B

FCoE Implementation
Overview
In this module, you will learn how to implement Fibre Channel over Ethernet (FCoE).

Module Objectives
Upon completing this module, you will be able to implement FCoE. You will be able to meet
these objectives:
 Describe FCoE and its advantages
 Configure FCoE on the Cisco MDS 9500 Series Multilayer Directors
5-2 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Lesson 1

Describing FCoE
Overview
This lesson explains Fibre Channel over Ethernet (FCoE) and its advantages. This lesson also
describes how the Data Center Bridging Exchange (DCBX) protocol eases the discovery and
configuration of Data Center Bridging (DCB)-capable devices. The features that a DCB-
capable device can provide are explained.

Objectives
Upon completing this lesson, you will be able to describe the FCoE, DCBX, and DCB features
of the Cisco MDS 9000 Series switch. This ability includes being able to meet these objectives:
 Describe the consolidation benefits of Unified I/O and the Cisco Data Center Ethernet
enhancements that provide a lossless fabric
 Explain the FCoE protocol and how Fibre Channel frames are encapsulated in Ethernet
frames
 Describe the FCoE adapters and software stack, and explain their features and benefits
 Explain the FCoE addressing scheme and how an FCoE node can acquire a unique MAC
address from the Fibre Channel Forwarder (FCF)
 Describe FCoE Initialization Protocol (FIP)
I/O Consolidation
This topic introduces I/O consolidation and describes the data center Ethernet enhancements
that enable Fibre Channel frames to be carried over a lossless Ethernet network.

Aggregation

SAN A SAN B

Access

Lossless
Ethernet

SAN Infrastructure Ethernet Converged Infrastructure (FCoE)

Ten Gigabit Ethernet is the basis for a new Unified Fabric with enhanced features that provide
a common platform for loss and lossless protocols carrying LAN and SAN.
In the future, a converged network with a unified fabric will not require the customer to
sacrifice any of the best features of current network technologies.
The domination of Ethernet means that FCoE can leverage the installed Ethernet base and
therefore reduce the number of I/O adapters, cables, and switches. This ability has important
ramifications on space, power, and cooling costs. You will be able to wire once with a media
with which all network operators are familiar.

5-4 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• SAN • LAN
- Traditionally - Traditionally
Fibre Channel Ethernet
- High bandwidth - Low Bandwidth –
– 8Gb Gigabit Ethernet
- Low latency – - High latency –
20uS 350uS
- Lossless - Lossy delivery
delivery SAN A SAN B

SAN Ethernet
Infrastructure Infrastructure

Modern data centers usually have two very different networks, which are LAN and SAN. The
LAN consists of three layers, which are core, aggregation, and access. Traditionally, Gigabit
Ethernet was defined by low bandwidth and relatively high latency. Ethernet frames are
dropped if there is congestion and Ethernet is regarded as a network subject to frame loss.
SAN consists of two layers, which are core and edge. Fibre Channel SANs are used to carry the
Small Computer Systems Interface (SCSI) protocol between servers and storage devices within
the payload of Fibre Channel frames. Fibre Channel is relatively high bandwidth with 8-Gb
links and low latency at around 20uS for Cisco MDS switch store and forward latency. Fibre
Channel has a lossless delivery that is provided by the credit-based flow control mechanism.
This mechanism ensures that frames are never transmitted unless the receiving port provides a
credit guaranteeing that the port has a buffer available. This guarantee ensures that frames are
never dropped, removing the need for recovery and keeping latency very low.

LAN
• Today:
- Too many cables 1 GE Ethernet
1 GE
1 GE Ethernet
1 GE
1 GE Ethernet
- Too many NICs and HBAs 1 GE
1 GE
1 GE Ethernet Nexus
- Separate Ethernet and FC FC
SAN
FC HBA
Fibre Channel switches Fibre
FC FC Channel
FC HBA
MDS

• Unified I/O with FCoE: LAN

- Reduced cables and adapters
- Simplified access layer
Nexus 7K
FC
CNA FCoE
FC
SAN
FC
CNA FCoE
FC

Nexus 5K

A server that is deployed in a data center today has many discrete I/O interfaces to support
specific application requirements:
 Ethernet network interface cards (NICs) (LAN): Used for client-to-server connectivity,
server-to-server connectivity, and out-of-band (OOB) management (typically 10/100/1000-
Mbps Ethernet NICs).
 Fibre Channel host bus adapters (HBAs) (SAN): Used for server-to-storage connectivity
and unique in their ability to combine lossless operation with the highest levels of network
resiliency and availability in the data center (typically 1/2/4-Gb/s Fibre Channel HBAs).

Typical servers in data centers have five to seven I/O interfaces. A unified I/O adapter that
adequately supports the unique and varied traffic requirements of data center applications can
reduce the number of network devices, server-network interfaces, and cables that are used to
interconnect them. Unified I/O can also lead to a major reduction in data center power
requirements. Power is the most limited resource that is available to data center managers
currently.

5-6 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
FC Traffic
FC Traffic FCoE SAN A
FCoE
FCoE
Enet Traffic FCoE SAN B
Enet Traffic

FCoE SAN

Fewer NICs and Cables Same Management

Model as Native Fibre Channel

Display
FC FCoE FCoE
HBA Adapter Adapter

FC Storage FC Switch FCoE Server

Switch

No Gateway Less Power and Cooling

I/O consolidation using FCoE provides the following benefits:

 FCoE reduces the number of NICs and cables that customers need to purchase, reducing
costs and simplifying cable management.
 Because FCoE follows the same model as Fibre Channel, storage administrators can
manage their fabrics in the same manner that they always have.
 FCoE does not have a stateful gateway but simply maps Fibre Channel traffic onto lossless
Ethernet. This mapping results in performance benefits over technologies that require a
gateway.
 Provides significant savings in the amount of power that is expended per rack result when
FCoE is employed. By consolidating I/O, less power is expended on NICs and cables.
Companies also save power on the cooling that is expended to compensate for the heat
produced.
 FCoE is managed like Fibre Channel at the initiator, target, and switch level.
 FCoE is an extension of Fibre Channel onto lossless Ethernet fabric.
 FCoE is aligned with the FC-BB-4 model, standardized in FC-BB-5.
 FCoE is easy to understand and is based on the Fibre Channel model.
 FCoE is the same operational model.
 FCoE has the same host-to-switch and switch-to-switch behavior.
 FCoE uses the same techniques of traffic management, for example, in order delivery or
Fabric Shortest Path First (FSPF) load balancing.
 FCoE uses the same management and security models. Examples include world wide
names (WWNs), Fibre Channel IDs (FCIDs), hard and soft zoning, Domain ID Database,
and Registered State Change Notification (RSCN).

Ethernet Enhancements
This topic describes the enhancements to the Ethernet quality of service (QoS) to support Fibre
Channel traffic requirements.

Traditional Fibre Channel Fibre Channel over Ethernet

FC-4 Upper-Layer Mapping FC-4 Upper-Layer Mapping

FC-3 Generic Services FC-3 Generic Services

FC-2 Framing and Flow Control FC-2 Framing and Flow Control

FC-1 Encoding FCoE Logical Endpoint

FC-0 Physical Interface Ethernet Media Access Control

Ethernet Physical Layer

The FCoE logical endpoint (FCoE_LEP) is responsible for the encapsulation and de-
encapsulation that is necessary to transport Fibre Channel frames over Ethernet. The figure
shows that FCoE_LEP has the standard Fibre Channel layers starting with FC-2 and continuing
up the Fibre Channel Protocol stack. This arrangement gives the appearance to higher-level
system functions that the FCoE network is, in fact, a standard Fibre Channel network. This
allows all of the same tools that are used in native Fibre Channel to be used in an FCoE
environment. Below the FCoE_LEP are the standard Ethernet media and physical layers for 10
Gigabit Ethernet with enhancements that allow Ethernet to be lossless. Using the Ethernet
standards allows FCoE to leverage a significant amount of existing technology.

5-8 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• DCB is a collection of new IEEE standards that provide a Converged
Enhanced Ethernet.
- Priority Flow Control (PFC) IEEE 802.1Qbb
• Provides lossless delivery for selected CoSs
- Enhanced Transmission Selection (ETS) IEEE 802.1Qaz
• Bandwidth management and priority selection
- QCN IEEE 802.1au
• Congestion awareness and avoidance (optional)
- DCBX IEEE 802.1AB
• Protocol for exchanging parameters between DCB devices
• Leverages functions provided by LLDP
- FIP T11 BB-5 standard
• Enables the establishment of point-to-point virtual Fibre Channel links
over a multiaccess network such as Ethernet

Cisco DCB architecture is based on a collection of open standards, Ethernet extensions that are
developed through the IEEE 802.1 working group. These standards are designed to improve
and expand Ethernet networking and management capabilities in the data center. DCB helps
ensure delivery over lossless fabrics and I/O convergence onto a unified fabric. Each element of
this architecture enhances the DCB implementation and creates a robust Ethernet infrastructure
to meet data center requirements now and in the future.
IEEE DCB builds on the strengths of classical Ethernet, adds several crucial extensions to
provide the next-generation infrastructure necessary for data center networks, and delivers
unified fabric. The main features of the DCB architecture contribute to a robust Ethernet
network capable of meeting the current growing application requirements and responding to
future data center network needs.
Different organizations created different names to identify the collection of the specifications,
all based on the same core specifications. IEEE has used the term “Data Center Bridging
(DCB).” IEEE typically calls a standard specification by a number, for example, IEEE 802.1az.
IEEE did not have a way to identify the group of specifications with a standard number, so the
organization grouped the specifications into DCB.
The term “Converged Enhanced Ethernet (CEE)” was coined by IBM, again to reflect the core
group of specifications, to gain consensus among industry vendors (including Cisco) as to what
a Version 0 list of the specifications would be, before they all become standards.
Cisco uses the term “Data Center Ethernet (DCE)” to refer to its architecture for a next-
generation Ethernet for the data center. Cisco Data Center Ethernet is a superset of the CEE and
DCB proposals, using the same three specifications. Those specifications are Priority-based
Flow Control (PFC), Enhanced Transmission Selection (ETS), and DCBX. In addition, Cisco
includes Layer 2 multipoint processor (L2MP), lossless fabric, and congestion notification.
The DCE designation will be used when a Cisco product or Cisco partner product includes
those specifications and meets the requirements as defined by the Cisco Data Center Ethernet
architecture.

Ethernet PAUSE:

Transmit Frame

STOP PAUSE

Fibre Channel Buffer-to-Buffer Credits:

R_RDY

Transmit Frame

With traditional Link Level Flow Control, a PAUSE capability exists that allows the receiving
port to send a PAUSE when its buffers start to fill. However, the PAUSE pauses the whole link
so all frames are stopped. Unfortunately, the pause may arrive too late and some frames may
arrive after all buffer space has run out. These frames will then be dropped, so even though
Ethernet has a basic flow control capability, Ethernet is not guaranteed to be lossless.
Fibre Channel has a different flow control mechanism. The receiving port is always in control
and sends a buffer credit with a receiver ready (R_RDY) to signal that the port has a frame
buffer available. When the transmitting port receives a buffer credit, the port can then send a
frame to the receiving port. When the transmitting port runs out of credits, the port must stop
transmitting.
This credit-based flow control mechanism ensures that Fibre Channel never loses a frame and
therefore is a lossless protocol.

5-10 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Enables lossless OR lossy fabrics for each CoS
• PAUSE sent per virtual lane when the limit of the buffers is exceeded

Transmit Queues Receive Buffers

Ethernet Link
One One Control Traffic

Two Two IPC Traffic

Three Three FCoE

Four Four
Eight
Virtual
Five Five
Lanes
Six STOP PAUSE Six

Seven Seven

Eight Eight

PFC is an enhancement to the existing pause mechanism in Ethernet. The current Ethernet
pause option stops all traffic on a link and works, essentially, as a link pause for the entire link.
PFC creates eight separate virtual links on the physical link and allows any of these links to be
paused and restarted independently. This capability enables the network to create a no-drop
class of service (CoS) for an individual virtual link. PFC also allows differentiated QoS policies
for the eight unique virtual links.
When Converged Enhanced Ethernet is used to carry multiple protocols across the same 10
Gigabit Ethernet link, the traditional pause mechanism cannot be used. This process is because
the pause mechanism would pause all the traffic flows and still would not guarantee a lossless
delivery. So, PFC creates eight channels or CoSs all sharing the same 10 Gigabit Ethernet link.
This capability lets individual channels pause and regulate the flow of each traffic flow.
Note that not all upper-level protocols require or desire a lossless fabric. TCP for example
requires packet loss for congestion management.

• Enables intelligent sharing of bandwidth between traffic classes
• 802.1Qaz Enhanced Transmission Selection

Offered Traffic 10 Gigabit Ethernet Link Realized Traffic Utilization

3G/s 3G/s 2G/s 3G/s HPC Traffic 2G/s

3G/s

3G/s Storage Traffic 3G/s

3G/s 3G/s 3G/s 3G/s

3G/s 4G/s 6G/s 3G/s LAN Traffic 5G/s

4G/s

t1 t2 t3 t1 t2 t3

ETS is also called priority grouping. Eight distinct virtual link types and CoSs can be created
by implementing PFC. ETS also has the advantage of having different traffic classes that are
defined within the different PFC traffic types.
ETS enables these differentiated treatments within the same priority classes of PFC.
ETS provides prioritized processing that is based on bandwidth allocation, low latency, or best
effort, resulting in per-group traffic class allocation.
For example, an Ethernet class of traffic may have a high-priority designation and a best effort
within that same class. ETS allows differentiation between traffic of the same priority class,
thus creating priority groups.

5-12 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Autonegotiation of capabilities between DCB devices
- Class-based Flow Control (PFC)
- Bandwidth Management (ETS)
- Congestion Notification (BCN/QCN)
- Logical Link Down
- Network Interface Virtualization
(NIV) Enhanced
Ethernet Links

DCBCXP Enhanced Ethernet

Links with Partial
Enhancements

DCBCXP

Legacy
Ethernet
Links

Legacy Ethernet DCBCXP

Converged Enhanced
Network Ethernet Cloud

DCBX is a discovery and capability exchange protocol that is used by devices that are enabled
for Cisco Data Center Ethernet to exchange configuration information. The following
parameters of the Data Center Ethernet features can be exchanged:
 Priority groups in ETS
 PFC
 Congestion notification (Backward Congestion Notification [BCN]/Quantized Congestion
Notification [QCN])
 Application types and capabilities
 Logical link down to signify the loss of a logical connection between devices even though
the physical link is still up
 Network interface virtualization (NIV)

Devices need to discover the edge of the enhanced Ethernet cloud:

 Each edge switch needs to learn that the edge switch is connected to a traditional switch.
 Servers need to learn if the servers are connected to an enhanced Ethernet device.
 Within the Enhanced Ethernet cloud, devices need to discover the capabilities of their
peers.
 DCBX Protocol (DCBXP) utilizes the data link layer discovery protocol and manages local
operational configuration for each feature.
 Link partners can choose supported features and their willingness to accept configuration
from a peer.

FCoE Protocol
This topic introduces and describes the FCoE protocol.

Requirements:
• Encapsulation of full Fibre Channel
frame into Jumbo Ethernet frame FCoE Traffic
10 Gigabit • SCSI
• Mapping of Fibre Channel pWWN • FICON
address to Ethernet MAC address Ethernet Link
Other
• FIP Networking
Traffic
• Lossless delivery of Fibre Channel • TCP/IP
frames • CIFS
• NFS
• iSCSI
Normal Ethernet Frame, EtherType = FCoE
Control Information: Version, Ordered Sets (SOF, EOF)
Same As a Physical Fibre Channel Frame
Ethernet

Channel
Header
Header

Header

CRC

EOF

FCS
FCoE

Fibre Channel Payload

Fibre

FCoE is a new protocol that is based upon the Fibre Channel layers defined by the ANSI T11
committee. This protocol replaces the lower layers of the Fibre Channel with CEE.
Minimum requirements for FCoE are as follows:
 Lossless delivery of Fibre Channel frames
 Support for jumbo frames, so that an entire Fibre Channel frame (length 2180 bytes) can be
carried in the payload of a single Ethernet frame
 The mapping of Fibre Channel port WWN (pWWN) addresses to Ethernet MAC addresses
 A minimum 10-Gb/s Ethernet platform

FCoE traffic consists of a Fibre Channel frame that is encapsulated within an Ethernet frame
with EtherType 0x8906. The Fibre Channel frame payload may in turn carry SCSI messages
and data, or in the future, fiber connectivity (FICON) for mainframe traffic.

SAN HBA
CNA with 10 Gigabit Ethernet
Enhanced Ethernet

PCIe
Link
4-Gb FC
Fibre Channel
4-Gb FC
Fibre Channel

PCIe
Link
10 GEE

10 GEE Ethernet

PCIe
Link

1 GE Ethernet
1 GE

LAN NIC

A converged network adapter (CNA) combines a NIC and Fibre Channel HBA into one card.
The traditional HBA manufacturers, QLogic and Emulex, also provide CNAs. First-generation
CNAs had three ASICs, which were 10 Gigabit Ethernet, Fibre Channel, and a convergence
ASIC provided by Cisco.
Second-generation CNAs were built using one custom ASIC to reduce power consumption and
reduce unit costs.

Combined Ethernet and Fibre Channel switch in one box
1. Encapsulated Fibre Channel frames arrive at Ethernet port.
- Destination MAC address is the FCoE_LEP.
2.The Ethernet header is removed.
3.The Fibre Channel frame is forwarded to the destination FCID address.

FC
Port
Fibre Channel Switch SAN
Traffic
FC
FCoE_LEP Port

FC
Port
Ethernet Switch
FC
Port

Ethernet EthernetLAN Ethernet Ethernet FCoE Ethernet

Ethernet Ethernet Ethernet
Port Port Port Port Port Port Port Port
Traffic Traffic

An FCoE Forwarder (FCF) is a switch that contains both a Fibre Channel switch and an
Ethernet switch. One or more FCoE_LEPs are used to attach the Fibre Channel switch to the
Ethernet switch. There is one universal MAC address per FCoE_LEP. The FCoE_LEP is
responsible for encapsulating and de-encapsulating Fibre Channel frames for FCoE transport.
Fibre Channel traffic flows into one end of the LEP, while FCoE flows into the other end.

Source MAC Address

(IEEE 802.1Q Tag)
EtherType = FCoE Version Reserved
Reserved
Reserved
Reserved Start of Frame (SOF)

Encapsulated Fibre Channel Frame (with CRC)

End of Frame (EOF) Reserved

Frame Check Sequence (FCS)
4 Bytes
Normal Ethernet Frame, EtherType = FCoE
Control Information: Version, Ordered Sets (SOF, EOF)
Same As a Physical Fibre Channel Frame
Channel
Ethernet

Header
Header

CRC

FCS
EOF
Header

FCoE

Fibre Channel Payload

Fibre

FCoE is encapsulated in an Ethernet frame with a dedicated EtherType, 0x8906. That packet
has a 4-bit version field. The other header fields in the frame (the source and destination MAC
addresses, virtual LAN [VLAN] tags, and frame markers) are all standard Ethernet fields.
Reserved bits pad the FCoE frame to the IEEE 802.3 minimum frame length of 64 bytes.
A Fibre Channel frame consists of 36 bytes of headers and up to 2112 bytes of data for a total
maximum size of 2148 bytes. The encapsulated Fibre Channel frame has all the standard
headers, which allow the frame to be passed to the storage network without further
modification. To accommodate the maximum Fibre Channel frame in an FCoE frame, the class
FCoE is defined with a default maximum transmission unit (MTU) of 2240 bytes.
The Ethernet frames that are sent by the switch to the adapter may include the IEEE 802.1Q
tag. This tag includes a field for the CoS value that is used by PFC. The IEEE 802.1Q tag also
includes a VLAN field.
The FCF expects frames from a FIP T-11-compliant CNA to be tagged with the VLAN tag for
the FCoE VLAN. Frames that are not correctly tagged are discarded.
The switch expects frames from a pre-FIP CNA to be a priority that is tagged with the FCoE
CoS value. The switch will still accept untagged frames from the CNA.

FCoE Addressing
This topic describes how an FCoE node (E Node) can acquire a unique MAC address using a
fabric-provided MAC address (FPMA).

• The FCoE standard requires support for FPMA by FCoE devices.

• The MAC address is assigned for each FCID and is consistent with the Fibre
Channel model.
• FC-MAP uses Organizationally Unique Identifiers (OUIs) with universal/local (U/L)
bit = 1 (local addressing).
• Multiple FC-MAPs are supported (one per physical SAN).
• No table is needed in FCF for translation of MAC addresses and Fibre Channel
addresses.
• Multiple MACs may be needed for NPIV.

24 Bits 24 Bits
FC-MAP FCID
(0E-FC-00) 07-08-09

MAC FC-MAP FCID

FPMAs create a direct mapping between the FCID that is assigned by the Fibre Channel Fabric
Services in the FCF and the Ethernet MAC address that is used as the E Node station address.
The 48-bit Ethernet MAC address consists of a fabric-wide FCoE MAC address prefix (FC-
MAP) value in the high-order 24 bits, with the assigned FCID in the low-order 24 bits. Fibre
Channel traffic can be encapsulated directly in FCoE frames with no table lookup. This process
is due to the fact the FC-MAP is a known quantity, and the destination ID (DID) in the Fibre
Channel frame (sent by FCF response during fabric login [FLOGI]) supplies the FCID.
FPMAs can create problems. The FCoE MAC address to be used by the station cannot be
determined until a Fibre Channel FLOGI is sent. The FCoE MAC address is therefore not
available for use as the source MAC address in the FLOGI itself. Furthermore, a mechanism
must be identified to determine the MAC address of the FCF so that the destination MAC
address for the FLOGI is known.
This mechanism is FIP, which is defined in the FCoE standards.

Domain ID: 3 Domain ID: 5

FCID: 01:01:01

Fibre Channel SAN

Domain ID: 1

FCID: 01:01:02

FCF-MAC(A) MAC(Host) Ethernet LAN

FCoE Forwarder CNA Source MAC = MAC(Host)

(Cisco Nexus 5000) Destination MAC = FCF-MAC(A)
Source FCID = 01:01:02
Destination FCID = 05:01:01

Currently, FCoE standards support the use of FCoE between an E Node and a directly attached
FCF that is deployed as an access switch.
The figure shows a Fibre Channel network (in green) that has been extended over an Ethernet
network (in blue) by using FCoE. The host with an FCoE node is attached directly to a switch
acting as an FCF, which is diagrammed as separate Ethernet and Fibre Channel switches
connected by the FCoE_LEP. Notice that the FCF and the E Node both have Ethernet and Fibre
Channel characteristics. You can see this function through the Fibre Channel addressing that is
associated with each device.

FCoE Initialization Protocol
This topic introduces FIP and how the protocol facilitates an FCoE node performing a FLOGI.

• FIP is used to perform the following:

- FIP VLAN Discovery (optional)
• Uses the native VLAN
• Discovers the FCoE VLAN that will be used by all other FIP protocols
- FIP FCF Discovery
• Sends a Discovery Solicitation message
• Used to discover the FCF MAC address used for subsequent logins
- FCoE Virtual Link Instantiation
• Encapsulates Fibre Channel FLOGI, FDISC, LOGO, and ELP messages
- FCoE Virtual Link Maintenance
• FIP periodically sends messages between the switch and CNA to ensure
the connection is still valid.
FIP FIP

FCoE Initiation Protocol

FIP allows the switch to discover and initialize FCoE-capable entities that are connected to an
Ethernet LAN. Cisco Nexus Operating System (NX-OS) switches support the CEE-DCBX
protocol for T11-compliant Gen-2 CNAs.

FIP Virtual Link Instantiation

FIP is used to perform device discovery, initialization, and link maintenance. FIP performs the
following protocols:
 FIP VLAN Discovery: Discovers the FCoE VLAN that will be used by all other FIP
protocols as well as by the FCoE encapsulation for Fibre Channel payloads on the
established virtual link. FIP VLAN discovery occurs in the native VLAN used by the
initiator or target to exchange Ethernet traffic. The FIP VLAN discovery protocol is the
only FIP protocol running on the native VLAN. All other FIP protocols run on the
discovered FCoE VLANs. You cannot map VLAN 1 or the native VLAN to an FCoE
VLAN.
 FIP FCF Discovery: When a FCoE device is connected to the fabric, the device sends out
a Discovery Solicitation message. An FCF or a switch responds to the message with a
Solicited Advertisement that provides an FCF MAC address to use for subsequent logins.
 FCoE Virtual Link instantiation: FIP defines the encapsulation of FLOGI, fabric
discovery (FDISC), logout (LOGO), and exchange link parameters (ELP) frames along
with the corresponding reply frames. The FCoE devices use these messages to perform a
fabric login.
 FCoE Virtual Link maintenance: FIP periodically sends maintenance messages between
the switch and the CNA to ensure that the connection is still valid.

5-20 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• A device directly connected to an FCF needs to obtain its dynamic
address.
- Sends a Discovery Solicitation message
- Obtains an FPMA from the attached switch, using FIP
- FIP creates Ethernet tunnels for FCoE logical links such as the following:
• VN-VF between CNA and FCF
• VE-VE between FCFs

FIP FIP

Obtain VN VF VE VE
FPMA
FCF FCF
VN-VF = Virtual node port to virtual fabric port
VE-VE = Virtual expansion port to virtual expansion port

FIP allows the switch to discover and initialize FCoE-capable entities that are connected to an
Ethernet LAN. Two versions of FIP are supported by the FCF:
 FIP: The CEE-DCBX protocol supports T11-compliant second-generation CNAs.
 Pre-FIP: The Cisco, Intel, Nuova Data Center Bridging Exchange (CIN-DCBX) protocol
supports first-generation CNAs. Pre-FIP is not supported by Cisco Nexus 7000 or MDS
9000 Series switches.

• Host solicitation is sent, indicating the type of MAC supported by the
CNA - SPMA, FPMA, or both
• Host solicitation is received by the FCF
• EtherType FIP

Domain ID: 10
Host Solicitation: FC-MAP: 0E:FC:00
To: All FCFs SAN
From: MAC(H2) FIP

FP=1
H2
F=0
FCF1 Classic
S=0 Ethernet
LAN

FIP starts with a multicast solicitation from the host. The solicitation includes bits that indicate
if the host can use FPMAs or server-provided MAC addresses (SPMAs). The EtherType of this
frame is FIP. This frame originates from a host, so the F bit is set to 0. This frame is not a
solicitation response, so the S bit is set to 0. The FP bit is set to 1, because this device is
capable of using FPMAs.
The most predominant role for MDS specific to FCoE will be single hop, as described by end
node – 5K – MDS – FC LUN.
Multihop is realistically relevant only when connecting MDS to FCoE-enabled storage devices.

5-22 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Switch provides the fabric-unique FC-MAP (used for the first 3 bytes of
the FPMA)
- FP = 1 indicates FPMA-capable
- F = 1 indicates the frame was generated by an FCF
- EtherType = FIP

Domain ID: 10
FCF Advertisement: FC-MAP: 0E:FC:00
To: MAC(H2) SAN
From: MAC(FCF1) FIP reply

FP=1
H2
F=1
FCF1 Classic
S=1 H2 FCF List: Ethernet
MAC(FCF1) – 0E:FC:00 LAN
FC-MAP=0E:FC:00

The FCF replies by sending an advertisement. The F bit is set to 0, indicating the FIP frame is
from an FCF. The S bit is set to 1, indicating that this reply is a response to a solicitation.

• Host has discovered the FCF Logical End-Point (LEP).

• Host sends a Fabric Login (FLOGI) to the Fibre Channel switch .
- At this point, host is still using the burned-in MAC address.
- EtherType = FIP

FCoE Virtual Link Domain ID: 10

Instantiation: FC-MAP: 0E:FC:00
To: MAC(FCF1) SAN
FLOGI
From: MAC(H2)

FLOGI to FF:FF:FE H2
From 00:00:00
FCF1 Classic
H2 FCF List: Ethernet
MAC(FCF1) – 0E:FC:00 LAN

Now that the host has an FC-MAP (a unique per-fabric MAC address prefix), the host can
continue with the FLOGI to finalize its FPMA. FLOGI frames are sent to the FCID FF.FF.FE,
which is the well-known address for fabric login.

• FCF provides an FCID for the device.
• FCF replies using the burned-in MAC address of the host.
- EtherType = FIP

Domain ID: 10
FLOGI Accept frame: FCID: 01:00:01 FC-MAP: 0E:FC:00
To: MAC(H2) SAN
Accept
From: MAC(FCF1)

ACC to 10:00:01 H2
From FF:FF:FE Classic
FCF1
H2 FCF List: Ethernet
MAC(FCF1) – 0E:FC:00 LAN

The switch assigns an FCID and responds with this ID. This FCID is appended to the
previously acquired FC-MAP to create the FPMA. The FPMA is used for future
communication.

• Host uses the FPMA for subsequent transmissions.

• FPMA is the FCID obtained through FLOGI and appended to the FC-
MAP provided by the FCF.
• EtherType FCoE

Domain ID: 10
Advertisement: FC-MAP: 0E:FC:00
To: MAC(FCF1) SAN
From: OE:FC:00 10:00:01 FLOGI

FC Frame……..
H2

FCF1 Classic
H2 FCF List: Ethernet
MAC(FCF1) – 0E:FC:00 LAN

Now that the device has a complete FPMA (FC-MAP + FCID), the device can communicate on
the fabric using FCoE frames.

5-24 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• Some of the consolidation benefits of using Unified I/O and FCoE are
reduced cabling, reduced number of switches at the access layer, lower
cost of ownership, and easier management.
• To support FCoE, the server requires either a CNA or an Intel Oplin 10
Gigabit Ethernet NIC and the FCoE software stack.
• Enhancements to the Ethernet QoS support Fibre Channel traffic
requirements.
• The ENode acquires a unique MAC address from the FCF and
incorporates its FCID.
• FIP is used to initialize the link and identify the FCF.

Configuring FCoE on Cisco

MDS 9500 and 9700 Series
Multilayer Directors
Overview
This lesson describes how to configure the Cisco MDS 9500 and 9700 Series Multilayer
Director Switch modules to enable Fibre Channel over Ethernet (FCoE) traffic in a SAN
environment.

Objectives
Upon completing this lesson, you will be able to configure the Cisco MDS 9500 and 9700
Series Directors for FCoE. This ability includes being able to meet these objectives:
 Describe the use of FCoE modules on the Cisco MDS 9500 and 9700 Series Directors
 Describe the support of virtual expansion (VE) interfaces
 Describe how FCoE can be configured to enable FCoE frames to traverse a unified fabric
Fibre Channel over Ethernet
This topic describes how FCoE frames are forwarded from the FCoE Forwarder (FCF) to
another FCF within a unified fabric.

• FCoE does not require a license on the Cisco MDS 9500 Series.
• You can only enable FCoE on an FCoE module.
• Do not install the FCoE module in the same switch where service
modules are installed
• You cannot enable FCoE on VLAN 1 (default) or the native VLAN.
• Cisco MDS 9500 supports only Generation-2 CNAs.
• Cisco MDS 9500 does not support LAN traffic and only accepts and
processes FCoE frames.
• The QoS policy must be the same on all Cisco FCoE switches in the
network.
• The MDS 9500 FCoE module requires MDS 9500 Supervisor-2A.
• The MDS 9700 FCoE module requires MDS 9710 Supervisor-1.

The figure describes the guidelines and limitations that should be considered when configuring
FCoE on MDS 9500 and MDS 9700 Series Director chassis.

5-28 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Three modules provide support for FCoE.
• MDS 9500 8-Port 10-Gb/s FCoE • MDS 9710 FCoE Module
Module - Supports deployment of FCoE in
- Multihop FCoE module for the the Cisco MDS 9710 Series
core chassis
- 48-Port 10 Gigabit Ethernet FCoE
Module

• Supervisor-2A Module
- Supports deployment of FCoE in
1HCY14
the Cisco MDS 9500 Series
chassis
- Nondisruptive upgrade from
Supervisor-2
- 2 GB of memory

Generation-4 FCoE Modules

Two fourth-generation modules provide support for multihop FCoE in Cisco MDS 9500 Series
chassis.
Developed for Cisco MDS 9500 Series Fibre Channel Directors, the Cisco MDS 9000 10-Gb/s
8-Port FCoE Module helps consolidate the network in the core to reduce complexity. The
module offers enterprise-class systems and topologies for FCoE deployments. By providing a
converged fabric, the module helps core Fibre Channel capacity to connect to the FCoE access
layer, while preserving storage network services.
The Cisco MDS 9500 Series Supervisor-2A Module delivers the latest advanced switching
technology. The module uses proven Cisco Nexus Operating System (NX-OS) software to
power a new generation of scalable and intelligent multilayer switching solutions for SANs.
Cisco NX-OS Release 5.2(1) and later is required to support FCoE.
The Cisco MDS 9500 Series Supervisor-2A Module provides these benefits:
 Enables converged, high-performance, intelligent, resilient, scalable, and secure multilayer
SAN switching solutions.
 Supports deployment of FCoE in the Cisco MDS 9500 Multilayer Director chassis.
 Integrates multiprotocol switching and routing, intelligent SAN services, and storage
applications onto highly scalable SAN switching platforms.
 Is supported in the Cisco MDS 9506 and 9513 Multilayer Director chassis.

MDS Director chassis
• Fibre Channel frames are contain FCoE modules
forwarded, hop by hop, from
FC
the server to Fibre Channel FC
Storage
storage: LAN Array SAN
- From CNA to Cisco Nexus Core
5000 FCF MDS

- From Cisco Nexus 5000 to Fiber Channel

VE-port
Cisco Nexus 7000 Bridge

- From Cisco Nexus 7000 to FCoE

Aggregation Layer
MDS 9500 Nexus 7000 FCF
- From MDS Director to
VE-port
Fibre Channel Storage
Array
- MDS is a Fibre Channel Access Layer FCF
Bridge, not a forwarder Nexus 5000
VF-port
FCoE

VN-port

In the example of multihop FCoE, frames are forwarded hop by hop from the converged
network adapter (CNA) in the server, through the Cisco Nexus 5000 FCF at the access layer,
through the Cisco Nexus 7000 Series switch at the aggregation layer, and on to the FCoE line
card module in the MDS 9500 Series Director. (Both the Cisco Nexus 5000 and 7000 Series do
multihop.)
The configuration of FCoE in and Fibre Channel out equals a Fibre Channel Bridge (FCB).
If the storage connects to the MDS via Fibre Channel, then it does not matter if the connection
coming into the MDS was single-op or multihop FCoE. From the perspective of the MDS, the
configuration is the same. MDS is a FCoE to Fibre Channel bridge.
 The access layer Cisco Nexus 5000 Series switches are FCoE FCFs.
 The aggregation layer Cisco Nexus 7000 Series switches are FCoE FCFs.
 The core MDS Director chassis are not FCoE FCFs.

5-30 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Multi-hop FCoE involving the
MDS MDS Director chassis FCoE
contain FCoE modules Storage
- CNA > Cisco Nexus 5000/7000 > Array
SAN
MDS > a storage device with an FC

FCoE adapter or stack

• In this case, the MDS is an FCF Core FCoE
MDS VF-port
9500/9700

FCF
VE-port
FCoE
VE-port

Access Layer
Nexus 5000 FCF

FCoE VF-port

VN-port

In this multihop FCoE example, frames are forwarded hop-by-hop from the CNA in the server,
through the Cisco Nexus 5000 Series Switch FCF at the access layer and on to the FCoE line
card module in the MDS 9500 Series Director. (The Cisco Nexus 5000 Series Switch and MDS
do multihop.)
Frames are forwarded hop-by-hop from the CNA through the Cisco Nexus 5000 Series Switch
FCF and on to the FCoE line card module in the MDS 9513 Multilayer Director and continue to
the storage with the FCoE software stack. (The MDS is an FCF doing multihop.)

VE Interfaces
This topic describes the new support of VE interfaces that provides for multihop FCoE
architectures.

• VE Ports refer to an FCoE ISL that runs between two FCoE-capable

switches:
- Defined by the FC-BB-5 standard as an entity that emulates a Fibre Channel
E Port over a non-Fiber Channel link
- Cisco NX-OS provides mechanisms to create VE Port-to-VE Port virtual links
• VE Ports offer the following benefits:
- Expand the FCoE fabric beyond a first-hop access solution
- Provide multihop FCoE connections between FCoE-capable switches

E Ports in Native Fibre Channel VE Ports in FCoE

E E VE VE

FC FCoE

A virtual expansion port (VE Port) is a port that emulates an expansion port (E Port) over a
non-Fibre Channel link. VE Port connectivity between FCFs is supported over point-to-point
links. These links can be individual Ethernet interfaces or members of an Ethernet port channel
interface.
VE Ports have the following guidelines:
 Automode on the virtual Fibre Channel (vFC) is not supported.
 VE Port trunking is supported over FCoE-enabled virtual LANs (VLANs).

5-32 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• A vFC interface is created after
receiving a unique FIP-FLOGI FC
FCoE

(VF Port) or FIP-ELP (VE Port).

• VE Ports are supported on or VN

between the Cisco Nexus 5000

VF
and Cisco Nexus 5500, 6000,
7000 Series and MDS Series.
• VE Ports are bound to the
VE VE
underlying 10-G infrastructure
VE VE
- Can be bound to a single 10
Gigabit Ethernet port or to a port
channel interface consisting of VF
multiple 10- Gigabit Ethernet links
VN

A vFC interface is created after receiving a unique FCoE Initialization Protocol (FIP)-fabric
login (FLOGI) (VF Port) or FIP-Exchange Link Parameters (ELP) (VE Port). This facility
creates a unique VF to VN or VE to VE link with the FCoE node (E Node) or peer FCF. Each
vFC interface is associated with only one virtual SAN (VSAN).
A vFC interface must be bound to an interface before the interface can be used. The binding is
to a physical Ethernet interface when the CNA is directly connected to the switch.

FCoE Configuration on the MDS 9500 and 9700
Series Multilayer Directors
This topic describes how FCoE modules can be configured to enable FCoE traffic to traverse a
unified fabric.

• FCoE feature is automatically DCNM-SAN Client

enabled when an FCoE module
is installed. Example in Device
• Additional features required Manager shows no
FCoE module
include the following:
currently
- LACP (a best practice)
- lldp

switch# config
switch(config)# feature lacp
Switch(config)# feature lldp

switch# show fcoe

Global FCF details
FCF-MAC is 00:0d:ec:6d:95:00
FC-MAP is 0e:fc:00
FCF Priority is 128
FKA Advertisement period for FCF is 8 seconds

To configure FCoE, enable the Link Aggregation Control Protocol (LACP) feature using the
feature lacp global configuration command.
The command feature lldp negotiates Data Center Bridging (DCB) point to point.
The Cisco MDS Director chassis switch automatically enables the FCoE feature when an
FCoE-capable module is present in the chassis. You must enable supporting features and the
appropriate quality of service (QoS) policy for FCoE.

5-34 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
• Default policy and classes:
- default-nq-8e-policy COS 3 (FCoE) is dropped
- default-nq-7e-policy COS 3 is NO DROP (required for FCoE)
- default-nq-6e-policy COS 3 is NO DROP, 4 is PRIORITY NO DROP
- default-nq-4e-policy COS 1-3 is NO DROP 4 is PRIORITY NO DROP

switch# config
switch(config)# system qos
switch(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
switch(config-sys-qos)# end
switch# copy run start

switch# show class-map type network-qos

switch# show policy-map system What’s currently active (best)
switch# show policy-map type What’s configured

• jumbo MTU requires the global command <system jumbomtu 9216>

The system qos command enables the QoS policy that supports FCoE traffic. The policy-name
default is default-nq-6e-policy.

1. Create appropriate VSANs.
switch# config
switch(config)# vsan database
switch(config-vsan-db)# vsan 200

2. Create VLANs and associate with the appropriate VSANs.

switch(config)# vlan 200
switch(config-vlan)# fcoe vsan 200

3. Create a DCB port channel between switches.

switch# interface port-channel 777
switch(config-if)# switchport mode trunk
switch(config-if)# no shutdown
VSAN
200
PC 777 VLAN
200 FCoE

Eth VE VE Eth Eth VF

Port Port Port

This example shows how to establish a VE-to-VE Inter-Switch Link (ISL) between switches.
Step 1 Create the appropriate VSANs.
Step 2 Create the appropriate VLANs and associate the VLANs with the appropriate
VSAN.
Step 3 Create the DCB port channel that will be used to connect the two switches.

5-36 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
4. Create a VFC interface that will be associated with the port channel.
switch(config)# interface vfc 3
switch(config-if)# bind interface port-channel 777
switch(config-if)# switchport mode E
switch(config-if)# switchport trunk allowed vsan 200
switch(config-if)# no shutdown
switch(config-if)# vsan database
switch(config-vsan-db)# vsan 200 interface vfc 3

5. Add Ethernet interfaces to the DCB port channel.

switch(config)# interface e1/1-2
switch(config-if-range)# switchport mode trunk
switch(config-if-range)# channel-group 777 mode active
switch(config-if-range)# no shutdown
switch(config-if-range)# exit

Step 4 Create the vFC interface that will be associated with the port channel, and then
specify the interface as an E Port and allow the appropriate VSANs.
Step 5 Add the Ethernet interfaces to the DCB port channel.

The figure describes the Cisco Data Center Network Manager (DCNM)-SAN Client FCoE
Configuration Wizard.

• The example shows how to display the FCoE database.
switch# show fcoe database
--------------------------------------------------------------INTERFACE FCID
PORT NAME MAC ADDRESS
--------------------------------------------------------------vfc3 0x490100
21:00:00:1b:32:0a:e7:b8 00:c0:dd:0e:5f:76

• The example shows how to display the FCoE interface settings.

switch# show interface ethernet 1/3 fcoe
Ethernet1/1 is FCoE UP
vfc3 is Up
FCID is 0x490100
PWWN is 21:00:00:1b:32:0a:e7:b8
MAC addr is 00:c0:dd:0e:5f:76

show vlan fcoe

show fcns database {vsan ##} Shows end points (initiator, target)
show lldp interface ethernet x/x
show lldp dcbx interface ethernet x/x
show system internal dcbx info interface ethernet x/x
show platform software fcoe_mgr info interface vfc<id>
show interface Ex/x flowcontrol non PFC Pause frames
show interface Ex/x priority-flow-control PFC pause frames

This example shows how to display the FCoE database and how to display the FCoE settings
for an interface.

5-38 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Summary
This topic summarizes the key points that were discussed in this lesson.

• The MDS 9500 and MDS 9700 Series directors support FCoE modules
and FCoE configuration.
• Multiple FCoE FCFs may be viewed as multihop FCoE.
• The MDS FCoE module behaves as an FCB with FCoE in and Fibre
Channel out. The MDS FCoE module behaves as an FCoE FCF with
FCoE in and FCoE out.

5-40 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Summary
This topic summarizes the key points that were discussed in this module.

• FCoE is a protocol that allows Fibre Channel frames to be encapsulated

within Ethernet frames in a lossless unified fabric. Some of the
consolidation benefits of using a unified fabric and FCoE are reduced
cabling, reduced number of switches at the access layer, lower cost of
ownership, and easier management.
• The Cisco MDS 9500 and 9700 Series Multilayer Director FCoE
modules can participate in a multihop topology by acting as FCFs. If the
frame entering the MDS 9500 or MDS 9700 Series Multilayer Director is
FCoE and the frame leaving the MDS is FCoE (to FCoE storage), then
the MDS is participating in multihop. If the frame entering the MDS 9500
or MDS 9700 Series Multilayer Director is FCoE and the frame leaving
the MDS is Fibre Channel (to FC Storage), then the MDS is not an
FCoE FCF and is not participating in multihop. In this case, the MDS is a
bridge.

5-42 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Module Self-Check
Use the questions here to review what you learned in this module. The correct answers and
solutions are found in the Module Self-Check Answer Key.
Q1) Which two adapters support FCoE on the server? (Choose two.) (Source: Describing
FCoE)
A) 10 Gigabit Ethernet NIC
B) 10 Gigabit Ethernet CNA
C) 1 Gigabit Ethernet CNA
D) 1 Gigabit Ethernet NIC
E) 10 Gigabit Ethernet Intel Oplin NIC
F) 10 Gigabit Ethernet Intel Oplin NIC, with FCoE software installed on the
server
Q2) How does an FCoE E Node acquire a MAC address to use when forwarding Fibre
Channel frames to the Cisco Nexus 5000 Series Switch? (Source: Describing FCoE)
A) The node uses a Server-Provided MAC Address.
B) The node uses the MAC address of the CNA or NIC installed.
C) The node uses the MAC address that is generated by the administrator.
D) The node uses a Fabric-Provided MAC Address.
Q3) Which feature of Cisco Data Center Ethernet supports CoS flow control? (Source:
Describing FCoE)
A) bandwidth management
B) DCBX
C) congestion notification
D) priority flow control
Q4) How does an FCoE E Node log into the fabric? (Source: Describing FCoE)
A) Using the PRLI protocol.
B) Using the FIP protocol.
C) FCoE E Nodes do not log into the fabric. The nodes communicate directly with
other FCoE devices.
D) The FCoE E Node does not need to log in. The FCF automatically assigns an
address.
Q5) Which three networks are converged in a unified fabric? (Choose three.) (Source:
Configuring FCoE on Cisco MDS 9500 and 9700 Series Multilayer Directors)
A) WAN
B) MAN
C) LAN
D) SAN
E) HPC
F) CPC
G) IBC

Q6) Which module and which software version is required to support FCoE on the Cisco
MDS 9500 Series Multilayer Directors? (Choose two.) (Source: Configuring FCoE on
Cisco MDS 9500 and 9700 Series Multilayer Directors)
A) Supervisor-2 module
B) Supervisor-2A module
C) Supervisor-2F module
D) Cisco NX-OS Release 5.0(1) and later
E) Cisco NX-OS Release 5.1(1) and later
F) Cisco NX-OS Release 5.2(1) and later
Q7) Which license must be installed for correct operation of FCoE on the Cisco MDS 9500
Series Multilayer Director? (Source: Configuring FCoE on Cisco MDS 9500 and 9700
Series Multilayer Directors)
A) Enterprise License
B) Advanced License
C) Advanced Fabric License
D) FCoE does not require an additional license.
Q8) Which topology supports the FCoE Forwarder behavior of the MDS 9513 Multilayer
Director with an FCoE module? (Source: Configuring FCoE on Cisco MDS 9500 and
9700 Series Multilayer Directors)
A) Server with HBA directly attached to the MDS 9513 Multilayer Director
B) Server with CNA and Fibre Channel storage directly attached to MDS 9513
Multilayer Director
C) Cisco Nexus 7000 Series Switch and FCoE storage directly attached to MDS
9513 Multilayer Director
D) Cisco Nexus 7000 Series Switch and Fibre Channel storage directly attached to
MDS 9513 Multilayer Director

5-44 Configuring Cisco MDS 9000 Series Switches (DCMDS) v2.0 © 2013 Cisco Systems, Inc.
Self-Check Answer Key
Q1) B, F
Q2) D
Q3) D
Q4) B
Q5) C, D, E
Q6) B, F
Q7) D
Q8) C

Cisco UCS Manager Troubleshooting Reference Guide
No ratings yet
Cisco UCS Manager Troubleshooting Reference Guide
122 pages
Implementing Cisco UCS Solutions - Second Edition
From Everand
Implementing Cisco UCS Solutions - Second Edition
Anuj Modi
No ratings yet
Ameron Calculation Manual For Bondstand GRE Pipe Systems
No ratings yet
Ameron Calculation Manual For Bondstand GRE Pipe Systems
16 pages
Cisco MDS Student Guide DCMDS
No ratings yet
Cisco MDS Student Guide DCMDS
482 pages
Cege0009 - PP2 10-12-2020
No ratings yet
Cege0009 - PP2 10-12-2020
3 pages
FL DCMDS 2.0.3 LG PDF
33% (3)
FL DCMDS 2.0.3 LG PDF
483 pages
Cisco MDS CLI Quick Reference v15
100% (1)
Cisco MDS CLI Quick Reference v15
2 pages
Cisco UCS
100% (1)
Cisco UCS
120 pages
Cisco MDS 32G HPE OSM Training - 062018 PDF
100% (1)
Cisco MDS 32G HPE OSM Training - 062018 PDF
101 pages
03 - ADMIN SC HW and Cabling 6.7C PDF
100% (2)
03 - ADMIN SC HW and Cabling 6.7C PDF
48 pages
How To Install ASA 9.1.5-16 Clustering/Security Contexts Enabled Appliance in GNS3 and EVE-NG
No ratings yet
How To Install ASA 9.1.5-16 Clustering/Security Contexts Enabled Appliance in GNS3 and EVE-NG
4 pages
Otis Engineering Center Gien Field Component Manual
100% (5)
Otis Engineering Center Gien Field Component Manual
13 pages
Cisco MDS Cookbook 31a
No ratings yet
Cisco MDS Cookbook 31a
272 pages
DCMDS20LG
100% (1)
DCMDS20LG
336 pages
DCUFI50SG Vol1
No ratings yet
DCUFI50SG Vol1
378 pages
Dcict Datacenter Part1 CCSP - Ir
100% (1)
Dcict Datacenter Part1 CCSP - Ir
364 pages
FL Dcmds 1.0.1 LG
No ratings yet
FL Dcmds 1.0.1 LG
386 pages
Cisco MDS 9000 Family Cook Book PDF
No ratings yet
Cisco MDS 9000 Family Cook Book PDF
244 pages
MDS Course Lab01
No ratings yet
MDS Course Lab01
28 pages
Cisco DCUCI v4.0 Student Guide Volume 2
No ratings yet
Cisco DCUCI v4.0 Student Guide Volume 2
346 pages
Unified Storage Management Final Lab Guide VNX Lab Guide 2014
100% (1)
Unified Storage Management Final Lab Guide VNX Lab Guide 2014
233 pages
Unity Implementation and Administration Lab Guide - Final PDF
100% (2)
Unity Implementation and Administration Lab Guide - Final PDF
218 pages
Cisco MDS 9000 Family NX-OS Fundamentals Configuration Guide
No ratings yet
Cisco MDS 9000 Family NX-OS Fundamentals Configuration Guide
244 pages
DCUCI-v6 1 0-SG
No ratings yet
DCUCI-v6 1 0-SG
242 pages
Vxrail Apresentação11
No ratings yet
Vxrail Apresentação11
15 pages
ES101CPX02007+ +VxRail+7.0.XXX+Concepts+-+Downloadable+Content
100% (1)
ES101CPX02007+ +VxRail+7.0.XXX+Concepts+-+Downloadable+Content
67 pages
Cisco Mds
No ratings yet
Cisco Mds
12 pages
SAN Management - Lab Guide 3.3
No ratings yet
SAN Management - Lab Guide 3.3
107 pages
Dobimigrate-6 4 3
100% (1)
Dobimigrate-6 4 3
203 pages
SAN Troubleshooting PDF
No ratings yet
SAN Troubleshooting PDF
510 pages
Understanding VxRail Network Integration Requirements
No ratings yet
Understanding VxRail Network Integration Requirements
8 pages
VxRail Networking Guide With Dell EMC S4148-On Switches
No ratings yet
VxRail Networking Guide With Dell EMC S4148-On Switches
41 pages
Nutanix - Hardware Admin Ref AOS v511
No ratings yet
Nutanix - Hardware Admin Ref AOS v511
89 pages
Connectrix MDS-Series Architecture and Management Overview - SRG
100% (1)
Connectrix MDS-Series Architecture and Management Overview - SRG
67 pages
VMWare Hol 2046 01 Hci - PDF - en
No ratings yet
VMWare Hol 2046 01 Hci - PDF - en
195 pages
MDS Troubleshooting
No ratings yet
MDS Troubleshooting
161 pages
Zoning Brocade Switches
No ratings yet
Zoning Brocade Switches
6 pages
Dell EMC Avamar For VMware
No ratings yet
Dell EMC Avamar For VMware
164 pages
SAN Switch Cheat Sheet For B-Series - M-Series - MDS-series
No ratings yet
SAN Switch Cheat Sheet For B-Series - M-Series - MDS-series
4 pages
UCS Director Student Guide PDF
No ratings yet
UCS Director Student Guide PDF
206 pages
NSX-T Reference Design Guide v2
No ratings yet
NSX-T Reference Design Guide v2
247 pages
Brocade FC Switch
No ratings yet
Brocade FC Switch
7 pages
VMware VSphere Troubleshooting 5.5
No ratings yet
VMware VSphere Troubleshooting 5.5
224 pages
VNX Unified Storage Management - Lab Guide
No ratings yet
VNX Unified Storage Management - Lab Guide
219 pages
Deployment Guide-Dell EMC ECS
No ratings yet
Deployment Guide-Dell EMC ECS
29 pages
Cisco MDS Training Vol1
No ratings yet
Cisco MDS Training Vol1
32 pages
Exam Prep Guide 3V0-623 v1.2
No ratings yet
Exam Prep Guide 3V0-623 v1.2
18 pages
NSX 63 Troubleshooting
No ratings yet
NSX 63 Troubleshooting
232 pages
Storage
No ratings yet
Storage
86 pages
PowerStore+Upgrades+ +Participant+Guide
No ratings yet
PowerStore+Upgrades+ +Participant+Guide
45 pages
VCAP Study Guide - Published Version PDF
No ratings yet
VCAP Study Guide - Published Version PDF
181 pages
VMware On NetApp Solutions Student Guide and Labs
No ratings yet
VMware On NetApp Solutions Student Guide and Labs
314 pages
Dse6321 Examen
No ratings yet
Dse6321 Examen
49 pages
ICSNS - Implementing Cisco Storage Networking Solutions v4.0 - Lab
No ratings yet
ICSNS - Implementing Cisco Storage Networking Solutions v4.0 - Lab
366 pages
VCF 4.3 On VxRail Architecture Guide
No ratings yet
VCF 4.3 On VxRail Architecture Guide
92 pages
Field Installation Guide
No ratings yet
Field Installation Guide
35 pages
Brocade BCFA 16 Gbps Training PDF
No ratings yet
Brocade BCFA 16 Gbps Training PDF
156 pages
Mastering Netscaler VPX: Learn how to deploy and configure all the available Citrix NetScaler features with the best practices and techniques you need to know
From Everand
Mastering Netscaler VPX: Learn how to deploy and configure all the available Citrix NetScaler features with the best practices and techniques you need to know
Marius Sandbu
No ratings yet
VMware Horizon View Essentials
From Everand
VMware Horizon View Essentials
Peter von Oven
No ratings yet
Implementing NetScaler VPX™ - Second Edition
From Everand
Implementing NetScaler VPX™ - Second Edition
Sandbu Marius
No ratings yet
Storage Optimization with Unity All-Flash Array: Learn to Protect, Replicate or Migrate your data across Dell EMC Unity Storage and UnityVSA
From Everand
Storage Optimization with Unity All-Flash Array: Learn to Protect, Replicate or Migrate your data across Dell EMC Unity Storage and UnityVSA
Wu Victor
5/5 (1)
Configuring Cisco MDS 9000 Series Switches
No ratings yet
Configuring Cisco MDS 9000 Series Switches
2 pages
Dcmds 3.6
No ratings yet
Dcmds 3.6
2 pages
Configuring Cisco Mds 9000 Series Switches DCMDS 2
No ratings yet
Configuring Cisco Mds 9000 Series Switches DCMDS 2
4 pages
7 Network Programmability Concepts
No ratings yet
7 Network Programmability Concepts
48 pages
4 IP Services
No ratings yet
4 IP Services
190 pages
Topics Covered: Virtualization: Trainer: Sagar Dhawan
No ratings yet
Topics Covered: Virtualization: Trainer: Sagar Dhawan
18 pages
0.1 Vlan
No ratings yet
0.1 Vlan
4 pages
8 Wireless
No ratings yet
8 Wireless
32 pages
3 IP Routing
No ratings yet
3 IP Routing
131 pages
Forticlient 7.0.0 Windows Release Notes
No ratings yet
Forticlient 7.0.0 Windows Release Notes
19 pages
0.2 VTPv2 and v3
No ratings yet
0.2 VTPv2 and v3
4 pages
Ise Tacacs Configuration Guide For Cisco Asa Secure Access How To User Series
No ratings yet
Ise Tacacs Configuration Guide For Cisco Asa Secure Access How To User Series
22 pages
Capsule Labs
No ratings yet
Capsule Labs
131 pages
QoS Values Calculator v2
No ratings yet
QoS Values Calculator v2
1 page
Mass Transfer
No ratings yet
Mass Transfer
75 pages
Design of Beam For Torsion
No ratings yet
Design of Beam For Torsion
14 pages
2 Quarter Science 10 Lesson 9 Lesson: Electromagnetic Waves: A. Content Standards
No ratings yet
2 Quarter Science 10 Lesson 9 Lesson: Electromagnetic Waves: A. Content Standards
10 pages
1.4057 Aisi 431 Data Sheet PDF
No ratings yet
1.4057 Aisi 431 Data Sheet PDF
2 pages
Certificate Generation Using Blockchain Report-1
No ratings yet
Certificate Generation Using Blockchain Report-1
31 pages
Full Text 01
No ratings yet
Full Text 01
113 pages
Metasomatic Zones in Metamorphic Rocks
No ratings yet
Metasomatic Zones in Metamorphic Rocks
13 pages
Scada: Fig: Simplified Block Diagram of A SCADA System
100% (1)
Scada: Fig: Simplified Block Diagram of A SCADA System
14 pages
2a - Mathematics Preboard Exam April 2024
No ratings yet
2a - Mathematics Preboard Exam April 2024
6 pages
ECN 303 Practice Problem Solutions For Ch.5
No ratings yet
ECN 303 Practice Problem Solutions For Ch.5
2 pages
Exercise #1 - Manipulating Strings
No ratings yet
Exercise #1 - Manipulating Strings
4 pages
INVERTER - AC Tonnage Calculator - All India
0% (1)
INVERTER - AC Tonnage Calculator - All India
37 pages
CS609-GDB No.1 Solution by M.junaid Qazi
No ratings yet
CS609-GDB No.1 Solution by M.junaid Qazi
2 pages
New1
No ratings yet
New1
14 pages
Bodyspace Anthropometry Ergonomics and The Design ... - (10 Anthropometric Data)
No ratings yet
Bodyspace Anthropometry Ergonomics and The Design ... - (10 Anthropometric Data)
3 pages
1Z0-1050-24-Demo
No ratings yet
1Z0-1050-24-Demo
4 pages
Ewon x104
No ratings yet
Ewon x104
2 pages
Practice Test: 41 Marks (55 Minutes) Additional Problem: 19 Marks (27 Minutes)
No ratings yet
Practice Test: 41 Marks (55 Minutes) Additional Problem: 19 Marks (27 Minutes)
12 pages
CHAPTER-6 FORECASTING TECHNIQUES - Formatted PDF
No ratings yet
CHAPTER-6 FORECASTING TECHNIQUES - Formatted PDF
46 pages
English Language and Literature Code Number 184 Class IX (2021-22) Syllabus of Term 1 Reading Section
No ratings yet
English Language and Literature Code Number 184 Class IX (2021-22) Syllabus of Term 1 Reading Section
6 pages
MNP Call Flow
No ratings yet
MNP Call Flow
2 pages
Maths, Science & ICT A A F S: Ssessment Ctivity Ront Heet
No ratings yet
Maths, Science & ICT A A F S: Ssessment Ctivity Ront Heet
22 pages
Harmony In Chopin David Damschroder instant download
100% (2)
Harmony In Chopin David Damschroder instant download
89 pages
Basic English Grammar Rules and Tips You Absolutely Need To Know
No ratings yet
Basic English Grammar Rules and Tips You Absolutely Need To Know
9 pages
Minerals Engineering: E. Forbes, K.J. Davey, L. Smith
No ratings yet
Minerals Engineering: E. Forbes, K.J. Davey, L. Smith
9 pages
Chapter 9 Multicore Systems
No ratings yet
Chapter 9 Multicore Systems
203 pages
ASTM Tables 6 (Good)
No ratings yet
ASTM Tables 6 (Good)
185 pages

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.