h18064.1 PowerEdge MX SmartFabric Config PDF

Dell EMC PowerEdge MX SmartFabric
Configuration and Troubleshooting Guide

H18064.2
Abstract
This document provides the steps for configuring and troubleshooting the Dell EMC
PowerEdge MX networking switches in SmartFabric mode. It includes examples for
ethernet connections to Dell EMC Networking, Cisco Nexus, and Fibre Channel
networks. This document covers PowerEdge MX releases up to and including 1.20.10.
Dell Technologies Networking Infrastructure Solutions
September 2020
Notes, cautions, and warnings
NOTE: A NOTE indicates important information that helps you make better use of your product.
CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid
the problem.
WARNING: A WARNING indicates a potential for property damage, personal injury, or death.
© 2020 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be trademarks of their respective owners.
Contents
Chapter 1: Introduction to Dell EMC PowerEdge MX......................................................................8

Overview................................................................................................................................................................................8
Terminology...........................................................................................................................................................................9
Audience................................................................................................................................................................................ 9
Scope......................................................................................................................................................................................9
Typographical conventions................................................................................................................................................9
Support and feedback.......................................................................................................................................................10
Chapter 2: Overview of SmartFabric Services for PowerEdge MX.................................................11

Dell EMC SmartFabric OS10............................................................................................................................................ 11
Operating modes................................................................................................................................................................ 12
Full Switch mode.......................................................................................................................................................... 12
SmartFabric mode........................................................................................................................................................12
Changing operating modes.............................................................................................................................................. 14
MX9116n Fabric Switching Engine: virtual ports........................................................................................................ 14
Virtual Link Trunking..........................................................................................................................................................15
Fibre channel connectivity ..............................................................................................................................................16
NPIV Proxy Gateway...................................................................................................................................................16
Direct attached (F_Port)........................................................................................................................................... 16
FCoE (FSB)....................................................................................................................................................................17
FCoE load balancing.......................................................................................................................................................... 18
Networks and automated QoS....................................................................................................................................... 18
Server templates, virtual identities, networks, and deployment............................................................................ 19
Templates.......................................................................................................................................................................19
Virtual identities and identity pools......................................................................................................................... 20
Deployment................................................................................................................................................................... 20
Dell EMC Demo Center....................................................................................................................................................20
Chapter 3: SmartFabric mode requirements, guidelines, and restrictions.....................................21

Create multichassis management group...................................................................................................................... 21
Upstream network requirements....................................................................................................................................21
Physical connectivity...................................................................................................................................................21
Ethernet – No Spanning Tree uplink...................................................................................................................... 22
Spanning Tree Protocol - Legacy Ethernet uplink...............................................................................................24
VLAN scaling guidelines................................................................................................................................................... 26
Enabling support for large VLAN counts..................................................................................................................... 26
Configuring port speed and breakout...........................................................................................................................29
IOM slot placement in SmartFabric mode...................................................................................................................29
Two MX9116n Fabric Switching Engines in different chassis...........................................................................29
Two MX5108n Ethernet switches in the same chassis......................................................................................29
Two MX9116n Fabric Switching Engines in the same chassis..........................................................................30
Switch-to-switch cabling................................................................................................................................................ 30
VLT backup link.............................................................................................................................................................31
NIC teaming guidelines..................................................................................................................................................... 31
Contents 3
Uplink failure detection.................................................................................................................................................... 32
Verify UFD configuration........................................................................................................................................... 35
Forward error correction.................................................................................................................................................35
Verify FEC configuration............................................................................................................................................37
Maximum transmission unit behavior............................................................................................................................37
Other restrictions and guidelines...................................................................................................................................37
Chapter 4: SmartFabric Creation.................................................................................................38

Overview of steps to create a SmartFabric............................................................................................................... 38
Physically cable PowerEdge MX chassis and upstream switches.........................................................................38
Define VLANs..................................................................................................................................................................... 39
Define VLANs for FCoE............................................................................................................................................. 39
Create the SmartFabric................................................................................................................................................... 40
Configure uplink port speed or breakout......................................................................................................................41
Create Ethernet – No Spanning Tree uplink.............................................................................................................. 42
Ethernet – No Spanning Tree upstream switch configuration.............................................................................. 44
Configure fibre channel universal ports....................................................................................................................... 44
Create fibre channel uplinks........................................................................................................................................... 45
Configuring the upstream switch and connecting uplink cables........................................................................... 45
Chapter 5: Server Deployment.....................................................................................................47

Deploying a server............................................................................................................................................................. 47
Server preparation............................................................................................................................................................ 47
Create a server template.................................................................................................................................................47
Create identity pools........................................................................................................................................................ 48
Associate server template with networks – without FCoE.................................................................................... 49
Associate server template with networks - with FCoE........................................................................................... 50
Deploy a server template................................................................................................................................................ 50
Chapter 6: SmartFabric Operations............................................................................................. 52

Viewing the fabric............................................................................................................................................................. 52
Editing the fabric...............................................................................................................................................................54
Editing the uplinks.............................................................................................................................................................54
Edit VLANs..........................................................................................................................................................................56
Edit VLANs on deployed servers with OME-M 1.20.00 and later....................................................................56
Edit VLANs on a deployed Server with OME-M 1.10.20 and earlier .............................................................. 58
Edit VLANs on a port to connect an external device to the IOM....................................................................58
Connect non-MX Ethernet devices to fabric.............................................................................................................59
Manage Fibre Channel Zones on MX9116n FSE........................................................................................................ 60
Configure FC aliases for server and storage adapter WWPNs........................................................................60
Create FC zones..........................................................................................................................................................60
Create a zone set.........................................................................................................................................................61
Activate a zone set......................................................................................................................................................61
Delete SmartFabric............................................................................................................................................................61
SmartFabric mode IOM replacement process............................................................................................................62
Expand from single-chassis to dual-chassis configuration......................................................................................62
Step 1: Management module cabling...................................................................................................................... 62
Step 2: Create Multichassis management group.................................................................................................62
Step 3: Add second MX Chassis to the MCM Group......................................................................................... 62
4 Contents
Step 4: Move MX9116n FSE from first chassis to second chassis..................................................................62
Step 5: Validation........................................................................................................................................................ 64
Chapter 7: Switch Operations..................................................................................................... 65

PowerEdge MX switch operations................................................................................................................................65
Switch Management page overview............................................................................................................................ 65
Switch Overview......................................................................................................................................................... 66
Hardware tab................................................................................................................................................................ 67
Firmware tab................................................................................................................................................................ 69
Alerts tab....................................................................................................................................................................... 69
Settings tab.................................................................................................................................................................. 69
Configure Ethernet ports........................................................................................................................................... 71
Upgrade Dell EMC SmartFabric OS10.......................................................................................................................... 73
Chapter 8: SmartFabric Deployment Validation........................................................................... 75

View MCM group topology............................................................................................................................................. 75
View SmartFabric status................................................................................................................................................. 76
View port status................................................................................................................................................................ 78
View or extract log from MX7000 via OME-M..........................................................................................................79
CLI commands....................................................................................................................................................................79
CLI commands for validating Ethernet - No Spanning Tree...................................................................................84
Chapter 9: SmartFabric Troubleshooting.....................................................................................90

Troubleshooting SmartFabric issues............................................................................................................................ 90
Troubleshooting port group breakout errors.............................................................................................................. 90
Troubleshooting STP.........................................................................................................................................................91
Verify VLT and vPC configuration on upstream switches...................................................................................... 92
Discovery of FEM and compute sleds..........................................................................................................................93
Troubleshoot VLTi between two MX9116n FSE........................................................................................................ 94
Troubleshoot uplink errors.............................................................................................................................................. 94
Troubleshoot FC and FCoE.............................................................................................................................................98
Rebalance FC and FCoE sessions................................................................................................................................. 99
SmartFabric Services troubleshooting commands.................................................................................................. 102
Troubleshooting common issues.................................................................................................................................. 107
Chapter 10: Uplink Configuration Scenarios............................................................................... 109

Scenario 1: SmartFabric deployment with Z9100-ON upstream switches with Ethernet - No
Spanning Tree uplink.................................................................................................................................................. 109
Configure SmartFabric.............................................................................................................................................. 110
Dell EMC PowerSwitch Z9100-ON configuration................................................................................................111
Dell EMC PowerSwitch Z9100-ON validation......................................................................................................112
Scenario 2: SmartFabric connected to Cisco Nexus 3232C switches with Ethernet - No Spanning
Tree uplink......................................................................................................................................................................113
Configure SmartFabric.............................................................................................................................................. 114
Cisco Nexus 3232C switch configuration.............................................................................................................115
Configuration validation............................................................................................................................................ 117
Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode........................119
Configuration validation........................................................................................................................................... 120
Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC Direct Attach...........................................121
Contents 5
Configuration validation............................................................................................................................................122
Scenario 5: Connect MX5108n to Fibre Channel storage - FSB..........................................................................123
SmartFabric configuration steps............................................................................................................................124
Scenario 6: Configure boot from SAN........................................................................................................................125
Configure NIC boot device...................................................................................................................................... 125
Configure BIOS settings...........................................................................................................................................127
Connect FCoE LUN................................................................................................................................................... 127
Set up and install media connection...................................................................................................................... 127
Use Lifecycle Controller to set up operating system driver for media installation.................................... 128
Scenario 7: SmartFabric deployment with Z9100-ON upstream switches with legacy Ethernet uplink.... 128
Dell EMC PowerSwitch Z9100-ON configuration.............................................................................................. 129
Dell EMC PowerSwitch Z9100-ON validation......................................................................................................131
Scenario 8: SmartFabric connected to Cisco Nexus 3232C switches with legacy Ethernet uplink............132
Cisco Nexus 3232C switch configuration............................................................................................................ 133
Configuration validation........................................................................................................................................... 135
Appendix A: Hardware................................................................................................................ 138

Hardware used in this guide..........................................................................................................................................138
Dell EMC PowerSwitch S3048-ON............................................................................................................................. 138
Dell EMC PowerSwitch Z9100-ON..............................................................................................................................139
Dell EMC PowerSwitch S4148U-ON...........................................................................................................................139
Dell EMC PowerSwitch Z9264F-ON...........................................................................................................................139
Dell EMC Unity 500F storage array............................................................................................................................ 139
Cisco Nexus 3232C......................................................................................................................................................... 140
Appendix B: Dell EMC PowerEdge MX I/O Module Full Switch mode Configuration – Key
Components............................................................................................................................ 141
Configure MX IOM in NPIV proxy gateway (NPG) mode.......................................................................................141
Configure MX IOM in FIP Snooping Bridge (FSB) mode....................................................................................... 142
Appendix C: Dell EMC PowerSwitch S4148U-ON Configuration.................................................. 144

Switch configuration commands..................................................................................................................................144
Appendix D: Dell EMC Unity Information.................................................................................... 147

About Dell EMC Unity..................................................................................................................................................... 147
Determine Unity 500F storage array FC WWPNs................................................................................................... 147
Determine CNA FCoE port WWPNs........................................................................................................................... 148
Configure Unity FC storage.......................................................................................................................................... 149
Create a storage pool............................................................................................................................................... 149
Add hosts.....................................................................................................................................................................150
Create LUNs and configure host access............................................................................................................. 150
Appendix E: Additional Information............................................................................................ 152

Delete MCM group..........................................................................................................................................................152
Reset chassis using RACADM...................................................................................................................................... 152
Set admin password guidelines.................................................................................................................................... 152
Reset SmartFabric OS10 switch to factory defaults.............................................................................................. 153
Reset Cisco Nexus 3232C to factory defaults......................................................................................................... 154
Connect to IO Module console port using RACADM.............................................................................................. 154
6 Contents
Appendix F: Hardware Components and Supported Versions......................................................155
Scenarios 1, 2, 7, and 8...................................................................................................................................................155
Dell EMC PowerSwitches........................................................................................................................................ 155
Dell EMC PowerEdge MX7000 chassis and components................................................................................155
Cisco Nexus switches...............................................................................................................................................156
Scenarios 3 through 6.................................................................................................................................................... 156
Appendix G: References............................................................................................................. 158

Dell EMC documentation............................................................................................................................................... 158
Dell Technologies Networking Infrastructure Solutions documentation............................................................158
Contents 7
1
Introduction to Dell EMC PowerEdge MX
Topics:
• Overview
• Terminology
• Audience
• Scope
• Typographical conventions
• Support and feedback
Overview
The vision at Dell Technologies is to be the essential technology company for the data era. Dell Technologies ensures
modernization for today's applications and the emerging cloud-native world. Our Networking team is committed to disrupting
the fundamental economics of the market with an open strategy that gives you the freedom of choice for networking operating
systems and top-tier merchant silicon. The Dell Technologies strategy enables business transformations that maximize the
benefits of collaborative software and standards-based hardware, including lowered costs, flexibility, freedom, and security. Dell
Technologies provides further customer enablement through validated deployment guides that demonstrate these benefits while
maintaining a high standard of quality, consistency, and support.
The Dell EMC PowerEdge MX is a unified, high-performance data center infrastructure. PowerEdge MX provides the agility,
resiliency, and efficiency to optimize a wide variety of traditional and new, emerging data center workloads and applications.
With its kinetic architecture and agile management, PowerEdge MX dynamically configures compute, storage, and fabric,
increases team effectiveness, and accelerates operations. The responsive design delivers the innovation and longevity that
customers need for their IT and digital business transformations.
As part of the PowerEdge MX platform, Dell EMC SmartFabric OS10 includes SmartFabric Services. SmartFabric Services is a
network automation and orchestration solution that is fully integrated with the MX Platform.
Figure 1. Dell EMC PowerEdge MX7000 chassis
8 Introduction to Dell EMC PowerEdge MX

This document provides information about SmartFabric OS10 SmartFabric Services running on the PowerEdge MX platform.
This document also provides examples for the deployment of two PowerEdge MX7000 chassis and the setup and configuration
of SmartFabric Services. In SmartFabric mode, switches operate as a Layer 2 I/O aggregation fabric and are managed through
the Open Manage Enterprise - Modular (OME-M) user interface or console.
This guide also demonstrates connectivity with different upstream switch options, including:
● Dell EMC PowerSwitch Z9100-ON
● Cisco Nexus 3232C
● Fibre Channel connectivity methods: NPG, Direct Attach, and FSB modes
NOTE: The examples in document assume that the MX7000 chassis are configured in a multichassis management group
and that no errors have been found. Also, this guide assumes that the reader has a basic understanding of the PowerEdge
MX platform.
Terminology
Four important terms and their definitions are as follows:
Scalable Fabric This is exclusive to the PowerEdge MX platform. This is an architecture that is comprised of the Dell EMC
Networking MX9116n Fabric Switching Engine (FSE) and Dell EMC Networking MX7116n Fabric Expander
Module (FEM) allowing a fabric to span up to ten MX7000 chassis. This creates a single network fabric
enabling efficient east and west traffic flows between participating chassis. Scalable Fabric is supported
in both SmartFabric and Full Switch modes.
SmartFabric SmartFabric mode leverages SmartFabric Services (see below) to create a Layer 2 network leveraging
mode one to ten MX7000 chassis. Switches operating in SmartFabric mode are administered through the
OpenManage Enterprise - Modular (OME-M) GUI interfaces that provide complete life cycle management
of the network fabric.
Full Switch mode When operating in Full Switch mode, the switch can perform any functionality that is supported by the
version of SmartFabric OS10 running on the switch. Most of the configuration is performed using the CLI,
not the OME-M GUI.
SmartFabric In PowerEdge MX, SFS technology provides the underlying network automation and orchestration to
Services (SFS) support all automated network operations. SFS is the underlying technology for all Dell EMC SmartFabric
OS10 automation efforts including PowerEdge MX, Isilon back-end storage networking, VxRail network
automation, and so on.
Audience
This guide assumes that the reader has a basic understanding of the PowerEdge MX platform.
Scope
This guide is a reference for the most used features of SmartFabric operating mode. It does not document all features of the
PowerEdge MX platform. This guide supports, but does not take precedence over, the Release Notes for the PowerEdge MX
platform.
NOTE: For a general overview and details of PowerEdge MX networking concepts, see the Dell EMC PowerEdge MX
Network Architecture Guide.
Typographical conventions
The CLI and GUI examples in this document use the following conventions:
Monospace text CLI examples
Introduction to Dell EMC PowerEdge MX 9

Underlined monospace text CLI examples that wrap the page, or to highlight information in
CLI output
Italic monospace text Variables in CLI examples
Bold text GUI fields and information entered in the GUI
Support and feedback

For technical support, visit http://www.dell.com/support or call (USA) 1-800-945-3355.
We encourage readers to provide feedback on the quality and usefulness of this publication by sending an email to
Dell_Networking_Solutions@Dell.com.
10 Introduction to Dell EMC PowerEdge MX

2
Overview of SmartFabric Services for
PowerEdge MX
Topics:
• Dell EMC SmartFabric OS10
• Operating modes
• Changing operating modes
• MX9116n Fabric Switching Engine: virtual ports
• Virtual Link Trunking
• Fibre channel connectivity
• FCoE load balancing
• Networks and automated QoS
• Server templates, virtual identities, networks, and deployment
• Dell EMC Demo Center
Dell EMC SmartFabric OS10

The networking market is transitioning from a closed, proprietary stack to open hardware supporting various operating systems.
Dell EMC SmartFabric OS10 is designed to allow multilayered disaggregation of the network functionality. While OS10
contributions to Open Source provide users with freedom and flexibility to pick their own third-party networking, monitoring,
management, and orchestration applications; SmartFabric OS10 bundles an industry-hardened networking stack featuring
standard Layer 2 and Layer 3 protocols over a standard and well-accepted CLI interface.
Figure 2. Dell EMC SmartFabric OS10 high-level architecture
Overview of SmartFabric Services for PowerEdge MX 11

Operating modes
The Dell EMC Networking MX9116n Fabric Switching Engine (FSE) and MX5108n Ethernet Switch operate in one of two modes:
● Full Switch mode (Default) – All switch-specific SmartFabric OS10 capabilities are available.
● SmartFabric mode – Switches operate as a Layer 2 I/O aggregation fabric and are managed through the Open Manage
Enterprise-Modular (OME-M) console.
The following SmartFabric OS10 CLI commands have been added specifically for the PowerEdge MX platform:
● show switch-operating-mode – displays the current operating mode (SmartFabric or Full Switch) of a supported
switch.
● show discovered-expanders – displays the MX7116n FEMs attached to the MX9116n FSEs.
● show unit-provision – displays or configures the unit ID and service tag of a MX7116n FEM attached to a MX9116n
FSE.
NOTE: For more information, see the Dell EMC SmartFabric OS10 User Guide on the Dell EMC Networking MX9116n and
Dell EMC Networking MX5108 web pages.
Full Switch mode

In Full Switch mode, all SmartFabric OS10 features and functions that are supported by the hardware are available to the user.
In other words, the switch operates the same way as any other SmartFabric OS10 switch. Configuration is primarily done using
the CLI, however, the following items can be configured or managed using the OME-M GUI:
● Initial switch deployment: Configure hostname, password, SNMP, NTP, and so on
● Monitor health, logs, alerts, and events
● Update the SmartFabric OS10 firmware
● View physical topology
● Switch power management
Full Switch mode is typically used when a desired feature or function is not available when operating in SmartFabric mode. For
more information about Dell EMC SmartFabric OS10 operations, see the Dell Technologies Networking Solutions Info Hub.
SmartFabric mode
A SmartFabric is a logical entity that consists of a collection of physical resources, such as servers and switches, and logical
resources such as networks, templates, and uplinks. The OpenManage Enterprise – Modular (OME-M) console provides a
method to manage these resources as a single unit.
In the PowerEdge M1000e and FX2 platforms, I/O Aggregation (IOA) was implemented to simplify the process to connect blade
servers to upstream networks, so server administrators and generalists could manage uplinks, downlinks, and VLAN assignments
without needing to be fluent with the CLI.
SmartFabric Services mode builds on this IOA functionality providing:
● Data center modernization
○ I/O aggregation
○ Plug-and-play fabric deployment
○ Single interface to manage all switches in the fabric
● Lifecycle management
○ Fabric-wide SmartFabric OS10 updates
○ Automated or user-enforced rollback to last well-known state
● Fabric automation
○ Physical topology compliance
○ Server networking managed using templates
○ Automated QoS assignment per VLAN
○ Automated storage networking
● Failure remediation
○ Dynamically adjusts bandwidth across all interswitch links in the event of a link failure
12 Overview of SmartFabric Services for PowerEdge MX

○ Automatically detects fabric misconfigurations or link level failure conditions
○ Automatically heals the fabric on failure condition removal
NOTE: In SmartFabric mode, MX series switches operate entirely as a Layer 2 network fabric. Layer 3 protocols are not
supported.
When operating in SmartFabric mode, access to certain CLI commands is restricted to SmartFabric OS10 show commands and
can run the following subset of CLI configuration commands:
● clock – Configure clock parameters
● end – Exit to the EXEC mode
● exit – Exit from the current mode
● fc alias – Set fibre channel name
● fc zone – Set fibre channel zone name
● fc zoneset – Set fibre channel zone set name
● help – Display available commands
● hostname – Set the system hostname
● host-description – Define the description for the host
● interface – Configure or select an interface
● ip nameserver – Configure nameserver
● ip ssh server – Configure SSH server
● ip telnet server – Configure Telnet server
● login concurrent-session – Start concurrent session and login
● login statistics – Enable timeframe for the login session
● logging – Configure system logging
● management route – Configure the IPV4/IPv6 management route
● no – Delete or disable commands in configuration mode
● ntp – Configure the network time protocol
● snmp-server – Configure the SNMP server
● tacacs-server – Configure the TACACS server
● username – Create or modify user credentials
● Spanning-tree commands:
○ disable – Disable spanning tree globally
○ mac-flush-timer – Set the time used to flush MAC address entries
○ mode – Enable a spanning-tree mode, such as RSTP or MST
○ rstp – Configure rapid spanning-tree protocol (RSTP) mode
○ vlan – Configure spanning-tree on a VLAN range
The following table outlines the differences between the two operating modes and apply to both the MX9116n FSE and the
MX5108n switches.
Table 1. IOM operating mode differences

Full Switch mode SmartFabric mode
Configuration changes are persistent during power cycle Only the configuration changes made using the OS10
events. commands below are persistent across power cycle events. All
other CLI configuration commands are disabled.
clock
fc alias
fc zone
fc zoneset
hostname
host-description
interface
ip nameserver
ip ssh server
ip telnet server
login concurrent-session
login statistics
logging

Table 1. IOM operating mode differences (continued)
Full Switch mode SmartFabric mode
management route
ntp
snmp-server
tacacs-server
username
spanning-tree
vlan
All switch interfaces are assigned to VLAN 1 by default and Layer 2 bridging is disabled by default. Interfaces must join a
are in the same Layer 2 bridge domain. bridge domain (VLAN) before being able to forward frames.
All configuration changes are saved in the running Verify configuration changes using feature-specific show
configuration by default. To display the current configuration, commands, such as show interface and show vlan,
use the show running-configuration command. instead of show running-configuration.
Changing operating modes

In both Full Switch and SmartFabric modes, only configuration changes you make using the OME-M GUI are retained when you
switch modes. Dell EMC recommends using the graphical user interface for switch configuration in SmartFabric mode and the
OS10 CLI for switch configuration in Full Switch mode.
By default, a switch is in Full Switch mode. When that switch is added to a fabric, it automatically changes to SmartFabric mode.
When you change from Full Switch to SmartFabric mode, all Full Switch CLI configurations are deleted except for the subset of
CLI commands that are supported in SmartFabric mode.
To change a switch from SmartFabric to Full Switch mode, you must delete the fabric. At that time, only the configuration
changes you make using the OME-M GUI, such as admin password, hostname, and management IP address, will be retained.
NOTE: There is no CLI command to switch between operating modes. Delete the fabric to change from SmartFabric to Full
Switch mode.
The CLI command show switch-operating-mode displays the currently configured operating mode of the switch. This
information is also available on the switch landing page in the OME-M GUI.
MX9116n Fabric Switching Engine: virtual ports

A virtual port is a logical switch port that connects to a downstream server and has no physical hardware location on the switch.
Virtual ports are created when an MX9116n Fabric Switching Engine (FSE) on-boards an MX7116n Fabric Expander Module
(FEM). The onboarding process consists of discovery and configuration.
NOTE: If the servers in the chassis have dual-port NICs, only the first QSFP28-DD port FEM must be connected. Do not
connect the second QSFP28-DD port.
To verify the autodiscovered Fabric Expander Modules, enter the show discovered-expanders command.
OS10# show discovered-expanders

Service- Model Type Chassis- Chassis-slot Port-group Virtual-
tag service-tag Slot-Id
------------------------------------------------------------------
403RPK2 MX7116n 1 SKY003Q A2 1/1/1 71
FEM
If the FSE is in SmartFabric mode, the attached FEM is automatically configured and virtual ports on the Fabric Expander
Module and a virtual slot ID are created and mapped to 8x 25 GbE breakout interfaces in FEM on the Fabric Engine.
MX9116n FSE in Full Switch mode automatically discovers the FEM when these conditions are met:
● The FEM is connected to the FSE by attaching a cable between the QSFP28-DD ports on both devices.
● The interface for the QSFP28-DD port-group connected to the FSE is in 8x 25 GbE FEM mode.
● At least one blade server is inserted into the MX7000 chassis containing the FEM and is turned ON.

NOTE: If the FSE is in Full Switch mode, you must manually configure the Unit ID of the FEM. See the Virtual ports and
slots section of the PowerEdge MX Networking Architecture Guide for implementation.
Once the FSE discovers the FEM, it creates virtual ports by mapping each 8x 25 GbE FEM breakout interface in port groups 1 to
10 to a FEM virtual port. The following table shows an example of this mapping.
Table 2. Virtual port mapping

FEM service tag FSE QSFP28-DD port FSE 25 G interfaces FEM unit ID (virtual FEM virtual ports
group slot ID)
12AB3456 portgroup1/1/1 1/1/17:1 71 1/71/1
1/1/17:2 1/71/2
1/1/17:3 1/71/3
1/1/17:4 1/71/4
1/1/18:1 1/71/5
1/1/18:2 1/71/6
1/1/18:3 1/71/7
1/1/18:4 1/71/8
When a QSFP28-DD port group is mapped to a FEM, in the show interface status output, the eight interfaces display dormant
instead of up until a virtual port starts to transmit server traffic:
OS10# show interface status

--------------------------------------------------------------------------------
Port Description Status Speed Duplex Mode Vlan Tagged-Vlans
--------------------------------------------------------------------------------
...
Eth 1/1/17:1 dormant
...
The port status shows dormant when running the show interface status command on the MX9116n FSE. This means
that the logical ports are part of a Link Aggregation Group (LAG) internally. These ports are physically connected, but do not
pass any traffic. Once they are connected to the internal port of the compute sled and start passing traffic, the port status
moves to Up. For more information, see the Dell EMC SmartFabric OS10 User Guide.
You can also use the show interface command to display the Fabric Engine physical port-to-Fabric Expander virtual port
mapping, and the operational status of the line:
OS10# show interface ethernet 1/1/30:3

Ethernet 1/1/30:3 is up, line protocol is dormant
Interface is mapped to ethernet1/77/7
NOTE: If you move a FEM by cabling it to a different QSFP28-DD port on the Fabric Engine, all software configurations on
virtual ports are maintained. Only the QSFP28-DD breakout interfaces that map to the virtual ports change.
Virtual Link Trunking

Virtual Link Trunking (VLT) aggregates two identical physical switches to form a single logical extended switch. However, each
of the VLT peers has its own control and data planes and can be configured individually for port, protocol, and management
behaviors. Though the dual physical units act as a single logical unit, the control and data plane of both switches remain isolated,
ensuring high availability and high resilience for all its connected devices. This differs from the legacy stacking concept, where
there is a single control plane across all switches in the stack, creating a single point of failure.

With the critical need for high availability in modern data centers and enterprise networks, VLT plays a vital role connecting with
rapid convergence, seamless traffic flow, efficient load balancing, and loop free capabilities.
With the instantaneous synchronization of MAC and ARP entries, both the nodes remain active/active and continue to forward
the data traffic seamlessly.
VLT is required when operating in SmartFabric mode.
For more information about VLT, see the Virtual Link Trunking chapter in the Dell EMC SmartFabric OS10 User Guide.
Fibre channel connectivity

PowerEdge MX Ethernet I/O modules support Fibre Channel, or FC connectivity in three different ways: Direct Attach (also
called F_Port), NPIV Proxy Gateway (NPG), and FIP Snooping Bridge (FSB). The method to implement depends on the existing
infrastructure and application requirements. Consult your Dell EMC representative for more information.
Configuring FC connectivity in SmartFabric mode is simple and is almost identical across the three connectivity types.
NPIV Proxy Gateway

The most common connectivity method, NPIV Proxy Gateway mode (NPG) is used when connecting PowerEdge MX to a
storage area network that hosts a storage array. NPG mode is simple to implement as there is little configuration that must be
done. The NPG switch converts FCoE from the server to native FC and aggregates the traffic into an uplink. The NPG switch is
effectively transparent to the FC SAN, which “sees” the hosts themselves. This mode is supported only on the MX9116n FSE.
Figure 3. Fibre channel NPG network to Dell EMC Unity
For more information about configuration and deployment, see Scenario 3: Connect MX9116n FSE to Fibre Channel storage -
NPIV Proxy Gateway mode on page 119.
Direct attached (F_Port)

Direct Attached mode, or F_Port, is used when FC storage needs to be directly connected to the switch. The switch supports
the required services such as name server and zoning that are typical of FC switches.
This example demonstrates the fibre channel directly attaching to the Dell EMC Unity 500F storage array. MX9116n FSE
universal ports 44:1 and 44:2 are required for FC connections and operate in F_Port mode, which allows for an FC storage array
to be connected directly to the MX9116n FSE. The uplink type enables F_Port functionality on the MX9116n unified ports,
converting FCoE traffic to native FC traffic and passing that traffic to a directly attached FC storage array.
This mode is supported only on the MX9116n FSE.

Figure 4. Fibre Channel (F_Port) direct connect network to Dell EMC Unity
For more information on configuration and deployment, see Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC
Direct Attach on page 121.
FCoE (FSB)
FCoE Transit, or FSB mode is used when connecting PowerEdge MX to an upstream switch that accepts FCoE and converts it
to native FC, such as the Dell EMC PowerSwitch S4148U. This mode is typically used when an existing FCoE infrastructure is in
place that PowerEdge MX must connect to. In the following figure, uplink type enables NPG FC Gateway functionality on the
MX9116n FSE unified ports, converting FCoE traffic to native FC traffic on PowerSwitch S4148U-ON and passing that traffic to
an external FC switch.
When operating in FSB mode, the switch snoops FIP packets on FCoE-enabled VLANs and discovers the following information:
● End nodes (ENodes)
● Fibre channel forwarders (FCFs)
● Connections between ENodes and FCFs
● Sessions between ENodes and FCFs
Using the discovered information, the switch installs ACL entries that provide security and point-to-point link emulation.
This mode is supported on both the MX9116n FSE and the MX5108n Ethernet Switch.
Figure 5. FCoE (FSB) Network to Dell EMC Unity through NPG switch
For more information about configuration and deployment, see Scenario 5: Connect MX5108n to Fibre Channel storage - FSB on
page 123.

FCoE load balancing
The FC uplinks from the MX9116n follow industry-standard protocols. Unlike the Ethernet LACP Link Aggregation Group (LAG)
protocol, there is no industry-standard mechanism for bonding multiple FC uplinks together. Because of this, Cisco and Brocade
independently developed their own proprietary mechanisms that are not interoperable. This prevents the MX9116n FC uplinks to
be bonded to either a Cisco or Brocade FC switch using their native/proprietary protocols.
Instead, load-balancing rules are used and are listed below:
● Load is calculated based on number of server sessions that are connected to the fibre channel forwarder (FCF). The FCF
runs in OS10 and provides the FC gateway functionality. There is one FCF for each physical uplink.
● If only one FCF is available, then all the servers form FCoE sessions with that FCF.
● In the case of multiple FCFs, The NPG module running in OS10 will provide the least loaded FCF available at that time to the
next server that will log in to the FC fabric.
● Load balancing is performed only during the server login process.
● If a new FCF/uplink is created, existing server sessions will not be automatically balanced across the new session. New
server sessions will leverage the new FCF.
● Once a server is logged into an FCF, it will not shift to least loaded FCF until there is a disruption to the existing session.
NOTE: As of OME-M 1.20.00 and OS10.5.0.7, it is possible to rebalance FCoE sessions across FCFs. Please see Rebalance
FC and FCoE sessions on page 99.
Networks and automated QoS

In addition to assigning VLANs to server profiles, SmartFabric automates QoS settings based on the Network Type specified.
Figure 6 shows that when defining a VLAN, one of 11 options is pre-defined.
Figure 6. Network types available in SmartFabric mode

The following table lists the network types and related settings. The QoS group is the numerical value for the queues available in
SmartFabric mode. Available queues include 2 through 5. Queues 1, 6, and 7 are reserved.
NOTE: In SmartFabric mode, an administrator cannot change the default weights for the queues. Weights for each queue
can be seen using the show queuing weights interface ethernet command that is described in SmartFabric Deployment
Validation on page 75.
Table 3. Network types and default QoS settings

Network type Description QoS group
General Purpose (Bronze) Used for low-priority data traffic 2
General Purpose (Silver) Used for standard/default-priority data traffic 3
General Purpose (Gold) Used for high-priority data traffic 4
General Purpose (Platinum) Used for extremely high-priority data traffic 5
Cluster Interconnect Used for cluster heartbeat VLANs 5
Hypervisor Management Used for hypervisor management connections such as the ESXi 5
management VLAN
Storage - iSCSI Used for iSCSI VLANs 5
Storage - FCoE Used for FCoE VLANs 5
Storage - Data Replication Used for VLANs supporting storage data replication such as for 5
VMware VSAN
VM Migration Used for VLANs supporting vMotion and similar technologies 5
VMware FT Logging Used for VLANs supporting VMware Fault Tolerance 5
Server templates, virtual identities, networks, and

deployment
For detailed information about templates, identities, and deployment, see the OpenManage Enterprise - Modular documentation
and the technical paper PowerEdge MX7000: Templates and Profiles.
Templates
A template is a set of system configuration settings referred to as attributes. A template may contain a small set of attributes
for a specific purpose, or all the attributes for a full system configuration. Templates allow for multiple servers to be configured
quickly and automatically without the risk of human error.
Networks (VLANs) are assigned to NICs as part of the server template. When the template is deployed, those networks are
programmed on the fabric for the servers that are associated with the template.
NOTE: Network assignment through template only functions for servers connected to a SmartFabric. If a template with
network assignments is deployed to a server connected to a switch in Full Switch mode, the network assignments are
ignored.
The OME-M GUI provides the following options for creating templates:
● Most frequently, templates are created by getting the current system configuration from a server that has been configured
to the exact specifications required (referred to as a Reference Server).
● Templates may be cloned, or copied and edited.
● A template can be created by importing a Server Configuration Profile (SCP) file. The SCP file may be from a server or
exported by OpenManage Essentials, OpenManage Enterprise, or OME-M.
● OME-M comes prepopulated with several templates for specific purposes.

Virtual identities and identity pools
Some of the attributes that are included in a template are referred to as identity attributes. Identity attributes identify a device
and distinguish it from all other devices on the network. Since identity attributes must uniquely identify a device, it is imperative
that each device has a unique network identity. Otherwise, devices cannot communicate with each other over the network.
Devices come with unique manufacturer-assigned identity values preinstalled, such as a factory-assigned MAC address. Those
identities are fixed and never change. However, devices can assume a set of alternate identity values, called a “virtual identity.”
A virtual identity functions on the network using that identity, as if the virtual identity was its factory-installed identity. The use
of virtual identity is the basis for stateless operations.
OME-M console uses identity pools to manage the set of values that can be used as virtual identities for discovered devices. It
controls the assignment of virtual identity values, selecting values for individual deployments from predefined ranges of possible
values. This allows the customer to control the set of values which can be used for identities. The customer does not have to
enter all needed identity values with every deployment request, or remember which values have or have not been used. Identity
pools make configuration deployment and migration easier to manage.
Identity pools are used in conjunction with template deployment and profile operations. They provide sets of values that can be
used for virtual identity attributes for deployment. After a template is created, an identity pool may be associated with it. Doing
this directs the identity pool to get identity values whenever the template is deployed to a target device. The same identity pool
can be associated with, or used by, any number of templates. Only one identity pool can be associated with a template.
Each template has specific virtual identity needs, based on its configuration. For example, one template may have iSCSI
configured, so it needs the appropriate virtual identities for iSCSI operations. Another template may not have iSCSI configured,
but may have FCoE configured, so it needs virtual identities for FCoE operations but not for iSCSI operations.
Deployment
Deployment is the process of applying a full or partial system configuration on a specific target device. In OME-M, templates are
the basis for all deployments. Templates contain the system configuration attributes that get provisioned to the target server,
then the iDRAC on the target server applies the attributes contained in the template and reboots the server if necessary. Often,
templates contain virtual identity attributes. As mentioned above, identity attributes must have unique values on the network.
Identity Pools facilitate the assignment and management of unique virtual identities.
Dell EMC Demo Center

The Dell EMC Demo Center is a highly scalable, multi-cloud-based service that provides 24/7 self-service access to virtual labs,
hardware labs, and interactive product simulations. Several interactive demos are available on the Demo Center for PowerEdge
MX deployments. Go to Dell EMC Interactive Demo: PowerEdge MX Series to quickly become familiar with deploying MX
Networks.

3
SmartFabric mode requirements, guidelines,
and restrictions
Before deploying a SmartFabric, ensure that the following requirements, guidelines, and restrictions are followed. Failure to do
so may impact your network.
Topics:
• Create multichassis management group
• Upstream network requirements
• VLAN scaling guidelines
• Enabling support for large VLAN counts
• Configuring port speed and breakout
• IOM slot placement in SmartFabric mode
• Switch-to-switch cabling
• NIC teaming guidelines
• Uplink failure detection
• Forward error correction
• Maximum transmission unit behavior
• Other restrictions and guidelines
Create multichassis management group

For a scalable fabric that uses more than one MX chassis, the chassis must be in a MultiChassis Management (MCM) Group.
For more information about how to create MCM Group, see the PowerEdge MX7000 Documentation page.
NOTE: SmartFabric mode can be enabled on a single chassis having two MX9116n FSEs or two MX5108n switches. For a
SmartFabric implemented using a single chassis, creating an MCM group is not mandatory but recommended. The chassis
must be in an MCM group for a SmartFabric containing more than one MX7000 chassis.
NOTE: It is recommended, but not required for the lead/backup chassis to have switches installed in them in an MCM
Group.
Upstream network requirements

This section describes the requirements and guidelines for connecting an upstream network to the PowerEdge MX platform.
Physical connectivity
All physical Ethernet connections within an uplink from a SmartFabric are automatically grouped into a single LACP LAG. This
configuration requires all ports on the upstream switches to be in a single LACP LAG. Failure to do so may create network loops.
A minimum of one physical uplink from each MX switch to each upstream switch is required and the uplinks must be connected
in a mesh design. For example, if you have two upstream switches, you need two uplinks from each MX9116n FSE, as shown in
the following figure.
A maximum of two Ethernet - No Spanning Tree uplinks or three legacy Ethernet uplinks are supported on a SmartFabric. If
multiple uplinks are going to be used, you cannot use the same VLAN ID on more than one uplink without creating a network
loop.
SmartFabric mode requirements, guidelines, and restrictions 21

NOTE: The upstream switch ports must be in a single LACP LAG as shown in the figure below. Creating multiple LAGs
within a single uplink results in a network loop and is not supported.
Figure 7. Required upstream network connectivity
The maximum number of uplinks supported in SmartFabric are detailed in the following table.
Table 4. Number of uplinks supported

OME-M version Uplink type supported Number of uplinks
1.20.10 and later Ethernet - No Spanning Tree 2
Legacy Ethernet (with Spanning Tree) 3
Fibre Channel 1 (from each switch)
1.20.00 Ethernet - No Spanning Tree 2 (Only QSFP28 interfaces are currently
supported for Ethernet - No Spanning Tree
uplinks.)
Legacy Ethernet (with Spanning Tree) 3
1.10.20 and earlier Ethernet 3
If multiple uplinks are to be used, you cannot use the same VLAN ID on more than one uplink without creating a network loop.
Ethernet – No Spanning Tree uplink

OME-M 1.20.00 and OS10.5.0.7 and later supports a new uplink type: Ethernet - No Spanning Tree. This uplink type allows
Ethernet uplinks to represent a SmartFabric as an end host with multiple adapters to the upstream network with spanning tree
being disabled on the uplink interfaces.
A loop free topology without STP is achieved by not allowing overlapping VLANs across uplinks. Supported use cases are shown
in the following figures.
NOTE: For PowerEdge MX systems using OME-M 1.20.00 and OS10.5.0.7 and later, Ethernet - No Spanning Tree uplinks
should be used instead of the legacy Ethernet uplink.
Supported scenarios:
The Ethernet - No Spanning Tree feature supports uplinks to both Dell EMC and non-Dell EMC switches in a vPC/VLT. Each
uplink must be in a single LACP LAG. No more than two Ethernet - No Spanning Tree uplinks are supported with OME-M
1.20.00. Refer to the previous table for more information.
22 SmartFabric mode requirements, guidelines, and restrictions

Guidelines:
● All legacy Ethernet uplinks must be deleted before creating Ethernet - No Spanning Tree uplinks to avoid the possibility of
creating a network loop.
● Ethernet-No Spanning Tree uplinks cannot co-exist with legacy Ethernet uplinks in the same SmartFabric.
● VLANs (tagged/untagged) must not overlap.
● FCoE Uplinks require separate untagged VLAN IDs.
● With OME-M 1.20.00 and earlier, on the MX9116n FSE, only QSFP28 interfaces for Ethernet - No Spanning Tree uplinks is
supported. With OME-M 1.20.10 and later, QSFP28-DD interfaces for Ethernet - No Spanning Tree are also supported.
Use Case 1: Standard uplink configuration (maximum of 2 uplinks)
Figure 8. Standard uplink configuration
Use Case 2: Uplink with FC gateway
Figure 9. Uplink with FC gateway
Use Case 3: Uplink with direct attached FC

Figure 10. Uplink with direct attached FC
Use Case 4: Ethernet - No Spanning Tree uplink with FCoE FSB
Figure 11. Uplink in FSB scenario
Configuring Ethernet - No Spanning Tree uplinks

Creating an Ethernet – No Spanning Tree uplink is the same process as the legacy Ethernet uplink except the upstream switch
configuration is different. Configuration examples for upstream switches can be found in this guide under Uplink Configuration
Scenarios on page 109. Instructions for how to create an uplink are included in this guide under Create Ethernet – No Spanning
Tree uplink on page 42.
Spanning Tree Protocol - Legacy Ethernet uplink

By default, SmartFabric OS10 uses Rapid per-VLAN Spanning Tree Plus (RPVST+) across all switching platforms including
PowerEdge MX networking IOMs. SmartFabric OS10 also supports RSTP.

NOTE: Dell EMC recommends using RSTP instead of RPVST+ when more than 64 VLANs are required in a fabric to avoid
performance problems.
Caution should be taken when connecting an RPVST+ to an existing RSTP environment. RPVST+ creates a single topology per
VLAN with the default VLAN, typically VLAN 1, for the Common Spanning Tree (CST) with RSTP.
For non-native VLANs, all bridge protocol data unit (BPDU) traffic is tagged and forwarded by the upstream, RSTP-enabled
switch with the associated VLAN. These BPDUs use a protocol-specific multicast address.
Any other RPVST+ tree that is attached to the RSTP tree might process these packets accordingly leading to the potential of
unexpected trees.
NOTE: When connecting to an existing environment that is not using RPVST+, Dell EMC recommends changing to the
existing spanning tree protocol before connecting a SmartFabric OS10 switch. This change ensures that the same type of
Spanning Tree is run on the SmartFabric OS10 MX switches and the upstream switches.
To switch from RPVST+ to RSTP, use the spanning-tree mode rstp command:
MX9116N-A1(config)# spanning-tree mode rstp

MX9116N-A1(config)# end
To validate the STP configuration, use the show spanning-tree brief command:
MX9116N-A1#show spanning-tree brief

Spanning tree enabled protocol rstp with force-version rstp
Executing IEEE compatible Spanning Tree Protocol Root ID Priority 0, Address
4c76.25e8.f2c0 Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID
Priority 32768, Address 2004.0f00.cd1e Configured hello time 2, max age 20, forward
delay 15 Flush Interval 200 centi-sec, Flush Invocations 95 Flush Indication threshold 0
(MAC flush optimization is disabled)
NOTE: STP is required when using legacy Ethernet uplinks. Operating a SmartFabric with STP disabled and the legacy
Ethernet uplink may create a network loop and is not supported. Use the Ethernet - No Spanning Tree uplink instead.

VLAN scaling guidelines
Because SmartFabric mode provides network automation capabilities that Full Switch mode does not, the number of supported
VLANs differs between the modes. The following table provides the recommended maximum number of VLANs per fabric,
uplink, and server port.
NOTE: These are recommendations, not enforced maximums.
Table 5. Recommended maximum number of VLANs in SmartFabric mode

OS10 version Parameter Value
10.5.1.6 Recommended max VLANs per fabric 512
Recommended max VLANs per uplink 512 across all uplinks
Recommended max VLANs per server port 256
Maximum number of MX9116n FSEs in a single MCM group 12 a
Maximum number of MX5108n Ethernet switches in a single 8
MCM group
10.5.0.1 through 10.5.0.7 Recommended max VLANs per fabric 256
Recommended max VLANs per uplink 64 across all uplinks
10.4.0.R3S Recommended max VLANs per fabric 128
10.4.0.R4S Recommended max VLANs per uplink 128 across all uplinks
a. From SmartFabric OS10.5.1.6 and later, twelve FSEs in a single MCM group and eight MX5108 switches in a single MCM
group is supported, but twelve FSEs and eight MX5108 switches together in single MCM group is not supported.
NOTE: VLANs 4001 to 4020 are reserved for internal switch communication and cannot be assigned to any interface in Full
switch or SmartFabric mode. VLAN 4020 is a Management VLAN and is enabled by default. Do not remove this VLAN, and
do not remove the VLAN tag or edit Management VLAN on the Edit Uplink page. In a Full-Switch mode, you can create a
VLAN, enable it, and define it as a Management VLAN in global configuration mode on the switch. For more information on
Configuring VLANs in Full switch mode, see the Dell EMC SmartFabric OS10 User Guide.
NOTE: In SmartFabric mode, a VLAN can be created using the CLI, but cannot be deleted or removed. Therefore, all VLAN
configuration must be done in OME-M GUI while in SmartFabric mode.
Enabling support for large VLAN counts

A SmartFabric created on PowerEdge MX versions 1.20.10 or later has support for large VLAN counts enabled by default. To
enable support for VLAN counts larger than 256 per fabric on existing SmartFabrics, a PowerShell script is used to enable the
OS10 scale-vlan-profile.
To enable this support, perform the following steps:
1. Download the script titled Set-ScaleVLANProfile.ps1 from the GitHub repository.
2. Copy this script to any folder or directory.
3. Open PowerShell and change the path to the directory where the script was copied.
4. Execute the script.

Figure 12. PowerShell - execute script
5. Enter the IP address of the OME-M instance that manages the switch being replaced. In this example, the OME-M Instance
IP is 100.67.XX.XX.
Figure 13. PowerShell - enter OME-M instance IP address

6. Provide the credentials for the OME-M instance.
Figure 14. PowerShell - enter credentials

7. When Prompted, enter Enabled to enable the scale-vlan-profile. To disable the profile, enter Disabled.
Figure 15. PowerShell - enable scale-vlan-profile

8. Using the cursor, select the SmartFabric to enable the scale-vlan-profile on.

Figure 16. PowerShell - select fabric
9. If successful, the script will indicate success.
Figure 17. PowerShell - execution of script successful

10. To verify that the scale-vlan-profile has been enabled, access a switch CLI that is part of the fabric and execute the show
running-configuration command. If successful, the entry scale-profile vlan will be listed in the configuration.
Figure 18. Show running-configuration output on IOM to verify

Configuring port speed and breakout
If you need to change the default port speed and/or breakout configuration of an uplink port, you must complete this task
before creating the uplink.
For example, the QSFP28 interfaces that belong to port groups 13, 14, 15, and 16 on MX9116n FSE are typically used for uplink
connections. By default, the ports are set to 1x 100 GbE. The QSFP28 interface supports the following Ethernet breakout
configurations:
● 1x 100 GbE – One 100 GbE interface
● 1x 40 GbE – One 40 GbE interface
● 2x 50 GbE – Breakout a QSFP28 port into two 50 GbE interfaces
● 4x 25 GbE – Breakout a QSFP28 port into four 25 GbE interfaces
● 4x 10 GbE – Breakout a QSFP28 port into four 10 GbE interfaces
The MX9116n FSE also supports fibre channel (FC) capabilities using universal ports on port-groups 15 and 16. For more
information about configuring FC storage on the MX9116n FSE, see Scenario 3 and Scenario 4 in Uplink Configuration Scenarios
on page 109.
For more information on interface breakouts, see the Dell EMC SmartFabric OS10 User Guide.
IOM slot placement in SmartFabric mode

SmartFabric mode supports three specific switch placement options. Attempts to use placements different than described here
is not supported and may result in unpredictable behavior and/or data loss.
NOTE: The cabling shown in this section is the VLTi connections between the MX switches.
Two MX9116n Fabric Switching Engines in different chassis

This is the required IOM placement when creating a SmartFabric on top of a Scalable Fabric Architecture. Placing the FSE
modules in different chassis provides redundancy in the event of a chassis failure. This configuration supports placement in
Chassis1 Slot A1 and Chassis 2 Slot A2 or Chassis1 Slot B1 and Chassis 2 Slot B2. A SmartFabric cannot include a switch in
Fabric A and a switch in Fabric B.
Figure 19. IOM placement – 2x MX9116n in different chassis
Two MX5108n Ethernet switches in the same chassis

The MX5108n Ethernet Switch is only supported in single chassis configurations, with the switches in either slots A1/A2 or slots
B1/B2. A SmartFabric cannot include a switch in Fabric A and a switch in Fabric B.

Figure 20. IOM placement – 2x MX5108n in the same chassis
Two MX9116n Fabric Switching Engines in the same chassis

This placement should only be used in environments with a single chassis, with the switches in either slots A1/A2 or slots B1/B2.
A SmartFabric cannot include a switch in Fabric A and a switch in Fabric B.
As of OME-M 1.20.00, an MX deployment can start with a single MX7000 chassis with a pair of MX9116n FSEs and grow to two
or more chassis. The instructions for this can be found in this document in Expand from single-chassis to dual-chassis
configuration on page 62.
Figure 21. IOM placement – 2x MX9116n in the same chassis
Switch-to-switch cabling
When operating in SmartFabric mode, each switch pair runs a VLT interconnect (VLTi) between them. For the MX9116n FSE,
QSFP28-DD port groups 11 and 12 (eth1/1/37-1/1/40) are used.
For the MX5108n, ports 9 and 10 are used. Port 10 operates at 40 GbE instead of 100 GbE because all VLTi links must run at the
same speed.
NOTE: The VLTi ports are not user selectable, and the SmartFabric engine enforces the connection topology.
Figure 22. MX9116n SmartFabric VLTi cabling

Figure 23. MX5108n SmartFabric VLTi cabling
VLT backup link

A pair of cables is used to provide redundancy for the VLTi link. A third redundancy mechanism, a VLT backup link, is
automatically created when the SmartFabric is created. This link exchanges VLT heartbeat information between the two
switches to avoid a split-brain scenario should the external VLTi links go down. Based on the node liveliness information, the
VLT LAG/port is in up state in the primary VLT peer and in down state in the secondary VLT peer. When only the VLTi link fails,
but the peer is alive, the secondary VLT peer shuts down the VLT ports. When the node in primary peer fails, the secondary
becomes the primary peer.
To see the status of VLT backup link, run show vlt domain-id backup-link.
For example:
OS10# show vlt 255 backup-link

VLT Backup Link
------------------------
Destination : fde1:53ba:e9a0:de14:2204:fff:fe00:a267
Peer Heartbeat status : Up
Heartbeat interval : 30
Heartbeat timeout : 90
Destination VRF : default
NIC teaming guidelines

While NIC teaming is not required, it is suggested for redundancy unless a specific implementation recommends against it.
There are two main kinds of NIC teaming:
● Switch dependent: Also referred to as LACP, 802.3ad, or Dynamic Link Aggregation, this teaming method uses the LACP
protocol to understand the teaming topology. This teaming method provides active/active teaming and requires the switch
to support LACP teaming.
● Switch independent: This method uses the operating system and NIC device drivers on the server to team the NICs. Each
NIC vendor may provide slightly different implementations with different pros and cons.
NIC Partitioning (NPAR) can impact how NIC teaming operates. Based on restrictions that are implemented by the NIC vendors
that are related to NIC partitioning, certain configurations preclude certain types of teaming.
The following restrictions are in place for both Full Switch and SmartFabric modes:
● If NPAR is NOT in use, both switch-dependent (LACP) and switch-independent teaming methods are supported.
● If NPAR IS in use, only switch-independent teaming methods are supported. Switch-dependent teaming is NOT supported.
If switch dependent (LACP) teaming is used, the following restrictions are in place:
● The iDRAC shared LAN on motherboard (LOM) feature can only be used if the “Failover” option on the iDRAC is enabled.
● If the host OS is Windows, the LACP timer MUST be set to “slow” (also referred to as “normal”).
Refer to the network adapter or operating system documentation for detailed NIC teaming instructions.
● Microsoft Windows 2012 R2, see Instructions
● Microsoft Windows 2016, see Instructions
NOTE: If using VMware ESXi and LACP, it is recommended to use VMware ESXi 6.7.0 Update 2 or later.
The following table shows the options that the MX Platform provides for NIC teaming:

Table 6. NIC teaming options on the MX Platform
Teaming option Description
No teaming No NIC bonding/teaming or switch-independent teaming
LACP teaming LACP (Also called 802.3ad or dynamic link aggregation.)
Other Other (Not recommended. Can have performance impact on link management.)
NOTE: LACP Fast timer is not currently supported.
Uplink failure detection

Uplink failure detection (UFD) detects the loss of upstream connectivity from switch uplinks to the next-hop switch. If the
switch loses upstream connectivity, the related downstream server-facing interfaces are shut down so the host can use a
different path to send data out of the fabric. By default, the attached hosts continue to send traffic to that switch without a
direct path to the destination. The VLTi link to the peer leaf switch handles traffic during such a network outage, but this is not
considered a best practice. The downstream devices do not generally receive an indication that the upstream connectivity was
lost because connectivity to the local switch is still operational. To solve this issue, use UFD.
In SmartFabric mode, UFD is automatically enabled with OME-M 1.10.20. UFD is user-configurable with OME-M 1.20.00 and
later. In Full Switch mode, UFD is NOT enabled by default and must be configured at the switch CLI.
Dell Technologies recommends enabling UFD, which detects the loss of upstream connectivity. An uplink state group is
configured on each switch, which creates an association between the uplinks to the upstream devices and the downlink
interfaces. In the event that all uplinks fail on a switch, UFD automatically shuts down the downstream interfaces. This
propagates to the hosts attached to the switch. Each host then uses its link to the remaining switch to continue sending traffic
across the network. An interface in an uplink-state group can be a physical interface or a port channel (LAG) aggregation of
physical interfaces.

Figure 24. UFD topology
For example, in the MX scenario that is mentioned in Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy
Gateway mode on page 119, when an uplink is set as FC gateway, UFD associates the set of downstream interfaces which are
part of the corresponding FCoE VLAN into a UFD group. In this scenario, the VLANs are VLAN 30 and VLAN 40 on each switch
respectively. The downstream interfaces are the ones connected to the MX740c compute sleds.
In SmartFabric mode with an FC uplink failure situation, where all FC uplink ports are down (for example, removing the fibre
channel transceiver from the switch), the switch operationally disables the downstream interfaces which belong to that UFD
group AND have the FCoE VLAN provisioned to them. A server that does not have an impacted FCoE VLAN is not disturbed.
Once the downstream ports are set operationally down, the traffic on these server ports is stopped, giving the operating system
the ability to fail traffic over to the other path. In a scenario with MX9116n FSEs, a maximum of eight FC ports can be part of an
FC Gateway uplink.
This is resolved by shutting down only the corresponding compute sled downstream ports which provide an alternate path to
the compute sleds. Bring up at least one FC port that is part of the FC gateway uplink so that the FCoE traffic can transition
through another FC port on the NIC or an IOM in the fabric. Remove FCoE VLANs from Ethernet-only downstream ports to
avoid an impact on Ethernet traffic.

Figure 25. UFD in an MX scenario
NOTE: In SmartFabric mode, one uplink-state-group is created and is enabled by default. In Full switch mode, up to 16
uplink-state groups can be created, the same as any SmartFabric OS10 switch. By default, no uplink-state groups are
created in full switch mode. Physical port channels can be assigned to an uplink-state group.
To include uplinks into a UFD group in SmartFabric mode, perform the following steps.
Steps
1. Access the OME-M console.
2. Select Devices > Fabric. Choose created fabric.
3. The UFD group can be included in two ways. If uplinks are not created, select Add Uplink. Enter Name, Description, and
Uplink type.
4. Mark the check box Include in Uplink Failure Detection Group.
Figure 26. UFD under Add Uplink

5. If uplinks are created, choose an uplink, select Edit.
6. Under Edit Uplink, mark the check box Include in Uplink Failure Detection Group.
7. This enables UFD and includes the uplink into the UFD group.

Figure 27. UFD under Edit Uplink
Verify UFD configuration

To validate UFD on the FSE, run the following commands on I/O module CLI. This will show that UFD is up and running.
MX9116n-1# show uplink-state-group 1

Uplink State Group: 1, Status: Enabled, Up
MX9116n-1# show uplink-state-group detail

(Up): Interface up (Dwn): Interface down
(Dis): Interface disabled (NA): Not Available
*: VLT Port-channel V: VLT Status P: Peer Operational Status ^: Tracking
Status
Uplink State Group : 1 Status : Enabled,up
Upstream Interfaces : Fc 1/1/44:1(Up), Fc 1/1/44:2(Up)
Downstream Interfaces: Eth 1/1/1(Up), Eth 1/1/3(Up), Eth
1/71/2(Up), Eth 1/71/7(Up)
Forward error correction

Forward error correction (FEC) is a technique used for controlling errors in data transmission at high speeds. With FEC, the
destination recognizes only the data with no errors from the source that is sending redundant error correcting code with the
data frame. This technique extends the range of the signal by correcting error without retransmission. FEC enhances data
reliability.
FEC modes:
● CL91-RS - Supports 100 GbE
● CL108-RS – Supports 25 GbE and 50 GbE
● CL74-FC – Supports 25 GbE and 50 GbE
● Auto
● Off
In SmartFabric mode, configuring FEC is supported on OME-M 1.20.00 and later. FEC options CL91-RS, CL108-RS, CL74-FC,
Auto, and Off are available. The options displayed in the GUI vary depending on the speed of the selected interface.
The following table shows the default FEC and auto negotiation values for optics and cables for the QSFP28-DD and QSFP28
ports at 200 and 100 GbE speeds.
Table 7. Media, Auto negotiation, and default FEC values for 200 and 100 GbE
Media Auto negotiation FEC
200 and 100 GbE DAC Enabled CL91-RS
200 and 100 GbE Fiber/AOC, except LR related optics Disabled CL91-RS
200 and 100 GbE LR related optics Disabled Disabled

The following table shows the default FEC and auto negotiation values for optics and cables for the QSFP28-DD and QSFP28
ports at 200, 100, 50, and 25 GbE speeds.
Table 8. Media, cable type, auto negotiation, and default FEC values
Media DAC cable type Auto negotiation FEC
200, 100, 50, and 25 GbE DAC CR-L Enabled CL-108-RS
CR-S Enabled CL-74-FC
CR-N Enabled Disabled
200, 100, 50, and 25 GbE Fiber/AOC, except LR N/A Disabled CL108-RS
related optics
200, 100, 50, and 25 GbE LR related optics N/A Disabled Disabled
In Full switch mode, see the Dell EMC SmartFabric OS10 User Guide to configure FEC.
To configure FEC in the OME-M GUI, perform the following steps.
Steps
1. Access the OME-M console.
2. Choose Devices > I/O Modules > Click on an I/O Module.
3. In an I/O Module options, Choose Hardware > Port Information. This list out the IOM ports and its information.
4. Select a port to configure FEC and click Configure FEC option at the top.
NOTE: FEC options are not supported for compute sled facing ports and FEM ports (breakout FEM, virtual ports).
Figure 28. Configure FEC option

5. This shows Current and Auto negotiated FEC Settings. Choose FEC Type for the port selected from the list.
Figure 29. Select FEC Type

Verify FEC configuration
FEC can be verified on I/O Module CLI in Full switch and SmartFabric mode. Run the following command to verify.
The show interface ethernet 1/1/41 command shows the current and negotiated FEC for port 1/1/41.
MX9116N-A1# show interface ethernet 1/1/41

Ethernet 1/1/41 is up, line protocol is up
Port is part of Port-channel 2
Hardware is Eth, address is 20:04:0f:21:d4:f1
Current address is 20:04:0f:21:d4:f1
Pluggable media present, QSFP28 type is QSFP28 100GBASE-SR4-NOF
Wavelength is 850
Receive power reading is not available
Interface index is 112

Internet address is not set
Mode of IPv4 Address Assignment: not set
Interface IPv6 oper status: Disabled
MTU 9216 bytes, IP MTU 9184 bytes
LineSpeed 100G, Auto-Negotiation off
Configured FEC is cl91-rs, Negotiated FEC is cl91-rs
(Output Truncated)
Maximum transmission unit behavior

The default maximum transmission unit (MTU) for SmartFabric OS10 interfaces is 1532 bytes. When a SmartFabric is created,
the default MTU for the switch is set to Jumbo (9216 bytes). This introduces the following behaviors:
● If the MTU is not individually set on a specific interface, the MTU is 9216 bytes.
● If the MTU has been specifically set on an individual interface, the MTU is the value that has been specified.
● If a FCoE VLAN is assigned to an interface, the MTU is set to 2500 bytes even if the MTU has been manually set to a
different value before the FCoE VLAN was assigned. It is recommended that you set the MTU back to 9216 bytes after the
FCoE VLAN is assigned.
See Configure Ethernet ports on page 71 for instructions on setting the MTU.
Other restrictions and guidelines

The following additional restrictions and guidelines are in place when operating in SmartFabric mode:
● Interconnecting switches in Slots A1/A2 with switches in Slots B1/B2, regardless of chassis, is not supported.
● When operating with multiple chassis, switches in Slots A1/A2 or Slots B1/B2 in one chassis must be interconnected only
with other Slots A1/A2 or Slots B1/B2 switches respectively. Connecting switches that reside in Slots A1/A2 in one chassis
with switches in Slots B1/B2 in another is not supported.
● Uplinks must be symmetrical. If one switch in a SmartFabric has two uplinks, the other switch must have two uplinks of the
same speed.
● You cannot have a pair of switches in SmartFabric mode uplink to another pair of switches in SmartFabric mode. A
SmartFabric can uplink to a pair of switches in Full Switch mode.
● VLANs 4001 to 4020 are reserved for internal switch communication and must not be assigned to an interface.
● In SmartFabric mode, although you can use the CLI to create VLANs 1 to 4000 and 4021 to 4094, you cannot assign
interfaces to them. For this reason, do not use the CLI to create VLANs in SmartFabric mode.
● VLAN 1 is automatically created as the Default/Native VLAN, but it is not required to be used. See Define VLANs on page
39 for more information.
● If you assign a specific VLAN to the OME-M management network, do not create that VLAN on the data path of the switch.

4
SmartFabric Creation
Topics:
• Overview of steps to create a SmartFabric
• Physically cable PowerEdge MX chassis and upstream switches
• Define VLANs
• Create the SmartFabric
• Configure uplink port speed or breakout
• Create Ethernet – No Spanning Tree uplink
• Ethernet – No Spanning Tree upstream switch configuration
• Configure fibre channel universal ports
• Create fibre channel uplinks
• Configuring the upstream switch and connecting uplink cables
Overview of steps to create a SmartFabric

The procedures in this section make the following assumptions:
● All MX7000 chassis and management modules are cabled correctly and in a MultiChassis Management group.
● The VLTi cables between switches have been connected.
● Open Manage Enterprise - Modular is at version 1.20.00 and OS10 is version 10.5.0.7 or later.
NOTE: All server, network, and chassis hardware must be updated to the latest firmware. See Hardware Components and
Supported Versions on page 155 for the minimum recommended firmware versions.
To walk through the steps of creating a SmartFabric, see the interactive demos for MX at Dell EMC Interactive Demo:
PowerEdge MX Series.
Physically cable PowerEdge MX chassis and upstream

switches
There are multiple areas of cabling for the PowerEdge MX chassis that must be completed. It is required to cable the
PowerEdge MX chassis and upstream switches before creating the SmartFabric.
Table 9. Cable requirements and instructions

Cable requirement Instructions
Management module cabling PowerEdge MX7000 Management Networking Cabling
VLTi cabling options See IOM placement – 2x MX9116n in different chassis on page
29, IOM placement – 2x MX5108n in the same chassis on
page 30, and IOM placement – 2x MX9116n in the same
chassis on page 30.
Cabling the PowerEdge MX chassis upstream See the example topologies in Uplink Configuration Scenarios
on page 109.
Console cable access, in-band and out-of-band management Management Networks for Dell EMC Networking
networks
For more information about cabling the PowerEdge MX, see the Dell EMC PowerEdge MX Networking Architecture Guide.
38 SmartFabric Creation
Define VLANs
Before creating the SmartFabric, the initial set of VLANs should be created. The first VLAN to be created should be the default,
or native VLAN, typically VLAN 1. The default VLAN must be created for any untagged traffic to cross the fabric.
NOTE: VLAN 1 will be created as a Default VLAN when the first fabric is created.
To define VLANs using the OME-M console, perform the following steps.
1. Open the OME-M console.
2. From the navigation menu, click Configuration > VLANs.
NOTE: In OME-M 1.10.20 and earlier, the VLANs screen is titled Networks.
3. In the VLANs pane, click Define.

4. In the Define Network window, complete the following:
a. Enter a name for the VLAN in the Name box. In this example, VLAN0010 was used.
b. Optionally, enter a description in the Description box. In this example, the description was entered as “Company A
General Purpose”.
c. Enter the VLAN number in the VLAN ID box. In this example, 10 was entered.
d. From the Network Type list, select the desired network type. In this example, General Purpose (Bronze) was used.
e. Click Finish.
The following figure shows VLAN 1 and VLAN 10 after being created using the previous steps.
Figure 30. Defined VLANs list
Define VLANs for FCoE
NOTE: Define VLANs for FCoE if implementing fibre channel configurations as per requirement.
A standard Ethernet uplink carries assigned VLANs on all physical uplinks. When implementing FCoE, traffic for SAN path A and
SAN path B must be kept separate. The storage arrays have two separate controllers which create two paths, SAN path A and
SAN path B, connected to the MX9116n FSE. For storage traffic to be redundant, two separate VLANs are created for that
traffic.
Using the same process described in Define VLANs on page 39, create two additional VLANs for FCoE traffic.
Table 10. FCoE VLAN attributes

Name Description Network type VLAN ID SAN
FC A1 FCOE A1 Storage - FCoE 30 A
FC A2 FCOE A2 Storage - FCoE 40 B
SmartFabric Creation 39
Figure 31. Defined FCoE VLANs list
NOTE: To create VLANs for FCoE, from the Network Type list, select Storage – FCoE, and then click Finish. VLANs to be
used for FCoE must be configured as the Storage – FCoE network type.
NOTE: In OME-M 1.10.20 and earlier, the VLANs screen is titled as Networks.
Create the SmartFabric

To create a SmartFabric using the OME-M console, perform the following steps:
2. From the navigation menu, click Devices > Fabric.
3. In the Fabric pane, click Add Fabric.
4. In the Create Fabric window, complete the following:
a. Enter a name for the fabric in the Name box. In this example, "SmartFabric" was entered.
b. Optionally, enter a description in the Description box. In this example, the description was entered as “SmartFabric
using MX9116n/MX7116n in Fabric A.”
c. Click Next.
d. From the Design Type list, select the appropriate type. In this example, “2x MX9116n Fabric Switching Engine in
different chassis” was selected.
e. From the Chassis-X list, select the first MX7000 chassis.
f. From the Switch-A list, select Slot-IOM-A1.
g. From the Chassis-Y list, select the second MX7000 chassis to join the fabric.
h. From the Switch-B list, select Slot-IOM-A2.
i. Click Next.
j. On the Summary page, verify the proposed configuration and click Finish.
NOTE: From the Summary window, a list of the physical cabling requirements can be printed.
Figure 32. SmartFabric deployment design window
The SmartFabric deploys. The process of Fabric creation can take up to 20 minutes to complete. During this time, all related
switches are rebooted, and the operating mode changes from Full Switch to SmartFabric mode.
NOTE: After the fabric is created, the fabric health is critical until at least one uplink is created.
The following figure shows the new SmartFabric object and some basic information about the fabric.
Figure 33. SmartFabric post-deployment without defined uplinks
NOTE: In an MX environment of multiple chassis, only twelve fabrics per group of 20 chassis are supported when using the
MX9116n FSE, and eight fabrics per group when using the MX5108n.
Configure uplink port speed or breakout

NOTE: Users should only perform this task if needed.
If the uplink ports must be reconfigured to a different speed or breakout setting from the default, you must complete this before
creating the uplink.
To configure the Ethernet breakout on port groups using OME-M Console, perform the following steps:
2. From the navigation menu, click Devices > I/O Modules.
3. Select the switch that you want to manage. In this example, a MX9116n FSE in slot IOM-A1 is selected.
4. Choose Hardware > Port Information.
5. In the Port Information pane, choose the desired port group. In this example port-group1/1/13 is selected.
6. Choose Configure Breakout. In the Configure Breakout dialog box, select HardwareDefault.
NOTE: Before choosing the breakout type, you must set the Breakout Type to HardwareDefault and then set the
desired configuration. If the desired breakout type is selected before setting HardwareDefault, an error occurs.
7. Click Finish.
Figure 34. Set the breakout type to HardwareDefault
8. Once the job is completed, choose Configure Breakout. In the Configure Breakout dialog box, select the required
Breakout Type. In this example, the Breakout Type for port-group1/1/13 is selected as 1x40GE. Click Finish.
Figure 35. Select the desired breakout type
9. Configure the remaining breakout types on additional uplink port groups as needed.
Create Ethernet – No Spanning Tree uplink

As of OME-M 1.20.00 and OS10.5.0.7, the preferred uplink type is the Ethernet - No Spanning Tree Protocol uplink. The legacy
Ethernet uplink is still available but is no longer recommended. The process for creating a legacy Ethernet uplink is the same as
below except for selecting Ethernet as the uplink type.
An Ethernet - No Spanning Tree uplink represents a SmartFabric as an end host with multiple adapters to the upstream
network. For this, STP is disabled on the uplink interfaces. A loop-free topology without STP is achieved by not allowing
overlapping VLANs across uplinks.
NOTE: Ethernet – No Spanning Tree uplink is supported with Dell EMC and non-Dell EMC switches in a vPC/VLT. Each
uplink must be a single LACP LAG. Uplinking to a port extender such as a Cisco FEX is not currently supported.
NOTE: To change the port speed or breakout configuration, see Configure uplink port speed or breakout on page 41 and
make those changes before creating the uplinks.
After initial deployment, the new fabric shows Uplink Count as ‘zero’ and shows a warning (yellow triangle with exclamation
point). The lack of a fabric uplink results in a failed health check (red circle with x). To create the uplink, perform the following
steps.
3. Click on the fabric name. In this example, SmartFabric is selected.
4. In the Fabric Details pane, click Uplinks.
5. Click the Add Uplinks button.
6. In the Add Uplink window, complete the following:
a. Enter a name for the uplink in the Name box. In this example, Uplink01 is entered.
b. Optionally, enter a description in the Description box.
c. From the Uplink Type list, select the desired type of uplink. In this example, Ethernet – No Spanning Tree is selected.
Figure 36. Create Ethernet – No Spanning Tree uplink
d. Click Next.
e. From the Switch Ports list, select the uplink ports on both the Mx9116n FSEs. In this example, ethernet 1/1/41 and
ethernet 1/1/42 are selected for both MX9116n FSEs.
NOTE: The show inventory CLI command can be used to find the I/O Module service tag information (for example,
8XRJ0T2).
f. From the Tagged Networks list, select the desired tagged VLANs. In this example, VLAN0010 is selected.
g. From the Untagged Network list, select the untagged VLAN. In this example, VLAN0001 is selected.
Figure 37. Create Ethernet uplink
7. Click Finish.
8. At this point, SmartFabric creates the uplink object and the status for the fabric changes to OK (green box with checkmark).
NOTE: VLAN1 will be assigned to Untagged Network by default.
Ethernet – No Spanning Tree upstream switch

configuration
If using Ethernet – No Spanning Tree uplinks, refer to the following table to configure your uplink switches. Configurations for
Dell EMC Networking OS10 (Z9100-ON) and Cisco Nexus 9000-series were used for these examples.
Table 11. Dell EMC OS10 and Cisco Nexus Ethernet – No Spanning Tree configuration
Dell EMC Networking OS10 Cisco Nexus OS
Global Settings Global Settings
spanning-tree mode RSTP spanning-tree port type edge bpduguard default
Port-channel spanning-tree port type network default
switchport mode trunk Port-channel
switchport trunk allowed vlan xy switchport mode trunk
spanning-tree bpdu guard enable switchport trunk allowed vlan xy
spanning-tree guard root channel-group <channel-group-id> mode active
spanning-tree port type edge Interface
Interface switchport mode trunk
no shutdown switchport trunk allowed vlan xy
channel-group <channel-group-id> mode active spanning-tree port type edge trunk
no switchport spanning-tree bpduguard enable
spanning-tree guard root
Configure fibre channel universal ports

NOTE: Configure fibre channel universal ports, if implementing fibre channel configurations as per requirement.
NOTE: Fibre channel port speed must be specifically configured. Autonegotiation is not supported.
On the MX9116n FSE, port-group 1/1/15 and 1/1/16 are universal ports capable of connecting to FC devices at various speeds
depending on the optic being used. In this example, we are configuring the universal port speed as 4x16G FC. To enable FC
capabilities, perform the following steps on each MX9116n FSE.
NOTE: Port-group 1/1/16 is used for FC connections in this example.

2. From the navigation menu click Devices, then click I/O Modules.
3. In the Devices panel, click to select the IOM to configure.
4. In the IOM panel, click Hardware, then click Port Information.
NOTE: See the SmartFabric Services for PowerEdge MX Port-Group Configuration Errors video for more information
about configuration errors.
5. Click the port-group 1/1/16 check box, then click Configure breakout.
6. In Configure breakout panel, select HardwareDefault as the breakout type.
7. Click Finish.
8. To set the port group 1/1/16 to 4X16GFC, select the port-group 1/1/16 check box, then click Configure breakout.
9. In Configure breakout panel, select 4X16GFC as the breakout type.
10. Click Finish.
NOTE: When enabling fibre channel ports, they are set administratively down by default. Select the ports and click the
Toggle Admin State button. Click Finish to administratively set the ports to up.
Create fibre channel uplinks

NOTE: Create fibre channel uplinks for FCoE, if implementing fibre channel configurations as per requirement.
NOTE: The steps in this section allow you to connect to an existing FC switch using NPG mode, or directly attach an FC
storage array. The uplink type is the only setting within the MX7000 chassis that distinguishes between the two
configurations.
To create uplinks, perform the following steps.
2. From the navigation menu click Devices, then click Fabric.
3. Click the SmartFabric fabric name.
4. In the Fabric Details panel, click Uplinks, then click the Add Uplinks button.
5. From the Add Uplinks window, use the information in the following table to enter an uplink name in the Name box.
6. Optionally, enter a description in the Description box.
7. From the Uplink Type list, select Type, then click Next. In this example, FCoE is selected as Uplink type. Choose Uplink
type as per your configuration from FC Gateway, FC Direct Attach, or FCoE options.
8. From the Switch Ports list, select the FC ports as defined in the following table. Select the appropriate port for the
connected uplink.
9. From the Tagged Networks list, select VLAN defined in the following table, then click Finish. SmartFabric creates the
uplink object, and the status for the fabric changes to OK.
NOTE: Fibre channel ports are administratively disabled by default. Make sure to set the fibre channel ports to Enabled by
toggling the Admin State of the ports. This is done by choosing Devices > I/O Modules > select MX9116n FSE switch >
Hardware and Port Information. Select the port and choose Toggle Admin State.
NOTE: Make sure to have MTU set up on the internal Ethernet ports leveraging FCoE. If the MTU is not set, configure the
MTU by selecting port under Port Information and choosing Configure MTU. Enter the MTU size between 2500 and 9216
bytes.
For the examples shown in Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode on page
119 and Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC Direct Attach on page 121, the uplink attributes are
defined in the following table.
Table 12. Uplink attributes

Uplink name Description Ports VLAN (tagged)
FCoE A1 FC Uplink for switch in Slot A1 Switch model dependent 30
FCoE A2 FC Uplink for switch in Slot Switch model dependent 40
A2
NOTE: Do not assign the same FCoE VLAN to both switches. They must be kept separate.
Configuring the upstream switch and connecting

uplink cables
The upstream switch ports must be configured in a single LACP LAG. This document provides six example configurations. The
first two scenarios are for Ethernet uplinks, and the remaining scenarios are for storage.
● Scenario 1: SmartFabric deployment with Dell EMC PowerSwitch Z9100-ON upstream switches with Ethernet - No Spanning
Tree uplink
● Scenario 2: SmartFabric connected to Cisco Nexus 3232C switches with Ethernet - No Spanning Tree uplink
● Scenario 3: Connect MX9116n FSE to fibre channel storage – NPIV proxy gateway mode
● Scenario 4: Connect MX9116n FSE to fibre channel storage – FC direct attach mode
● Scenario 5: Connect MX5108n to fibre channel storage - FSB mode
● Scenario 6: Configure boot from SAN
● Scenario 7: SmartFabric deployment with Dell EMC PowerSwitch Z9100-ON upstream switches with legacy Ethernet uplink
● Scenario 8: SmartFabric connected to Cisco Nexus 3232C switches with legacy Ethernet uplink
5
Server Deployment
Topics:
• Deploying a server
• Server preparation
• Create a server template
• Create identity pools
• Associate server template with networks – without FCoE
• Associate server template with networks - with FCoE
• Deploy a server template
Deploying a server
Before beginning, ensure that all server firmware, especially the NIC/CNA, has been updated to the latest version. For additional
information about validated components and firmware, see Hardware Components and Supported Versions on page 155.
Server preparation
The examples in this guide reference the Dell EMC PowerEdge MX740c compute sled with QLogic (model QL41262HMKR)
Converged Network Adapters (CNAs) installed. CNAs are required to achieve FCoE connectivity. Use the steps below to
prepare each CNA by setting them to factory defaults (if required) and configuring NIC partitioning (NPAR) if needed. Not
every implementation requires NPAR.
NOTE: iDRAC steps in this section may vary depending on hardware, software, and browser versions used. See the
documentation for your Dell EMC server for instructions on connecting to the iDRAC.
Create a server template

Before creating the template, select a server to be the reference server and configure the hardware to the exact settings
required for the implementation.
NOTE: In SmartFabric mode, you must use a template to deploy a server and to configure networking.
A server template contains parameters that are extracted from a server and allows these parameters to be quickly applied to
multiple compute sleds. A server template contains all server settings for a specific deployment type including BIOS, iDRAC,
RAID, NIC/CNA, and so on. The template is captured from a reference server and can then be deployed to multiple servers
simultaneously. The server template allows an administrator to associate VLANs to compute sleds.
The templates contain settings for the following categories:
● Local access configuration
● Location configuration
● Power configuration
● Chassis network configuration
● Slot configuration
● Setup configuration
To create a server template, perform the following steps.
2. From the navigation menu, click Configuration, then click Deploy.
Server Deployment 47
3. From the center panel, click Create Template, then click From Reference Device to open the Create Template window.
4. In the Template Name box, enter a name. In this example, “MX740c with FCOE CNA” is entered.
Figure 38. Create Template dialog box
5. Optionally, enter a description in the Description box, then click Next.

6. In the Device Selection section, click Select Device.
7. From the Select Devices window, choose the previously configured server, then click Finish.
8. From the Elements to Clone list, select all the elements, and then click Finish.
Figure 39. Select the elements to clone
A job starts and the new server template displays on the list. When complete, the Completed successfully status displays.
Create identity pools

Dell Technologies recommends the use of identity pools. Virtual identity pools are used in conjunction with server templates to
automate network onboarding of compute sleds. Perform the following steps to create an ID pool.
2. From the navigation menu, click Configure, then click Identity Pools.
3. In the Network panel, click Create. The Create Identity Pool window displays.
4. Type Ethernet CNA into the Pool Name box.
5. Optionally, enter a description in the Description box.
6. Click Next.
48 Server Deployment
7. Click to select the Include Ethernet Virtual MAC Addresses option.
8. In the Starting MAC Address box, type a unique MAC address (for example, 06:3C:F9:A4:CC:00).
9. Type 255 in the Number of Virtual MAC Identities box, click Next, then click Next again.
10. Select the Include FCoE Identity option if using FCoE.
11. In the Starting MAC Address box, type a unique MAC address (for example, 06:3C:F9:A4:CD:00).
12. Type 255 in the Number of FCoE identities box for FCoE scenarios.
Figure 40. Include FCoE identity
13. Click Finish, then click Finish again.
Associate server template with networks – without

FCoE
After successfully creating a new template, associate the template with a network.
1. From the Deploy pane, select the template to be associated with VLANs. In this example, the MX740c with Intel
mezzanine server template is selected.
2. Click Edit Network.
3. In the Edit Network window, complete the following:
a. Optionally, from the Identity Pool list, choose the desired identity pool. In this example, the Ethernet ID Pool is
selected.
b. Optionally, from the NIC Teaming option, choose the desired NIC Teaming option.
c. The NIC teaming option can be selected as No Teaming, LACP teaming, and Other, as detailed in NIC teaming
options on the MX Platform on page 32.
d. For both ports, from the Untagged Network list, select the untagged VLAN. In this example, VLAN0001 is selected.
e. For both ports, from the Tagged Network list, select the tagged VLAN. In this example, VLAN0010 is selected.
f. Click Finish.
The following figure shows the associated networks for the server template.
Figure 41. Server template network settings - no FCoE
Associate server template with networks - with FCoE

After successfully creating a new template, associate the template with a network.
1. From the Deploy pane, select the template to be associated with VLANs. In this example, the MX740c with FCoE CNA
server template is selected.
2. Click Edit Network.
3. In the Edit Network window, complete the following:
a. To choose FCoE VLANs, select Ethernet CNA from the Identity Pool list.
b. For NIC in Mezzanine 1A Port 1, select FC A1 from the Tagged Network list.
c. For NIC in Mezzanine 1A Port 2, select FC A2 from the Tagged Network list.
d. Click Finish.
The following figure shows the associated networks for the server template.
Figure 42. Server template network settings - FCoE
Deploy a server template

To deploy the server template, perform the following steps.
1. From the Deploy pane, select the template to be deployed. In this example, the MX740c with FCOE CNA server template
is selected.
2. Click Deploy Template.
3. In the Deploy Template window, complete the following:
50 Server Deployment
a. Click the Select button to choose which slots or compute sleds to deploy the template to.
b. Select the Do not forcefully reboot the host OS option.
c. Click Next. Choose Run Now. Click Finish.
The interfaces on the switch are updated automatically. SmartFabric configures each interface with an untagged VLAN and any
tagged VLANs. Also, SmartFabric deploys associated QoS settings. See Networks and automated QoS for more information.
To monitor the deployment progress, go to Monitor > Jobs > Select Job > View Details. This shows the progress of the
server template deployment.
Figure 43. Job details displaying deployment of server template
6
SmartFabric Operations
Topics:
• Viewing the fabric
• Editing the fabric
• Editing the uplinks
• Edit VLANs
• Connect non-MX Ethernet devices to fabric
• Manage Fibre Channel Zones on MX9116n FSE
• Delete SmartFabric
• SmartFabric mode IOM replacement process
• Expand from single-chassis to dual-chassis configuration
Viewing the fabric

VIew the SmartFabric using OME-M. The green checkmark next to the fabric name indicates that the fabric is in a healthy state.
In this example, the fabric created is named Fabric01.
3. To view the Fabric components, select the fabric. This can also be achieved by clicking the View Details button on the
right.
Figure 44. SmartFabric details screen

Fabric components include:
● Uplinks
● Switches
● Servers
● ISL links
Uplinks connect the MX9116n switches with upstream switches. In this example, the uplink is named as Uplink1.
52 SmartFabric Operations
Figure 45. Uplinks information within Fabric Details
Switches are the I/O modules that are part of the fabric. In this example, the fabric has two MX9116n switches.
NOTE: Fabric Expander Modules are transparent and therefore do not appear on the Fabric Details page.
Figure 46. Switches listing within Fabric Details
Servers are the compute sleds that are part of the fabric. In this example, two PowerEdge MX740c compute sleds are part of
the fabric.
Figure 47. Servers listing within Fabric Details
ISL Links are the VLT interconnects between the two switches. The ISL links should be connected on port groups 11 and 12 on
MX9116n switches, and ports 9 and 10 on MX5108n switches.
CAUTION: This connection is required. Failure to connect the defined ports results in a fabric validation error.
SmartFabric Operations 53
Figure 48. ISL Links within Fabric Details
Editing the fabric

A fabric has four components:
● Uplinks
● Switches
● Servers
● ISL Links
To edit the fabric that is discussed in this section, edit the fabric name and description using the following steps:
3. On the right, click the Edit button.
Figure 49. Edit fabric name and description screen

4. In the Edit Fabric dialog box, change the name and description, then click Finish.
Editing the uplinks

Editing the uplinks on the created fabric is done using the following steps:
3. Select the fabric.
4. Select the Uplink to edit and click Edit.
NOTE: In this example, Uplink1 is selected.
5. In the Edit Uplink dialog box, modify the Name and Description as necessary.
NOTE: The uplink type cannot be modified once the fabric is created. If the uplink type must be changed after the
fabric is created, delete the uplink and create a new uplink with the wanted uplink type.
Figure 50. Edit Uplink dialog box
NOTE: The Include in Uplink Failure Detection Group box under Uplink Type will only be seen on OME-M 1.20.00 and
later.
6. Click Next.
7. Edit the Uplink ports on the MX switches that connect to the upstream switches. In this example, ports 41 and 42 that are
on the MX9116n switches, connect to upstream switches, and are displayed.
NOTE: Carefully modify the uplink ports on both MX switches. Select the IOM to display the respective uplink switch
ports.
Figure 51. Edit uplink ports and VLAN networks

8. If necessary, modify the tagged and untagged VLANs.
NOTE: If you have changed OME-M to use a VLAN other than the default, make sure that you do not add that VLAN to
an uplink.
9. Click Finish.
Edit VLANs
The following sections describe this task for deployed servers with different versions of OME-M.
Edit VLANs on deployed servers with OME-M 1.20.00 and later

OME-M 1.20.00 adds the ability to edit VLANs on multiple servers at the same time. This section describes how to edit VLANs
and deploy settings from a reference server to multiple target servers in SmartFabric mode. After deployment of SmartFabric
and deployment of server templates, network settings can be changed by the following instructions.
2. From the navigation menu, click Device > Fabric.
4. Select Servers from the left pane.
5. Choose Edit Networks.
Figure 52. Edit Networks

6. Select Reference Server, click Next. The Reference Server settings will be deployed on one more target servers in the
fabric. In this example, Sled-1 is chosen as the Reference Server.
Figure 53. Select Reference Server

7. Choose NIC teaming from LACP, No Teaming, and Other options.
8. Modify the VLAN selections as required by defining the tagged and untagged VLANs.
9. Select VLANs on Tagged and Untagged Network for each Mezzanine card port. Click Next.
Figure 54. Modify VLANs
10. Select Target Server(s).
11. To select multiple Servers click Add and choose multiple servers from the list. Click Add again.
Figure 55. Select multiple target servers

12. Select the servers.
Figure 56. Select target servers

13. Click Finish.
NOTE: VLAN settings will be pushed to the selected servers and will overwrite any existing settings.
Edit VLANs on a deployed Server with OME-M 1.10.20 and earlier

NOTE: Instructions in this section are supported until OME-M 1.10.20. If you are using the updated Firmware OME-M
1.20.00 and later, follow the instructions in the next section.
The OME-M Console is used to add/remove VLANs on the deployed servers in a SmartFabric. Perform the following steps to
add/remove VLANs on the deployed servers.
2. From the navigation menu, click Device > Fabric.
4. Select Servers from the left pane.
Figure 57. Add and remove VLANs

5. Choose the wanted server. In this example, the PowerEdge MX740C with service tag 8XQP0T2 is selected.
6. Choose Edit Networks.
7. Choose NIC teaming from LACP, No Teaming, and Other options.
8. Modify the VLAN selections as required by defining the tagged and untagged VLANs.
9. Select VLANs on Tagged and Untagged Network for each Mezzanine card port.
10. Click Save.
Figure 58. Modify VLANs
NOTE: At this time, only one server can be selected at a time in the GUI.
Edit VLANs on a port to connect an external device to the IOM

You can connect non-MX Ethernet devices to an IOM running in SmartFabric mode and assign one or more VLANs to those
interfaces. Only VLAN assignment is possible, and protocols such as DCB will not be enabled.
To edit VLANs on an I/O module using the OME-M console, perform the following steps.
3. Choose the desired I/O module, click Hardware > Port Information.
4. Select the port on which the VLANs need to be changed, click Edit VLANs.
Figure 59. Edit VLANs for a port in IOM
5. Select Tagged and Untagged VLANs. While selecting Tagged Network, choose VLANs from Available VLANs, and shift
your selection to Selected VLANs.
Figure 60. Edit VLANs dialog box

6. Click Finish.
Connect non-MX Ethernet devices to fabric

As of SmartFabric OS10.5.0.1 and OME-M 1.10.00, PowerEdge MX Ethernet switches allow the connection of non-MX devices
such as rack servers to the fabric, so long as that device provides a physical interface that is supported by the switch. Once
connected, VLANs must be assigned to each port that the device is connected. This capability does not allow non-MX devices
to support FCoE.
To connect a non-MX device to a switch running in SmartFabric mode, perform the following steps:
2. To configure the breakout on the port-group, see the Configure uplink port speed or breakout section, if needed.
3. Once the breakout on the port-group is complete, select the port. by selecting Edit VLANs.
NOTE: Make sure that the port is not in use for any other purpose.
4. Click Edit VLANs and then select Default VLAN 1, which is shown as Untagged Network in the example below.
5. Select any of the other VLANs as the Tagged Network.
Figure 61. Selection of VLANs in Edit VLANs section
6. Click Finish.
7. Repeat these steps for any other port or IOM.
Manage Fibre Channel Zones on MX9116n FSE

When a storage array is directly connected to the MX9116n FSE, Fibre Channel Zones can be used to improve security and
performance.
Preparation of the servers is the same as mentioned in Chapter 5. Determine the FC WWPNs for the compute sleds and storage
array as discussed in Appendix B.
These examples assume that the storage array has been successfully connected to the MX9116n FSE’s FC uplinks and there are
no errors.
Below are examples of the steps and commands to configure FC Zoning. For more information about the Dell EMC SmartFabric
OS10 Fibre Channel capabilities and commands, see the Dell EMC SmartFabric OS10 User Guide.
Configure FC aliases for server and storage adapter WWPNs

An FC alias is a human defined name that references a WWN. This allows users to refer to those devices by the easy to
remember alias instead of the long WWN. In this example, aliases for two MX740c compute sleds and a Dell EMC Unity storage
array is defined.
The WWNs for the servers are obtained using the OME-M console.
MX9116n-A1 MX9116n-A2
configure terminal configure terminal
fc alias mx740c-1p1 fc alias mx740c-1p2

member wwn 20:01:00:0E:1E:09:A2:3A member wwn 20:01:00:0E:1E:09:A2:3B
fc alias mx740c-2p1 fc alias mx740c-2p2

member wwn 20:01:00:0E:1E:09:B8:F6 member wwn 20:01:00:0E:1E:09:B8:F7
fc alias SpA-0 fc alias SpA-1

member wwn 50:06:01:66:47:E0:1B:19 member wwn 50:06:01:67:47:E0:1B:19
fc alias SpB-0 fc alias SpB-1

member wwn 50:06:01:6E:47:E0:1B:19 member wwn 50:06:01:6F:47:E0:1B:19
Create FC zones
Server and storage adapter WWPNs, or their aliases are combined into zones to allow communication between devices in the
same zone. Dell EMC recommends single-initiator zoning. In other words, no more than one server HBA port per zone. For high
availability, each server HBA port should be zoned to at least one port from SP A and one port from SP B. In this example, one
zone is created for each server HBA port. The zone contains the server port and the two storage processor ports that are
connected to the same MX9116n FS.
MX9116n-A1 MX9116n-A2
fc zone mx740c-1p1zone fc zone mx740c-1p2zone

member alias-name mx740c-1p1 member alias-name mx740c-1p2
member alias-name SpB-0 member alias-name SpB-1
member alias-name SpA-0 member alias-name SpA-1
fc zone mx740c-2p1zone fc zone mx740c-2p2zone

member alias-name mx740c-2p1 member alias-name mx740c-2p2
member alias-name SpB-0 member alias-name SpB-1
member alias-name SpA-0 member alias-name SpA-1
Create a zone set

A zone set is a collection of zones. A zone set named zoneset1 is created on each switch, and the zones are added to it.
MX9116n-A1 MX9116n-A2
fc zoneset zoneset1 fc zoneset zoneset1

member mx740c-1p1zone member mx740c-1p2zone
member mx740c-2p1zone member mx740c-2p2zone
exit exit
Activate a zone set

Once the zone set is created and members are added, activating the zone set is the last step in the process. After the zone set
is activated, save the configuration using the write memory command.
MX9116n-A1 MX9116n-A2
vfabric 1 vfabric 1
zoneset activate zoneset1 zoneset activate zoneset1
exit exit
write memory write memory
Delete SmartFabric
To remove the SmartFabric using the OME-M console, perform the following steps:
3. Select SmartFabric.
4. Click the Delete button.
5. In the delete fabric dialog box, click Yes.
All participating switches reboot to Full Switch mode.
CAUTION: Any configuration that is not completed by the OME-M console is lost when switching between IOM
operating modes.
SmartFabric mode IOM replacement process
The Dell EMC PowerEdge MX platform gives you the ability to replace an I/O module in a SmartFabric if required. The MX9116n
FSE and MX5108n can only be replaced with another I/O Module of the same type.
The process used depends on the version of OS10 installed and should be run with Dell Technical Support engaged. For
technical support, go to https://www.dell.com/support or call (USA) 1-800-945-3355.
Expand from single-chassis to dual-chassis

configuration
Starting with OME-Modular 1.20.00 and OS10.5.0.7, a single MX7000 chassis with a pair of MX9116n switches can be expanded
to two MX7000 chassis with MX9116n FSEs and M7116n FEMs while running in SmartFabric mode. As shown in the following
steps, this process will not require any reconfiguration and is not destructive.
NOTE: Before beginning this process, ensure that server redundancy is configured and working correctly. While this
process is not destructive, it will disrupt the network path for NIC ports connected to the switch being moved
Step 1: Management module cabling

Connect network cables to the MX7000 Management Modules on both chassis. For more information on management module
cabling, see the PowerEdge MX Chassis Management Networking Cabling.
NOTE: Make sure to have both chassis powered on and have IP address assigned to the management module using LCD
panel or KVM ports.
Step 2: Create Multichassis management group

Create a Multichassis Management (MCM) Group on the single MX chassis configuration. For a scalable fabric that uses more
than one MX chassis, the chassis must be in an MCM Group. For more information on how to create an MCM Group, see
<Create multichassis management group.
NOTE: This step can be skipped if MCM Group is already created.
Step 3: Add second MX Chassis to the MCM Group

Perform the following steps:
1. Access OME-M GUI.
2. Select Chassis. Choose Configure > Add member.
3. Select second MX7000 Chassis from the available chassis to be added as a member to the existing MCM group.
4. Click Finish.
Step 4: Move MX9116n FSE from first chassis to second chassis

Access OME-M GUI from the lead MX Chassis. Choose I/O Modules under Devices.
Figure 62. Select IOM under Devices > I/O Modules
Select I/O Module in slot A2 from the first chassis. Power off the IOM from the Power Control drop-down menu.
Figure 63. Power off the IOM
1. Once the MX9116n FSE in Chassis 1-Slot A2 in the first chassis is powered off, physically move the switch to Slot A2 of the
second MX7000 chassis, but do NOT insert it completely at this time.
2. Insert a MX7116n FEM in Chassis 1-Slot A2 and another FEM in Chassis 2-Slot A1.
3. Connect QSFP28-DD cables between FSE and FEM as shown in the following figure.
Figure 64. Connection between FSE and FEM
4. Once cabled, fully insert the MX9116n FSE in Chassis 2-Slot A2 and it will power on automatically.
5. These steps can be repeated for IOMs in slots B1/B2 as well.
Step 5: Validation
Perform the following steps to validate the environment.
1. Make sure that all MX9116n FSEs and MX7116n FEMs on both chassis appear in the OME-M GUI. Restart the second
MX9116n FSE if you do not see it in the correct chassis.
2. Check the SmartFabric configuration to ensure that nothing has changed.
3. Make sure all internal switch ports on the MX9116n FSE and MX7116n FEMs are enabled and up. Check link lights for the
external ports to make sure that they are illuminated.
7
Switch Operations
Topics:
• PowerEdge MX switch operations
• Switch Management page overview
• Upgrade Dell EMC SmartFabric OS10
PowerEdge MX switch operations

Dell EMC PowerEdge MX switches can be managed using the OME-M console. From the Switch Management page, you can
view activity, health, and alerts. The Switch Management page also allows you to perform operations such as power control,
firmware update, and port configuration. Some of these operations can also be performed in Full Switch mode.
Switch Management page overview

To access the Switch Management page:
3. Select the preferred switch.
NOTE: In the following example, the MX9116n FSE IOM-A1 is selected.
Switch Operations 65
Figure 65. IOM Overview page on OME-M
Switch Overview
The Overview page provides a convenient location to view the pertinent data on the IOM such as:
● Chassis information
● Recent alerts
● Recent activity
● IOM subsystems
● Environment
The Power Control drop-down button provides three options:
● Power Off: Turns off the IOM
● Power Cycle: Power cycles the IOM
● System Reset: Initiates a cold reboot of the IOM
66 Switch Operations
Figure 66. Power Control options
The Blink LED drop-down button provides an option to turn the ID LED on the IOM on or off. To turn on the ID LED, select
Blink LED > Turn On. This selection activates a blinking blue LED which provides easy identification. To turn off the blinking ID
LED, select Blink LED > Turn Off.
Figure 67. Blink LED button
Hardware tab
The Hardware tab provides information about the following IOM hardware:
● FRU
● Device Management Info
● Installed software
● Port information
Figure 68. Hardware tab
The Port Information tab provides useful operations such as:

● Configuring port-group breakout
● Toggling the admin state of ports
● Configuring MTU of ports
● Toggling Auto Negotiation
NOTE: Do not use the OME-M GUI to manage ports of a switch in Full Switch Mode.
Figure 69. Port Information
Firmware tab
The Firmware tab provides options to manage the firmware on the IOM. The Dell Update Package (DUP) file is used to update
the firmware of the IOM.
Figure 70. Firmware tab
Alerts tab
The Alerts tab provides information about alerts and notifies the administrator. The advanced filter option can be leveraged to
quickly filter out alerts. Various operations can be performed on an alert or several alerts such as:
● Acknowledge
● Unacknowledged
● Ignore
● Export
● Delete
Figure 71. Alerts tab
Settings tab
The Settings tab provides options to configure the following settings for the IOMs:
● Network
● Management
● Monitoring
● Advanced Settings
Figure 72. Settings tab
Network
The Network option includes configuring IPv4, IPv6, DNS Server, and Management VLAN settings.
Figure 73. Network Settings
Management
The Management option includes setting the hostname and admin account password.
NOTE: Beginning with OME-M 1.20.00 and OS 10.5.0.7, this field will set the admin account password. For versions OME-
M 1.10.20 and OS10.5.0.5 and earlier, the field name Root Password will set the OS10 linuxadmin account password. The
default username for CLI access is admin and the password is admin.
Figure 74. Management Settings
Monitoring
The Monitoring section provides options for SNMP settings.
Figure 75. Monitoring Settings option
Advanced Settings
The Advanced Settings tab offers the option for time configuration replication and alert replication. Select the Replicate
Time Configuration from Chassis check box to replicate the time settings that are configured in the chassis to the IOM.
Select the Replicate Alert Destination Configuration from Chassis check box to replicate the alert destination settings that
are configured in the chassis to the IOM.
Figure 76. Advanced Settings option
Configure Ethernet ports

The MX switches can be accessed using the OME-M console. Various operations such as port breakout, altering the MTU size,
enabling/disabling auto negotiation, and so forth. Follow the below steps to gain insight into modifying various entities.
NOTE: In SmartFabric mode, the configuration of the interfaces using the CLI should not be performed. Use the OME-
Modular GUI instead.
1. From the Switch management page, choose Hardware > Port Information.
Figure 77. IOM Overview page on OME-M
Figure 78. Port information section

2. To configure MTU, select the port that is listed under the respective port-group.
3. Click Configure MTU. Enter MTU size in bytes.
Figure 79. Configure MTU screen
4. Click Finish.
5. To configure Auto Negotiation, select the port that is listed under the respective port-group and then click Toggle
AutoNeg. This changes the Auto Negotiation of the port to Disabled/Enabled.
6. Click Finish.
Figure 80. Enable/Disable Auto Negotiation

7. To configure the administrative state (shut/no shut) of a port, select the port that is listed under the respective port-group.
Click Toggle Admin State. This toggles the port administrative state to aDisabled/Enabled state.
8. Click Finish.
Upgrade Dell EMC SmartFabric OS10

Upgrading the IOMs in the fabric can be done using the OME-M console. The upgrade is carried out using a Dell Update
Package (DUP). A DUP is a self-contained package format that updates a single element on a system. Using DUPs, you can
update a wide range of system components simultaneously and apply scripts to similar sets of Dell systems to levels. The OS10
DUP is available for download from https://www.dell.com/support/.
NOTE: To access the complete inventory of drivers and other downloads specific for your system, sign in to your Dell
Support account.
NOTE: When downloading new versions of OME-M software (from https://www.dell.com/support/software/), always
check for a corresponding SmartFabric OS10 release.
Figure 81. Download page file for MX9116n FSE
NOTE: If an IOM is in SmartFabric mode, all the switches that are part of the fabric are updated. Do not select both of the
switches in the fabric to update.
NOTE: If an IOM is in Full Switch mode, the firmware upgrade is completed only on the specific IOMs that are selected in
the GUI.
For step-by-step instructions about how to upgrade OS10 on PowerEdge MX IO modules along with a version-to-version
upgrade matrix, see the OME-M User's Guide on the PowerEdge MX700 Documentation page.
8
SmartFabric Deployment Validation
Topics:
• View MCM group topology
• View SmartFabric status
• View port status
• View or extract log from MX7000 via OME-M
• CLI commands
• CLI commands for validating Ethernet - No Spanning Tree
View MCM group topology

The OME-M console can be used to show the physical cabling of the SmartFabric.
2. In the left navigation panel, click View Topology.
3. Click the lead chassis and then click Show Wiring.
4. To show the cabling, click the light-blue checkmark icons.
SmartFabric Deployment Validation 75

Figure 82. SmartFabric cabling
View SmartFabric status

The OME-M console can be used to show the overall health of the SmartFabric.
3. Select SmartFabric1 to expand the details of the fabric. The following figure shows the status of the fabric.
76 SmartFabric Deployment Validation

Figure 83. Fabric status details
The Overview tab shows the current inventory, including switches, servers, and interconnects between the MX9116n FSEs in
the fabric. The image below shows the SmartFabric switch in a healthy state.
Figure 84. SmartFabric switch inventory
The following image shows the participating servers in a healthy state.
Figure 85. SmartFabric server inventory
The image below shows the content of the Topology tab and the VLTi that the SmartFabric mode created.

Figure 86. SmartFabric Topology overview
Within the Topology tab, you can also view the Wiring Diagram table as shown in the image below.
Figure 87. Wiring Diagram table
View port status

The OME-M console can be used to show the port status. In this example, the figure displays ports for an MX9116n FSE.
3. Select an IOM and click the View Details button to the right of the Inventory screen. The IOM Overview displays for that
device.
4. From IOM Overview, click Hardware.
5. Click to select the Port Information tab.
The image below shows Ethernet 1/1/1, 1/1/3, 1/71/1, and 1/72/1 in the correct operational status, which is Up. The interfaces
correspond to the MX740c compute sleds in slots 1 and 2 in both chassis. The figure also shows the VLT connection (port
channel 1000) and the uplinks (port channel 1) to the Z9100-ON leaf switches.

Figure 88. IOM Port information
View or extract log from MX7000 via OME-M

This section briefly describes a method for collecting Extract Logs to troubleshoot any hardware or firmware issues in an MX
environment. Dell EMC PowerEdge MX7000 comes with a Management Module that provides chassis management. An integral
feature of the management firmware is to keep a detailed log of events from managed devices and software events in the
management firmware. Firmware logs collected from Management Module components, which can be used for troubleshooting,
are grouped as Extract Logs.
It is important to note that Extract Logs are on-demand (user-initiated) from the Management Module and are always stored in
a network share that the customer configures.
For step-by-step instructions about how to view and collect these logs, see the OME-M User's Guide on the PowerEdge MX700
Documentation page.
CLI commands
show switch-operating-mode
Use the show switch-operating-mode command to display the current operating mode:
MX9116N-1# show switch-operating-mode
Switch-Operating-Mode : Smart Fabric Mode

show discovered-expanders
The show discovered-expanders command is only available on the MX9116n FSE. The MX7116n FEMs display the service
tag that is attached to the MX9116n FSEs, the associated port group, and the virtual slot.
MX9116N-1# show discovered-expanders

Service Model Type Chassis Chassis-slot Port-group Virtual
tag service-tag Slot-Id
--------------------------------------------------------------------------
D10DXC2 MX7116n 1 SKY002Z A1 1/1/1 71
FEM
show unit-provision
The show unit-provision command is only available on the MX9116n FSE and displays the unit ID, the provision name, and
the discovered name of the MX7116n FEM that is attached to the MX9116n FSE.
MX9116N-1# show unit-provision

Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+---------------------------------+-------|
1 | 71 | D10DXC2 | D10DXC2 | up |
show lldp neighbors

The show lldp neighbors command shows information about LLDP neighbors. The iDRAC that is in the PowerEdge MX
compute sled produces LLDP topology packets that contain specific information that the SmartFabric Services engine uses to
determine the physical network topology regardless of whether a switch is in Full Switch or SmartFabric mode. For servers that
are connected to switches in SmartFabric mode, the iDRAC LLDP topology feature is required. Without it, the fabric does not
recognize the compute sled and the user cannot deploy networks to the sled.
The iDRAC MAC address can be verified by selecting iDRAC Settings > Overview > Current Network Settings from the
iDRAC GUI of a compute sled as shown in the following example:
Figure 89. IOM Port information
Alternately, the iDRAC MAC information can be obtained from the System Information on the iDRAC Dashboard page.

Figure 90. System Information on iDRAC Dashboard
When viewing the LLDP neighbors, the iDRAC MAC address in addition to the NIC MAC address of the respective mezzanine
card are shown.
MX9116N-1# show lldp neighbors

Loc PortID Rem Host Name Rem Port Id Rem Chassis Id
--------------------------------------------------------------------------------
ethernet1/1/1 Not Advertised 98:03:9b:65:73:b2 98:03:9b:65:73:b4
ethernet1/1/1 iDRAC-8XQP0T2 8XQP0T2 NIC.Mezzanine.1A-1-1 d0:94:66:87:ab:40
---- OUTPUT TRUNCATED -----
In the example deployment validation of LLDP neighbors, Ethernet1/1/1, ethernet 1/1/3, and ethernet
1/1/71-1/1/72 represent the two MX740c sleds in one chassis. The first entry is the iDRAC for the compute sled. The
iDRAC uses connectivity to the mezzanine card to advertise LLDP information. The second entry is the mezzanine card itself.
Ethernet 1/71/1 and ethernet 1/71/2 represent the MX740c compute sleds connected to the MX7116n FEM in the
other chassis.
Ethernet range ethernet1/1/37-1/1/40 are the VLTi interfaces for the SmartFabric. Last, ethernet1/1/41-1/1/42
are the links in a port channel that is connected to the Dell EMC Networking Z9100-ON leaf switches.
MX9116N-1# show lldp neighbors

----------------------------------------------------------------------------
ethernet1/1/1 iDRAC-CBMP9N2 CBMP9N2 NIC.Mezzanine.1A-1-1 d0:94:66:2a:07:2f
ethernet1/1/1 Not Advertised 24:6e:96:9c:e3:50 24:6e:96:9c:e3:50
ethernet1/1/3 iDRAC-1S35MN2 1S35MN2 NIC.Mezzanine.1A-1-1 d0:94:66:29:fa:f4
ethernet1/1/3 Not Advertised 24:6e:96:9c:e5:48 24:6e:96:9c:e5:48
ethernet1/1/37 C160A2 ethernet1/1/37 20:04:0f:00:a1:9e
ethernet1/1/41 Z9100-Leaf1 ethernet1/1/3 4c:76:25:e8:f2:c0
ethernet1/1/42 Z9100-Leaf2 ethernet1/1/3 4c:76:25:e8:e8:40
ethernet1/71/1 Not Advertised 24:6e:96:9c:e5:d8 24:6e:96:9c:e5:d8
ethernet1/71/1 iDRAC-CF52XM2 CF52XM2 NIC.Mezzanine.1A-1-1 d0:94:66:29:fe:b4
ethernet1/71/2 Not Advertised 24:6e:96:9c:e5:da 24:6e:96:9c:e5:da
ethernet1/71/2 iDRAC-1S34MN2 1S34MN2 NIC.Mezzanine.1A-1-1 d0:94:66:29:ff:27

show qos system
The show qos system command displays the QoS configuration that is applied to the system. The command is useful to
verify the service policy that is created manually or automatically by a SmartFabric deployment.
MX9116N-1# show qos system

Service-policy (input): PM_VLAN
ETS Mode : off
show policy-map
Using the service policy from show qos system, the show policy-map type qos PM_VLAN command displays QoS policy
details including associated class maps, for example, CM10, and QoS queue settings, qos-group 2.
MX9116N-1# show policy-map type qos PM_VLAN

Service-policy (qos) input: PM_VLAN
Class-map (qos): CM10
set qos-group 2
show class-map
The show class-map command displays details for all the configured class-maps. For example, the association between CM10
and VLAN 10 is shown.
MX9116N-1# show class-map

Class-map (application): class-iscsi
Class-map (qos): class-trust
Class-map (qos): CM10(match-any)
Match: mac vlan 10
Class-map (qos): CM2(match-any
show vlt domain-id vlt-port-detail

The show vlt domain-id vlt-port-detail command shows the VLT port channel status for both VLT peers. The VLT
in this example is connected to the Cisco ACI vPC. It is automatically configured in port channel 1, and it consists of two ports
on each switch.
MX9116n-1# show vlt 255 vlt-port-detail

vlt-port-channel ID : 1
VLT Unit ID Port-Channel Status Configured ports Active ports
-------------------------------------------------------------------------------
* 1 port-channel1 up 2 2
2 port-channel1 up 2 2
show interface port channel summary

The show interface port-channel summary command shows the LAG number (VLT port channel 1 in this example),
the mode, status, and ports used in the port channel.
MX9116n-1# show interface port-channel summary

LAG Mode Status Uptime Ports
1 L2-HYBRID up 00:29:20 Eth 1/1/43 (Up)
Eth 1/1/44 (Up)

show queuing weights interface ethernet
The show queuing weights interface ethernet command shows the queue and weights for each queue in
percentage. These queues belong to the QoS groups mentioned in Networks and automated QoS. For example, queue 2 belongs
to Bronze and queue 3 belongs to Silver.
MX9116N-1# show queuing weights interface ethernet 1/1/41

Interface ethernet1/1/41
Queue Weight(In percentage)
--------------------------------
0 1
1 2
2 3
3 4
4 5
5 10
6 25
7 50
The example of QoS group, its related queue, and weight is shown here.
QoS Group Queue Weight(In percentage)

--------------------------------
0 0 1
1 1 2
2(Bronze) 2 3
3(Silver) 3 4
4(Gold) 4 5
5(Platinum) 5 10
6 6 25
7 7 50
show lldp dcbx interface ethernet ets detail

The show lldp dcbx interface ethernet ets detail command shows each port group, its priorities and bandwidth
in admin, remote and local mode. Ensure that you have dcbx enabled to run the command. Bandwidth is in percentage. The
minimum and maximum bandwidth can be changed in OME-M under the Edit Network option for the created server template.
MX9116N-1# show lldp dcbx interface ethernet 1/1/1 ets detail

Max Supported PG is 8
Number of Traffic Classes is 8
Admin mode is on
Admin Parameters :
------------------
Admin is enabled
PG-grp Priority# Bandwidth TSA

------------------------------------------------
0 0,1,2,5,6,7 1% ETS
1 0% SP
2 0% SP
3 3 98% ETS
4 4 1% ETS
5 0% SP
6 0% SP
7 0% SP
Remote Parameters :
-------------------
Remote is enabled
------------------------------------------------
0 0,1,2,5,6,7 1% ETS
1 0% SP
2 0% SP

3 3 98% ETS
4 4 1% ETS
5 0% SP
6 0% SP
7 0% SP
Remote Willing Status is enabled

Local Parameters :
-------------------
Local is enabled

------------------------------------------------
0 0,1,2,5,6,7 1% ETS
1 0% SP
2 0% SP
3 3 98% ETS
4 4 1% ETS
5 0% SP
6 0% SP
7 0% SP
Oper status is init

ETS DCBX Oper status is Up
State Machine Type is Asymmetric
Conf TLV Tx Status is enabled
Reco TLV Tx Status is enabled
4 Input Conf TLV Pkts, 55 Output Conf TLV Pkts, 2 Error Conf TLV Pkts
0 Input Reco TLV Pkts, 55 Output Reco TLV Pkts, 0 Error Reco TLV Pkts
CLI commands for validating Ethernet - No Spanning

Tree
If using Ethernet – No Spanning Tree uplinks, use the CLI commands in this section to validate the configuration.
show port-channel summary on MX9116n FSE

From the MX I/O module, use the show port-channel summary command to confirm the port-channel is created for the
uplink with spanning-tree disabled on MX Switches
MX9116N-A1# show port-channel summary
Flags: D - Down I - member up but inactive P - member up and active U - Up (port-

channel) F - Fallback Activated
------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
--------------------------------------------------------------------------------
2 port-channel2 (U) Eth DYNAMIC 1/1/41(P)
1000 port-channel1000 (U) Eth STATIC 1/1/37(P) 1/1/38(P) 1/1/39(P) 1/1/40(P)
show spanning-tree brief

From the MX switch, use the show spanning-tree brief CLI command to verify RSTP is enabled on the switch.
MX9116N-A1# show spanning-tree brief

Spanning tree enabled protocol rstp with force-version rstp
Executing IEEE compatible Spanning Tree Protocol
Root ID Priority 32768, Address 2004.0f21.d480
Root Bridge hello time 2, max age 20, forward delay 15
Bridge ID Priority 32768, Address 2004.0f21.d480
We are the root

Configured hello time 2, max age 20, forward delay 15
Flush Interval 200 centi-sec, Flush Invocations 93
Flush Indication threshold 65535
Interface Designated
Name PortID Prio Cost Sts Cost Bridge ID PortID
---------------------------------------------------------------------------
ethernet1/1/1 128.8 128 800 FWD 0 32768 2004.0f21.d480 128.8
ethernet1/1/2 128.16 128 200000000 BLK 0 0 0000.0000.0000 0.0
ethernet1/1/3 128.24 128 800 FWD 0 32768 2004.0f21.d480 128.24
ethernet1/1/4 128.32 128 200000000 BLK 0 0 0000.0000.0000 0.0
show port-channel summary on upstream switch

From the upstream switch, run the show port-channel summary CLI command to verify port-channel is up and running and no
STP BPDUs are received on the upstream switch.
Figure 91. show port-channel summary command
show spanning-tree interface port-channel

From the upstream switch, run the show spanning-tree interface port- channel CLI. command to verify that no
BPDUs are received on the port.
Figure 92. show spanning-tree interface port-channel command

show running-configuration interface port-channel
From the upstream switches, use the show running-configuration interface port-channel CLI command to
verify spanning tree is enabled on the port-channel interface. Then run the show lldp neighbors command to show that
no BPDU packets are received on the interface.
Figure 93. show running-configuration interface port-channel command

show lldp neighbors
After running the show running-configuration interface port-channel command above, use the show lldp
neighbors CLI command to verify that no BPDU packets are received on the interface.
Figure 94. show lldp neighbors command

Validation on Cisco switch
From an upstream Cisco Nexus switch, run the show port-channel summary CLI command to verify port-channel is up and
running and no STP BPDUs are received on the upstream switch.
Figure 95. show port-channel summary

Run the show running-config interface port-channel {port-channel ID} command on the Cisco Nexus to
show spanning tree configuration.
Figure 96. show running-config interface port-channel command

9
SmartFabric Troubleshooting
Topics:
• Troubleshooting SmartFabric issues
• Troubleshooting port group breakout errors
• Troubleshooting STP
• Verify VLT and vPC configuration on upstream switches
• Discovery of FEM and compute sleds
• Troubleshoot VLTi between two MX9116n FSE
• Troubleshoot uplink errors
• Troubleshoot FC and FCoE
• Rebalance FC and FCoE sessions
• SmartFabric Services troubleshooting commands
• Troubleshooting common issues
Troubleshooting SmartFabric issues

This section provides information about errors that might be encountered while working with a SmartFabric. Troubleshooting
and remediation actions are also included to help with resolving errors.
Troubleshooting port group breakout errors

The creation of a SmartFabric requires you to perform specific steps in a recommended order. The SmartFabric deployment
consists of four broad steps that are all completed using the OME-M console:
1. Create the VLANs to be used in the fabric.
2. Select the switches and create the fabric that is based on the preferred physical topology.
3. Create uplinks from the fabric to the existing network and assign VLANs to those uplinks.
4. Create and deploy the appropriate server templates to the compute sleds.
For cases where breakout of port groups is required, the breakout must be configured after the SmartFabric creation and before
adding the uplinks.
Figure 97. Recommended order of steps for port breakout while creating SmartFabric
90 SmartFabric Troubleshooting
If the recommended order of steps is not followed, you may encounter the following errors:
Table 13. Troubleshooting port group breakout errors

Scenario Error display
Configuration of the breakout requires you to create the
SmartFabric first. When attempting to configure the breakout
before creating a SmartFabric, the following error displays:
Configuration of the breakout requires you to select the

HardwareDefault breakout type first. If the breakout type is
directly selected without first selecting HardwareDefault, the
following error displays:
Once the uplinks are added, they are most often associated with
tagged or untagged VLANs. When attempting to configure the
breakout on the uplink port-groups after adding uplinks
associated with VLANs to the fabric, the following error displays:
Troubleshooting STP
Spanning Tree Protocol (STP) prevent loops in the network. Loops can occur when multiple redundant parts are available
between the switches. To prevent the network from going down due to loops, various flavors of STP are available. When using
the legacy Ethernet uplink type, it is essential to ensure that network loops are prevented by using appropriate type of STP on
the upstream switches as well as the MX9116n FSE switches.
When using the Ethernet – No Spanning Tree Protocol uplink, STP is not required on the upstream switch interfaces
SmartFabric Troubleshooting 91
Verify STP is enabled on upstream switches
STP is required when connecting a SmartFabric to the upstream network when using the legacy Ethernet uplink. Turning off
Spanning Tree in the upstream switches will result in network loops and may cause downtime. Enable the appropriate STP type
on the upstream switches.
Verify STP type is identical on MX and upstream switches

Check the upstream switch if STP is enabled and verify that the type of STP matches the type of STP running on the MX
switches. By default, the MX switches run RPVST+ as shown below:
OS10# show spanning-tree brief

Spanning tree enabled protocol rapid-pvst
VLAN 1
The following example shows the STP on the upstream switches, Cisco Nexus 3232C, is configured to run MST:
Nexus-3232C-Leaf1(config)# do show spanning-tree summary

Switch is in mst mode (IEEE Standard)
Root bridge for: MST0000
Port Type Default is disable
The recommended course of action is to change the STP type to RPVST+ on the upstream Cisco Nexus switches.
Nexus-3232C-Leaf1(config)# spanning-tree mode rapid-pvst

Nexus-3232C-Leaf1(config)# do show spanning-tree summary
Switch is in rapid-pvst mode
--- OUTPUT TRUNCATED -----
Another course of action in the above case can be to change the spanning tree type on the MX switches operating in
SmartFabric mode to match the STP type on the upstream switches. Make the change using the SmartFabric OS10 CLI. The
options available on the type of STP are as follows:
MX9116N-A1(config)# spanning-tree mode ?

rstp Enable rstp
rapid-pvst Enable rapid-pvst
Verify VLT and vPC configuration on upstream

switches
Configuring a single VLT domain with Dell EMC upstream switches or a single vPC domain with Cisco upstream switches is
required. Creating two VLT/vPC domains may cause a network loop. See Scenario 1: SmartFabric deployment with Z9100-ON
upstream switches with Ethernet - No Spanning Tree uplink on page 109 and Scenario 2: SmartFabric connected to Cisco
Nexus 3232C switches with Ethernet - No Spanning Tree uplink on page 113 for the topology that is used in the deployment
example.
The following example shows a mismatch of the VLT domain IDs on VLT peer switches. To resolve this issue, ensure that a
single VLT domain is used across the VLT peers.
Z9100-Leaf1# show vlt 1

Domain ID : 1
Unit ID : 1
Role : primary
Version : 1.0
Local System MAC address : 4c:76:25:e8:f2:c0

Domain ID : 30
Unit ID : 1
Role : primary
Version : 1.0
The following example shows a mismatch of the vPC domain IDs on vPC peer switches. To resolve this issue, ensure that a
single vPC domain is used across the vPC peers.
Nexus-3232C-Leaf1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer link is down
vPC keep-alive status : peer is alive, but domain IDs do not match
3232C-Leaf2# show vpc

Legend:
vPC domain id : 255
Peer status : peer link is down
vPC keep-alive status : peer is alive, but domain IDs do not match
Discovery of FEM and compute sleds

The following can be verified if server or FEM discovery does not happen:
● If a link is not indicated on the FSE port, switch the auto negotiation settings for that port.
● Verify that the compute sled is properly seated in the compute slot in the MX7000 chassis.
● Verify that the compute sled is turned on.
● Confirm that the drivers and firmware for the BIOS, iDRAC, NICs, and CNAs on the compute sleds are up to date.
● If a QLogic 41262 or 41232 adapter used in the compute sled, the link speed setting on the adapter should be set to
SmartAN.
● Check the Topology LLDP settings. You can verify the settings by selecting iDRAC Settings > Connectivity from the
iDRAC GUI that is on the compute sled. Check that this setting is set to Enabled as shown in the figure below.
Figure 98. Ensure that Topology LLDP is enabled
Troubleshoot VLTi between two MX9116n FSE
Problem
Once SmartFabric is created, the following error displays:Warning: Unable to validate the fabric because the design link
ICL-1_REVERSE not connected as per design and Unable to validate the fabric because the design link
ICL-1_FORWARD not connected as per design
An example is shown below. To see the warning message, go to the OME-M GUI and click Devices > Fabric. Choose View
Details next to Warning. You can view the details of the warning message choosing the SmartFabric that was created, and
clicking Topology. The warnings are displayed in Validation Errors section.
Figure 99. Warning for VLTi connections using QSFP28 100 GbE cables
Figure 100. Warning messages
Solution
This occurs because the VLTi connections between two MX9116n FSEs are using QSFP28 cables instead of QSFP28-DD cables.
Make sure QSFP28-DD cables are connected between port group 11 and 12 (ports 1/1/37 through 1/1/40) on both FSEs for
VLTi connections.
Troubleshoot uplink errors

There might be additional settings that are enabled or disabled after uplinks are added to the fabric.
Toggle auto negotiation
Enabling or disabling auto negotiation from the OME-M console can bring up the uplinks connecting to the upstream switches.
For example, when deploying the SmartFabric with the Cisco Nexus 3232C (see Scenario 2: SmartFabric connected to Cisco
Nexus 3232C switches with Ethernet - No Spanning Tree uplink on page 113), disable auto negotiation on uplink ports on the
MX switches to bring the link up.
The OME-M console is used to disable/enable auto negotiation ports on MX switches. The following steps illustrate turning
disabling auto negotiation on ports 41 and 42 of a MX9116n.
1. From switch management page, choose Hardware > Port Information.
2. Select the ports on which auto negotiation must be disabled. In this example, ports 1/1/41 and port 1/1/42 are selected.
3. Click Toggle AutoNeg > Finish.
Figure 101. Toggle AutoNeg dialog box
Set uplink ports to administratively up

The uplink ports on the switch might be administratively down. Enabling the uplink ports can be carried out from the OME-M
console. The uplink ports can be administratively down when a port group breakout happens, especially for FC breakouts.
The OME-M console can be used to disable/enable the ports on MX switches. The following steps illustrate turning setting the
administrative state on ports 41 and 42 of an MX9116n.
1. From switch management page, choose Hardware > Port Information.
2. Select the ports.
NOTE: In this example, ports 1/1/41 and port 1/1/42 are selected.
3. Click Toggle Admin State > Finish .
Figure 102. Toggle Admin State dialog box
Verify MTU size

Keep the same MTU size on the ports that connect the MX switches, the ports on the upstream switches, and the server NICs.
To set the MTU size from the OME-M console, see Configuring Ethernet ports.
Verify auto negotiation settings on upstream switches

Verify the auto negotiation settings on the upstream switches. In the case where auto negotiation settings are modified, the
links might not come up. Change the auto negotiation on upstream switches to resolve the issue.
For example, if the auto negotiation was disabled on the Cisco Nexus upstream switches, the setting can be turned on. To
enable the autonegotiation on an Ethernet interface on Cisco Nexus switches, run the following commands:
switch# configure terminal

switch(config)# interface ethernet <interface-number>
switch(config-if)# negotiate auto
The following example shows interface ethernet 1/2 with auto negotiation enabled on the interface:
Nexus-3232C-Leaf1(config-if)# do show int eth 1/2

Ethernet1/2 is down (XCVR not inserted)
admin state is down, Dedicated Interface
Hardware: 40000/100000 Ethernet, address: 00fe.c8ca.f367 (bia 00fe.c8ca.f36c)
MTU 1500 bytes, BW 100000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
auto-duplex, auto-speed
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Verify LACP
The interface status of the upstream switches can provide valuable information for the link being down. The following example
shows interfaces 1 and 3 on upstream Cisco Nexus switches as members of port channel 1:
3232C-Leaf2# show interface status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- connected routed full 1000 --
Eth1/1 To MX Chassis 1 suspended trunk full 100G QSFP-100G-SR4
Eth1/2 -- xcvrAbsen routed auto auto --
Eth1/3 To MX Chassis 2 suspended trunk full 100G QSFP-100G-SR4
Checking interface 1 reveals that the ports are not receiving the LACP PDUs as shown in the following example:
3232C-Leaf2# show int eth 1/1

Ethernet1/1 is down (suspended(no LACP PDUs))
admin state is up, Dedicated Interface
Belongs to Po1
NOTE: Within the Dell EMC PowerSwitch, use the show interface status command to view the interfaces and
associated status information. Use the show interface ethernet interface number to view the interface
details.
In the following example, the errors listed above occurred because an uplink was not created on the fabric.
Figure 103. Fabric topology with no uplinks
The following image shows the Topology with QSFP28 100 GbE connection on ports 37 and 39 instead of QSFP28-DD
connection, an unsupported configuration.
Figure 104. Fabric topology with uplinks and QSFP28 100 VLTi connection
The resolution is to add the uplinks and verify that the fabric turns healthy.
Figure 105. Healthy fabric
Troubleshoot FC and FCoE
When troubleshooting FC and FCoE, consider the following:
● Verify that the firmware and drivers are up to date on the CNAs.
● Check the storage guide to confirm that the CNAs are supported by the storage that is used in the deployment. For qualified
support matrix, see the following resources:
○ Dell Technologies E-Lab Navigator
○ Dell EMC Storage Compatibility Matrix for SC Series, PS Series, and FS Series storage solutions
● Verify that port group breakout mode is appropriately configured.
● Make sure that the FC port-groups that are broken out on the unified ports in the MX9116n switches are made
administratively up once the ports are changed from Ethernet to FC.
● MX9116n switches operating in SmartFabric mode support various commands to verify the configuration. Use the following
commands to verify FC configurations from MX9116n CLI:
OS10# show fc
alias Show FC alias
ns Show FC NS Switch parameters
statistics Show FC Switch parameters
switch Show FC Switch parameters
zone Show FC Zone
zoneset Show fc zoneset
● Use the following commands to verify FCoE configurations from MX9116n CLI:
OS10# show fcoe

enode Show FCOE enode information
fcf Show FCOE fcf information
sessions Show FCOE session information
statistics Show FCOE statistics information
system Show FCOE system information
vlan Show FCOE vlan information
● Verify that the FC ports are up, for example:
OS10# show interface status | grep 1/43

Fc 1/1/43:1 up 16G auto -
Fc 1/1/43:2 up 16G auto -
Fc 1/1/43:3 down 0 auto –
Fc 1/1/43:4 down 0 auto –
The show vfabric command output provides various information including the default zone mode, the active zone set, and
interfaces that are members of the vfabric.
OS10# show vfabric

Fabric Name New vfabric
Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
FKA-Adv-Period Enabled,8
Config-State ACTIVE
Oper-State UP
==========================================
Switch Config Parameters
==========================================
Domain ID 1
==========================================
Switch Zoning Parameters
==========================================
Default Zone Mode: Allow
Active ZoneSet: None
==========================================
Members
fibrechannel1/1/44:1
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2
The show fcoe sessions command shows active FCoE sessions. The output includes MAC addresses, Ethernet interfaces,
the FCoE VLAN ID, FC IDs, and WWPNs of logged-in CNAs.
NOTE: Due to the width of the command output, each line of output is shown on two lines below.
OS10# show fcoe sessions

Enode MAC Enode Interface FCF MAC FCF interface VLAN FCoE
MAC FC-ID PORT WWPN PORT WWNN
-----------------------------------------------------------------------------------------
----------------------------------------------------------------
06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:00
NOTE: For more information about FC and FCoE, see the Dell EMC SmartFabric OS10 User Guide .
Rebalance FC and FCoE sessions

Beginning with OME-M 1.20.00 and OS10.5.0.7, the capability to rebalance FC and FCoE sessions across FC uplinks has been
added. This can be validated on Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode.
The system performs an end-node based rebalancing when the CLI command is run. Factors for rebalancing are the current
session count on the uplink includes Fabric Login requests (FLOGI) plus Fabric Discovery Requests (FDISC), and the speed of
the uplink. Rebalance can be applied once the FC fabric is up and running and uplinks sessions are established in them.
Requirements and configuration guidelines

When a new uplink is added to an IOM operating in NPG mode, or when an uplink having FC/FCoE sessions established in them
goes down, the system will go to an unbalanced state. A manual rebalance can be performed when the system is found to be
unbalanced.
The new uplink added must be operationally up before the rebalance is triggered. When an uplink goes down and rebalance is
triggered, all the sessions associated with that uplink will be torn down and will be reestablished on the other available uplinks.
Rebalancing is done at the vFabric level.
NOTE: Sessions that are torn down and reestablished appear to the host as an FC path failure until the session is
reestablished. Ensure that MPIO functionality on the host is operational before performing the rebalance.
Because FC session rebalancing is path disruptive, the command provides the ability to perform a dry run to provide a list of
servers that will be affected.
Below are the steps to perform rebalancing of uplinks.
System in balanced state

The following command shows that the system is in unbalanced state. Run the show npg device brief and show npg
uplink-interfaces commands to see the unbalanced state of the system.
Figure 106. System in unbalanced state

Trigger rebalance command with dry-run
A rebalance of the uplinks with FC and FCoE sessions can be done by triggering a dry run and actual run. Run the re-balance
npg sessions vfabric 10 dry-run command for the test run.
Figure 107. Rebalance using dry-run command

Trigger re-balance command
Rebalance of uplinks with FC and FCoE sessions can be done with actual run. Run command re-balance npg sessions
vfabric 10.
Figure 108. Rebalance with actual run
System in a balanced state

The following figure shows the system in a balanced state.
Figure 109. System in balanced state
SmartFabric Services troubleshooting commands

The following commands allow user to view various SmartFabric Services configuration information. These commands can also
be used as troubleshooting purpose on SmartFabric OS10.
For information related to Support release for commands, see the Dell EMC SmartFabric OS10 User Guide.

show smartfabric personality
The show smartfabric personality command is used on a node to view the personality of SmartFabric Services
configured. The possible values can be PowerEdge MX, Isilon, VxRail, L3 fabric .
show smartfabric cluster

The show smartfabric cluster command is used to see if node is part of the cluster. This displays the cluster
information of the node such as node role, service, virtual ip address, and node domain. It can also be used to verify role of the
node as either Backup or Master.
OS10# show smartfabric cluster
----------------------------------------------------------
CLUSTER DOMAIN ID : 50
VIP : fde1:53ba:e9a0:de14:0:5eff:fe00:150
ROLE : MASTER
SERVICE-TAG : CBJXLN2
NOTE: New features may not appear in the MSM GUI until the master is upgraded to the version that supports the new
features. The example above shows how the show smartfabric cluster command determines which I/O module is
the master, and which I/O module role is the back-up.
show smartfabric cluster member

The show smartfabric cluster member command is used to see the member details of the cluster. This displays the
cluster member information such as service-tag, ip address, status, role, type of each node and service tag of the chassis where
the node belongs to.
OS10# show smartfabric cluster member

Service-tag IP Address Status Role Type
Chassis-Service-Tag Chassis-Slot
-----------------------------------------------------------------------------------------
--------------------------------
CBJXLN2 fde1:53ba:e9a0:de14:2204:fff:fe00:cde7 ONLINE MASTER MX5108n
SKY002Z A1
BZTQPK2 fde1:53ba:e9a0:de14:2204:fff:fe00:19e5 ONLINE BACKUP MX5108n
SKY002Z B1
6L59XM2 fde1:53ba:e9a0:de14:2204:fff:fe00:3de5 ONLINE BACKUP MX5108n
SKY002Z B2
F13RPK2 fde1:53ba:e9a0:de14:2204:fff:fe00:a267 ONLINE BACKUP MX5108n
SKY003Z A2
show smartfabric details

The show smartfabric details command is used to see the all configured fabric details. This command displays the
nodes that are part of the fabric, status of the fabric, and design type associated with the fabric.
OS10# show smartfabric details

----------------------------------------------------------
Name : Fabric 1
Description :
ID : 74b3d3a4-7804-4c15-b6d3-5e4e7c364f82
DesignType : 2xMX9116n_Fabric_Switching_Engines_in_different_chassis
Validation Status: VALID
VLTi Status : VALID
Placement Status : VALID
Nodes : CBJXLN2, F13RPK2
----------------------------------------------------------

show smartfabric uplinks
The show smartfabric uplinks command is used to verify the uplinks configured across the nodes in the fabric. This
command displays the following information that is associated with the fabric:
● Name
● Description
● ID
● Media type
● Native VLAN
● Configured interfaces
● Network profile
OS10# show smartfabric uplinks

----------------------------------------------------------
Name : FCoE Path A
Description :
ID : 1b328dc2-b99c-466e-b87c-b84c9c342225
Media Type : FC
Native Vlan : 0
Untagged-network :
Networks : 6a161bae-788f-4d65-8b0c-69b404c477dc
Configured-Interfaces : CBJXLN2:fibrechannel1/1/44:1, CBJXLN2:fibrechannel1/1/44:2
----------------------------------------------------------
----------------------------------------------------------
Name : Uplink1
Description :
ID : d493fee2-9680-41c7-989d-cf0347aab4fd
Media Type : ETHERNET
Native Vlan : 1
Untagged-network :
Networks : e6189b88-7f19-4b05-98b5-0c05ff7ff8c8, 284dae93-b91f-4593-9cff-
c8521cd7ae90
Configured-Interfaces : CBJXLN2:ethernet1/1/42:1, F13RPK2:ethernet1/1/41:1,
F13RPK2:ethernet1/1/42:1, CBJXLN2:ethernet1/1/41:1
----------------------------------------------------------
----------------------------------------------------------
Name : FCoE Path B
Description :
ID : 0f7ad3a2-e59e-4a07-9a74-4e57558f0a4d
Media Type : FC
Native Vlan : 0
Untagged-network :
Networks : e2c35ec5-c177-46f1-9a69-75d8b202d739
Configured-Interfaces : F13RPK2:fibrechannel1/1/44:1, F13RPK2:fibrechannel1/1/44:2
show smartfabric networks

The show smartfabric networks command is used to view the various network profiles configured. The command displays the
VLANs that are configured, QoS Priority, and the network type for each network profile.
OS10# show smartfabric networks

Name Type QosPriority
Vlan
--------------------------------------------------------------------------------
FCoE A1 STORAGE_FCOE PLATINUM 998
VLAN1 GENERAL_PURPOSE BRONZE 1
FCoE A2 STORAGE_FCOE PLATINUM 999
VLAN10 GENERAL_PURPOSE SILVER 10
UPLINK VLAN GENERAL_PURPOSE SILVER 2491

show smartfabric validation-error
The show smartfabric validation-error displays all the information about the validation errors such as category,
subcategory, recommended action, severity, timestamp, and recommended link to each error.
show smartfabric nodes

The show smartfabric nodes command is used to view the details of the nodes that are part of the cluster. This
command helps the user to view the status of a node, chassis details the node belongs to.
OS10# show smartfabric nodes

Service-Tag Type Status Mode Chassis-Service Chassis-
Slot
Tag
--------------------------------------------------------------------------
F13RPK2 MX9116n ONLINE FABRIC SKY003Z A2
110DXC2 MX7116n NOT-APPLICABLE SKY002Z A2
CBJXLN2 MX9116n ONLINE FABRIC SKY002Z A1
6L59XM2 MX5108n ONLINE FULL-SWITCH SKY002Z B2
D10DXC2 MX7116n NOT-APPLICABLE SKY003Z A1
BZTQPK2 MX5108n ONLINE FULL-SWITCH SKY002Z B1
show smartfabric configured-servers

The show smartfabric configured-servers command displays the list of deployed servers and details such as service tag, model
(MX740c/MX840c) of compute sled, chassis slot and chassis service tag. It also shows that the compute sled has been
discovered, onboarded and configured or not.
OS10# show smartfabric configured-server

**********************************************************
Service-Tag : 8XQP0T2
Server-Model : PowerEdge MX740c
Chassis-Slot : 1
Chassis-Model : PowerEdge MX7000
Chassis-Service-Tag : 8XXJ0T2
Is-Discovered : TRUE
Is-Onboarded : TRUE
Is-Configured : TRUE
**********************************************************
**********************************************************
Service-Tag : DTQHMR2
Chassis-Slot : 1
Is-Onboarded : TRUE
**********************************************************
**********************************************************
Service-Tag : 8XRH0T2
Chassis-Slot : 1
Chassis-Service-Tag : F7PQ0T2
Is-Onboarded : TRUE
**********************************************************
**********************************************************
Service-Tag : ST0000W
Chassis-Slot : 1

Is-Onboarded : TRUE
show smartfabric configured-servers configured-

server-interface
The show smartfabric configured-servers configured-server-interface <compute-sled service
tag> command shows details of one deployed server such as NIC-ID, Switch Interface and Fabric. It also shows tagged and
untagged VLANs on NIC Mezzanine Card ports.
OS10# show smartfabric configured-server configured-server-interface DTQHMR2

**********************************************************
----------------------------------------------------------
Nic-Id : NIC.Mezzanine.1A-2-1
Switch-Interface : 87QNMR2:ethernet1/71/2
Fabric : SF (abdeec7f-3a83-483a-929e-aa102429ae86)
Is-Onboarded : TRUE
NicBonded : FALSE
Native-vlan : 1
Static-onboard-interface:
Networks : 40, 1611
----------------------------------------------------------
Nic-Id : NIC.Mezzanine.1A-1-1
Switch-Interface : 8XRJ0T2:ethernet1/1/3
Fabric : SF (abdeec7f-3a83-483a-929e-aa102429ae86)
Is-Onboarded : TRUE
NicBonded : FALSE
Native-vlan : 1
Static-onboard-interface:
Networks : 30, 1611
show smartfabric discovered-servers

The show smartfabric discovered-servers shows the list of servers present in the cluster and discovered by IOMs.
OS10# show smartfabric discovered-server

----------------------------------------------------------
Service-Tag : 8XQP0T2
Chassis-Slot : 1
----------------------------------------------------------
----------------------------------------------------------
Chassis-Slot : 1
----------------------------------------------------------
----------------------------------------------------------
Service-Tag : 8XRH0T2
Chassis-Slot : 1
----------------------------------------------------------
----------------------------------------------------------
Service-Tag : ST0000W

Chassis-Slot : 1
show smartfabric discovered-servers discovered-

server-interface
The show smartfabric discovered-servers discovered-server-interface <compute-sled service
tag> command shows list of discovered servers NIC connections.
OS10# show smartfabric discovered-server discovered-server-interface DTQHMR2

Nic-Id : Switch-Interface
------------------------------------------------------
NIC.Mezzanine.1A-1-1 8XRJ0T2:ethernet1/1/3
NIC.Mezzanine.1A-2-1 87QNMR2:ethernet1/71/2
Troubleshooting common issues

This section discusses the various issues that you may encounter when configuring the scenarios and examples that are
mentioned in this guide. A problem statement is given for each scenario, along with one or more possible solutions.
Table 14. Problem and resolution examples

Problem Scenario Solution
MX7116n FEMs are Two MX7000 chassis are connected in an MCM When resolving the issue, consider the following:
not discovered group with MX9116n FSEs and MX7116n FEMs. 1. Do not enable LLDP under the Discovery option
when creating a MX9116n FSEs are connected to Upstream in the distributed virtual switch settings. LLDP is
SmartFabric. switches. Upstream switches are connected to rack not a supported Discovery protocol on a
servers, and vCenter is deployed in this scenario. Distributed Virtual Switch in ESXi on the MX
VMs are also deployed on the ESXi hosts MX platform.
Compute sleds and Rack servers. 2. Disable Beacon Probing and revert to Link
The Link Layer Discovery Protocol (LLDP) Status only on all port groups. This can be done
advertisements from the blade NICs may not be under Port-group settings > Teaming and
visible to the IOMs. Running the show lldp Failover.
neighbors command from the IOM does not list
the NICs shown here
In the blade iDRAC, the NICs status shows as

Unknown, and the Switch Connection ID and
Switch Port Connection ID are shown as Not
Applicable.
3. If the NICs are configured for Jumbo Frames,

try turning this off.
4. Set up the Traffic Filtering (ACL) to drop LLDP
This issue may prevent MX7116n from being packets in ingress and egress direction. Verify
discovered when creating a SmartFabric. that the same ACL does not exist on any
physical switch or virtual switch where the
SmartFabric is expected to be interconnected.
Dropped packets Two MX7000 chassis are connected in an MCM The issue is when one of the MX9116n FSE on
between VMs for group with MX9116n FSEs and MX7116n FEMs. MX7000 chassis becomes the Spanning Tree root.
15 seconds after MX9116n FSEs are connected to Upstream
the switch reboots

Table 14. Problem and resolution examples (continued)
Problem Scenario Solution
switches. Upstream switches are connected to rack To resolve this issue, make an upstream switch the
servers, and vCenter is deployed in this scenario. STP Root, not the MX9116n FSE. In the topology
VMs are also deployed on the ESXi hosts MX mentioned here, the switch with the lower priority
Compute sleds and Rack servers. Verify that STP is number increases the likelihood that the bridge to it
enabled. becomes the STP Root.
Rebooting the MX9116n FSE on the MX7000 Run the commands mentioned in the SmartFabric
chassis while passing traffic between the VMs OS10 User Guide to make upstream switch the STP
deployed on the MX compute sleds and the VMs root.
that are deployed on rack servers causes three to
five requested time outs and dropped packets for
up to 15 seconds.
Not able to set QoS Scenario: I/O Modules MX9116n FSE or MX5108 is By default, the MX9116n FSE and MX5108n IOMs
on a compute sled connected to MX740c compute sled with Intel support the DCBx protocol and can be used to push
connected to XXV710 ethernet controller. IOMs are connected to their QoS configuration to the server NIC. The NIC
MX9116n FSE or upstream switches must be configured to accept these QoS settings
MX5108 from the switch by setting their Remote Willing
Running show lldp dcbx interface ethernet <node/
Status to Enable.
slot/port> pfc detail command shows Remote
willingness status is disabled on server facing ports. In Full switch mode, user can configure DCBx as
mentioned in SmartFabric OS10 User Guide.
OS10# show lldp dcbx interface ethernet 1/1/1 pfc
detail In SmartFabric mode, DCBX configuration is tied to
FCOE UPLINK and it will enable only after FCOE
Uplink configured on this switch.
Admin mode is on
Once DCBX configuration applied on switch side, it
Admin is enabled, Priority list is 4,5,6,7 will push to remote end and remote end must
accept this configuration by “Remote Willing Status
Remote is enabled, Priority list is 4,5,6,7
Enabled”.
Remote Willing Status is disabled
(Output Truncated)
The NIC on the server that is attached to the
switch is not configured to receive DCBx or any
QoS configurations, which is what causes the
Remote Willing Status is disabled message. Some
server NICs will only receive a QoS configuration
(scheduling, bandwidth, priority queues, etc.) from
the switch they are attaching to. The drivers for
these NICs do not support this configuration via
software, but only from a peer via the DCBx
protocol.
Removing the To reproduce the scenario with MX IOMs In a Full-Switch mode, user can create a VLAN,
management VLAN connected to Upstream switches: enable it and define as a Management VLAN in
tag under Edit global configuration mode on switch. For more
1. Create management VLAN.
Uplinks removes information on Configuring VLANs in Full switch
the management 2. After creating SmartFabric and adding uplinks,
mode, see the Dell EMC SmartFabric OS10 User
VLAN the VLANs can be edited from the Edit Uplinks
Guide.
page.
3. Go to OME-M Console > Devices > Fabric > In SmartFabric mode, management VLAN 4020 will
Select a fabric > Select uplink > Edit. be created by default.
4. Click Next to access Edit Uplink page. Make sure not to add management VLAN by Add
5. Add Network and add management VLAN Network or remove tag on management VLAN.
6. Tag the management VLAN. The GUI accepts
This removes the management VLAN itself.
the change but no change in device. Access the
CLI to confirm.
7. Remove the tag on management VLAN, this in
turn deletes the management VLAN as well.

10
Uplink Configuration Scenarios
Topics:
• Scenario 1: SmartFabric deployment with Z9100-ON upstream switches with Ethernet - No Spanning Tree uplink
• Scenario 2: SmartFabric connected to Cisco Nexus 3232C switches with Ethernet - No Spanning Tree uplink
• Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode
• Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC Direct Attach
• Scenario 5: Connect MX5108n to Fibre Channel storage - FSB
• Scenario 6: Configure boot from SAN
• Scenario 7: SmartFabric deployment with Z9100-ON upstream switches with legacy Ethernet uplink
• Scenario 8: SmartFabric connected to Cisco Nexus 3232C switches with legacy Ethernet uplink
Scenario 1: SmartFabric deployment with Z9100-ON

upstream switches with Ethernet - No Spanning Tree
uplink
The following figure shows a topology using a pair of Dell EMC PowerSwitch Z9100-ON upstream switches, but any SmartFabric
OS10 switches can be used. This section details configuration of the Z9100-ON with Ethernet - No Spanning Tree uplink as well
as validation of the Z9100-ON configuration. It also includes instructions on how to configure SmartFabric.
Uplink Configuration Scenarios 109

Figure 110. SmartFabric with Dell EMC PowerSwitch Z9100-ON leaf switches
NOTE: See the Dell EMC PowerEdge MX Network Architecture Guide for more information about the QSFP28-DD cables.
Configure SmartFabric
Perform the following steps to configure SmartFabric:
1. Physically cable the MX9116n FSE to the Z9100-ON upstream switch. Make sure that chassis are in a Multi-Chassis
Management group. To create a MCM Group, see Chapter 5.
2. Define VLANs to use in the Fabric. For instructions, see Define VLANs on page 39.
3. Create the SmartFabric as per instructions in Create the SmartFabric on page 40.
4. Configure uplink port speed or breakout. For more instructions, see Configuring port speed and breakout on page 29.
5. After the SmartFabric is created, create the Ethernet - No Spanning Tree uplink. See Create Ethernet – No Spanning Tree
uplink on page 42 for more information.
6. Set MX I/O modules global spanning tree configurations to Rapid Spanning Tree Protocol (RSTP).
7. Create and deploy the appropriate server templates to the compute sleds. See Server Deployment on page 47 for more
information.
110 Uplink Configuration Scenarios

Dell EMC PowerSwitch Z9100-ON configuration
This section outlines the configuration commands issued to the Dell EMC PowerSwitch Z9100-ON switches with Ethernet - No
Spanning Tree uplink connected from MX9116n FSE to Z9100-ON. The switches start with their factory default settings as
indicated in the Reset SmartFabric OS10 switch to factory defaults section.
NOTE: With Ethernet - No Spanning Tree uplink, spanning tree is disabled on the upstream port channel on the MX I/O
modules. To disable spanning tree on ports connected to MX I/O modules, run the commands below on the Dell EMC
PowerSwitch Z9100-ON.
NOTE: For information related to the same scenario using the legacy Ethernet uplink with Spanning Tree Protocol, see
Scenario 7: SmartFabric deployment with Z9100-ON upstream switches with legacy Ethernet uplink on page 128.
There are four steps to configure the Z9100-ON upstream switches:
1. Set the switch hostname and management IP address. Enable spanning-tree mode as RSTP.
2. Configure the VLT between the switches.
3. Configure the VLANs.
4. Configure the port channels to connect to the MX switches.
Use the following commands to set the hostname, and to configure the OOB management interface and default gateway.
Z9100-ON Leaf 1 Z9100-ON Leaf 2
hostname Z9100-Leaf1 hostname Z9100-Leaf2

spanning-tree mode rstp spanning-tree mode rstp
interface mgmt 1/1/1 interface mgmt 1/1/1

no ip address dhcp no ip address dhcp
no shutdown no shutdown
ip address 100.67.XX.XX/24 ip address 100.67.YY.YY/24
management route 0.0.0.0/0 100.67.XX.XX management route 0.0.0.0/0 100.67.YY.YY
Configure the VLT between switches using the following commands. VLT configuration involves setting a discovery interface
range and discovering the VLT peer in the VLTi. The vlt-domain command configures the peer leaf-2 switch as a back-up
destination.
interface range ethernet1/1/29-1/1/31 interface range ethernet1/1/29-1/1/31

description VLTi description VLTi
no switchport no switchport
vlt-domain 1 vlt-domain 1
backup destination 100.67.YY.YY backup destination 100.67.XX.XX
discovery-interface ethernet1/1/29-1/1/31 discovery-interface ethernet1/1/29-1/1/31
Configure the required VLANs on each switch. In this deployment example, the VLAN used is VLAN 10 and the Untagged VLAN
used is VLAN 1.
interface vlan1 interface vlan1

description “Default VLAN” description “Default VLAN”
description “Company A General Purpose” description “Company A General Purpose”

Configure the port channels that connect to the downstream switches. The LACP protocol is used to create the dynamic LAG.
Trunk ports allow tagged VLANs to traverse the trunk link. In this example, the trunk is configured to allow VLAN 10. Disable the
spanning tree on port channels and run the commands related to Ethernet - No Spanning Tree uplinks as mentioned in the
following.
interface port-channel1 interface port-channel1

description "To MX Chassis" description "To MX Chassis"
switchport mode trunk switchport mode trunk
switchport access vlan 1 switchport access vlan 1
switchport trunk allowed vlan 10 switchport trunk allowed vlan 10
vlt-port-channel 1 vlt-port-channel 1
mtu 9216 mtu 9216
spanning-tree bpduguard enable spanning-tree bpduguard enable
spanning-tree guard root spanning-tree guard root
spanning-tree disable spanning-tree disable
spanning-tree port type edge spanning-tree port type edge
interface ethernet1/1/1 interface ethernet1/1/1

description "To MX Chassis-1" description "To MX Chassis-1"
channel-group 1 mode active channel-group 1 mode active

end end
Dell EMC PowerSwitch Z9100-ON validation

This section contains validation commands for the Dell EMC PowerSwitch Z9100-ON leaf switches.
show vlt
The show vlt command validates the VLT configuration status when the VLTi Link Status is up. The role of one switch in the
VLT pair is primary, and its peer switch (not shown) is assigned the secondary role.

Domain ID : 1
Unit ID : 1
Role : primary
Version : 1.0
VLT MAC address : 4c:76:25:e8:f2:c0
IP address : fda5:74c8:b79e:1::1
Delay-Restore timer : 90 seconds
Peer-Routing : Disabled
Peer-Routing-Timeout timer : 0 seconds
VLTi Link Status
port-channel1000 : up
VLT Peer Unit ID System MAC Address Status IP Address Version

--------------------------------------------------------------------------------
2 4c:76:25:e8:e8:40 up fda5:74c8:b79e:1::2 1.0

show lldp neighbors
The show lldp neighbors command provides information about connected devices. In this case, ethernet1/1/1 and
ethernet1/1/3 connect to the two MX9116n FSEs, C160A2 and C140A1 . The remaining links, ethernet1/1/29, and
ethernet 1/1/31, represent the VLTi connection.
Z9100-Leaf1# show lldp neighbors

----------------------------------------------------------------
ethernet1/1/3 C140A1 ethernet1/1/41 20:04:0f:00:cd:1e

displays name, description, id, media type, native vlan, configured interfaces, and network profile associated with fabric. Run this
command on MX9116n FSE. The following output shows that the uplink created is an Ethernet - No Spanning Tree uplink.
MX9116n-A1# show smartfabric uplinks

----------------------------------------------------------
Name : Uplink 1
Description :
ID : 3d4f2222-f082-43c1-b034-b14a8df3a172
Media Type : Ethernet - No Spanning Tree
Native Vlan : 1
Untagged-network :
Networks : 9418125b-5f1f-48d7-8b5d-648b0977c643
Configured-Interfaces : 87QNMR2:ethernet1/1/41, 87QNMR2:ethernet1/1/42
8XRJ0T2:ethernet1/1/41, 8XRJ0T2:ethernet1/1/42
----------------------------------------------------------
Scenario 2: SmartFabric connected to Cisco Nexus

3232C switches with Ethernet - No Spanning Tree
uplink
The figure below shows a topology using a pair of Cisco Nexus 3232C as leaf switches, but other Cisco Nexus switches may be
used. This section details configuration of the Cisco Nexus switch with Ethernet - No Spanning Tree uplink, validation of the
topology with Cisco Nexus switches, and creation of a SmartFabric with the corresponding uplinks.

Figure 111. SmartFabric with Cisco Nexus 3232C leaf switches
Configure SmartFabric
Perform the following steps to configure SmartFabric:
1. Physically cable the MX9116n FSE to the Cisco Nexus upstream switch. Make sure that chassis are in a Multi-Chassis
Management group. To create a MCM Group, see Chapter 5.
2. Define VLANs to use in the Fabric. For instructions, see Define VLANs on page 39.
3. Create the SmartFabric as per instructions in Create the SmartFabric on page 40.
4. Configure uplink port speed or breakout. For more instructions, see Configuring port speed and breakout on page 29.
5. After the SmartFabric is created, create the Ethernet - No Spanning Tree uplink. See Create Ethernet – No Spanning Tree
uplink on page 42 for more information.
6. Set MX I/O modules global spanning tree configurations to Rapid Spanning Tree Protocol (RSTP).
7. Create and deploy the appropriate server templates to the compute sleds. See Server Deployment on page 47 for more
information.

Cisco Nexus 3232C switch configuration
The following section outlines the configuration commands that are issued to the Cisco Nexus 3232C leaf switches with
Ethernet - No Spanning Tree uplink connected from MX9116n FSE to the Cisco Nexus switch.
NOTE: While this configuration example is specific to the Cisco Nexus 3232C switch, the same concepts apply to other
Cisco Nexus and IOS switches.
The switches start at their factory default settings, as described in the Reset Cisco Nexus 3232C to factory defaults section.
NOTE: With Ethernet - No Spanning Tree Uplink, spanning tree is disabled on upstream port channel on MX I/O modules.
To disable spanning tree on ports connected to MX I/O modules, run the commands below on the Cisco Nexus switches. In
this deployment example, default VLAN is VLAN 1 and the created VLAN is VLAN 10. See the Cisco Nexus 3000 Series NX-
OS Configuration Guide for more details.
NOTE: For information related to the same scenario using the legacy Ethernet uplink with Spanning Tree Protocol, see
Scenario 8: SmartFabric connected to Cisco Nexus 3232C switches with legacy Ethernet uplink on page 132.
There are four steps to configure the 3232C upstream switches:
1. Set switch hostname, management IP address, enable features vPC, LLDP, LACP, and interface-vlan.
2. Configure vPC between the switches.
4. Configure the downstream port channels to connect to the MX switches.
Enter the following commands to set the hostname and enable required features. Configure the management interface and
default gateway. Also run the global setting commands for Spanning Tree Protocol as mentioned in the following.
NOTE: The MX IOMs run Rapid per-VLAN Spanning Tree Plus (RPVST+) by default. Cisco Nexus switches run RSTP by
default. Ensure the Dell and non-Dell switches are both configured to use RSTP. For the Ethernet - No spanning Tree
uplinks from MX9116n FSE to the Cisco Nexus switches, spanning tree must be disabled on ports connected to MX I/O
modules.
Cisco Nexus 3232C Leaf 1 Cisco Nexus 3232C Leaf 2
hostname 3232C-Leaf1 hostname 3232C-Leaf2
feature vpc feature vpc

feature lldp feature lldp
feature lacp feature lacp
feature interface-vlan feature interface-vlan
spanning-tree port type edge bpduguard spanning-tree port type edge bpduguard
default default
spanning-tree port type network default spanning-tree port type network default
interface mgmt0 interface mgmt0

vrf member management vrf member management
vrf context management vrf context management

ip route 0.0.0.0/0 100.67.XX.XX ip route 0.0.0.0/0 100.67.YY.YY
Enter the following commands to create a virtual port channel (vPC) domain and assign the keepalive destination to the peer
switch management IP. Then create a port channel for the vPC peer link and assign the appropriate switchport interfaces.
vpc domain 255 vpc domain 255

peer-keepalive destination 100.67.YY.YY peer-keepalive destination 100.67.XX.XX

switchport switchport

vpc peer-link vpc peer-link
interface Ethernet1/29 interface Ethernet1/29

description vPC Interconnect description vPC Interconnect

Configure the required VLANs on each switch. In this deployment example, the Tagged VLAN used is VLAN 10 and Untagged
VLAN used is VLAN 1. Disable spanning tree on VLANs.

description “Default VLAN” description “Default VLAN”
no spanning-tree mode no spanning-tree mode

no spanning-tree mode no spanning-tree mode
Enter the following commands to configure the port channels to connect to the downstream MX9116n FSEs. Then, exit
configuration mode and save the configuration. Disable spanning tree on the port channel connected to MX9116n FSE.

description To MX Chassis description To MX Chassis
switchport trunk allowed vlan 1,10 switchport trunk allowed vlan 1,10
spanning-tree bpduguard enable spanning-tree bpduguard enable
spanning-tree port type edge spanning-tree port type edge
spanning-tree guard root spanning-tree guard root
vpc 255 vpc 255

description To MX Chassis 1 description To MX Chassis 1

end end
copy running-configuration startup- copy running-configuration startup-
configuration configuration

NOTE: If the connections to the MX switches do not come up, see the SmartFabric Troubleshooting chapter for
troubleshooting steps.
Trunk ports on switches allow tagged traffic to traverse the links. All flooded traffic for the VLAN is sent across trunk ports to
all the switches, even if those switches do not have an associated VLAN. This takes up the network bandwidth with
unnecessary traffic. VLAN or VTP Pruning is the feature that can be used to eliminate this unnecessary traffic by pruning the
VLANs.
Pruning restricts the flooded traffic to only those trunk ports with associated VLANs to optimize the usage of network
bandwidth. If the existing environment is configured for Cisco VTP or VLAN pruning, ensure that the Cisco upstream switches
are configured appropriately. See the Cisco Nexus 3000 Series NX-OS Configuration Guide for additional information.
NOTE: Do not use switchport trunk allow vlan all on the Cisco interfaces. The VLANs must be explicitly
assigned to the interface.
Configuration validation
This section covers the validation of the Cisco Nexus 3232C leaf switches. For information about the Dell EMC Networking MX
switch validation commands, see the CLI commands section.
show vpc
The show vpc command validates the vPC configuration status. The peer adjacency should be OK, with the peer should show
as alive. The end of the command shows which VLANs are active across the vPC.
NX3232C-Leaf1# show vpc

Legend:
vPC domain id : 255

Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 inconsistency reason : Consistency Check Not Performed
vPC role : secondary, operational primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po255 up 1,10
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
255 Po1 up success success 1,10
show vpc consistency-parameters

The show vpc consistency-parameters command displays the configured values on all interfaces in the vPC. The
displayed configurations are only those configurations that limit the vPC peer link and vPC from coming up.
NX3232C-Leaf1# show vpc consistency-parameters vpc 255

Legend:

Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------
STP Port Type 1 Normal Port Normal Port
STP Port Guard 1 Default Default
STP MST Simulate PVST 1 Default Default
lag-id 1 [(1000, [(1000,
20-4-f-0-cd-1e, 1, 0, 20-4-f-0-cd-1e, 1, 0,
0), (7f9b, 0), (7f9b,
0-23-4-ee-be-ff, 80ff, 0-23-4-ee-be-ff, 80ff,
0, 0)] 0, 0)]
mode 1 active active
delayed-lacp 1 disabled disabled
Speed 1 100 Gb/s 100 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Dot1q Tunnel 1 no no
Switchport Isolated 1 0 0
vPC card type 1 N9K TOR N9K TOR
Allowed VLANs - 1,10 1,10
Local suspended VLANs - - -
show lldp neighbors

The show lldp neighbors command provides information about lldp neighbors. In this example, Eth1/1 and Eth1/3 are
connected to the two MX9116n FSEs, C160A2 and C140A1. The remaining links, Eth1/29 and Eth1/31, represent the vPC
connection.
NX3232C-Leaf1(config)# show lldp neighbors

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
S3048-ON mgmt0 120 PBR ethernet1/1/45
C160A2 Eth1/1 120 PBR ethernet1/1/41
NX3232C-Leaf2 Eth1/29 120 BR Ethernet1/29
Total entries displayed: 5

displays name, description, id, media type, native vlan, configured interfaces, and network profile associated with fabric. Run this
command on MX9116n FSE. The following output shows that the uplink created is an Ethernet - No Spanning Tree uplink.
MX9116n-A1# show smartfabric uplinks

----------------------------------------------------------
Name : Uplink 1
Description :
ID : 3d4f2222-f082-43c1-b034-b14a8df3a172
Media Type : Ethernet - No Spanning Tree
Native Vlan : 1
Untagged-network :
Networks : 9418125b-5f1f-48d7-8b5d-648b0977c643
Configured-Interfaces : 87QNMR2:ethernet1/1/41, 87QNMR2:ethernet1/1/42
8XRJ0T2:ethernet1/1/41, 8XRJ0T2:ethernet1/1/42
----------------------------------------------------------

Scenario 3: Connect MX9116n FSE to Fibre Channel
storage - NPIV Proxy Gateway mode
This chapter discusses a method for connecting the MX9116n FSE to an FC storage array connected to existing FC switches
using the NPIV Proxy Gateway (NPG) mode for the connection. NPG mode allows for larger SAN deployments that aggregate
I/O traffic at the NPG switch.
Figure 112. FC (NPG) network to Dell EMC Unity
This scenario shows attachment to a brownfield FC switch infrastructure. Configuration of the existing FC switches is beyond
the scope of this document.
NOTE: The MX5108n Ethernet Switch does not support this feature.
This example assumes that an existing SmartFabric has been created and is fully operational. For instructions on creating a
SmartFabric, see Chapter 4.
To configure NPG mode on an existing SmartFabric, the following steps are completed using the OME-M console:
1. Connect the MX9116n FSE to the FC SAN.
CAUTION: Ensure that the cables do not criss-cross between the switches
.
2. Define FCoE VLANs to use in the fabric. For instructions, see the Define VLANs section for information about defining the
VLANs.
3. If necessary, create the Identity Pools. See the Create identity pools section for more information about how to create
identity pools.
4. Configure the physical switch ports for FC operation. See the Configure Fibre Channel universal ports section for
instructions.
5. Create the FC Gateway uplinks. For instruction, see the Create Fibre Channel uplinks section for steps on creating uplinks.
6. Create and deploy the appropriate server templates to the compute sleds. See Chapter 5 for more information.
Once the server operating system loads the FCoE driver, the WWN appears on the fabric and on the FC SAN. The system is
now ready to connect to Fibre Channel storage. See Appendix B for setting up storage logical unit numbers (LUNs).
NOTE: For information related to use cases and configuring Ethernet – No Spanning Tree uplink with different tagged and
untagged VLANs, see Ethernet – No Spanning Tree Uplink.
NOTE: When MX9116n FSEs are in NPG mode, connecting to more than one SAN is possible by creating multiple vFabrics
each with their own NPG gateway only in Full Switch mode. However, an individual server can only connect to one vFabric
at a time, so one server cannot see both SANs.

show fcoe sessions

C140A1# show fcoe sessions

-----------------------------------------------------------------------------------------
----------------------------------------------------------------
06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:0c
show vfabric
C140A1# show vfabric

Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
Config-State ACTIVE
Oper-State UP
==========================================
==========================================
Domain ID 1
==========================================
==========================================
==========================================
Members
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2
show fc switch
The show fc switch command verifies the switch mode (for example, F_Port) for FC traffic.
C140A1# show fc switch

Switch Mode : FPORT
Switch WWN : 10:00:e4:f0:04:6b:04:42

Scenario 4: Connect MX9116n FSE to Fibre Channel
storage - FC Direct Attach
This chapter discusses a method for connecting an FC storage array directly to the MX9116n FSE.
On PowerEdge MX platform, the difference between configuring NPG mode or FC Direct Attach mode on the MX9116n FSE is
selecting different uplink type desired.
Figure 113. Fibre Channel (F_Port) Direct Attach to Dell EMC Unity
This example shows directly attaching a Dell EMC Unity 500F storage array to the MX9116n FSE using universal ports 44:1 and
44:2.
To configure NPG mode on an existing SmartFabric, the following steps are completed using the OME-M console:
1. Connect the storage array to the MX9116n FSE. Each storage controller is connected to each MX9116n FSE. Define FCoE
VLANs to use in the fabric. For instructions, see Define VLANs for defining VLANs.
2. If necessary, create Identity Pools. See the Create identity pools section for more information about how to create identity
pools.
3. Configure the physical switch ports for FC operation. See the Configure Fibre Channel universal ports section for
instructions.
4. Create the FC Direct Attached uplinks. For more information about creating uplinks, see the Create Fibre Channel uplinks
section.
6. Configure zones and zone sets. See the Manage Fibre Channel Zoning on the MX9116n FSE section for instructions.
Once the server operating system loads the FCoE, the WWN appears on the fabric and on the FC SAN. The system is now
ready to connect to Fibre Channel storage. See Appendix B for setting up storage logical unit numbers (LUNs).
NOTE: For information related to use cases and configuring Ethernet - No Spanning Tree uplink with different tagged and
untagged VLANs, see the Ethernet - No Spanning Tree Uplink section.

show fc ns switch
The show fc ns switch command shows all device ports that are logged into the fabric. In this deployment, four ports are
logged in to each switch: two storage ports and two CNA ports.
C140A1# show fc ns switch
Total number of devices = 3

Switch Name 10:00:20:04:0f:00:cd:1e
Domain Id 1
Switch Port fibrechannel1/1/44:1
FC-Id 01:00:00
Port Name 50:06:01:61:47:e4:1b:19
Node Name 50:06:01:60:c7:e0:1b:19
Class of Service 8
Symbolic Port Name UNITY::::SPA13::FC::::::
Symbolic Node Name UNITY::::SPA::FC::::::
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

Domain Id 1
Switch Port ethernet1/71/1
FC-Id 01:01:00
Port Name 20:01:06:c3:f9:a4:cd:03
Node Name 20:00:06:c3:f9:a4:cd:03
Class of Service 8
Symbolic Port Name
Symbolic Node Name
Port Type N_PORT

Domain Id 1
Switch Port ethernet1/1/1
FC-Id 01:02:00
Port Name 20:01:f4:e9:d4:73:d0:0c
Node Name 20:00:f4:e9:d4:73:d0:0c
Class of Service 8
Symbolic Port Name QLogic qedf v8.24.8.0
Symbolic Node Name QLogic qedf v8.24.8.0
Port Type N_PORT
show fcoe sessions

C140A1# show fcoe sessions

-----------------------------------------------------------------------------------------
----------------------------------------------------------------
06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:0c

show vfabric
C140A1# show vfabric

Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
Config-State ACTIVE
Oper-State UP
==========================================
==========================================
Domain ID 1
==========================================
==========================================
==========================================
Members
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2
show fc switch
The show fc switch command verifies the switch mode (for example, F_Port) for FC traffic.
C140A1# show fc switch

Switch Mode : FPORT
Switch WWN : 10:00:e4:f0:04:6b:04:42
Scenario 5: Connect MX5108n to Fibre Channel

storage - FSB
This chapter provides instructions for connecting either the MX5108n or MX9116n to a Fibre Channel SAN using native FCoE
uplinks. This connection type would be used in an environment where an existing switch such as the Dell EMC PowerSwitch
S4148U has the capability to accept native FCoE and connect to native FC.
Dell EMC SmartFabric OS10 uses a FIP Snooping Bridge (FSB) to detect and manage FCoE traffic and discovers the following
information:
● End nodes (E_Nodes)
● Fibre Channel forwarder (FCF)
● Connections between E_Nodes and FCFs
● Sessions between E_Nodes and FCFs
Using the discovered information, the switch installs ACL entries that provide security and point-to-point link emulation to
ensure that FCoE traffic is handled appropriately.
NOTE: The examples in this chapter use the Dell EMC Networking MX5108n. The same instructions may also be applied and
used with the MX9116n.
The FSB switch can connect to an upstream switch operating in NPG mode:

Figure 114. FCoE (FSB) Network to Dell EMC Unity through NPG mode switch
Or operating in F_Port mode:
Figure 115. FCoE (FSB) Network to Dell EMC Unity through F_Port mode switch
NOTE: See the Dell EMC SmartFabric OS10 Documentation for configuring NPG mode globally on the Dell EMC Networking
S4148U-ON switches.
SmartFabric configuration steps

1. To configure FCoE mode on an existing SmartFabric, the following steps are completed using the OME-M console: Connect
the MX switch to the S4148U.
CAUTION: Verify that the cables do not criss-cross between the switches.
2. Define FCoE VLANs to use in the fabric. For instructions, see the Define VLANs section for more information about defining
the VLANs.
3. If necessary, create Identity Pools. See the Create identity pools section for more information.
4. Create the FCoE uplinks. See the Create Fibre Channel uplinks section for more information about creating uplinks.
6. Configure the S4148U switch. See the Dell EMC Networking Fibre Channel Deployment with S4148U-ON in F_port Mode
knowledge base article for more information.

Once the server operating system loads the FCoE driver, the WWN displays on the fabric and on the FC SAN. The system is
now ready to connect to Fibre Channel storage. See Appendix B for setting up storage logical unit numbers (LUNs).
To validate the configuration, use the same commands that are mentioned in Chapter 8.
Scenario 6: Configure boot from SAN

The host operating system of MX Server can boot from a remote FC storage array using the IOMs. Booting to an operating
system through FC direct attach (F_port), FC (NPG), and FCoE (FSB) scenarios are supported.
Figure 116. Boot from SAN
The figure below shows the example topology that is used in this chapter to demonstrate Boot from SAN. The required steps
are provided to configure NIC partitioning, system BIOS, an FCoE LUN, and an OS install media device required for Boot from
SAN.
Figure 117. FCoE boot from SAN
NOTE: See the OS10 Enterprise Edition User Guide for configuring NPG mode globally on the S4148U-ON switches.
Configure NIC boot device

In this section, each QLogic CNA port is partitioned into one Ethernet and one FCoE partition.
NOTE: This is only done on CNA ports that carry converged traffic. In this example, these are the two 25GbE QLogic CNA
ports on each server that attach to the switches internally through an orthogonal connection.
1. Connect to the server's iDRAC in a web browser and launch the virtual console.
2. In the virtual console, select BIOS Setup from the Next Boot menu.
3. Reboot the server.
4. On the System Setup Main Menu, select Device Settings.

5. Select the first CNA port.
6. Select Device Level Configuration.
7. Set the Virtualization Mode to NPAR (if not already set), and click Back.
Figure 118. Virtualization mode to NPAR

8. Choose NIC Partitioning Configuration.
9. Select Partition 1 Configuration.
10. Set NIC + RDMA Mode to Disabled.
Figure 119. Set the value of NIC and RDMA mode

11. Click Back to return.
12. Select Partition 2 Configuration.
13. Set FCoE Mode to Enabled as shown.
Figure 120. FCoE mode to Enabled

14. Click Back and select Back to go to Main Configuration Page.
15. Select NIC Configuration, then set the Boot Protocol to UEFI FCoE, and then click Back.
Figure 121. Set value of Boot Protocol to UEFI FCoE

16. If present, select Partition 3 Configuration in NIC Partitioning Configuration.
17. Set all modes to Disabled and then click Back.

18. If present, select Partition 4 Configuration in NIC Partitioning Configuration.
19. Set all modes to Disabled and then click Back.
20. Select FCoE Configuration.
NOTE: It is not required to have a Virtual LAN ID setup in the CNA, as the CNA uses FIP discovery on the untagged
VLAN to obtain the FCoE VLAN.
21. Set Connect 1 to Enabled.
22. Set the World Wide Port Name Target 1 to the connected port on Unity.
Figure 122. FCoE configuration

23. Click Back and then click Finish.
24. When prompted, answer Yes to save changes and click OK in the Success window.
25. Select the second CNA port and repeat the steps in this section for port 2.
26. Click Finish to exit to the System Setup Main Menu.
Configure BIOS settings

To allow boot from SAN, perform the following steps in the system BIOS settings to disable the PXE BIOS.
1. Select System BIOS from the System Setup Main Menu.
2. Select Network Settings.
3. Click Disable for all PXE Devices.
4. Click Back.
5. Click Finish, click Finish again, then select Yes to exit and reboot.
NOTE: As previously documented, this server configuration may be used to generate a template to deploy to other servers
with identical hardware. When a template is not used, repeat the steps in this chapter for each MX server sled that requires
access to the FC storage.
Connect FCoE LUN

The server should be provisioned to connect to an FCoE boot LUN before moving on. Follow the procedures in Appendix B to
configure and connect to an FCoE LUN. Once connected, continue to the steps below to complete the Boot from SAN
configuration.
Set up and install media connection

NOTE: The steps in this section were completed using the iDRAC Java Virtual Console.
1. Connect to the server’s iDRAC in a web browser and launch the virtual console.
2. In the virtual console, from the Virtual Media menu, select Virtual Media.
3. In the virtual console, from the Virtual Media menu, select Map CD/DVD.
4. Click Browse to find the location of the operating system install media then click Map Device.
5. In the virtual console, from the Next Boot menu, select Lifecycle Controller.

6. Reboot the server.
Use Lifecycle Controller to set up operating system driver for

media installation
The installation media for some operating systems do not contain the necessary FCoE drivers to boot from a FCoE LUN. Use
this procedure to create an internal operating system install media device.
NOTE: For VMware ESXi, see the Dell customized media instructions provided on the Dell EMC Support website.
1. In Lifecycle Controller, select OS Deployment, then select Deploy OS.

2. From the Select an Operating System screen, verify that Boot mode is set to UEFI.
3. Select an operating system to install to the boot LUN.
Figure 123. Lifecycle Controller operating system deployment menu

4. Click Next.
5. Click the Manual Install check box, then click Next.
6. Click Next on the Insert OS Media screen.
7. Click Finish when prompted on the Reboot System screen.
8. System reboots to Virtual Media. Press any key to boot install media when prompted.
9. Follow the operating system prompts to install the operating system to the FCoE storage LUN.
Scenario 7: SmartFabric deployment with Z9100-ON

upstream switches with legacy Ethernet uplink
The following figure shows a topology using a pair of Dell EMC PowerSwitch Z9100-ONs as upstream switches. This section
walks through configuring the Z9100-ON and validating the Z9100-ON configuration, but any SmartFabric OS10 switches can be
used. This section details configuration of the Z9100-ON as well as validation of the Z9100-ON configuration.
NOTE: For information related to the same scenario using the Ethernet - No Spanning Tree uplink (recommended), see
Scenario 1: SmartFabric deployment with Z9100-ON upstream switches with Ethernet - No Spanning Tree uplink on page
109.

Figure 124. SmartFabric with Dell EMC PowerSwitch Z9100-ON leaf switches
Dell EMC PowerSwitch Z9100-ON configuration

This section outlines the configuration commands issued to the Dell EMC PowerSwitch Z9100-ON switches. The switches start
with their factory default settings as indicated in the Reset SmartFabric OS10 switch to factory defaults section.
NOTE: The MX IOMs run Rapid Per-VLAN Spanning Tree Plus (RPVST+) by default. RPVST+ runs RSTP on each VLAN
while RSTP runs a single instance of spanning tree across the default VLAN. The Dell EMC PowerSwitch Z9100-ON used in
this example runs SmartFabric OS10 and has RPVST+ enabled by default.
There are four steps to configure the Z9100-ON upstream switches:
1. Set the switch hostname and management IP address.
2. Configure the VLT between the switches.
4. Configure the port channels to connect to the MX switches.

Use the following commands to set the hostname, and to configure the OOB management interface and default gateway.
hostname Z9100-Leaf1 hostname Z9100-Leaf2

NOTE: Use the spanning-tree {vlan vlan-id priority priority-value} command to set the bridge
priority for the upstream switches. The bridge priority ranges from 0 to 61440 in increments of 4096. For example, to make
Z9100-ON Leaf 1 as the root bridge for VLAN 10, enter the command spanning-tree vlan 10 priority 4096.
Configure the VLT between switches using the following commands. VLT configuration involves setting a discovery interface
range and discovering the VLT peer in the VLTi. vlt-domain configures the peer leaf-2 switch as a back up destination.
interface range ethernet1/1/29-1/1/31 interface range ethernet1/1/29-1/1/31

description VLTi description VLTi
vlt-domain 1 vlt-domain 1
backup destination 100.67.YY.YY backup destination 100.67.XX.XX
discovery-interface ethernet1/1/29-1/1/31 discovery-interface ethernet1/1/29-1/1/31
Configure the required VLANs on each switch. In this deployment example, the VLAN used is VLAN 10.

Configure the port channels that connect to the downstream switches. The LACP protocol is used to create the dynamic LAG.
Trunk ports allow tagged VLANs to traverse the trunk link. In this example, the trunk is configured to allow VLAN 10.

description "To MX Chassis" description "To MX Chassis"
switchport trunk allowed vlan10 switchport trunk allowed vlan10
vlt-port-channel 1 vlt-port-channel 1



end end
Dell EMC PowerSwitch Z9100-ON validation

This section contains validation commands for the Dell EMC PowerSwitch Z9100-ON leaf switches.
show vlt
The show vlt command validates the VLT configuration status when the VLTi Link Status is up. The role of one switch in the
VLT pair is primary, and its peer switch (not shown) is assigned the secondary role.

Domain ID : 1
Unit ID : 1
Role : primary
Version : 1.0
VLT MAC address : 4c:76:25:e8:f2:c0
IP address : fda5:74c8:b79e:1::1
Delay-Restore timer : 90 seconds
Peer-Routing : Disabled
Peer-Routing-Timeout timer : 0 seconds
VLTi Link Status
port-channel1000 : up
VLT Peer Unit ID System MAC Address Status IP Address Version

--------------------------------------------------------------------------------
2 4c:76:25:e8:e8:40 up fda5:74c8:b79e:1::2 1.0
show lldp neighbors

The show lldp neighbors command provides information about connected devices. In this case, ethernet1/1/1 and
ethernet1/1/3 connect to the two MX9116n FSEs, C160A2 and C140A1 . The remaining links, ethernet1/1/29, and
ethernet 1/1/31, represent the VLTi connection.
Z9100-Leaf1# show lldp neighbors

----------------------------------------------------------------
ethernet1/1/3 C140A1 ethernet1/1/41 20:04:0f:00:cd:1e
show spanning-tree brief

The show spanning-tree brief command validates that STP is enabled on the leaf switches. All the interfaces are
forwarding (FWD), as shown in the Sts column.
Z9100-Leaf1# show spanning-tree brief

Spanning tree enabled protocol rapid-pvst
VLAN 1
Root ID Priority 32768, Address 2004.0f00.a19e
Bridge ID Priority 32769, Address 4c76.25e8.f2c0

Flush Indication threshold 0 (MAC flush optimization is disabled)
--------------------------------------------------------------------------------
port-channel1 128.2517 128 50 FWD 0 32768 2004.0f00
Interface
Name Role PortID Prio Cost Sts Cost Link-type Edge
--------------------------------------------------------------------------------
port-channel1 Root 128.2517 128 50 FWD 0 AUTO No
VLAN 10
Root ID Priority 32778, Address 4c76.25e8.e840
Bridge ID Priority 32778, Address 4c76.25e8.f2c0
Flush Indication threshold 0 (MAC flush optimization is disabled)
--------------------------------------------------------------------------------
port-channel1 128.2517 128 50 FWD 1 32768 2004.0f00
Interface
Name Role PortID Prio Cost Sts Cost Link-type Edge
--------------------------------------------------------------------------------port-
channel1 Root 128.2517 128 50 FWD 1 AUTO No
Scenario 8: SmartFabric connected to Cisco Nexus

3232C switches with legacy Ethernet uplink
The figure below shows a topology using a pair of Cisco Nexus 3232C as leaf switches, but other Cisco Nexus switches may be
used. This section details coniguration of the Cisco Nexus 3232Cs and creatin of a SmartFabric with the corresponding legacy
Ethernet uplinks.
NOTE: For information related to the same scenario using Ethernet - No Spanning Tree uplink, refer to Scenario 2:
SmartFabric connected to Cisco Nexus 3232C switches with Ethernet - No Spanning Tree uplink on page 113.

Figure 125. SmartFabric with Cisco Nexus 3232C leaf switches
Cisco Nexus 3232C switch configuration

This section outlines the configuration commands that are issued to the Cisco Nexus 3232C leaf switches.
NOTE: While this configuration example is specific to the Cisco Nexus 3232C switch, the same concepts apply to other
Cisco Nexus and IOS switches.
The switches start at their factory default settings, as described in the Reset Cisco Nexus 3232C to factory defaults section.
NOTE: The MX IOMs run Rapid per-VLAN Spanning Tree Plus (RPVST+) by default. Ensure the Cisco and Dell switches are
configured to use compatible STP protocols. The mode of STP on the Cisco switch can be set using the command
spanning-tree mode, which is shown below. In this deployment example, default VLAN is VLAN 1 and the created VLAN is
VLAN 10. See the Cisco Nexus 3000 Series NX-OS Configuration Guide for more details.
There are four steps to configure the 3232C upstream switches:

1. Set switch hostname, management IP address, enable features and spanning tree.
2. Configure vPC between the switches.
4. Configure the downstream port channels to connect to the MX switches.
Enter the following commands to set the hostname, enable required features, and enable RPVST spanning tree mode. Configure
the management interface and default gateway.
hostname 3232C-Leaf1 hostname 3232C-Leaf2
feature vpc feature vpc

feature lldp feature lldp
feature lacp feature lacp
spanning-tree mode rapid-pvst spanning-tree mode rapid-pvst
interface mgmt0 interface mgmt0

vrf member management vrf member management
vrf context management vrf context management

ip route 0.0.0.0/0 100.67.XX.XX ip route 0.0.0.0/0 100.67.YY.YY
Enter the following commands to create a virtual port channel (vPC) domain and assign the keepalive destination to the peer
switch management IP. Then create a port channel for the vPC peer link and assign the appropriate switchport interfaces.
vpc domain 255 vpc domain 255

peer-keepalive destination 100.67.YY.YY peer-keepalive destination 100.67.XX.XX

vpc peer-link vpc peer-link


Enter the following commands to configure the port channels to connect to the downstream MX9116n FSEs. Then, exit
configuration mode and save the configuration.

description To MX Chassis description To MX Chassis
vpc 255 vpc 255



end end
copy running-configuration startup- copy running-configuration startup-
configuration configuration
NOTE: If the connections to the MX switches do not come up, see the SmartFabric Troubleshooting chapter for
troubleshooting steps.
Trunk ports on switches allow tagged traffic to traverse the links. All flooded traffic for the VLAN is sent across trunk ports to
all the switches, even if those switches do not have an associated VLAN. This takes up the network bandwidth with
unnecessary traffic. VLAN or VTP Pruning is the feature that can be used to eliminate this unnecessary traffic by pruning the
VLANs.
Pruning restricts the flooded traffic to only those trunk ports with associated VLANs to optimize the usage of network
bandwidth. If the existing environment is configured for Cisco VTP or VLAN pruning, ensure that the Cisco upstream switches
are configured appropriately. See the Cisco Nexus 3000 Series NX-OS Configuration Guide for additional information.
NOTE: Do not use switchport trunk allow vlan all on the Cisco interfaces. The VLANs must be explicitly
assigned to the interface.
This section covers the validation of the Cisco Nexus 3232C leaf switches. For information about the Dell EMC Networking MX
switch validation commands, see the CLI commands section.
show vpc
The show vpc command validates the vPC configuration status. The peer adjacency should be OK, with the peer should show
as alive. The end of the command shows which VLANs are active across the vPC.
NX3232C-Leaf1# show vpc

Legend:
vPC domain id : 255

Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 inconsistency reason : Consistency Check Not Performed
vPC role : secondary, operational primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po255 up 1,10
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
255 Po1 up success success 1,10
show vpc consistency-parameters

The show vpc consistency-parameters command displays the configured values on all interfaces in the vPC. The
displayed configurations are only those configurations that limit the vPC peer link and vPC from coming up.
NX3232C-Leaf1# show vpc consistency-parameters vpc 255

Legend:
Type 1 : vPC will be suspended in case of mismatch
Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------
STP Port Type 1 Normal Port Normal Port
STP Port Guard 1 Default Default
STP MST Simulate PVST 1 Default Default
lag-id 1 [(1000, [(1000,
20-4-f-0-cd-1e, 1, 0, 20-4-f-0-cd-1e, 1, 0,
0), (7f9b, 0), (7f9b,
0-23-4-ee-be-ff, 80ff, 0-23-4-ee-be-ff, 80ff,
0, 0)] 0, 0)]
mode 1 active active
delayed-lacp 1 disabled disabled
Speed 1 100 Gb/s 100 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Dot1q Tunnel 1 no no
Switchport Isolated 1 0 0
vPC card type 1 N9K TOR N9K TOR
Allowed VLANs - 1,10 1,10
Local suspended VLANs - - -
show lldp neighbors

The show lldp neighbors command provides information about lldp neighbors. In this example, Eth1/1 and Eth1/3 are
connected to the two MX9116n FSEs, C160A2 and C140A1. The remaining links, Eth1/29 and Eth1/31, represent the vPC
connection.
NX3232C-Leaf1(config)# show lldp neighbors

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
S3048-ON mgmt0 120 PBR ethernet1/1/45
Total entries displayed: 5

show spanning-tree summary
The show spanning-tree summary command validates that STP is enabled on the leaf switches. All interfaces are shown
as forwarding.
NX3232C-Leaf1# show spanning-tree summary

Switch is in rapid-pvst mode
Root bridge for: VLAN0010
Port Type Default is disable
Edge Port [PortFast] BPDU Guard Default is disabled
Edge Port [PortFast] BPDU Filter Default is disabled
Bridge Assurance is enabled
Loopguard Default is disabled
Pathcost method used is short
STP-Lite is disabled
Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0001 0 0 0 2 2
VLAN0010 0 0 0 2 2
---------------------- -------- --------- -------- ---------- ----------
2 vlans 0 0 0 4 4

A
Hardware
Topics:
• Hardware used in this guide
• Dell EMC PowerSwitch S3048-ON
• Dell EMC PowerSwitch Z9100-ON
• Dell EMC PowerSwitch S4148U-ON
• Dell EMC PowerSwitch Z9264F-ON
• Dell EMC Unity 500F storage array
• Cisco Nexus 3232C
Hardware used in this guide

This section covers the rack-mounted networking switches used in the examples in this guide.
Table 15. Hardware and roles

Hardware Role
Dell EMC PowerSwitch S3048-ON One S3048-ON switch supports out-of-band (OOB) management traffic for all
examples.
Dell EMC PowerSwitch Z9100-ON A pair of Z9100-ON switches are used as leaf switches in Scenario 1: SmartFabric
deployment with Z9100-ON upstream switches with Ethernet - No Spanning Tree
uplink on page 109.
Dell EMC PowerSwitch S4148U-ON Two S4148U-ON switches support storage traffic, and are the first of two leaf
switch options.
Dell EMC PowerSwitch Z9264F-ON This switch may be used as a leaf or spine switch in a Leaf-spine topology. It is
optimized for nonblocking 100 GbE leaf/spine fabrics and high-density 25/50 GbE
in-rack server and storage connections. It provides up to 64 ports of 100 GbE
QSFP28 or up to 128 ports of 1/10/25/40/50 GbE ports using breakout cables.
Dell EMC Unity 500F storage array This array is used for the FC connections. Additional 2U Disk Array Enclosures
(DAEs) may be added, providing twenty-five additional drives each.
Cisco Nexus 3232C A pair of Cisco Nexus 3232C switches are used as leaf switches in Scenario 2:
SmartFabric connected to Cisco Nexus 3232C switches with Ethernet - No
Spanning Tree uplink on page 113.
More detail about each of these devices is provided in the following sections.
For detailed information about Hardware components related to the MX Platform, please see the Dell EMC PowerEdge MX
Networking Architecture Guide.
Dell EMC PowerSwitch S3048-ON

The Dell EMC PowerSwitch S3048-ON is a 1U switch with forty-eight 1GbE BASE-T ports and four 10GbE SFP+ ports. This
guide uses one S3048-ON switch in each rack for out-of-band (OOB) management traffic.
138 Hardware
Figure 126. Delll EMC PowerSwitch S3048-ON
Dell EMC PowerSwitch Z9100-ON

The Dell EMC PowerSwitch Z9100-ON is a 1U multilayer switch with 32x QSFP28 ports supporting 10/25/40/50/100 GbE and
2x 10 GbE SFP+ ports.
Figure 127. Dell EMC PowerSwitch Z9100-ON
Dell EMC PowerSwitch S4148U-ON

The Dell EMC PowerSwitch S4148U-ON is a 1U switch with 48x SFP+ ports, 2x QSFP+ ports, and 4x QSFP28 ports.
Figure 128. Dell EMC PowerSwitch S4148U-ON
Dell EMC PowerSwitch Z9264F-ON

The Dell EMC PowerSwitch Z9264F-ON is a 2U, multilayer switch with 64x 100 GbE QSFP28 ports and 2x 10 GbE SFP+ ports.
Figure 129. Dell EMC PowerSwitch Z9264F-ON
Dell EMC Unity 500F storage array

The Unity 500F storage platform delivers all-flash storage with up to 8 PB of raw capacity. It has concurrent support for NAS,
iSCSI, and FC protocols. The Disk Processing Enclosure (DPE) has a 2U form factor, redundant Storage Processors (SPs), and
support for up to twenty-five 2.5" drives.
Hardware 139
Figure 130. Dell EMC Unity 500F front view
Figure 131. Dell EMC Unity 500F rear view
Cisco Nexus 3232C

The Cisco Nexus 3232C is a 1U fixed form-factor 100 GbE switch with thirty-two QSFP28 ports supporting 10/25/40/50/100
GbE.
140 Hardware
B
Dell EMC PowerEdge MX I/O Module Full
Switch mode Configuration – Key
Components
Topics:
• Configure MX IOM in NPIV proxy gateway (NPG) mode
• Configure MX IOM in FIP Snooping Bridge (FSB) mode
Configure MX IOM in NPIV proxy gateway (NPG)

mode
This section only focuses on key components to be considered while configuring MX I/O Module in Full Switch mode. In a
scenario mentioned in Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode, SAN
(FCoE/FC) traffic will get redirected to the FC Switch via Fibre Channel uplinks. NPG mode allows for larger SAN deployments
that aggregate I/O module traffic at the NPG switch.
Configuring MX I/O Modules in Full Switch mode with NPG enabled requires commands mentioned below. These are the key
components to enable NPG on an IOM.
Enable DCBX
Make sure to enable DCBX in global configuration mode:
C140A1# dcbx enable
Enable NPG
Enable the NPG feature in global configuration mode:
C140A1# feature fc npg
Fabric VLAN
Enable VLAN for vFabric services.
C140A1# interface vlan 1001
Dell EMC PowerEdge MX I/O Module Full Switch mode Configuration – Key Components 141
Ethernet VLAN
Make sure to enable VLANs for Ethernet traffic.

vFabric
Enabling vFabric, assigning vFabric VLAN and FCoE map information.
C140A1# vfabric 1
C140A1# vlan 1001
C140A1# fcoe fcmap 0xEFC00
Make sure to add vFabric under interface configuration. Configure Port-channels, VLANs, DCB parameters, QoS settings and
interface settings as mentioned in the Dell EMC SmartFabric OS10 User Guide.
Configure MX IOM in FIP Snooping Bridge (FSB)

mode
This section only focuses on key components to be considered while configuring MX I/O Module in Full Switch mode. In a
scenario mentioned in Scenario 5: Connect MX5108n to Fibre Channel storage - FSB, I/O Module with FSB enabled is
connected to servers and switches on a converged ethernet link carries both Local Area Network (LAN) and Storage Area
Network (SAN) traffic.
Configuring MX I/O Modules in Full Switch mode with FSB enabled requires commands mentioned below. These are the key
components to enable FSB on an IOM.
Enable DCBX
Make sure enable DCBX in global configuration mode:
C140A1# dcbx enable
Enable FIP-Snooping
Enable the FIP snooping feature in global configuration mode:
C140A1# feature fip-snooping
FCoE VLAN
The SAN (FCoE) traffic will be directed to upstream switches in NPG mode or F-Port mode.
Make sure to enable fip-snooping for SAN (FCoE) traffic under fabric VLAN interface:

C140A1# description FCoE_VLAN
C140A1# fip-snooping enable
142 Dell EMC PowerEdge MX I/O Module Full Switch mode Configuration – Key Components
FIP Snooping under Port-Channel
Make sure to enable FIP Snooping under port channel interface, members of this port channel are FCoE Forwarder (FCF) facing
ports:
C140A1# interface port-channel 1

C140A1# fip-snooping port-mode fcf
Configure Port-channels, VLANs, DCB parameters, QoS settings and interface settings as mentioned in the Dell EMC
SmartFabric OS10 User Guide.
Dell EMC PowerEdge MX I/O Module Full Switch mode Configuration – Key Components 143
C
Dell EMC PowerSwitch S4148U-ON
Configuration
This section covers the switch configuration for S4148U-ON switches running OS10. In the Scenario 5: Connect MX5108n to
Fibre Channel storage - FSB topology, S4148U-ON switches are connected to the MX9116n FSE in the MX7000 chassis and to
the FC switches. Run the commands in the following sections to complete the configuration of both leaf switches.
Topics:
• Switch configuration commands
Switch configuration commands

Run the following commands to configure the hostname, OOB management IP address, and default gateway.
General settings
NOTE: The MX I/O Modules run Rapid Per-VLAN Spanning Tree Plus (RPVST+) by default. RPVST+ runs RSTP on each
VLAN while RSTP runs a single instance of spanning tree across the default VLAN. Dell EMC PowerSwitch S4148U-ON used
in this example runs SmartFabric OS10 and has RPVST+ enabled by default. See Spanning Tree Protocol recommendation in
Dell EMC SmartFabric OS10 User Guide for more information.
S4148U-ON Leaf 1 S4148U-ON Leaf 2
hostname S4148U-Leaf1 hostname S4148U-Leaf2

NOTE: Use the spanning-tree {vlan vlan-id priority priority-value} command to set the bridge
priority for the upstream switches. The bridge priority ranges from 0 to 61440 in increments of 4096. The switch which has
lowest bridge priority becomes STP root.
Configure VLANs
Run the commands in this section to configure VLANs. In this deployment example, the VLANs used are VLAN 30 and VLAN 40.
Set the MTU as 9216 Bytes

mtu 9216 mtu 9216
144 Dell EMC PowerSwitch S4148U-ON Configuration

Configure DCBx, NPG, and vFabric
Configure and enable DCBx feature, NPG as a FC feature and vFabric.
NOTE: Remove all the FC configuration, vFabric global configuration and vFabric configuration under interface or port-
channels prior to config FC feature.
dcbx enable dcbx enable

feature fc npg feature fc npg
vfabric 101 vfabric 102
vlan 40 vlan 30
fcoe fcmap 0xEFC00 fcoe fcmap 0xEFC00
Configure QoS
Configure class-map, policy-map and define QoS parameters. In this example, queue 3 is defined as Output queue in policy map.
The bandwidth is also defined as 50%. Configure the QoS parameters as mentioned below.
class-map type network-qos class-map type network-qos

class_Dot1p_3 class_Dot1p_3
match qos-group 3 match qos-group 3
class-map type queuing class-map type queuing
map_ETSQueue_0 map_ETSQueue_0
match queue 0 match queue 0
class-map type queuing class-map type queuing

map_ETSQueue_3 map_ETSQueue_3
match queue 3 match queue 3
trust dot1p-map trust dot1p-map

map_Dot1pToGroups map_Dot1pToGroups
qos-group 0 dot1p 0-2,4-7 qos-group 0 dot1p 0-2,4-7
qos-group 3 dot1p 3 qos-group 3 dot1p 3
qos-map traffic-class qos-map traffic-class
map_GroupsToQueues map_GroupsToQueues
queue 0 qos-group 0 queue 0 qos-group 0

queue 3 qos-group 3 queue 3 qos-group 3
policy-map type network-qos policy-map type network-qos

policy_Input_PFC policy_Input_PFC
class class_Dot1p_3 class class_Dot1p_3

pause pause
pfc-cos 3 pfc-cos 3
policy-map type queuing policy-map type queuing

policy_Output_BandwidthPercent policy_Output_BandwidthPercent
class map_ETSQueue_0 class map_ETSQueue_0

bandwidth percent 50 bandwidth percent 50
class map_ETSQueue_3 class map_ETSQueue_3
bandwidth percent 50 bandwidth percent 50
system qos system qos
trust-map dot1p trust-map dot1p

map_Dot1pToGroups map_Dot1pToGroups
qos-map traffic-class qos-map traffic-class
map_GroupsToQueues map_GroupsToQueues
Dell EMC PowerSwitch S4148U-ON Configuration 145

Configure interfaces
In this topology, Interface 1/1/1 and 1/1/3 on both leafs are connected to the FC switches. Interfaces 1/1/11 and 1/1/12 are
connected to MX9116n FSEs. Configure the interfaces as mentioned below. Make sure to configure port-groups before
configuring interfaces.
interface fibrechannel 1/1/1 interface fibrechannel 1/1/1

interface fibrechannel 1/1/3 interface fibrechannel 1/1/3


priority-flow-control mode on priority-flow-control mode on
service-policy input type service-policy input type
network-qos policy_Input_PFC network-qos policy_Input_PFC
service-policy output type service-policy output type
queuing queuing
ets mode on ets mode on

priority-flow-control mode on priority-flow-control mode on
service-policy input type service-policy input type
network-qos policy_Input_PFC network-qos policy_Input_PFC
service-policy output type service-policy output type
queuing queuing
ets mode on ets mode on
end end
146 Dell EMC PowerSwitch S4148U-ON Configuration

D
Dell EMC Unity Information
Topics:
• About Dell EMC Unity
• Determine Unity 500F storage array FC WWPNs
• Determine CNA FCoE port WWPNs
• Configure Unity FC storage
About Dell EMC Unity

This section shows how an administrator can determine the World Wide Port Names (WWPNs) of Converged Network Adapters
(CNAs) and storage targets. The WWPNs are used to connect FC storage targets to specific servers for file storage or OS
boot.
Determine Unity 500F storage array FC WWPNs

The WWPNs of FC adapters in storage arrays are also used for FC configuration. Perform the following steps to determine
WWPNs on Unity storage arrays.
1. Connect to the Unisphere GUI in a web browser and log in.
2. Click the Settings (gear icon) near the top-right corner of the page.
3. In the left pane of the Settings window, select Access, then Fibre Channel.
The Fibre Channel Ports page is displayed as shown. A zoomed-in view of the area inside the red box is shown in the
following figure.
Figure 132. Unisphere Fibre Channel Ports page
Figure 133. Zoomed-in view of SP A and SP B WWNs
Two WWNs are listed for each port. The World Wide Node Name (WWNN), outlined in black, identifies this Unity storage
array (the node). The WWPNs, outlined in blue, identify the individual ports.
4. Record the WWPNs as shown in the following table:
Dell EMC Unity Information 147

Table 16. Storage array FC adapter WWPNs
Service processor Port WWPN
SP A 0 50:06:01:66:47:E0:1B:19
SP A 1 50:06:01:67:47:E0:1B:19
SP B 0 50:06:01:6E:47:E0:1B:19
SP B 1 50:06:01:6F:47:E0:1B:19
Determine CNA FCoE port WWPNs

In this example, the MX740c server's FCoE adapter WWPNs are used for FC connection configuration. Perform the following
steps to determine Adapter WWPNs.
1. Connect to the first server's iDRAC in a web browser and log in.
2. Select System, then click Network Devices.
3. Click the CNA. In this example, it is NIC Mezzanine 1A. Under Ports and Partitioned Ports, the FCoE partition for each
port is displayed as shown in the following figure.
Figure 134. FCoE partitions in iDRAC
4. The first FCoE partition is Port 1, Partition 2. Click the (+) icon to view the MAC Addresses as shown in the following
figure.
148 Dell EMC Unity Information

Figure 135. MAC address and FCoE WWPN for CNA port 1
5. Record the MAC Address and WWPN, outlined in red in the previous figure.
NOTE: A convenient method is to copy and paste these values into a text file.
6. Repeat steps 4 and 5 for the FCoE partition on port 2.

7. Repeat the steps in this section for the remaining MX740c servers.
The FCoE WWPNs and MAC addresses used in this deployment example are shown in the following table:
Table 17. Server CNA FCoE port WWPNs and MACs

Server Port WWPN MAC
MX740c-1 1 20:01:18:66:DA:71:50:AD 18:66:DA:71:50:AC
MX740c-1 2 20:01:18:66:DA:71:50:AF 18:66:DA:71:50:AE
MX740c-2 1 20:01:18:66:DA:77:D0:C3 18:66:DA:77:D0:C2
MX740c-2 2 20:01:18:66:DA:77:D0:C5 18:66:DA:77:D0:C4
Configure Unity FC storage

This section covers configuration of a Dell EMC Unity 500F storage array. Refer to the storage system documentation for other
FC storage devices.
Create a storage pool

Perform the following steps to create a storage pool.
1. Connect to the Unisphere GUI in a web browser and log in using the required credentials.
2. In the left pane under STORAGE, select Pools.
3. Click the (+) icon.
4. In the Create Pool dialog box, provide a name in the required field, then click Next.
5. Select appropriate storage tiers and RAID configuration for the pool, then click Next.
6. Under Select Amount of Storage, select the desired number of drives. The total number of drives and the total capacity is
displayed next to Totals. Click Next.
7. The Capability Profile Name section is optional. Click Next.
8. Review selections on the Summary page and click Finish to create the pool. Once the Overall status shows 100%, click
Close.
The pool is displayed on the STORAGE > Pools page as shown in the following figure.

Figure 136. Storage pool created
Add hosts
Perform the following steps to add hosts to the Unity FC storage array configuration.
1. In the Unisphere left pane under ACCESS, select Hosts.
2. On the Hosts tab, click the (+) icon, then click Host.
3. Enter the Name of the server in the field provided, then click Next.
4. Select the Initiator IQN/WWN checkbox in the Discovered Initiators panel.
5. Click Next.
6. Click Finish in the Review the host configuration page.
NOTE: Additional hosts may be added as needed by clicking the (+) icon from the Hosts tab.
Create LUNs and configure host access

Perform the following steps to create LUNs and configure host access.
1. In the Unisphere left pane under STORAGE, select Block.
2. On the LUNs tab, click the (+) icon to open the Create LUNs dialog box.
3. On the Configure LUN(s) page, select the Number of LUNs.
4. Provide a name in the field provided, then select the Storage Pool.
5. From the Size section, make the required modifications, then click Next.
6. On the Access page, click the (+) icon and the select host(s) to be granted access to the LUN.
150 Dell EMC Unity Information

7. Click OK, then click Next.
8. On the Snapshot page, leave settings at their defaults and click Next.
9. On the Replication page, leave settings at their defaults and click Next.
10. On the Summary page, review the details and click Finish to create the LUN.
11. On the Results page, click Close when overall status shows 100% completed.
The newly created LUN is now visible on the LUNs tab, as shown in the following figure. In this example, a LUN named
FC-80GB that is 80 GB in size has been created.
Figure 137. LUN created

12. Create additional LUNs and grant access (map) to hosts as needed.
NOTE: To modify host access at any time, check the box next to the LUN to select it. Click the Edit (pencil) icon, and
select the Host Access tab.

E
Additional Information
Topics:
• Delete MCM group
• Reset chassis using RACADM
• Set admin password guidelines
• Reset SmartFabric OS10 switch to factory defaults
• Reset Cisco Nexus 3232C to factory defaults
• Connect to IO Module console port using RACADM
Delete MCM group

To remove an MCM group using the OME-M console, perform the following steps.
2. In the MCM group pane, click the name of the lead chassis.
3. From the Configure menu, select Delete Group.
4. In the Delete Group dialog box, click Confirm.
At this point, the OME-M console removes the MCM group. To manage the chassis, use the individual IP addresses assigned to
each.
Reset chassis using RACADM

To reset the chassis to factory default settings, perform the following steps.
1. Connect to the MX9002m IP address using SSH. The default username is root, and the default password is calvin.
2. In the RACADM shell, run the racadm racresetcfg command.
3. The factory reset process is initiated, and a status message displays.
NOTE: The process takes several minutes to complete.
Optionally, after the reset process is complete, use the LCD screen to reassign a static IP address.
Set admin password guidelines
Configure password on OME-M

To configure the I/O Module ‘admin’ account password, access the OME-M GUI. Choose Devices > I/O Module > Select an
IOM and choose Settings.
152 Additional Information

Figure 138. Set password on OME-M
NOTE: Passwords require a minimum of nine characters.
Configuration of password failure

Error alerts will be shown if the password requirements are not met.
Figure 139. Error for password requirements failure
Access IOM CLI

Access the IOM CLI using the new password to ensure that the new password has been set.
Reset SmartFabric OS10 switch to factory defaults

To reset SmartFabric OS10 switches back to the factory default configuration, enter the following commands:
OS10# delete startup-configuration
Proceed to delete startup-configuration [yes/no(default)]:yes

OS10# reload
System configuration has been modified. Save? [yes/no]:no
Proceed to reboot the system? [confirm yes/no]:yes
The switch reboots with default configuration settings.
Additional Information 153

Reset Cisco Nexus 3232C to factory defaults
To reset the Cisco Nexus 3232C switches to the factory default configuration, enter the following commands:
3232C# write erase

Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y
After the next reboot, the switch loads with default configuration settings.
Connect to IO Module console port using RACADM

To connect to an IOM console port, first connect to the OME-Modular IP address using SSH using the same credentials used to
log in to the OME-M GUI.
Use the RACADM command from the MX9002m management module:
racadm connect [-b] -m <module>
-b is for Binary mode.

-m is the Module option. The module option can be one of the following:
● server-<n>: where n = 1 to 8
● switch-<n>: where n = 1 to 6 or <a1 | a2 | b1 | b2 | c1 | c2>
For example:
● Connect to I/O Module 1 serial console:
racadm connect -m switch-1
● Connect to Server 1 serial console:
racadm connect -m server-1
154 Additional Information

F
Hardware Components and Supported
Versions
Topics:
• Scenarios 1, 2, 7, and 8
• Scenarios 3 through 6
Scenarios 1, 2, 7, and 8
The following tables include the hardware components and supported software and firmware versions for Scenario 1, Scenario 2,
Scenario 7, and Scenario 8.
Dell EMC PowerSwitches

Table 18. Dell EMC PowerSwitch switches and OS versions – Scenarios 1, 2, 7, and 8
Qty Item Version
2 Dell EMC PowerSwitch Z9100-ON leaf switches 10.5.0.5
1 Dell EMC PowerSwitch S3048-ON OOB management switch 10.4.0E(R3P2)
Dell EMC PowerEdge MX7000 chassis and components

Table 19. Dell EMC PowerEdge MX7000 chassis and components – Scenarios 1, 2, 7, and 8
Qty Item Version
2 Dell EMC PowerEdge MX7000 chassis -
4 Dell EMC PowerEdge MX740c sled See the following table
4 Dell EMC PowerEdge M9002m modules 1.20.10
2 Dell EMC Networking MX9116n FSE 10.5.1.6
2 Dell EMC Networking MX7116n FEM -
Table 20. MX740c sled details – Scenarios 1, 2, 7, and 8

Qty per sled Item Firmware Version
1 Intel(R) Xeon(R) Silver 4114 CPU @ 2.20 GHz -
12 16 GB DDR4 DIMMs (192 GB total) -
1 Boot Optimized Storage Solution (BOSS) Controller with 2x 240 GB SATA 2.6.13.3011
SSDs
1 PERC H730P MX 25.5.3.0005
3 600 GB SAS HDD -
1 Intel(R) Ethernet 25 G 2P XXV710 mezzanine card 18.8.9
Hardware Components and Supported Versions 155

Table 20. MX740c sled details – Scenarios 1, 2, 7, and 8 (continued)
- BIOS 2.3.10
- iDRAC with Lifecycle Controller 3.20.21.20
- iDRAC Service Module Installer 3.3.1
- System CPLD 1.0.0
Cisco Nexus switches

Table 21. Nexus switches and OS versions – Scenarios 1, 2, 7, and 8
Qty Item Version
2 Cisco Nexus 3232C 7.0(3)I4(1)
Scenarios 3 through 6
The tables in this section include the hardware components and supported software and firmware versions for Scenario 3
through Scenario 6 in this document.
Table 22. Switches and OS versions

Qty Item Software Version
1 Dell EMC PowerSwitch S3048-ON management switch 10.4.0E(R3P2)
2 Dell EMC Networking MX9116n FSE 10.5.1.6
2 Dell EMC Networking MX5108 10.5.1.6
2 Dell EMC PowerSwitch S4148U-ON 10.4.3.3
2 Dell EMC Networking MX7116n FEM -
Table 23. MX-series components

Qty Item Version
4 Dell EMC PowerEdge M9002m modules 1.20.10
4 Dell EMC PowerEdge MX740c compute sleds See the following table
Table 24. MX740c compute sled details

1 QLogic QL41262HMKR (25 G) mezzanine CNA 15.10.0.4
2 Intel(R) Xeon(R) Silver 4114 CPU @ 2.20 GHz -
12 16 GB DDR4 DIMMs (192 GB total) -
1 Boot Optimized Storage Solution (BOSS) Controller with 2x 240 GB SATA 2.6.13.3011
SSDs
1 PERC H730P MX 25.5.5.0005
3 600 GB SAS HDD -
- Dell OS Driver Pack 18.10.17
- BIOS 2.3.10
- iDRAC with Lifecycle Controller 3.20.21.20
156 Hardware Components and Supported Versions

Table 24. MX740c compute sled details (continued)
- iDRAC Service Module Installer 3.3.1
- System CPLD 1.0.0
Hardware Components and Supported Versions 157

G
References
Topics:
• Dell EMC documentation
• Dell Technologies Networking Infrastructure Solutions documentation
Dell EMC documentation

The following Dell EMC documentation provides additional and relevant information. Access to these documents may depend on
your log in credentials. If you do not have access to a document, contact your Dell EMC representative.
● Dell EMC Networking Guides
● Dell EMC PowerEdge MX IO Guide
● Dell EMC PowerEdge MX Network Architecture Guide
● Dell EMC Interactive Demo: PowerEdge MX Series
● Dell EMC PowerEdge MX SmartFabric and Cisco ACI Integration Guide
● Dell EMC PowerEdge MX SmartFabric Deployment Video
● Dell EMC PowerEdge MX SmartFabric Deployment with Cisco ACI Video
● MX Port-Group Configuration Errors Video
● MX Port-Group Configuration Video
● Dell EMC OpenManage Enterprise-Modular Edition User's Guide v1.00.01
● OS10 Enterprise Edition User Guide for PowerEdge MX IO Modules Release 10.4.0E R3S
● Dell EMC SmartFabric OS10 User Guide
● Manuals and documents for Dell EMC PowerEdge MX7000
● Manuals and documents for Dell EMC PowerSwitch MX9116n
● Manuals and documents for Dell EMC PowerSwitch S3048-ON
● Manuals and documents for Dell EMC PowerSwitch Z9100-ON
● Dell EMC OME-M v1.00.01 for PowerEdge MX7000 Chassis User's Guide
● Dell EMC Networking Layer 3 Leaf-Spine Deployment and Best Practices with OS10
● Fibre Channel Deployment with S4148U-ON in F_Port Mode
● FCoE-to-Fibre Channel Deployment with S4148U-ON in F_Port Mode
● Manuals and documentation for Dell EMC Networking S4148U-ON
Dell Technologies Networking Infrastructure

Solutions documentation
The following documentation provides additional networking solutions information.
NOTE: Access to the documentation may require user credentials. If you do not have access to a document, contact your
Dell Technologies representative.
Networking solutions: https://infohub.delltechnologies.com/t/networking-solutions-57/
158 References

h18064.1 PowerEdge MX SmartFabric Config PDF

Uploaded by

Copyright:

Available Formats

h18064.1 PowerEdge MX SmartFabric Config PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

h18064.1 PowerEdge MX SmartFabric Config PDF

Uploaded by

Copyright:

Available Formats

Dell EMC PowerEdge MX SmartFabric

Configuration and Troubleshooting Guide

Dell Technologies Networking Infrastructure Solutions

Chapter 1: Introduction to Dell EMC PowerEdge MX......................................................................8

Chapter 2: Overview of SmartFabric Services for PowerEdge MX.................................................11

Chapter 3: SmartFabric mode requirements, guidelines, and restrictions.....................................21

Chapter 4: SmartFabric Creation.................................................................................................38

Chapter 5: Server Deployment.....................................................................................................47

Chapter 6: SmartFabric Operations............................................................................................. 52

Chapter 7: Switch Operations..................................................................................................... 65

Chapter 8: SmartFabric Deployment Validation........................................................................... 75

Chapter 9: SmartFabric Troubleshooting.....................................................................................90

Chapter 10: Uplink Configuration Scenarios............................................................................... 109

Appendix A: Hardware................................................................................................................ 138

Appendix C: Dell EMC PowerSwitch S4148U-ON Configuration.................................................. 144

Appendix D: Dell EMC Unity Information.................................................................................... 147

Appendix E: Additional Information............................................................................................ 152

Appendix G: References............................................................................................................. 158

Figure 1. Dell EMC PowerEdge MX7000 chassis

8 Introduction to Dell EMC PowerEdge MX

Monospace text CLI examples

Introduction to Dell EMC PowerEdge MX 9

Support and feedback

10 Introduction to Dell EMC PowerEdge MX

Dell EMC SmartFabric OS10

Figure 2. Dell EMC SmartFabric OS10 high-level architecture

Overview of SmartFabric Services for PowerEdge MX 11

Full Switch mode

12 Overview of SmartFabric Services for PowerEdge MX

Table 1. IOM operating mode differences

Overview of SmartFabric Services for PowerEdge MX 13

Changing operating modes

MX9116n Fabric Switching Engine: virtual ports

OS10# show discovered-expanders

14 Overview of SmartFabric Services for PowerEdge MX

Table 2. Virtual port mapping

OS10# show interface status

OS10# show interface ethernet 1/1/30:3

Virtual Link Trunking

Overview of SmartFabric Services for PowerEdge MX 15

Fibre channel connectivity

NPIV Proxy Gateway

Figure 3. Fibre channel NPG network to Dell EMC Unity

Direct attached (F_Port)

16 Overview of SmartFabric Services for PowerEdge MX

Overview of SmartFabric Services for PowerEdge MX 17

Networks and automated QoS

Figure 6. Network types available in SmartFabric mode

18 Overview of SmartFabric Services for PowerEdge MX

Table 3. Network types and default QoS settings

Server templates, virtual identities, networks, and

Overview of SmartFabric Services for PowerEdge MX 19

Dell EMC Demo Center

20 Overview of SmartFabric Services for PowerEdge MX

Create multichassis management group

Upstream network requirements

SmartFabric mode requirements, guidelines, and restrictions 21

Figure 7. Required upstream network connectivity

Table 4. Number of uplinks supported