BCSDNutshell VC

Brocade
Product Training
BCSD-in-a-Nutshell Virtual Classroom Version
Part 1
Brocade Education Services
2006 Brocade Communications Systems, Incorporated.

Revision March 2006

Revision March 2006
Page 1-1
Objectives
Review the BCSD-in-a-Nutshell Training
Review the key business applications of SANs
Identify the specific business problems being addressed by the SAN
solution
Capture information that defines the current SAN environment
Capture information that defines the requirements for the target SAN
environment
Review performance design methodology for SANs and MetaSANS

Revision March 2006
We will identify the key pieces of information that Brocade believes should be
collected before designing a SAN. This information covers both business issues and
requirements, as well as more technically-oriented information about servers,
storage, device availability, and performance.
Note - Brocade recognizes that many of its partners use their own data gathering
tools, including questionnaires and spreadsheets. The methods demonstrated in this
course are not meant to supercede those of Brocade partners, but instead are
provided as examples.

Revision March 2006
Page 1-2
BCSD Overview
This training prepares you to pass the certification exam for the
Brocade Certified SAN Designer (BCSD)

This level of certification indicates that you have mastered the
concepts and intricacies of building a SAN from the basic
components through the integration of industry applications and
state-of-the-art storage components
BCSD is focused on SAN Designers, Pre-Sales Support, and SAN
Managers

Revision March 2006

Revision March 2006
Page 1-3
Training Objectives
The BCSD-in-a-Nutshell training focuses on the four main tasks for a
Brocade Certified SAN Designer:

1. Assess SAN Requirements
Given a specific customer business problem, relate how a SAN installation
can meet and solve customer needs

Given a specific customer business problem, identify what information
needs to be collected about their current and target environments to

characterize the proposed architecture
Map existing SAN technologies and protocols to an appropriate solution
Define the key availability characteristics which need to be identified to
determine an appropriate SAN architecture

Revision March 2006

Revision March 2006
Page 1-4
Training Objectives (cont.)

2. Create a SAN design that meets specific requirements, while
considering design trade-offs

Choose or build an SAN architecture to meet manageability, availability,
scalability, and performance requirements

Given a topology, classify the availability and distribution model of each Fabric
as single or redundant, resilient or non-resilient, and the fault tolerance level

Define the appropriate device attachment scheme to meet availability and
performance requirements
Given a particular configuration with performance information, evaluate locality,
ISL/trunks, quads/octets and link speed to determine performance and

capacity requirements/limitations

Revision March 2006

Revision March 2006
Page 1-5

2. Create a SAN design that meets specific requirements, while
considering design trade-offs (cont.)

Given an initial design and the customer's growth plan, select which
topology and switch types are appropriate

Given a proposed SAN architecture, identify whether hop count
restrictions have been violated

Given a varying set of SAN designs/components and customer
requirements, identify which SAN components will need to be integrated.

Given a SAN architecture, identify the most appropriate Brocade switches
Select the zoning implementation that best meets the SAN design
requirements, based on Brocade guidelines

Revision March 2006

Revision March 2006
Page 1-6

3. Translate a SAN design to a functional deployment plan
Identify what physical connections or placements would be required to
integrate the SAN into an existing data center
Integrate the SAN into the existing mgmt. network infrastructure
Select the appropriate equipment, actions, or configurations to complete
the MAN/WAN implementation
Create the appropriate switch and fabric configuration planning
documentation to implement a given SAN architecture
Create the appropriate physical or logical SAN layout documentation
required to implement a given SAN architecture

Revision March 2006

Revision March 2006
Page 1-7

3. Translate a SAN design to a functional deployment plan (cont.)
Create an appropriate logical connection diagram, including logical site
groupings and topology
Create a SAN migration plan, given current and targeted SAN
Document the issues and consequences of a particular SAN/Fabric
management application given a set of customer requirements
Describe specific growth plans for a proposed SAN architecture
4. Optimize and Tune SAN Design

Select methodologies to optimize and tune a deployed SAN architecture

Revision March 2006

Revision March 2006
Page 1-8
Training Agenda
The BCSD-in-a-Nutshell virtual class training is delivered in (2) two-
hour sessions which focuses on the four main tasks of a Brocade

Certified SAN Designer:
Assessing SAN Requirements
Creating a SAN Design
Documenting and Deploying SANs
Optimizing and Tuning SANs

Revision March 2006

Revision March 2006
Page 1-9
SAN Applications - Overview

Fibre Channel-based Storage Area Networks are suited to a wide variety
of business applications and initiatives

Improved connectivity better sharing, utilization, and scalability
Improved availability better up-times, supports DR
Dedicated data movement infrastructure centralizes management
Improved bandwidth better performance and scalability
Server/Storage Centralized
Consolidation Management
Increase
Resource
Utilization
Improved Data
Availability
Business
Continuity
Performance
& Scalability
Simplify
management:
7-10 times more
storage managed
per administrator
Reduce
backup and
restore times
by 50-90%
Extend
SANs over
distance for
Disaster
Recovery
Accelerate
Mission critical
applications

Revision March 2006
10
Storage Area Networks drive benefits to a number of applications and initiatives,

including:
Server/storage consolidation, often doubling utilization levels, especially for
storage;
Centralized management, allowing system administrators to manage significantly
more data;
Improved data availability, through faster backups and restores;
Enhanced business continuity, through long-distance disaster recovery;
Enhanced performance and scalability, through improved performance and
scalability (as compared with a purely-SCSI solution).

Revision March 2006
Page 1-10
SAN Applications - Overview (cont.)

Some Key SAN-enabled Business Applications
Server consolidation
Migrate to fewer, larger, faster servers
Storage applications
Consolidation, utilization, and data sharing, and centralized

management initiatives
Backups
Move data to disk or tape storage systems without using the

existing LAN/WAN
High-availability applications
Clustering, business continuance
Disaster tolerance
Keep data not only at a local site, but also at a remote site
Performance
Improve data rates and response times
Security
Ensure that specified servers can access specified storage
Scalability
Add servers and storage without taking costly downtime for

network reconfigurations

Revision March 2006
11
Storage Area Networks drive benefits to a number of applications and initiatives,

including:
Server/storage consolidation, often doubling utilization levels, especially for
storage;
Centralized management, allowing system administrators to manage significantly
more data;
Improved data availability, through faster backups and restores;
Enhanced business continuity, through long-distance disaster recovery;
Enhanced performance and scalability, through improved performance and
scalability (as compared with a purely-SCSI solution).

Revision March 2006
Page 1-11
SAN Applications
What SANs Add
Storage Applications and Server Consolidation:
Improved connectivity (fan-in)
Improved bandwidth (vs. SCSI)
Improved availability
Better utilization of storage systems
Backups:
Separate infrastructure that off-loads the existing WAN/LAN
infrastructure
SCSI-based infrastructure better suited to large-block I/Os
Avoids IP stack for better performance
Long-distance connectivity to facilitate remote site operations

Revision March 2006
12
Storage Applications: As storage systems become larger and more robust, users
are migrating data from numerous small storage systems (either SCSI-based or FCbased) to smaller numbers of large storage systems. The various servers that
accessed the distributed storage must now share fewer storage ports when accessing
the consolidated storage. The availability of each storage port is thus more
important, and connectivity to the storage port must be more flexible so that the
storage can be highly utilized. The connection to the storage port must support
higher levels of the bandwidth per storage port. In a storage consolidation solution,
we choose Storage Area Networks that connect servers to highly available and
reliable storage. The switches to which the storage is connected must support
improved resource utilization by providing additional connectivity and high
performance.
Server Consolidation: The similarities between storage applications and server
consolidation are clear the only difference is whether we focus on the server or the
storage.
Backups: In a SAN-based backup, the SAN is used to move the backup data, hence
the definition of LAN-free. In reality, though, a LAN is used to send a few bytes
of information about each file being backed up.

Revision March 2006
Page 1-12
SAN Applications
What SANs Add (cont.)
High-availability applications
Highly-available data network
Easier sharing of storage
Better long-distance connectivity for clusters
Disaster tolerance
Fibre Channel high-availability features
Longer-distance storage access through the SCSI driver (better
performance)
Performance
Improved Fibre Channel bandwidth (vs. SCSI)
SilkWorm switch features (trunking)

Revision March 2006
13
Disaster Tolerance: The continuing availability and integrity of corporate servers

and storage have become a crucial parts of many business operations. A Fibre
Channel-based SAN includes numerous high-availability solutions, and can easily
extend to 120 kilometers (75 miles), far enough to support a variety of disaster
tolerance solutions. The high data rates of the Fibre Channel connection can be
driven by applications that transfer data using the SCSI protocol, reducing system
overhead and ensuring maximum use of the available bandwidth.

Revision March 2006
Page 1-13
SAN Applications
What SANs Add (cont.)
Security needs
Fabric zoning ensures that servers access only the storage to
which they are granted access

FC-FC Routing extends device access security across fabrics
Secure Fabric OS provides physical and management security
Scalability
High-port-count Fibre Channel directors and switches support
expandable fabric topologies

Revision March 2006
14
Scalability: Examples of high-port-count Fibre Channel directors and switches are

the Brocade SilkWorm 3900, 12000, and 24000.

Revision March 2006
Page 1-14
Data Collection - Overview

When designing a SAN, you must have information from several key
subject categories:
Business
Requirements
What are the business requirements of the SAN?

What are the technical challenges to be
addressed by the SAN solution?
Current SAN
Requirements
What are the business requirements of the SAN?

What are the technical challenges to be
addressed by the SAN solution?
Target SAN
Requirements
SAN infrastructure? Servers and storage?

Availability? Performance? Future growth? New
SAN-enabled applications?

Revision March 2006
15
In the slides that follow, we detail these main points.

Revision March 2006
Page 1-15
Business Requirements
What are the business challenges that need to be addressed by a
(potential) SAN solution?

We have too many slow servers. We are moving to fewer, faster
servers.
We have too many small, underused storage systems!
Our backups are still running in the morning when our employees
begin entering new orders.
We have to become a 24/7 shop we cannot afford doing weekly
rebuilds.
We need a more rigorous disaster recovery solution.
Our data mining activities take too long.

Revision March 2006
16
As much as we have been discussing SAN devices and terminology, it is ultimately

business problems that drive the introduction of technologies like a Storage Area
Network. The above slide lists typical business problems (e.g., device consolidation,
performance, and high availability) that may require a SAN solution.

Revision March 2006
Page 1-16
Business Requirements (cont.)

Phrase the problem as a quantitative technical requirement, against which
success/failure is measured:
Server and Storage Consolidation
High Availability
Backup Needs
Disaster Tolerance and Business Continuance
Performance
SAN-Enabled Applications
Manageability
Scalability
Security Needs
Reduced Total Cost of Ownership
Storage Utilization

Revision March 2006
17
Business requirements are a key part of the SAN design process. For both the end user and
the SAN designer, they establish standards by which success and failure can be ascertained.
Server/Storage Consolidation The following servers shall access the following storage
systems:.
High Availability No more than X minutes of downtime per year, including Y minutes
for scheduled maintenance.
Backup Needs The following servers must complete their backups according to this
schedule:.
Disaster Tolerance and Business Continuance In the event of failures at site A, site B
shall assume control of all applications and data within C minutes.
Performance On server E, application F shall load database G within H minutes.
SAN-Enabled Applications I will need to use Multiple Pathing Software, Volume
Management, Shared Storage, Clustering.
Manageability X Management Station will run Y applications and access the
infrastructure using Z methods.
Scalability My network will grow A% over the next B time frame, so my interface will
need to provide C number of ports.
Security Needs Our Depth of Defense is X and we require Y security needs.
Reduced Total Cost of Ownership Overall SAN costs, including hardware, software,
and labor, will be reduced by X% over the next 12 months.
Storage Utilization - Disk storage should be at least Y% utilized, up from the current level
of X%.

Revision March 2006
Page 1-17
Current SAN Requirements

Overview
Any SAN design must take into consideration the existing SAN infrastructure
Before creating a SAN design, collect information on the following requirements:
Facilities
SAN Infrastructure
Servers
Storage (Disk and Tape)
Availability (Data and Application)
Performance
Security
LAN Infrastructure
Multiprotocol Infrastructure

Revision March 2006
18
The following slides details each of these requirements:

Revision March 2006
Page 1-18

Facilities
Gather physical facilities information:
Customer Info - Customer name and location; technical contact
(SE, support); region

Environmental - Power, cooling, rack space
Geographic requirements - For each location:
Data center(s) and distance between them
Cage(s), Communication closest, nearby Fibre Channel
rings/pulls/patch panels (as needed)
Long Distance communication components, protocols and topologies

Revision March 2006
19
Documenting the geographic information at an early stage gives you time to prepare
for additional changes that may be required (e.g., acquiring additional floor space
for new equipment, adding fiber optic cable pulls, ensuring sufficient power and
cooling, etc.).

Revision March 2006
Page 1-19

SAN Infrastructure
Gather data about the existing SAN
infrastructure; for each fabric:

Number of user ports
Number of total ports
Switch types and counts
Fabric OS revision(s)
Topology (Core/Edge, Mesh,
etc.)
Maximum hop count
SilkWorm feature licenses
Across the overall SAN, other
information is needed:
Number of fabrics in SAN
Number of sites in environment
Expected growth rate

Revision March 2006
20
You can obtain most of this information by running the supportShow command
and uploading the switch configuration file. Brocade also provides services to help
with this type of data collection more on this later in this module.

Revision March 2006
Page 1-20
Current SAN Requirements SAN-Enabled Applications

In the current SAN, what SAN-enabled applications that are to be
implemented?
Server consolidation
Storage consolidation and storage utilization
High availability
Backup Needs
Disaster tolerance
Performance
Security Needs
Scalability

Revision March 2006
21
Here are some of the server, storage, and switch hardware and software decisions
that may be affected by the SAN-enabled applications implemented on the current
SAN:
Server consolidation Fewer Faster Servers, Clustering, Volume Management,
Multi-Pathing, Persistent Binding
Storage consolidation and storage utilization Selective Presentation, LUN
Masking, LUN Security, Virtualization
High availability Clustering, Multi-Pathing
Backup Needs Tape Consolidation, LAN-Free, Serverless, Virtualization
Disaster tolerance Disaster recovery/DR, Remote take-over, Extended Fabrics,
Remote Switch or ISL_RDY Mode, FC-to-FC Routing, FCIP
Performance Trunking, Multi-Pathing
Security Needs Secure Fabric OS, Zoning, LUN Security
Scalability Fabric Topology selection, switch/Director selection

Revision March 2006
Page 1-21

Servers
For each server, gather the following information:
Server system
Operating system
Ethernet
Host bus adapters (HBAs)
Software applications
Disk storage
Tape storage
SAN issues
Physical location

Revision March 2006
22
Consider the above list to be the absolute minimum information to be gathered about
each server. Some additional notes on the values above:
Server system The server name, manufacturer, and the manufacturers model
name.
Operating system The name of the operating system, and the specific release of
the OS (and any service packs, patches, etc.).
Ethernet requirements The number of Ethernet interfaces needed (higher for
clustered servers), and the required speeds.
HBAs The quantity, manufacturer, model, and driver revision information.
Software Applications The main application run by this server; for multipleserver applications (cluster software, multi-server databases, etc.), note the partner
server.
Disk storage The storage system used; the amount of storage used; the number of
storage-system connection points used by the server to access the storage; and the
number of separate logical units (or LUNs) through which the server data is
accessed.
Tape storage The tape library that contains the drive(s), and the speed of each
drives.
SAN Issues Whether some type of dynamic multi-pathing (DMP) software is
installed on this server, and additional SAN data security (LUN masking on the
server or disk storage, zoning).
Physical location Room, building, city, etc. where the server is installed.
Revision March 2006
Page 1-22

Storage
Gather data about each disk storage system:
Storage system
Connections: Count, media (Cu/SW/LW/ELW), connector
(SC/LC/HSSDC2), type (loop/fabric), I/O profile
Storage capacity: Usable capacity, capacity in use
SAN issues: Remote mirroring, SAN security
Physical location
If possible, obtain the server-to-storage map (LUNs in use)

Usually developed with the storage vendor

Revision March 2006
23
As with the server information, consider this to be the absolute minimum

information to be gathered for each server. Some additional notes on the values
above:
Storage system name, manufacturer, and model The name used to identify the
make and model of the storage system.
Connection Points The number of separate connectors available for users to
access data from the storage system, and the media type of the connections.
Storage capacity The total capacity available for data storage after mirroring of
parity is organized; and the amount of data currently being stored.
SAN Issues - Whether the storage can mirror data to a remote source without
server intervention, and the level of SAN security supplied by the storage (target,
LUN, file, or none).
Physical location Room, building, city, etc. where the storage is installed.

Revision March 2006
Page 1-23

Storage (cont.)
Gather data about the tape storage:
Storage system
Drives
Storage capacity
Physical location
Backup software and architecture
Backup hardware
Backup schedule
Storage support
If possible, obtain the server-to-tape-drive map
Usually developed with the storage vendor

Revision March 2006
24
As with the server information, consider this to be the absolute minimum

information to be captured for each server. Some additional notes on the values
above:
Storage System The storage name, manufacturer, and the manufacturers model
name.
Drives The number of drives, the type of each drive, the media supported (Cu,
SW, LW, ELW), and the connector type (SC, LC, HSSDC2).
Storage capacity The total internal capacity for storing tape cartridges.
Physical location Room, building, city, etc. where the storage is installed.
Backup hardware The manufacturer of any additional hardware, including SCSIto-FC bridges.
Backup schedule The details of all backups on a per-server basis, including
full/incremental collection, scheduled time, backup window, and data quantities.
Storage support Whether data backup is performed from live data volumes,
snapshots, or split mirrors.

Revision March 2006
Page 1-24

Availability
The desired data availability for the current SAN must be clearly
defined can strongly impact the desired solution

Downtime maximums Both for unplanned and planned downtime
Data path definitions Resilient (multi-path per fabric) or Redundant
(multi-fabric per SAN)

Service Level Agreement (SLA) requirements (if applicable)
Availability is also defined for applications as well
Uptime/downtime
Maintenance window requirements

Revision March 2006
25
Data and application availability depend on servers being able to access the data
stored on the SAN. This availability must endure implementation, routine
maintenance, and any repairs or upgrades. If you implement a dual-fabric
redundant SAN solution, then application availability will be improved.

Revision March 2006
Page 1-25

Performance
Quantifying performance has several key factors:
Identify peak and sustained performance thresholds
Peak: Driven by end-of-cycle, Web events, etc.
Sustained: Often far lower than the peak
When will the peak(s) occur?
Backups usually create peaks
Allow more bandwidth in proportion to increasing numbers of
simultaneously occurring peaks
Changing peak times can better distribute required peak performance,
reducing the final cost of the solution
For long-distance applications, latency or response time may
matter more than throughput or I/Os per second

Revision March 2006
26
Any SAN design must always support sustained performance requirements and
provide for peak needs. Gathering relevant information at an early stage helps to
identify peak times, and encourages planning for additional capacity or modifying
processes to avoid network overloading.

Revision March 2006
Page 1-26

Performance (cont.)
Collect the current SAN performance data in a table
Try to measure peak and sustained requirements
Initiator
SERV1
SERV2
SERV3
Target
Peak
(MB/sec)
When is peak?
Sustained
(MB/sec)
STOR2
10
1st, 1800
TAPE6
20
Sun, 0100
STOR1
10
1st, 1800
STOR2
10
1st, 1800
TAPE3
20
Sun, 0100
STOR1
40
30th, 2300
20
TAPE3
20
Sun, 0100

Revision March 2006
27
The above example represents a performance table for three servers within a
proposed SAN. We can see that the connection to TAPE3 has a large peak every
Sunday at 0100 (possibly a full backup by SERV2 and SERV3?), and that STOR2
has a large peak on the first of every month at 1800 (perhaps an end-of-month
process?). As we gather data about other servers in the SAN, we can develop
greater insight about the requirements of the final design.

Revision March 2006
Page 1-27

Security
Security policies are defined at three levels:
1. Host Level - Volume Management, Multipathing, HBA Drivers
(persistent binding)
2. Fabric Level - Hard Zoning, Secure Fabric OS
3. Storage Level - Selective LUN Presentation, LUN Security
SAN data security is rarely just implemented as a single-level solution
look for multiple levels

Revision March 2006
28
SAN security is implemented on both servers and storage, as well as in the fabric or
SAN itself. By implementing a multi-level SAN security solution, simple evasive
techniques (like password theft or WWN spoofing) are thwarted.

Revision March 2006
Page 1-28

LAN Infrastructure
Data collected on the current LAN/MAN/WAN infrastructure should
focus on key SAN-related items:

Management: Reserved IP addresses for SAN-attached devices
on a separate VLAN or subnet? Application bandwidth needs?
Data: Connectivity needed for SAN gateway switches or FCIP?
Speeds and distances over the MAN and WAN?

Revision March 2006
29
When documenting the current LAN infrastructure, focus on two key areas:
To manage the SAN-attached devices, what are the IP addresses reserved, and are
they in separated VLANs or subnets? In addition, do the SAN management
applications require constant device access (for live monitoring)?
If the SAN devices are going to transfer data via SAN gateway switches or an
MPR (over FCIP), what are the speed and distance requirements for the
WAN/MAN?

Revision March 2006
Page 1-29

Multiprotocol Infrastructure
Depending on design decisions, a Multiprotocol Router may be
included, so identify the current multiprotocol environment:

FCIP Tunneling Service Information
IP addresses
Distances
MAN/WAN connection details
iSCSI Gateway Service
iSCSI hosts and FC storage
LAN interface
Application and performance information

Revision March 2006
30
The MetaSAN infrastructure need to be documented as well.

Revision March 2006
Page 1-30
Current SAN Requirements Multiprotocol Infrastructure (cont.)

For FC-to-FC Routing, collect the following information:
Front domains per edge fabric
Translate domains per edge fabric
Real domains (i.e. switches) per edge fabric
Total domains per edge fabric (physical domains + front domains +
translate domains)
Local and remote devices per edge fabric
Imported devices per edge fabric
Edge fabrics connected to a routed fabric
Fibre Channel routers per routed fabric
Maximum hops between switches

Revision March 2006
31
For a Multiprotocol Router that implements FC-to-FC Routing, collect additional

information.

Revision March 2006
Page 1-31
Target SAN Requirements

Overview
For the target SAN environment, collect the same information as for
the current SAN environment

Facilities
SAN Infrastructure
Servers and Storage (Disk and Tape)
Availability (Data and Application)
Performance
Security
LAN and Multiprotocol Infrastructure
with a focus on new initiatives, growth, and planning

Revision March 2006
32
Before creating a SAN design, the target SAN environment must be clearly defined.
When collecting information on the current SAN environment, you may also be
able to collect the information listed above on the target SAN. The focus here
should be on new initiatives, growth estimates, and planning details for the target
SAN solution.

Revision March 2006
Page 1-32

Growth
Growth is often measured in several ways:
Next 12 months
Over the next 12-24 months
Maximum numbers of servers, site-to-site connections
Maximum storage capacity (disk and tape)
Also capture how often changes will be made during maintenance
windows only, or at any time?

In each rack, how much space is dedicated to future cabling?

Revision March 2006
33
These questions can play a major role in determining the final structure of a
solution.

Revision March 2006
Page 1-33

Planning Details
In addition, collect information that can affect SAN design and
planning details
Customer schedule: target time for pilot, first implementation, any
information relating to the timeline of the SAN project

Switch vendor and SAN partners
Support providers for all components

Revision March 2006
34
To ensure timely delivery of the target SAN, as well as correct interlock with your
switch vendor and SAN partners, cover vital business items up front.

Revision March 2006
Page 1-34
Data Collection - Organizing the Data

Ensure that your data is well-organized before creating your SAN
design
As needed, create your own spreadsheet or database
Many vendors use their own proprietary spreadsheets
Brocade also provides the SAN Health tool that automates data collection
from the current SAN

Revision March 2006
35
The organization of current and target SAN requirements are vital to ensuring a
valid, accurate SAN design. Depending on the size of the data set, use either a basic
spreadsheet or a more complex database. Depending on your switch vendor, you
may be required to use a specific spreadsheet. The Brocade SAN Health tool
(downloadable from http://www.brocade.com) automates collection of data
from the current SAN.

Revision March 2006
Page 1-35
SAN Design Overview

What are the key considerations in designing a SAN?
SAN availability must ensure the delivery of data from servers to
storage - if a server cannot write data to SAN-attached storage, that

data is lost
SAN performance should ensure sufficient bandwidth
SAN topologies should leverage Brocade best practices
SilkWorm switches should be selected to meet availability,
performance, and topology requirements, as well as scalability needs

Revision March 2006
36
The above points are essential to planning and design a SAN solution that can be
implemented and managed successfully.

Revision March 2006
Page 1-36
SAN Design Suggestions

Availability
A well-connected SAN should avoid single points of failures (SPOFs)
ISLs, ports, switches, HBAs, storage ports, fabric itself
Fabrics with no SPOFs are considered resilient fabrics
Always at least two paths between any two switches
Switch 1
Switch 2
Switch A
SPOF =
Switch B
Switch 3
Switch 4
Switch C
Switch D
Resilient Fabric
Non-Resilient Fabric

Revision March 2006
37
Single points of failure, or SPOFs, are any component hardware or software

that, if failed, results in server(s), storage, and/or switch(es) disconnected from the
fabric. An SPOF could be a switch or ISL in a non-resilient topology, or a host with
no clustering software installed. In an environment where uptime is absolutely
critical, even distributed fabric services (such as the Name Server) can be viewed as
potential single points of failure. In the right-hand fabric above, the path for traffic
between Switch A or B and Switch D is dependent on Switch B, making Switch B a
potential SPOF.
An example of a resilient fabric is the left-hand fabric shown above. This fabric
allows for multiple paths between any two switches that do not share hardware, and
has no SPOFs. Because the right-hand fabric above includes a SPOF, that fabric is
said to be non-resilient.

Revision March 2006
Page 1-37

Availability (cont.)
For best availability of the overall SAN solution, create a Redundant
SAN
A dual resilient-fabric SAN with no SPOFs
Switch 1A
Switch 2A
Switch 1B
Switch 2B
Switch 3A
Switch 4A
Switch 3B
Switch 4B
Resilient Fabric A
Resilient Fabric B

Revision March 2006
38
Creating a redundant SAN implies a certain level of hardware investment:

At least two HBAs per server
At least two storage ports per storage system
Each server and storage system must be connected to both fabrics
Remember - a SAN does not equal a single fabric! Since the fabric exists in both
hardware and software, a dual-fabric SAN guards against any potential failure in the
software.

Revision March 2006
Page 1-38

Even with the higher availability in the Brocade SilkWorm 3900 and
12000, a redundant SAN (based on two resilient fabrics) is essential

Long-term maintenance test new actions of one fabric while
maintaining data access through the second fabric
Make each fabric resilient to survive hardware, software, user, site
failures

Revision March 2006
39
A redundant SAN, comprised of at least two resilient fabrics, is a key part of any
long-term high-availability solution. With a redundant SAN solution, all SAN
installations and maintenance can be performed on one fabric at a time without
affecting ongoing data traffic that continues in the second fabric.
Making each fabric resilient (e.g., by the addition of multiple paths between
switches, dual-HBA servers with path failover, and dual-ported storage) ensures
that any device failures do not compromise data access. Multiple resilient fabrics
ensure that hardware failures, software failures, human errors, and site problems
which might fail a switch or fabric do not fail the SAN.

Revision March 2006
Page 1-39

With dual-fabric designs that include the
SilkWorm 12000, ensure best availability

by designing for only one Fabric per
SilkWorm 12000 chassis
Better ability to survive environmental
hazards
Protects against propagation of
firmware faults and chassis-level
operator errors
Two fabrics per chassis should be
considered only on an exceptional
basis

Revision March 2006
40
The above slide shows a dual-fabric solution in which each fabric contains a single
Brocade SilkWorm 12000 chassis with two logical switches each.
Environmental hazards: The Brocade SilkWorm 12000 chassis is a robust
enclosure, but it may not survive all physical hazards (including fires, water
damage, earthquakes, etc.). To ensure best availability, only one fabric should be
reside in a single Brocade SilkWorm 12000 chassis. With dual-fabric designs, try to
keep the Brocade SilkWorm 12000 chassis for each fabric in separate locations
within the data center or the enterprise to provide even better chances of survival.
Firmware faults: Within a single Brocade SilkWorm 12000 chassis, the same
version of Brocade Fabric OS runs on both logical switches. During a firmware
upgrade, we want to test new firmware in one fabric before taking both fabrics
operational. The logical switches in a Brocade SilkWorm 12000 chassis should be
kept within the same fabric so that new firmware (or any firmware faults) can be
isolated to only one of the two fabrics in the SAN.
Chassis-level operator errors: As we have seen, there are certain Brocade
SilkWorm 12000 operations that can affect both logical switches within the chassis.
To guard against any errors resulting from these operations from affecting both
fabrics in a SAN, keep the logical switches in a Brocade SilkWorm 12000 chassis
within the same fabric.

Revision March 2006
Page 1-40

Single-chassis SilkWorm Directors offer the highest availability
The SilkWorm 24000 as one logical 128-port switch
One logical SilkWorm 12000 or two logical SilkWorm 12000s
connected using ISLs

A two-chassis SilkWorm 24000 core is the most available solution
Single core core/edge fabrics are more likely and acceptable
Should be deployed in a redundant fabric SAN
Not recommended for single fabric SANs

Revision March 2006
41
A computer system or application is only as available as the weakest link. To build

a highly available computer system, it is not sufficient to only have a highly
available SAN. It is necessary to account for availability throughout the entire
computer system: dual HBAs, multi-pathing software, highly available and multiported storage subsystems, and clustering software are some of the components that
may make up such a system.

Revision March 2006
Page 1-41
MetaSAN Design Concepts

Availability
What are some of the key techniques used to design highly-available
SAN solutions?
Make each fabric resilient (at least two paths between any two
switches in the fabric)
Have redundant fabrics (at least two fabrics per SAN)
Attach devices redundantly to the SAN (to at least two fabrics)
When designing a highly-available MetaSAN solution, leverage the
same techniques used to design highly-available SAN solutions
Focus: Avoid creating single fault domains

Revision March 2006
42
The list of techniques above should already be familiar from SAN design:
Fabric resiliency ensures that any failures within a fabric do not interrupt data
flow.
Redundant fabrics protect against any failures of an entire fabric.
Redundant device attachment defends against failures within the attached devices,
or in any cables.
These techniques result in a SAN design that avoids single points of failure that
could result in loss of data flow. When designing MetaSANs, seek to leverage
resiliency and redundancy, but in the context of Edge Fabrics and Routed Fabrics.

Revision March 2006
Page 1-42

The key techniques for
increasing the availability of a

MetaSAN are:
1. Attach each fabric in a
redundant fabric SAN to a
separate Backbone Fabric in a
redundant Routed Fabric
Corollary: Avoid attaching
both fabrics in a redundant
fabric SAN to the same
Backbone Fabric

Revision March 2006
43
Attaching dual-fabric SANs: In the example above, SAN-1 and SAN-2 are dual
fabric SANs (Fabric A and Fabric B). Each SAN fabric is connected to
separate Routed Fabrics (A and B), ensuring that devices in the two SANs can
communicate, but that fault isolation is maintained between fabrics in the same
SAN.

Revision March 2006
Page 1-43

2. Attach each Edge Fabric
resiliently to the Backbone

Fabric, with more than one IFL
Ensures that an IFL failure
does not impact fabric-tofabric routing
4
2
4
2

Revision March 2006
44
Attaching Edge Fabrics with resiliency: In the example above, the Edge Fabrics in
SAN-1 are each attached to a Routed Fabric by 4 IFLs. This ensures that the
connection between these Edge Fabrics and the Backbone Fabric is resilient. In a
similar manner, the Edge Fabrics in SAN-2 are each attached to a Backbone Fabric
by 2 IFLs the minimum needed to ensure resiliency.

Revision March 2006
Page 1-44

3. For the highest levels of
availability, create redundant

Routed Fabrics
Use two Backbone Fabrics
per Routed Fabric
Attach each Edge Fabric to
each Backbone Fabric with
at least one IFL

Revision March 2006
45
The example above expands on the concepts presented in the previous slide. By
connecting each Edge Fabric to two Backbone Fabrics, Routed Fabric maintenance
(and other issues) can be handled without interrupting Edge Fabric access an
essential requirement for solutions that require the highest levels of availability.

Revision March 2006
Page 1-45

Device Attachment
Strategies for attaching devices is influenced by the selected topology
Cascade, Ring: Maximize server-storage locality
ISLs mainly for Fabric management not for data traffic
Full Mesh: Locality is preferred
Still relatively few ISLs, so beware of ISL over-subscription
Core/Edge: Servers and storage should be attached to Edge switches
All I/Os pass through the Core switches, increasing the number of
equal-cost data paths

Can add storage to the Core this limits scalability

Revision March 2006
46
Whenever attaching devices, locality always achieves the best performance, as

there are no congestion points in the SAN to overcome. As SAN sizes increase
though, the physical distribution of existing devices may make it difficult to find
sufficient switch ports for localized connections. Controller-based storage may be
shared by so many servers that complete locality may be impossible. Similarly,
servers using storage virtualization techniques may access so many storage systems
as to make complete locality difficult to achieve.
The Core/Edge topology provides an acceptable number of ISLs, so that there is
good bandwidth between non-local servers and storage. Here, locality should be
used strategically, between high-performance servers and storage focused on these
servers. In this way, the heaviest traffic remains local to the switches, with SAN
bandwidth used by the majority of lower-performance servers.

Revision March 2006
Page 1-46

Device Attachment (cont.)
To best leverage locality on different SilkWorm switches, connect
servers and storage as follows:

Eight-port and sixteen-port switches: Same quad
SilkWorm 3900: Same octet
SilkWorm 12000: Same quad on the same Port Card
With the SilkWorm 3900 and 12000, use locality to limit congestion
from high-performance servers and storage
Servers and storage that intercommunicate localize on the same
SilkWorm 3900 octet or SilkWorm 12000 quad
Servers and storage that do not intercommunicate mix with
lower-performance servers and storage across several
quads/octets

Revision March 2006
47
Brocade 8-port and 16-port switches are non-blocking and congestion free.
The SilkWorm 3900 and 12000 are non-blocking switches that can encounter
congestion under certain conditions, particularly if the sustained SilkWorm 12000
quad-to-quad traffic is expected to exceed 4 Gbit/sec, or sustained SilkWorm 3900
octet-to-octet traffic is expected to exceed 8 Gbit/sec. This happens only if several
2 Gbit/sec devices connected to different quads/octets are expected to communicate
simultaneously at a sustained rate of greater than 1 Gbit/sec. This is a high
requirement, as a device that connects at 2 Gbit/sec rarely sustains the full
bandwidth over time. In this case, try one of these options:
If multiple high-performance devices are communicating with each other
simultaneously and at full bandwidth, localize all the high-performance 2
Gbit/sec devices together on the same quad or octet. This is leverages the
congestion-free connects available within the same quad/octet.
If the high-performance are not communicating with each other or are shared
by many other devices, then these devices should be connected to quads/octets
with lower-performing devices. By limiting the number of high-performance
devices per quad/octet, we manage the quad-to-quad and octet-to-octet traffic
within the limits discussed earlier.

Revision March 2006
Page 1-47

With the Brocade SilkWorm 12000,
add these rules:

1. For best availability, distribute
devices and ISLs across 16-port
cards
Advantage: Limits the impact of
the failure (or removal) of a 16port card
Caution: For storage, increases
impact of storage planning on
I/O traffic; for ISLs, limits use of
trunking
Distribute high port

count devices (such
as arrays or tape
libraries) across
multiple blades
Storage
Tape
Storage
Tape
Servers
Servers
ISLs
ISLs
Distribute Devices and

ISLs Across Blades
Revision March 2006
48
When attaching devices to Brocade SilkWorm 12000 logical switches, connect

HBAs and storage ports to different port cards to eliminate a potential single point
of failure.

Revision March 2006
Page 1-48

2. For best performance on ISLs
between Brocade SilkWorm 12000

switches, use a mix of rules 1 and 2
above.
Distribute the ISLs across at least
two 16-port cards
With multiple ISLs on a port card,
concentrate them on the same
quad
2-4
2-4

Revision March 2006
49
Connections between Brocade SilkWorm 12000 switches are first distributed across
port cards to ensure that the high levels of switch availability are not lost through
over-concentrated connections. After distributing these ISLs across at least two port
cards, we can then leverage trunking by concentrating any additional ISLs within
the same quad of a port card.

Revision March 2006
Page 1-49

Performance
The Brocade method for quantifying
performance is the ISL over-subscription

ratio
Actually a measure of the potential for
network contention
For each switch, the ratio of the
bandwidth needed by HBAs to the
bandwidth available on the ISLs
With all 2 Gbit/sec devices and
switches, # servers / # ISLs
SERV1 SERV2 SERV3 SERV4
SW1
SW2
STOR2

Revision March 2006
50
The ISL over-subscription ratio is defined in terms of bandwidth, not just HBA and
ISL counts. This recognizes the difference in performance of HBAs and ISLs,
depending on the speed of the switches in the Fabric and the HBAs in the attached
servers. In the example above, the four servers SERV1-4 connected to switch SW1
must share a single ISL to reach storage system STOR2, making for a 4:1 ISL oversubscription ratio.

Revision March 2006
Page 1-50

Performance (cont.)
Most successful SAN designs start with an ISL over-subscription ratio
of 7:1
One ISLs per eight-port switch
Two ISLs per 16-port card or switch
Four ISLs per SilkWorm 3900 switch
High-performance SANs or conservative designs may consider an ISL
over-subscription ratio of 3:1
Edge
Core
Revision March 2006
51
Brocade has found that SAN designs with an ISL over-subscription ratio of 7:1 are
suitable for many user environments. This corresponds to two ISLs per SilkWorm
12000/24000 port card, four ISLs per 32-port SilkWorm 3900, two ISLs per 16-port
SilkWorm 3800/3850, and one ISL per eight-port SilkWorm 3200/3250. The
example above demonstrates a 7:1 ISL over-subscription ratio between the
SilkWorm 3850 Edge switches and the Core switches.
To provide additional ISL bandwidth for large-block applications, add ISLs in
parallel with existing ISLs. A common ISL over-subscription ratio for higherbandwidth solutions (or more conservative designs) is 3:1.

Revision March 2006
Page 1-51

Performance (cont.)
To enhance performance, focus on two techniques:
1. Locality Leverages internal bandwidth; requires lots of planning
Focus locality on the largest servers: Large-bandwidth, streaming
applications through 100 MBytes/sec or 200 MBytes/sec HBAs

and storage
2. Adding ISLs to leverage trunking - Watch port counts
Edge
Core

Revision March 2006
52
Locality between servers and storage has the lowest cost of any of the techniques
discussed today. By carefully selecting attach points from the available switch
ports, we can keep server-storage I/O traffic away from the ISLs. By some
measures, this may be the most difficult technique to implement, as the planning
needed for locality grows even the size of the SAN grows. As discussed earlier,
high-traffic servers are best suited to locality here, we refine this definition to
large-bandwidth servers, as these servers tend to share the available ISL bandwidth
poorly.
Adding ISLs adds bandwidth immediately, and can leverage trunking if you plan
carefully. In the example above, two additional ISLs have been added between the
left-most Edge and left-most Core switches.

Revision March 2006
Page 1-52

Performance (cont.)
To leverage trunking in Edge switches, allocate ports for ISLs, servers,
and storage on a quad basis

Quad: Group of four SilkWorm 3000-series or 12000 switch ports;
color-coded on the switch or port card
If possible, leave other ports in the quad open for performance or
scaling growth
ISL
Quad
Server Server Storage

Quad Quad Quad

Revision March 2006
53
Ports within the same quad share the same port control circuitry. If we connect
multiple ISLs between two switches that share the same quad on both switches, the
SilkWorm port circuitry merges them into a single trunk. On most SilkWorm
switches, the port numbers are color-coded to indicate quad membership.
For the SilkWorm 3000-series and 12000 switches, the following port groups are
members of the same quad:
SilkWorm 3200: Ports 0-3; ports 4-7.
SilkWorm 3800 (image, above): Ports 0-3; ports 4-7; ports 8-11; ports 1215.
SilkWorm 3900: Ports 0-3; ports 4-7; ports 8-11; ports 12-15; ports 16-19;
ports 20-23; ports 24-27; ports 28-31.
SilkWorm 12000 (16-port card): Ports 0-3; ports 4-7; ports 8-11; ports 1215.
In the example above, one quad has been allocated for ISLs, two quads for servers,
and one quad for storage. We will refine this methodology later in this module.

Revision March 2006
Page 1-53

Performance (cont.)
On a SilkWorm 3900 Edge switch, allocate ISL ports from quads on
opposite corners on the same switch

Leverages internal connections between port hardware
For locality, seek to use ports within the same octet (neighboring
quads)
Octet
Octet
Quad
ISL
Quad
Quad
Quad
ISL
Quad
Quad
Quad
Quad

Revision March 2006
Octet
Octet
54
In the slide above, we see a SilkWorm 3900 with the recommended allocation of
ISL quads.
On the SilkWorm 3900, adjacent pairs of quads share additional connections,
providing even better levels of performance. These octets, shown in the drawing
above, are the following group of ports: ports 0-7, ports 8-15; ports 16-23, and ports
24-31.

Revision March 2006
Page 1-54

Performance (cont.)
On a SilkWorm 12000 or 24000
Edge switch, allocate one quad per

Port Card for ISLs
Use different quads on each
Port Card (allocate diagonally)

Revision March 2006
55
In the slide above, the port blades in a SilkWorm 12000 logical switch are shown.
The strengths of this strategy are:
Distributing the ISLs across the blades maximizes availability by minimizing the
impact of a failed Port Card.
Using different quads on each Port Card provides for a slight improvement in
performance.

Revision March 2006
Page 1-55

Performance
Use these techniques to assess
and enhance performance in a

MetaSAN:
1. Fabric Locality Whether a host
and storage are attached to the
same Edge Fabric
High fabric locality Traffic
kept local to the Edge Fabric
No fabric locality IFLs are
used for traffic

Revision March 2006
Host A->Storage B:
High fabric locality
Host X->
Tape Y:
No fabric
locality
56
Hosts and storage are considered fabric local if they are connected to the same
Edge Fabric. In the example above, host A and storage B have a high degree of
fabric locality, as both are connected to SAN-1 Fabric A; thus, IFLs in the Routed
Fabric do not carry traffic between these devices. However, host X and tape Y are
connected to separate Edge Fabrics, so that any traffic between these devices would
be carried by IFLs (as shown by the arrows above).

Revision March 2006
Page 1-56

Performance (cont.)
2. IFL Oversubscription Ratio Ratio of devices with non-fabric-local
traffic to IFLs
Similar to the ISL Oversubscription Ratio concept used in SAN
design
Check fan-in and fan-out profiles
If actual bandwidth data is available, use the formula: (sum of
server bandwidths) / (sum of IFL bandwidths)
Recommendations for connecting and provisioning IFLs:
Connect at a 15:1 IFL oversubscription ratio
Provision for a 7:1 IFL oversubscription ratio
To provide resiliency, connect each Edge Fabric to each Backbone
Fabric with at least two 2 Gbit/sec IFLs

Revision March 2006
57
In SAN design, the ISL Oversubscription Ratio determines the number of ISLs
needed to connect a switch to the fabric, based on the number of devices attached to
the switch that may need to share the ISLs. In a similar manner, the IFL
Oversubscription Ratio determines the number of IFLs need to connect an Edge
Fabric to a Backbone Fabric, based on the number of devices with non-fabric-local
traffic that must share the available IFL bandwidth.
The IFL oversubscription ratio above translate to IFLs as:
15:1 one IFL per SilkWorm Multiprotocol Router
7:1 two IFLs per SilkWorm Multiprotocol Router
This technique should yield a conservative estimate for the number of IFLs, but
may not be suitable for specific MetaSAN solutions. As in SAN-based solutions,
use tools that track actual port traffic metrics (Fabric Watch, Web Tools, etc.) to
fine-tune IFL counts.
As with ISLs, provision for future IFLs by pre-allocating ports on Edge Fabric
switches and Backbone Fabric routers. For good resiliency, connect each Edge
Fabric to each Backbone Fabric by at least two 2 Gbit/sec IFLs.

Revision March 2006
Page 1-57

BCSDNutshell VC

Uploaded by

Copyright:

Available Formats

BCSDNutshell VC

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BCSDNutshell VC

Uploaded by

Copyright:

Available Formats

Brocade

Brocade Education Services

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Brocade Certified SAN Designer (BCSD)

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Brocade Certified SAN Designer:

can meet and solve customer needs

needs to be collected about their current and target environments to

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Training Objectives (cont.)

considering design trade-offs

scalability, and performance requirements

as single or redundant, resilient or non-resilient, and the fault tolerance level

ISL/trunks, quads/octets and link speed to determine performance and

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Training Objectives (cont.)

considering design trade-offs (cont.)

topology and switch types are appropriate

restrictions have been violated

requirements, identify which SAN components will need to be integrated.

requirements, based on Brocade guidelines

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Training Objectives (cont.)

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Training Objectives (cont.)

4. Optimize and Tune SAN Design

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

hour sessions which focuses on the four main tasks of a Brocade

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

SAN Applications - Overview

of business applications and initiatives

2006 Brocade Communications Systems, Incorporated.

Storage Area Networks drive benefits to a number of applications and initiatives,

2006 Brocade Communications Systems, Incorporated.

SAN Applications - Overview (cont.)

Migrate to fewer, larger, faster servers

Consolidation, utilization, and data sharing, and centralized

Move data to disk or tape storage systems without using the

Clustering, business continuance

Improve data rates and response times

Ensure that specified servers can access specified storage

Add servers and storage without taking costly downtime for

2006 Brocade Communications Systems, Incorporated.

Storage Area Networks drive benefits to a number of applications and initiatives,

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

2006 Brocade Communications Systems, Incorporated.

Disaster Tolerance: The continuing availability and integrity of corporate servers