#CLUS

Cisco Nexus 7000 /

7700 Switch
Architecture
Shankar Gopalkrishnan
Principal Engineer
BRKARC-3470

#CLUS
Session Abstract
This session presents an in-depth study of the architecture of the
latest generation of Nexus 7000 and Nexus 7700 data center
switches.
Topics include supervisors, fabrics, I/O modules, forwarding engines,
and physical design elements, as well as a discussion of key
hardware-enabled features that combine to implement high-
performance data center network services. Focus will be on the
Nexus 7700, especially on the new hardware additions.

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Goal
• To provide a thorough understanding of the Nexus 7000 / Nexus
7700 switching architecture, chassis, supervisor, fabric, and I/O
module design, key forwarding engine functions, and complete
packet flows
• This session will not examine NX-OS software architecture or other
Nexus platform architectures

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-3470

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Introduction to Nexus 7000 / Nexus 7700 Platform
Data-center class Ethernet switches designed to deliver:
• High performance
Chassis
• High availability
• System scale Fabric
Modules
• Investment protection
• Feature-rich
• High density of 10G/40G/100G
Line
Supervisor Cards

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Nexus 7000 / Nexus 7700 – Common Foundation
Nexus 7000 Nexus 7700
General purpose DC switching w/10/40/100G Targeted at Dense 40G/100G deployments

Common Foundation

• Same release vehicles, versioning, feature-sets

• Common configuration model
• Common operational model

• Common fabric ASICs (Fab2) and architecture

• Same central arbitration model
• Same VOQ/QOS model

• Identical forwarding ASICs (F3, M3)

• Consistent hardware feature sets
• Consistent hardware scale

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Nexus 7700 Series – Next Generation Modules

Supervisor
I/O Modules Fabric Modules
Modules

Nexus 7700 F4 Series Nexus 7700 Supervisor 3E

Nexus 7700 Fabric 3 *
30-port 40/100G QSFP28 New Multi-Core CPU
2.4 Tbps per Slot
More DRAM

FCS FCS FCS

F4: Compatible with Supervisor 2E and Fabric 2 Modules | VDC Interoperability with F3 or M3 I/O Modules

* For Nexus 7700 6-slot and 10-slot chassis

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Nexus 7700 Chassis Family NX-OS 6.2(2) and later

Nexus 7718 Chassis

NX-OS 6.2(2) and later

Back
Nexus 7710
Back

26RU

14RU
Front

Front

Front Rear Front Rear

N77-C7710 N77-C7718

NX-OS 6.2(6) and later NX-OS 7.2(0)D1(1) and later

Nexus 7706 Back
Back Nexus 7702

3RU

9RU Front Rear

Front
Front N77-C7702
Front Rear
N77-C7706 #CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Supervisor 3E Engine
SUP
SUP

• Provides all control plane and management functions

• Connects to fabric via 10G inband interface.
• Interfaces with I/O modules via 1G switched EOBC
• Onboard central arbiter ASIC N77-SUP2E
Controls access to fabric bandwidth via dedicated arbitration path to I/O modules

Feature Supervisor Engine 2E Supervisor Engine 3E

CPU/Mem Two quad-core 2.1GHz, 32 GB 8-core 2.0 GHz, 64 GB

Co-processor None Co-processor/Offload engine
SSD None 240GB
Inband ports 1 Gbps 10 Gbps
ID and Status USB Expansion
LEDs Flash

Console Port Management

Ethernet

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Supervisor 3E Engine Architecture
SUP
SUP

To Module CPUs To Fabric Modules To Module VOQs

Dedicated
Switched
Arbitration
1GE EOBC Fabric ASIC Path
Dedicated Central
Arbitration Arbiter
Switched Path
EOBC VOQs

10GE EOBC 10GE Inband

I/O Controller

Bootflash
SSD NVRAM
(eUSB)
10 Gbps
4 GB
Main CPU
240 GB
With FSA
DRAM

2.0 GHz 64 GB
Console Mgmt0 Mgmt1
8-Core
USB device port

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 M1 M2 LC
1G / 10G 10G / 40G / 100G

M3 delivers best of M-
and F-series capabilities
M-Series Modules
L2/L3/L4 with large forwarding tables and rich feature set
M3

10G / 40G / 100G

F1 F2/F2E
10G
10G
F3
10G / 40G / 100G
F4 increases
F-Series Modules
High performance, low latency with streamlined feature set
F3 closes the
F/M feature gap!
F3
100G port density
F4

F2E 100G
10G
10G / 40G / 100G
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
19
 LC

Nexus 7700 F3 I/O Modules

• 10G / 40G / 100G F3 I/O modules
• Share common hardware architecture N77-F348XP-23
• SOC-based forwarding engine design
• 6 independent SOC ASICs per 10G module
• 12 independent SOC ASICs per 40G module
• 12 independent SOC ASICs per 100G module

• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS) N77-F324FQ-25

and advanced features (FP/MPLS/OTV/GRE/VXLAN etc.)
• Onboard Fabric Services Accelerator (FSA)
Module Port Density Optics Bandwidth
F3 10G 48 x 1G/10G SFP+ 480G
F3 40G 24 x 40G (or up to 76 x 10G + 5 x 40G via QSFP+ 960G
breakout)
F3 100G 12 x 100G CPAK 1.2T N77-F312CK-26

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Nexus 7700 F3 48-Port 1G/10G Module Architecture
LC

N77-F348XP-23 To Fabric Modules

EOBC To Central Arbiters

FSA Arbitration
CPU Aggregator
…
x6
1G switch
… Fabric ASIC Fabric ASIC
x6
to FSA
LC Inband
CPU
to ARB

8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G 8 X 10G

SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 6

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports (SFP/SFP+)

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Nexus 7700 F3 24-Port 40G Module Architecture
LC

N77-F324FQ-25
EOBC To Fabric Modules To Central Arbiters

FSA Arbitration
CPU Aggregator
…
x 12
1G switch
… Fabric ASIC Fabric ASIC
x 12
to FSA
LC Inband
CPU
to ARB

2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G 2 X 40G
SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 6 SOC 7 SOC 8 SOC 9 SOC 10 SOC 11 SOC 12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Front Panel Ports (QSFP+)

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 Nexus 7700 F3 12-Port 100G Module Architecture
LC

N77-F312CK-26
EOBC To Fabric Modules To Central Arbiters

FSA Arbitration
CPU Aggregator
…
x 12
1G switch
… Fabric ASIC Fabric ASIC
x 12
to FSA
LC Inband
CPU
to ARB

1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G 1 X 100G
SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 6 SOC 7 SOC 8 SOC 9 SOC 10 SOC 11 SOC 12

1 2 3 4 5 6 7 8 9 10 11 12
Front Panel Ports (CPAK)

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 LC

Nexus 7700 M3 I/O Modules

• 10G / 40G / 100G M3 I/O modules

• Share common hardware architecture

• SOC-based forwarding engine design

N77-M348XP-23L
• 2 independent SOC ASICs per 10G module
• 4 independent SOC ASICs per 40G module
• 6 independent SOC ASICs per 100G module

• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS) and advanced

features (FP/MPLS/OTV/GRE/VXLAN etc.)

• Large forwarding tables (2M FIB/128K ACL)

N77-M324FQ-25L
• 256-bit 802.1AE LinkSec on all ports, all speeds

• Onboard Fabric Services Accelerator (FSA)

Module Port Density Optics Bandwidth
M3 10G 48 x 1G/10G SFP+ 480G
N77-M312CQ-26L
M3 40G 24 x 40G QSFP+ 960G
M3 100G 12 x 100G QSFP28 1.2T
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 LC

Nexus 7700 M3 48-Port 1G/10G Module Architecture

EOBC To Fabric Modules To Central Arbiters

10G FSA Arbitration

CPU Aggregator

to SOC 2 Fabric ASIC Fabric ASIC to SOC 1

TCAM Buffer TCAM Buffer

to FSA
to ARB CPU

24 X 10G 24 X 10G
SOC 1 SOC 2

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48
Front Panel Ports (SFP/SFP+)
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 LC

Nexus 7700 M3 24-Port 40G Module Architecture

EOBC To Fabric Modules To Central Arbiters

10G FSA Arbitration

CPU Aggregator
…
to SOC 4 Fabric ASIC Fabric ASIC to SOC 1 x4
to SOC 3

TCAM Buffer TCAM Buffer

to FSA to FSA
to ARB TCAM Buffer TCAM Buffer CPU CPU

6 X 40G 6 X 40G 6 X 40G 6 X 40G

SOC 1 SOC 2 SOC 3 SOC 4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Front Panel Ports (QSFP+)
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 LC

Nexus 7700 M3 12-Port 100G Module Architecture

EOBC To Fabric Modules To Central Arbiters

10G FSA Arbitration

CPU Aggregator
…
Fabric ASIC Fabric ASIC to SOC x6

TCAM Buffer TCAM Buffer TCAM Buffer TCAM Buffer to FSA

to ARB TCAM Buffer TCAM Buffer CPU

2 X 100G 2 X 100G 2 X 100G 2 X 100G 2 X 100G 2 X 100G

SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 6

1 2 3 4 5 6 7 8 9 10 11 12
Front Panel Ports (QSFP28)
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 LC

Nexus 7700 F4 I/O Modules

• 40G / 100G F4 I/O modules

• Share common hardware architecture

• SOC-based forwarding engine design

• 10 independent SOC ASICs per 100G module

• Layer 2/Layer 3 forwarding with L3/L4 services (ACL/QOS) and advanced

features (FP/MPLS/OTV/GRE/VXLAN etc.)

• forwarding tables (96K FIB/32K ACL) Nexus 7700 F4 Series

30-port 40/100G QSFP28
• Onboard Fabric Services Accelerator (FSA)

Module Port Density Optics Bandwidth

F4 40G 30 x 40G QSFP+ 1.2T
F4100G 30 x 100G QSFP28 2.4T

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
 LC

Nexus 7700 F4 30-Port 100G Module Architecture

N77K-F430CQ-36
EOBC To Fabric Modules To Central Arbiters

FSA Arbitration
CPU Aggregator

x 10 …
Fabric ASIC Fabric ASIC
SOC 1 SOC 10 to SOC
to FSA
CPU
to ARB

3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G 3 X 100G
SOC 1 SOC 2 SOC 3 SOC 4 SOC 5 SOC 6 SOC 7 SOC 8 SOC 9 SOC 10

1 3 4 6 7 9 10 12 13 15 16 18 19 21 22 24 25 27 28 30

2 5 8 11 14 17 20 23 26 29
Front Panel Ports (QSFP28)
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 LC

Fabric Services Accelerator (FSA) for F4

EOBC
• High-performance module CPU with on-board
acceleration engines
FSA CPU
• High-bandwidth inband connectivity from SOCs to
FSA (20 Gbps)
• Multi-Mpps packet processing
6 Core Acceleration
• 2 X 4GB dedicated DRAM
LC CPU Engines
• Performance/scale boost for distributed fabric
services
• High-rate sampled Netflow – 50Kpps sampled 4GB 4GB

copies per module

DRAM DRAM
I/O

• Distributed BFD – 15 msec hello timer, 45 msec

dead time for 250 sessions per module
2 x 10Gbps
• Similar FSA for F3 and M3 Module Inband
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
 LC

I/O Module Interoperability

• General module interoperability rule is: “+/-1 generation” in same
Virtual Device Context (VDC)
• F3 interoperability: coexists with either F4 or M3 in same VDC
• M3 interoperability: coexists with either F4 or F3 in same VDC
• F4 interoperability: coexists with either M3 or F3, in same VDC
• Interoperability model for current modules is “Ingress Forwarding
with Lowest Common Denominator”

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Introducing the F4 -Series into existing chassis
Lowest common denominator details

Fabric
VDC Type Layer 2 Layer 3 VXLAN MPLS OTV LISP Table Sizes
Path

✓ ✓ ✓ ✓ ✓ ✓ ✗
F4 F4

M3 + F4 ✓ ✓ ✓ ✓ ✓ ✓ ✗ F4 size

F3 + M3 + F4 ✓ ✓ ✓ ✓ ✓ ✓ ✗ F3 size

F3 + F4 ✓ ✓ ✓ ✓ ✓ ✓ ✗ F3 size

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 FM

Crossbar Switch Fabric Modules

• Provide interconnection of I/O modules
• Nexus 7700 fabrics based on Fabric 2 and Fabric 3 ASIC
• Each installed fabric increases available per-payload slot bandwidth
Per-fabric module Max fabric
Fabric Module Supported Chassis Total bandwidth per slot
bandwidth modules
Nexus 7700 Fabric 2 7706 / 7710 / 7718 220Gbps per slot 6 1.32Tbps per slot
Nexus 7700 Fabric 3 7706 / 7710 440Gbps per slot 6 2.64Tbps per slot

• Different I/O modules leverage different amount of available fabric bandwidth

• Access to fabric bandwidth controlled using QOS-aware central arbitration
with VOQ

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 Multistage Crossbar FM
FM

Nexus 7700s implement 3-stage crossbar switch fabric

• Stages 1 and 3 on I/O modules

• Stage 2 on fabric modules 2nd stage

Fabric Modules (FAB2) Fabric Modules (FAB3)

Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric
ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC
Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric
ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC ASIC
1 2 3 4 5 6 1 2 3 4 5 6

1.32T 2.64T
440G
220G
Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC

Ingress Module Egress Module Ingress Module Egress Module

1st stage
1st stage
3rd stage 3rd stage
Nexus 7700 Nexus 7700
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 I/O Module Capacity – Nexus 7700 FM
Fabric 2 Modules

1100Gbps
1320Gbps
880Gbps
660Gbps
440Gbps
220Gbps Local Fabric
Fabric
1
Fabric
ASICs
per slot bandwidth #2
#1 (480G)
One fabric:
2
• Any port can pass traffic to any other port Fabric
ASICs
in VDC
Three fabrics: Fabric
Local Fabric 3
#2
#1 (960G) Fabric
• 480G F3/M3 10G module has maximum ASICs

bandwidth
4
Five fabrics: Fabric
ASICs
Fabric
• 960G F3/M3 40G module has maximum Local Fabric
#2
bandwidth #1 (1.2T) 5
Fabric
ASICs
Six fabrics:
• 1.2T F3/M3 100G module has maximum 6
Fabric
bandwidth ASICs

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 I/O Module Capacity – Nexus 7700 FM
Fabric 3 Modules
2.64Tbps
2.20Tbps
1.32Tbps
880Gbps
440Gbps
1.76Tbps Local Fabric
Fabric
1
Fabric
ASICs
per slot bandwidth #2
One fabric: #1 (2.4T)

2
• Any port can pass traffic to any Fabric

other port in VDC

ASICs

Fabric
Three fabrics: Local Fabric
#2 3
#1 (960G) Fabric

M3 40/100G module has maximum

ASICs
•
bandwidth
4
Six fabrics:
Fabric
ASICs
Fabric
Local Fabric
• F4 100G module has maximum #1 (1.2T)
#2
5
bandwidth Fabric
ASICs

6
Fabric
ASICs

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 FM

What About Nexus 7702?

• Nexus 7702 has no fabric modules
• Single I/O module – all traffic locally switched
• Two fabric channels connect to supervisor engine

F3/M3
Module Fabric Fabric
ASIC ASIC

1* 55G
fabric channel
Supervisor
Fabric ASIC

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Hardware Forwarding Process
• Ingress queuing and scheduling LC

• Perform forwarding lookups LC

• Forward through fabric FM

• Egress queuing and scheduling LC

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Ingress / Egress Queuing and Scheduling
• Nexus 7000 / Nexus 7700 use queuing policies and
network-QOS policies to define queuing and scheduling
behavior
• Default policies always in effect in absence of any user
configuration
• Queuing model varies by I/O module architecture
• F-series modules / M3 modules: pure ingress-buffered
architecture

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 LC

M3/F3/F4 – Ingress Buffered

Central Arbiter

Ingress Buffering Multistage Crossbar Fabric

Egress buffer – Receives frames
from fabric and schedules traffic
toward egress ports
For more information about queuing and scheduling on Nexus 7000/7700,
EGRESS QUEUING POLICY
please visit BRKDCT-3346 - End-to-End
Independent q1
QoS Implementation and Operation with Cisco
scheduling for each VOQ buffer
destination/priority
q2
q3
Nexuscarved
Switches
by source
q4 and priority
4/8 priority levels
e2/9… per port (VQI)
e1/25
SP
DWRR
Virtual queuing – Virtual Ingress
Congestion management Queuing Buffer e2/9
Ingress buffer –
and local scheduling toward Buffers traffic for Egress Buffer
egress destinations (VQIs) congested egress
destinations (VQIs)
NETWORK-QOS POLICY
INGRESS QUEUING POLICY

N7700 10G F3 module used as example

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 LC

Hardware Forwarding Lookups

• Layer 2 and Layer 3 packet flow virtually identical in hardware
• Forwarding engine / decision engine pipeline provides consistent
L2 and L3 lookup performance
• Pipelined architecture also performs ingress and egress ACL, QOS,
and Netflow lookups, affecting final forwarding result

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 F4 Decision Engine • Return final result
(destination + priority) to
Ingress Buffer

To Ingress Final Results

Buffer
• Egress MAC lookups
L2 Lookup (post-L3)
• FIB TCAM and adjacency table
lookups for Layer 3 forwarding
• ECMP hashing

FIB
Layer 3 FIB Policing
• Egress ACL/QOS TCAM • Egress policing
classification
• Ingress policing

MAC CL Classification Egress lookup

Table TCAM (ACL/QOS/SNF) pipeline

Ingress lookup
• Ingress ACL/QOS/SNF pipeline
classification
• Ingress MAC table lookups
L2 Lookup (pre-L3) • Port-channel hash result

Ingress Parser Decision Engine

• Receive packet from Port Logic block PKT HDR

• Send payload to Ingress Buffer
• Send header to Decision Engine From Ingress F4 SOC
Port Logic
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 M3 Decision Engine LC

To Ingress Final Results

Buffer

L2 Lookup (post-L3)

Large FIB TCAM

and CL TCAM
external to SOC
FIB
Layer 3 FIB Policing
TCAM

CL Classification Egress lookup

TCAM (ACL/QOS/SNF) pipeline

MAC
Table Ingress lookup
pipeline

L2 Lookup (pre-L3)

Ingress Parser Decision Engine

PKT HDR
From Ingress M3 SOC
Port Logic
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 LC

Layer 2 Hardware Forwarding

• Layer 2 forwarding driven by MAC table lookups
• Source and destination MAC lookups performed for each frame,
based on {VLAN,MAC} pairs
• Source MAC lookup drives new learns and refreshes aging timers
• Destination MAC lookup returns destination port

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 MAC Table Lookup F3 module used as example

Ingress frame
From Ingress Parser header
Extract VLAN and
MAC address HDR Decision Engine
(VLAN,MAC) pair fed
into two separate hash
functions
L2 Lookup
MAC Table
V100,MAC D
(2*16*2048 = 64K entries)

MAC Table MAC Table

Hash #1 Hash #2
Bank 1 Bank 2
Compare (VLAN,MAC)
17 V10,MAC B 16 pages pair to selected row on
each page of each bank
18 V10,MAC B
19 V10,MAC B
20 V10,MAC B
1 V10,MAC B 21 V10,MAC B
2 V10,MAC B 22 V10,MAC B
3 V10,MAC B 23 V10,MAC B
4 V10,MAC B 24 V10,MAC B Matching entry returns
5 V10,MAC B 25 V10,MAC B destination index
6 V10,MAC B 26 V10,MAC B
7 V10,MAC B 27 V10,MAC B
8 V10,MAC B 28V10,MAC B
V10,MAC B
29

2048 rows
9 V10,MACV10,MAC
B A V30,MAC
V10,MAC
30 B C
10 V10,MAC B V10,MAC B
31
11 V10,MAC B To Ingress
Hash result #1 12 V68,MAC
V10,MAC B B V100,MAC D
V100,MAC
32
Lookup Pipeline
MATCH!
selects row on each 13 V10,MAC B
page of Bank 1 14 V10,MAC B V22,MAC E
15 V10,MAC B
16 V100,MAC D Hash result #2 V621,MAC F
selects row on each
page of Bank 2
#CLUS BRKARC-3470
BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
76
Layer 3 Hardware Forwarding
• Layer 3 forwarding driven by FIB table lookups
• Forwarding tables built by control plane on supervisor
engine
• OSPF, EIGRP, IS-IS, BGP, statics, etc.
• Tables downloaded to forwarding engine hardware for
data-plane forwarding
• FIB TCAM lookup based on longest-match destination
IP prefix lookup
• FIB match returns rewrite (next-hop) information in
adjacency table
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 IP FIB TCAM Lookup
Flow data from packet
header fed into load-
Generate lookup key based on sharing hash function
destination IP and compare to
From Ingress FIB TCAM entries
IP FIB Lookup
HDR
Lookup Pipeline Forwarding
Modulo of hash result
Engine
and # next-hops selects
10.1.1.10 SIP + DIP + exact ADJ entry

SPort + DPort
Ingress unicast IP 10.1.1.2 Adj Index, # next-hops Next-hop 1 (IF, MAC)
packet header Load-Sharing
10.1.1.3 Adj Index, # next-hops Hash Next-hop 2 (IF, MAC)

10.1.1.4 Adj Index, # next-hops

10.10.0.10 Adj Index, # next-hops

10.10.0.100 Adj Index, # next-hops Hash Next-hop 3 (IF, MAC)

Result Offset Return lookup
Mask out “don’t
10.10.0.33 Adj Index, # next-hops
mod result
care” bits while
comparing key
10.1.1.xx
10.1.2.xx Adj Index, # next-hops

10.1.3.xx Adj Index, # next-hops Next-hop 4 (IF, MAC)

# next-
10.10.100.xx Adj Index, # next-hops
hops
Next-hop 5 (IF, MAC)
To Ingress
HIT! 10.1.1.xx Adj Index, # next-hops Next-hop 6 (IF, MAC) Lookup
Adj Index Pipeline
10.100.1.xx Adj Index, # next-hops Next-hop 7 (IF, MAC) Result
Hit in FIB
returns result in 10.10.0.xx Adj Index, # next-hops
FIB DRAM
10.100.1.xx Adj Index, # next-hops
Adjacency index
identifies ADJ
block to use
FIB TCAM FIB DRAM Adjacency Table
#CLUS BRKARC-3470
BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
79
 Security ACL

CL TCAM Lookup – ACL ip access-list

permit ip any
example
host 10.1.2.100
Generate lookup key
From Ingress/Egress based on packet fields Decision Engine deny ip any host 10.1.68.44
Lookup Pipeline and compare to CL deny ip any host 10.33.2.25
Packet header TCAM entries
HDR deny ip any host 10.24.77.7
deny ip any host 10.24.77.1
Mask out “don’t permit tcp any any eq 22
care” bits while Classification deny tcp any any eq 23
comparing key
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 deny udp any any eq 514
permit tcp any any eq 80
xxxxxxx
xxxxxxx
| 10.1.2.100
| 10.2.2.2 || xx
xx|| xxx
xxx|| xx
xxx Permit
permit udp any any eq 161
xxxxxxx | 10.1.68.44 | xx | xxx | xxx Hit in CL TCAM
Deny
ACEs from returns result in
xxxxxxx | 10.33.2.25 | xx | xxx | xxx CL SRAM
Deny Actions from
security ACL
security ACL
(x = don’t care)
xxxxxxx | 10.24.77.7| xx | xxx | xxx Deny
xxxxxxx | 10.24.77.1 | xx | xxx | xxx Deny
Return lookup result,
affecting final packet
xxxxxxx
xxxxxxx || xxxxxxx
xxxxxxx || tcp
tcp || xxx
xxx || 80
22 Permit handling (forward or drop)

xxxxxxx | xxxxxxx | tcp | xxx | 23 Deny

xxxxxxx | xxxxxxx | udp | xxx | 514 Deny To Ingress/
HIT!
xxxxxxx | xxxxxxx | tcp | xxx | 80 Permit Egress Lookup
Pipeline
xxxxxxx | xxxxxxx | udp | xxx | 161 Permit

CL TCAM CL SRAM

Fields to match:
src IP | dst IP | protocol | src port | dst port

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 QOS Policy ACLs

CL TCAM Lookup – QOS ip access-list

permit ip any
police
10.3.3.0/24
Generate lookup key
From Ingress/Egress based on packet fields Decision Engine permit ip any 10.4.12.0/24
Lookup Pipeline and compare to CL permit ip any 10.10.1.0/24
Packet header TCAM entries
HDR permit ip any 10.20.2.0/24
ip access-list remark-dscp-32
permit udp 10.0.1.0/24 any
Mask out “don’t
care” bits while Classification permit udp 10.1.1.0/24 any
comparing key
10.1.1.1 | 10.2.2.2 | tcp | 33992 | 80 ip access-list remark-dscp-40
permit tcp 10.0.1.0/24 any
xxxxxxx
xxxxxxx| |10.3.3.xx
10.2.2.xx| |xx
xx| |xxx
xxx| |xxx
xx Policer ID 1 permit tcp 10.1.1.0/24 any
xxxxxxx | 10.4.12.xx | xx | xxx | xxx Policer ID 1 ip access-list remark-prec-3
Hit in CL TCAM
ACEs from QOS returns result in permit tcp any 10.2.3.0/24 eq 23
xxxxxxx | 10.10.1.xx | xx | xxx | xxx CL SRAM Policer ID 1
policy ACLs permit tcp any 10.5.5.0/24 eq 23
(x = don’t care)
xxxxxxx | 10.20.2.xx | xx | xxx | xxx Policer ID 1
10.0.1.xx
10.1.1.xx||xxxxxxx
xxxxxxx ||udp
tcp | xxx | xxx Remark DSCP 32 Actions from
QOS policy
10.1.1.xx | xxxxxxx | udp | xxx | xxx Remark DSCP 32
10.0.1.xx | xxxxxxx | tcp | xxx | xxx Remark DSCP 40
HIT! To Ingress/
10.1.1.xx | xxxxxxx | tcp | xxx | xxx Remark DSCP 40 Egress Lookup
Pipeline
xxxxxxx | 10.2.3.xx | tcp | xxx | 23 Remark IP Prec 3
xxxxxxx | 10.5.5.xx | tcp | xxx | 23 Remark IP Prec 3 Return lookup result,
affecting final packet
handling (police or remark)
CL TCAM CL SRAM

Fields to match:
src IP | dst IP | protocol | src port | dst port

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 LC

F3/M3/F4 Sampled Netflow

• Hardware-based sampling with software-based Netflow cache in
FSA
• Classification lookup selects Netflow sampler-table entry
• Sampler table defines which sampler to use (defines M:N)
• Copy of randomly sampled packets sent to FSA via module inband
• Sampled copies rate-limited to 50K packets per second to avoid
over-running FSA

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 Sup CPU transmits NDE
packets either via via mgmt0
NDE on F3/M3/F4 Modules mgmt0 or via Inband to
collector(s)

Supervisor Engine
Fabric Modules Sup
LC CPU builds NDE via Supervisor Inband CPU
packets (IP+UDP+NDE)
and sends them to Sup Fabric
via EOBC

EOBC
LC CPU builds DRAM DRAM DRAM
software Netflow FSA FSA FSA
cache based on NF Fabric NF Fabric NF Fabric
samples and Table CPU Table CPU Table CPU
periodically ages
out entries
Data-plane traffic
Module Module Module
traverses decision
Packets marked for Inband engine on each SOC Inband Inband
sampling copied to
LC inband Decision Engine Ingress Decision Engine Ingress Decision Engine Ingress
Buffer Buffer Buffer
Rate Limiting Rate Limiting Rate Limiting
Samples subjected
to HW rate limiter

Sampler Table Sampler Table Sampler Table

Sampler marks M:N
Hardware Hardware Hardware
random packets to Forwarding Forwarding Forwarding
sample
Classification Classification Classification
(ACL/QOS/SNF) (ACL/QOS/SNF) (ACL/QOS/SNF)
Classification block
in Decision Engine
selects sampler SOC SOC SOC

M3 Module M3 Module F3 Module

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 F3/M3/F4 System Architecture Supervisor Engine N7700 10G F3 module used as example

Central Arbiter

Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4 Fabric Module 5 Fabric Module 6

Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC

Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC

Decision Engine
q1 MAC
q2 Table
q3 Layer 2 Lookups
q4 Layer 3 Lookups FIB
e2/9… Classification for TCAM
e1/25 ACL / QOS / SNF SP
CL DWRR
Virtual TCAM
Queuing e2/9
Ingress
Buffer Ingress Parser Egress Buffer
F3 SOC 1
F3 SOC 2
F3 SOC 3

F3 SOC 5
F3 SOC 6

F3 SOC 1

F3 SOC 3
F3 SOC 4
F3 SOC 5
F3 SOC 6
F3 SOC 4

F3 SOC 2
Port Logic Port Logic

Module 1 Module 2
e1/25 e2/9
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 Return buffer credit

F3/M3/F4 Packet Flow Buffer credit

granted Supervisor Engine
(destination port +
priority)
N7700 10G F3 module used as example

Credit Central Arbiter

Transmit
Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4 Fabric Module 5 Fabric Module 6
to fabric
Request buffer Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC
credit for
destination port
+ priority

Fabric ASIC Fabric ASIC Fabric ASIC Fabric ASIC

Final lookup result:
destination port + priority

Req Credit
Receive
Decision Engine from fabric

q1 MAC
q2 Table
Dst+Pri
PKTq3 HDR Layer 2 Lookups Ingress/egress L2 and L3
q4 Layer 3 Lookups FIB lookups, ACL/QOS
e2/9… Classification for TCAM lookups, Netflow sampling
e1/25 ACL / QOS / SNF
Queue packet SP
descriptor in VOQ CL DWRR
(destination port + Virtual TCAM
Buffer on egress
priority) Queuing based on destination e2/9
Ingress Packet headers
port + priority VLAN
Buffer Ingress Parser
PKT HDR sent to DE Egress Buffer translation,
etc.
Payload queued in
ingress buffer based
F3 SOC 1
F3 SOC 2
F3 SOC 3

F3 SOC 5
F3 SOC 6

F3 SOC 1

F3 SOC 3
F3 SOC 4
F3 SOC 5
F3 SOC 6
F3 SOC 4

F3 SOC 2
Schedule for
on COS / DSCP CRC, storm Port Logic transmission
Port Logic
control, VLAN
translation, etc.

Ingress packet
Module 1 Receive packet parsing Module 2
from wire PKT HDR Transmit on
e1/25 wire e2/9
#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Agenda
• Introduction to Nexus 7000 / Nexus 7700
• Chassis Architecture
• Supervisor Engine and I/O Module Architecture
• Fabric Architecture
• Hardware Forwarding
• Packet Walks
• Conclusion

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
 Conclusion
• You should now have a thorough understanding of the
Nexus 7000 / Nexus 7700 switching architecture –
including chassis, supervisors, I/O modules, and fabrics,
as well as forwarding lookups and complete system-level
packet flows…
• Any questions?

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKARC-3470 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Thank you

#CLUS
#CLUS