IOS XR Platform Hardware

Architectures
LJ Wobker, Principal Engineer
Lane Wigley, Technical Marketing
BRKSPG-2404
Agenda/Abstract
Introduction building a forwarding path
Platform Design & Building Blocks
IOS XR Platforms
Hardware
Modern demands such as cloud computing, mobility, and media delivery are driving
Virtualized greater bandwidth and service requirements into provider networks. To address this,
Cisco offers a broad portfolio of systems for provider networks. These systems all run
the IOS XR operating system, but have meaningful differences in how they are
designed from a system/hardware standpoint. This presentation discusses system
architectures for the NCS-6000, ASR-9000, NCS-5500, and other IOS XR platforms
with a specific focus on the high-level design decisions. A significant part of the talk is
around silicon selection (custom vs. merchant, for example) and the mechanical and
logical design of linecards for these larger systems.
Should I be here?
Todays topics and not-topics.

Yes!
Hardware architectures
No!
System-level design (or at least not much)
Data planes
Routing protocols / design
Packet forwarding
Network level designs
High-speed, complex,
Control plane / OS infrastructure
(expensive!) systems
Selling a specific product

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
You cant always get what you want
But if you try sometimes well you just might find you get what you need

Nothing is free
Some things are closer to free

The further from the middle,

the more things cost

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
The IOS XR Router Family

CRS

NCS 5000 NCS 5500 NCS 6000 ASR 9000

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Whats needed to build a forwarding path?
1. Optical to electrical
2. Transport a signal from optics to NPU
3. Ingress forwarding operations
4. Transport a signal from ingress NPU to egress NPU (fabric)

5. Egress forwarding operations

4
6. NPU to optics
3/5
7. Electrical to optical 2/6

1/7

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Logical view of forwarding path components
LASERS / RECEIVERS FAST MEMORY SILICON

Tables
Optics SERDES
TCAM MEMORY TCAM TM
Fabric
NPU Interface
FAST MEMORY
ASIC
Packet
SILICON Fabric
Optics Buffers
nPower X1

Optics nPower X1 FIA

Optics nPower X1 FIA

Optics nPower X1 FIA

Optics nPower X1 FIA

SERDES Ethernet Control

DRAM CPU Switch
Ethernet

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 Router
Hardware
Liquid cooling

Junction temp
Cooling
Acoustics

Building
Heat Sinks
Airflow

NEBS
Filters

Routing Slot pitch

Materials Cable Management
Busbar N:N
Power Blocks

IPoDWDM

Ops/sec
Chassis
Signal Integrity

PCB
SERDES

QSFP28

Capacity
MPO
Backplane Interposer
Optics Bandwidth

Die size
Fabric
Connectors

Process FIB size

On-chip
Stats
CPAK LR4
PPS
HMC
Silicon

Serial
Features Memory SRAM

DDR3
HBM

LLDRAM
Programmability
Pipeline

Fast Convergence Buffering TCAM

Run to completion
ACL scale GDDR5

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Challenge: Scale routers faster than components
Note: exponential scale

4096x
1024x
256x Moores Law ?
64x
?
16x
?
4x
1x
?

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Technology Trends POS/Ethernet Standards
Fastest Interface Bandwidth

4096x
1024x
400G
256x
100G
64x Interface Speeds 40G
Moores Law
16x 10G

4x OC-48
GE
1x OC-12

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Technology Trends POS/Ethernet Standards
Fastest Interface & Router Bandwidth
58T
4096x Buffered Router Bandwidth
24T
16T
1024x 8T
400G
256x 2.4T
100G
64x 640G 40G
Interface Speeds Moores Law
16x 10G

4x OC-48
GE 28G
1x 8G
OC-12

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Technology Trends - Optics
Bandwidth/Volume
Driven by physics, not Moores Law, costs shifting
SerDes dependency (NPU interface)
4096x Cooling challenge 30C lower max temp vs. ASICs

1024x
256x
64x
16x
4x
100G CPAK
1x GBIC 10G XFP 10G SFP+ 100G CFP 40G QSFP+ 100G QSFP28

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Silicon & Modular Router PPS Per NPU/LC
*Mpps for Standalone and Modular/Buffered
*represents a combination of memory, Silicon & SerDes
4096x
1024x 3200
256x 1200
800 700x6
64x 280x5 NCS 5500
125 NCS 6000 150x4
16x 75
CRS-3 ASR 9000
CRS-1 45x4
4x 14x4
ASR 9000 SoC no buffer
16
4 ASR 9000 Modular buffered
1x CPU 12000

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Technology Trends Memory
Commodity & Custom Bandwidth

4096x Scaling FIB above ~256K IPv4

usec vs. msec buffering NCS 6000
1024x Operations / second critical for FIB 325
HMC 160
256x HBM
NCS 6000 100
64x 55
GDDR5 28
25
16x DDR3
DDR4
DDR2 10
4x 5 ASR 9000
DDR
SDR 2nd & 3rd gen
1x 0.8 1.6

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Technology Trends SerDes
High-performance Electrical Link (Speed in GHz)

4096x
Optics to NPU
1024x NPU to fabric
NPU to TCAM
256x NPU to serial memory

64x
16x 25
15
5 11.5
4x
2.5
1x 1.25 1.25

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Technology Trends Compared

4096x
1024x
256x
Interfaces
64x PPS

16x Memory
Optics
4x SerDes

1x

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Modular Buffered Router Bandwidth
Systems growing much faster than components exponential scale
58T
4096x 24T
16T
1024x 8T

256x 2.4T

640G Interfaces
64x PPS

16x Memory
Optics
4x SerDes
28G
1x 8G

1998 2000 2002 2004 2006 2008 2010 2012 2014 2016

*similar trends for conductive materials, connectors, fans, power supplies, CPUs,
BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Platform Design
Why not one platform?
Requirements have a butterfly effect
An individual requirement may dramatically impact architecture
Every component ahead of the curve adds complexity

Key Drivers
Time / Cost
Buffering
FIB scale
System Scale
Features/Flexibility

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Time Terabit LCs in 2013 vs. 2016
NCS 6000 1T 200G NPU w/ full FIB, deep buffers, programmable
Significant investment in nPower X1 silicon development
Custom memory for FIB and buffering extremely high bandwidth and ops/sec
$271M acquisition of Lightwire for small, low-power 100G optics

ASR 9000 1.2T similar speed, flexibility, and power 3 years later
Commercial NPU, customized for Cisco, commodity memories
Cisco fabric, backward compatible with earlier generations
QSFP28 optics, full range of chassis sizes

NCS 5500 3.6/2.4T lighter silicon architecture

Commodity memory, partial bandwidth buffering, VoQ
TCAM and non-TCAM options (density change)
Reduced counters, QoS, queueing complexity

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 Ports
Buffering

Ports

Ports
On-chip limited to microseconds NPU
But Off-chip requires
FIB

ASIC I/O pins fewer interfaces Ports

Board space and power
Commodity bandwidth is limited
Many devices may be needed
Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write)
Stalled High-end graphics and networking moving to custom
Ports
Custom memories
Fewer devices are needed to reach bandwidth, fewer pins
Development and per-unit costs Extremely expensive

Ports

Ports
NPU
FIB

Buffers

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
FIB Scale FIB

On-chip FIB limited to ~256K LPM entries

Ports

Ports
Usually an on-chip TCAM or lookup tree NPU
Can be scaled up to ~1M with well-known prefixes
Combination of flat tables and LPM
Buffers

Larger FIB tables require external memory:

ASIC I/O pins redirected from interfaces to memory
4-8 reads for every packet high operations per second
Board space (density) and power

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
System Scale FIB

Four common approaches

Fabric
Fixed single NPU / Forwarding ASIC

Ports
NPU
All resources dedicated to network interfaces
May or may not have external memories
Buffers
Fixed multiple NPUs
Connected via mesh or fabric chips

Modular Expand with line cards

Resources for fabric and usually external memories

Multi-chassis
Adds fiber connections to fabric cards (more power & board space)
Increases software complexity

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 Features and Flexibility

Packet Processing Engine (PPE)

C programmable
Run to completion
Anything is possible, no pure optimization
CRS PPES
Wide range of pipeline programmability
NCS 5000 < NCS 5500 < ASR 9000 < NCS6K

Packet rate is strongly correlated to Ingress Egress

Network Interface
cost, power, and flexibility Port
Parser
Link VLAN
Tunnel
Service Fwd PMF
FEC FEC
Term Layer Trans Res Res
Term

Counting stuff is much harder than

most people realize
NCS 5500 PIPELINE

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
IOS XR Platforms
The IOS XR Router Family

CRS

NCS 5000 NCS 5500 NCS 6000 ASR 9000

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
NCS 6000 First Dense 100G
Highest router bandwidth capacity
128 Tbps first generation multi-chassis (2013)
Only Tbps card on the market in 2013-2015
With buffers and full FIB
Dramatic improvement in power efficiency
$6000/100G/month power savings over CRS-3 64x100
Merchant fabric
Highly programmable forwarding, massive stats
Custom memories for FIB and buffering
Custom CPAK optics

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
NCS 6000 Line Card Architecture

Slice architecture Optics

Tables

T
Optics, NPU & FIA per slice C T Fabric
Run to Completion NPU w/ PPEs A PPEs M Interface
M
ASIC
1 Generation (so far) Optics
Packet
Buffers

1T line cards w/ 200G NPUs (40 nm) nPower X1

400G NPUs in lab Optics nPower X1 FIA

Optics nPower X1 FIA
TCAM for ACL/QoS scale Optics nPower X1 FIA
Optics nPower X1 FIA
Deep buffers (50+ msec)
DRAM CPU

10X 100GE

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
ASR 9000 Series

Full range of chassis and interfaces

40G, 200G, and 1T generations

Highly programmable forwarding

Partnered for a customized NPU cisco software
Highest scale, feature, and QoS capabilities

Cisco fabric
Commodity memories for FIB and buffering
Mostly commodity optics

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
ASR 9000 System Design
Backplane or midplane
Flexible switch fabric options
RP and fabric may be integrated into RSP
Variable # of fabrics for increased capacity and redundancy
First stage of fabric on line card

Multiple airflow designs

Front-to-back 9922, 9912, 9010
Side-to-back 9006
Side-to-side 9001, 9004

Modular options for lower-speed and legacy interfaces

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
ASR 9000 Line Card Architecture
FIB
Slice architecture TCAM Packet Buffers

CPAK
Optics, NPU & FIA per slice NP-5c FIA
CPAK
Flexible pipeline NPU
1st stage of fabric on line card CPAK
NP-5c FIA

Fabric
CPAK
3 Generations (so far)
CPAK
40G line cards w/ 15G NPUs (90 nm) NP-5c FIA
200-360G line cards w/ 60G NPUs (55 nm) CPAK
800G-1.2T line cards with 240G NPUs (28 nm) CPAK
NP-5c FIA
CPAK
TCAM for ACL/QoS scale
DRAM CPU
Huge FIB (5M+) & buffers (200 msec)
8X 100GE

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
ASR 9000 Modular Line Card Architecture
10G SFP+ 10G SFP+
10G SFP+
FIB TCAM Packet Buffers
10G SFP+
10G SFP+
10G SFP+ 10G SFP+
10G SFP+
Bay
PHY

PHY

PHY

PHY
10G SFP+ 10G SFP+ 10G SFP+ 10G SFP+
10G SFP+

10G SFP+
10G SFP+

10G SFP+
10G SFP+ 10G SFP+
10G SFP+ 0 NP-5c FIA
10G SFP+
10G SFP+

Fabric
10G SFP+
10G SFP+ 10G SFP+

2/4/8X 10GE 20X 10G FIB TCAM Packet Buffers

1G SFP
Bay
1G SFP
1G SFP 40G 100G
1 NP-5c FIA
1G SFP QSFP+ CPAK
PHY

PHY
PHY

PHY
PHY
1G SFP 40G 100G
1G SFP QSFP+ CPAK
1G SFP 40G 100G
1G SFP QSFP+ CPAK DRAM CPU
1G SFP
1G SFP

20X 1G 1/2X 40G 1/2X100G MODULAR 3RD GENERATION LC

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
NCS 5500 3rd generation 100G
Dense 100GE with deep buffers
8 & 16-slot modular, 1 & 2 RU fixed
Up to 57.6 Tbps in rack in 2016
2.4T - 3.6T line cards
Options for route and ACL scale via TCAM
Highly integrated Silicon
Single ASIC for forwarding & fabric interface
Dramatic power reduction to 0.24 W/Gbps
QSFP28 optics
GDDR5 commodity buffers

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
NCS 5500 System Design AIR INLET

New chassis design for XR

Orthogonal direct connect
Horizontal line cards and vertical fabric
Direct connection between line cards and fabric cards
No midplane
FAN
Distributed air intake between cards REMOVED

Fans cover fabric cards

Provides cool air equally to all optics

Optics require ~30C cooler operation than silicon
Avoid preheating air from module to module

REAR VIEW

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
NCS 5500 Line Card Architecture Buffers

QSFP28
QSFP28
QSFP28 Forwarding
Slice architecture QSFP28 ASIC

Optics & Pipelined Forwarding ASIC per slice integrated FIA QSFP28
QSFP28
Optional TCAM for FIB/ACL scale
Optics x 6 FA
1st Generation
Optics x 6 FA
3.6T line cards FA at 600G
2.9T line cards FA at 720G + TCAM Optics x 6 FA

Deep VoQ buffers (50+ msec) Optics x 6 FA

Optics x 6 FA

DRAM CPU
36X 100GE

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
 TCAM Buffers
Forwarding
NCS 5501/5502 Architecture

CPU
ASIC

NCS 5501

QSFP28
QSFP28
QSFP28

QSFP28

DRAM
SFP+
SFP+
SFP+
SFP+
SFP+

SFP+
1RU Single 800 Gbps Forwarding ASIC
4x 100G + 40x 10G
40x10G 4x100G

NCS 5502 NCS 5501 800G

2RU 8 600Gbps Forwarding ASICs NCS 5502 4.8T

Integrated switch fabric Switch Fabric Switch

48x 100G

TCAM Buffers
Forwarding

FA
FA

FA
FA
FA
FA
FA

CPU
Optional TCAM for scale ASIC

Deep VoQ buffers (50+ msec)

QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP x 6
QSFP28

QSFP28
QSFP28
QSFP28

QSFP28
QSFP28

DRAM
48x100G
2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
NCS 5000 Platforms
Extending IOS XR for satellite, ToR, and beyond

Maximizing Silicon capabilities

No external memories small FIB, usec buffers
NCS 5001
All bandwidth to ports, no fabric

Low cost and power

Full IOS XR Routing
NCS 5002

Forwarding ASIC FIB

32x100
Buffers

NCS 5011

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
CRS-X
Scaling CRS into 100G
4x 100G
40x 100G
2x 100G + 5x 40G
3x 100G + 1x 100G IPoDWDM

Up to 51.2T via multi-chassis

Fully compatible with CRS-1 & CRS-3

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Virtualized IOS XR
IOS XR on x86 hardware with Linux VM & containers
Router creation in seconds rapid service deployment
Runs NCS 6000 software (recompiled) in emulated NPU
Hosted & small PE with L2 & L3 VPNs
Ideal route reflector
64 bit
10+ M routes

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Virtualized IOS XR Performance & QoS
Targeted for 5-50 Gbps forwarding
20+ Gbps with features per socket
3-layer Hierarchical QoS

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
One serious thing before we go...
Im grateful for the opportunity to be here and to talk about something (at least I think) is
interesting. Its important to me personally that attendees find these presentations
helpful and/or valuable. Certainly not everyone will enjoy every presentation -- but you
guys spend a lot of money to come here and Cisco and the speakers do a lot of work to
put on the event and the presentations. If the material is excellent, weve done our
jobs. If theres something we can do better, please let us know... We really do read the
comments. ;-)

Thanks for your time today.

Lawrence J Wobker, Cisco

February 2017
Q&A
 Cisco Spark
Ask Questions, Get Answers, Continue the Experience

Use Cisco Spark to communicate with the Speaker and fellow

participants after the session

Download the Cisco Spark app from iTunes or Google Play

1. Go to the Cisco Live Berlin 2017 Mobile app
2. Find this session
3. Click the Spark button under Speakers in the session description
4. Enter the room, room name = BRKSPG-2404
5. Join the conversation!

The Spark Room will be open for 2 weeks after Cisco Live

Presentation ID 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Complete Your Online Session Evaluation
Please complete your Online
Session Evaluations after each
session
Complete 4 Session Evaluations &
the Overall Conference Evaluation
(available from Thursday) to receive
your Cisco Live T-shirt
All surveys can be completed via
the Cisco Live Mobile App or the
Dont forget: Cisco Live sessions will be available
Communication Stations for viewing on-demand after the event at
CiscoLive.com/Online

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Continue Your Education
Demos in the Cisco campus
Walk-in Self-Paced Labs
Lunch & Learn
Meet the Engineer 1:1 meetings
Related sessions

BRKSPG-2404 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public 46