BRKDCN 2035 PDF

#CLMEL
VXLAN BGP EVPN

Based Multi-Site
Lukas Krattiger – Principal Engineer

BRKDCN-2035
#CLMEL
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKDCN-2035
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Objectives
At the end of the session, the participants should be able to:

 Articulate the value proposition of the new VXLAN Multi-
Site architecture and list several use cases for which it
should be positioned
 Understand the functionalities and specific design
considerations associated to VXLAN Multi-Site
Initial assumption:
 The audience already has a good knowledge of the VXLAN
EVPN technology and its use to deploy modern Data
Centre Fabrics
#CLMEL BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 Data Centre Interconnect (DCI) Evolution
Agenda  VXLAN Multi-Site Introduction
• Functional Components and Use Cases
• HW/SW Support and Scalability Values
• Supported Topologies
 VXLAN Multi-Site Deep Dive
• Border Gateway Deployment Considerations
• Inter-Site BUM Traffic Handling
• Control and Data Planes
• Failure Detection on BGWs
• Connectivity to the External Layer 3 Domain
• Network Services Integration
• Legacy Site Integration
• Configuration Specifics
 Conclusions
Data Centre
Interconnect (DCI)
Evolution
#CLMEL
Overlays Evolve and Spread
DC Local Overlay
End-to-End Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
Single Logical Data Centre
Back Then
VXLAN for Interconnecting Networks
Changing the Paradigm with Overlays
DC Local Overlay
Multi-Site Overlay
SS SS SS SS
S S S S S S S S
L L L L .... L L L L L .... L
Multiple Logical Data Centre
VXLAN Evolves as the Control Plane Evolves!
Before Yesterday
Yet Another Encapsulation
 Flood and Learn (Multicast-based)
 Data-Plane only Yesterday
VXLAN for the Data Centre – Intra-DC
 Control-Plane
 Active VTEP Discovery
Today
 Multicast and Unicast
VXLAN for DCI – Inter-DC
 DCI Ready
 ARP/ND caching/suppress
 Multi-Homing
 Failure Domain Isolation
 Loop Protection
Inter-X Connectivity
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site
EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2
Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P
Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal
DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane
 Single Fabric with End-to-  Multiple Fabrics – Normalised  Multiple Fabrics with
End Encapsulation through Ethernet Integrated DCI (DCI2)
 Build Hierarchy in the  Multiple Fabrics Interconnect  Integrated DCI – Scaling
Underlay – Flatten it in the using DCI (Layer 2 and Layer 3) within and between
Overlay Fabrics
VXLAN Multi-Site
Introduction
#CLMEL
Functional Components
and Use Cases
#CLMEL
VXLAN Multi-Site
Functional Components https://tools.ietf.org/html/draft-sharma-multi-site-evpn
Site-External DCI
(IP Routing and Increased
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site-Internal Fabric
Site 1 (Common VXLAN and Site n
BGP-EVPN Functions)
VXLAN Multi-Site Characteristics
 Multiple Overlay Domains – Interconnected and Controlled

 Multiple Overlay Control-Plane Domains – Interconnected and Controlled
 Multiple Underlay Domains - Isolated
 Multiple Replication Domains for BUM – Interconnected and Controlled
 Multiple VNI Administrative Domains – Phase 2
Underlay Isolation – Overlay Hierarchies
VXLAN Multi-Site
Main Use Cases
Scale-Up Model to Build a

Large Intra-DC Network
Data Centre Interconnect (DCI)
Integration with Legacy Networks

(Coexistence and/or Migration)
VXLAN Multi-Site Inter-Site Network
Underlay Isolation Routing Table
Border Site1: Border Site2:
10.1.1.101 10.2.2.101
10.1.1.102 10.2.2.102
10.1.1.111 10.2.2.222
Multi-Site VIP Multi-Site VIP

10.1.1.111 Site-External DCI 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine
Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP
10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
Site 1 10.1.1.6 Site n 10.2.2.6
10.1.1.7 10.2.2.7
VXLAN Multi-Site
Introducing the Border Gateway
Overlay Multi-Site
Border Gateway (BGW)

- Anycast Cluster -
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine
Any VTEP
Site 1 Site n
Multi-Site – VXLAN Tunnel Adjacencies
BG102# show nve peers
Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1
Overlay
10.1.1.1
Multi-Site
30000 00:12:16
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:23

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
VTEP VTEP VTEP

10.1.1.1 VTEP VTEP
10.1.1.4 VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP
VTEP VTEP VTEP VTEP VTEP VTEP
Leaf1-1# show nve peers

Leaf2-7# show nve peers
Site 1
---------- ----------- ------ ---------- Site n------
---------- ----------- ----------
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:25
nve1 10.1.1.111 30000 00:12:23
HW/SW Support and
Scalability Values
#CLMEL
VXLAN Multi-Site
HW/SW Support
 Minimum hardware and software requirements for Border Gateways

Item Requirement
• Cisco Nexus 9300 EX platform
• Cisco Nexus 9300 FX platform
• Cisco Nexus 9300 FX2 platform
Cisco Nexus Hardware
• Cisco Nexus 9364C platform
• Cisco Nexus 9500 platform with X9700-EX line card
• Cisco Nexus 9500 platform with X9700-FX line card
Cisco Nexus Software Cisco NX-OS Software Release 7.0(3)I7(1) or later
 The hardware and software requirements for the Site-Internal nodes of a

VXLAN BGP EVPN site remain the same as those without the EVPN Multi-Site
BGW
VXLAN Multi-Site
Scalability Values as of 9.2(2) Release
Multi-Site Scale
Number of Sites 10
Number of BGWs per Site 4 (Anycast), 2 (vPC)
VTEPs per Site 256
Border Gateway (BGW) Scale EX/FX/FX2 N9364C

Number of Layer-2 VNI (VLAN) 2,000
Number of Layer-3 VNI (VRF) 1,000
MAC per BGW 90,000 64,000
IPv4 Host Routes per BGW* ~530,000 ~60,000
IPv4 Network Routes per BGW* ~530,000 ~8,000
IPv6 Host Routes per BGW* ~24,000 ~7,000
IPv6 Network Routes per BGW* ~260,000 ~2,000
*The values provided in these tables focus on the scalability of one particular route scale at a time
Supported Topologies
#CLMEL
BGW-to-Cloud
Layer-3
Network
BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
BGWs Between Spine and Super-Spine
Super-Spine Super-Spine
Spine Spine Spine Spine Spine Spine
BGWs on Spine
Super-Spine Super-Spine
Spine Spine Spine Spine Spine Spine Spine Spine Spine Spine
BGWs Back-to-Back
BGW BGW BGW BGW

Site 1 Site 2
Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
VXLAN Multi-Site
Deep Dive
#CLMEL
Border Gateway
Deployment
Considerations
#CLMEL
VXLAN Multi-Site
Border Gateways Deployment Considerations
 Border Gateways used for two main functions: Anycast Border Gateways
Interconnecting each site to the Inter-Site network (for

BGW BGW BGW BGW
• VTEP VTEP VTEP VTEP
East-West traffic flows)

• Connecting each site to the external Layer 3 domain (for
North-South traffic flows)
• May also be used to connect endpoints and/or network
service nodes (FWs, ADCs) Site 1
 Possible deployment models:

VPC Border Gateways
• Anycast Border Gateways BGW BGW
VPC Border Gateways (from ACI release 9.2(1))

VTEP VTEP
 BGW function enablement in the VXLAN EVPN fabric:

• BGWs as leaf nodes
• BGWs as spine nodes (Border-Spines)
Site 1
Anycast Border
Gateways
#CLMEL
VXLAN Multi-Site
Anycast Border Gateway (1)
Anycast Border Gateway
 Up to 4 Border Gateways
 Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1)
VTEP VTEP VTEP VTEP
• Deploying at Spine – 7.0(3)I7(2)
Site 1
VXLAN Multi-Site
 Common Multi-Site Virtual IP (Multi-Site VIP)
Multi-Site VIP
10.1.1.111 across BGWs
BGW BGW BGW BGW

• Multi-Site VIP for communication between
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
the Border Gateways in different Sites
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• Multi-Site VIP for communication between
Border Gateways and Leaf nodes within a
Multi-Site VIP
10.1.1.111
Site
 Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
endpoints (routed only), intra- and inter-
Site
Site 1
VXLAN Multi-Site
Type: 00 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:07 VNI: 30010  Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a set of
Layer-2 VNIs
BGW BGW BGW BGW
VTEP VTEP VTEP VTEP • DF election and assignment is automatic
DF
30010
DF
30011
DF
30012
DF
30099
 Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 00)
BGP EVPN • Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
• Multi-Site Discriminator (Ethernet-Segment:
RR RR 00:00:07)
Spine Spine
• Originators IP Address (PIP): 10.1.1.101

• Layer-2 VNI: 30010
Site 1
VXLAN Multi-Site
External
Connectivity Anycast Border Gateway
Point-to-Point L3 Links
(Physical/Sub-Interfaces)  Single-Homed End-Points only connected with
L3 links
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW BGW BGW
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
• External routers
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• No SVI support on BGW nodes
.1 .1
 Advertised and Reachable through Individual
Primary IP Address (PIP)
(Physical/Sub-Interfaces)
ADC ADC
• Intra-Site: Leaf nodes use PIP to reach the device
ADC ADC
connected to Border Gateways
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102 • Inter-Site: Remote Border Gateways use PIP to
reach the device connected to Border Gateways
VTEP
Site 1
VPC Border Gateways
NXOS Release 9.2(1)
#CLMEL
NXOS Release
9.2(1)
Anycast BGW vs. VPC Border Gateway
Anycast Border Gateway VPC Border Gateway

• Up to 4 BGW • 2 BGW with physical VPC Peer-Link
• Shared Nothing • Small Deployments
• Simple Failure Scenarios • End-Point or Network Services
• Any Deployments Connectivity on BGW
• No End-Point or Network Services • Migration Use-Cases (Brownfield)
Connectivity on BGW • Pseudo-BGW to BGW
• Greenfield Deployments • Classic Ethernet/FabricPath to
VXLAN EVPN
NXOS Release
9.2(1)
Multi-Site Border Gateway – Anycast vs. vPC

• Both Anycast and vPC Border Gateway needs to be configured with a common Multi-Site VIP address and an
individual Primary IP (PIP) address
• vPC Border Gateways share a secondary IP address to be used as vPC virtual IP (vPC VIP)
Anycast BGW vPC BGW

vPC VIP
11.11.11.11
100.100.100.100 100.100.100.100
PIP1 PIP1 PIP1 PIP1

10.1.1.1 10.1.2.1 10.1.1.1 10.1.2.1
….
VTEP VTEP VTEP VTEP
BGW1 BGW4 BGW1 BGW2

Fabric Fabric
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
NXOS Release
VXLAN Multi-Site 9.2(1)
VPC Border Gateway and Transit Traffic

VPC Border Gateway
 Common Multi-Site Virtual IP (Multi-Site VIP)
across BGWs
Multi-Site VIP
10.1.1.111
• Multi-Site VIP for Inter-Site transit communication

BGW BGW
VTEP VTEP
(transit)
PIP-BGW1 PIP-BGW2
10.1.1.101 VPC VIP
10.1.1.121
10.1.1.102 • Common VPC Virtual IP (VPC VIP) across BGWs
• Used by default for communication with external
Multi-Site VIP networks
10.1.1.111
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
 Individual Primary IP (PIP) per BGW
• Used for communication with external networks
when “advertised-pip” is configured
Site 1
NXOS Release
VPC Border Gateway and Locally Attached End-Points

VPC Border Gateway
 Single- or Dual-Homed End-Points
Multi-Site VIP
10.1.1.111 • Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW
• Physical or Virtual Servers
VTEP VTEP
• Anycast Gateway function offered to the
Anycast
endpoints
Anycast
Gateway VPC VIP Gateway
10.1.1.121
 Advertised and Reachable through VPC Virtual

Multi-Site VIP
10.1.1.111 IP Address (VPC VIP)
ADC Baremetal • Intra-Site: Leaf nodes use VPC VIP to reach End-
Points connected to Border Gateways
ADC EP
0000.3010.1102
192.168.10.102
0000.3010.1101
192.168.10.101
• Inter-Site: Remote Border Gateways use VPC VIP
to reach End-Points connected to Border
Gateways
• Traffic potentially traverses VPC Peer-Link
Site 1
NXOS Release
VPC Border Gateway and Designated BUM Forwarder

VPC Border Gateway
 VPC-based Designated Forwarder Election
 Per-Site Designated Forwarder (DF) election
BGW BGW • Using same approach as in VPC
• Best Path to Rendezvous-Point or VPC Primary
VTEP VTEP
DF VPC VIP
10.1.1.121 Node
• Same VPC node is elected DF for all the Layer-2
VNIs
Site 1
NXOS Release
VPC Border Gateways 9.2(1)
Traffic between Locally Connected Endpoints across Sites

Src Dst
VXLAN
vPC VIP1 vPC VIP2 Original Packet
Header
Inter-Site
Network
vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
Site 1 Site 2
NXOS Release
Traffic between Locally Connected Endpoints and Remote L3Out

Src Dst
VXLAN
Header
Inter-Site
Network
vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
L3
Site 1 Site 2
NXOS Release
Traffic between Site Connected Endpoints across Sites

Src Dst
Multi-Site Multi-Site VXLAN
Original Packet
VIP1 VIP2 Header
Inter-Site
Network
Multi-Site VIP1 Multi-Site VIP2

100.100.100.100 200.200.200.200
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
VTEP VTEP
Site 1 Site 2
NXOS Release
BUM Traffic across Sites

Src Dst
VXLAN
Header
Inter-Site
Network
vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
DF DF
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
BUM Traffic redirected

via vPC peer-link
toward the DF
VTEP VTEP
Site 1 Site 2
Inter-Site BUM Traffic
Handling
#CLMEL
VXLAN Multi-Site
BUM Traffic Forwarding
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
BUM
Site 1 Site n
Baremetal
VXLAN Multi-Site
BUM Replication Modes (Multicast Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Multicast Multicast
Site 1 Site n
VXLAN Multi-Site
BUM Replication Modes (Ingress Replication Only)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Ingress Replication
Site 1 Site n
VXLAN Multi-Site
BUM Replication Modes (Mixed Mode Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW

Spine Spine Spine
Ingress Replication Multicast
Site 1 Site n
VXLAN Multi-Site
BUM Traffic Policing
Overlay Multi-Site
Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP
BGW BGW
Unknown Unicast 0-100% BGW BGW
Multicast 0-100%
Spine Spine Spine
BUM
Site 1 Site n
Baremetal
Control and Data Planes
#CLMEL
VXLAN Multi-Site
Control Plane
#CLMEL
VXLAN Multi-Site
Control Plane Deployment Considerations
 MP-eBGP EVPN only inter-Sites
• Next-hop behavior (VXLAN tunnel termination and reorigination) and loop protection
(as-path attribute)
 Two main options for underlay and overlay control plane deployment
1. I-E-I (Recommended)
• Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP
• Inter-Sites: eBGP for both underlay and overlay CPs
2. E-E-E*
• Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs
 Full mesh of MP-eBGP EVPN adjacencies across sites

• Recommended to deploy a couple of Route-Servers with 3 or more sites
• RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”,
IETF RFC 7947)
• RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite
*For more information on why eBGP for both underlay and overlay CP is not a good idea:
https://learningnetwork.cisco.com/blogs/community_cafe/2017/10/17/the-magic-of-super-spines-and-rfc7938-with-overlays-guest-post
VXLAN Multi-Site
Underlay Control Plane
DC Core
(Layer-3 Unicast)
DCI
…. ….
VTEP VTEP VTEP VTEP
Fabric BGW BGW BGW BGW
Spine
VXLAN EVPN Spine Spine
VXLAN EVPN Spine
Site1 Site2
VXLAN Multi-Site
Overlay Control Plane (L3 Core)
Route Server (eBGP ”Route Reflector”)
RS
DC Core
(Layer-3 Unicast)
DCI
…. ….
VTEP VTEP VTEP VTEP
Spine
VXLAN EVPN
iBGP-EVPN
Spine Spine
VXLAN EVPN
iBGP-EVPN
Spine
RR RR
Site1 Site2
VXLAN Multi-Site
Overlay Control Plane (L3 Core, no RS)
eBGP-EVPN
DC Core
(Layer-3 Unicast)
DCI
…. ….
VTEP VTEP VTEP VTEP
Spine
VXLAN EVPN
iBGP-EVPN
Spine Spine
VXLAN EVPN
iBGP-EVPN
Spine
RR RR
Site1 Site2
VXLAN Multi-Site
Overlay Control Plane
RS
L3VNI: 50001
L3VNI: 50001
Route-Target: DC Core Route-Target: 65502:50001
65501:50001
(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Spine
VXLAN EVPN
iBGP-EVPN
Spine Spine
VXLAN EVPN
iBGP-EVPN
Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
VXLAN Multi-Site
Overlay Control Plane (Site 1)
RS
L3VNI: 50001
L3VNI: 50001
65501:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
VRF VRF
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1
Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32

Tenant1 10.1.1.111
50001, 65501:50001
DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
…. ….
10.1.1.111 10.2.2.222
Spine
VXLAN EVPN Spine
RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
VXLAN Multi-Site
Overlay Control Plane (Site 2)
RS
L3VNI: 50001
L3VNI: 50001
65502:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 VRF 30010, 65502:30010

0000.3010.1101/48 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF
2 Tenant1 30020, 65502:30020
0000.3020.2101/48 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1
DCI
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
…. ….
10.1.1.111 10.2.2.222
Spine
VXLAN EVPN Spine
RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
VXLAN Multi-Site
Overlay Control Plane (DCI)
RS
L3VNI: 50001
L3VNI: 50001
65501:50001
(Layer-3 Unicast)
VRF VRF
Tenant1 Tenant1
DCI
…. ….
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW 2 0000.3010.1101/48BGW30010, 65599:30010 192.168.10.101/32 BGW
50001, 65599:50001 10.1.1.111 BGW
2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222
Spine
VXLAN
2 EVPN Spine30010, 65599:30010
0000.3010.1102/48 192.168.10.102/32 VXLAN
50001, 65599:50001
Spine
EVPN
10.2.2.222
Spine
RR RR
L2VNI: 30010 (VLAN 10)
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
VXLAN Multi-Site Data
Plane
#CLMEL
VXLAN Multi-Site
Overlay Data Plane
Inter-site VXLAN
Data Plane
De-capsulation and DC Core

Re-encapsulation on
BGW (L2 or L3 lookup)
(Layer-3 Unicast) De-capsulation and
Re-encapsulation on
BGW (L2 or L3 lookup)
DCI
…. ….
VTEP Multi-Site VIP1 VTEP VTEP Multi-Site VIP2 VTEP
10.1.1.111 10.2.2.222
Spine
VXLAN EVPN Spine
Intra-site VXLAN
Data Plane
Site1 Site2
Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
Multi-Site Packet Walk
(BUM)
#CLMEL
VXLAN Multi-Site Packet Walk
Layer 2 (BUM) – Site 1
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
Traffic is replicated VTEP 30010 VTEP
intra-Site BGW11 BGW21
2
VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20
DF
1 Host 1 sends a
VTEP 30010 VTEP
BGW12 BGW22
L2 BUM frame
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 1
Bridge

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule

Layer 2 (BUM) – DCI
Bridge
BGW11- BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

PIP Payload

PIP
VTEP
DF
30010 BGW11 replicates traffic VTEP
BGW11
inter-Sites toward BGW BGW21
nodes
VTEP BUM Forward 3 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – DCI
Bridge

PIP Payload

PIP
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP

DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge

Payload
BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP VTEP
30010
Traffic is replicated
BGW11 BGW21 intra-Site
VTEP
4 VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (DF and Split Horizon) – Site 2
Bridge

Payload
BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2
192.168.10.101 192.168.10.102

Bridge
DF
VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP
DF
VTEP 30010 VTEP
BGW12 BGW22
5
Leaf20 sends traffic
to local Host 2
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Multi-Site Packet
Walk (Bridging)
#CLMEL
Layer 2 (Host 1 to Host 2) – Site 1
Bridge

Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
Leaf10 performs L2 lookup

and encapsulates toward VTEP VTEP
local BGW VIP1 address BGW11 BGW21
2
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
1 Host 1 sends traffic

VTEP VTEP
BGW12 BGW22
destined to remote Host 2
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (Host 1 to Host 2) – DCI
Bridge

Payload
BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP
BGW11 performs L2 lookup

VTEP VTEP
and encapsulates toward
BGW11 remote BGW VIP2 address BGW21
VTEP
3 VTEP
VTEP VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Bridge

Payload
BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP
VTEP VTEP
BGW11 BGW21
destination L20 node
VTEP
4 VTEP
VIP1 VIP2
VTEP VTEP
BGW12 BGW22
5
Leaf20 bridges traffic
to local Host 2
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Bridge

Payload
L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP
Leaf20 performs L2 lookup

VTEP VTEP
BGW11 BGW21 local BGW VIP2 address
VTEP
7 VTEP
VIP1 VIP2
VTEP VTEP
BGW12 BGW22
6
Host 2 replies to
remote Host 1
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer 2 (Host 2 to Host 1) – DCI
Bridge

Payload
BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP
BGW21 performs L2
VTEP lookup and encapsulates VTEP
BGW11 toward remote BGW VIP1 BGW21

address
VTEP
8 VTEP
VTEP VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Bridge

Payload
BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP

and encapsulates toward VTEP VTEP
destination L10 node BGW11 BGW21
9
VTEP VTEP
VTEP VTEP
10 Leaf10 bridges traffic
BGW12 BGW22
toward Host 1
Baremetal Baremetal
Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Failure Detection on
BGWs
#CLMEL
Anycast BGWs
#CLMEL
VXLAN Multi-Site
Failure Detection on BGWs – Fabric Isolation
 The Site-Internal interfaces on BGW nodes are

constantly tracked to determine their status
Site-External
Multi-Site VIP
(‘evpn multisite fabric-tracking’ command)
10.111.111.1
BGW BGW BGW BGW

VTEP VTEP VTEP VTEP
PIP-BGW2 PIP-BGW3 PIP-BGW4

10.200.200.22 10.200.200.23 10.200.200.24
Site-Internal
Spine Spine
Site 1
VXLAN Multi-Site

Site-External
Multi-Site VIP
10.111.111.1
 If all the Site-Internal interfaces are detected as
BGW BGW BGW BGW down:
The isolated BGW stops advertising PIP/VIP
VTEP VTEP VTEP VTEP
1.
PIP-BGW2 PIP-BGW3 PIP-BGW4 addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
2. The remaining BGWs perform new DF elections for
Site-Internal
the L2VNIs owned by the isolated BGW

Spine Spine
Site 1
VXLAN Multi-Site

Site-External
Multi-Site VIP
10.111.111.1
BGW BGW BGW BGW down:
The isolated BGW stops advertising PIP/VIP
VTEP VTEP VTEP VTEP
1.
PIP-BGW2 PIP-BGW3 PIP-BGW4 addresses toward the Site-External network
10.200.200.22 10.200.200.23 10.200.200.24
2. The remaining BGWs perform new DF elections for
Site-Internal
the L2VNIs owned by the isolated BGW

Spine Spine
 As a result, the BGW becomes isolated from
both the Site-Internal and Site-External
Site 1 networks
 Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
VXLAN Multi-Site
Failure Detection on BGWs – DCI Isolation
DC Core
(Layer-3 Unicast)  The Site-External interfaces on BGW nodes are
also tracked to determine their status (‘evpn
Site-External
multisite dci-tracking’ command)

 If all the Site-External interfaces are detected as
BGW BGW BGW BGW down, the isolated BGW node:
Stops advertising VIP VTEP address toward the
VTEP VTEP VTEP VTEP
1.
PIP-BGW1 PIP-BGW2 PIP-BGW3 PIP-BGW4
Site-Internal network
10.200.200.21 10.200.200.22 10.200.200.23 10.200.200.24 2. Withdraws BGP EVPN Type-4 advertisements
Site-Internal
Multi-Site VIP (triggering a new DF election between other BGWs)

10.111.111.1
3. Starts functioning as a regular VTEP (PIP still up)
 As a result, the BGW continues to operate as a

Site 1 Site-Internal VTEP
 Seamless BGW node reinsertion using a “delay-
restore” timer for the VIP address
vPC BGWs
#CLMEL
NXOS Release
Failure Detection on vPC BGWs – Fabric Isolation

Site-External
Multi-Site VIP (‘evpn multisite fabric-tracking’ command)

10.111.111.1
VPC VIP
10.1.1.121
BGW BGW
VTEP VTEP
PIP-BGW1 PIP-BGW2
10.1.1.101 10.1.1.102
Site-Internal
Spine Spine
Site 1
NXOS Release

Site-External

10.111.111.1
VPC VIP  If all the Site-Internal interfaces are detected as
BGW
10.1.1.121
BGW down:
The isolated BGW keeps advertising PIP/VIP
VTEP VTEP
PIP-BGW1 PIP-BGW2 •
10.1.1.101 10.1.1.102
addresses toward the Site-Internal and Site-
External network
Site-Internal
Spine Spine
Site 1
NXOS Release

Site-External

10.111.111.1
BGW
10.1.1.121
BGW down:
The isolated BGW keeps advertising PIP/VIP
VTEP VTEP
10.1.1.101 10.1.1.102
addresses toward the Site-Internal and Site-
External network
Site-Internal
 As a result, the VPC Peer-Link will be used for

Spine Spine Site-External to Site-Internal communication
Site 1
NXOS Release

Site-External

10.111.111.1
BGW
10.1.1.121
BGW down:
The isolated BGW keeps advertising PIP/Multi-Site
VTEP VTEP
10.1.1.101 10.1.1.102
VIP/VPC VIP addresses toward the Site-Internal
and Site-External network (via VPC Peer-Link)
Site-Internal
 As a result, the VPC Peer-Link will be used for

ADC
Spine Spine
Baremetal
Site-External to Site-Internal communication
ADC EP
0000.3010.1101
 Locally attached Single or Dual-Connected End-
0000.3010.1102 Site 1
192.168.10.102 192.168.10.101 Points stay reachable from Site-External and
from Site-Internal via VPC Peer-Link
NXOS Release
Failure Detection on vPC BGWs – DCI Isolation
DC Core
Site-External
VPC VIP  If all the Site-External interfaces are detected as

BGW
10.1.1.121
BGW down:
The isolated BGW keeps advertising PIP/VPC VIP
VTEP VTEP
addresses toward the Site-External and Site-
10.1.1.101 10.1.1.102
Internal network (for External Connectivity and Local

Site-Internal
Multi-Site VIP
10.111.111.1 Hosts)
• The isolated BGW stops advertising the Multi-Site
VIP address toward the Site-Internal network and
Site-External (via VPC Peer-Link)
Site 1  Seamless BGW node reinsertion by re-
advertising via VPC Peer-Link (ensure eBGP
multi-hop peering)
NXOS Release
Failure Detection on vPC BGWs – DCI Isolation
DC Core
Site-External
VPC VIP  If all the Site-External interfaces are detected as

BGW
10.1.1.121
BGW down:
The isolated BGW keeps advertising PIP/VPC VIP
VTEP VTEP
10.1.1.101 10.1.1.102
Internal network (for External Connectivity and Local

Site-Internal
Multi-Site VIP
10.111.111.1 Hosts)
• The isolated BGW stops advertising the Multi-Site
ADC Baremetal
ADC EP
Single or Dual-Connected End-Points stay

0000.3010.1102 Site 1 0000.3010.1101
192.168.10.102 192.168.10.101 
reachable from Site-External and from Site-
Internal via VPC Peer-Link
NXOS Release
Failure Detection on vPC BGWs – ZigZag Isolation
DC Core
(Layer-3 Unicast)  If all the Site-External interfaces are detected as
down on BGW1 (Leaf-Mode):
Site-External
Multi-Site VIP • The isolated BGW keeps advertising PIP/VPC VIP

10.111.111.1 addresses toward the Site-External and Site-
VPC VIP Internal network (for External Connectivity and Local
BGW
10.1.1.121
BGW Hosts)
VTEP
PIP-BGW1
VTEP
PIP-BGW2 • The isolated BGW stops advertising the Multi-Site
10.1.1.101 10.1.1.102
Site-Internal
Multi-Site VIP
10.111.111.1
Spine Spine down on BGW2:
• The isolated BGW keeps advertising PIP/VPC VIP
Site 1 Internal network (via VPC Peer-Link)
NXOS Release
Failure Detection on vPC BGWs – ZigZag Isolation
DC Core
(Layer-3 Unicast)  If all the Site-External interfaces are detected as
down on BGW1 (Leaf-Mode):
Site-External
Multi-Site VIP • The isolated BGW keeps advertising PIP/VPC VIP

10.111.111.1 addresses toward the Site-External and Site-
VPC VIP Internal network (for External Connectivity and Local
BGW
10.1.1.121
BGW Hosts)
VTEP
PIP-BGW1
VTEP
PIP-BGW2 • The isolated BGW stops advertising the Multi-Site
10.1.1.101 10.1.1.102
Site-Internal
Multi-Site VIP
10.111.111.1
ADC
Spine Spine
Baremetal
down on BGW2:
• The isolated BGW keeps advertising PIP/VPC VIP
ADC EP
0000.3010.1101
0000.3010.1102 Site 1 Internal network (via VPC Peer-Link)
192.168.10.102 192.168.10.101
 Single or Dual-Connected End-Points stay

reachable from Site-External and from Site-
Internal via VPC Peer-Link
Connectivity to the
External Layer 3 Domain
#CLMEL
VXLAN Multi-Site
Connectivity to the External Layer 3 Domain
 The BGW nodes can also be used to provide Layer-3 external

connectivity to each site
 Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared
Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites
 External Layer-3 network may be different from the DCI
network used for inter-site communication
VXLAN Multi-Site
Border Gateways and VRF-Lite to External Routers
Dedicated physical Separate IPv4/IPv6 routing

interfaces / sub- peering for each VRF (IGP
interfaces for each VRF External
VRF-AVRF-B VRF-C or eBGP)  Separate IPv4/IPv6 routing peering for
Connectivity each VRF established with the
Site-External
external routers on dedicated physical

Multi-Site
Overlay
interfaces/sub-interfaces
 Must use separate interfaces for
BGW BGW BGW BGW inter-site communication
No support for VXLAN encapsulated
VTEP VTEP VTEP VTEP

traffic on sub-interfaces
Site-Internal
Site 1
VXLAN Multi-Site
Border Gateway and Shared Border (aka ‘GOLF’)
External router operates like a
traditional VXLAN EVPN VTEP
(Layer 3 only)
 Single MP-BGP EVPN peering
External
VRF-A
VRF-B VRF-C
established with the external routers
Connectivity
Single MP-BGP EVPN routing
to exchange routes for all the VRFs
instance to exchange routes
Routed interface extending
‘underlay’ connectivity to
for all VRFs
 VXLAN Data-Plane between the
the external routers
Multi-Site BGWs and the external routers
Site-External
Overlay
 Same spine uplinks used for all
VXLAN encapsulated traffic (North-
South and East-West)
BGW BGW BGW BGW
VXLAN Data Plane
 Required because of the use of DCI link
VTEP VTEP VTEP VTEP
between BGW and WAN tracking
Edge Router
Site-Internal
 Various northbound hand-off options

depending on specific HW support:
Site 1
VRF-Lite, MPLS-VPN, LISP
Legacy Site Integration
#CLMEL
VXLAN Multi-Site
Legacy Site Integration IR for BUM +
aggregated BUM Pair of vPC BGWs
containment (EX/FX Switches)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine Baremetal ADC
Greenfield Site Legacy Site
 Coexistence and/or migration use cases

• Extend Layer-2 and Layer-3 multi-tenant connectivity across sites
 Deploy a pair of vPC BGWs in the legacy site
• Simplified configuration required on vPC BGW nodes
• Still offering native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.)
Multi-Site and Legacy Site Integration
Default Gateway Deployment – Recommended
Default Gateway
migrated to the Border
Greenfield VXLAN Gateways (VXLAN EVPN
EVPN Fabric offers L2 VTEP VTEP L3 VTEP VTEP
Anycast Gateway)
and L3 services for the BGW BGW BGW BGW
stretched IP subnets L2
Legacy
infrastructure offers
only L2 services
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP
L2
Distributed Anycast Greenfield Site Legacy Site

Gateway function
 Recommended approach is to migrate the default gateway from the legacy

aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway)
 Optimise routing between endpoints deployed across sites
Layer-2 Control Plane Exchange across Sites
eBGP-EVPN
MAC NH MAC NH
0000.3010.1101 Leaf1 0000.3010.1101 VIP1
0000.3010.1102 VIP2 VIP1 VIP2 0000.3010.1102 Po1

10.1.1.111 vPC Anycast 10.2.2.222
VTEP VTEP
VTEP VTEP VTEP
BGW BGW BGW BGW All End-Points in the

legacy site are learned as
Po1 directly connected to the
BGW
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
Layer-3 Control Plane Exchange across Sites
eBGP-EVPN
IP NH IP L3VNI
192.168.10.101 Leaf1 192.168.10.101 VIP1
192.168.20.101 VIP2 VIP1 VIP2 192.168.20.101 Po1

10.1.1.111 10.2.2.222
VTEP VTEP L3 VTEP VTEP
BGW BGW
L2 BGW BGW All End-Points in the
legacy site are learned as
Po1 directly connected to the
BGWs
Baremetal Baremetal

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.101
VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (1)
Pair of vPC BGWs
Pair of vPC BGWs (EX/FX Switches)
(EX/FX Switches) BGW
VTEP VTEP VTEP VTEP
BGW BGW
Legacy Site 1 Legacy Site 2
 A pair of vPC BGWs inserted in each legacy site to extend Layer-2 and Layer-3
connectivity between sites
• Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …)
 Slowly phase out the legacy networks and replace them with VXLAN EVPN fabrics
Convert the nodes to
full BGWs functions
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
VTEP VTEP VTEP VTEP VTEP VTEP
‘Mixed’ Site 1 ‘Mixed’ Site 2
 Introduce VXLAN EVPN spines and additional VTEPs in each site

 Migrate endpoints between the legacy network and the new VXLAN EVPN fabric
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Greenfield Site 1 Greenfield Site 2
 Decommission the legacy networks and leave only the VXLAN EVPN
fabrics in place
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Greenfield Site 1 Greenfield Site 2
 Move endpoints directly connected to the vPC BGW nodes (if any) to
regular leaf nodes and migrate to the Anycast BGW model
 Anycast BGW is the recommended deployment options
 The migration can be done in a non disruptive way, one node at the time
Conclusions
#CLMEL
VXLAN EVPN – Multi-Site
Multi-Site Core
• Border Gateway (BGW) to Border Gateway (BGW)
reachability required
• Reachability Back-to-Back (full-mesh) or via Layer-3
transport network
• Any Routing Protocol for BG reachability No Underlay Extension
• IPv4 Unicast Transport
VTEP
(Ingress
VTEP
Replication) VTEP VTEP
• BGP full-mesh or Route-Server (eBGP ”Route Reflector”)
for Overlay Control-Plane Multi-Site Border Gateway (BGW):
• Seamless insertion into existing VXLAN EVPN Fabrics
(Border Gateways require Nexus 9x00-EX/-FX) Spine Spine Spine Spine
• Layer-2 and Layer-3 extension to other Sites

• BGP- or VPC-based Border Gateway (BGW) Cluster (up
to 4 nodes when using BGP)
• All Border Gateways (BGW) are representing a common
Anycast VTEP
• Failure containment through Broadcast, Unknown Unicast
and Layer-2 Multicast limiter (off or rate-based)

• Co-Existence with VRF-Lite for External Connectivity
• Core and Fabric link tracking
Site 1 Site n
Multi-Site Advantages – ”The Multiple”
 Multiple Overlay Domains – Interconnected and Controlled

• Scaling and Segregating VXLAN EVPN Networks
 Multiple Overlay Control-Plane Domains – Interconnected and Controlled
• Limited Overlay Control-Plane Update Propagation
 Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
 Multiple Replication Domains for BUM – Interconnected and Controlled
• Individual BUM flooding domain with Traffic control
Inter-X Connectivity
Multi-Pod Multi-Fabric Multi-Site
Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains
Overlay Control Plane Separated Overlay Control-Plane Domains
Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes
Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)
ARP Flood Suppression (DCI) yes yes yes
Unknown Unicast Flood

no yes yes
Suppression (DCI)
Broadcast Suppression/Limit
no yes yes
(DCI)
Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)
Resources
• VXLAN EVPN Multi-Site Design and Deployment White Paper

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-
739942.html
• Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide - Configuring VXLAN
EVPN Multi-Site
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_VXLAN_Configuration_Guide_7x_chapter_01100.html
• Cisco Live Online - VXLAN BGP EVPN based Multi-POD, Multi-Fabric and Multi-Site
- BRKDCN-2035
https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035&showMyInterest=false#/
• Cisco DCNM 11.1(1) - Multi-Site Domain for VXLAN BGP EVPN Fabrics
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/11_1_1/config_guide/lanfabric/b_dcnm_fabric_lan/cont
rol.html#concept_nhz_lfc_yfb
In Summary…
Q&A
#CLMEL
Complete Your Online Session Evaluation
• Give us your feedback and receive a
complimentary Cisco Live 2019 Power
Bank after completing the overall event
evaluation and 5 session evaluations.
• All evaluations can be completed via
the Cisco Live Melbourne Mobile App.
• Don’t forget: Cisco Live sessions will
be available for viewing on demand
after the event at:
https://ciscolive.cisco.com/on-demand-library/
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Thank you
#CLMEL
#CLMEL

BRKDCN 2035 PDF

Uploaded by

Copyright:

Available Formats

BRKDCN 2035 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 2035 PDF

Uploaded by

Copyright:

Available Formats

What are the functional components and use cases of VXLAN Multi-Site?

What are the functional components and use cases of VXLAN Multi-Site?

What are the agenda items covered in the session?

What are the agenda items covered in the session?

#CLMEL

VXLAN BGP EVPN

Lukas Krattiger – Principal Engineer

At the end of the session, the participants should be able to:

EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Overlay Overlay Overlay Overlay Overlay Overlay

Bar Bar Bar Bar

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine Spine Spine Spine Spine

 Multiple Overlay Domains – Interconnected and Controlled

Underlay Isolation – Overlay Hierarchies

Scale-Up Model to Build a

Data Centre Interconnect (DCI)

Integration with Legacy Networks

Multi-Site VIP Multi-Site VIP

BGW BGW BGW BGW

Site 1 Underlay Site n Underlay

10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

Border Gateway (BGW)

BGW BGW BGW BGW

Spine Overlay Site 1

Multi-Site VIP Multi-Site VIP

BGW BGW BGW BGW

Spine Overlay Site 1

VTEP VTEP VTEP

Leaf1-1# show nve peers

 Minimum hardware and software requirements for Border Gateways

Cisco Nexus Software Cisco NX-OS Software Release 7.0(3)I7(1) or later

 The hardware and software requirements for the Site-Internal nodes of a

Border Gateway (BGW) Scale EX/FX/FX2 N9364C

BGW BGW BGW BGW

Interconnecting each site to the Inter-Site network (for

• VTEP VTEP VTEP VTEP

East-West traffic flows)

 Possible deployment models:

VPC Border Gateways (from ACI release 9.2(1))

 BGW function enablement in the VXLAN EVPN fabric:

BGW BGW BGW BGW

• Originators IP Address (PIP): 10.1.1.101

Anycast BGW vs. VPC Border Gateway

Anycast Border Gateway VPC Border Gateway

Multi-Site Border Gateway – Anycast vs. vPC

Anycast BGW vPC BGW

PIP1 PIP1 PIP1 PIP1

BGW1 BGW4 BGW1 BGW2

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

VPC Border Gateway and Transit Traffic

• Multi-Site VIP for Inter-Site transit communication

VPC Border Gateway and Locally Attached End-Points

 Advertised and Reachable through VPC Virtual

VPC Border Gateway and Designated BUM Forwarder

Traffic between Locally Connected Endpoints across Sites

vPC VIP1 vPC VIP2