#CiscoLive

Building Data Center Networks

with VXLAN EVPN Overlays – Segment 1

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Overlay Taxonomy
Underlay

S S S S

L L L L .... L

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Overlay Taxonomy
Underlay

S S S S

Underlay

Edge Device L L L L .... L

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Overlay Taxonomy
Underlay

S S S S
Layer-3
Point-to-Point
Underlay

Edge Device L L L L .... L

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Overlay Taxonomy
Underlay

S S S S
Layer-3
Point-to-Point
Underlay

Edge Device L L L L .... L

LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Workload

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Overlay Taxonomy
Overlay

S S S S

Overlay

Edge Device L L L L .... L

LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Workload

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Overlay Taxonomy
Overlay
VTEP: VXLAN Tunnel End-Point
VNI/VNID: VXLAN Network Identifier

S S S S

Overlay

VTEP VTEP VTEP VTEP VTEP

Edge Device L L L L .... L
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Workload

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Overlay Taxonomy
Overlay
VTEP: VXLAN Tunnel End-Point
VNI/VNID: VXLAN Network Identifier

S S S S
Tunnel Encapsulation (VNI Namespace)

Overlay

VTEP VTEP VTEP VTEP VTEP

Edge Device L L L L .... L
LAN
Segment
Hypervisor Baremetal Hypervisor Hypervisor Baremetal Hypervisor Baremetal Baremetal
Workload

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 Understanding Overlay Technologies
Overlay Services
• Layer-2 Underlay Transport
Tunnel Encapsulation
• Layer-3 Network
• Layer-2 and Layer-3

Data-Plane
Control-Plane
• Overlay Layer-2/Layer-3 Unicast
• Route Learning
• Local Learning
Forwarding
• Remote Learning • Overlay Broadcast, Unknown Unicast,
• Route Distribution Multicast (BUM) Forwarding
• Peer Discovery • Unicast-based (Ingress Replication)
• Multicast-based (PIM)

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Understanding Overlay Technologies

S S S S
Control-Plane
• Route Learning
• Local Learning
• Remote Learning
• Route Distribution
• Peer Discovery VTEP VTEP VTEP VTEP VTEP

L L L L .... L

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Understanding Overlay Technologies

S S S S
Control-Plane
• Route Learning
• Local Learning
• Remote Learning
• Route Distribution
• Peer Discovery VTEP VTEP VTEP VTEP VTEP

L L L L .... L
M1/IP1 Eth1/1 M2/IP2 Eth1/5 M3/IP3 Eth1/12

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Understanding Overlay Technologies

S S S S
Control-Plane
• Route Learning at
e
d
• Local Learning Up
PN
• Remote Learning EV
• Route Distribution
• Peer Discovery VTEP VTEP VTEP VTEP VTEP

L L L L .... L
M1/IP1 Eth1/1 M2/IP2 Eth1/5 M3/IP3 Eth1/12
+M1/IP1 VTEP1 +M1/IP1 VTEP1

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Understanding Overlay Technologies
RR: BGP Route Reflector

RR RR
S S S S
Control-Plane
• Route Learning
• Local Learning
• Remote Learning
• Route Distribution
• Peer Discovery VTEP VTEP VTEP VTEP VTEP

L L L L .... L
M1/IP1 Eth1/1 M2/IP2 Eth1/5 M3/IP3 Eth1/12
+M2/IP2 VTEP2 +M1/IP1 VTEP1 +M1/IP1 VTEP1
+M3/IP3 VTEPn +M3/IP3 VTEPn +M2/IP3 VTEP2

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Understanding Overlay Technologies

S S S S
VTEP Peer Discovered
Control-Plane
• Route Learning
• Local Learning
• Remote Learning
• Route Distribution
• Peer Discovery VTEP VTEP VTEP VTEP VTEP

L L L L .... L
M1/IP1 Eth1/1 M2/IP2 Eth1/5 M3/IP3 Eth1/12
+M2/IP2 VTEP2 +M1/IP1 VTEP1 +M1/IP1 VTEP1
+M3/IP3 VTEPn +M3/IP3 VTEPn +M2/IP3 VTEP2

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Understanding Overlay Technologies

Data-Plane
• Overlay Layer-2/Layer-
S S S S
3 Unicast Forwarding
• Overlay Broadcast,
Unknown Unicast,
Multicast (BUM)
Forwarding
• Unicast-based VTEP VTEP VTEP VTEP VTEP
(Ingress Replication)
• Multicast-based L L L L .... L
(PIM)

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Understanding Overlay Technologies

Data-Plane
• Overlay Layer-2/Layer-
S S S S
3 Unicast Forwarding
• Overlay Broadcast,
Unknown Unicast,
Multicast (BUM)
Forwarding
• Unicast-based VTEP VTEP VTEP VTEP VTEP
(Ingress Replication)
• Multicast-based L L L L .... L
(PIM)

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 Understanding Overlay Technologies

Data-Plane
• Overlay Layer-2/Layer-
S S S S
3 Unicast Forwarding
• Overlay Broadcast,
Unknown Unicast,
Multicast (BUM)
Forwarding
• Unicast-based VTEP VTEP VTEP VTEP VTEP
(Ingress Replication)
• Multicast-based L L L L .... L
(PIM)

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Understanding Overlay Technologies

Data-Plane
• Overlay Layer-2/Layer-
S S S S
3 Unicast Forwarding
• Overlay Broadcast,
Unknown Unicast,
Multicast (BUM)
Forwarding
• Unicast-based VTEP VTEP VTEP VTEP VTEP
(Ingress Replication)
• Multicast-based L L L L .... L
(PIM)

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive
Building Data Center Networks
with VXLAN EVPN Overlays – Segment 2

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 What is it?
VXLAN EVPN
• Standards based Encapsulation • Standards based Control-Plane
• RFC 7348 • RFC 8365 (and RFC 7432)
• Uses UDP-Encapsulation • Uses Multiprotocol BGP
• Transport Independent • Uses Various Data-Planes
• Layer-3 Transport (Underlay) • VXLAN (EVPN-Overlay), MPLS, Provider
• Flexible Namespace Backbone (PBB)

• 24-bit field (VNID) provides ~16M • Many Use-Cases Covered

unique identifier • Bridging, MAC Mobility, First-Hop &
• Allows Segmentations Prefix Routing, Multi-Tenancy (VPN)

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Introducing Ethernet VPN (EVPN)

Provider Backbone
MPLS Overlay (NVO3)
Bridges
(RFC 7432) (RFC 8365)
(RFC 7623)
ata
(i .e. VX LAN) for D
nnels
v er NVO Tu r
• E VPN o
ri c E n c a psulation verla y Service ove
b O
Center Fa er-2 and Layer-3
L a y
• Provides
Network
simple IP

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 I
FY
VXLAN and EVPN related RFCs & Drafts (IETF)
ID Title Category

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane for MPLS

RFC 8365 A Network Virtualization Overlay Solution using EVPN Control Plane for NVO

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane / Data Plane

draft-ietf-bess-evpn-prefix-advertisement IP Prefix Advertisement in EVPN Control Plane / Data Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
EVPN Use-Cases for Data Center Fabrics

VLAN- Asymmetric
Layer-2
Aware IRB

EVPN

Layer-2 & VLAN- Symmetric

IP VRF
Layer-3 Based IRB

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
EVPN Use-Cases for Data Center Fabrics

VLAN- Asymmetric
Layer-2
Aware IRB

EVPN

Layer-2 & VLAN- Symmetric

IP VRF
Layer-3 Based IRB

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
 EVPN Layer-2 Service Interface
Layer-2 & VLAN- Symmetric
EVPN IP VRF
Layer-3 Based IRB

• VLAN-Based Model
• ‘Uses EVPN Route-Type2
• Single Subnet per EVPN Virtual Instance (EVI)
• Called VLAN-Based
VID
• Unique Route Distinguisher (RD)
10 EVI
• Unique Router Target (RT)
• BGP Route-Target constrain mechanism to limit
propagation (import/export
• 1:1 Mapping
Route Target: 65000:30000
• EVI to Broadcast Domain (Bridge Domain)
• Ethertag Must be Zero
• RFC 8365 – Section 5.1.2 - Option 1

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
 VLAN-
EVPN Layer-2 Service Interface Based

S S S S

VNI: 30000 (EVI 2)

L L L L .... L
RT: 65500:30000

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 VLAN-
EVPN Layer-2 Service Interface Based

S S S S
RD: 10.10.10.1:32769
[2]:[0]:[0]:[48]:[M1]:[0]:[0]
L2 VNI: 30000
L2 RT: 65000:30000

VNI: 30000 (EVI 2)

L L L L .... L
RT: 65500:30000

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 VLAN-
EVPN Layer-2 Service Interface Based

S S S S
RD: 10.10.10.1:32769
[2]:[0]:[0]:[48]:[M1]:[0]:[0]
L2 VNI: 30000
L2 RT: 65000:30000

VNI: 30000 (EVI 2)

L L L L .... L
RT: 65500:30000

VLAN2 - M1/IP1 - VTEP1

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 VLAN-
EVPN Layer-2 Service Interface Based

S S S S
RD: 10.10.10.255:32770
[2]:[0]:[0]:[48]:[M3]:[0]:[0]
L2 VNI: 30001
L2 RT: 65000:30001

VNI: 30000 (EVI 2)

L L L L .... L
RT: 65500:30000

VNI: 30001 (EVI 3)

RT: 65500:30001 VLAN2 - M1/IP1 - VTEP1
VLAN3 – M3/IP3 - VTEPn

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 IP-VRF-to-IP-VRF Model in EVPN
Layer-2 & VLAN- IP VRF Symmetric
EVPN
Layer-3 Based (interface-less) IRB

• Interface-Less Model

• Route-Type 5 only

• Next-Hop is remote VTEP

• Two extended communities

• Encapsulation Extended Community
• Router’s MAC Address (remote VTEP)

• IETF Draft “IP Prefix Advertisement in EVPN”

• draft-ietf-bess-evpn-prefix-advertisement – Section 4.4.1

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Interface-
IP-VRF-to-IP-VRF Model in EVPN Less

S S S S

L L L L .... L
VNI: 50000 (IP-VRF A)
RT: 65500:50000

Baremetal Baremetal Baremetal

Net1 Net2 Net3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Interface-
IP-VRF-to-IP-VRF Model in EVPN Less

S S S S
RD: 10.10.10.1:1
[5]:[0]:[0]:[24]:[NET1]
L2 VNI: 50000
L2 RT: 65000:50000
RMAC: MAC-L1

L L L L .... L
VNI: 50000 (IP-VRF A)
RT: 65500:50000

NET1/24 -> L1 (VNI50000)

Baremetal Baremetal Baremetal

Net1 Net2 Net3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Interface-
IP-VRF-to-IP-VRF Model in EVPN Less

S S S S
RD: 10.10.10.2:1
[5]:[0]:[0]:[24]:[NET2]
L2 VNI: 50000
L2 RT: 65000:50000
RMAC: MAC-L2

L L L L .... L
VNI: 50000 (IP-VRF A)
RT: 65500:50000

NET1/24 -> L1 (VNI50000)

NET2/24 -> L2 (VNI50000)

Baremetal Baremetal Baremetal

Net1 Net2 Net3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 Interface-
IP-VRF-to-IP-VRF Model in EVPN Less

S S S S

L L L L .... L
VNI: 50000 (IP-VRF A)
RT: 65500:50000
NET1/24 -> L1 (VNI50000)
NET1/24 -> L1 (VNI50000)
NET2/24 -> L2 (VNI50000)
NET1/24 -> L1 (VNI50000)
NET2/24 -> L2 (VNI50000)
NET1/24 -> L1 (VNI50000)
NET2/24 -> L2 (VNI50000) NET1/24 -> L1 (VNI50000)
NET2/24 -> L2 (VNI50000) NET2/24 -> L2 (VNI50000)

Baremetal Baremetal Baremetal

Net1 Net2 Net3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Integrated Routing and Bridging (IRB) in EVPN
Layer-2 & VLAN- Symmetric
EVPN IP VRF
Layer-3 Based IRB

• Symmetric Inter-Subnet Forwarding

• Bridge->Route/Route->Bridge

• Symmetric VNI in both directions

• Adjacency contains Remote VTEP,VRF
• Optimal for Scale

• Flexible Configuration

• IETF Draft “Integrated Routing and Bridging in

EVPN”
• draft-ietf-bess-evpn-inter-subnet-forwarding – Section 3.2

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Symmetric
Integrated Routing and Bridging (IRB) IRB

S S S S

VNI: 50000 (IP-VRF A)

RT: 65500:50000
L L L L .... L
VNI: 30002 (EVI 4) VNI: 30001 (EVI 3)
RT: 65500:30002 RT: 65500:30001

Baremetal Baremetal

M2/IP2 M3/IP3
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 Symmetric
Integrated Routing and Bridging (IRB) IRB

S S S S
RD: 10.10.10.2:32769
[2]:[0]:[0]:[48]:[M2]:[32]:[IP2]
L2 VNI: 30002 | L3 VNI: 50000
L2 RT: 65000:30002 | L3 RT: 65000:50000

VNI: 50000 (IP-VRF A)

RT: 65500:50000
L L L L .... L
VNI: 30002 (EVI 4) VNI: 30001 (EVI 3)
RT: 65500:30002 RT: 65500:30001
ARP Table Routing Table
M2 – IP2 – VLAN 4 Baremetal
IP2 /48– VTEP2 (VNI 50000) Baremetal

M2/IP2 M3/IP3
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Symmetric
Integrated Routing and Bridging (IRB) IRB

S S S S
RD: 10.10.10.255:32769
[2]:[0]:[0]:[48]:[M3]:[32]:[IP3]
L2 VNI: 30001 | L3 VNI: 50000
L2 RT: 65000:30001 | L3 RT: 65000:50000

VNI: 50000 (IP-VRF A)

RT: 65500:50000
L L L L .... L
VNI: 30002 (EVI 4) VNI: 30001 (EVI 3)
RT:
Routing 65500:30002
Table RT: 65500:30001
ARP Table
IP3/48– VTEP255 (VNI 50000) M3 – IP3 – VLAN 3
ARP Table Routing Table
M2 – IP2 – VLAN 4 Baremetal
IP2 /48– VTEP2 (VNI 50000) Baremetal

M2/IP2 M3/IP3
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 Symmetric
Integrated Routing and Bridging (IRB) IRB

S S S S

VNI: 50000 (IP-VRF A)

RT: 65500:50000
L L L L .... L
VNI: 30002 (EVI 4) VNI: 30001 (EVI 3)
RT: 65500:30002 RT: 65500:30001
Routing Table Routing Table
IP3/48– VTEP255 (VNI 50000) Baremetal
IP2 /48– VTEP2 (VNI 50000) Baremetal

M2/IP2 M3/IP3
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Integrated Routing and Bridging (IRB) in EVPN
Layer-2 & VLAN- Symmetric
EVPN IP VRF
Layer-3 Based IRB

• Symmetric IRB and Asymmetric IRB is NOT interoperable per-se

• IETF Draft “EVPN Interoperability Modes” – Section 4
• Routing is implemented different
• Symmetric IRB: Bridge -> Route -> Route -> Bridge
• Asymmetric IRB: Bridge -> Route -> Bridge
• Symmetric IRB uses Route-Type 2 with two VNI
• L3VNI for routing and L2VNI for bridging
• Asymmetric IRB uses Route-Type2 and Route-Type 5
• Type 2 with L2VNI for bridging and inter-subnet forwarding (known VNI/VTEP)
• Type 5 with L3VNI for inter-subnet forwarding (see IP-VRF Services)
• If implemented

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive
Building Data Center Networks
with VXLAN EVPN Overlays – Segment 3

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001,
2 0000.3001.1101 / 48 10.200.200.101
65500:30001

L L L L .... L

Baremetal

Host A
MAC: 0000.3001.1101
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001,
2 0000.3001.1101 / 48 10.200.200.101
65500:30001

30001,
2 0000.3001.1102 / 48 10.200.200.104
65500:30001

L L L L .... L

Baremetal Baremetal

Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001,
2 0000.3001.1101 / 48 10.200.200.101
65500:30001

30001,
2 0000.3001.1102 / 48 10.200.200.104
65500:30001

30002,
2 0000.3002.2101 / 48 10.200.200.255
65500:30002

L L L L .... L

Baremetal
Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 Host Advertisements
S S S S

Host MAC (Route Type 2)

• MAC Address
• MPLS Label1 (L2VNI*)
• Route Target for MAC-VRF

MAC attributes are Mandatory

L L L L .... L

Baremetal
Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1101 MAC: 0000.3001.1102 MAC: 0000.3002.2101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001, 50000,
2 0000.3001.1101 / 48 192.168.10.101 /32 10.200.200.101 2
65500:30001 65500:50000

L L L L .... L

Baremetal

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101 #CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001, 50000,
2 0000.3001.1101 / 48 192.168.10.101 /32 10.200.200.101
65500:30001 65500:50000

30001, 50000,
2 0000.3001.1102 / 48 192.168.10.102 /32 10.200.200.104
65500:30001 65500:50000

L L L L .... L

Baremetal Baremetal

Host A Host B
MAC: 0000.3001.1101 MAC: 0000.3001.1102
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Host Advertisements
Type MAC / Length S
L2VNI / RT S IP / Length S S
L3VNI / RT Next-Hop Seq.

30001, 50000,
2 0000.3001.1101 / 48 192.168.10.101 /32 10.200.200.101
65500:30001 65500:50000

30001, 50000,
2 0000.3001.1102 / 48 192.168.10.102 /32 10.200.200.104
65500:30001 65500:50000

30002, 50000,
2 0000.3002.2101 / 48 192.168.20.101 /32 10.200.200.107
65500:30002 65500:50000

L L L L .... L

Baremetal
Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Host Advertisements
S S S S
Host MAC (Route Type 2)
• MAC and IP
• MPLS Label1 (L2VNI)
• Route Target for MAC-VRF
• MPLS Label2 (L3VNI*)
• Route Target for IP-VRF
• Router MAC

L
IP Attributes are Optional L L L .... L
Populated through ARP/ND

Baremetal
Baremetal Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Subnet Route Advertisements
Type S
IP / Length S
L3VNI / RT S Next-Hop S Seq.

50000,
5 192.168.10.0 /24 10.200.200.101
65500:50000

L L L L .... L

Subnet A
192.168.10.0/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Subnet Route Advertisements
S S S S
Internal and External Subnet
Prefixes (Route Type 5)
• IP Prefix
• MPLS Label (L3VNI)
• Route Target for IP-VRF
• Router MAC

Populated throughL ExternalL L L .... L

Routing Protocol

Subnet A
192.168.10.0/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Introducing VXLAN

MAC 802.1q IP Payload CRC

Src and Dst
Src, Dst VTEP VTEP IP
and Hop-by- UDP Dst VXLAN
Address Port 4789 VNI
Hop MAC Original Layer-2 Frame

Data-Plane
Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC
(VXLAN)

20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

of total overhead

UDP Src Port

Hash of L2/L3/L4
headers of
original Frame

*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 VXLAN and BGP EVPN – Putting it Together
Control-Plane (BGP EVPN)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

3001 5000
2 0000.3001.1102/48 192.168.10.102/32 10.200.200.103
65500:3001 65500:5000

Dst VTEP IP L2VNI Dst MAC Dst IP

10.200.200.103 3001 0000.3001.1101 192.168.10.102

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC

Data-Plane (VXLAN)

Bridging

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 EVPN Control-Plane

Type 2

Packet Walk – Bridging MAC / Length

L2VNI / RT
0000.3001.1102 / 48

3001 / 65500:3001

IP / Length 192.168.10.102 / 32

L3VNI / RT 5000 / 65500:5000

VLAN 101 (Green)

TOR1 TOR2
Next-Hop 10.200.200.103
Host
Leaf Leaf
Ext. Community 0200.0ade.de03
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine
SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SIP DIP VXLAN SMAC DMAC SIP DIP

Spine
Payload
10.200.200.101 10.200.200.103 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

SMAC DMAC SIP DIP Host C

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102
MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 VXLAN and BGP EVPN – Putting it Together
Extended Community
Router MAC
Control-Plane (BGP EVPN) 0200.0ade.de07

Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

3001 5000
2 0000.3001.2101/48 192.168.20.101/32 10.200.200.104
65500:3001 65500:5000

Dst VTEP IP L3VNI Router MAC Dst IP

10.200.200.104 5000 0200.0ade.de07 192.168.20.101

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC

Data-Plane (VXLAN)

Routing

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
 Routing and the Router MAC – Ethernet
Router
MAC

SMAC DMAC SIP DIP

Payload
0200.0ade.de01 0200.0ade.de04 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Switch Switch
SVI10 SVI20
192.168.10.1 192.168.20.1

interface: Eth2/1 interface: Eth2/1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de04
Host IP: 10.200.200.1 IP: 10.200.200.4 Host

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 Routing and the Router MAC – VXLAN
Router
MAC

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
10.200.200.1 10.200.200.4 5000 0200.0ade.de01 0200.0ade.de04 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP SMAC DMAC SIP DIP

Payload Payload
0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101 2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

SVI10 SVI20
192.168.10.1 VTEP
VXLAN VTEP 192.168.20.1

interface: NVE1 interface: NVE1

MAC: 0200.0ade.de01 MAC: 0200.0ade.de04
Host IP: 10.200.200.1 IP: 10.200.200.4 Host

Host A Host C
MAC: 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.10.101 IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 EVPN Control-Plane

Type 2

Packet Walk – Routing MAC / Length

L2VNI / RT
0000.3002.2101 / 48

3002 / 65500:3002

IP / Length 192.168.20.101 / 32

L3VNI / RT 5000 / 65500:5000

VLAN 101 (Green)

TOR1 TOR2
Next-Hop 10.200.200.104
Host
Leaf Leaf
Ext. Community 0200.0ade.de04
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC DMAC SIP DIP Spine

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
Spine
10.200.200.101 10.200.200.104 5000 0200.0ade.de01 0200.0ade.de04 192.168.10.101 192.168.20.101

TOR3 TOR4
SMAC DMAC SIP DIP
VLAN 101 (Green)

Leaf Leaf
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
What about ARP
resolution?
 Packet Walk – ARP Request

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Request for 192.168.10.102 Spine

SMAC: DMAC: SIP DIP VXLAN SMAC DMAC ARP Request for
0000.3001.1101 FFFF.FFFF.FFFF
192.168.10.102
10.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

ARP Request for 192.168.10.102

SMAC: DMAC:
Host C
0000.3001.1101 FFFF.FFFF.FFFF MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Packet Walk – ARP Response

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
ARP Response for 192.168.10.102 Spine

SMAC: DMAC:
0000.3001.1102 0000.3001.1101
SIP DIP VXLAN SMAC DMAC ARP Response for
192.168.10.102
10.200.200.103 10.200.200.101 3001 0000.3001.1102 0000.3001.1101
Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

ARP Response for 192.168.10.102

SMAC: DMAC:
Host C
0000.3001.1102 0000.3001.1101 MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Silent Host Discovery
…and here ARP
continues
 EVPN Control-Plane

Type 5

Packet Walk – Routing IP / Length

L3VNI / RT
192.168.20.0 / 24

5000 / 65500:5000

Silent Host Discovery Next-Hop 10.200.200.102 10.200.200.104

Ext. Community 0200.0ade.de02 0200.0ade.de04

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SIP DIP VXLAN SMAC Spine DMAC SIP DIP

Payload
10.200.200.101 10.200.200.102 5000 0200.0ade.de01 0200.0ade.de02 192.168.10.101 192.168.20.101

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 EVPN Control-Plane

Type 5

Packet Walk – Routing IP / Length

L3VNI / RT
192.168.20.0 / 24

5000 / 65500:5000

Silent Host Discovery Next-Hop 10.200.200.102 10.200.200.104

Ext. Community 0200.0ade.de02 0200.0ade.de04

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

SIP DIP VXLAN SMAC DMAC ARP Request for
192.168.20.101
10.200.200.102 239.0.0.1 3002 AGM FFFF.FFFF.FFFF

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf
ARP Request for 192.168.20.101
Host SMAC: DMAC:
AGM FFFF.FFFF.FFFF
Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 EVPN Control-Plane

Type 5

Packet Walk – Routing IP / Length

L3VNI / RT
192.168.20.0 / 24

5000 / 65500:5000

Silent Host Discovery Next-Hop 10.200.200.102 10.200.200.104

Ext. Community 0200.0ade.de02 0200.0ade.de04

VLAN 101 (Green)

TOR1 TOR2
Host
Leaf Leaf

Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
ARP Response for 192.168.20.101MAC: 0000.3002.2101
IP: 192.168.20.101
SMAC: DMAC:
0000.3002.2102 AGM
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 EVPN Control-Plane

Type 2

Packet Walk – Routing MAC / Length

L2VNI / RT
0000.3002.2101 / 48

3002 / 65500:3002

Silent Host Discovery IP / Length 192.168.20.101 / 32

L3VNI / RT 5000 / 65500:5000

VLAN 101 (Green)

TOR1 TOR2
Next-Hop 10.200.200.104
Host
Leaf Leaf
Ext. Community 0200.0ade.de04
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
Spine

VLAN 202 (Blue)

Spine

TOR3 TOR4
VLAN 101 (Green)

Leaf Leaf

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
ARP Response for 192.168.20.101MAC: 0000.3002.2101
IP: 192.168.20.101
SMAC: DMAC:
0000.3002.2102 AGM
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 EVPN Control-Plane

Type 2

Packet Walk – Routing MAC / Length

L2VNI / RT
0000.3002.2101 / 48

3002 / 65500:3002

Silent Host Discovered IP / Length 192.168.20.101 / 32

L3VNI / RT 5000 / 65500:5000

VLAN 101 (Green)

TOR1 TOR2
Next-Hop 10.200.200.104
Host
Leaf Leaf
Ext. Community 0200.0ade.de04
Host A
MAC: 0000.3001.1101
IP: 192.168.10.101
SMAC DMAC SIP DIP Spine

VLAN 202 (Blue)

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
Spine
10.200.200.101 10.200.200.104 5000 0200.0ade.de01 0200.0ade.de04 192.168.10.101 192.168.20.101

TOR3 TOR4
SMAC DMAC SIP DIP
VLAN 101 (Green)

Leaf Leaf
2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Host

Host B
MAC: 0000.3001.1102
VXLAN EVPN

VLAN 202 (Blue)

IP: 192.168.10.102
Host

Host C
MAC: 0000.3002.2101
IP: 192.168.20.101

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Distributed Anycast
Gateway
Distributed Anycast Gateway
S S S S

L L L L .... L
SVI 101
192.168.10.1/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Distributed Anycast Gateway
S
Distributed First-Hop Routing on Edge Device
S S S
• All Edge Device share same Gateway IP and MAC
address
• Pervasive Gateway approach

Gateway is always active

• No First-Hop redundancy protocol for hello or
state exchange

L
Distributed and smaller state L L L .... L
• Only local End-Points ARP entries

SVI 101
192.168.10.1/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Distributed Anycast Gateway
S S S S

L L L L .... L

Anycast Gateway MAC

2020.0000.AAAA
SVI 101
192.168.10.1/24

SVI 202
192.168.20.1/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Distributed Anycast Gateway
S S S S

fabric forwarding anycast-gateway-mac 2020.0000.AAAA

!
interface Vlan101
vrf member myvrf_5000
ip address 192.168.10.1/24
fabric forwarding mode anycast-gateway
!
interface Vlan202
vrf member myvrf_5000

L
ip address 192.168.20.1/24
L
fabric forwarding mode anycast-gateway L L .... L

Anycast Gateway MAC

2020.0000.AAAA
SVI 101
192.168.10.1/24

SVI 202
192.168.20.1/24

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Anycast – One-to-Nearest Association
S S S S
• Local Ethernet Segment-based ARP
Resolution for First-Hop Gateway

L L L L .... L
ARP request
ARP reply ARP request ARP reply

Baremetal
Baremetal

Host A Host C
MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 #CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Anycast – One-to-Nearest Association
S S S S
• Local Ethernet Segment-based ARP
Resolution for First-Hop Gateway

• No ARP resolution for SVI from Overlay side

L L L L .... L

Baremetal
Baremetal

Host A Host C
MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 #CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Anycast – One-to-Nearest Association
S S S S

• Network Addressing and Routing

Methodology

• Datagrams sent from a single Sender to the

Topologically Nearest Node

L L L L
• .... L
Group of potential Receivers, all identified by
the same Destination Address

Baremetal
Baremetal

Host A Host C
MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 #CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Anycast – One-to-Nearest Association + vPC
S S S S

vpc

L L L L .... L

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Anycast – One-to-Nearest Association + vPC
S S S S
• Active / Active Forwarding with no First-Ho
redundancy protocol (ie HSRP, VRRP)
• ARP / ND Synchronized via VPC
• Fast Failover

vpc • Port-Channel Hashing avoids duplicate

• Broadcast, Unknown Unicast, Multicast
L L L L ....
(BUM) hashed on single link
L

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Anycast – One-to-Nearest Association + vPC
S S S S

vpc

L L L L .... L
ARP reply

ARP request

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Virtual Port-Channel
(vPC)
Virtual Port-Channel (vPC) Gateway Redundancy
S S S S

vpc

L L L L .... L

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 Virtual Port-Channel (vPC) Gateway Redundancy

• VPC – Virtual Port-Channel

S S S S
• Multi-Chassis Link Aggregation
• Layer-2 Multihoming
• Extended for VXLAN

• Host-side
• Dual-Connect Hosts
• Using Port-Channels
vpc

• Fabric-side
•
•
L
Individual VTEPs
Using a common Anycast VTEP
L L L .... L
• Seen as one VTEP from remote
Nodes

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Virtual Port-Channel (vPC) Gateway Redundancy
S S S S
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.123

50000,
2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 10.200.200.123
65500:5000

vpc

L L L L .... L

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Virtual Port-Channel (vPC) Gateway Redundancy
S S S S
• Independent Devices in the EVPN
Control-Plane
• Individual Router and Peering
• Unique Route Distinguisher (RD)
• Independent Underlay Routing
Devices
vpc
• Common VXLAN Device
•
• L
Next-Hop is Anycast VTEP
Underlay ECMP Load Share to L L L .... L
Anycast VTEP

Baremetal Baremetal
Baremetal

Host A Host B Host C

MAC: 0000.3001.1102 MAC: 0000.3002.2101
MAC: 0000.3001.1101 IP: 192.168.20.101
IP: 192.168.10.101 IP: 192.168.10.102
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
vPC Gateway Redundancy for VXLAN
Spine Spine

Individual Identity vPC1 Individual Identity vPC2

10.200.200.104 10.200.200.105
Anycast VTEP (vip)
10.200.200.123
vPC1 vPC2

Server Server

Server

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 vPC Gateway Redundancy for VXLAN
Spine Spine

Individual Identity vPC1

interface loopback0 Individual
interface Identity vPC2
loopback0
description RID description RID
10.200.200.104 10.200.200.105
ip address 10.10.10.104/32 ip address 10.10.10.105/32
Anycast VTEP (vip)
interface loopback1 interface loopback1
description VTEP 10.200.200.123 description VTEP
ip address 10.200.200.104/32 vPC1 vPC2 ip address 10.200.200.105/32
ip address 10.200.200.123/32 secondary ip address 10.200.200.123/32 secondary

Server Server

Server

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
vPC Gateway Redundancy for VXLAN
Spine Spine

Individual Identity vPC1 Individual Identity vPC2

10.200.200.104 10.200.200.105
Anycast VTEP (vip)
10.200.200.123
vPC1 vPC2

Server
EVPN Route Type Attachment Next-hop
Server

Type 5 vPC advertised by vip

(IP Prefix Routes) Orphan advertised by vip
Server
Type 2 vPC advertised by vip
(Host Routes) Orphan Port advertised by vip
DIGDCN-3378 #CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
 NX-OS 9.2(3)

vPC with Fabric Peering

Spine Spine

Virtual Peer Link

vPC Domain

vPC1 vPC2
Orphan Port Orphan Port

Peer Keepalive

Server Server

Server

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 NX-OS 9.2(3)

vPC with Fabric Peering

Spine Spine

Virtual Peer Link

Virtual Peer Link over Fabric (Layer-3) vPC Domain

• Uses Spines for Redundancy, Resiliency
and Performance vPC1 vPC2
• Doesn’t use VTEP IP address (loopback)

Peer Keepalive remains

Peer Keepalive • Out-of-Band (mgmt0 or dedicated link)
• In-Band (dedicated Loopback over Fabric)

Server Server

Server

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 NX-OS 9.2(3)

Host Attachment
Spine Spine

Virtual Peer Link

Individual Identity and Individual Identity and
VTEP vPC1 (pip) VTEP vPC2 (pip)
10.200.200.104 10.200.200.105
Anycast VTEP (vip)
10.200.200.123
vPC1 vPC2
Orphan Port Orphan Port

Peer Keepalive

vPC
Server
EVPN Route Type Attachment Next-hop
Server

Type 5 vPC advertised by pip

(IP Prefix Routes) Orphan advertised by pip
Server
Type 2 vPC advertised by vip
(Host Routes) Orphan Port advertised by pip
DGTL-BRKDCN-3378 #CiscoLive © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
 NX-OS 9.2(3)

Network Attachment
Spine Spine

Virtual Peer Link

Individual Identity and Individual Identity and
VTEP vPC1 (pip) VTEP vPC2 (pip)
10.200.200.104 10.200.200.105
Anycast VTEP (vip)
10.200.200.123
vPC1 vPC2

Subnet X Orphan Port Orphan Port Subnet Y

192.168.11.0/24 192.168.12.0/24

Peer Keepalive

vPC
EVPN Route Type Attachment Next-hop

Type 5 vPC advertised by pip

Subnet X (IP Prefix Routes) Orphan advertised by pip
192.168.11.0/24
Type 2 vPC advertised by vip
(Host Routes) Orphan Port advertised by pip
DGTL-BRKDCN-3378 #CiscoLive 57
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive
Building Data Center Networks
with VXLAN EVPN Overlays – Segment 4

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
MTU and Overlays
• Data Centre often require Jumbo MTU
• Most Server NIC support up to 9000 Bytes

• Network Switches support MTU up to 9216* Bytes

• Accommodates Jumbo MTU plus Overlay overhead (50/54bytes)

• Avoid Fragmentation
• Adjust the Transport Network with appropriate MTU

No Fragmentation Needed

*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 Traffic

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 Interface Principles

S S S S
• Routed Ports and Interfaces
• Layer-3 Interfaces as Uplink (no switchport)
• For each Point-2-Point (P2P) connection, minimum /31
required (IPv4)
• Alternative, use IPv4 Unnumbered (/32)
• IPv6 allows Link-Local and Global IP Addressing

• Use Loopback as Source-Interface for VTEP

(NVE)
L L L L .... L

Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 IP Addressing Principles
p2p Agg: 10.1.1.0/24 RP RP
10.1.1.0/30
10.1.1.4/30

•
10.1.1.8/30
Prepare
… a IP Addressing Plan S
RID
S
RID
S
RID
S
RID

10.1.1.251/30

• Separate
RID Interface
Agg: 10.10.10.0/24 functions through IP
10.10.10.1/32
Addressing (Aggregates)
10.10.10.2/32
• Unicast Routing – Routing Protocol Peering (p2p)
10.10.10.3/32
• …
Unicast Routing – Routing Identifier (RID)
10.10.10.255/32
• VTEP and VPC
Multicast
• Agg:
VTEP Rendezvous-Point (RP)
10.200.200.0/24 VTEP VTEP VTEP VTEP VTEP
10.200.200.1/32
• IPv4
10.200.200.2/32
and IPv6 (as per
10.200.200.3/32 L
NX-OS 9.3(1))
RID L
RID L
RID L
RID .... L
RID
…
10.200.200.255/32

RP Agg: 10.254.254.0/24
10.254.254.1/32
Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
IP Addressing Principles
p2p Agg: 10.1.1.0/24 RP RP
10.1.1.0/30
10.1.1.4/30
10.1.1.8/30
…
S
RID
S
RID
S
RID
S
RID

10.1.1.251/30

RID Agg: 10.10.10.0/24

10.10.10.1/32
10.10.10.2/32
10.10.10.3/32
…
10.10.10.255/32

VTEP Agg: 10.200.200.0/24 VTEP VTEP VTEP VTEP VTEP

10.200.200.1/32
10.200.200.2/32
10.200.200.3/32 L
RID L
RID L
RID L
RID .... L
RID
…
10.200.200.255/32

RP Agg: 10.254.254.0/24
10.254.254.1/32
Baremetal Baremetal Baremetal

M1/IP1 M2/IP2 M3/IP3

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 Unicast Routing – OSPF and IS-IS
• IS-IS – what was this CLNS?
• OSPF – watch your Network Type • Independent of IP (CLNS)
• Well suited for routed interfaces/ports
• Network Type Point-2-Point • No SPF calculation on Link change; only if
• Preferred (only LSA type-1) Topology changes
• No DR/BDR election • Consider Level-2 for Single Area
• Suits well for routed interfaces/ports (optimal from • Fast Re-convergence
a LSA DB perspective) • Not everyone is familiar with it
• Full SPF calculation on Link Change

Did you know?

There are IGP Enhancements for Flood Optimization and Leaf/Spine Networks
draft-ietf-lsr-isis-spine-leaf-ext & draft-li-lsr-dynamic-flooding

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Unicast Routing – BGP
• eBGP Underlay Routing – Service Provider style
• Intended for non-Overlay L3-Fabric (RFC7938)
• Two Different Models
• Two-AS
• Multi-AS

• BGP is a Distance Vector Protocol

• actually Path Vector Protocol
• AS* are used to calculate the Path (AS_Path)

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Unicast Routing – eBGP Two-AS Model

AS 65000 (All-Spine) • eBGP Two-AS, yes it works!

S S S S • eBGP peering for Underlay
• Spine is not a Route-Reflector (eBGP) – Retain
Route-Targets
• Disable BGP AS-Path check

• Underlay is Reachability!
• Advertise your Loopbacks
L L L L .... L
• Special Overlay Control-Plane
AS 65001 (All-Leaf) Treatment
• Next-Hop needs to be Unchanged
• Disable BGP AS-Path check

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Unicast Routing – eBGP Multi-AS Model

AS 65000 (All-Spine) • eBGP Multi-AS, rebuild the Internet

S S S S • eBGP peering for Underlay
• Spine is not a Route-Reflector (eBGP) – Retain
Route-Targets
• Disable BGP AS-Path check
• Next-Hop needs to be Unchanged

• Underlay is Reachability!
L L L L .... L • Advertise your Loopbacks

AS AS AS AS AS • Special Overlay Control-Plane

65001 65002 65003 65004 6500n Treatment
• Next-Hop needs to be Unchanged

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Unicast Routing – eBGP Model

S S S S • Two different BGP Peering

• eBGP peering for Underlay
• Global IPv4/v6 Address-Family
• Use Physical Interface IP

• eBGP peering for Overlay

L L L L .... L • Global EVPN Address-Family
• Use Loopback Interface IP
• BFD not so ok
Don
’t Forg
e t EC
MP

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive
Building Data Center Networks
with VXLAN EVPN Overlays – Segment 5

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Traditional
Forwarding in VXLAN
Overlays
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S R
SRC RCVR

VLAN 101 (Green)

VLAN 101 (Green) SRC-10 RCVR-10
VLAN 101 (Green)
224.10.10.10 10.10.10.10
10.10.10.100
VLAN 101 (Green)
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S R
SRC RCVR

VLAN 101 (Green)

VLAN 101 (Green) SRC-10 RCVR-10
VLAN 101 (Green)
224.10.10.10 10.10.10.10
10.10.10.100
VLAN 101 (Green)
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Same Subnet Forwarding no IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S R
SRC RCVR

VLAN 101 (Green)

VLAN 101 (Green) SRC-10 RCVR-10
VLAN 101 (Green)
224.10.10.10 10.10.10.10
10.10.10.100
VLAN 101 (Green)
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Same Subnet Forwarding with IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S R
SRC RCVR

VLAN 101 (Green)

VLAN 101 (Green) SRC-10 RCVR-10
VLAN 101 (Green)
224.10.10.10 10.10.10.10
10.10.10.100
VLAN 101 (Green)
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Same Subnet Forwarding with IGMP Snooping
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S R
RCVR
R
SRC
VLAN 101 (Green)
VLAN 101 (Green) RCVR
SRC-10 RCVR-10
VLAN 101 (Green)
224.10.10.10 10.10.10.10
RCVR-11 10.10.10.100
10.10.10.11 VLAN 101 (Green)
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S
VLAN 102 (Blue)
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S

L L L L .... L
S
VLAN 102 (Blue)
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 Different Subnet Forwarding – Router on-a-Stick
Traditional Forwarding in VXLAN Overlays

S S S S
• Multiple Copy in Core –
Treated as BUM
• Different Subnet possible –
RPF Challenges
• Pruning on Local Interface
• VXLAN is NOT pruned if
interest Receiver exists
behind one Remote VTEP L L L L .... L
S
VLAN 102 (Blue)
R
VLAN 101 (Green) SRC

RCVR
VLAN 101 (Green) 10.10.10.254 SRC-10
RCVR-21 10.20.20.254 224.10.10.10
10.20.20.21 10.10.10.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Tenant Routed
Multicast (TRM)
Functional
Components
Functional Components
Tenant Routed Multicast (TRM)

S S S S

https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-seamless-interop

L
DR L L
DR L .... L
DR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Functional Components
Tenant Routed Multicast (TRM)
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-seamless-interop

S S S S

Underlay:
• PIM-based Underlay Transport (PIM ASM)
• Separate Multicast Groups from Layer-2 VNI
• Leveraging same redundant Underlay Rendezvous-
Point (i.e. PIM Anycast-RP)
• Single Packet in Core

L
DR L L
DR L .... L
DR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
 Functional Components
Tenant Routed Multicast (TRM)
https://tools.ietf.org/html/draft-sajassi-bess-evpn-mvpn-seamless-interop

S S S S
Overlay:
Underlay: • BGP-based Control-Plane using ngMVPN (Next-
• PIM-based Underlay Transport (PIM ASM) Generation Multicast VPN)
• Separate Multicast Groups from Layer-2 VNI • Using existing BGP Route-Reflector
• Leveraging same redundant Underlay Rendezvous- • Rendezvous-Point-less
Point (i.e. PIM Anycast-RP) • Efficient Single Copy in Multicast Underlay
• Single Packet in Core • Always-Route approach (per-VLAN config)
• Distributed Anycast Designated Router (DR)

L
DR L L
DR •
• L ....
VPC – Virtual Port-Channel
DR
Integration with non-TRM VTEP L
S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
TRM Control- & Data-
Plane (RP-less)
 RP-less Tenant Route Multicast (TRM)
S S S S

RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
 MRoute: Underlay Multicast State (PIM)

*,G – (*, 239.1.1.2/32)

S S S S
IIF: Uplink (Underlay)
OIF: NVE1 (Underlay)

S,G – (10.0.0.1, 239.1.1.2/32) S,G – (10.0.0.255, 239.1.1.2/32)

IIF: NVE-LoopbackS,G – (10.0.0.2, 239.1.1.2/32)
(Underlay) IIF: NVE-Loopback (Underlay)
OIF: Uplink (Underlay)
IIF: NVE-LoopbackS,G – (10.0.0.3, 239.1.1.2/32)
(Underlay) OIF: Uplink (Underlay)
OIF: Uplink (Underlay)
IIF: NVE-LoopbackS,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: Uplink (Underlay)
RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 MRoute: Underlay Multicast State (PIM)
• PIM ASM required for Underlay
*,G – (*, 239.1.1.2/32)
S S S SeparateSGroups for BUM and MDT
•
IIF: Uplink (Underlay) • Default MDT initiates on VTEP startup (*, G)
OIF: NVE1 (Underlay)
• Per-VTEP (S,G) initiated on VTEP startup
S,G – (10.0.0.1, 239.1.1.2/32) • RP-less (Distributed Anycast
S,G – (10.0.0.255, RP) on VTEPs
239.1.1.2/32)
IIF: NVE-LoopbackS,G – (10.0.0.2, 239.1.1.2/32)
(Underlay) IIF: NVE-Loopback (Underlay)
OIF: Uplink (Underlay)
IIF: NVE-LoopbackS,G – (10.0.0.3, 239.1.1.2/32)
(Underlay) OIF: Uplink (Underlay)
OIF: Uplink (Underlay)
IIF: NVE-LoopbackS,G – (10.0.0.4, 239.1.1.2/32)
(Underlay)
OIF: Uplink (Underlay)
IIF: NVE-Loopback (Underlay)
OIF: Uplink (Underlay)
RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
 Action: Multicast Source Starts Sending Traffic
S S S S

RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR
FHR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 NGMVPN: Source Active Advertisement
MVPN Type 5

S S S S
Source Active (NGMVPN Type 5)
Originator: Leaf #255
Route-Target: 65502:50001
S,G: 10.10.10.100, 224.10.10.10

RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR
FHR

S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 Action: Receiver IGMP Join
S S S S

*,G – (*, 224.10.10.10)

IIF: VRF-L3VNI (Overlay)
OIF: VLAN102 (Host-facing)
RP RP RP
VNI: 50001 (VRF) S,G – (10.10.10.100, 224.10.10.10)
MDT: 239.1.1.2
RT: 65502:50001 L
DR LIIF: L
VRF-L3VNIDR
(Overlay)
OIF: VLAN100 (Host-facing) L .... L
DR
FHR
IGMP Join S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 NGMVPN: Source Tree Join
MVPN Type 7

S S S S
Source Tree Join (NGMVPN Type 7)
Originator: Leaf #3
S,G: 10.10.10.100, 224.10.10.10

RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR
FHR
IGMP Join S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 MRoute: Overlay Multicast State
S S S S

*,G – (*, 224.10.10.10)

IIF: VRF-L3VNI (Overlay)
OIF: VLAN102 (Host-facing)
RP RP RPS,G – (10.10.10.100, 224.10.10.10)
VNI: 50001 (VRF) S,G – (10.10.10.100, 224.10.10.10)
MDT: 239.1.1.2
RT: 65502:50001 L
DR LIIF: L
VRF-L3VNIDR
(Overlay)
OIF: VLAN100 (Host-facing) L .... DR L
IIF: VLAN100 (Host-facing)
OIF: VRF-L3VNI (Overlay)
FHR
IGMP Join S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 TRM Traffic Flow
S S S S

RP RP RP
VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
DR
FHR
IGMP Join S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Action: Receiver IGMP Join
S S S S

RP RP RPS,G – (10.10.10.100, 224.10.10.10)

VNI: 50001 (VRF)
MDT: 239.1.1.2
RT: 65502:50001 L
DR L L
DR L .... L
IIF: VLAN100 (Host-facing)
DR
OIF: VRF-L3VNI (Overlay),
FHR VLAN100 (Host-facing)
IGMP Join S R
S R SRC RCVR

RCVR
SRC
SRC-10 RCVR-10
RCVR-21 224.10.10.10 10.10.10.10
SRC-20
10.20.20.21 10.10.10.100
224.20.20.20
10.20.20.100
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive
Building Data Center Networks
with VXLAN EVPN Overlays – Segment 6

Lukas Krattiger, Principal Engineer

@CCIE21921
DGTL-BRKDCN-3378

#CiscoLive
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
 VXLAN Evolves as the Control Plane Evolves!
Before Yesterday
Yet Another Encapsulation
§ Flood & Learn (Multicast-based)
§ Data-Plane only

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
 VXLAN Evolves as the Control Plane Evolves!
Before Yesterday
Yet Another Encapsulation
§ Flood & Learn (Multicast-based)
§ Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
§ Control-Plane
§ Active VTEP Discovery
§ Multicast and Unicast

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Traditional Overlay Forwarding
SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L

Baremetal Baremetal

10.1.1.10 10.1.1.20

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Traditional Overlay Forwarding
SS SS SS SS
AS 65033

S S S S S S S S

AS 65001 AS 65002

L L L L .... L L L L L .... L

Baremetal Baremetal

10.1.1.10 10.1.1.20

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Traditional Overlay Forwarding
SS SS SS SS
AS 65033

S S S S S S S S

AS 65001 AS 65002

L L L L .... L L L L L .... L

Control-Plane
Host AS Next-Hop
Baremetal Baremetal

10.1.1.10 65001 10.1.1.1

10.1.1.10 10.1.1.20
10.1.1.20 65002 10.2.2.4
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Traditional Overlay Forwarding
SS SS SS SS
AS 65033

S S S S S S S S

AS 65001 AS 65002

L L L L .... L L L L L .... L

Control-Plane
Host AS Next-Hop
Baremetal Baremetal

10.1.1.10 65001 10.1.1.1

10.1.1.10 10.1.1.20
10.1.1.20 65002 10.2.2.4
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Multi-Pod: Overlay Spread and Extend

DC Local Overlay

End-to-End Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Single Logical Data Center

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
The Ugly Truth
Overlays Spread and Extend

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Multi-Site: A Paradigm Change with Hierarchical Overlays

DC Local Overlay

Multi-Site Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Multiple Logical Data Center

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Network Routing Forwarding
SS SS SS SS

BS BS BS BS BS BS BS BS

L L L L .... L L L L L .... L

Baremetal Baremetal

10.1.1.10 10.1.1.20

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Network Routing Forwarding
SS SS SS SS
AS 65033 (Core)

BS BS BS BS BS BS BS BS

AS 65001 (Site1) AS 65002 (Site2)

L L L L .... L L L L L .... L

Baremetal Baremetal

10.1.1.10 10.1.1.20

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
 Control-Plane (Core)
Network Routing Forwarding Host AS Next-Hop
10.1.1.10 65001 BS-Site1
10.1.1.20 65002 BS-Site2
SS SS SS SS
AS 65033 (Core)

BS BS BS BS BS BS BS BS

AS 65001 (Site1) AS 65002 (Site2)

L L L L .... L L L L L .... L

Control-Plane (Site1) Control-Plane (Site2)

Host AS Next-Hop Host AS Next-Hop
Baremetal Baremetal

10.1.1.10 65001 10.1.1.1 10.1.1.10 65001 BS-Site2

10.1.1.10 10.1.1.20
10.1.1.20 65002 BS-Site1 10.1.1.20 65002 10.2.2.2

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
 Control-Plane (Core)
Network Routing Forwarding Host AS Next-Hop
10.1.1.10 65001 BS-Site1
10.1.1.20 65002 BS-Site2
SS SS SS SS
AS 65033 (Core)

BS BS BS BS BS BS BS BS

AS 65001 (Site1) AS 65002 (Site2)

L L L L .... L L L L L .... L

Control-Plane (Site1) Control-Plane (Site2)

Host AS Next-Hop Host AS Next-Hop
Baremetal Baremetal

10.1.1.10 65001 10.1.1.1 10.1.1.10 65001 BS-Site2

10.1.1.10 10.1.1.20
10.1.1.20 65002 BS-Site1 10.1.1.20 65002 10.2.2.2

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
 VXLAN Evolves as the Control Plane Evolves!
Before Yesterday
Yet Another Encapsulation
§ Flood & Learn (Multicast-based)
§ Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
§ Control-Plane
§ Active VTEP Discovery
Today
§ Multicast and Unicast
VXLAN for DCI – Inter-DC
§ DCI Ready
§ ARP/ND caching/suppress
§ Multi-Homing
§ Failure Domain Isolation
§ Loop Protection
#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Control-Plane (Core)
Multi-Site Overlay Forwarding Host AS Next-Hop
MAC1/IP1 65001 Site1-VIP
MAC2/IP2 65002 Site2-VIP
SS SS SS SS
AS 65033 (Core)

BS BS BS BS BS BS BS BS

AS 65001 (Site1) AS 65002 (Site2)

L L L L .... L L L L L .... L

Control-Plane (Site1) Control-Plane (Site2)

Host AS Next-Hop Host AS Next-Hop
Baremetal Baremetal

MAC1/IP1 65001 10.1.1.1 MAC1/IP1 65001 Site2-VIP

10.1.1.10 10.1.1.20
MAC2/IP2 65002 Site1-VIP MAC2/IP2 65002 10.2.2.2

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 VXLAN Multi-Site
Hierarchical VXLAN
Overlay Multi-Site

Border Gateway
(BGW)
Multi-Site VIP Multi-Site VIP
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Any VTEP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
VXLAN Multi-Site Characteristics

§ Multiple Overlay Domains – Interconnected and Controlled

§ Multiple Overlay Control-Plane Domains – Interconnected and Controlled
§ Multiple Underlay Domains - Isolated
§ Multiple Replication Domains for BUM – Interconnected and Controlled
§ Multiple VNI Administrative Domains

Underlay Isolation – Overlay Hierarchies

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Agenda
Segment 1 - Overlays Segment 2 – Standards & Implementation Segment 3 – Control & Data-Plane

Segment 4 – Underlay Considerations Segment 5 – Tenant Routed Multicast Segment 6 – VXLAN EVPN Multi-Site

#CiscoLive DGTL-BRKDCN-3378 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
If you haven’t had
enough VXLAN BGP
EVPN
Thank you

#CiscoLive
#CiscoLive