Control Plane Troubleshooting

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
• Inter-controller control connections establishment

• vEdge to controllers control connection establishment

• <17.2.0, essentially Command / CLI-driven troubleshooting

• 17.2 and 17.2.1 adds significant capabilities in troubleshooting from within vManage

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
Components
• vBond – vEdge/vSmart/vManage Connection
• DTLS and internal primitives

• vEdge – vSmart and vSmart – vSmart Connection

• Overlay Management Protocol

• vEdge / vSmart – vManage Connection

• Netconf

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vBond Configuration
vBond1# show running-config system
system
host-name vBond1
system-ip 1.1.1.1 Required
organization-name Marketing-Demo
clock timezone America/Los_Angeles
upgrade-confirm 15
vbond 10.0.2.116 local vbond-only
……
vBond1# sh run vpn 0
vpn 0
interface ge0/0
ip address 10.0.2.116/24
no shutdown
!
ip route 0.0.0.0/0 10.0.2.1
!

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vBond Configuration
• Show orchestrator local-properties
vBond# show orchestrator local-properties
personality vbond
sp-organization-name Marketing-Demo
Required
organization-name Marketing-Demo
system-ip 1.1.1.1
certificate-status Installed
root-ca-chain-status Installed
certificate-validity Valid
certificate-not-valid-before Jun 21 22:54:43 2017 GMT
certificate-not-valid-after Jun 21 22:54:43 2018 GMT
chassis-num/unique-id 1cc6f68a-7281-48c6-9b22-3732a97c1187
serial-num 12345C5E
vedge-list-version 9223372036854775807
vsmart-list-version 9223372036854775807 Confirmation
number-active-wan-interfaces 2
protocol dtls
INSTANCE INDEX PORT VSMARTS VMANAGES STATE
----------------------------------------------
0 0 12346 2 8 up
1 0 12347 2 8 up

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vBond
• Show orchestrator connections / show orchestrator connections-history
0 vsmart dtls 1.1.1.10 110 1 10.0.2.237 12346 52.6.192.219 12346 default up Marketing-Demo 115:12:45:40
0 vsmart dtls 1.1.1.10 110 1 10.0.2.237 12446 52.6.192.219 12446 default up Marketing-Demo 115:12:45:40
0 vmanage dtls 1.1.1.1 101 0 10.0.2.44 12346 34.199.49.113 12346 default up Marketing-Demo 65:19:16:14
0 vmanage dtls 1.1.1.1 101 0 10.0.2.44 12446 34.199.49.113 12446 default up Marketing-Demo 5:19:01:11

0 unknown dtls - 0 0 :: 0 38.140.50.254 12346 default tear_down BIDNTVRFD/NOERR 33652 2017-09-25T15:20:07-0700

0 unknown dtls - 0 0 :: 0 52.41.148.34 12386 default tear_down BDSGVERFL/NOERR 63 2017-09-25T15:19:57-0700
0 unknown dtls - 0 0 :: 0 38.140.50.254 43932 default tear_down BIDNTVRFD/NOERR 1162 2017-09-25T15:19:55-0700
0 unknown dtls - 0 0 :: 0 38.140.50.254 43932 default tear_down ORPTMO/NOERR 0 2017-09-25T05:19:58-0700

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vBond
• Show orchestrator summary
vBond# show orchestrator summary
VMANAGE VSMART VEDGE LISTENING LISTENING LISTENING
INSTANCE COUNTS COUNTS COUNTS PROTOCOL IP IPV6 PORT
------------------------------------------------------------------------------
0 8 2 4 dtls 0.0.0.0 :: 12346
1 8 2 1 dtls 0.0.0.0 :: 12446

• Show orchestrator valid-vsmarts

vBond1# show orchestrator valid-vsmarts
SERIAL NUMBER ORG
--------------------------------------------------
074B7AE1DEA4F9678FA42C5A766845EF Marketing-Demo
0E4782A4EBCFA2DA47EB0DD9AE097C9A Marketing-Demo
10CC1C0B71C3775885BF8B0F3E79ECC7 Marketing-Demo

• Show orchestrator valid-vedges

vBond# show orchestrator valid-vedges
CHASSIS NUMBER SERIAL NUMBER VALIDITY ORG
----------------------------------------------------------------------------------------------------
05AB5C5F-101B-12C4-B2DF-B7342C0B20D3 af492837 valid Marketing-Demo
1B1C2B1A-A91B-12D4-B2DF-B7342C0B20D3 bf73576f31eb57e5774e19701a853b42 valid Marketing-Demo

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vBond – Show orchestrator-connections: error legend
vBond1# show orchestrator connections-history
Legend for Errors
ACSRREJ - Challenge rejected by peer. NOVMCFG - No cfg in vmanage for device.
BDSGVERFL - Board ID Signature Verify Failure. NOZTPEN - No/Bad chassis-number entry in the ZTP.
BIDNTPR - Board ID not Initialized. ORPTMO - Server's peer timed out.
BIDNTVRFD - Peer Board ID Cert not verified. RMGSPR - Remove Global saved peer.
CERTEXPRD - Certificate Expired RXTRDWN - Received Teardown.
CRTREJSER - Challenge response rejected by peer. RDSIGFBD - Read Signature from Board ID failed.
CRTVERFL - Fail to verify Peer Certificate. SSLNFAIL - Failure to create new SSL context.
CTORGNMMIS - Certificate Org name mismatch. SERNTPRES - Serial Number not present.
DCONFAIL - DTLS connection failure. SYSIPCHNG - System IP changed.
DEVALC - Device memory Alloc failures. TMRALC - Timer Object Memory Failure.
DHSTMO - DTLS HandShake Timeout. TUNALC - Tunnel Object Memory Failure.
DISCVBD - Disconnect vBond after register reply. TXCHTOBD - Failed to send challenge to BoardID.
DISTLOC - TLOC Disabled. UNMSGBDRG - Unknown Message type or Bad Register msg.
DUPSER - Duplicate Serial Number. UNAUTHEL - Recd Hello from Unauthenticated peer.
DUPCLHELO - Recd a Dup Client Hello, Reset Gl Peer. VBDEST - vDaemon process terminated.
HAFAIL - SSL Handshake failure. VECRTREV - vEdge Certification revoked.
IP_TOS - Socket Options failure. VSCRTREV - vSmart Certificate revoked.
LISFD - Listener Socket FD Error. VB_TMO - Peer vBond Timed out.
MGRTBLCKD - Migration blocked. Wait for local TMO.
MEMALCFL - Memory Allocation Failure. VM_TMO - Peer vManage Timed out.
NOACTVB - No Active vBond found to connect. VP_TMO - Peer vEdge Timed out.
NOERR - No Error. VS_TMO - Peer vSmart Timed out.
NOSLPRCRT - Unable to get peer's certificate. XTVSTRDN - Extra vSmart tear down.
LRNTPEER - Delete learnt peer entry. CGNIDCHNGD- Vbond teardown CGN ID.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vEdge/vSmart/vManage Configuration
• Show control local-properties
Gateway-VPC-Edge1# show control local-properties

Required
personality vedge
sp-organization-name Marketing-Demo
organization-name Marketing-Demo
certificate-status Installed
root-ca-chain-status Installed
certificate-validity Valid
certificate-not-valid-before Aug 19 21:16:54 2017 GMT
certificate-not-valid-after Aug 17 21:16:54 2027 GMT
dns-name 52.5.226.23
site-id 8001
domain-id 1
protocol dtls
tls-port 0
system-ip 5.5.5.15
chassis-num/unique-id 5a5b5c5f-a91b-12c4-b2df-b7342c0b20d3
serial-num 878CAAF9
token Invalid
keygen-interval 1:00:00:00
retry-interval 0:00:00:19
no-activity-exp-interval 0:00:00:12
dns-cache-ttl 0:00:02:00
port-hopped TRUE
time-since-last-port-hop 37:01:20:07

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vEdge/vSmart/vManage Configuration
• Show control local-properties
number-vbond-peers 1
INDEX IP PORT Confirmation
-----------------------------------------------------
0 52.5.226.23 12346
number-active-wan-interfaces 1
NAT TYPE: E -- indicates End-point independent mapping
A -- indicates Address-port dependent mapping
N -- indicates Not learned
Note: Requires minimum two vbonds to learn the NAT type
PUBLIC PUBLIC PRIVATE PRIVATE PRIVATE MAX RESTRICT/ LAST SPI TIME NAT VM
INTERFACE IPv4 PORT IPv4 IPv6 PORT VS/VM COLOR STATE CNTRL CONTROL/ LR/LB CONNECTION REMAINING TYPE CON
STUN PRF
-------------------------------------------------------------------------------------------------------------------------------------------
ge0/0 34.202.61.140 12366 10.0.5.126 :: 12366 1/1 3g up 2 no/yes/no No/No 0:00:00:00 0:10:39:52 N 5

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
vEdge/vSmart/vManage Control connections
• Show control connections / Show control connections-history
0 vedge dtls 1.1.1.8 108 1 172.31.255.21 12366 52.52.125.202 12366 public-internet up 15:13:44:27
0 vsmart dtls 1.1.1.3 103 1 10.0.2.211 12346 34.199.57.20 12346 default up 65:19:30:53
0 vsmart dtls 1.1.1.10 110 1 10.0.2.237 12346 52.6.192.219 12346 default up 115:12:59:11
0 vbond dtls 1.1.1.2 0 0 34.199.1.194 12346 34.199.1.194 12346 default up 65:19:30:53

0 vedge dtls 1.1.1.8 108 1 172.31.13.15 12366 52.9.139.131 12366 biz-internet up RXTRDWN VM_TMO 1 2017-09-07T10:01:00-0700
0 vbond dtls 1.1.1.11 0 0 34.204.213.179 12346 34.204.213.179 12346 default tear_down VB_TMO NOERR 1 2017-08-18T07:48:18-0700
0 vsmart dtls 1.1.1.3 103 1 10.0.2.211 12346 34.199.57.20 12346 default connect DCONFAIL NOERR 0 2017-07-27T07:37:23-0700
0 vbond dtls - 0 0 34.199.1.194 12346 34.199.1.194 12346 default connect DCONFAIL NOERR 10 2017-07-27T07:37:23-0700
0 vsmart dtls 1.1.1.3 103 1 10.0.2.211 12346 34.199.57.20 12346 default up RXTRDWN VM_TMO 0 2017-07-27T07:37:09-0700

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Control Plane Troubleshooting
OMP Sessions

DataCenter# show omp peers

R -> routes received
I -> routes installed
S -> routes sent

DOMAIN OVERLAY SITE

PEER TYPE ID ID ID STATE UPTIME R/I/S
------------------------------------------------------------------------------------------
1.1.1.3 vsmart 1 1 103 up 45:05:18:14 2/2/4
1.1.1.10 vsmart 1 1 110 up 40:04:51:25 2/0/4

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting
Components
• TLOC-table – Data Plane Destinations
• Private/Public IPs / Ports
• Encryption Keys / SPIs

• vRoutes
• TLOCs
• Labels

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting
• vEdge-to-vEdge only

• Typical Problem Sources:

• NAT Traversal (Symmetric NAT or Non-deterministic outside port allocation)
• NAT Device illicit header rewrite (affecting AH)
• Multiple vEdge-devices behind same NAT
• Dinky NATs
• Link/Path Quality
• Indication of a data plane problems
• Individual BFD Sessions being down/flapping
• BFD Session never coming up without other error indications

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – TLOC-table
DataCenter# show omp tlocs
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid.
PUBLIC PRIVATE
ADDRESS PSEUDO PUBLIC PRIVATE PUBLIC IPV6 PRIVATE IPV6 BFD
FAMILY TLOC IP COLOR ENCAP FROM PEER STATUS KEY PUBLIC IP PORT PRIVATE IP PORT IPV6 PORT IPV6 PORT STATUS
------------------------------------------------------------------------------------------------------------------------------------------------------
ipv4 1.1.1.4 biz-internet ipsec 1.1.1.3 C,I,R 1 34.192.241.51 12346 10.0.1.32 12346 :: 0 :: 0 up
1.1.1.10 C,R 1 34.192.241.51 12346 10.0.1.32 12346 :: 0 :: 0 up
1.1.1.4 public-internet ipsec 1.1.1.3 C,I,R 1 34.199.146.200 12346 10.0.3.232 12346 :: 0 :: 0 up
1.1.1.10 C,R 1 34.199.146.200 12346 10.0.3.232 12346 :: 0 :: 0 up

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – Encryption
DataCenter# show ipsec inbound-connections
SOURCE SOURCE DEST DEST REMOTE REMOTE LOCAL LOCAL NEGOTIATED
IP PORT IP PORT TLOC ADDRESS TLOC COLOR TLOC ADDRESS TLOC COLOR ENCRYPTION ALGORITHM TC SPIs
------------------------------------------------------------------------------------------------------------------------------------------------------
34.192.241.51 12346 172.31.8.114 12346 1.1.1.4 biz-internet 1.1.1.5 biz-internet AES-GCM-256 8
35.167.220.247 12346 172.31.8.114 12346 1.1.1.6 biz-internet 1.1.1.5 biz-internet AES-GCM-256 8
34.198.153.120 12346 172.31.8.114 12346 1.1.1.7 biz-internet 1.1.1.5 biz-internet AES-GCM-256 8
52.9.139.131 12366 172.31.8.114 12346 1.1.1.8 biz-internet 1.1.1.5 biz-internet AES-GCM-256 8
34.199.146.200 12346 172.31.255.23 12366 1.1.1.4 public-internet 1.1.1.5 public-internet AES-GCM-256 8
52.41.98.132 12346 172.31.255.23 12366 1.1.1.6 public-internet 1.1.1.5 public-internet AES-GCM-256 8
34.203.52.21 12346 172.31.255.23 12366 1.1.1.7 public-internet 1.1.1.5 public-internet AES-GCM-256 8
52.52.125.202 12366 172.31.255.23 12366 1.1.1.8 public-internet 1.1.1.5 public-internet AES-GCM-256 8

DataCenter# show ipsec local-sa

SOURCE SOURCE SOURCE
TLOC ADDRESS TLOC COLOR SPI IPv4 IPv6 PORT KEY HASH
----------------------------------------------------------------------------------------------------------------------
1.1.1.5 biz-internet 489 172.31.8.114 :: 12346 *****c5c5
1.1.1.5 biz-internet 490 172.31.8.114 :: 12346 *****90f7
1.1.1.5 public-internet 488 172.31.255.23 :: 12366 *****0e88
1.1.1.5 public-internet 489 172.31.255.23 :: 12366 *****d362

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – Encryption
DataCenter# show ipsec outbound-connections
SOURCE SOURCE DEST DEST REMOTE REMOTE AUTHENTICATION NEGOTIATED
IP PORT IP PORT SPI TUNNEL MTU TLOC ADDRESS TLOC COLOR USED KEY HASH ENCRYPTION ALGORITHM TC SPIs
------------------------------------------------------------------------------------------------------------------------------------------------------
172.31.8.114 12346 34.192.241.51 12346 286 1442 1.1.1.4 biz-internet AH_SHA1_HMAC *****1676 AES-GCM-256 8
172.31.8.114 12346 34.198.153.120 12346 487 1442 1.1.1.7 biz-internet AH_SHA1_HMAC *****96c8 AES-GCM-256 8
172.31.8.114 12346 35.167.220.247 12346 486 1442 1.1.1.6 biz-internet AH_SHA1_HMAC *****d1f8 AES-GCM-256 8
172.31.8.114 12346 52.9.139.131 12366 492 1442 1.1.1.8 biz-internet AH_SHA1_HMAC *****3c56 AES-GCM-256 8
172.31.255.23 12366 34.199.146.200 12346 286 1442 1.1.1.4 public-internet AH_SHA1_HMAC *****68f3 AES-GCM-256 8
172.31.255.23 12366 34.203.52.21 12346 487 1442 1.1.1.7 public-internet AH_SHA1_HMAC *****368c AES-GCM-256 8
172.31.255.23 12366 52.41.98.132 12346 488 1442 1.1.1.6 public-internet AH_SHA1_HMAC *****4589 AES-GCM-256 8
172.31.255.23 12366 52.52.125.202 12366 492 1442 1.1.1.8 public-internet AH_SHA1_HMAC *****5597 AES-GCM-256 8

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – VPN vRoutes
DataCenter# show ip routes vpn 1
Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive

PROTOCOL NEXTHOP NEXTHOP NEXTHOP

VPN PREFIX PROTOCOL SUB TYPE IF NAME ADDR VPN TLOC IP COLOR ENCAP STATUS
---------------------------------------------------------------------------------------------------------------------------------------------
1 5.5.5.5/32 static - - - - - - - B,F,S
1 10.0.1.0/24 omp - - - - 1.1.1.4 biz-internet ipsec F,S
1 10.0.1.0/24 omp - - - - 1.1.1.4 public-internet ipsec F,S
1 172.31.35.0/24 connected - ge0/2 - - - - - F,S

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting
Components – VPN vRoutes
DataCenter# show ip routes vpn 1 10.0.1.0/24 det
Codes Proto-sub-type:
IA -> ospf-intra-area, IE -> ospf-inter-area,
E1 -> ospf-external1, E2 -> ospf-external2,
N1 -> ospf-nssa-external1, N2 -> ospf-nssa-external2,
e -> bgp-external, i -> bgp-internal
Codes Status flags:
F -> fib, S -> selected, I -> inactive,
B -> blackhole, R -> recursive

--------------------------------------------
VPN 1 PREFIX 10.0.1.0/24
--------------------------------------------
proto omp
distance 250
metric 0
uptime 10:07:42:01
tloc-ip 1.1.1.4
tloc-color biz-internet
tloc-encap ipsec
nexthop-label 1
status F,S

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – VPN vRoutes
DataCenter# sh ip fib vpn 1
NEXTHOP NEXTHOP NEXTHOP SA
VPN PREFIX IF NAME ADDR LABEL INDEX TLOC IP COLOR
------------------------------------------------------------------------------------------------------------------------------
1 5.5.5.5/32 NULL0 0.0.0.0 - - - -
1 10.0.1.0/24 ipsec 34.192.241.51 1 1912 1.1.1.4 biz-internet
1 10.0.1.0/24 ipsec 34.199.146.200 1 1911 1.1.1.4 public-internet
1 172.31.35.0/24 ge0/2 - - - - -
1 172.31.35.235/32 ge0/2 - - - - -

DataCenter# show omp tlocs

PUBLIC PRIVATE
ADDRESS PSEUDO PUBLIC PRIVATE PUBLIC IPV6 PRIVATE IPV6 BFD
FAMILY TLOC IP COLOR ENCAP FROM PEER STATUS KEY PUBLIC IP PORT PRIVATE IP PORT IPV6 PORT IPV6 PORT STATUS
------------------------------------------------------------------------------------------------------------------------------------------------------
ipv4 1.1.1.4 biz-internet ipsec 1.1.1.3 C,I,R 1 34.192.241.51 12346 10.0.1.32 12346 :: 0 :: 0 up
1.1.1.10 C,R 1 34.192.241.51 12346 10.0.1.32 12346 :: 0 :: 0 up
1.1.1.4 public-internet ipsec 1.1.1.3 C,I,R 1 34.199.146.200 12346 10.0.3.232 12346 :: 0 :: 0 up
1.1.1.10 C,R 1 34.199.146.200 12346 10.0.3.232 12346 :: 0 :: 0 up

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
 Data Plane Troubleshooting
Components – BFD Sessions
DataCenter# show bfd sessions
SOURCE TLOC REMOTE TLOC DST PUBLIC DST PUBLIC DETECT TX
SYSTEM IP SITE ID STATE COLOR COLOR SOURCE IP IP PORT ENCAP MULTIPLIER INTERVAL UPTIME TRANSITIONS
------------------------------------------------------------------------------------------------------------------------------------------------------
1.1.1.4 104 up biz-internet biz-internet 172.31.8.114 34.192.241.51 12346 ipsec 7 1000 10:07:50:27 28
1.1.1.4 104 up public-inter public-inter 172.31.255.23 34.199.146.200 12346 ipsec 7 1000 14:07:57:53 0
1.1.1.6 106 up biz-internet biz-internet 172.31.8.114 35.167.220.247 12346 ipsec 7 1000 85:18:08:05 4
1.1.1.6 106 up public-inter public-internet 172.31.255.23 52.41.98.132 12346 ipsec 7 1000 1:03:43:26 1
1.1.1.7 107 up biz-internet biz-internet 172.31.8.114 34.198.153.120 12346 ipsec 7 1000 2:13:59:41 15
1.1.1.7 107 up public-inter public-internet 172.31.255.23 34.203.52.21 12346 ipsec 7 1000 14:07:57:53 0
1.1.1.8 108 up biz-internet biz-internet 172.31.8.114 52.9.139.131 12366 ipsec 7 1000 15:15:49:34 5
1.1.1.8 108 up public-inter public-internet 172.31.255.23 52.52.125.202 12366 ipsec 7 1000 14:07:57:53 0

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting
Your Best Friend – show system statistics diff

• Provides access to all internal system counters – Drops, Errors etc

• Issue multiple times with ’diff’ keyword to see the diff between show runs

• Source port not matching the received TLOC

DataCenter# show system statistics | incl rx_implicit_acl_drops
rx_implicit_acl_drops : 185

• Packet being modified by transit NAT – usually ID

DataCenter# show system statistics | incl replay_integrity_drops
replay_integrity_drops : 0

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Data Plane Troubleshooting
Other Tools

• Implicit ACL Logging

• Access-lists w/ Logging and/or counters

• Packet Mirroring (IPinIP tunneled to destination of choice or dumped locally w/ tcpdump

• TCPdump

• Debugs

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential