BRKRST 2600
BRKRST 2600
BRKRST 2600
Shashank Singh
Technical Leader, Cisco
BRKRST-2600
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Advanced platform visibility
• Zero Touch Provisioning (ZTP)
• PoE Innovations & StackPower
• Manageability Innovations
• Support Packages
• Appendix (Script Examples for Reference)
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Your Instructor Today…
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Cisco Catalyst 9000 switches at a glance
Enabling a new era of intent-based networking
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Cisco Catalyst 9000 innovations
Cisco DNA Center™
Converged
Open Cisco IOS XE
ASIC
Segmentation Automation Assurance
UADP ASIC
Single image
Common
licensing Analytics On-box app hosting
• Streaming telemetry • Monitoring security IoT
• Network monitoring
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Campus switching portfolio
Cisco Catalyst
Cisco Catalyst 9600 Series
9400 Series Cisco
Cisco Catalyst Catalyst Cisco Catalyst
9300 Series 9000 9500 Series
Cisco Catalyst
9200 Series
switch
platform
Cisco Catalyst Cisco Catalyst Cisco Catalyst Cisco Catalyst Cisco Catalyst Cisco Catalyst
2960-X/XR Series 3650/3850 Series 4500E Series 3850F/4500-X 6840-X/6880-X 6807-XL/6500-E
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Simplified operations and serviceability with
Cisco Catalyst 9000
Ease of serviceability with blue beacons on
Inventory management efficiency with built-in RFID
each component
Ergonomic design with industry-standard icons Wireless console access with Bluetooth
Icons
Mobile Laptop
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
Advanced Platform Visibility
Guestshell
Real life problems and tools
Application Hosting
TCL
Day 0/1 deployment
Cisco Guestshell
Network-wide configuration
Use this space to provide a bit of detail.
Cisco DNAC/PnP
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
On the box Python
Guestshell overview
• Access to bootflash.
• Bundled with Python- Cisco cli python library for CLI DMI = Data Model Interface = Netconf/Yang interface
PnP = Plug N Play = Zero Touch provisioning
operations and automated output collection.
Also Supported…
• Default access only permitted through Mgmt VRF.
• ZTP – Zero Touch Provisioning can retrieve a Python
Additional options to integrate with device data plane.
script via DHCP at boot time
• EEM – Use Embedded Event Manager to trigger a
Python script in response to an event
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Guestshell
Preparation
Cat9K#conf t
Cat9K(config)#iox Start IOx
Cat9K#show iox-service
IOx Infrastructure Summary:
--------------------------- Ensure service is running
IOx service (CAF) : Running
IOx service (HA) : Running
IOx service (IOxman) : Running
Libvirtd : Running Configuration required before activation
Configuration includes VPG interface for Guestshell
Activation is via simplified exec command
Cat9K#conf t
Cat9K(config)#app-hosting appid guestshell
Cat9K(config-app-hosting)#app-vnic management guest-interface 0
Cat9K#guestshell enable
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Guestshell
Verification
Ensure guestshell is up and
running
Cat9K#show app-hosting list
App id State
------------------------------------------------------
guestshell RUNNING Double-check that Linux
inherited mgmt
configuration
Cat9K#guestshell run sudo ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.30.2 netmask 255.255.255.248 broadcast 192.168.30.7
inet6 fe80::5054:ddff:fe20:e54c prefixlen 64 scopeid 0x20<link>
ether 52:54:dd:20:e5:4c txqueuelen 1000 (Ethernet)
RX packets 8 bytes 648 (648.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 00
Cat9K#guestshell destroy
Guestshell destroyed successfully
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Guestshell
Using cli library Cat9K#guestshell run python
Python 2.7.5 (default, Aug 4 2017, 00:39:18) Run python and
execute cisco CLI.
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Building configuration...
Current configuration : 38 bytes
!
interface GigabitEthernet1/0/1 Change configuration (i.e. go to interface
end gi1/0/1 and set description and IP address)
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Events?
Syslog
Custom Triggers
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Regular Expressions
Regex Cheatsheets & Activities
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Trigger off an interface statistic
EEM example
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Packet capture
EEM + Python – Automated data collection
Trigger is receive rate on Gi1/0/1
event manager applet detect_high_interface_rate
event interface name GigabitEthernet1/0/1 parameter receive_rate_bps entry-op ge
entry-val 1000 poll-interval 1 entry-type value
Variables can be passed to
action 0.01 syslog msg "High Rx rate detected, running automated packet capture." python script at runtime
action 0.02 cli command "enable" Python script called
action 0.03 cli command "guestshell run python /flash/gs_script/packet_capture.py Gi1/0/1"
packet_capturet.py
import cli,time,sys
Unique filename
t = time.localtime() using timestamp
timestamp = time.strftime('%b-%d-%Y_%H%M%S', t)
PATH_NAME = ("flash:/gs_script/" + timestamp + ".pcap")
INTERFACE_NAME = str(sys.argv[1])
Interface name received
cli.execute("enable") from EEM as variable
cli.execute("mon cap pack_cap int %s in file loc %s size 10 match any" % (INTERFACE_NAME, PATH_NAME))
cli.execute("monitor capture pack_cap start")
cli.execute("send log Capture running on %s for 10 sec" %INTERFACE_NAME)
cli.executep("show monitor capture pack_cap")
time.sleep(10)
cli.execute("monitor capture pack_cap stop")
cli.execute("send log Capture saved in %s" %PATH_NAME) 21
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Periodic Events
At certain time/day
Every N sec
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Common trigger mechanisms
Time before script Time before system
runs in seconds kills script in seconds
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Shut down a port at 6pm every Thursday.
EEM example
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ASIC level drops and exceptions
show platform hardware fed switch active fwd-asic drop exceptions
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Monitoring Cat9K forwarding asic drops
Guestshell example
import re,time,cli
import sys
Raw data
#Get output of- show platform hard fed sw active fwd-asic drop exceptions
#Check for non zero value against any field in delta column.
#Ignore rows NO_EXCEPTION PKT_DROP_COUNT BLOCK_FORWARD these rows as they are seen to increment without any
issue too
non_zero_values_delta = re.findall(r"\d+?\s+?\d+?\s+?(\S+?)\s+?\d+?\s+?\d+?\s+?([1-9]\d*?)\s",
sh_drop_exceptions)
non_zero = 0 Regex to match
if non_zero_values_delta: desired pattern
for name, non_zero_delta in non_zero_values_delta:
if str(name) != "NO_EXCEPTION" and str(name) != "PKT_DROP_COUNT" and str(name) != "BLOCK_FORWARD":
non_zero =1
cli.execute("send log" + "Non zero delta value found found %s, for %s. Check 'show platform hard
fed sw active fwd-asic drop exceptions'" % (non_zero_delta, name))
#If not non zero delta values found generating no problem found alert.
if not non_zero:
cli.execute("send log" + "No problem found")
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Demo
Forwarding asic drops
Troubleshooting Bot
(Tbot) Scripts
Troubleshooting bot scripts – IOS-XE 17.1.X
Bundled python bots
• Python scripts bundled with IOS-XE software that can be run from guestshell in
order to confirm hardware programming for Layer 2 and 3 entries.
• Collects the dumps of different components and analyzes them on the box.
• Integrated in the build image, scripts will be copied to the /flash/Tbot directory.
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Troubleshooting bot scripts – Demo
Run the L2DebugInfraAPi.py script, it will use the other .py files in the Tbot directory. Running Tbot
guestshell run python /flash/Tbot/L2DebugInfraAPIs.py 2182 00a7.428a.93d9
############################### Matm RP Active Mac Address Matches -- SUCCESS
#### Layer2 Debugging Tool #### Matm FP Active Table ID Matches -- SUCCESS
############################### Aom Vlan ID and Mac Address Matches -- SUCCESS
Aom Status Matches -- SUCCESS
@@@@ User Inputs @@@@
-> VLAN ID : 2182 @@@@ Hardware Programming @@@@
-> MAC ADDRESS : 00a7.428a.93d9 -> AsicHexList : []
-> ECIFIDIntf : Port-channel101
@@@@ L2 Forwarding Programming @@@@ -> FedMVID : 15
-> PoMembers : ['Fo6/0/9'] -> ECIFIDGpn : 2144
-> MacAddrindex : 0 -> AsicVlanList : []
-> PoMemStats : ['P'] -> AsicCoreList : ['0']
-> PoID : 101 -> AsicGpnList : []
-> PoNAME : Port-channel101 -> ECIFID : 0x00000106
-> PoProtocol : On -> MacHandle : 0x7f34d24dd178
-> DIInstIDList : ['4']
@@@@ Interface to Doppler D Instance Mapping @@@@ -> DIHandle : 0x7f34d271b758
-> IntfIfID_List : ['0x92'] -> DIPortIDList : ['20']
-> IntfAsic_List : ['2'] -> DICoreIDList : ['0']
-> IntfCore_List : ['0'] Contextual data -> DIAsicIDList : ['2']
-> IntfInst_List : ['4'] gathering and analysis -> SI_PortMapList : 0x00000000 0x00100000
-> SI_RIList : ['0x2']
MVID Value and Lead Vlan LookUp Value Matches -- SUCCESS -> SICoreIDList : ['0']
-> SIAsicIDList : ['2']
@@@@ Software Programming @@@@ -> SIHandle : 0x7f34d2713628
-> Matm_FPPortsList : 262 -> SI_DIList : ['0x5248']
-> Matm_RPPortsList : 262 -> SIPortIDList : ['20']
-> Matm_RPOmPtr : OM: 0x3480768298
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Zero-Touch Provisioning
Zero Touch Provisioning
What is this?
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Zero Touch Provisioning
ZTP workflow using option 67
ip dhcp pool ztp
network 10.1.1.0 255.255.255.0
default-router 10.1.1.254
option 150 ip 10.1.1.254
option 67 ascii ZTP_demo.py
!
tftp-server flash:ZTP_demo.py
TFTP/HTTP
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Zero touch provisioning workflow
ZTP device boots up & does not ZTP device gets IP address of
1 find the startup configuration. 4 HTTP/TFTP server, and
downloads Python script.
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Zero Touch Provisioning
Day 0 configuration
print "\n\n *** ZTP Python Script *** \n\n"
import cli,re
user = "cisco"
password = "cisco"
enable = "cisco"
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Zero Touch Provisioning
Day 1 configuration
print "\n\n *** ZTP Python Script Execution Complete *** \n\n"
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Demo
Zero Touch Provisioning
PoE Innovations
& StackPower
UPoE+ on Catalyst 9000
Cisco and
New partner
802.3bt* cloud
services for
Type 4 control
90W Cisco UPOE+©
UPOE+©
(low voltage) New
Devices
30 - 60 Watt devices 90 Watt devices
*Type 4 UPOE+ shipping on Catalyst 9400 and Type 3 shipping on Catalyst 9300.
Type 4 UPOE+ on 9300 is on roadmap. Catalyst 9200 is PoE+.
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
90W UPOE+ standardization is enabling a
growing ecosystem of Switches
UPOE+© Standardization Growing ecosystem
UPoE Powered Compact
switches
802.1AC Access Points New
VOIP Phones
90W
UPOE+ USB-C POE Displays Network HVAC VAV’s
60W © Dongle
Touchscreen IP Cameras Powered Light
PCs
30W UPOE
PoE+
15W
PoE
Badge Facial
Readers Recognition
Systems
2019 UPOE Powered Nurse call
CBRS Biometric
2011 Compact Switches systems
Private LTE
Environmental
Door Locks IP Call Tower
Sensor Hubs
2009
2003 IT OT
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Push architecture boundaries with UPOE+ 90W
IEEE 802.3bt compliant platforms Daisy-chaining
1 (for cable cost savings)
PoE+
USB-C powering
3 (laptop/phone charging data)
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Cisco innovations in PoE deliver a robust low
voltage infrastructure
2-event
Perpetual PoE Fast PoE
classification
• Fast power negotiation • Uninterrupted PoE • Bypasses IOS control
without LLDP power during control plane boot
• Physical layer plane reboot • Restores power to PD
negotiation < 1s within 30 sec of
power resumption
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
2-Event Classification
Class 4 PD gets 30W even before the link
Without 2-Event Classification: comes up, otherwise 15.4W
Type-2 PSE sends classification voltage (2nd
Handshake) pulse to Class 4 PD (PoE+,
12.96-25.5 W) confirming it is a high power
LLDP LLDP capable PSE
(Layer 2) 20-25 sec (Layer 2)
30W of power is allocated for PD (PD draws
up to 25.5W)
Both PSE and PD need to support 2-Event
No waiting for CDP/LLDP exchange for
With 2-Event Classification (Hardware Based): PoE+ power levels
Switch> enable
Switch# configure terminal
Electrical signal Switch(config)# interface gigabitethernet2/0/1
Electrical signal
(Layer 1) Switch(config-if)# power inline port 2-event
1 sec (Layer 1)
Switch(config-if)# end
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Perpetual PoE/UPOE
PoE devices connected to switch stay powered even on switch reload
Switch> enable
Switch# configure terminal
Switch(config)# interface gigabitethernet2/0/1
Switch(config-if)# power inline port perpetual-poe-ha
Switch(config-if)# end
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Fast PoE/UPOE
Power
2 event
Classification
• Works in stacking deployments * In case of UPOE, since the PD relies on LLDP to get
to higher power levels, PD may still need to wait till the
IOS comes up and LLDP packet exchanges happen
Switch> enable
Switch# configure terminal
Switch(config)# interface gigabitethernet2/0/1
Switch(config-if)# power inline port perpetual-poe-ha
[“perpetual-poe-ha” is a perquisite to “poe-ha”] Switch(config-if)# power inline port poe-ha
Switch(config-if)# end
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
StackPower - Overview
“Zero-Footprint” RPS deployment
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
How StackPower Works?
StackPower
715 W
• Pools power from all power supplies
1100
715 W
W
715
W
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Power Budget Modes Load shedding does not occur in redundant mode
unless two or more power supplies fail, because the
largest power supply is used as a backup power source.
1100 1100
715 W 715 W
W W
715 715
W W
Lost PS or Lost PS or
Power source Power source
Shed Load
Lost Shed Load
another PS
Dropped PD Dropped PD
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
StackPower CLIs For Your
Reference
Power supply
inputs, PIDs
PowerStack domain
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
StackPower Best Practice
Balance Power supplies across the stack
Total Input Power = 4400W
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
Support Packages
Support Packages – IOS-XE 16.9.3, 16.10.1
Layer3 unicast stream
show tech-support platform layer3 unicast [vrf <VRFname>] destIp
<destinationIP> srcIp <destinationIP> Context aware data
collection
Dot1x
show tech-support identity mac <mac address> interface <type_and_#>
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Serviceability Innovations
CPU Punt packet rate per interface
C9300#show platform software fed sw active ifm mappings
Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active
GigabitEthernet1/0/1 0x8 1 0 1 0 0 26 6 1 1 NIF Y
IF_ID mapping
C9300#show platform software fed sw active punt rates interfaces 0x8
Punt Rate on Single Interfaces Statistics
Interface : GigabitEthernet1/0/1 [if_id: 0x8]
Received Dropped
-------- -------
Total : 3263 Total : 0
10 sec average : 4340 10 sec average : 0
1 min average : 4340 1 min average : 0 Also works without specifying
5 min average : 450 5 min average : 0
interface number
Per CPUQ punt stats on the interface (rate averaged over 10s interval)
==========================================================================
Q | Queue | Recv | Recv | Drop | Drop |
no | Name | Total | Rate | Total | Rate |
==========================================================================
0 CPU_Q_DOT1X_AUTH 0 0 0 0
1 CPU_Q_L2_CONTROL 1582 0 0 0
2 CPU_Q_FORUS_TRAFFIC 0 0 0 0
3 CPU_Q_ICMP_GEN 0 0 0 0
4 CPU_Q_ROUTING_CONTROL 0 0 0 0
5 CPU_Q_FORUS_ADDR_RESOLUTION 482 0 0 0
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Manageability Innovations
Manageability that fits your network
Active switch
UP Ports
Switch Stack
Standby switch
POE status
WebUI
Cisco DNA Center
Small Branch - CPC Migration
Part of the Larger Network
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Embedded WebUI
Build Troubleshooting
Ease of Access Intuitive Interface Switch View
configurations made fun
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
WebUI - Switch view
Switch Stack
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Wireless manageability with Bluetooth
• Supported IOS XE version 16.12.1 onwards.
• Supported on all Catalyst 9000 series switches except C9500-32C, C9500-32QC,
C9500-48Y4C, and C9500-24Y4C models
• Bluetooth 4.0 version only.
• In a stack, external USB Bluetooth dongle needs to be enabled on active switch.
• After Stateful Switchover (SSO), external USB Bluetooth needs to be re-enabled on
the new active switch.
• External USB Bluetooth dongle is not supported with the following configurations:
Quality of Service (QoS)
Access Control List (ACL)
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Bluetooth Dongle for wireless management
Step 1 Connect an external USB Bluetooth dongle to the USB Type A port on the Switch.
Step 2
Switch# configure terminal
Switch(config)# interface bluetooth 0/4
Switch(config-if)# enable
Switch(config-if)# no shut
Step 3
Switch(config-if)#bluetooth pin 1111
OR
Switch(config-if)#exit Mobile Laptop
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Bluetooth Dongle for wireless management
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Summary
What?
Features that solve your problem
Which?
More than one way, which is the best?
How?
Configure, observe, evaluate, tweak
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Appendix - Reference Scripts
import cli
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Upgrade IOS-XE version on Catalyst 9000
EEM example
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Stack event checker
Guestshell example – Part 1/2
import re, time, cli
# SDP counter check for non zero count against Tx Fail or Rx Fail
pattern = re.compile(r'(\w+.*)\s+?\d+\s+?(((\d{2,}|[1-9])\s+?\d+\s+(0))|((0)\s+?\d+\s+(\d{2,}|[1-9])))')
for stack in stack_numbers:
show_cmd = 'show platform software stack-mgr switch %s r0 sdp-counters'%(stack)
sdp_counter_op = cli.execute(show_cmd)
if sdp_counter_op:
for line in sdp_counter_op.splitlines():
match = pattern.search(line)
if match:
message = match.group(1).strip()
tx_fail = match.group(4)
rx_fail = match.group(8)
if not tx_fail:
tx_fail = '0'
if not rx_fail:
rx_fail = '0'
cli.execute("send log" + " '%s' has %s Tx_Fail and %s Rx_Fail counters"%(message, tx_fail, rx_fail))
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Stack event checker
Guestshell example – Part 2/2
# Register's snapshot
cmd_list = ['show platform hardware fed switch %s fwd-asic register read register-name SifRacDataCrcErrorCnt asic %s', 'show
platform hardware fed switch %s fwd-asic register read register-name SifRacRwCrcErrorCnt asic %s', 'show platform hardware fed
switch %s fwd-asic register read register-name SifRacPcsCodeWordErrorCnt asic %s', 'show platform hardware fed switch %s fwd-asic
register read register-name SifRacInvalidRingWordCnt asic %s']
snapshot = {}
for stack in stack_numbers:
for asic in range(int(switch_asic_dict[stack])):
# Loop for 2 snapshot
for x in range(2):
snapshot.update({x:[]})
# collect value for all 4 commands
for cmd in cmd_list:
sh_cmd = cmd%(stack, asic)
output = cli.execute(sh_cmd)
value = re.search('count\s+?:\s(\w+)',output)
if value:
# check for hex 3+ digits
if int(value.group(1), 16) > int('0x99', 16):
cli.execute("send log" + " '%s' has '%s' count"%(sh_cmd, value.group(1)))
snapshot[x].append((sh_cmd, value.group(1)))
else:
snapshot[x].append([sh_cmd, 'None'])
# wait for 5 sec and collect again for above 4 commands
time.sleep(5)
# compare two snapshot values
for index, cmd in enumerate(cmd_list):
if snapshot[1][index][1] and snapshot[0][index][1]:
if snapshot[1][index][1] > snapshot[0][index][1]:
cli.execute("send log" + " '%s' has increased from '%s' to '%s' within 5 seconds"%(snapshot[0][index][0],
snapshot[0][index][1], snapshot[1][index][1] © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Monitoring Cat9K CPU Punt cause & drops
Guestshell example
import re,time,cli
import sys
sh_sw_punt_cause = cli.execute('show platform software fed switch active punt cause summary')
sh_sw_cpu_int = cli.execute('show platform hardware fed switch active qos queue stats internal cpu policer')
sh_sw_punt_cpuq = cli.execute('show platform software fed switch active punt cpuq all')
Regex to match
desired pattern
#Look for any non zero drop count. If found generate log along with the cause and non zero field.
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Monitoring Cat9K CPU Punt cause & drops
Guestshell example
##looking for any cpu policer queue dropping packets. If found generate alert along with the queue and
corresponding non zero field.
Sends log
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Monitoring Cat9K CPU Punt cause & drops
Guestshell example
#Look for any non zero count following fields -
#Send to IOSd failed count
#RX suspend count
#RX unsuspend send failed count
#RX dropped count
#RX non-active dropped count Regex to match
#RX conversion failure dropped desired pattern
#RX spurious interrupt
punt_cpuq = 0
non_zero_values_punt_cpuq = re.findall(r"(CPU Q Id\s+?\: (\d{1,})(?:(?!(?:CPU Q Id))[\s\S])*)", sh_sw_punt_cpuq)
if non_zero_values_punt_cpuq:
for cpq_entry, cpu_q_id in non_zero_values_punt_cpuq:
match = re.findall(r"(Send to IOSd failed count|RX suspend count|RX unsuspend send failed count|RX dropped count|RX
non-active dropped count|RX conversion failure dropped|RX spurious interrupt)\s+?\: ([1-9][0-9]{0,})", cpq_entry)
if match:
punt_cpuq = 1
for field, non_zero_value in match:
cli.execute("send log" + " Non zero value %s found for %s - CPU Q ID %s. Check 'show platform software fed
switch active punt cpuq all'" % (non_zero_value, field, cpu_q_id))
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Cat9K QoS drops checker
Use case
Cat9K#show platform hardware fed sw active qos queue stats int Gi2/0/1
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
0 0 0 0 756751
1 0 0 0 452
2 0 0 0 37645
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
------------------------------- Cat9K#show platform hardware fed sw active qos queue stats inter Gi2/0/1
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop -------------------------------
----- ----------- ----------- ----------- ----------- ----------- Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
0 0 0 9393 0 0 ----- ------- ----------- ----------- -----------
1 0 0 0 0 0 0 0 0 0 978374
2 0 0 0 0 0 1 0 0 0 934
2 0 0 0 37989
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 0 10393 0 0
1 0 0 0 0 0
2 0 0 0 0 0
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Cat9K QoS drops checker - continued
Guestshell example
import re,time,cli
import sys
Raw data
#Get show interface status output
show_int_status_op = cli.execute('show interface status')
show_running_op = cli.execute('show running-config')
#Find all interfaces that are in up/up state. Look for connected in show interface status output.
up_interfaces_list = re.findall(r"(Gi\d\/\d\/\d{1,2}|Te\d\/\d\/\d{1,2}).+?connected", show_int_status_op)
if not up_interfaces_list:
cli.execute("send log" + " Need interface is found to be up'") Regex to match
exit() desired pattern
#Check if drops are happening in any queue-threshold by comparing 2 snapshots with 2 secs time interval
for the up interfaces
intf_drop_traffic = []
for up_interface in up_interfaces_list:
snapshot_1 = cli.execute('show platform hard fed sw active qos queue stats interface ' +
up_interface)
time.sleep(2) Introduce delay
snapshot_2 = cli.execute('show platform hard fed sw active qos queue stats interface ' +
up_interface)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cat9K QoS drops checker - continued
Guestshell example
for queue in range (0, 8):
match_2 = re.search(r"Drop Counters[^!]+?(%d)\s+?(\d+)\s+?(\d+)\s+?(\d+)\s+?(\d+)\s+?(\d+)\n"
%(queue), snapshot_2)
match_1 = re.search(r"Drop Counters[^!]+?(%d)\s+?(\d+)\s+?(\d+)\s+?(\d+)\s+?(\d+)\s+?(\d+)\n"
%(queue), snapshot_1)
for threshold in range (2, 7):
if match_2: Regex to match
drop_count_2 = int(match_2.group(threshold)) desired pattern
if match_1:
drop_count_1 = int(match_1.group(threshold))
if match_2 and match_1: Check if drop counter
if drop_count_2 > drop_count_1: incremented between snapshots
intf_drop_trafffic.append(up_interface)
if threshold < 5:
cli.execute("send log" + " Drop-Th%d is dropping traffic in queue %d. Drop count is
%d. Check 'show platform hard fed sw active qos queue stats interface %s'" % (threshold-2, queue,
drop_count_2, up_interface))
elif threshold == 5:
cli.execute("send log" + " SBufDrop is incrementing queue %d. Drop count is %d. Check
'show platform hard fed sw active qos queue stats interface %s'." % (queue, drop_count_2, up_interface))
elif threshold == 6:
cli.execute("send log" + " QebDrop is incrementing in queue %d. Drop count is %d.
Check 'show platform hard fed sw active qos queue stats interface %s'." % (queue, drop_count_2,
up_interface))
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Continue your education
Demos in the
Walk-In Labs
Cisco Showcase
BRKRST-2600 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Thank you