LPTS and ASR9K Notes

jaisilva

© 2023 Cisco and/or its affiliates. All rights reserved.

reserved. Cisco
Cisco Confidential
Confidential

Cisco Confidential
LPTS checks
RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware police location 0/0/CPU0
Tue Nov 7 11:07:13.496 UTC
-------------------------------------------------------------
Node 0/0/CPU0: Drops are observed when rates
-------------------------------------------------------------
Each Entry matches one of the out-of-the-box Burst = 100ms for all flow types
are surpassed
Flow Types, entries are then grouped and treated -------------------------------------------------------------
under its matching Policer. FlowType Policer Type Cur. Rate Def. Rate Accepted Dropped TOS Value
---------------------- ------- ------- ---------- ---------- -------------------- -------------------- ----------
<snip>
HSRP 149 Local 2000 400 2411247773 5268586 01234567
<snip>
SNMP 125 Static 300 300 11978291 147 01234567
<snip>
BGP-known 106 Static 2500 2500 67632635 0 01234567
BGP-cfg-peer 107 Static 2000 2000 7113504 434 01234567
BGP-default 108 Static 1500 1500 73967562 16 01234567
RP/0/RSP0/CPU0:IOS-XR#
If Hardware usage is in constant high usage, it is
good to then validate which type of entries are RP/0/RSP0/CPU0:IOS-XR# sh lpts pifib hardware usage location 0/0/0
Tue Nov 7 08:28:39.846 UTC
predominant, later it is good to validate TCAM
overall usage and lately to consider restrict the Node: 0/0/CPU0:
amount of allowed Entries per flow. ----------------------------------------
Type Region ID Size Used Used(%)
---------- ---------- --------------- --------------- -------
IPv4 1 4997 4997 100.00
This commands shows how many entries are IPv6 1 599 599 100.00

found in 0/0/0, later a comparison on how many RP/0/RSP0/CPU0:IOS-XR#

of those 5611 are related to the predominant Flow
RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware entry brief location 0/0/CPU0 | b Offset | u wc -l
type(TCP/UDP Port) Wed Nov 8 10:52:18.175 UTC
5611
RP/0/RSP0/CPU0:IOS-XR#

Here, almost 45% of the total are RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware entry brief location 0/0/CPU0 | b Offset | u egrep ",161 " | u wc -l
Wed Nov 8 10:52:29.236 UTC
SNMP entries 1036
RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware entry brief location 0/0/CPU0 | b Offset | u egrep ",162 " | u wc -l
Wed Nov 8 10:52:48.433 UTC
1034
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
RP/0/RSP0/CPU0:IOS-XR#

Cisco Confidential
LPTS Entry deep dive
RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware entry statistics location 0/0/CPU0
Tue Nov 7 09:31:09.523 UTC

Node: 0/0/CPU0:
---------------------------------------- LU: Punt to active RP
L3 - L3 Protocol;L4 - Layer4 Protocol; Intf - Interface; LM: Punt to both active and Standby RP, used for
Dest - Destination Node;
LU - Local chassis fabric unicast; BGP-known when both RPs are installed and NSR
LM - Local chassis fabric multicast; Destination is the Local is enabled (there are other use-cases)
RU - Multi chassis fabric unicast; Node to where this packets
RM - Multi chassis fabric multicast; are punted, a RSP for
na - Not Applicable or Not Available example.
Offset L3 VRD id L4 Intf Dest Pkts/Drops laddr,Port raddr,Port acl name
------ ---- ------------ ------ --------------- ----------- ---------------- --------------------- -----------------------------------
<..SNIP>
1100 IPV4 000014EE TCP any LM[30] 33791/0 any,179 192.168.177.172,53135
1101 IPV4 0000128C TCP any LM[30] 32999/0 any,179 172.17.2.92,19653
1102 IPV4 00001414 TCP any LM[30] 32776/0 any,179 192.168.232.12,56262
1103 IPV4 000012AF TCP any LM[30] 189724/0 any,179 172.31.8.92,63155 RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware entry l4protocol udp location 0/0/CPU0
1104 IPV4 000011E8 TCP any LM[30] 185079/0 any,179 192.168.108.20,40069 <..SNIP..>
1105 IPV4 00001349 TCP any LM[30] 27237/0 any,179 192.168.145.76,26431 ----------------------------------------------------
1106 IPV4 0000102E TCP any LM[30] 0/0 any,179 192.168.243.124,65385VRF ID : 0x60000000
Destination IP : any M – Fabric Multicast
1107 IPV4 00001075 TCP any LM[30] 21274/0 any,179 192.168.120.92,12703 Source IP : any L – Listener Tag
<..SNIP> Is Fragment : 0 T – Min TTL
3191 IPV4 default UDP any LU(30) 11918356/147 any,161 any,any Interface : any F – Flow Type
M/L/T/F : 0/IPv4_LISTENER/0/SNMP
3192 IPV4 00001010 UDP any LU(30) 0/0 any,161 any,any DestNode : 48
DestNode: Destination Node
3193 IPV4 default UDP any LU(30) 33670/0 any,162 any,any DestAddr : 48 DestAddr – Destination Fabric queue
3194 IPV4 00001010 UDP any LU(30) 0/0 any,162 any,any SID : 9 SID – Stream ID
3195 IPV4 00001012 UDP any LU(30) 0/0 any,161 any,any L4 Protocol : UDP Po – Policer
Source port : Port:any HAr - Hardware Average rate limit
3196 IPV4 00001012 UDP any LU(30) 0/0 any,162 any,any Destination Port : 161 HBu - Hardware Burst
3197 IPV4 00001015 UDP any LU(30) 0/0 any,161 any,any Ct : 0x36588 Cir - Committed Information rate in HAL
3198 IPV4 00001015 UDP any LU(30) 0/0 any,162 any,any Accepted/Dropped : 0/0
Lp/Sp : 1/255
3199 IPV4 00001017 UDP any LU(30) 0/0 any,161 any,any # of TCAM entries : 1
3200 IPV4 00001017 UDP any LU(30) 0/0 any,162 any,any HPo/HAr/HBu/Cir/acl: 460403/300pps/150ms/300pps/
3201 IPV4 00001018 UDP any LU(30) 0/0 any,161 any,any State : Entry in TCAM
Rsp/Rtp : 3189/3204
3202 IPV4 00001018 UDP any LU(30) 0/0 any,162 any,any

This entry is not

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential receiving any traffic.
Cisco Confidential
Configure specific interfaces in MPP

Details of CSCtn85097 Chronicle of a problem Foretold

• Symptom is LPTS High utilization 1. Since, all interfaces and sub-interfaces are
• Conditions: considered “enabled” for all MGMT protocols,
• No MPP configuration OR MPP with LPTS opens a “SNMP Listener Flow” for each
interface all interface/VRF.
• SNMP enabled 2. Each Flow consumes a TCAM/LPTS entry.
• Several VRFs / Sub-interfaces configured 3. Previous condition in Line Cards with high
in the LC service utilization will swamp TCAM/LPTS
• Impact may vary, it may cause resources
BGP/HSRP/OSPF session flaps, or LPTS Packet 4. Now, in unusual conditions such as abusive
drops. SNMP requests, quickly flapping
BGP/HSRP/OSPF sessions, because LPTS
resources are short, services will start
behaving erratically.

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
 Configure specific interfaces in MPP (cont.)
Replace interface all by specific interface, this will fix(workaround) the tcam SNMP exhaustion.
By specifying an interface (or
several) any not-defined-
Found today in the Network Workaround. interface is prevented from
processing management
control-plane control-plane protocols requests, from
management-plane Some devices have no management-plane opening LPTS entries for SNMP
inband configurations related to inband Listener Flows.
CPP/MPP
interface all interface GigabitEthernet0/0/0/1
allow SSH peer allow SSH peer
address ipv4 <ip> address ipv4 <ip>
address ipv4 <ip2> !
allow SNMP peer
address ipv4 <ip2>
interface all enable VRF
!
interfaces as well
allow Telnet peer
address ipv4 <ip3>
!
Note: Keep in mind
▪ SNMP Process may be restarted via process restart snmpd command ▪ This behavior is addressed in the defect CSCtn85097
to release LPTS bindings (TCAM entries release).
Note
▪ Apparently applying an ACL have had similar results for MEX335AR10
but such workaround have not been tested internally.
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
About TCAM and A9K LC
The TCAM has a fixed size per line-card type.
Trident Cards
• -E Trident cards have 256k 144bit entries PID Description LC Gen
• -B Trident cards have 128k 144bit entries A9K-4T-L 4-Port 10GE Low Queue Line Card Trident
• -L Trident cards have 64k 144bit entries A9K-2T20GE-L 2-Port 10GE, 20-Port GE Low Queue Combo Line Card Trident
Typhoon Cards A9K-2T20GE-B 2-Port 10GE, 20-Port GE Medium Queue Combo Line Card Trident
A9K-MOD80-SE ASR 9000 Mod80 Modular Line Card, Service Edge Optimized Typhoon
• -TR Typhoon cards have 24k 160bit entries
A9K-MOD80-TR ASR 9000 Mod80 Modular Line Card, Packet Transport Optimized Typhoon
• -SE Typhoon cards have 96k 160bit entries ASR-9001 ASR-9001 Fixed chassis Typhoon
These are lab
one-dimensional Note: Hardware found in Network Profile
calculations.
Trident LCs got LDoS on May 31, 2020
Typhoon LCs get LDoS on May 31, 2024

Each v6 entry consumes 4 times the TCAM bits compared to v4 (640

bits vs 160 bits)

For example; from the ~24K TCAM entries A9K-MOD80-TR has RP/0/RSP0/CPU0:ios(admin-config)#hw-module profile tcam ?
available, ~20K are for v4 and ~4K for v6. default Default tcam partitions ods2:ods8 to 60:40
Estimates needs to be done because not all entries are the same tcam-part-30-70 Set tcam partitions ods2:ods8 to 30:70
size. tcam-part-40-60 Set tcam partitions ods2:ods8 to 40:60
tcam-part-50-50 Set tcam partitions ods2:ods8 to 50:50
By default, the hardware is configured to use on a 60:40 relation the tcam-part-70-30 Set tcam partitions ods2:ods8 to 70:30
v4:v6 entries.

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
LPTS Overview
LPTS is an application
LPTS is used to reliably route the LPTS has (default) Hardware policers in Incoming
control/management plane packets to Line Cards to limit traffic sent to local or packet
the right node/application. LPTS has an remote nodes
"internal" FIB or iFIB that directs certain LPTS entries in TCAM classifies packets
packets to various nodes. to select a policer to apply:
- Polices on protocol (BGP, OSPF, SSH) Packet received on the
IOS-XR can handle certain traffic on the and flow state (BGP established, BGP interface first go FIB
line card (such as BFD, Net flow and configured, and BGP listen). through FIB look up.
ARP) and these packets are instructed by - Policing done on the LC Hardware
LPTS to be handled by the local CPU ASIC before packets hit RP/LC CPU
rather than the RSP CPU.
Next No
For
Hop Us?

Yes

LPTS IFIB look up, based

on the look up result
packet will be punted to iFIB
applications running on
RSPs or LCs.

Keep in Mind
• LPTS: Local Packet Transport Services
• IFIB: Internal Forwarding Information Base
• FIB: Forwarding Information Base
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
LPTS Resources

• Punt Rate Policing • LPTS TCAM space

RP/0/RSP0/CPU0:IOS-XR#show lpts pifib hardware police location 0/0/CPU0 RP/0/RSP0/CPU0:IOS-XR# sh lpts pifib hardware usage location 0/0/0
Tue Nov 7 11:07:13.496 UTC
-------------------------------------------------------------
Tue Nov 7 08:28:39.846 UTC
Node 0/0/CPU0:
------------------------------------------------------------- Node: 0/0/CPU0:
Burst = 100ms for all flow types ----------------------------------------
-------------------------------------------------------------
FlowType Policer Type Cur. Rate Def. Rate Accepted Dropped TOS Value Type Region ID Size Used Used(%)
---------------------- ------- ------- ---------- ---------- -------------------- -------------------- ---------- ---------- ---------- --------------- --------------- -------
<snip> IPv4 1 4997 4997 100.00
HSRP 149 Local 2000 400 2411247773 5268586 01234567
<snip>
IPv6 1 599 599 100.00
SNMP 125 Static 300 300 11978291 147 01234567
<snip> RP/0/RSP0/CPU0:IOS-XR#
BGP-known 106 Static 2500 2500 67632635 0 01234567
BGP-cfg-peer 107 Static 2000 2000 7113504 434 01234567
BGP-default 108 Static 1500 1500 73967562 16 01234567
RP/0/RSP0/CPU0:IOS-XR#

• IOS-XR, out-of-the-box sets Default Rates to each pre-defined Flow • LPTS shares the TCAM space with other applications
• Default Rate is tunable on a Linecard basis • Limited by Hardware, not tunable.
• The TCAM Space relation between IPv4 and IPv6 is tunable.
• TCAM entries can be limited to a certain number.

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
Useful commands

RP/0/RSP0/CPU0:ios#show prm server tcam sum all all all location 0/4/CPU
<SNIP>
TCAM summary for NP0:

TCAM Logical Table: TCAM_LT_L2 (1)

Use show prm server tcam sum all all all location to validate TCAM Partition ID: 0, priority: 2, valid entries: 7, free entries: 2041
application utilization. Partition ID: 1, priority: 2, valid entries: 0, free entries: 2048
NP_APP_ID_IFIB is the application ID for LPTS Resource allocation Partition ID: 2, priority: 1, valid entries: 0, free entries: 2048
Partition ID: 3, priority: 1, valid entries: 28, free entries: 24548
This command requires the cisco-support task group READ attribute Partition ID: 4, priority: 0, valid entries: 175, free entries: 67409
TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 78536, resvd 128
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_IFIB (0)
Total: 1 vmr_ids, 19008 active entries, 19008 allocated entries.

If required, use lpts pifib hardware dynamic-flows location RP/0/RP0/CPU0:8201-1(config)#$flows location 0/0/0 flow snmp max ?
<0-4294967295> TCAM Entries NUM
configuration command to set the number of entries a specific flow can use RP/0/RP0/CPU0:8201-1(config)#$flows location 0/0/0 flow snmp max

You can use the following commands to validate the number of entries and
compare the specific usage of a certain Port:

sh int desc | e down | e \\. | i "Gi|Te" | u wc -l !<-counts the number of physical interfaces on the whole device
sh int desc | u egrep -v down | i "0/0/0" | i \\. | u wc -l !<-counts the number of sub0interfaces on a specific LC
sh lpts pifib hardware usage location 0/0/0 !<- Checks LPTS/TCAM utilization
show lpts pifib hardware entry brief location 0/0/0 | u wc -l !<-Counts the number of total entries
show lpts pifib hardware entry brief location 0/0/0 | i IPV6 | u wc -l !<-Counts the number of IPV6 Entries
show lpts pifib hardware entry brief location 0/0/0 | i IPV6 | u egrep ",161" | u wc -l !<-Counts the number of IPV6/SNMP(UDP 161) Entries
show lpts pifib hardware entry brief location 0/0/0 | i IPV6 | u egrep ",162" | u wc -l !<-Counts the number of IPV6/SNMP(UDP 162) Entries

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
LPTS

Notes

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential
Quick LC Memories Overview

NP
RLDRAM TCAM
- Lookup/Search - Classification (Vlan
Memory Tag (EVCs), QoS and Each NP has four main memories:
- Stores MAC, FIB, Security ACL) • Lookup/Search Memory (RLDRAM): Stores MAC, FIB, and Adjacencies
and Adjacencies Tables
Tables • TCAM: Classification (Vlan Tag (EVCs), QoS and Security ACL)
• Stats QDR memory: Interface and forwarding statistics, policers data, etc
• Frame memory: Buffer memory for Queues

In ASR9Ks, the primary applications that TCAMs are used for are:
1. Matching packets to an interface
2. Classifying iFIB/for-us traffic
Stats QDR memory Frame memory
3. QoS class maps
- Interface and - Buffer memory for
4. Security ACLs
forwarding statistics, Queues
policers data, etc
TCAM, QDR and Frame memory sizes depend on LC version
The size of them affects number of QoS queues and L2 sub-
interfaces supported

© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Confidential

Cisco Confidential