TCAM and DRAM usage on ASR 1001-HX. Need TCAM calculation formula

[Azimbekovich]

Hello! Sorry in advance for long text...
I got few questions about TCAM allocation on Cisco ASR 1001-HX Router.
I'm trying to resolve task that requires to know exact formula of TCAM calculation on 1001-HX basically.
My goal is to create an ACL used in NAT rule and ACEs in it could be very different (like in examples below).
I want to use TCAM only, but if it's not possible I want to know when the router be out of TCAM and use DRAM instead.

Here's my questions
1) Is anyone knows the formula where I can put all initial values of ACE like qty of SRC, DST, ports, etc in separate OGs and get correct result of TCAM usage on ASR 1001-HX?
I would like to get or develop something like this https://github.com/grindelwaldus/Cisco-TCAM-usage-calculator/tree/master but firstly I need a formula.
2) How to handle with DRAM leaking? I can't reboot ASR in production environment if something simullar would occur.
3) Where I can read comprehensive information about memory utilization on ASR 1000 series? I found only this presentation and it's not answering my questions:
https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKARC-3147.pdf 

Starting point
ASR1001HX#sh object-group - no object-groups on device
ASR1001HX#show platform resources slot r0 - DRAM utilization
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 2.53% 100% 80% 90% H
DRAM 3808MB(24%) 15828MB 88% 93% H
bootflash 3074MB(11%) 26686MB 88% 93% H

ASR1001HX#show platform hardware qfp active tcam resource-manager usage - looking on TCAM usage
QFP TCAM Usage Information

80 Bit Region Information
--------------------------
Name : Leaf Region #0
Number of cells per entry : 1
Current 80 bit entries used : 0
Current used cell entries : 0
Current free cell entries : 0

160 Bit Region Information
--------------------------
Name : Leaf Region #1
Number of cells per entry : 2
Current 160 bits entries used : 4
Current used cell entries : 8 - something already allocated but I don't know what it is and it doesn't matter
Current free cell entries : 4086

320 Bit Region Information
--------------------------
Name : Leaf Region #2
Number of cells per entry : 4
Current 320 bits entries used : 0
Current used cell entries : 0
Current free cell entries : 0

Here's my object-groups (OGs)
! Define Object-Groups
object-group network SRC_OG1
description Source Network Group 1
host 10.0.1.1
10.0.2.0 255.255.255.0
host 10.0.3.3
10.0.4.0 255.255.255.0
host 10.0.5.5

object-group network SRC_OG2
description Source Network Group 2
10.0.6.0 255.255.255.0
host 10.0.7.7
10.0.8.0 255.255.255.0
host 10.0.9.9
10.0.10.0 255.255.255.0

object-group network DST_OG1
description Destination Network Group 1
host 172.16.1.1
host 172.16.1.2
host 172.16.1.3
host 172.16.1.4
host 172.16.1.5

object-group network DST_OG2
description Destination Network Group 2
host 172.16.2.1
host 172.16.2.2
host 172.16.2.3
host 172.16.2.4
host 172.16.2.5

object-group service PORT_OG1
description Port Group 1
tcp-udp eq 100
tcp-udp eq 101
tcp-udp eq 102
tcp-udp eq 103

object-group service PORT_OG2
description Port Group 2
tcp-udp eq 200
tcp-udp eq 201
tcp-udp eq 202
tcp-udp eq 203
tcp-udp eq 204
tcp-udp eq 205
tcp-udp eq 206
tcp-udp eq 207
tcp-udp eq 208
tcp-udp eq 209
tcp-udp eq 210
tcp-udp eq 211
tcp-udp eq 212
tcp-udp eq 213
tcp-udp eq 214
tcp-udp eq 215
tcp-udp eq 216
tcp-udp eq 217
tcp-udp eq 218
tcp-udp eq 219
tcp-udp eq 220

Results of my practical research about TCAM usage. ###RULE +X (TCAM usage increase). All of ACEs below eventually allocated to Leaf Region #1 TCAM
###1 +2
ip access-list extended ACL_CUSTOM - empty ACL (with implicit deny)
###2 +2
! 1 SRC & 1 DST no ports
permit ip host 10.0.1.10 host 172.16.1.10
###3 +2
! ANY SRC & 1 DST no ports
permit ip any host 172.16.1.11
###4 +2
! 1 SRC & ANY DST no ports
permit ip host 10.0.2.10 any
###5 +2
! 1 SRC & 1 DST with 1 port
permit tcp host 10.0.3.10 host 172.16.1.12 eq 80
###6 +2
! 1 SRC & 1 DST range of 2 ports
permit tcp host 10.0.4.10 host 172.16.1.13 range 100 101
###7 +2
! 1 SRC & 1 DST range of 4 ports
permit tcp host 10.0.5.10 host 172.16.1.14 range 200 203
###8 +2
! 1 SRC & 1 DST range of 10 ports
permit tcp host 10.0.6.10 host 172.16.1.15 range 300 309
###9 +2
! 1 SRC & 1 DST range of 21 ports
permit tcp host 10.0.7.10 host 172.16.1.16 range 400 420
###10 +10
! 5 SRC OG & 1 DST with 1 port
permit tcp object-group SRC_OG1 host 172.16.1.17 eq 443
###11 +10
! 5 SRC OG & 1 DST range of 4 ports
permit tcp object-group SRC_OG1 host 172.16.1.18 range 500 503
###12 +40
! 5 SRC OG & 1 DST range of 21 ports
permit tcp object-group SRC_OG1 host 172.16.1.19 range 600 620
###13 +6
! 1 SRC & 5 DST OG with 1 port
permit tcp host 10.0.8.10 object-group DST_OG1 eq 8080
###14 +6
! 1 SRC & 5 DST OG range of 4 ports
permit tcp host 10.0.9.10 object-group DST_OG1 range 700 703
###15 +18
! 1 SRC & 5 DST OG range of 21 ports
permit tcp host 10.0.10.10 object-group DST_OG1 range 800 820
###16 +24
! 5 SRC OG & 5 DST OG with 1 port
permit tcp object-group SRC_OG2 object-group DST_OG2 eq 22
###17 +24
! 5 SRC OG & 5 DST OG range of 4 ports
permit tcp object-group SRC_OG2 object-group DST_OG2 range 900 903
###18 +96
! 5 SRC OG & 5 DST OG range of 21 ports
permit tcp object-group SRC_OG2 object-group DST_OG2 range 1000 1020
###19 +4
! 1 SRC & 1 DST with 4 ports OG
permit object-group PORT_OG1 host 10.0.1.20 host 172.16.2.6
###20 +16
! 1 SRC & 1 DST with 21 ports OG
permit object-group PORT_OG2 host 10.0.2.20 host 172.16.2.7
###21 +10
! 5 SRC OG & 1 DST with 4 ports OG
permit object-group PORT_OG1 object-group SRC_OG1 host 172.16.2.8
###22 +80
! 5 SRC OG & 1 DST with 21 ports OG
permit object-group PORT_OG2 object-group SRC_OG1 host 172.16.2.9
###23 +12
! 1 SRC & 5 DST OG with 4 ports OG
permit object-group PORT_OG1 host 10.0.3.20 object-group DST_OG2
###24 +48
! 1 SRC & 5 DST OG with 21 ports OG
permit object-group PORT_OG2 host 10.0.4.20 object-group DST_OG2
###25 +48
! 5 SRC OG & 5 DST OG with 4 ports OG
permit object-group PORT_OG1 object-group SRC_OG2 object-group DST_OG1
###26 +192
! 5 SRC OG & 5 DST OG with 21 ports OG
permit object-group PORT_OG2 object-group SRC_OG2 object-group DST_OG1

Strange thing with ACEs containing ports in OG
ACEs with ports in object-group affect TCAM only after NAT re-activation (delete & add NAT rule):
ASR1001HX(config)#ip access-list extended ACL_CUSTOM
ASR1001HX(config-ext-nacl)#permit object-group PORT_OG1 host 10.0.1.20 host 172.16.2.6 - ACE added
ASR1001HX(config-ext-nacl)#do show platform hardware qfp active tcam resource-manager usage - looking on TCAM usage
QFP TCAM Usage Information

80 Bit Region Information
--------------------------
Name : Leaf Region #0
Number of cells per entry : 1
Current 80 bit entries used : 0
Current used cell entries : 0
Current free cell entries : 0

160 Bit Region Information
--------------------------
Name : Leaf Region #1
Number of cells per entry : 2
Current 160 bits entries used : 5
Current used cell entries : 10 - no difference
Current free cell entries : 4086

320 Bit Region Information
--------------------------
Name : Leaf Region #2
Number of cells per entry : 4
Current 320 bits entries used : 0
Current used cell entries : 0
Current free cell entries : 0

ASR1001HX(config-ext-nacl)#exit
ASR1001HX(config)#$nside source list ACL_CUSTOM interface TenGigabitEthernet0/1/0.49 overload - delete NAT rule
ASR1001HX(config)#ip nat inside source list ACL_CUSTOM interface TenGigabitEthernet0/1/0.49 overload - add same NAT rule back
ASR1001HX(config)#do show platform hardware qfp active tcam resource-manager usage - looking on TCAM usage again
QFP TCAM Usage Information

80 Bit Region Information
--------------------------
Name : Leaf Region #0
Number of cells per entry : 1
Current 80 bit entries used : 0
Current used cell entries : 0
Current free cell entries : 0

160 Bit Region Information
--------------------------
Name : Leaf Region #1
Number of cells per entry : 2
Current 160 bits entries used : 7
Current used cell entries : 14 - counter increased
Current free cell entries : 4082

320 Bit Region Information
--------------------------
Name : Leaf Region #2
Number of cells per entry : 4
Current 320 bits entries used : 0
Current used cell entries : 0
Current free cell entries : 0

Total TCAM Cell Usage Information
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries : 14
Total tcam free cell entries : 1048562
Threshold status : below critical limit

Looking on processes that use TCAM:
ASR1001HX#show platform resources tcam sorted
TCAM Usage Information

Total cells in TCAM: 1048576
Free cells in TCAM: 1048146

CG-Id Name Client 160bitVMR 320bitVMR Total Cell Total% Label
---------------------------------------------------------------------------------------------------------------
nat:1001 --- NAT 211 0 422 0 46

CACE – TCAM offload
• When falling back to DRAM, the classification is done by CACE
    • CACE = Classification Engine
    • A slower, software version of the TCAM – little measurable impact in practice
• When a policy is moved to the CACE, it will not fall back to TCAM - Is it my case?
• Supplemental classification engine:
    • Uses DRAM resources
    • Costs QFP resources when classifying packets
    • May or may not be perceptible…

I've generated massive ACL (500001 entrys including implicit deny) with ACEs like that: permit tcp host 10.0.0.1 host 172.16.0.1 eq 80 (just a sample).
Each ACE will consume 2 cells of TCAM. This ACL used in NAT rule and looks like it utilizes DRAM instead of TCAM:
ASR1001HX#show platform resources slot r0
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
RP0 (ok, active) H
Control Processor 14.70% 100% 80% 90% H
DRAM 8819MB(55%) 15828MB 88% 93% H
bootflash 3096MB(11%) 26686MB 88% 93% H

I thought CACE is the case, but I got nothing in the output:
ASR1001HX#show platform resources exmem | in CACE
ASR1001HX#

TCAM also not used:
ASR1001HX#show platform hardware qfp active tcam resource-manager usage
QFP TCAM Usage Information

80 Bit Region Information
--------------------------
Name : Leaf Region #0
Number of cells per entry : 1
Current 80 bit entries used : 0
Current used cell entries : 0
Current free cell entries : 0

160 Bit Region Information
--------------------------
Name : Leaf Region #1
Number of cells per entry : 2
Current 160 bits entries used : 4
Current used cell entries : 8
Current free cell entries : 8184

320 Bit Region Information
--------------------------
Name : Leaf Region #2
Number of cells per entry : 4
Current 320 bits entries used : 0
Current used cell entries : 0
Current free cell entries : 0

Total TCAM Cell Usage Information
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries : 8
Total tcam free cell entries : 1048568
Threshold status : below critical limit

Summary
After all I did above, I decided to delete ACL with 500001 ACEs and create object-groups and rules that 100% used TCAM before (the 26 rules mentioned previously).
They didn't use TCAM anymore and honestly I don't know what they're using, because after deletion of the huge ACL DRAM usage didn't decrease (still 55%).
I tried to delete NAT rule in case that ACL cached (it couldn't be or am I wrong?), but it didn't work. Am I supposed to reboot device because of that or some mighty configuration command exists to resolve this issue?

[rostanchase14]

When working with TCAM allocation on the Cisco ASR 1001-HX, the task can get quite complex, especially when trying to contador de horas predict exactly when you'll run out of TCAM and start dipping into DRAM. Unfortunately, there's no single formula provided by Cisco that covers all scenarios, as TCAM usage depends on the specifics of your ACLs, NAT rules, and object groups.

From my experience, each ACE can consume a different number of TCAM entries depending on factors like the number of source and destination IPs, ports, and whether you're using object groups. For example, a simple permit statement with a single source and destination IP might use only 2 TCAM entries, but when you introduce object groups or port ranges, the usage can multiply significantly.

One approach to estimate TCAM usage is to perform practical testing, as you've done. This can give you a rough idea of how much TCAM each type of ACE will consume. However, if you need a more precise method, you could consider using tools or scripts, like the one mentioned in your question, that simulate TCAM usage.

As for handling DRAM leakage or when your router starts using DRAM instead of TCAM, monitoring tools and regular checks (show platform resources slot r0, show platform hardware qfp active tcam resource-manager usage) are your best friends. Unfortunately, in some cases, a reboot might be necessary to clear out stuck memory, but I'd recommend exhausting all other options first.