ipv4/ipv6 TCAM / TRIE resources on Nexus

[marcin.pietrkiewicz]

Hi all,

I have question regarding Nexus 9300

What is a difference between TCAM routes and trie routes that are presented in "show hardware internal forwarding table utilization" command.

Max IPv4 Trie route entries: 2000000
Max IPv6 Trie route entries: 628224
Max TCAM table entries : 24576
Max V4 Ucast TCAM table entries : 12288
Max V6 Ucast TCAM table entries : 4096

..

Percentage utilization of IPv4 trie routes : 40.30
Percentage utilization of IPv6 trie routes : 28.80
Percentage utilization of IPv4 TCAM routes : 99.88
Percentage utilization of IPv6 TCAM routes : 97.75
Percentage utilization of Nexthop entries : 0.61

1. Does both "TCAM routes" and "TRIE routes" refer to forwarding routes/prefixes and not for routing purposes?

2. if yes, when exactly "TCAM routes" region is utilized and when "trie routes" region is utilize? Why with regular BGP table and internet peering mode "TCAM routes" max-out whilst "trie routes" is half empty. Can it be due to some other configuration (eg. heavy utilization of SVI etc.)?

3. Are both of them (trie/TCAM) separated from other TCAM resoures (QoS,ACL,Netflow,...)?

Thanks in advance,
Marcin

[Joseph W. Doherty]

It might be difficult to find someone able and willing to answer all your questions as some might be considered proprietary information.  As best, you might only find Cisco public information for troubleshooting high TCAM utilization.

No support contract for this Nexus 9300?

As you mention Internet peering, this Nexus is working with a large number of Internet routes?  (If so, that alone might be problematic as I don't believe the Nexus series was designed for that.)

[marcin.pietrkiewicz]

Thanks, Joseph, for your response. According to the Cisco Nexus 9000 Series NX-OS Verified Scalability Guide, Release 10.5(2)F - Cisco

Nexus 9300-FX3/H2R/H1/GX/GX2 switches support up to 2,000,000 IPv4 LPM routes and 628,224 IPv6 LPM routes. I tested with routes with around 800k-900k IPv4 routes, I understand that there are better options for handling full Internet routing tables. However, my goal is to determine the actual limit in this scenario and to understand how many prefixes I can effectively forward.

[Joseph W. Doherty]

Skimming that reference, your limits appear to be in the unidimensional section, which probably are pointless in most real world cases.

The multi dimensional section values are more likely better, and they appear to be much smaller values.

This looks like a case of "your mileage might vary".

I'm assuming you're concerned about the high TCAM utilization values.  I would be too.  Possibly your configuration could be tweaked to better optimize hardware resources, but I suspect it may boil down to this Nexus isn't the ideal device for how you're using it.

@Ramblin Tech anything you could contribute?

[Ramblin Tech]

Hey @Joseph W. Doherty / @marcin.pietrkiewicz 

Some Nexus products have been built on merchant Broadcom XGS/DNX and some on Cisco custom SiliconOne NPUs, both of which I had a passing familiarity. It now appears that some NX 9300 & 9500 platforms are built on a different Cisco custom NPU, "Cloud Scale", with which I am not familiar at all, so I hesitate to make any definitive comments about its scale or how its FIB might work (NX 9300-FX3 appears to use Cloud Scale). That said, I suspect that the trie data structure is in some kind of RAM (HBM maybe?) and not in TCAM. Not sure how the TCAM and trie entries are related wrt to route scale, but HBM on some NPUs is used to store overflow table data that does not fit in TCAM.

Anyway... when it comes to determining max scale for this particular use-case, Joe's preference for using multi-dimensional scale numbers is the right way. When Cisco's CVT (Customer Verification Testing) org performs a SIT on a customer use-case to verify scale, they use the customer's intended configuration in a test bed that mimic's the customer's. All that to say: if you want to "determine the actual limit in this scenario and to understand how many prefixes I can effectively forward" then definitely treat this as a SIT and not a unit test. That is, in your scale testing configure everything as you would in production: same number of peers, same route policies, same timer expiry intervals, same ACLs, etc, and then there should only be one free variable, the route scale for you to determine for this use-case.

[Giuseppe Larosa]

Hello @marcin.pietrkiewicz ,

@Joseph W. Doherty is right a Nexus 9300 is not the right device to manage a BGP internet peering when you receive a full Internet table with 880,000 prefixes.

TRIE if I remember correctly is the name given to an algorythm used to find the longest prefix match in the IP routing table.

if the TCAM is full you have found the upper limit in the data plane for your device and there is no process switching fallback.

LPM and host routes should refer to BGP EVPN scenarios.

Hope to help

Giuseppe