Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
3457
Views
55
Helpful
13
Replies

IPv4/IPv6 TCAM Routes

axelhauguel
Level 1
Level 1

‎05-06-2021 08:04 AM

Dear all,

 

I'have a problem with my router. Some routes are stucked in BGP, my neighbor do not announce but my router always announce the network.

 

Also, i'have some errors in log and I want to know how to : 

 

Percentage utilization of IPv4 native host routes : 2.63   
Percentage utilization of IPv6 native host routes : 2.19   
Percentage utilization of IPv6 ND/local routes : 2.19   
Percentage utilization of IPv6 host /128 learnt routes : 0.00   
Percentage utilization of IPv4 trie routes : 82.15  
Percentage utilization of IPv6 trie routes : 21.60  
Percentage utilization of IPv4 TCAM routes : 99.82  
Percentage utilization of IPv6 TCAM routes : 94.92  
Percentage utilization of nexthop entries : 3.74

 

How to adjust "Percentage utilization of IPv4 TCAM routes " ? Seem full, but If I check system ressources, it's okay : 

 

 

 PAR-N9K-1(config)# sh resource 

     Resource                   Min       Max       Used      Unused    Avail    
     --------                   ---       ---       ----      ------    -----    
     vlan                       16        4094      23        0         4071     
     vrf                        2         4096      2         0         4094     
     port-channel               0         511       1         0         510      
     u4route-mem                768       768       127       641       641      
     u6route-mem                512       512       31        481       481      
     m4route-mem                58        58        1         57        57       
     m6route-mem                8         8         1         7         7        

PAR-N9K-1(config)#

 

 

 

 

vdc PAR-N9K-1 id 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 511
  limit-resource u4route-mem minimum 768 maximum 768
  limit-resource u6route-mem minimum 512 maximum 512
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8

 

 

 

PAR-N9K-1(config)# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or 
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or 
Lesser General Public License (LGPL) Version 2.0. 
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.

Software
  BIOS: version 05.39
 NXOS: version 9.3(3)
  BIOS compile time:  08/30/2019
  NXOS image file is: bootflash:///nxos.9.3.3.bin
  NXOS compile time:  12/22/2019 2:00:00 [12/22/2019 16:00:37]


Hardware
  cisco Nexus9000 C93180YC-FX Chassis 
  Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 65808192 kB of memory.
  Processor Board ID FDO22300362

  Device name: PAR-N9K-1
  bootflash:  115805356 kB
Kernel uptime is 4 day(s), 1 hour(s), 6 minute(s), 20 second(s)

Last reset at 594450 usecs after Sun May  2 15:57:19 2021
  Reason: Reset Requested by CLI command reload
  System version: 9.3(3)
  Service: 

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):
	
PAR-N9K-1(config)#

 

 

 

Lot of errors in logging :

 

PAR-N9K-1(config)# sh logging last 10
2021 May  6 17:05:16 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:17 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a06:e881:2606::/48, flags:0x0, intf:0x10041c, Error: Hw Trie full(201)
2021 May  6 17:05:18 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:19 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:19 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:20 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:21 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:21 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:22 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c0:193::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:22 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:53c0:2000::/36, flags:0x0, intf:0x100040, Error: Hw Trie full(201)
PAR-N9K-1(config)#

Anyone have a magic command to fix all errors ?

 

Thanks !

 

Axel 

0 Helpful
13 Replies 13

Harold Ritter
Spotlight
Spotlight

‎05-06-2021 10:25 AM

Hi Axel,

 

"show resource" is for route memory utilization, not for TCAM utilization.

"show hardware internal forwarding table utilization" is for TCAM utilization and it looks like you have issues with the TCAM utilization.

 

Can you provide the output from the following command, to see what profile you are using:

 

sh runn | incl "system routing"

 

Ideally, you should be using the "system routing template-internet-peering".

 

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

axelhauguel
Level 1
Level 1
In response to Harold Ritter

‎05-06-2021 10:52 AM

Hi

 

Thanks for your reply. I already use system routing template : 

 

 

PAR-N9K-1# sh runn | incl "system routing"
system routing template-internet-peering
PAR-N9K-1#

 

0 Helpful

Harold Ritter
Spotlight
Spotlight

‎05-06-2021 12:42 PM

Hi Axel,

 

Two things to consider then.

 

1. Reduce the number of prefixes that you receive from your Service Provider. Are you receiving the full Internet routing table from them. If so, you might want to consider only receiving partial routes.

 

2. Replace the Nexus by something else. The Nexus9k was not originally designed as an Internet edge device and therefore comes with its limitations.

 

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

axelhauguel
Level 1
Level 1
In response to Harold Ritter

‎05-06-2021 02:03 PM

Hi

 

Thanks for your reply. I'have around 880 000 routes.

 

Can you know what is the error "Hw trie full" ? Also strange, because route memory (u4mem) is not full

 

Axel

0 Helpful

Harold Ritter
Spotlight
Spotlight
In response to axelhauguel

‎05-06-2021 02:34 PM

Hi Axel,

 

HW trie is related to TCAM. As I mentioned before, u4mem is routing memory (urib). The issue you are having is with ufib.

 

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

axelhauguel
Level 1
Level 1
In response to Harold Ritter

‎05-07-2021 02:02 AM

Hello Harold,

 

Thanks against for your reply.

 

Small last question. For example, i'have a error for a prefix but If I check route, I see the path. Normal ? Not normal ?

 

 

PAR-N9K-1# sh logging last 10
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 62.24.100.0/23, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 62.24.100.0/24, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.166.0.0/15, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.166.0.0/16, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 197.156.137.0/24, flags:0x0, intf:0x100138, Error:
 Hw Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.135.169.0/24, flags:0x0, intf:0x100138, Error:
 Hw Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 154.123.0.0/16, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:448c::/32, flags:0x0, intf:0x10041c, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:4ea8::/32, flags:0x0, intf:0x10041c, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 45.148.160.0/24, flags:0x0, intf:0x1000d1, Error: 
FIB TCAM FULL For IP Routes(235)
PAR-N9K-1# show bgp ipv4 unicast 154.123.0.0
BGP routing table information for VRF default, address family IPv4 Unicast
BGP routing table entry for 154.123.0.0/18, version 5085384
Paths: (1 available, best #1)
Flags: (0x08001a) (high32 00000000) on xmit-list, is in urib, is best urib route, is in HW

  Advertised path-id 1
  Path type: external, path is valid, received and used, is best path, no labeled nexthop, in rib
  AS-Path: 49434 174 3257 8966 12455 , path sourced external to AS
    100.64.70.5 (metric 0) from 100.64.70.5 (193.178.0.1)
      Origin incomplete, MED not set, localpref 100, weight 0
      Community: 174:21100 174:22008 49434:101 49434:201 49434:301 49434:1000 49434:1001 

  Path-id 1 advertised to peers:
    185.171.202.183    185.171.202.204    185.171.202.246

PAR-N9K-1#

 

0 Helpful

Harold Ritter
Spotlight
Spotlight

‎05-07-2021 07:58 AM

Hi Axel,

 

It is normal to see the path in the BGP table. You will also see the route in the routing table (show ip route <prefix/prefixlen>). You will not see the route in the TCAM (show forwarding ipv4 route <prefix/prefixlen>).

 

Regards,

 

 

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

axelhauguel
Level 1
Level 1
In response to Harold Ritter

‎05-07-2021 07:59 AM

Hi

 

Okay, thanks ! Small last question

 

There is an impact in production to not see route in TCAM ? 

What impact if route is not present in TCAM ?

 

Thanks

0 Helpful

Harold Ritter
Spotlight
Spotlight
In response to axelhauguel

‎05-07-2021 08:46 AM

The Nexus 9k does not do software forwarding for transit traffic (page 121 of the below presentation). So if you don't have a summary route installed in the TCAM (including a default route), traffic to the specific destination for which the error was seen in the log will be dropped.

 

For more information about the Nexus9k architecture, you can refer to the following document:

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKDCT-3101.pdf

 

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

Giuseppe Larosa
Hall of Fame
Hall of Fame
In response to Harold Ritter

‎05-07-2021 09:13 AM

Hello Harold,

it is interesting that the Nexus will drop packets for destinations not installed in the TCAM instead of using software based forwarding.

 

In the past for older platforms like Cat 6500 or Cisco 7600 (Sup720 based )  it was enough to change a linecard and to plug in the new linecard with the wrong daughter card DFC (non XL version ) and the whole system could experience a sudden increase in CPU usage of main CPU caused by process switching of traffic for destinations not stored on that linecard CEF table.

 

Best Regards

Giuseppe

 

 

 

Harold Ritter
Spotlight
Spotlight
In response to Giuseppe Larosa

‎05-07-2021 09:31 AM

Hi Giuseppe,

 

Long time no speak. Hope you are doing well.

 

Software forwarding was indeed supported as a fallback on those platforms, but created a lot of issues, as RP was able to support limited throughput and ended up causing high cpu in most cases. That was probably taken in consideration when designing the Nexus 9k family.

 

Regards,

Regards,
Harold Ritter, CCIE #4168 (EI, SP)

marcin.pietrkiewicz
Level 1
Level 1

‎03-13-2025 03:05 AM

Hello Harold,

What is a difference between TCAM routes and trie routes that are presented in "show hardware internal forwarding table utilization" command.

Max IPv4 Trie route entries: 2000000
Max IPv6 Trie route entries: 628224
Max TCAM table entries : 24576
Max V4 Ucast TCAM table entries : 12288
Max V6 Ucast TCAM table entries : 4096

..

Percentage utilization of IPv4 trie routes : 40.30
Percentage utilization of IPv6 trie routes : 28.80
Percentage utilization of IPv4 TCAM routes : 99.88
Percentage utilization of IPv6 TCAM routes : 97.75
Percentage utilization of Nexthop entries : 0.61

1. Does both "TCAM routes" and "TRIE routes" refer to forwarding routes/prefixes and not for routing purposes?

2. if yes, when exactly "TCAM routes" region is utilized and when "trie routes" region is utilize? Why with regular BGP table and internet peering mode "TCAM routes"  max-out whilst "trie routes" is half empty. Can it be due to some other configuration (eg. heavy utilization of SVI etc.)?

3. Are both of them (trie/TCAM) separated from other TCAM resoures (QoS,ACL,Netflow,...)?

Best regards,

Marcin

0 Helpful

Harold Ritter
Spotlight
Spotlight
In response to marcin.pietrkiewicz

‎03-13-2025 10:16 AM

Hi @marcin.pietrkiewicz ,

I would suggest you open a new post to get assistance on this specific question, as the original post is quite old.

Regards,

Harold

Regards,
Harold Ritter, CCIE #4168 (EI, SP)
0 Helpful
Customers Also Viewed These Support Documents