IPv4/IPv6 TCAM Routes

[axelhauguel]

Dear all,

I'have a problem with my router. Some routes are stucked in BGP, my neighbor do not announce but my router always announce the network.

Also, i'have some errors in log and I want to know how to : 

Percentage utilization of IPv4 native host routes : 2.63   
Percentage utilization of IPv6 native host routes : 2.19   
Percentage utilization of IPv6 ND/local routes : 2.19   
Percentage utilization of IPv6 host /128 learnt routes : 0.00   
Percentage utilization of IPv4 trie routes : 82.15  
Percentage utilization of IPv6 trie routes : 21.60  
Percentage utilization of IPv4 TCAM routes : 99.82  
Percentage utilization of IPv6 TCAM routes : 94.92  
Percentage utilization of nexthop entries : 3.74   

How to adjust "Percentage utilization of IPv4 TCAM routes " ? Seem full, but If I check system ressources, it's okay : 

 PAR-N9K-1(config)# sh resource 

     Resource                   Min       Max       Used      Unused    Avail    
     --------                   ---       ---       ----      ------    -----    
     vlan                       16        4094      23        0         4071     
     vrf                        2         4096      2         0         4094     
     port-channel               0         511       1         0         510      
     u4route-mem                768       768       127       641       641      
     u6route-mem                512       512       31        481       481      
     m4route-mem                58        58        1         57        57       
     m6route-mem                8         8         1         7         7        

PAR-N9K-1(config)# 

vdc PAR-N9K-1 id 1
  limit-resource vlan minimum 16 maximum 4094
  limit-resource vrf minimum 2 maximum 4096
  limit-resource port-channel minimum 0 maximum 511
  limit-resource u4route-mem minimum 768 maximum 768
  limit-resource u6route-mem minimum 512 maximum 512
  limit-resource m4route-mem minimum 58 maximum 58
  limit-resource m6route-mem minimum 8 maximum 8

PAR-N9K-1(config)# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or 
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or 
Lesser General Public License (LGPL) Version 2.0. 
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.

Software
  BIOS: version 05.39
 NXOS: version 9.3(3)
  BIOS compile time:  08/30/2019
  NXOS image file is: bootflash:///nxos.9.3.3.bin
  NXOS compile time:  12/22/2019 2:00:00 [12/22/2019 16:00:37]


Hardware
  cisco Nexus9000 C93180YC-FX Chassis 
  Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 65808192 kB of memory.
  Processor Board ID FDO22300362

  Device name: PAR-N9K-1
  bootflash:  115805356 kB
Kernel uptime is 4 day(s), 1 hour(s), 6 minute(s), 20 second(s)

Last reset at 594450 usecs after Sun May  2 15:57:19 2021
  Reason: Reset Requested by CLI command reload
  System version: 9.3(3)
  Service: 

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):
	
PAR-N9K-1(config)# 

Lot of errors in logging :

PAR-N9K-1(config)# sh logging last 10
2021 May  6 17:05:16 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:17 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a06:e881:2606::/48, flags:0x0, intf:0x10041c, Error: Hw Trie full(201)
2021 May  6 17:05:18 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:19 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:19 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:20 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:21 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:21 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2607:6100:100b::/48, flags:0x0, intf:0x10041c, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:22 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c0:193::/48, flags:0x0, intf:0x100040, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 May  6 17:05:22 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:53c0:2000::/36, flags:0x0, intf:0x100040, Error: Hw Trie full(201)
PAR-N9K-1(config)# 

Anyone have a magic command to fix all errors ?

Thanks !

Axel 

[Harold Ritter]

Hi Axel,

"show resource" is for route memory utilization, not for TCAM utilization.

"show hardware internal forwarding table utilization" is for TCAM utilization and it looks like you have issues with the TCAM utilization.

Can you provide the output from the following command, to see what profile you are using:

sh runn | incl "system routing"

Ideally, you should be using the "system routing template-internet-peering".

Regards,

[axelhauguel]

Hi

Thanks for your reply. I already use system routing template : 

PAR-N9K-1# sh runn | incl "system routing"
system routing template-internet-peering
PAR-N9K-1# 

[Harold Ritter]

Hi Axel,

Two things to consider then.

1. Reduce the number of prefixes that you receive from your Service Provider. Are you receiving the full Internet routing table from them. If so, you might want to consider only receiving partial routes.

2. Replace the Nexus by something else. The Nexus9k was not originally designed as an Internet edge device and therefore comes with its limitations.

Regards,

[axelhauguel]

Hi

Thanks for your reply. I'have around 880 000 routes.

Can you know what is the error "Hw trie full" ? Also strange, because route memory (u4mem) is not full

Axel

[Harold Ritter]

Hi Axel,

HW trie is related to TCAM. As I mentioned before, u4mem is routing memory (urib). The issue you are having is with ufib.

Regards,

[axelhauguel]

Hello Harold,

Thanks against for your reply.

Small last question. For example, i'have a error for a prefix but If I check route, I see the path. Normal ? Not normal ?

PAR-N9K-1# sh logging last 10
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 62.24.100.0/23, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 62.24.100.0/24, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.166.0.0/15, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.166.0.0/16, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 197.156.137.0/24, flags:0x0, intf:0x100138, Error:
 Hw Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 102.135.169.0/24, flags:0x0, intf:0x100138, Error:
 Hw Trie full(201)
2021 May  7 11:02:53 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 154.123.0.0/16, flags:0x0, intf:0x100138, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:448c::/32, flags:0x0, intf:0x10041c, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:4ea8::/32, flags:0x0, intf:0x10041c, Error: H
w Trie full(201)
2021 May  7 11:02:54 PAR-N9K-1 %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 45.148.160.0/24, flags:0x0, intf:0x1000d1, Error: 
FIB TCAM FULL For IP Routes(235)
PAR-N9K-1# show bgp ipv4 unicast 154.123.0.0
BGP routing table information for VRF default, address family IPv4 Unicast
BGP routing table entry for 154.123.0.0/18, version 5085384
Paths: (1 available, best #1)
Flags: (0x08001a) (high32 00000000) on xmit-list, is in urib, is best urib route, is in HW

  Advertised path-id 1
  Path type: external, path is valid, received and used, is best path, no labeled nexthop, in rib
  AS-Path: 49434 174 3257 8966 12455 , path sourced external to AS
    100.64.70.5 (metric 0) from 100.64.70.5 (193.178.0.1)
      Origin incomplete, MED not set, localpref 100, weight 0
      Community: 174:21100 174:22008 49434:101 49434:201 49434:301 49434:1000 49434:1001 

  Path-id 1 advertised to peers:
    185.171.202.183    185.171.202.204    185.171.202.246

PAR-N9K-1# 

[Harold Ritter]

Hi Axel,

It is normal to see the path in the BGP table. You will also see the route in the routing table (show ip route <prefix/prefixlen>). You will not see the route in the TCAM (show forwarding ipv4 route <prefix/prefixlen>).

Regards,

[axelhauguel]

Hi

Okay, thanks ! Small last question

There is an impact in production to not see route in TCAM ? 

What impact if route is not present in TCAM ?

Thanks

[Harold Ritter]

The Nexus 9k does not do software forwarding for transit traffic (page 121 of the below presentation). So if you don't have a summary route installed in the TCAM (including a default route), traffic to the specific destination for which the error was seen in the log will be dropped.

For more information about the Nexus9k architecture, you can refer to the following document:

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKDCT-3101.pdf

Regards,

[Giuseppe Larosa]

Hello Harold,

it is interesting that the Nexus will drop packets for destinations not installed in the TCAM instead of using software based forwarding.

In the past for older platforms like Cat 6500 or Cisco 7600 (Sup720 based )  it was enough to change a linecard and to plug in the new linecard with the wrong daughter card DFC (non XL version ) and the whole system could experience a sudden increase in CPU usage of main CPU caused by process switching of traffic for destinations not stored on that linecard CEF table.

Best Regards

Giuseppe

[Harold Ritter]

Hi Giuseppe,

Long time no speak. Hope you are doing well.

Software forwarding was indeed supported as a fallback on those platforms, but created a lot of issues, as RP was able to support limited throughput and ended up causing high cpu in most cases. That was probably taken in consideration when designing the Nexus 9k family.

Regards,