Solved: Re: N9K crashing every months

axelhauguel · ‎08-03-2021

Hello,

I'have a nexus Nexus9000 C93180YC-FX. I using it to do BGP and some small routing.

It crashing every months, with error :

edge01.par9k# show ver
Cisco Nexus Operating System (NX-OS) Software
Software
  BIOS: version 05.39
 NXOS: version 9.3(3)
  BIOS compile time:  08/30/2019
  NXOS image file is: bootflash:///nxos.9.3.3.bin
  NXOS compile time:  12/22/2019 2:00:00 [12/22/2019 16:00:37]


Hardware
  cisco Nexus9000 C93180YC-FX Chassis 
  Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 65808192 kB of memory.
  Processor Board ID FDO22300362

  Device name: edge01.par9k
  bootflash:  115805356 kB
Kernel uptime is 0 day(s), 3 hour(s), 31 minute(s), 43 second(s)

Last reset at 123969 usecs after Tue Aug  3 06:21:02 2021
  Reason: Reset Requested due to Fatal Module Error
  System version: 9.3(3)
  Service: System manager

plugin
  Core Plugin, Ethernet Plugin

Active Package(s):
        
edge01.par9k#

edge01.par9k#  show logging nvram 
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.64.0/20, flags:0x0, intf:0x10043d, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:68c::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a04:5b05::/32, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:da00:404::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:da00:400::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c1:f17::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:7240::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:7240:2008::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400c:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400c::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4001::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2401:b180:2800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:6000::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4004:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400a::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2401:b180:2000::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:6800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4001:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4000:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400b::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4004::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400a:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400b:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0b:b86:f::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:24c::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:224::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:228::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:240::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:2c0::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:8400::/38, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:b8c0::/32, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:93c0:100::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:c000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:4000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc::/33, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:8000::/33, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:8000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c0:dee::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:dc40::/32, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2602:fd20::/36, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a05:4140:103::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a05:dfc7:44::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:bc00::/30, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0b:e640:1::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:bc04::/30, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2620:0:2f0::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a02:e981:10::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:2280:147::/48, flags:0x0, intf:0x100060, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b103:b19::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b103:b18::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:6f40::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:654::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:4e4::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:718::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:7c8::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2406:cf00::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:5ec0:1000::/40, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:5ec0:1100::/40, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c1:a90::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b107:b85::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:db40::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:780::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a06:cdc0:2007::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2001:1398:16::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:18:06 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:2280:147::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.44.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.10.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.8.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.42.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 194.98.240.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 195.189.220.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.80.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.46.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.12.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.112.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.32.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.0.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.64.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.96.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 193.32.101.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.54.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.16.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 212.155.238.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.20.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201)
2021 Aug  3 06:18:14 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0c:7a80::/29, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:18:14 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0c:7a80::/29, flags:0x0, intf:0x100060, Error: FIB TCAM FULL For IPV6 Routes(237)
2021 Aug  3 06:18:22 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:371c:8000::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201)
2021 Aug  3 06:19:00 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "tahusd" (PID 7452) hasn't caught signal 6 (core will be saved).
2021 Aug  3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "tahusd" in vdc 1 has had a hap failure
2021 Aug  3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 23536 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xF
FFFFFFF)  . 
2021 Aug  3 06:21:01 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 22180 with message filename = 0x102_tahusd_log.7452.tar.gz . 
2021 Aug  3 06:21:02 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: copy_cores_to_logflash: PID 22180 with message filename = 1624182126_0x102_tahusd_log.7332.tar.gz . 
2021 Aug  3 06:23:20 edge01 %$ VDC-1 %$ netstack: Registration with cli server complete
2021 Aug  3 06:23:39 edge01 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp
2021 Aug  3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2021 Aug  3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
edge01.par9k#

I have tried quite a few things and I can never find where it came from. this bothers me a lot because it is unusable for production. Can someone help me ? it seems related to the TAHUSD service, maybe by disabling it we could work around the problem?

I'have checked bug page but.. always, it's not for my NX-OS version :

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvg24055

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvp34985

I have the impression that it is related to an SFP + which is not of the good brand .. for information.

Thanks

marce1000 · ‎08-03-2021

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv84849

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

View solution in original post

Leo Laohoo · ‎08-03-2021

Could be CSCvx27498.

Raise a TAC Case.

axelhauguel · ‎08-03-2021

Hello,

Not sure that that. Because error of routes, i'have this every time and not only in the crash.

2021 Aug  3 06:19:00 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "tahusd" (PID 7452) hasn't caught signal 6 (core will be saved).
2021 Aug  3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "tahusd" in vdc 1 has had a hap failure
2021 Aug  3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 23536 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xF
FFFFFFF)  . 
2021 Aug  3 06:21:01 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 22180 with message filename = 0x102_tahusd_log.7452.tar.gz . 
2021 Aug  3 06:21:02 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: copy_cores_to_logflash: PID 22180 with message filename = 1624182126_0x102_tahusd_log.7332.tar.gz . 
2021 Aug  3 06:23:20 edge01 %$ VDC-1 %$ netstack: Registration with cli server complete
2021 Aug  3 06:23:39 edge01 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp
2021 Aug  3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary
2021 Aug  3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary

This is linked to crash

Leo Laohoo · ‎08-03-2021

@axelhauguel wrote:

I'have checked bug page but.. always, it's not for my NX-OS version :

The two bug IDs you've identified look promising. I still recommending engaging TAC.

NOTE: Cisco rarely updates Bug ID information.

marce1000 · ‎08-03-2021

- Could be related to a resource problem , you may want to review this thread too :

https://community.cisco.com/t5/routing/error-hw-trie-full-201/m-p/4396229

And : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn57068

Use advisory software release, check if that can help.

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

MHM Cisco World · ‎08-03-2021

N9K as I get from discussion here is full memory, please not that the BGP cause this.
if your N9K is work as transit AS then all router form one ISP will forward to other ISP make the N9K high utilize even if the N9K do use this routes.
so can you config that this N9K is connect to two BGP peer ? and you don't config any inbound outbound policy?

axelhauguel · ‎08-03-2021

Hello,

As discussed in another topic, I always use "soft-reconfiguration inbound". Delete this function can fix ?

I monitor RAM of my router and seem not full, and stable.

edge01.par9k(config-router)# show resource 

     Resource                   Min       Max       Used      Unused    Avail    
     --------                   ---       ---       ----      ------    -----    
     vlan                       16        4094      24        0         4070     
     vrf                        2         4096      2         0         4094     
     port-channel               0         511       1         0         510      
     u4route-mem                768       768       127       641       641      
     u6route-mem                512       512       34        478       478      
     m4route-mem                58        58        1         57        57       
     m6route-mem                8         8         1         7         7        

edge01.par9k(config-router)# show system re
redundancy     reset-reason   resources      
edge01.par9k(config-router)# show system res
reset-reason   resources      
edge01.par9k(config-router)# show system resources 
Load average:   1 minute: 0.50   5 minutes: 0.44   15 minutes: 0.49
Processes   :   714 total, 2 running
CPU states  :   2.01% user,   1.67% kernel,   96.30% idle
        CPU0 states  :   3.03% user,   3.03% kernel,   93.93% idle
        CPU1 states  :   4.95% user,   4.95% kernel,   90.09% idle
        CPU2 states  :   4.08% user,   2.04% kernel,   93.87% idle
        CPU3 states  :   4.08% user,   1.02% kernel,   94.89% idle
        CPU4 states  :   3.03% user,   4.04% kernel,   92.92% idle
        CPU5 states  :   2.00% user,   2.00% kernel,   96.00% idle
        CPU6 states  :   3.00% user,   2.00% kernel,   95.00% idle
        CPU7 states  :   1.02% user,   1.02% kernel,   97.95% idle
        CPU8 states  :   0.00% user,   0.00% kernel,   100.00% idle
        CPU9 states  :   0.00% user,   0.00% kernel,   100.00% idle
        CPU10 states  :   0.00% user,   0.00% kernel,   100.00% idle
        CPU11 states  :   1.00% user,   0.00% kernel,   99.00% idle
Memory usage:   65808192K total,   9179364K used,   56628828K free
Current memory status: OK
edge01.par9k(config-router)#

To open TAC in cisco, is required to have a cisco contract, no ?

I'have get the NX-OS update from another people, I can update the router without forget licence ?

Thanks

Axel

MHM Cisco World · ‎08-03-2021

this monitor after crush ?

check it after two three days see the unused how much.

127 route used are this route install in RIB?

axelhauguel · ‎08-03-2021

Hi MHM,

Ok. What is 127 route ? It's 127MB route no ?

edge01.par9k# show hardware internal forwarding table utilization

slot  1
=======


IPv4/IPv6  hosts and routes summary on module : 1 
--------------------------------------------------

Configured System Routing Mode: Internet Peering

Dynamic V6 Trie : True 
--------------------------------------------------
Max IPv4 Trie route entries: 1000448 
Max IPv6 Trie route entries: 500224 
Max TCAM table entries : 16384 
Max V4 Ucast DA TCAM table entries : 10240 
Max V6 Ucast DA TCAM table entries : 2048 
Max native host route entries (shared v4/v6) : 32768 
Max ARP entries: 32768 
Max ND entries (Entries might overflow into tcam as Host-As-Route): 16384 
Total number of IPv4 host trie routes used : 0 
Total number of IPv4 host tcam routes used : 30 
Total number of IPv4 LPM trie routes used : 686607 
Total number of IPv4 LPM tcam routes used : 4413 
Total number of IPv6 host trie routes used : 0 
Total number of IPv6 host tcam routes used: 0 
Total number of IPv6 LPM trie routes used : 123635 
Total number of IPv6 LPM tcam routes used : 1270 

Total number of IPv4 host native routes used in native tiles : 691 
Total number of IPv6 host native routes used in native tiles : 353 
Total number of IPv6 ND/local routes used in native tiles : 353 
Total number of IPv6 host /128 learnt routes used in native tiles : 0 
IPv4 Host-as-Route count : 2 
IPv6 Host-as-Route count : 0 
Nexthop count : 1165 

Percentage utilization of IPv4 native host routes : 2.10   
Percentage utilization of IPv6 native host routes : 2.15   
Percentage utilization of IPv6 ND/local routes : 2.15   
Percentage utilization of IPv6 host /128 learnt routes : 0.00   
Percentage utilization of IPv4 trie routes : 68.62  
Percentage utilization of IPv6 trie routes : 24.71  
Percentage utilization of IPv4 TCAM routes : 43.40  
Percentage utilization of IPv6 TCAM routes : 62.01  
Percentage utilization of nexthop entries : 3.55   


IPv4/Ipv6 Mcast host entry summary 
----------------------------------

Max Mcast Route Entries Limit = 0 
Max Mcast V4 SA TCAM entries (S/m) = 0 
Max Overflow Mcast v4 SA TCAM entries = 0
Max Overflow Mcast V4 DA TCAM entries = 0 
Total number of IPv4 Multicast SA LPM routes used = 0 
Percentage utilization of IPv4 Multicast SA LPM routes = 0.00   
Used Mcast Entries = 0 
Used MCIDX Count = 4 
Used *,G Entries in HRT = 0 
Used *,G Entries in LPM = 0 
Used (S,G) Entries = 0 
Used Mcast S/32 Entries in HRT = 0 
Max Reserved Mcast S/32 and G/32 in LPM = 0
Used Mcast S/32 Entries in SA LPM = 0 
Percentage utilization of S/32 in SA LPM = 0.00   
Used Mcast G/32 Entries in DA LPM = 0 
Percentage utilization of G/32 in DA LPM = 0.00   

Max IPv6 Mcast Route Entries Limit = 0 
Used IPv6 Mcast Entries = 0 
Used IPv6 *,G Entries in HRT = 0 
Used IPv6 *,G Entries in LPM = 0 
Used IPv6 (S,G) Entries = 0 
Used IPv6 Mcast S/128 Entries in HRT = 0 


Mcast Mac entry summary 
----------------------------------

Max Mcast Mac Route Entries Limit = 0 
Used Mcast Mac Entries = 0 


-------------------MPLS Hardware Resources------------------

Max Label Entries ( without ECMP ) = 1000 
Max Label Entries ( with ECMP ) = 500 
Used Label Entries = 0 
No. of MPLS VPN Labels Used = 0 


-------------------ECMP GROUP/TILE INFO-------------------

ECMP group supported : Yes
Max ECMP groups : 1022
ECMP groups used : 2
Max FC ECMP groups : 1024
FC ECMP groups used : 0
Num of ECMP group member tiles : 2
Num of ECMP group tiles : 1

axelhauguel · ‎08-03-2021

Also found that :

edge01.par9k(config-router-neighbor)# show logging onboard internal platform
----------------------------
    Module:  1
----------------------------


Tue Aug  3 06:21:02 2021 (123848 us) 
Critical failure due to Service on linecard had a hap-reset in device 134
 (device error code: 0x0, extra info: tahusd hap reset)

marce1000 · ‎08-03-2021

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv84849

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

axelhauguel · ‎08-03-2021

Very interesting.

System version notified on the TAC is : System version: 7.0(3)I7(8)

But I see, the bug is fixed in 9.3(5.80) / 9.3(6)

I'am in 9.3(3). But i'have not contract with cisco. Is cisco compliant to ask user if they have NX-OS version for me ?

Thanks for advance,

marce1000 · ‎08-03-2021

- Normally you need a service contract for downloading software for particular models that you own, you may rase a TAC call and mention the bug report , as your business may be considered important they may offer that version to you themselves.

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

Leo Laohoo · ‎08-03-2021

@axelhauguel wrote:

I'am in 9.3(3). But i'have not contract with cisco. Is cisco compliant to ask user if they have NX-OS version for me ?

There is a dodgy way of getting the firmware files.

First, read Cisco NX-OS Software Unauthenticated Arbitrary File Actions Vulnerability security bulletin.
Next, scroll down to the Customers Without Service Contracts section.
Read and understand the first two paragraphs very carefully.

NOTE:

When contacting TAC, make sure to be ready to provide the exact firmware you are looking for by specifying the filename and link to where the file(s) are located.

IMPORTANT:

Do not waste any time -- Someone has been removing the Customers Without Service Contracts from a lot of the Security Bulletins.

axelhauguel · ‎08-04-2021

Hello,

Thanks. Cisco have send me the firmware with fix.

Note for me, today :

Dynamic V6 Trie : True 
--------------------------------------------------
Max IPv4 Trie route entries: 1000448 
Max IPv6 Trie route entries: 500224 
Max TCAM table entries : 16384 
Max V4 Ucast DA TCAM table entries : 10240 
Max V6 Ucast DA TCAM table entries : 2048 
Max native host route entries (shared v4/v6) : 32768 
Max ARP entries: 32768 
Max ND entries (Entries might overflow into tcam as Host-As-Route): 16384 
Total number of IPv4 host trie routes used : 0 
Total number of IPv4 host tcam routes used : 32 
Total number of IPv4 LPM trie routes used : 686706 
Total number of IPv4 LPM tcam routes used : 6758 
Total number of IPv6 host trie routes used : 0 
Total number of IPv6 host tcam routes used: 0 
Total number of IPv6 LPM trie routes used : 122579 
Total number of IPv6 LPM tcam routes used : 1939 

Total number of IPv4 host native routes used in native tiles : 714 
Total number of IPv6 host native routes used in native tiles : 366 
Total number of IPv6 ND/local routes used in native tiles : 366 
Total number of IPv6 host /128 learnt routes used in native tiles : 0 
IPv4 Host-as-Route count : 2 
IPv6 Host-as-Route count : 0 
Nexthop count : 1178 

Percentage utilization of IPv4 native host routes : 2.17   
Percentage utilization of IPv6 native host routes : 2.23   
Percentage utilization of IPv6 ND/local routes : 2.23   
Percentage utilization of IPv6 host /128 learnt routes : 0.00   
Percentage utilization of IPv4 trie routes : 68.63  
Percentage utilization of IPv6 trie routes : 24.50  
Percentage utilization of IPv4 TCAM routes : 66.32  
Percentage utilization of IPv6 TCAM routes : 94.67  
Percentage utilization of nexthop entries : 3.59