08-03-2021 01:00 AM
Hello,
I'have a nexus Nexus9000 C93180YC-FX. I using it to do BGP and some small routing.
It crashing every months, with error :
edge01.par9k# show ver Cisco Nexus Operating System (NX-OS) Software Software BIOS: version 05.39 NXOS: version 9.3(3) BIOS compile time: 08/30/2019 NXOS image file is: bootflash:///nxos.9.3.3.bin NXOS compile time: 12/22/2019 2:00:00 [12/22/2019 16:00:37] Hardware cisco Nexus9000 C93180YC-FX Chassis Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 65808192 kB of memory. Processor Board ID FDO22300362 Device name: edge01.par9k bootflash: 115805356 kB Kernel uptime is 0 day(s), 3 hour(s), 31 minute(s), 43 second(s) Last reset at 123969 usecs after Tue Aug 3 06:21:02 2021 Reason: Reset Requested due to Fatal Module Error System version: 9.3(3) Service: System manager plugin Core Plugin, Ethernet Plugin Active Package(s): edge01.par9k#
edge01.par9k# show logging nvram 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.64.0/20, flags:0x0, intf:0x10043d, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:68c::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a04:5b05::/32, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:da00:404::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:da00:400::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c1:f17::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:7240::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:7240:2008::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400c:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400c::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4001::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2401:b180:2800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:6000::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4004:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400a::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2401:b180:2000::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:b200:6800::/37, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4001:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4000:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400b::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:4004::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400a:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2408:400b:8000::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0b:b86:f::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:24c::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:224::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:228::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:240::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:2c0::/46, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:7fc0:8400::/38, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:b8c0::/32, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:93c0:100::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:c000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:4000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc::/33, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:8000::/33, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:32cc:8000::/34, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c0:dee::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2400:dc40::/32, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2602:fd20::/36, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a05:4140:103::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a05:dfc7:44::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:bc00::/30, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0b:e640:1::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:bc04::/30, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2620:0:2f0::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a02:e981:10::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:2280:147::/48, flags:0x0, intf:0x100060, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b103:b19::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b103:b18::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:6f40::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:654::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:4e4::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:718::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:7c8::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2406:cf00::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:5ec0:1000::/40, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:5ec0:1100::/40, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c1:a90::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:b107:b85::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2402:db40::/33, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0e:97c5:780::/46, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a06:cdc0:2007::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:17:56 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2001:1398:16::/48, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:18:06 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2404:2280:147::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.44.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.10.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.8.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.42.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 194.98.240.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 195.189.220.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.80.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.46.0/23, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.12.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.112.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.32.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.0.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.64.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.96.0/20, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 193.32.101.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.54.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.16.0/21, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 212.155.238.0/24, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:10 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 171.18.20.0/22, flags:0x0, intf:0x100083, Error: Hw Trie full(201) 2021 Aug 3 06:18:14 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0c:7a80::/29, flags:0x0, intf:0x100036, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:18:14 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2a0c:7a80::/29, flags:0x0, intf:0x100060, Error: FIB TCAM FULL For IPV6 Routes(237) 2021 Aug 3 06:18:22 edge01.par9k %$ VDC-1 %$ %IPFIB-SLOT1-2-UFIB_ROUTE_CREATE: Unicast route create failed for INS unit 0, VRF: 1, 2804:371c:8000::/48, flags:0x0, intf:0x100036, Error: Hw Trie full(201) 2021 Aug 3 06:19:00 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "tahusd" (PID 7452) hasn't caught signal 6 (core will be saved). 2021 Aug 3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "tahusd" in vdc 1 has had a hap failure 2021 Aug 3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 23536 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xF FFFFFFF) . 2021 Aug 3 06:21:01 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 22180 with message filename = 0x102_tahusd_log.7452.tar.gz . 2021 Aug 3 06:21:02 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: copy_cores_to_logflash: PID 22180 with message filename = 1624182126_0x102_tahusd_log.7332.tar.gz . 2021 Aug 3 06:23:20 edge01 %$ VDC-1 %$ netstack: Registration with cli server complete 2021 Aug 3 06:23:39 edge01 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp 2021 Aug 3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary 2021 Aug 3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary edge01.par9k#
I have tried quite a few things and I can never find where it came from. this bothers me a lot because it is unusable for production. Can someone help me ? it seems related to the TAHUSD service, maybe by disabling it we could work around the problem?
I'have checked bug page but.. always, it's not for my NX-OS version :
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvg24055
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvp34985
I have the impression that it is related to an SFP + which is not of the good brand .. for information.
Thanks
Solved! Go to Solution.
08-03-2021 09:03 AM
- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv84849
M.
08-03-2021 01:25 AM
Could be CSCvx27498.
Raise a TAC Case.
08-03-2021 01:56 AM
Hello,
Not sure that that. Because error of routes, i'have this every time and not only in the crash.
2021 Aug 3 06:19:00 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "tahusd" (PID 7452) hasn't caught signal 6 (core will be saved). 2021 Aug 3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-HAP_FAILURE_SUP_RESET: Service "tahusd" in vdc 1 has had a hap failure 2021 Aug 3 06:19:03 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: fsm_action_become_offline: PID 23536 with message Could not turn off console logging on vdc 1 error: mts req-response with syslogd in vdc 1 failed (0xF FFFFFFF) . 2021 Aug 3 06:21:01 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 22180 with message filename = 0x102_tahusd_log.7452.tar.gz . 2021 Aug 3 06:21:02 edge01.par9k %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: copy_cores_to_logflash: PID 22180 with message filename = 1624182126_0x102_tahusd_log.7332.tar.gz . 2021 Aug 3 06:23:20 edge01 %$ VDC-1 %$ netstack: Registration with cli server complete 2021 Aug 3 06:23:39 edge01 %$ VDC-1 %$ %USER-0-SYSTEM_MSG: end of default policer - copp 2021 Aug 3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: IOFPGA booted from Primary 2021 Aug 3 06:23:41 edge01 %$ VDC-1 %$ %CARDCLIENT-2-FPGA_BOOT_PRIMARY: MIFPGA booted from Primary
This is linked to crash
08-03-2021 02:05 AM
@axelhauguel wrote:
I'have checked bug page but.. always, it's not for my NX-OS version :
The two bug IDs you've identified look promising. I still recommending engaging TAC.
NOTE: Cisco rarely updates Bug ID information.
08-03-2021 02:59 AM
- Could be related to a resource problem , you may want to review this thread too :
https://community.cisco.com/t5/routing/error-hw-trie-full-201/m-p/4396229
And : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvn57068
Use advisory software release, check if that can help.
M.
08-03-2021 04:39 AM
N9K as I get from discussion here is full memory, please not that the BGP cause this.
if your N9K is work as transit AS then all router form one ISP will forward to other ISP make the N9K high utilize even if the N9K do use this routes.
so can you config that this N9K is connect to two BGP peer ? and you don't config any inbound outbound policy?
08-03-2021 05:06 AM
Hello,
As discussed in another topic, I always use "soft-reconfiguration inbound". Delete this function can fix ?
I monitor RAM of my router and seem not full, and stable.
edge01.par9k(config-router)# show resource Resource Min Max Used Unused Avail -------- --- --- ---- ------ ----- vlan 16 4094 24 0 4070 vrf 2 4096 2 0 4094 port-channel 0 511 1 0 510 u4route-mem 768 768 127 641 641 u6route-mem 512 512 34 478 478 m4route-mem 58 58 1 57 57 m6route-mem 8 8 1 7 7 edge01.par9k(config-router)# show system re redundancy reset-reason resources edge01.par9k(config-router)# show system res reset-reason resources edge01.par9k(config-router)# show system resources Load average: 1 minute: 0.50 5 minutes: 0.44 15 minutes: 0.49 Processes : 714 total, 2 running CPU states : 2.01% user, 1.67% kernel, 96.30% idle CPU0 states : 3.03% user, 3.03% kernel, 93.93% idle CPU1 states : 4.95% user, 4.95% kernel, 90.09% idle CPU2 states : 4.08% user, 2.04% kernel, 93.87% idle CPU3 states : 4.08% user, 1.02% kernel, 94.89% idle CPU4 states : 3.03% user, 4.04% kernel, 92.92% idle CPU5 states : 2.00% user, 2.00% kernel, 96.00% idle CPU6 states : 3.00% user, 2.00% kernel, 95.00% idle CPU7 states : 1.02% user, 1.02% kernel, 97.95% idle CPU8 states : 0.00% user, 0.00% kernel, 100.00% idle CPU9 states : 0.00% user, 0.00% kernel, 100.00% idle CPU10 states : 0.00% user, 0.00% kernel, 100.00% idle CPU11 states : 1.00% user, 0.00% kernel, 99.00% idle Memory usage: 65808192K total, 9179364K used, 56628828K free Current memory status: OK edge01.par9k(config-router)#
To open TAC in cisco, is required to have a cisco contract, no ?
I'have get the NX-OS update from another people, I can update the router without forget licence ?
Thanks
Axel
08-03-2021 05:30 AM - edited 08-03-2021 05:38 AM
this monitor after crush ?
check it after two three days see the unused how much.
127 route used are this route install in RIB?
08-03-2021 08:29 AM - edited 08-03-2021 08:36 AM
Hi MHM,
Ok. What is 127 route ? It's 127MB route no ?
edge01.par9k# show hardware internal forwarding table utilization
slot 1
=======
IPv4/IPv6 hosts and routes summary on module : 1
--------------------------------------------------
Configured System Routing Mode: Internet Peering
Dynamic V6 Trie : True
--------------------------------------------------
Max IPv4 Trie route entries: 1000448
Max IPv6 Trie route entries: 500224
Max TCAM table entries : 16384
Max V4 Ucast DA TCAM table entries : 10240
Max V6 Ucast DA TCAM table entries : 2048
Max native host route entries (shared v4/v6) : 32768
Max ARP entries: 32768
Max ND entries (Entries might overflow into tcam as Host-As-Route): 16384
Total number of IPv4 host trie routes used : 0
Total number of IPv4 host tcam routes used : 30
Total number of IPv4 LPM trie routes used : 686607
Total number of IPv4 LPM tcam routes used : 4413
Total number of IPv6 host trie routes used : 0
Total number of IPv6 host tcam routes used: 0
Total number of IPv6 LPM trie routes used : 123635
Total number of IPv6 LPM tcam routes used : 1270
Total number of IPv4 host native routes used in native tiles : 691
Total number of IPv6 host native routes used in native tiles : 353
Total number of IPv6 ND/local routes used in native tiles : 353
Total number of IPv6 host /128 learnt routes used in native tiles : 0
IPv4 Host-as-Route count : 2
IPv6 Host-as-Route count : 0
Nexthop count : 1165
Percentage utilization of IPv4 native host routes : 2.10
Percentage utilization of IPv6 native host routes : 2.15
Percentage utilization of IPv6 ND/local routes : 2.15
Percentage utilization of IPv6 host /128 learnt routes : 0.00
Percentage utilization of IPv4 trie routes : 68.62
Percentage utilization of IPv6 trie routes : 24.71
Percentage utilization of IPv4 TCAM routes : 43.40
Percentage utilization of IPv6 TCAM routes : 62.01
Percentage utilization of nexthop entries : 3.55
IPv4/Ipv6 Mcast host entry summary
----------------------------------
Max Mcast Route Entries Limit = 0
Max Mcast V4 SA TCAM entries (S/m) = 0
Max Overflow Mcast v4 SA TCAM entries = 0
Max Overflow Mcast V4 DA TCAM entries = 0
Total number of IPv4 Multicast SA LPM routes used = 0
Percentage utilization of IPv4 Multicast SA LPM routes = 0.00
Used Mcast Entries = 0
Used MCIDX Count = 4
Used *,G Entries in HRT = 0
Used *,G Entries in LPM = 0
Used (S,G) Entries = 0
Used Mcast S/32 Entries in HRT = 0
Max Reserved Mcast S/32 and G/32 in LPM = 0
Used Mcast S/32 Entries in SA LPM = 0
Percentage utilization of S/32 in SA LPM = 0.00
Used Mcast G/32 Entries in DA LPM = 0
Percentage utilization of G/32 in DA LPM = 0.00
Max IPv6 Mcast Route Entries Limit = 0
Used IPv6 Mcast Entries = 0
Used IPv6 *,G Entries in HRT = 0
Used IPv6 *,G Entries in LPM = 0
Used IPv6 (S,G) Entries = 0
Used IPv6 Mcast S/128 Entries in HRT = 0
Mcast Mac entry summary
----------------------------------
Max Mcast Mac Route Entries Limit = 0
Used Mcast Mac Entries = 0
-------------------MPLS Hardware Resources------------------
Max Label Entries ( without ECMP ) = 1000
Max Label Entries ( with ECMP ) = 500
Used Label Entries = 0
No. of MPLS VPN Labels Used = 0
-------------------ECMP GROUP/TILE INFO-------------------
ECMP group supported : Yes
Max ECMP groups : 1022
ECMP groups used : 2
Max FC ECMP groups : 1024
FC ECMP groups used : 0
Num of ECMP group member tiles : 2
Num of ECMP group tiles : 1
08-03-2021 08:46 AM
Also found that :
edge01.par9k(config-router-neighbor)# show logging onboard internal platform ---------------------------- Module: 1 ---------------------------- Tue Aug 3 06:21:02 2021 (123848 us) Critical failure due to Service on linecard had a hap-reset in device 134 (device error code: 0x0, extra info: tahusd hap reset)
08-03-2021 09:03 AM
- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvv84849
M.
08-03-2021 09:08 AM
Very interesting.
System version notified on the TAC is : System version: 7.0(3)I7(8)
But I see, the bug is fixed in 9.3(5.80) / 9.3(6)
I'am in 9.3(3). But i'have not contract with cisco. Is cisco compliant to ask user if they have NX-OS version for me ?
Thanks for advance,
08-03-2021 09:20 AM
- Normally you need a service contract for downloading software for particular models that you own, you may rase a TAC call and mention the bug report , as your business may be considered important they may offer that version to you themselves.
M.
08-03-2021 03:58 PM - edited 08-03-2021 04:00 PM
@axelhauguel wrote:
I'am in 9.3(3). But i'have not contract with cisco. Is cisco compliant to ask user if they have NX-OS version for me ?
There is a dodgy way of getting the firmware files.
NOTE:
When contacting TAC, make sure to be ready to provide the exact firmware you are looking for by specifying the filename and link to where the file(s) are located.
IMPORTANT:
Do not waste any time -- Someone has been removing the Customers Without Service Contracts from a lot of the Security Bulletins.
08-04-2021 08:44 AM
Hello,
Thanks. Cisco have send me the firmware with fix.
Note for me, today :
Dynamic V6 Trie : True -------------------------------------------------- Max IPv4 Trie route entries: 1000448 Max IPv6 Trie route entries: 500224 Max TCAM table entries : 16384 Max V4 Ucast DA TCAM table entries : 10240 Max V6 Ucast DA TCAM table entries : 2048 Max native host route entries (shared v4/v6) : 32768 Max ARP entries: 32768 Max ND entries (Entries might overflow into tcam as Host-As-Route): 16384 Total number of IPv4 host trie routes used : 0 Total number of IPv4 host tcam routes used : 32 Total number of IPv4 LPM trie routes used : 686706 Total number of IPv4 LPM tcam routes used : 6758 Total number of IPv6 host trie routes used : 0 Total number of IPv6 host tcam routes used: 0 Total number of IPv6 LPM trie routes used : 122579 Total number of IPv6 LPM tcam routes used : 1939 Total number of IPv4 host native routes used in native tiles : 714 Total number of IPv6 host native routes used in native tiles : 366 Total number of IPv6 ND/local routes used in native tiles : 366 Total number of IPv6 host /128 learnt routes used in native tiles : 0 IPv4 Host-as-Route count : 2 IPv6 Host-as-Route count : 0 Nexthop count : 1178 Percentage utilization of IPv4 native host routes : 2.17 Percentage utilization of IPv6 native host routes : 2.23 Percentage utilization of IPv6 ND/local routes : 2.23 Percentage utilization of IPv6 host /128 learnt routes : 0.00 Percentage utilization of IPv4 trie routes : 68.63 Percentage utilization of IPv6 trie routes : 24.50 Percentage utilization of IPv4 TCAM routes : 66.32 Percentage utilization of IPv6 TCAM routes : 94.67 Percentage utilization of nexthop entries : 3.59
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide