Cisco 3650 Crash

paul amaral · ‎12-11-2023

HI recently I had an issue where one of my 3650 switches in the stack crashed. It looks like it was related to spanning-tree process. Does anyone know if this is a known bug. I couldn't find anything related to it. I'm using IOS XE version, 16.9.5.

TIA, P

Exception to Fastpath Thread:
Frame pointer 0xFF8FAF1DA0, PC = 0xFFE19D55FC

-Traceback= 1#29f72ee29020089947733224e5b632bb   ld:FFF6B47000+38010 tdllib:FFE192B000+AA5FE tdllib:FFE192B000+AA8AC iosd_ngwc_unix:FFD4197000+49178 iosd_ngwc_unix:FFD4197000+3A848 iosd_ngwc_unix:FFD4197000+2EC14 prelib:FFF6AC2000+374C p
thread:FFD3A4A000+B240 c:FFD3F86000+11C0DC

IOS Thread backtrace:
UNIX-EXT-SIGNAL: User defined signal 2(17), Process = Spanning Tree
-Traceback= 1#29f72ee29020089947733224e5b632bb   pthread:FFD3A4A000+E730 iosd_ngwc_unix:FFD4197000+43028 :AAB5CB5000+1E31B34 :AAB5CB5000+78E0068 :AAB5CB5000+78DA09C :AAB5CB5000+78DA25C :AAB5CB5000+76A6FAC :AAB5CB5000+78E96F8

Auxiliary Thread backtrace:
-Traceback= 1#29f72ee29020089947733224e5b632bb   iosd_ngwc_unix:FFD4197000+47F2C prelib:FFF6AC2000+374C pthread:FFD3A4A000+B240 c:FFD3F86000+11C0DC

$0 : 0x00000000, AT : 0xFFE1B34DD0, v0 : 0x00000000, v1 : 0xFF8FAF4C48
a0 : 0xAAD2707348, a1 : 0xAAD269A890, a2 : 0x00000020, a3 : 0x00000000
t0 : 0xFFC2749000, t1 : 0xFF8FAFB540, t2 : 0x00000200, t3 : 0xFFFFFF00
t4 : 0x00000000, t5 : 0x80080018, t6 : 0x81320FA0, t7 : 0x8000000109124000
s0 : 0x4047D6F8, s1 : 0xFFE1AD8550, s2 : 0x0000000C, s3 : 0xFFE1B2EFA8
s4 : 0xD00000037, s5 : 0xAAD2707348, s6 : 0xFFD4231B40, s7 : 0xEA0FFEB70
t8 : 0x00000000, t9 : 0xFFE19D5588, k0 : 0x00000000, k1 : 0x00000000
gp : 0xFFE1B34DD0, sp : 0xFF8FAF1DA0, s8 : 0x00000001, ra : 0xFFE19D58AC
EPC : 0xFFE19D55FC,  SREG : 0x00000000 Cause : 0xFFFFFFFF
Error EPC : 0x00000000, BADVADDR = 0x00000000


========= Start of Crashinfo Collection (10:40:33 est Mon Dec 11 2023) =========

For image:
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 30-Jan-20 18:17 by mcpre


Uptime = 4w3d



========= Exception Tracebacks =================================================

Exception to Fastpath Thread:
Frame pointer 0xFF8FAF1DA0, PC = 0xFFE19D55FC
IOS Thread backtrace:
UNIX-EXT-SIGNAL: User defined signal 2(17), Process = Spanning Tree
-Traceback= 1#29f72ee29020089947733224e5b632bb   pthread:FFD3A4A000+E730 iosd_ngwc_unix:FFD4197000+43028 :AAB5CB5000+1E31B34 :AAB5CB5000+78E0068 :AAB5CB5000+78DA09C :AAB5CB5000+78DA25C :AAB5CB5000+76A6FAC :AAB5CB5000+78E96F8
Fastpath Thread backtrace:
-Traceback= 1#29f72ee29020089947733224e5b632bb   ld:FFF6B47000+38010 tdllib:FFE192B000+AA5FE tdllib:FFE192B000+AA8AC iosd_ngwc_unix:FFD4197000+49178 iosd_ngwc_unix:FFD4197000+3A848 iosd_ngwc_unix:FFD4197000+2EC14 prelib:FFF6AC2000+374C p
thread:FFD3A4A000+B240 c:FFD3F86000+11C0DC
Auxiliary Thread backtrace:
-Traceback= 1#29f72ee29020089947733224e5b632bb   iosd_ngwc_unix:FFD4197000+47F2C prelib:FFF6AC2000+374C pthread:FFD3A4A000+B240 c:FFD3F86000+11C0DC




========= Event Log ============================================================


3426   1848295862: Dec 14 06:03:59.171 RF_PROG_STANDBY_HOT(105) SSM(85) op=0 rc=11
3427   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) Switch Backup Interface client(507) op=0 rc=11
3428   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) REP Protocol(212) op=0 rc=11
3429   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) DHCP Snooping(105) op=0 rc=11
3430   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) Call-Home RF(1510) op=0 rc=11
3431   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) TCP(1601) op=0 rc=11
3432   1848296862: Dec 14 06:03:59.172 RF_PROG_STANDBY_HOT(105) LDP HA(73) op=0 rc=11
3433   1848297862: Dec 14 06:03:59.173 RF_PROG_STANDBY_HOT(105) IP Tunnel RF(151) op=0 rc=11
3434   1848298862: Dec 14 06:03:59.173 RF_PROG_STANDBY_HOT(105) Config Verify RF client(94) op=0 rc=11
3435   1848298862: Dec 14 06:03:59.174 RF_PROG_STANDBY_HOT(105) Platform Inline Power(10005) op=0 rc=11
3436   1848298862: Dec 14 06:03:59.174 RF_PROG_STANDBY_HOT(105) Platform Manager RF Client IPC(10008) op=0 rc=11
3437   1848298862: Dec 14 06:03:59.174 RF_PROG_STANDBY_HOT(105) Inline Power rf client(505) op=0 rc=11
3438   1848299862: Dec 14 06:03:59.175 RF_PROG_STANDBY_HOT(105) EnergyWise rf client(516) op=0 rc=11
3439   1848299862: Dec 14 06:03:59.175 RF_PROG_STANDBY_HOT(105) Igmp Snooping(506) op=0 rc=11
3440   1848299862: Dec 14 06:03:59.175 RF_PROG_STANDBY_HOT(105) Port Security(508) op=0 rc=11
3441   1848299862: Dec 14 06:03:59.175 RF_PROG_STANDBY_HOT(105) SISF table(515) op=0 rc=11
3442   1848299862: Dec 14 06:03:59.175 RF_PROG_STANDBY_HOT(105) CRYPTO RSA(130) op=0 rc=11
3443   1848300862: Dec 14 06:03:59.176 RF_PROG_STANDBY_HOT(105) PKI RF Client(131) op=0 rc=11
3444   1848300862: Dec 14 06:03:59.176 RF_PROG_STANDBY_HOT(105) WSMAN(161) op=0 rc=11
3445   1848300862: Dec 14 06:03:59.176 RF_PROG_STANDBY_HOT(105) DHCPv6 Relay(148) op=0 rc=0
3446   1848301862: Dec 14 06:03:59.177 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Relay(148) op=8 rc=0
3447   1848302862: Dec 14 06:03:59.178 RF_EVENT_PEER_PROG_DONE(506) DHCPv6 Relay(148) op=105 rc=11
3448   1848303862: Dec 14 06:03:59.179 RF_PROG_STANDBY_HOT(105) DHCPv6 Server(149) op=0 rc=0
3449   1848303862: Dec 14 06:03:59.179 RF_EVENT_CLIENT_PROGRESSION(503) DHCPv6 Server(149) op=8 rc=0
3450   1848304862: Dec 14 06:03:59.180 RF_EVENT_PEER_PROG_DONE(506) DHCPv6 Server(149) op=105 rc=11
3451   1848304862: Dec 14 06:03:59.180 RF_PROG_STANDBY_HOT(105) ISSU Test Client(4005) op=0 rc=11
3452   1848304862: Dec 14 06:03:59.180 RF_PROG_STANDBY_HOT(105) Network RF 2 Client(93) op=0 rc=11
3453   1848304862: Dec 14 06:03:59.180 RF_PROG_STANDBY_HOT(105) LAN-Switch PAgP/LACP(510) op=0 rc=11
3454   1848304862: Dec 14 06:03:59.180 RF_PROG_STANDBY_HOT(105) DATA DESCRIPTOR RF CLIENT(141) op=0 rc=11
3455   1848305862: Dec 14 06:03:59.181 RF_PROG_STANDBY_HOT(105) CTS HA(1000) op=0 rc=11
3456   1848305862: Dec 14 06:03:59.181 RF_PROG_STANDBY_HOT(105) Keystore(1001) op=0 rc=11
3457   1848305862: Dec 14 06:03:59.181 RF_PROG_STANDBY_HOT(105) IOS Config SHELL(4022) op=0 rc=11
3458   1848305862: Dec 14 06:03:59.181 RF_PROG_STANDBY_HOT(105) IOS Config ARCHIVE(4020) op=0 rc=11
3459   1848305862: Dec 14 06:03:59.181 RF_PROG_STANDBY_HOT(105) IOS Config ROLLBACK(4021) op=0 rc=11
3460   1848306862: Dec 14 06:03:59.182 RF_PROG_STANDBY_HOT(105) ONEP(4060) op=0 rc=11
3461   1848306862: Dec 14 06:03:59.182 RF_PROG_STANDBY_HOT(105) PASSWD RF Client(4058) op=0 rc=11
3462   1848306862: Dec 14 06:03:59.182 RF_PROG_STANDBY_HOT(105) Smart_Agent_RF_Client(1376) op=0 rc=11
3463   1848306862: Dec 14 06:03:59.182 RF_PROG_STANDBY_HOT(105) License Core HA Client(20001) op=0 rc=11
3464   1848306862: Dec 14 06:03:59.182 RF_PROG_STANDBY_HOT(105) License Agent HA Client(20011) op=0 rc=11
3465   1848307862: Dec 14 06:03:59.183 RF_PROG_STANDBY_HOT(105) MLD Snooping(521) op=0 rc=11
3466   1848307862: Dec 14 06:03:59.183 RF_PROG_STANDBY_HOT(105) Online Diag HA(4056) op=0 rc=11

Dec 11 10:40:33 PVD-ACCESS-3650-Stack-SW_1_RP_0 pman[11707]: %PMAN-3-RPSWITCH: RP switch initiated. Critical process linux_iosd-image has failed (rc 0)
Dec 11 10:40:34 PVD-ACCESS-3650-Stack-SW_1_RP_0 kernel: device_release(80000001041bcee8,80000000ea694500)
Dec 11 10:40:36 PVD-ACCESS-3650-Stack-SW_1_RP_0 pvp[13022]: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Dec 11 10:43:30 PVD-ACCESS-3650-Stack-SW_1_RP_0 kernel: LSMPI: Deregister dual stack diverter
Dec 11 10:43:30 PVD-ACCESS-3650-Stack-SW_1_RP_0 pman[14231]: %PMAN-3-PROCHOLDDOWN: The process linux_iosd-image has been helddown (rc 139)
Dec 11 10:43:33 PVD-ACCESS-3650-Stack-SW_1_RP_0 pvp[14276]: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code

Kasun Bandara · ‎12-11-2023

@paul amaral hi, there is few bugs in this version. not sure which one hitting your device. try upgrading to latest recommended version 16.12.10a

check this for bug details Bug Search Tool (cisco.com)

Please rate this and mark as solution/answer, if this resolved your issue
Good luck
KB

paul amaral · ‎12-12-2023

Yes, there is a few bugs on the version I uses, then again there's bugs on all versions :). I just can't find any bugs similar to the one that caused my crash. I guess I should update to the version you mentioned.

Paul

balaji.bandi · ‎12-12-2023

similar bugs found on the other IOS code :

https://bst.cisco.com/quickview/bug/CSCvb05362

but as suggested upgrade to latest version and check.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

paul amaral · ‎12-12-2023

Thanks, but the affected version is 15, am in 16. I haven't found any bug matching the issue i had that is closer to the code train am on.

P