4500X VSS Unexpected reload of both switches

[FuadG]

Hello everyone,

We have a VSS pair of 4500x switches that started to reload once in day, standby and active both reloading separately 

IOS version is Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.03.E RELEASE SOFTWARE (fc3)

The error says: Last reload reason: Critical software exception

Interestingly, there is no crashinfo being generated

Directory of bootflash:/

126722 -rwx 178257396 Jan 29 2016 06:06:04 +04:00 cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.bin
126723 -rw- 7425 Mar 14 2016 05:18:17 +04:00 RUN-IPadd.txt
126724 -rw- 10373 Mar 14 2016 07:01:57 +04:00 startup-config.converted_vs-20160314-030157
126725 -rw- 17069 Oct 11 2016 04:55:03 +04:00 startup-config.saved_vs-20161011-005503
126726 -rw- 17170 Oct 13 2016 11:39:56 +04:00 prePAGP.txt

Directory of slavebootflash:/

126722 -rwx 178257396 Jan 29 2016 06:04:58 +04:00 cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.bin
126723 -rw- 7595 Mar 14 2016 05:19:16 +04:00 RUN-IPadd.txt
126724 -rw- 10457 Mar 14 2016 07:03:22 +04:00 startup-config.converted_vs-20160314-030322
126725 -rw- 10457 Mar 14 2016 07:09:46 +04:00 startup-config.saved_vs-20160314-030946

This VSS pair has been working fine for 6 years now, the problem started a couple of days ago

BCT-VSS-01 uptime is 6 years, 9 weeks, 4 days, 8 hours, 44 minutes
Uptime for this control processor is 39 minutes

Seems like this is not a hardware issue since both of them are reloading.

What do you think? 

Thank you in advance

[marce1000]

 - Look into https://software.cisco.com/download/home/284867339/type/282046477/release/3.11.7E  , check if upgrading help , I tend to make abstraction on a current appearing issue because it can always come from outside such as induced by a networking issue (or client traffic) , 

 M.

[balaji.bandi]

Unfortunately, there are no crash logs we see on the switches. when rebooted, what logs you noticed on the switches? what was the recent changes ?

what sup module you have? how many in each chassis?

[FuadG]

Hello,

When rebooted, just ordinary VSS failover logs and interfaces going up and down 

Pls see the modules info below

Switch Number: 1 Role: Virtual Switch Standby

Chassis Type : WS-C4500X-16

Power consumed by backplane : 0 Watts

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------------+-----------
1 16 4500X-16 10GE (SFP+) WS-C4500X-16 JAE20030D4A

M MAC addresses Hw Fw Sw Status
--+--------------------------------+---+------------+----------------+---------
1 00c8.8b31.0640 to 00c8.8b31.064f 1.1 15.0(1r)SG11 03.06.03.E Ok

Mod Redundancy role Operating mode Redundancy status
----+-------------------+-------------------+----------------------------------
1 Standby Supervisor SSO Standby hot

Switch Number: 2 Role: Virtual Switch Active

Chassis Type : WS-C4500X-16

Power consumed by backplane : 0 Watts

Mod Ports Card Type Model Serial No.
---+-----+--------------------------------------+------------------+-----------
1 16 4500X-16 10GE (SFP+) WS-C4500X-16 JAE20030AUT

M MAC addresses Hw Fw Sw Status
--+--------------------------------+---+------------+----------------+---------
1 00c8.8b30.e240 to 00c8.8b30.e24f 1.1 15.0(1r)SG11 03.06.03.E Ok

Mod Redundancy role Operating mode Redundancy status
----+-------------------+-------------------+----------------------------------
1 Active Supervisor SSO Active

[Leo Laohoo]

Post the complete output to the following commands:

1. dir crashinfo:
2. dir slavecrashinfo:
3. sh log on up detail

[FuadG]

Hello Leo,

dir crashinfo:
Directory of crashinfo:/

58466 -rw- 0 Feb 16 2023 19:16:01 +04:00 cilogs
58467 -rw- 0 Jan 29 2016 05:27:05 +04:00 koops.dat
58468 -rw- 192 Feb 16 2023 19:13:27 +04:00 deleted_crash_files
58469 -rwx 3881765 Feb 16 2023 19:13:27 +04:00 crashinfo_iosd_20230216-151157-UTC
58470 -rw- 44 Feb 16 2023 19:13:28 +04:00 last_crashinfo
58471 -rwx 67468601 Feb 16 2023 19:13:28 +04:00 fullcore_iosd_20230216-151157-UTC
58472 -rwx 3926504 Sep 30 2022 03:59:06 +04:00 crashinfo_iosd_20220929-235735-UTC
58473 -rwx 68408336 Sep 30 2022 03:59:07 +04:00 fullcore_iosd_20220929-235735-UTC
58474 -rwx 3894207 Feb 13 2023 06:00:24 +04:00 crashinfo_iosd_20230213-015854-UTC
58475 -rwx 68149154 Feb 13 2023 06:00:25 +04:00 fullcore_iosd_20230213-015854-UTC

dir slavecrashinfo:
Directory of slavecrashinfo:/

62498 -rw- 0 Feb 16 2023 20:39:11 +04:00 cilogs
62499 -rw- 0 Jan 29 2016 05:27:17 +04:00 koops.dat
62500 -rw- 576 Feb 16 2023 20:36:47 +04:00 deleted_crash_files
62501 -rwx 3916846 Feb 16 2023 20:36:47 +04:00 crashinfo_iosd_20230216-163517-UTC
62502 -rw- 44 Feb 16 2023 20:36:47 +04:00 last_crashinfo
42337 -rwx 67380146 Feb 16 2023 20:36:48 +04:00 fullcore_iosd_20230216-163517-UTC
62504 -rwx 3933106 Oct 25 2022 08:17:37 +04:00 crashinfo_iosd_20221025-041607-UTC
62505 -rwx 68160469 Oct 25 2022 08:17:38 +04:00 fullcore_iosd_20221025-041607-UTC
62506 -rwx 3922734 Feb 14 2023 12:51:50 +04:00 crashinfo_iosd_20230214-085019-UTC
62507 -rwx 67841300 Feb 14 2023 12:51:50 +04:00 fullcore_iosd_20230214-085019-UTC

show log onboard uptime detail

Executing the command on VSS member switch role = VSS Active, id = 2

--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 01/17/2016 21:09:17
Total uptime : 6 years 48 weeks 4 days 2 hours 25 minutes
Total downtime : 0 years 8 weeks 1 days 10 hours 46 minutes
Number of resets : 42
Number of slot changes : 2
Current reset reason : 0x9
Current reset timestamp : 02/16/2023 19:20:30
Current slot : 11
Current subslot : 0
Current uptime : 0 years 0 weeks 0 days 15 hours 0 minutes
--------------------------------------------------------------------------------
Reset | |
Reason | Count |
--------------------------------------------------------------------------------
No historical data to display
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
01/17/2016 21:09:17 0x0 0 0 0 0 0
01/17/2016 21:17:21 0x0 0 0 0 0 0
01/29/2016 05:29:35 0x0 0 0 0 0 20
01/29/2016 05:48:35 0x9 0 0 0 0 5
01/29/2016 06:13:16 0x0 0 0 0 0 0
01/29/2016 06:29:33 0x0 0 0 0 0 0
03/14/2016 04:21:39 0x0 0 0 0 0 0
03/14/2016 04:55:21 0x0 0 0 0 0 20
03/14/2016 07:10:09 0x9 0 0 0 2 0
03/28/2016 07:46:46 0x0 0 1 6 23 0
05/24/2016 07:28:38 0x9 0 8 0 23 0
05/24/2016 08:15:45 0x9 0 0 0 0 20
05/25/2016 02:21:35 0x9 0 0 0 17 0
05/25/2016 05:47:56 0x9 0 0 0 0 30
05/25/2016 05:58:39 0x0 0 0 0 0 0
05/25/2016 06:17:10 0x0 0 0 0 0 5
05/25/2016 06:44:05 0x9 0 0 0 0 15
05/25/2016 07:27:06 0x0 0 0 0 0 20
05/25/2016 07:51:37 0x9 0 0 0 0 15
05/25/2016 08:04:22 0x9 0 0 0 0 5
05/25/2016 08:16:08 0x9 0 0 0 0 5
05/25/2016 08:47:05 0x9 0 0 0 0 20
05/25/2016 09:28:04 0x9 0 0 0 0 30
10/10/2016 08:42:58 0x0 0 19 4 23 0
10/10/2016 09:36:03 0x0 0 0 0 0 15
10/10/2016 09:50:59 0x0 0 0 0 0 5
10/10/2016 10:06:20 0x9 0 0 0 0 5
10/11/2016 04:06:30 0x9 0 0 0 17 0
10/13/2016 08:56:14 0x0 0 0 2 4 0
10/13/2016 18:20:26 0x9 0 0 0 1 0
10/13/2016 18:48:21 0x9 0 0 0 0 20
10/13/2016 19:09:10 0x9 0 0 0 0 10
12/12/2016 12:32:11 0x9 0 8 3 17 0
12/17/2016 14:27:23 0x9 0 0 5 1 0
07/12/2017 19:32:58 0x9 0 29 4 4 0
07/12/2017 19:43:46 0x9 0 0 0 0 0
05/03/2019 21:44:07 0x9 1 42 1 1 0
03/06/2021 11:49:41 0x0 1 43 6 13 0
03/06/2021 11:58:59 0x0 0 0 0 0 0
09/30/2022 04:05:58 0x9 1 29 4 15 0
02/13/2023 06:07:21 0x9 0 19 3 1 0
02/16/2023 19:20:30 0x9 0 0 3 13 0
--------------------------------------------------------------------------------

Executing the command on VSS member switch role = VSS Standby, id = 1

--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 01/17/2016 21:09:20
Total uptime : 6 years 48 weeks 4 days 2 hours 15 minutes
Total downtime : 0 years 8 weeks 1 days 18 hours 44 minutes
Number of resets : 44
Number of slot changes : 1
Current reset reason : 0x9
Current reset timestamp : 02/16/2023 20:43:33
Current slot : 1
Current subslot : 0
Current uptime : 0 years 0 weeks 0 days 14 hours 0 minutes
--------------------------------------------------------------------------------
Reset | |
Reason | Count |
--------------------------------------------------------------------------------
No historical data to display
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
01/17/2016 21:09:20 0x0 0 0 0 0 0
01/17/2016 21:17:24 0x0 0 0 0 0 0
01/29/2016 05:29:46 0x0 0 0 0 0 20
01/29/2016 05:47:40 0x9 0 0 0 0 5
01/29/2016 06:12:26 0x0 0 0 0 0 0
01/29/2016 06:28:40 0x0 0 0 0 0 0
03/14/2016 01:40:38 0x0 0 0 0 0 0
03/14/2016 02:37:53 0x0 0 0 0 0 10
03/14/2016 07:09:13 0x9 0 0 0 4 0
03/28/2016 07:00:53 0x0 0 1 6 23 0
05/01/2016 23:17:49 0x0 0 4 6 16 0
05/24/2016 07:29:57 0x0 0 3 1 8 0
05/24/2016 08:23:13 0x9 0 0 0 0 40
05/25/2016 06:34:31 0x9 0 0 0 22 0
05/25/2016 06:57:26 0x9 0 0 0 0 15
05/25/2016 09:27:09 0x9 0 0 0 2 0
10/10/2016 08:42:09 0x0 0 19 4 23 0
10/10/2016 09:13:12 0x0 0 0 0 0 15
10/11/2016 04:05:39 0x9 0 0 0 18 0
10/11/2016 04:18:44 0x9 0 0 0 0 5
10/11/2016 04:55:26 0x9 0 0 0 0 30
10/11/2016 16:20:50 0x0 0 0 0 3 0
10/13/2016 08:55:20 0x0 0 0 2 0 0
10/13/2016 18:40:32 0x0 0 0 0 1 0
10/13/2016 18:56:50 0x9 0 0 0 0 5
12/12/2016 12:31:20 0x9 0 8 3 17 0
12/17/2016 13:02:32 0x9 0 0 5 0 0
05/27/2017 16:32:00 0x9 0 23 0 3 0
07/13/2017 10:06:01 0x9 0 6 4 17 0
08/31/2017 03:21:33 0x0 0 6 6 16 0
12/12/2017 22:50:57 0x0 0 14 5 19 0
07/03/2018 00:30:56 0x0 0 28 6 1 0
07/03/2018 03:04:14 0x0 0 0 0 0 10
07/03/2018 03:41:33 0x0 0 0 0 0 30
07/03/2018 04:41:59 0x0 0 0 0 0 10
07/03/2018 19:50:23 0x0 0 0 0 14 0
07/20/2018 22:57:32 0x0 0 2 3 2 0
01/27/2019 13:09:27 0x0 0 27 1 13 0
04/07/2019 09:10:50 0x0 0 9 6 19 0
02/13/2021 21:36:09 0x9 1 44 5 12 0
02/23/2022 08:25:50 0x9 1 1 2 10 0
10/25/2022 08:24:35 0x9 0 34 5 23 0
02/14/2023 12:58:47 0x9 0 16 0 4 0
02/16/2023 20:43:33 0x9 0 0 2 7 0
--------------------------------------------------------------------------------

[Leo Laohoo]

---

@FuadG

58469 -rwx 3881765 Feb 16 2023 19:13:27 +04:00 crashinfo_iosd_20230216-151157-UTC
58471 -rwx 67468601 Feb 16 2023 19:13:28 +04:00 fullcore_iosd_20230216-151157-UTC
58474 -rwx 3894207 Feb 13 2023 06:00:24 +04:00 crashinfo_iosd_20230213-015854-UTC
58475 -rwx 68149154 Feb 13 2023 06:00:25 +04:00 fullcore_iosd_20230213-015854-UTC​

---

That's a lot of crash files!
From the primary supervisor card, please attach these files so we can have a peep.

[FuadG]

Hello Leo,

Please find attached the files you requested.

[marce1000]

      - You may for instance also connect to your device(s) with  https://cway.cisco.com/cli/ , then at the top left you can run 'System Diagnostics' at the top right you can run 'Crashdump Analyzer'

 M.

[Leo Laohoo]

---

@FuadG wrote:

IOSD-EXT-SIGNAL: Segmentation fault(11), Process = DHCPD Receive

---

Even matches the IOS version:  CSCuy18386 

NOTE:  IOS version 15.2(2)E3 equates to IOS-XE version 3.6.3.

[FuadG]

Hello Leo,

Thank you for your efforts but we do not run any DHCP server on VSS

[Leo Laohoo]

---

@FuadG wrote:
Thank you for your efforts but we do not run any DHCP server on VSS

---

Are there any "ip helper-address" commands?

[FuadG]

Yes, sure. What exactly in the DHCP packets is triggering this? The configuration for this VSS was working fine for years

[Leo Laohoo · Accepted solution]

---

@FuadG wrote:
What exactly in the DHCP packets is triggering this?

---

The crash is caused by the DHCP daemon.  I don't work for Cisco so I cannot provide any clue as to why this is just happening now after several years of uptime.  

The fix is to upgrade the firmware.   Upgrade to either 3.8.10 or 3.11.7.  

[Leo Laohoo]

Another thing:  Currently, the VSS pair is running IOS-XE version 3.6.3 - Last digit (highlighted in RED) is a "3".
Personally, I would never use an IOS/IOS-XE version with a last digit of 0, 1, 2 or 3.  I would start considering an IOS/IOS-XE version that starts with 4 and higher.  

Take, for example, IOS-XE version 3.8.10 or 3.11.7.  

Last thing, whether or not you are going to upgrade, I'd recommend downloading these two firmware files now while it is still possible.