Did you try removing the

Carlos Rocha · ‎09-10-2015

Hi all,

We have a pair of 4500X switches running in VSS mode. They are running Version 03.04.06.SG RELEASE SOFTWARE (fc1).

Suddenly, the secondary unit crashed and started rebooting in a loop, as showed below. I would like to know if there is a way to recover this secondary unit. Any help is welcome!

Program Done!
##############
Linux version 2.6.24.4.3.70.k10 (abhakat@build-lnx-036) (gcc version 4.2.1 p7 (Cisco c4.2.1-p7)) #1 SMP Mon May 4 01:47:35 PDT 2015
Starting System Services

diagsk10-post version 5.1.4.0

prod: WS-C4500X-32 part: 73-14959-05 serial: JAE1731077E

Power-on-self-test for Module 1: WS-C4500X-32

CPU Subsystem Tests ...
seeprom: Pass

Traffic: L3 Loopback ...
Test Results: Pass

Traffic: L2 Loopback ...
Test Results: Pass
post done
Exiting to ios...
Starting IOS Services
Starting EOBC bridge eobcbr (mac = 00:22:bd:cd:d2:02)
Platform Manager: acquire hwlock chassis(2)
Platform Manager: starting in VSS mode

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.06.SG RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 04-May-15 02:44 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2010, 2012 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

%SYS-3-CPUHOG: Task is running for (101128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (102128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (103128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (104128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (105128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (106128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (107128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (108128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (109128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (110128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (111128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (112128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (113128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (114128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
%SYS-3-CPUHOG: Task is running for (115128)msecs, more than (2000)msecs (33/33),process = Cat4k Mgmt HiPri.
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
IOSD-WATCHDOG: Process = Cat4k Mgmt HiPri
-Traceback= 1#45ffad4ec0d49987340f66b7b5756e32 :126DC000+609C4FC :126DC000+54ECD00 :126DC000+54ECD00 :126DC000+4ABC66C :126DC000+4ABC6F8 :126DC000+4ABCAF0 :126DC000+4ABCBBC :126DC000+4AA59D0 :126DC000+3E823F4 :126DC000+3E92FFC :126DC000+3E97150 :126DC000+3E98688 :126DC000+3E64664 :126DC000+3E6B6EC :126DC000+3E6BA80 :126DC000+3E6BCA8
Crashinfo file not found.
Early Notification of a Critical job [iosd] exiting abnormally! Initiating cleanup sequence.
Please stand by while rebooting the system...
Restarting system.

Rommon (G) Signature verification PASSED
Rommon (P) Signature verification PASSED
FPGA (P) Signature verification PASSED

************************************************************
* *
* Welcome to Rom Monitor for WS-C4500X-32 System. *
* Copyright (c) 2008-2013 by Cisco Systems, Inc. *
* All rights reserved. *
* *
************************************************************

Rom Monitor (P) Version 15.0(1r)SG10
CPU Rev: 2.2, Board Rev: 9, Board Type: 107
CPLD Mobat Rev: 2.0x549a.0x59a4
Chassis: WS-C4500X-32

MAC Address : f0-f7-55-ce-20-4f
Ip Address : Not set.
Netmask : Not set.
Gateway : Not set.
TftpServer : Not set.

Non-Redundant system or peer not running IOS
System Uplinks & Linecards have been reset!!

***** The system will autoboot in 5 seconds *****

Type control-C to prevent autobooting.
. . . . .

******** The system will autoboot now ********

config-register = 0x2101

Autobooting using the first file from bootflash.....

Loading image !!!!!!!!!!!!!!!!

Checking digital signature....
[bootflash:/cat4500e-universalk9.SPA.03.04.06.SG.151-2.SG6.bin]
Digitally Signed Release Software with key version A

Rommon reg: 0x00084F80
Reset2Reg: 0x00004200

Image load status: 0x00000003
####
Radtrooper 065 controller 0x0481ADD2..0x049B696C Size:0x00661EDC Program Done!
##############
Linux version 2.6.24.4.3.70.k10 (abhakat@build-lnx-036) (gcc version 4.2.1 p7 (Cisco c4.2.1-p7)) #1 SMP Mon May 4 01:47:35 PDT 2015
Starting System Services

diagsk10-post version 5.1.4.0

prod: WS-C4500X-32 part: 73-14959-05 serial: JAE1731077E

Power-on-self-test for Module 1: WS-C4500X-32

CPU Subsystem Tests ...
seeprom: Pass

Traffic: L3 Loopback ...
Test Results: Pass

Traffic: L2 Loopback ...
Test Results: Pass
post done
Exiting to ios...
Starting IOS Services
Starting EOBC bridge eobcbr (mac = 00:22:bd:cd:d2:02)
Platform Manager: acquire hwlock chassis(2)
Platform Manager: starting in VSS mode

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.06.SG RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 04-May-15 02:44 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2010, 2012 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

Mark Malone · ‎09-10-2015

hi you could try recover it in rommon from console

break the auto reboot

***** The system will autoboot in 5 seconds *****

Type control-C to prevent autobooting.
. . . . .

******** The system will autoboot now ********

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/24061-149.html

Carlos Rocha · ‎09-11-2015

Thanks Mark for the post! But unfortunately, it didn't work. I've tried the whole procedure, but the result is always the same. I've even tried to load a newer software release, through the USB, but with no success... Below is the result:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2015.09.11 16:27:05 =~=~=~=~=~=~=~=~=~=~=~=
boot usb0:cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.bin
Loading image !!!!!!!!!!!!!!!!!!!!!!!

Checking digital signature....
[usb0:cat4500e-universalk9.SPA.03.06.03.E.152-2.E3.bin]
Digitally Signed Release Software with key version A

Rommon reg: 0x00084F80
Reset2Reg: 0x00004200

Image load status: 0x00000003
####
Radtrooper 064 controller 0x047F4140..0x0498FCDA Size:0x00661EDC
Program Done!
######################
[ 0.090918] pci 0000:00:00.0: ignoring class b20 (doesn't match header type 01)
[ 0.181039] pci 0001:04:00.0: ignoring class b20 (doesn't match header type 01)
[ 0.273989] pci 0002:0c:00.0: ignoring class b20 (doesn't match header type 01)
Starting System Services

diagsk10-post version 5.1.4.1

prod: WS-C4500X-32 part: 73-14959-05 serial: JAE1731077E

Power-on-self-test for Module 1: WS-C4500X-32

CPU Subsystem Tests ...
seeprom: Pass

Traffic: L3 Loopback ...
Test Results: Pass

Traffic: L2 Loopback ...
Test Results: Pass
post done(66 secs)
Exiting to ios...
Sep 11 12:36:59 %IOSXE-3-PLATFORM: process kernel: [ 98.948644] mpc85xx_pci_err_probe: Unable to requiest irq 0 for MPC85xx PCI err
Sep 11 12:36:59 %IOSXE-3-PLATFORM: process kernel: [ 98.960812] mpc85xx_pcie_err_probe: Unable to requiest irq 0 for MPC85xx PCIe err
Loading gsbu64atomic as gdb64atomic
Failed to bring interface "eth1" up
Using 1 for MTS slot
Platform Manager: starting in standalone mode (active)

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.03.E RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 26-Aug-15 06:05 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

IOSD-WATCHDOG: Process = Cat4k Mgmt HiPri
-Traceback= 1#7f5c8b9
<Fri Sep 11 12:40:59 2015> Message from sysmgr: Reason Code:[2] Reset Reason:Service [iosd] pid:[4742] terminated abnormally [6].
Details:
--------
Service: IOSd service
Description: IOS daemon
Executable: /tmp/sw/mount/cat4500e-universalk9.SPA.152-2.E3.pkg//usr/binos/bin/iosd

Started at Fri Sep 11 12:37:14 2015 (765918 us)
Stopped at Fri Sep 11 12:40:59 2015 (518296 us)
Uptime: 3 minutes 45 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 4742
Exit code: signal 6

CWD: /var/sysmgr/work

PID: 4742
UUID: 512
/etc/rc6.d/S01reboot: line 198: umount_cisco_filesystems: command not found
Please stand by while rebooting the system...
[ 464.893303] Restarting system.

Rommon (G) Signature verification PASSED

Rommon (P) Signature verification PASSED

FPGA (P) Signature verification PASSED

************************************************************
* *
* Welcome to Rom Monitor for WS-C4500X-32 System. *
* Copyright (c) 2008-2013 by Cisco Systems, Inc. *
* All rights reserved. *
* *
************************************************************

Rom Monitor (P) Version 15.0(1r)SG10
CPU Rev: 2.2, Board Rev: 9, Board Type: 107
CPLD Mobat Rev: 2.0x549a.0x59a4
Chassis: WS-C4500X-32

MAC Address : f0-f7-55-ce-20-4f
Ip Address : Not set.
Netmask : Not set.
Gateway : Not set.
TftpServer : Not set.

Non-Redundant system or peer not running IOS
System Uplinks & Linecards have been reset!!

***** The system will autoboot in 5 seconds *****

Type control-C to prevent autobooting.

InayathUlla Sharieff · ‎09-11-2015

Did you try removing the switch from VSS and bring it up as standalone?

Traceback leading towards the hardware failure...

Rolf Fischer · ‎09-10-2015

Hi,

there is still the "Ask the Expert - VSS" right here in the LAN, Switching and Routing section:

https://supportforums.cisco.com/discussion/12599066/ask-expert-implementing-and-troubleshooting-virtual-switching-system-vss-cisco

... a perfect opportunity to discuss such issues!

You may also want to ask there for a software recommendation. We had a lot of trouble with 3.4 and 3.5 and running an 3.6 (Engineering Special if i recall correctly) at the moment.

HTH

Rolf

Carlos Rocha · ‎09-11-2015

Thanks Rolf for the post! I was about to post on the "Ask the Expert - VSS", but now, I don't think it's really related to VSS. I'm trying to reboot the switch with factory configuration, but it doesn't load the IOS until the end...

About the software release, I tried to use the latest release this afternoon, but the result was the same. The switch freezes when trying to load the IOS and after some time reboots again, in a loop...

4500X VSS - secondary unit crash