Catalyst 4500 Series Switch VSS Member Issue

[xafarali19901]

Hi,

we have Cisco Catalyst 4500 Series Switch at our distribution layer with VSS configuration, problem we faced last time is our second switch in VSS goes down all the devices connected with the switch got disconnected but the switch is power up and running but it is not showing in the switch configuration, we have to reboot both switches to get it online. can anyone help me what was the issue.

[Reza Sharifi]

Hi,

What type of sup do you have and that version of IOS are you running?

Are the access switches connected to both VSS chassis? 

What is the output of "sh redun" when VSS functioning correctly?

HTH

[xafarali19901]

Ji Reza,

We have Cisco WS-C4500X-16 witches in VSS config, version info is as follows

Show version = Software (cat4500e-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)

Some of access access switches are connected with both chassis and some are with only one of them.

Show redundancy output is as follows

DIST_VSS_SW#sh redundancy
Redundant System Information :

------------------------------
       Available system uptime = 1 day, 12 hours, 9 minutes
Switchovers system experienced = 0
              Standby failures = 0
        Last switchover reason = none

Hardware Mode = Duplex
    Configured Redundancy Mode = Stateful Switchover
     Operating Redundancy Mode = Stateful Switchover
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
------------------------------
               Active Location = slot 1/1
        Current Software state = ACTIVE
       Uptime in current state = 1 day, 12 hours, 7 minutes
                 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 16-Dec-16 21:17 by prod
               BOOT = bootflash:cat4500e-universalk9.SPA.03.06.06.E.152-2.E6.bin,1;
        Configuration register = 0x2102

Peer Processor Information :
------------------------------
              Standby Location = slot 2/1
        Current Software state = STANDBY HOT
       Uptime in current state = 1 day, 12 hours, 6 minutes
                 Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch  Software (cat4500e-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 16-Dec-16 21:17 by pr
               BOOT = bootflash:cat4500e-universalk9.SPA.03.06.06.E.152-2.E6.bin,1;
        Configuration register = 0x2102

[Reza Sharifi]

Hi,

Ok, so you have 4500x series. The recommended version for this platform running VSS is

03.06.05.E.152-2.E5.bin. And yes, this version is actually a downgrade version of what you are running. Also, I am not saying that this will for sure fix your particular issue but it and it may help.

You may also want to open a ticket with TAC and consult with them about the software versions.

HTH

[xafarali19901]

Hi Reza,

We have downgrade ios version because we were facing some routing issue with our VSS switch which is also acting as a inter vlan router, so Cisco support recommended us to use this version of ios.

[Reza Sharifi]

Hi,

From you description of the issue, it sounds like an IOS bug. I have seen the same behavior in Nexus but not on the 4500x series.  The only way to fix it was to upgrade a couple versions, so the BIOS gets upgraded as well. Have you contacted Cisco about this?

Also, can you post your vss config?

HTH

[xafarali19901]

Hi Reza,

We did not contacted Cisco yet, about IOS upgrade we are not planning any now.

Show virtual switch output is as follow.

DIST_VSS_SW#show switch virtual

Executing the command on VSS member switch role = VSS Active, id = 1


Switch mode                  : Virtual Switch
Virtual switch domain number : 100
Local switch number          : 1
Local switch operational role: Virtual Switch Active
Peer switch number           : 2
Peer switch operational role : Virtual Switch Standby

Executing the command on VSS member switch role = VSS Standby, id = 2


Switch mode                  : Virtual Switch
Virtual switch domain number : 100
Local switch number          : 2
Local switch operational role: Virtual Switch Standby
Peer switch number           : 1
Peer switch operational role : Virtual Switch Active

sh

[Leo Laohoo]

second switch in VSS goes down all the devices connected with the switch got disconnected but the switch is power up and running but it is not showing in the switch configuration

Are you sure the second supervisor card did not boot up in ROMmon?

What exact firmware version is the VSS running on?  Post the complete output to the command "sh version".

[xafarali19901]

Hi leo,

When we reboot the switches the come back online just fine, and things are back to normal.

Show version = Software (cat4500e-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)

[Leo Laohoo]

Can you please post the complete output to the command "sh version".  I am not just looking for the IOS version.

[xafarali19901]

Hi Leo,

The complete output of show version is as follow

DIST_VSS_SW#sh version
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 16-Dec-16 21:17 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.0(1r)SG11
DIST_VSS_SW uptime is 3 days, 11 hours, 34 minutes
Uptime for this control processor is 3 days, 11 hours, 36 minutes
System returned to ROM by reload
System restarted at 04:54:22 GMT Sat Jun 3 2017
System image file is "bootflash:cat4500e-universalk9.SPA.03.06.06.E.152-2.E6.bin"
Jawa Revision 2, Winter Revision 0x0.0x41

Last reload reason: Reload Shelf CLI

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Information for 'WS-C4500X-16'
License Level: ipbase Type: Permanent
Next reboot license Level: ipbase

cisco WS-C4500X-16 (MPC8572) processor (revision 9) with 4194304K bytes of physical memory.
Processor board ID JAE190603ZU
MPC8572 CPU at 1.5GHz, Cisco Catalyst 4500X
Last reset from Reload
4 Virtual Ethernet interfaces
48 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

[Leo Laohoo]

Output looks good.  

Can you post the complete output to the command "dir crashinfo:" and "dir slavecrashinfo:".

[xafarali19901]

Hi,

DIST_VSS_SW#dir crashinfo:
Directory of crashinfo:/

12098 -rw- 0 Jun 3 2017 04:52:58 +04:00 cilogs
12099 -rw- 0 Mar 1 2015 21:29:09 +04:00 koops.dat
12100 -rw- 272 Jun 3 2017 04:50:28 +04:00 last_systemreport_log

DIST_VSS_SW#dir slavecrashinfo:
Directory of slavecrashinfo:/

6050 -rw- 0 Jun 3 2017 04:52:57 +04:00 cilogs
6051 -rw- 0 Mar 1 2015 21:30:00 +04:00 koops.dat
6052 -rw- 0 Mar 22 2015 14:30:03 +04:00 deleted_crash_files
6053 -rwx 4880237 Mar 22 2015 14:30:03 +04:00 crashinfo_eicored_20150322-102942-UTC
6054 -rw- 47 Mar 22 2015 14:30:03 +04:00 last_crashinfo
6055 -rwx 2279424 Mar 22 2015 14:30:03 +04:00 fullcore_eicored_20150322-102942-UTC
6056 -rw- 544 Jun 3 2017 04:50:28 +04:00 last_systemreport_log

[Leo Laohoo]

If the date and time of the VSS system is to be trusted, then I don't see any significant issue pointing to a crash. 

If there was a system crash, the output would've been present in these two sub-directories.  But they are not.  Which leads me to the following conclusion:  System outage was either cause by a power outage or a silent crash (which doesn't leave any crashinfo file/files).

[serge123]

Did you find what triggered the standby CP to reload? I got same issue yesterday on my 4503-E running same Code as yours. thanks,