C4507 VSS strange issue

[Martynas]

Hello,

Recently I encountered strange problem at my work. In the network main routing is done by two c4507 switches stacked with VSS. All hosts on one of the switches can't reach subnet B, but this happens only on one of the c4507 in the stack, the other one doesn't have this problem. To understand it more clearly, I attached a picture

Also other strange thing is: while the host has a problem reaching subnet B, it doesn't have any problems reaching tens of other subnets, or the internet, but once again this problem can be solved by connecting the host to the other switch in VSS stack.

Tried to do packet capture and the traffic from host connected to C4507 SW1 arrives at the interface but doesn't go out the anywhere. I check dropped packet counters and they stay at 0.It feels like traffic trying to reach subnet B, just disapears if it's coming from hosts connected to C4507 SW1

I tried to reset switching cards with hw-module reset, but it didn't solved my problem. Next thing that comes to mind is rebooting the problematic switch in the stack. Obviously rebooting will be done during scheduled downtime. Is there anything else that can be done before trying to reboot? 

 

Any advice what do to, besides rebooting switch would be helpful.

EDIT: Solution was switch 1 reboot.

[paul driver]

Hello
sh switch virtual
sh switch virtual role
sh switch virtual link

[Martynas]

Hi Paul,

There's the output of the commands:

c4507-1#sh switch virtual

Executing the command on VSS member switch role = VSS Active, id = 1

Switch mode : Virtual Switch
Virtual switch domain number : 100
Local switch number : 1
Local switch operational role: Virtual Switch Active
Peer switch number : 2
Peer switch operational role : Virtual Switch Standby

Executing the command on VSS member switch role = VSS Standby, id = 2

Switch mode : Virtual Switch
Virtual switch domain number : 100
Local switch number : 2
Local switch operational role: Virtual Switch Standby
Peer switch number : 1
Peer switch operational role : Virtual Switch Active

c4507-1#sh switch virtual role

Executing the command on VSS member switch role = VSS Active, id = 1

RRP information for Instance 1

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Preempt Priority Role Local Remote
Number Oper(Conf) Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 1 UP FALSE(N ) 100(100) ACTIVE 0 0
REMOTE 2 UP FALSE(N ) 100(100) STANDBY 4909 9594

Peer 0 represents the local switch

Flags : V - Valid
In dual-active recovery mode: No

Executing the command on VSS member switch role = VSS Standby, id = 2

RRP information for Instance 2

--------------------------------------------------------------------
Valid Flags Peer Preferred Reserved
Count Peer Peer

--------------------------------------------------------------------
TRUE V 1 1 1

Switch Switch Status Preempt Priority Role Local Remote
Number Oper(Conf) Oper(Conf) SID SID
--------------------------------------------------------------------
LOCAL 2 UP FALSE(N ) 100(100) STANDBY 0 0
REMOTE 1 UP FALSE(N ) 100(100) ACTIVE 9594 4909

Peer 0 represents the local switch

Flags : V - Valid
In dual-active recovery mode: No

c4507-1#sh switch virtual link

Executing the command on VSS member switch role = VSS Active, id = 1

VSL Status : UP
VSL Uptime : 1 year, 17 weeks, 1 day, 10 hours, 17 minutes
VSL Control Link : Te1/3/1

Executing the command on VSS member switch role = VSS Standby, id = 2

VSL Status : UP
VSL Uptime : 1 year, 17 weeks, 1 day, 10 hours, 18 minutes
VSL Control Link : Te2/3/1

[paul driver]

Hello
Is subnet B allowed over the trunk to via sw1, and is it in its vlan D/B ?
Sw1
sh int trunk
sh vlan bri

[Martynas]

The link between VSS stack and SW3 is port-channel(Updated the topology to represent it) and it has all needed VLAN. Also VSS switch routes traffic from subnet A from subnet B.

I did a packet capture and I could see the traffic coming from hosts connected to VSS SW 2 entering and going out of the stack, but as soon as the host is connected to VSS SW 1, it's traffic arrives to the stack, but never leaves it.

I don't think there's is any other explanation to this except hardware/software failure.

[MHM Cisco World]

hmm, interesting issue, 
the GW in VSS ?
the GW MAC address in Access SW point to ?

[Martynas]

VSS is gateway to subnet A. It routes traffic from subnet A to subnet B.

I did a packet capture and I could see the traffic coming from hosts connected to VSS SW 2 entering and going out of the stack, but as soon as the host is connected to VSS SW 1, it's traffic arrives to the stack, but never leaves it.

I don't think there's is any other explanation to this except hardware/software failure.

[MHM Cisco World]

nop,I dont think  that.
I will analysis the issue today 

[Martynas]

I'm pretty sure it's software problem.

Imagine two completely identical configuration ports, the only difference is one is on switch 1 and other is on switch 2 of VSS stack.

One port can reach subnet B, the other one can not. There is not other logical explanation.

We scheduled some downtime today, and will try to reboot the switch 1 chassis. Will let you know if it solves my problem.

[Leo Laohoo · Accepted solution]

---

@Martynas wrote:
and will try to reboot the switch 1 chassis

---

Look, if there is a possibility for both chassis to be COLD reboot (completely turn it off and then on), do it.  

[Martynas]

Rebooted switch 1 and all hosts on subnet A were able to reach subnet B. Problem solved.

Thanks for help.

[MHM Cisco World]

one Q, are the client that previous can connect after reboot can connect ?

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/11072-inter-vlan-11072.html

one issue that not relate to SW or HW, 
you must analysis the issue and see where packet stop.

[Leo Laohoo]

Post the complete output to the command "sh version".
NOTE:  Erase the hostname of the switch.  Leave everything else alone.

[Martynas]

c4507-1#sh version
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500es8-UNIVERSALK9NPE-M), Version 03.06.06.E RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Sat 17-Dec-16 04:25 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.1(1r)SG5
c4507-1 uptime is 5 years, 6 weeks, 1 day, 13 hours, 9 minutes
Uptime for this control processor is 4 years, 40 weeks, 3 days, 14 hours, 50 minutes
System returned to ROM by SSO Switchover
System restarted at 16:27:53 EEST Tue Oct 17 2017
System image file is "bootflash:cat4500es8-universalk9npe.SPA.03.06.06.E.152-2.E6.bin"
Jawa Revision 3, RadTrooper Revision 0x0.0x41, Conan Revision 0x1449

Last reload reason: Stateful Switchover

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Information for 'WS-X45-SUP8-E'
License Level: entservices Type: Permanent
Next reboot license Level: entservices

cisco WS-C4507R+E (P5040) processor (revision 2) with 4194304K bytes of physical memory.
Processor board ID xxxxxxxxxx
P5040 CPU at 2.2GHz, Supervisor 8-E
Last reset from Reload
16 Virtual Ethernet interfaces
192 Gigabit Ethernet interfaces
56 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.

Configuration register is 0x2102