Re: Virtual Switching System(VSS) problem

aleks · ‎09-18-2018

Hello.

Network scheme is attached below.

Almost 3 months ago we've built VSS pair on 6880-X for VLAN termination (hereafter referred to as "Core").

Cisco IOS Software, c6880x Software (c6880x-ADVENTERPRISEK9-M), Version 15.2(1)SY4, RELEASE SOFTWARE (fc4)

Technical Support: http://www.cisco.com/techsupport

Compiled Mon 10-Apr-17 14:35 by prod_rel_team

ROM: System Bootstrap, Version 15.1(02)SY01 [ Rel 1.1], RELEASE SOFTWARE

BOOTLDR:

CORE uptime is 16 weeks, 6 days, 15 hours, 37 minutes

Uptime for this control processor is 16 weeks, 6 days, 15 hours, 27 minutes

System returned to ROM by Stateful Switchover at 14:27:39 EEDT Wed May 23 2018

System restarted at 15:52:10 EEDT Wed May 23 2018

System image file is "bootdisk:/c6880x-adventerprisek9-mz.SPA.152-1.SY4.bin"

Cisco C6880-X-LE ( Intel(R) Core(TM) i3- CPU @ 2.00GHz ) processor (revision ) with 1686527K/409600K bytes of memory.

Processor board ID SAL1948U2UG

Processor signature 0xA7060200

Last reset from s/w reset

327 Virtual Ethernet interfaces

1 Gigabit Ethernet interface

64 Ten Gigabit Ethernet interfaces

1955824K bytes of USB Flash bootdisk (Read/Write)

Configuration register is 0x2102

Recently we've encountered a problem with traffic bypass. Here's an example (here and further I am referring to the scheme for easier understanding):

- on the Po3 (Te1/1/2+Te2/1/2) which is connected to AGG2 L2 switch suddenly half of clients become unreachable

- AGG2 becomes inaccessible for management

- half of clients behind AGG2 are still working

- no logs on AGG2 nor on Core

Solution: interface Te2/1/2 shutdown. Port-channel 3 operates in single physical port Te1/1/2. All clients are brought back, management becomes accessible again. And due to VSS we've got traffic partially running through VSL link, and it is not good afaik.

Speculation: looks like all traffic passing through Te2/1/2 is being... blocked? Where and how? The issue is being compounded by the fact there're no logs (including SEA logs) on both enclosures, related to loops, OSPF state changes, etc.

. Progression: - Later, the same problem appeared on AGG3. Same solution: bringing Port-channel down to 1 physical link with interface Te1/5/10 shutdown. - Shutting down Port-channel and bringing in back again has no effect. - If we try to bring back disabled interfaces, problem immediately appears - half of the traffic is blocked. Here how it looks like in logs:

// Core
Sep 19 06:38:22.574 EEDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to down
Sep 19 06:38:23.362 EEDT: %LINK-SW2-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to up
Sep 19 06:38:23.854 EEDT: %DTP-SW2-5-TRUNKPORTON: Port Te1/1/3 has become dot1q trunk
Sep 19 06:38:26.098 EEDT: %EC-SW2-5-BUNDLE: Interface Te1/1/3 joined port-channel Po4
Sep 19 06:38:26.198 EEDT: %LINEPROTO-SW2-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/3, changed state to up
Sep 19 06:38:26.183 EEDT: %EC-SW1_STBY-5-BUNDLE: Interface Te1/1/3 joined port-channel Po4

// AGG3
Sep 19 06:38:23.816 EEDT: %LINK-3-UPDOWN: Interface TenGigabitEthernet4/1, changed state to up
Sep 19 06:38:24.308 EEDT: %DTP-SP-5-TRUNKPORTON: Port Te4/1 has become dot1q trunk
Sep 19 06:38:26.148 EEDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet4/1, changed state to up
Sep 19 06:38:26.100 EEDT: %EC-SP-5-BUNDLE: Interface Te4/1 joined port-channel Po4

At this point we've lost management access to AGG3 (yet no logs about OSPF neighbor down) and ~half of the active clients behind AGG3 become non-operational. Port-channels from both sides are up and running. I would appreciate any suggestions how to debug and solve the issue.

aleks · ‎09-19-2018

Absolutely no idea what it might be??? :(

andrewswanson · ‎09-20-2018

Hi

What mode are you using for your mutichassis etherchannels? From the document below:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6500-virtual-switching-system-1440/109547-vss-best-practices.html

Do not use on and off options with PAgP or LACP or Trunk protocol negotiation.

PAgP Run Desirable-Desirable with MEC links.
LACP Run Active-Active with MEC links.
Trunk Run Desirable-Desirable with MEC links.

hth

Andy

Rene Mueller · ‎07-23-2019

Hi aleks, did you get the problem solved? We are facing the same problems and I would like to understand what the problem is.

Thanks,

René

aleks · ‎07-26-2019

Hi René.

I apologize for the long answer, I was on vacation. Our problem is described here Cisco Bug: CSCvg81219, so we can say that this is not a VSS problem but a problem with с-6880. Since we have no support, we are still looking for a new IOS that can solve this problem (the problem occurs somewhere every three months). What IOS do you use?

Rene Mueller · ‎08-08-2019

Hi,

we are facing this issue on a WS-C4506-E Switch with SUP v8 installed. Our current IOS version is 152-2.E6. I think I have to open a TAC case for that :-(

aleks · ‎08-08-2019

Hi

Then this problem does not apply the bug indicated in the past message. Of course, if there is support, it is better to go there.
If possible, write what the problem was, thanks.

tudor bucenica · ‎10-23-2019

Hi,

Do you have any update on this issue?
I was just curious if there was any resolution in both cases (6800s & 4500s).

Thank you in advance!

Regards,

Tudor

aleks · ‎02-11-2020

Half a year ago, we upgraded to a new firmware c6880x-adventerprisek9-mz.SPA.155-1.SY2.bin
, but today the same problem again appeared.
Prior to this, the problem appeared every 3 months. Can anyone come across this problem and have solutions?
Our problem is identical as in bug CSCvg81219 or CSCve08995. But the cisco solution is just humiliating!

Rod-Pires · ‎09-12-2023

Hi aleks,

did you have progress on this case?

Regards

aleks · ‎09-12-2023

@Rod-Pires wrote:
Hi aleks,
did you have progress on this case?

Regards

Hi, yep we turned off 6880! Because it is hardware problem. CSCve08995