cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Cisco Community Designated VIP Class of 2020

213
Views
5
Helpful
7
Replies
Beginner

Cisco 3650 stack separation caused L2 loop

Can this happen ?

A 3650 stack of two switches, call them SW1 (master) and SW2 ( member ), this stack is connected to a distribution layer switch with two links, where each switch in the stack has direct uplink to the distribution switch.

Those two uplinks are not in port-channel, but one of them is forwarding ( on SW1) , and the other is in blocking by STP ( on SW2).

Now the master switch in the stack goes down, the master switch in the stack has the active uplink connected to it, at this time the other switch in the stack (member) became a master and the uplink port that was blocking by STP starts forwarding traffic.

Every thing is fine by now, however one switch is down, after 10 minutes or more, the switch that went down (SW1) goes up again, the SW2 sense that the stack port goes up, we could see a log message of it, however the stack functionality never re-established, both are masters and stack port shows down when we check the command 
show switch stack-ports.

 

So the new situation here, we have two 3650 connected by a stack cable but they are not behaving as a one stack switch with one control plane.

Seconds later, when SW1 fully converged, the SW1 uplink to the distribution goes up, seconds later the CPU at the distribution switch goes up to 99% and the whole network goes down !

This actually happened, and my question is, can the stack port forward data plane traffic, and block control plane traffic ? 
Why those switches, even when they were separate could n't calculate STP the right way and block one of their ports ?

Have you seen this before ?

How to prevent this scenario from happening again ( without configuring a port-channel) ?

Can this be a bug ? 

 

Thank you ! 



 

7 REPLIES 7
Hall of Fame Community Legend

Re: Cisco 3650 stack separation caused L2 loop


@Networker A! wrote:

Can this be a bug ? 


Sounds like a bug.  What firmware is the stack running on?

Post the complete output to the command "dir crashinfo-1:" and "dir crashinfo-2:".

Beginner

Re: Cisco 3650 stack separation caused L2 loop

The stack is using 03.03.05SE version.

I will check about the output of the commands.

Beginner

Re: Cisco 3650 stack separation caused L2 loop

The stack is using 03.03.05SE version.
I will check about the output of the commands.
Hall of Fame Community Legend

Re: Cisco 3650 stack separation caused L2 loop

Don't bother checking the "dir" commands.
The issue is the firmware. That is a very, very old firmware.

What is the total uptime of the entire stack? 
Post the complete output to the command "sh proc cpu sort | ex 0.00".

Beginner

Re: Cisco 3650 stack separation caused L2 loop

Do you have a reference that shows someone before us got the same type issue ?
An error code or something, please ?
Even if you know about some bug fixes, anything concrete fact would be appreciated.

The stack had a run time of 3+ years

Here is the output of the command :

#sho processes cpu sorted | exclude 0.00
Core 0: CPU utilization for five seconds: 32%; one minute: 14%; five minutes: 14%
Core 1: CPU utilization for five seconds: 43%; one minute: 9%; five minutes: 8%
Core 2: CPU utilization for five seconds: 23%; one minute: 8%; five minutes: 8%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
5819 2321440 53029597 367 3.82 3.74 3.76 0 stack-mgr
5817 1153490 22690755 80 2.01 1.54 1.52 1088 fed
10504 2095108 13509749 110 0.59 1.43 1.33 0 iosd
5818 1938140 49343159 39 0.15 0.10 0.10 0 platform_mgr
937 41400 684994 60 0.05 0.03 0.04 0 octeon-ethernet
6462 10060 374731 26 0.05 0.05 0.01 0 profiled
6482 595470 703434 846 0.05 0.05 0.05 0 cpumemd
6505 17160 705397 24 0.05 0.05 0.02 0 ngdumper_provid

VIP Mentor

Re: Cisco 3650 stack separation caused L2 loop

that software is 4 years old , definitely the issue
It has a deferral notice on the Cisco website if you try to download it saying not to use it , you could open with TAC or run the show tech through Cisco cli analyzer if you want bug id or check the below release notes that match your version for an ID

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/3se/release_notes/OL30563.html
Highlighted
Hall of Fame Community Legend

Re: Cisco 3650 stack separation caused L2 loop


@Networker A! wrote:

The stack had a run time of 3+ years


Stack uptime is >3 years and running 3.3.X?  
That's a recipe for disaster right there.  

Upgrade the firmware or raise a TAC Case.  

CreatePlease to create content
Content for Community-Ad