Cisco 3750 Switch Stack Stop Forwarding traffic / Crash

[dtran]

Hi Everyone,

I had a network melt down last week where I have a stack of 5 3750G's running IOS version 12.2.55.SE10 connecting to another stack of 4 3750v2 running IOS version 15.0.2.SE via a trunk link and suddenly both stacks just crash stop forwarding traffic. I was not seeing anything suspicious from the logs so I went ahead rebooted both stacks and the network recovers after the reboot. 

Has anyone run into this issue in the past ?

Thanks in advance !!! I appreciate any inputs / suggestions !!

Danny

[Reza Sharifi]

Hi,

It is not very common to see to stacks with 2 different versions of IOS crash at the same time.  I would look for signs of a broadcast storm on your network that could have caused all switches to melt down.

HTH

[dtran]

Hi Reza, thanks for responding to my post !!!
Yeah, that's what I suspect too. The switch stack was still responding as I was able to console into it and check out the logs but users were not able to access any resources on the network. How would you track down if a broadcast storm was happening and how to control or prevent broadcast storm ?
Thanks Reza !!
Danny

[Reza Sharifi]

Hi Danny,

So, broadcast storms are kind of tough to track.  If you have hubs on user desks or anywhere connected to your network that is one thing you want to look at. Now, it could be that someone looped a couple of ports together accidentally and as soon as the network was effected he or she unplugged the cable.

I also have seen 3rd party devices with operating systems connected to the network malfunction and cause a broadcast storm. As for how to prevent it, there are a couple of things you can do.

1-Add broadcast and multicast control to user-facing ports.

example:

storm-control broadcast include multicast
storm-control broadcast level 1.00

2-Also adding spanning-tree portfast bpduguard default

will prevent looping loop.

HTH

[dtran]

I do have spanning-tree bpduguard enabled but I don't think the issue was spanning-tree related. If it was spanning-tree that caused the issue, the issue wouldn't last that long correct ? The network should recover on its own once spanning-tree re-converged ?
One thing I notice that was not normal before I rebooted the stack was all the port status lights were blinking really fast, don't look like normal patterns.

[Reza Sharifi]

One thing I notice that was not normal before I rebooted the stack was all the port status lights were blinking really fast, don't look like normal patterns.

What I have seen is that port status lights blinking really fast is a sign of a broadcast storm happening.

HTH

[dtran]

Thanks Reza !!!

If the broadcast storm was the issue. I would think the issue would re-occur even after rebooting the stack ?

Do you have best practice config on how to control broadcast storm that you can share ?

Thanks Reza !!

Danny

[Reza Sharifi]

Hi Danny,

 I am not sure if I have seen any best practice document on storm control but I think you want to set it as low as possible.  So, for example, "storm-control broadcast level 1.00" is 1% of the total bandwidth of an interface.  So, if the bandwidth to end devices is Gig, 1% would be 10Mb. If it goes above 1%, the broadcast packets will be dropped.  There is also an option to shut the port if there is a broadcast storm associated with on interface. Here is a document on how to configure it. 

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/configuration/guide/book/storm.html

HTH

[dtran]

Thanks Reza !!!

Another thing I was thinking, if broadcast storm is the root cause then why would it affect all networks across the switch stack ? as I understand broadcast storm is contained to its own broadcast domain but in my scenario the issue affected all networks across both switch stacks.

Thanks Reza !!!

Danny