Cisco WS-C4510R+E stopped forwarding

[BristolWaterIT]

Hi

We have two Cisco WS-C4510R+E acting as a VSS pair running Version 03.04.00.SG software, both Core switches have been up for over well over 7 years without issues previously. Attached to these cores switches we have multiple WS-C2960L-24TS switches as depicted:

The active supervisor is normally in Core SW 1 but recently we have experienced major traffic disruption/loss and noticed the ports normally blocking on the 2960 port-channel had gone forwarding and no other blocked ports anywhere. Also the other ports on the 2960's were all going through the STP stages of blocking, listening, learning and forwarding constantly. These are connected to HyperV servers for management.

The only way to restore connectivity was to force the standby sup in Core SW 2 to become active. Once this was carried out STP appeared to become 'normal' and blocked ports were seen on the 2960's again.

The logs on the Core switches showed other ports/port-channels going up/down but nothing else. CPU also appeared normal at around 30-40%. The only access we had to the Core during the issue was via console.

Has anybody had similar experiences or ideas what may have gone on? I appreciated it's all 'after the horse has bolted' but I'm getting asked for a root cause and am finding it hard to come up with anything concrete.

Appreciate any help that could be offered.

[balaji.bandi]

is this 2960 stacked switches ?

as per the diagram - if the 2960 stacked you should not see the issue at all, i was think you may have stack ring issue - so please check the cables

is this issue only with this Cat 2960 switch ? instead of VSS pair can you turn off SW1 or SW2 of Cat 2960 ?

if this was configured trunk port only allowed vlan in the trunk to start limited other get hammered with STP changes that caused inconsistency.

make sure right device act as root and priroty setup is good option

On the last upgrade to latest code on all devices some bugs might have fixed in the long time.

[BristolWaterIT]

Hi,

Thanks for getting back to me. The 2960s are not stacked, they are 2 separate switches connected using 2x1G ports configured as a port-channel trunking one vlan.

The core switch is configured and acting as root for all vlans. 

I realise we are running old software and they should probably be updated, would the switches being up for several years be an issue? The debate as to whether to reboot switches regularly always seems to divide opinion.

Paul.

[balaji.bandi]

STP evil, of that is not stacking, make STP cost and priority so it uses right Links rather going parent switch, if this switch required back to back connection.

also if that VLAN Locally required make sure Root bridge in right place.,

post the config for validate and suggest in better.

what STP you using  ?

[Leo Laohoo]

Firmware version is 3.4.0 and the last digit is a "0".  

I'd say it is a firmware bug and I would recommend upgrading to a firmware with a last-digit value of >5 like 3.8.X or 3.11.X.

[BristolWaterIT]

Hi

Upgrading to 3.11.x is one of the things we are planning to carry out. Would there be any indication to say it's a firmware bug or anything I can look out for?

Thanks.

Paul.

[balaji.bandi]

read the release notes always before upgrade, (some time i dont go facny latest firm just like that, some are very critical bug may effect your config or environment).

[Leo Laohoo]

---

@BristolWaterIT wrote:
Would there be any indication to say it's a firmware bug or anything I can look out for?

---

The issue went away after there was a failover.  I find that as a sign it was indeed a bug.  If it was a configuration error, it would not have gone away.  

Plus, the version ends with a "0".  

[BristolWaterIT]

Thanks for that. An upgrade is on the cards ay some point so fingers crossed we won't see the problems again.

[balaji.bandi]

we need to upgrade and monitor, in some cases the bugs may re-appears.