"spanning-tree vlan 224" brought the entire network down

[Andreas Schneider]

Hello,

having a plain L2 network with almost 60 (sometimes older) switches. On all switches rapid pvst is configured, VTP is also configured. No comments please

The "VTP-Server" is an 3750g Stack with 12.2(55)SE1. There are 128 Vlan's configured. Now when adding one more Vlan (for e.g. 224) the switch throws a message that he could not start a spanning-tree instance for vlan 224 because the maximum of spanning-tree instances is already reached. At the same time the switch writes "no spanning-tree vlan 224" to it's running-config AND to all running-configs of all switches in the same VTP-Domain... and a loop occourred - So far so bad.

So, after deleting some "never used" vlan's (now there are 124 left) I decided to give vlan 224 a second try. 

On the VTP-Server switch I typed in "spanning-tree vlan 224"... an the network "stops". The cpu went to almost 100% and the syslog fills up with messages of flapping mac and ports. It feels like that spanning-tree starts an entire recalculation...

I never tought that (on "per vlan spanning-tree") the command "spanning-tree vlan 224" (which just starts a spanning-tree instance just for vlan224) has such impact...

Any thoughts or ideas are very appreciated!

Kind regards,

Andreas

[Georg Pauwen]

Hello,

what is the root switch for the new Vlan ?

show spanning-tree vlan 224 root

[Andreas Schneider]

Hello,

the switch itself is the root for that vlan :

switch#sh spanning-tree vlan 224 root
Vlan                   Root ID          Cost    Time  Age Dly  Root Port
---------------- -------------------- --------- ----- --- ---  ------------
VLAN0224         32992 001d.e63e.2980         0    2   20  15

switch#sh spanning-tree vlan 224

VLAN0224
  Spanning tree enabled protocol rstp
  Root ID    Priority    32992
             Address     001d.e63e.2980
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32992  (priority 32768 sys-id-ext 224)
             Address     001d.e63e.2980
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/25            Desg BKN*4         128.25   P2p *LOOP_Inc
Gi1/0/26            Desg BKN*4         128.26   P2p *LOOP_Inc
Gi1/0/27            Desg BKN*4         128.27   P2p *LOOP_Inc
Po1                 Desg FWD 3         128.488  P2p
St1                 Desg FWD 4         128.872  P2p
Gi4/0/25            Desg FWD 4         128.187  P2p
Gi4/0/26            Desg BKN*4         128.188  P2p *LOOP_Inc
Gi4/0/27            Desg BKN*4         128.189  P2p *LOOP_Inc

Kind regards,

Andreas

[Georg Pauwen]

Hello,

what if you delete and recreate the Vlan ?

no vlan 224

vlan 224

[Andreas Schneider]

Hi,

sorry but this can't be done, because it's in use already.

I did not having problems creating vlans. The thing is, that the command "spanning-tree vlan 224" caused that the entire network went down. It feels like that STP was recalculating for all (124)vlan's. And this brings the switch to 100% and then the switch seems to flood all ports, resulting in loops, mac- and port flaps...

Kind regards

Andreas

[paul driver]

Hello
If you have exceeded the stp limitation < sh stp summary total), things like broadcast storms can occur, as stp isn’t able to be cope with the extra stp instances.

Manually pruning vlans off the trunks could cut down on the virtual port totasl but long term it a matter of upgrading your switches or migrating on the MST

[Andreas Schneider]

Hello Paul,

yes, I know about the limitation. For long term this network will be migrated to a L3 topology with NO spanning-tree anymore...

But for now I need to figure out the root cause of the network outage. I never imagined that (with 124 configured vlans - so 4 left until the living hell) bringing another stp instance (for vlan 224) up, would cause in a total network "meltdown"...

Could it be the lack of the old 3750g to handle the per vlan spanning-tree?

Kind regards,

Andreas

[paul driver]

Hello

---

@Andreas Schneider wrote:

But for now I need to figure out the root cause of the network outage. I never imagined that (with 124 configured vlans - so 4 left until the living hell) bringing another stp instance (for vlan 224) up, would cause in a total network "meltdown"...

---

It was probably down to a stp loop, as stp couldnt support that additional vlan as such its basically the same a turning stp off for that vlan and allowing it to propagate through your network without it getting blocked

[Andreas Schneider]

Hi,

yes it looks like an stp loop, but I don't understand why this happens for all vlans and not just for the vlan 224...

Kind regards,

Andreas

[paul driver · Accepted solution]

Hello
It’s not just the stp loop, it what it effects, broadcast storms increase exponentially multiple process not only large amount of stp transitions, interface, CPU/memory utilization , switch cam and router arp tables will also get flooded from consistently being updated, IGP peering will flap,all this will have a drastic effect on your network, until that loop is broken and stp convergence is completed.

[Andreas Schneider]

Hello Paul, sorry for the delay. 

ok, I understand the CPU/mem, cam-table etc. But I would expect (especially if running pvst) that behavior just for the vlan 224, not for all vlans...

Kind regards,
Andreas