Loop - broadcast storm in network

[Michael Turkstra]

Good day to you all, i'm with some problem and i can't seem to find the right solution.

at our company we have arround 300 2960 switches, also in some areas of the factory they are using 3com hubs or other hub devices.

i am trying to take them all out, but the factory is to big and there are more then 100 on places i dont know.

My problem is that many times we have a broadcast storm or loop in the network.

users just put in 2 cables in a hub, or the cisco phone both cables in the hub.

the hub is connected to a 2960 switch.

My port configuration is:

interface FastEthernet0/3

switchport access vlan 27

switchport mode access

switchport voice vlan 244

spanning-tree portfast

spanning-tree bpduguard enable

end

the STP settings global are:

spanning-tree mode pvst

spanning-tree loopguard default

spanning-tree portfast bpduguard default

no spanning-tree optimize bpdu transmission

spanning-tree extend system-id

in my opinion the port that have the 3com connected should go in to err-disable when a loop is created because it receive BPDU packets.

unfortuinatly this does not happens and my whole network goes down.

the logging in the switch only indentify that there is mac flapping.

Mar  1 07:28:02: %SW_MATM-4-MACFLAP_NOTIF: Host 0026.18d6.e3d6 in vlan 27 is flapping between port Fa0/2 and port Gi0/1

Mar  1 07:28:18: %SW_MATM-4-MACFLAP_NOTIF: Host e05f.b9e5.acba in vlan 27 is flapping between port Fa0/45 and port Gi0/1

Mar  1 07:28:38: %SW_MATM-4-MACFLAP_NOTIF: Host e05f.b9e5.acba in vlan 27 is flapping between port Fa0/45 and port Gi0/1

Mar  1 07:28:42: %SW_MATM-4-MACFLAP_NOTIF: Host 0026.18d6.e3d6 in vlan 27 is flapping between port Fa0/2 and port Gi0/1

Mar  1 07:28:50: %SW_MATM-4-MACFLAP_NOTIF: Host 0026.18d6.e3d6 in vlan 27 is flapping between port Fa0/2 and port Gi0/1

Mar  1 07:28:50: %SW_MATM-4-MACFLAP_NOTIF: Host e05f.b9e5.acba in vlan 27 is flapping between port Fa0/45 and port Gi0/1

Mar  1 07:29:03: %SW_MATM-4-MACFLAP_NOTIF: Host 0026.18d6.e3d6 in vlan 27 is flapping between port Fa0/2 and port Gi0/1

Mar  1 07:29:06: %SW_MATM-4-MACFLAP_NOTIF: Host e05f.b9e5.acba in vlan 27 is flapping between port Fa0/45 and port Gi0/1

Mar  1 07:29:16: %SW_MATM-4-MACFLAP_NOTIF: Host 0026.18d6.e3d6 in vlan 27 is flapping between port Fa0/2 and port Gi0/1

Mar  1 07:29:18: %SW_MATM-4-MACFLAP_NOTIF: Host e05f.b9e5.acba in vlan 27 is flapping between port Fa0/45 and port Gi0/1

Does someone have an idea to prefent this from happening ??

Thanks a lot!

[Vasilii Mikhailovskii]

Hello, Michael.

Could you also please provide configuration for G0/1 interface?

Per my understanding the issue you faced could be caused by some BPDU filtering device.

If I'm wrong about BPDU filtering device, then remove portfast status from user ports will help you.

You also could apply storm-control configuration per user port with a threshold of 3-5 %, or apply port-security with an appropriate number of MACs per access portl; but this would be in vain.

I guess the best thing to do is to redesign interconnectivity between your Office and Factory (to have no more than 2 links to STP capable device[s]).

PS: could you please share your topology, as I guess, having 300 switches could cause other issues.

[Michael Turkstra]

to prefent all network failure by broadcast storm we have already defided our factory in parts.

We have 2 4507 as core, then connected to 11 3560x switch by L3 connection. (no trunking)

This prefent that the broadcast storm or loop stay in only that area of the network.

then we have 2960 access switches connected to the 3560X gb ports.

and then in some areas they connected 3com switch or other hub devices when they don't have sufficient network ports.

I know that the best would be not use, but here seems impossible.

Factory is changing many times layout.

resume to for example one part of factory: 4507-->(L3 LINK) 3560X --> (trunk) 2960 access switch ---> some parts 3com.

interface GigabitEthernet0/1

switchport mode trunk

end

[Michael Turkstra]

Hello Mikhailovsky.

You are talking about storm controle settings with a treshold of 3 - 5 %.

Could you explain a little to me? i am not familiar with these settings but what i read about it seems to be what i am searching for.

My question is should i only set on the interface "storm-control broadcast level ??"

or do i also need to set multicast and unicast ?

and why is the 3 to 5 %, so it will drop the storm when reach 95 % on interface ?

Thanks for your answer!!

Greetings

[paul driver]

Hello

My question is should i only set on the interface "storm-control broadcast level ??"

or do i also need to set multicast and unicast ? - All depends on what traffic you have traversing your links you need to be sure you dont set the levels to low has to prohibit legitimate IGP/broadcast/mulitcast/unicast traffic this includes any bespoke application traffic that utilzies any of the above

and why is the 3 to 5 %, so it will drop the storm when reach 95 % on interface ? - 5% of an 100mb link would be reached at 5 mb utilization of whatever traffic you define, the higher rate the less effective stom controll is.

To protect against layer 1 devices such are hubs and say access ports with attached switches(managed/unmanaged) you can also apply port-security running along side your current stp bpduguard.

switchport nonegotiate ( disables DTP)

switchport port-security ( enables port security)

switchport port-security aging type inactivity ( ageing of mac- address)

switchport port-security aging time xx  ( mins the mac address will age out)

Switchport port-security violation restrict| shutdown ( violation action of port-security)

Switchport port-security max xx ( number of mac- address allowed on port)

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

[Leo Laohoo]

spanning-tree bpduguard enable

If you have BPDU Guard enabled in every access port and you're still getting a loop then there's a very strong chance someone's snuck this naughty line "errdisable recovery cause bpduguard".

[Michael Turkstra]

Hello all, thanks for the replys.

the port security will not be an option for me because i don't know the quantity of users some times on a port.

And also it is difficult to controle.

i would like to use the storm control future but can some one tel me if the settings are correct i am using?

interface FastEthernet0/4

switchport access vlan 25

switchport mode access

switchport voice vlan 244

storm-control broadcast level 5.00 2.00

storm-control multicast level 5.00 2.00

storm-control unicast level 5.00 2.00

storm-control action shutdown

spanning-tree portfast

i am using unicast as well because in the past we had some problems with NIC's of users.

please i am open for any suggestion.

Thanks!

[Vasilii Mikhailovskii]

Hello, Michael.

"storm-control unicast level 5" is really risky configuration, as it means that whenever your 100M link is utilized for 5M, traffic will be dropped for the rest of the interval.

So, per my understanding, if you want to use unicast, you would better set value around 70-85.

I thought about your case.

The problem is: you extend your Office L2 to uncontrolled (untrusted) factory device. So the solution should be to isolate them from your L2 domain - it means the use of L3 links between Office and Factory.

So, you run routing with factory having only a couple of L3 links without VLAN extension (between factory and office).

---

One more option to think about: configure all the access port with port-security, but set maximum MAC number (let's say 200), aging time equal to 1 minutes and aging mode inactivity.

So, if any MAC tries to bounce between ports (loop over factory devices), one of the ports will go down.

The only drawback I see here: if anybody on the factory is fast enough to disconnect laptop from one switch and connect to another within 1 minute, the port will go down as well.

PS: I recommend L3 connectivity with Factory + police/rate-limit incoming traffic on both links.