cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
8573
Views
44
Helpful
14
Replies

STP - loops protection mechanism - Plz advise

Hi all,

Recently, I faced an incident where (by accident)  an external partner of my company used an ethernet cable and connected two switchport interfaces, of one of our access switches .

I have to mention that these access ports are configured with Root, BDPU guard as well as Portfast.

This had a severe effect (unavailability) to our LAN. L2 loops caused our L2, Core Switches and Routers to reach 100% CPU!

I thought that BPDU and Root guard were enough to protect the access ports of our LAN but finally it was not true.

I found the next :

" It should be noted that short-living loops are not prevented by Root or BPDU Guards, if two portfast-enabled ports are connected directly or through the hub."

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

Portfast has been configured to all access ports since these  ports are used by the end users to connect to the network.

My task now is to find a solution to prevent L2 loops to our LAN by similar events.

Similar events could be if a user accidentally  connect his switch to two ports of one of our access switches (to remind that all access-ports are Port Fast enabled) or if an end-user connects a hub to one or two ports of the same or different switches.

After a investigation I found that the next commands

switchport port-security maximum 1

switchport port-security violation {restrict | shutdown}

I believe that if apply these commands to all  access interfaces (Port fast enabled ports) of our switches I could overcome L2 loops and broadcast storms due to the incidents described above.

What do you think? How other companies protect their network from similar events?

I would appreciate if you can share your thoughts and i will be generous with many  points

Thanks!

Vasilis

14 Replies 14

Peter Paluch
Cisco Employee
Cisco Employee

Hi Vasilis,

The outage you have experienced is quite rare but it does show that the PortFast ports are vulnerable to short-lived switching loops, even if protected by the BPDU Guard. The reason is obvious: a PortFast port becomes immediately Forwarding as soon as it comes up, i.e., it is unblocked from the very moment it is declared connected. If two PortFast ports are interconnected, in the moment of their mutual connection, you have an existing switching loop (no frame may be caught in it but it still exists). Theoretically, the ports should start sending BPDUs very soon and lose their PortFast status as a result, or even go into err-disabled state if BPDU Guard is configured. However, if for some reason, the BPDUs are not sent or processed, this loop may never go away.

Using the port-security as you suggested is not going to guaranteedly solve the problem. You see, the port-security is going to react if it detects multiple MAC addresses being learned on a port that allows only for a single MAC address, or if a MAC address learned on a secure port is being learned by a different secure port. However, again, with PortFast ports connected together, the possibility of frames being looped still exists. The port security may allow for only a single MAC address to be learned on a port (and to not be learned on any other secure port), but until it reacts, some time may pass during which a huge inrush of broadcast/multicast/unknown unicast frames can overwhelm the CPU.

Using this port security, you are certainly adding yet another layer of protection that can decrease the probability of a switching loop growing into dangerous sizes. However, it is not going to prevent the switching loop in the first place.

The only proper way is to disable the PortFast on ports which can not be trusted, i.e. on which the user may connect another switch, or which can be inadvertently or deliberately connected together. Those ports will suffer from longer times to become Active, and will suffer from topological changes inside the network in the case of RSTP (because of the Proposal/Agreement mechanism), but the loops will not be created. Indeed, the PortFast is inteded to be configured only on a port where it is certain that no loop can be created.

My two cents on this.

Best regards,

Peter

Hi,

I continue my investigation to this problem and I found an other option that could eliminate the impact of L2 loops. This may be achieved if Iconfigure storm control to the ports.

Storm control prevents traffic on a LAN from being disrupted by a broadcast, multicast, or unicast storm on a port.

http://www.cisco.com/en/US/docs/switches/lan/catalyst2950/software/release/12.1_22ea/SCG/swtrafc.html

What do you think?

Hi Vasil,

The storm control can help to alleviate the impact of a broadcast/multicast storm but it is not going to prevent it. Just like the port security setting you have suggested earlier, the storm control is just another layer of protection that makes it harder for a switching loop to create a dangerous condition, but it does not prevent the switching loop from occuring. In addition, with particular respect to the storm control, setting the appropriate threshold level can be a difficult task. How are you going to differentiate between a correct high-load multicast traffic and a multicast traffic caught in a loop?

Try to see this from a fundamental perspective. STP never opens a port until it is absolutely sure that the port can be opened without creating a switching loop. If a port is put into Forwarding state by STP, it is already established that it can be done safely. Thus, STP is not reacting to the loop creation - on the contrary, it is preventing it. However, if you have PortFast enabled on your ports, STP bypasses the procedures that make sure that the port can indeed be put into Forwarding state, and it just makes it Forwarding right way. This way, interconnecting two PortFast ports necessarily leads to loop creation. Any mechanisms that try to disable it after it has been created may fail because the CPUs and interfaces may get so overloaded that the mechanisms may not be able to work anymore.

Best regards,

Peter

Hi Guys,

There are few points that I need to point out.

1) Portfast is a mode in which a switchport will go directly into forwarding mode, completely elimanating the listenong and learning states.

2) If a port is enabled with portfast and BPDU guard, it will definately go into forwarding state directly,  but if a port is also enabled with BPDUGurard it WILL go into error-disabled mode. Regardless whether it is in portfast or no, the BPDU sensor will keep sensing for BPDUs on the configured switchport.

The following link from the Cisco site suports my claim.

From my perspective the following poblem is result of a misconfiguration.

If a switch is self looped, the higher port number would automatically move towards blocking state. the Keepalive function of a switchport makes sure of that.

I'd suggest you entirely analyse the issue from vlan to vlan basis.

Regards,

Rustom Billimoria

CCIE RS#24798

Hello Rustom,

1) Portfast is a mode in which a switchport will go directly into  forwarding mode, completely elimanating the listenong and learning  states.

Completely correct, and I believe I have stated the same multiple times in my previous replies.

2) If a port is enabled with portfast and BPDU guard, it will definately  go into forwarding state directly,  but if a port is also enabled with  BPDUGurard it WILL go into error-disabled mode. Regardless whether it is  in portfast or no, the BPDU sensor will keep sensing for BPDUs on the  configured switchport.

This is also correct. However, note that without a BPDU arriving on a BPDU Guard-protected port, the BPDU Guard protection will not be tripped. There are situations in which the BPDUs may not be received or processed - think overloaded interfaces, CPUs, filtering of BPDUs, etc.

What I am trying to explain here is that the STP is proactive in prevention of switching loops. All other tools mentioned here - BPDU Guard, Port Security, Storm Control - they are all reactive, i.e. they step into action only after the switching loop has already been created and potentially began to do its damaging work. That is also why I am stressing that with PortFast, switching loops are not prevented, merely detected and reacted to - and this detection/reaction is not reliable.

If a switch is self looped, the higher port number would automatically  move towards blocking state. the Keepalive function of a switchport  makes sure of that. 

Do not confuse the LOOP frames (a.k.a. keepalives) with normal STP operation on a self-looped switch. What you have described is not the keepalive function but merely the STP operation. The keepalive mechanism on switches is effective only against self-looped ports (i.e. a port receiving its own frames back), and in that case, the port gets err-disabled and not just blocked.

Best regards,

Peter

Hi Peter,

M apologies for the mistake. Maybe i should have read the entire post rather than just the initial post. The self looped ports are a pain in the ***. Sometimes physical security of a network is equaly important as compared to access security.

But i think when i worked with the cisco 2900xl, the slef induced ports would be blocked automatically. Or maybe it was the HP series or maybe the  ....ARRHHH!!! Multivendor confusion!!!!!  :-( Sure enough i tried in on my Cisco 3750 and viola!!!! loop thunderstorm :-) . So I am assuming that I'm missing something. But we're humans ...and humans tends to get mislead.

Thanks a lot to Peter and Vasileios for correcting me. Maybe I can get to work on this an come up with a solution.

In case anyone has an update, please do post it here.

Regards

Rustom Billimoria

CCIE RS#24798

I promise five golden points if you can find a solution with this problem.

It is very strange because this problem can appear to all networks with L2 STP topologies (I assume that can be thousands), however as I can see there is not a common solution. If that is true, Cisco should take that into consideration.

And i do not think that disabling port fase is a kind of solution...I remember some problems to end users with PCs connected to a non port fast enabled port. I will investigate it further and i return back

Hi Rustom,

Which link from cisco site supports your claim? Probably, you forgot to add it.

The only link that I have found from Cisco about this issue is the next (already posted)

" It should be noted that short-living loops are not prevented by Root or BPDU Guards, if two portfast-enabled ports are connected directly or through the hub."

http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a0080136673.shtml

and it does not seem to agree with your points.

Thank you

Vasilis

Pulkit Nagpal
Cisco Employee
Cisco Employee

I agree with Peter's remark, I have seen cases like these where customer has looped its own switch (accidentally). The best way to avoid such issues is to make sure that you have all your access port guarded, by guarded I mean no one should be able to play with these ports which unfortunately is not possible in very large networks or networks which are managed only remotely.

So we are only left with one option that is to disable portfast on those unprotected ports.

Regards,

Pulkit

Pulkit,

Thank you!

Best regards,

Peter

Hi all,

I do not think that disabling portfast to access switch port is an option to overcome the L2 loops for incidents as described above.

Portfast is a requirment and problems can appear to the end users without it.

I found from Cisco the next link about problems to end users with Pfast.

Sample:

  • Unable to log in to a network domain, either Microsoft Windows NT or Novell
  • Unable to obtain a DHCP address

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a00800b1500.shtml

So, it seems that the only solution is to minimize the possibility and the impact to this problem than to find a solution to overcome the problem.

Do you agree?

Thanks!

Vasilis

Hi!,

Please shut the unsued ports on the switch, that way the customer cannot loop the N/W also the N/W administrator has control over the N/W

Regards,

Ranjit

Hello Vasilis,

It is very strange because this problem can appear to all networks with  L2 STP topologies (I assume that can be thousands), however as I can see  there is not a common solution. If that is true, Cisco should take that  into consideration.

There is no common solution because fundamentally and principially, there is none if you allow yourself the luxury of putting a connected port to Forwarding state immediately, without first verifying if no switching loops occur.

I can imagine a certain workaround: after a PortFast port is enabled, do not put it into Forwarding state immediately but keep it Discarding, and send a certain kind of BPDUs in fast succession (say, 2 BPDUs in a second for a duration of 3-5 seconds). If during this time, no BPDU is received, then jump to Forwarding and revert to normal Hello interval for sending BPDUs - otherwise, drop the PortFast operational mode and proceed according to normal (R)STP rules. This modification would prevent the loops and allow the port to become Forwarding reasonably fast. However, note that this modification is not about putting a port into Forwarding state immediately. Therefore, my former statement holds: with PortFast as currently implemented, by its very nature, it is impossible to prevent switching loops with guarantee.

Best regards,

Peter

yang yang
Level 1
Level 1

hi

 i fund a thing maybe help.

switchport protected

this use under interface ,

A protected port does not forward any traffic (unicast, multicast, or broadcast) to any other port that is also a protected port. Data traffic cannot be forwarded between protected ports at Layer 2; only control traffic, such as PIM packets, is forwarded because these packets are processed by the CPU and forwarded in software. All data traffic passing between protected ports must be forwarded through a Layer 3 device.

Forwarding behavior between a protected port and a nonprotected port proceeds as usual.

Protected ports are supported on IEEE 802.1Q trunks.

since this will only react with other protected port, it will not effect any daily use, only for some one come in to plug one user port to other near acces port that will make protect port complete and will block unicast, multicast, or broadcast and only allow contril go thought than mean BPDU and with BPDU guard and storm-control. i think will have full protection on this problem, but i am not sure is the protect port need to send out some control in the fist place or have to wait till other pack go thought.

all the best

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco