Solved: How does a loop form in a misconfigured Etherchannel? - Page 2

Peter Paluch · ‎12-04-2010

Dear friends,

It is a commonly seen and practically proven issue that if two switches are interconnected by a number of parallel links which are bundled into an Etherchannel on one switch (obviously using the on mode) while being unbundled on the second switch, a Layer2 loop may well be created. However, I do not understand the exact mechanism of the formation of this loop.

I am well aware of the basic principles behind: I know that STP treats the Port-channel interface as a single interface, and thus all member links bundled in this Etherchannel share the same STP state/role. I also understand that a broadcast/multicast/unknown unicast frame sent by a port in the Etherchannel will reach the opposite switch and get flooded over all remaining links, eventually arriving back at the switch with the configured Etherchannel.

And this precise moment is where my understanding ends: The frame arrived back and its destination is still unknown. However, from the viewpoint of the switch, the frame came in through a particular Port-channel interface. If this switch floods the frame, it will flood it through all remaining ports except the port through which the frame came in, meaning that the frame will never be sent back through the Port-channel. How does the loop get created, then?

Thank you very much for helping me out with this!

Best regards,

Peter

Edison Ortiz · ‎12-05-2010

Jon,

It will send it back out (it isn't unknown) but I don't see how that situation would happen. Hmm, maybe if it was the STP Root? well, not even ...

Jon Marshall · ‎12-05-2010

Edison

This post just got me thinking about loops and etherchannel etc. I think that the broacast issue we have been discussing is more to do with the broadcast packet setting up the loop rather than actually what happens when the loop is setup. Have a look at the following scenario, which is not uncommon ie. L2 switches with router and see what you think -

R1 (fa0/0) -> (gi0/4) sw1 (po1) -> gi0/1 sw2 -> vlan 10 clients h1, h2, h3 etc..

gi0/2

gi0/3

eveything is in vlan 10. R1 fa0/0 interace is default-gateway for clients. R1 fa0/0 mac-address is aa.b. all clients for vlan 10 are attached to sw2.

R1 arps out for h3 mac-address. Destination mac is broadcast so -

1) sw1 receives arp on gi0/4 interface and records aa.bb to gi0/4 in cam table

2) sw1 floods packet to sw2 on gi0/1. sw1 records this in cam table ie. aa.bb to gi0/1

3) sw2 receives it floods it out and sends it back on gi0/2 and gi0/3.

4) sw1 receives it on gi0/2 & 3 and so updates it's cam table ie. aa.bb is now no longer reachable via gi0/4 but po1.

so sw1 thinks aa.bb is reachable via po1 and sw2 thinks it is reachable via gi0/1.

h2 wants to send traffic to R1. It sends the packet with dst mac of aa.bb which sw2 receives and forwards out of gi0/1. sw1 receives the packet and it's cam table says this address is reachable via po1 so it selects gi0/3 for example and sends it back to sw2. sw2 sends it back to sw1 and so on.

Jon

Peter Paluch · ‎12-05-2010

Jon,

h2 wants to send traffic to R1. It sends the packet with dst mac of aa.bb which sw2 receives and forwards out of gi0/1. sw1 receives the packet and it's cam table says this address is reachable via po1 so it selects gi0/3 for example and sends it back to sw2. sw2 sends it back to sw1 and so on.

Indeed, this would be true if a frame received on the Po1 interface (one of its members) was allowed to be resent through some member link in the Po1 interface. So far, I believe that such behavior is not permitted under any circumstances but I shall probably dive into the IEEE 802.1ad standard to see the gory details.

I have to stress one thing: My original motivation behind this thread was a network collapse caused by an Etherchannel misconfiguration which I attributed to the actual looping of frames in the network - a true looping of frames where a frame is retransmitted in a circle ad infinitum. I have a feeling (but only a feeling, not a definitive evidence) that I have even seen such a loop in motion - network going down, LEDs blinking wildly, switches complaining of rapid MAC relearning, and all that in a small network topology with essentially no other traffic (just a lab network, not a production net) - but I cannot confirm that this was indeed related to an Etherchannel misconfiguration.

From what has been written here, I have learned that a loop with Etherchannel as I imagined it is actually not possible. That leaves me a bit dumbfounded because now I am not sure what exact process is responsible for such a catastrophic outage in networks with misconfigured Etherchannels. I am even starting to doubt whether the outages I have seen were indeed caused by Etherchannel misconfigs.

Best regards,

Peter

Jon Marshall · ‎12-05-2010

Peter

I am even starting to doubt whether the outages I have seen were indeed caused by Etherchannel misconfigs.

I know how you feel I was involved in a discussion on uplinkfast and transit switches recently and for the life of me i still cannot quite get a topology that supports the contention that uplinkfast can lead to loops on transit switches.

And now this. Speaking for myself only, it's at times like this that i realise just how much more there is to learn even on something as fundamental as L2 switching. It makes you question what you think you know.

Fortunately i have 2 3550 switches working there way to me after moving so i suspect i will be spending quite a few hours confirming or indeed relearning the things i knew.

Jon

Peter Paluch · ‎12-05-2010

Edison,

It will send it back out (it isn't unknown)

Really??? If a frame is received on an Etherchannel member link and the source is learned on the Port-channel, the frame will be reflected back? If that is so I am left slack-jawed (actually, my jaw has just broken my keyboard )

I don't see how that situation would happen.

For example, LOOP frames - sent from and destined for the same MAC if my memory serves.

Best regards,

Peter

Edison Ortiz · ‎12-05-2010

Peter,

You left the part out "but I don't see how that situation would happen". Thanks for the editing

How a switch is going to receive a frame with a destination MAC address pointing back to the switch where the frame came from?

Peter Paluch · ‎12-06-2010

Hi Edison,

You left the part out "but I don't see how that situation would happen". Thanks for the editing

I did not leave it out, on the contrary - I have quoted it right below and responded specifically to it by referencing the LOOP frames.

How a switch is going to receive a frame with a destination MAC address pointing back to the switch where the frame came from?

Simply - just perform the clear mac address-table dynamic on the switch where the frame came from. Or wait for the LOOP frames (sent every 10 seconds), as I indicated earlier: they are both sourced and targeted to the same MAC address. Or cause an RSTP TCN condition on the neighboring switch that forces it to flush its CAM table... I think that these situations are actually far more common than it may appear.

Best regards,

Peter

Peter Paluch · ‎12-05-2010

Jon,

Logic dictates that a frame received by a link in an Etherchannel bundle must not be sent back through any of the links constituting the bundle. That is just an extension of the usual rule that a frame is never going to be sent back its own ingress port, even if the destination is learned on that same port.

I may perform a lab experiment tomorrow and get back with the results but I do not expect to learn anything new (although I should not be so biased).

Best regards,

Peter

Jon Marshall · ‎12-05-2010

paluchpeter wrote:

Jon,

Logic dictates that a frame received by a link in an Etherchannel bundle must not be sent back through any of the links constituting the bundle. That is just an extension of the usual rule that a frame is never going to be sent back its own ingress port, even if the destination is learned on that same port.

I may perform a lab experiment tomorrow and get back with the results but I do not expect to learn anything new (although I should not be so biased).

Best regards,

Peter

Logic dictates that a frame received by a link in an Etherchannel bundle must not be sent back through any of the links constituting the bundle.

That was my thinking as well but i wondered if it could happen due to some peculiarities of etherchannel. I am strugging to see how a loop where packets are literally transmitted back and forth would happen because of this etherchannel setup. Unfortunately not having any switches i am unable to verify these sort of things at the moment.

Jon

Jon Marshall · ‎12-05-2010

ediortiz wrote:

Peter,

Packets will be flooded out of all interfaces for unknown broadcast/unicast packets with the exception of the interface where the flood came from.

In a 2 switch topology, Switch A (which has the bundle) will flood the packet out of the members of the bundle expecting Switch B to receive the packet on its bundle.

However Switch B will only receive the packet on one of its physical interfaces and can potentially flood back to Switch B out of its other connected physical interface causing the STP loop.

Regards,

Edison.

Edison

Okay, i'm confused now.

Does etherchannel send a broadcast across all of it links or just pick one of them ?

You also say when switch A floods the broadcast out of all the members of the bundle, switch B will only receive it one interface. Not understanding this as they are still physical connections so if A sends on all links B will receive on all links ?

Could you explain a bit more ?

Jon

Edison Ortiz · ‎12-05-2010

Jon,

Your explanation and mine are the same - I don't understand how you can be confused.

The EC will hash out of one link but the logical link will be treated as one for the STP loop so it will eliminate this interface as a candidate for the egress flood.

*** Edited for Clarification

Jon Marshall · ‎12-05-2010

Peter

And this precise moment is where my understanding ends: The frame arrived back and its destination is still unknown. However, from the viewpoint of the switch, the frame came in through a particular Port-channel interface. If this switch floods the frame, it will flood it through all remaining ports except the port through which the frame came in, meaning that the frame will never be sent back through the Port-channel. How does the loop get created, then?

My undestanding is -

sw1 = switch with etherchannel gi0/1 - 4

sw2 = switch without etherchannel gi0/1 - 4 not etherchanneled

sw1 selects a port in the etherchannel and transmits the broadcast eg gi0/2. It arrives at sw2 as you say and because the there is no etherchannel configured sw2 floods the broadcast on ports gi0/1, 3 & 4 but not 2 because it received it on that port.

sw1 receives the packet back on gi0/1, 3 & 4 but not gi0/2. So sw1 then refloods the packet on gi0/2 back to sw2 and the whole process starts over again.

Jon

Peter Paluch · ‎12-05-2010

Hi Jon,

Thank you very much for joining this thread!

sw1 receives the packet back on gi0/1, 3 & 4 but not gi0/2. So sw1 then refloods the packet on gi0/2 back to sw2 and the whole process starts over again.

Right, the sw1 will receive the frame back on Gi0/1, Gi0/3 and Gi0/4. But for the sw1, all these ports are just members of the Port-channel interface. A frame arriving through any of the member links is treated as arriving through the Port-channel itself. Even the MAC addresses are learned on Port-channel interfaces, not on physical member ports. Therefore, when sw1 forwards the frame, it - by an elementary switch logic - cannot forward it back through the original ingress interface which is Port-channel.

And this is my trouble here.

Best regards,

Peter

Jon Marshall · ‎12-05-2010

Peter

Etherchannel can be configured to learn addresses in 2 ways -

1) aggregate port learning ie. the mac-addresses are associated with the port-channel interface

2) physical port learning - speaks for itself really

If the switch was using physical port learning then there could indeed be a loop created by the method Edison and I described as far as i can see.

However, having said that,i noticed your recent post (yesterday) on the etherchannel broadcast storm and after checking the config guides for both 2960 and 3750 switches it states that physical port learning is not supported on those switches so it doesn't answer the previous posts issue.

Jon

Peter Paluch · ‎12-05-2010

Hi Jon,

Thanks for your insights. Correct, I have almost forgotten about the physical learning method for Etherchannels. But even if taking that into account, I still do not believe that it could cause frames to be caught in a loop. Let me explain.

We all agree that a frame can be returned back to switch A (with the Etherchannel) if the neighboring switch B does not have the Etherchannel configured. No doubt about that. Jon, now, your supposition is that because the unknown frame is forwarded back to switch A through almost all member links of the bundle, the switch A will somehow re-flood the frame back to switch B. Is that correct?

Note that this situation is not specific only for Etherchannel misconfiguration. The switch B can have the Etherchannel configured just fine but it suddenly receives a burst of frames with different destination MACs, all unknown, or its load-balancing mechanism may react to something different that just the source/destination MAC combination. From the switch A viewpoint, the situation is no different, and yet, no looping ensues - even with physical address learning, otherwise the Etherchannel would be unusable per se.

Best regards,

Peter