Etherchannel and STP

[chenson]

Recently there was a debate among some folks about how STP is handled within an Etherchannel. I understand that once an Etherchannel is defined, regardless of PAGP or LACP, that the new port-channel is introduced into the spanning tree and the physical ports are removed. That makes sense. The question is, how are BPDUs passed across the port-channel? I found an article (below) that references that for 2900s and 3550s, it uses the default port. That makes sense. But I can't find any documentation for other switches. 3560s? 4k? 6k? Nexus? Is the same functionality applied across the product lines?

http://www.cisco.com/en/US/customer/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

2900/2950/2550 section:

The default port is used to transmit       traffic, such as Spanning Tree Protocol (STP), multicasts, and       unknown unicasts. The default port can be identified from the       output of the command show etherchannel summary by a       notation of d.

[Giuseppe Larosa]

Hello CHenson,

implementation of ethechannel on C2950 is peculiar and I would not consider it for higher end switches.

STP traffic can be subject to L2 hashing algorithm as they are L2 frames with a multicast destination and so it can use a different member link in different directions. ( on higher end switches I have this impression)

What is important to note is that STP does not handle ALL member links at the same time, and so the use of LACP ( I would consider PAGP legacy now) is highly recommended combined with STP.

Hope to help

Giuseppe

[chenson]

Hello Giuseppe. Thanks for the reply.

I understand your position on the 2950 so that makes sense. However, not being able to find anything else on the other platforms confused me.

As for the STP hashing, you are saying that because a BPDU has a MAC in the frame (and the destination is mcast), it is subject to the same traffic distribution as other packets in the port-channel? OK. That makes sense as well.

Your last statement confuses me. "STP does not handle ALL member links at the same time". Can you elaborate? STP does not see the member links right? Regardless of PAGP or LACP?

NOTE: Yes, we are an LACP standard shop. PAGP is not encouraged in our environment.

--chenson

[Giuseppe Larosa · Accepted solution]

Hello Chenson,

what I would say is: STP can update the member link cost as soon as some mechanism detects that one member link is down, but STP does not provide member link detection as the links are seen as a whole:

if a single BPDU is sent in each direction and always uses the same link, only those two links are "monitored" by STP not the other members of the bundle.

Hope to help

Giuseppe

[chenson]

Thank you. It makes sense now.

[danrya]

The STP "cost" of a portchannel doesn't change when a member link goes down.  STP is only monitoring the reachability through the port channel, not individual links.  When a member link "goes down" the link will be removed from the port channel and STP will NOT need to re-converge, since the "port" is still up and active. An STP event will not occur unless all member ports go "down".

I would recommend PAGP or LACP for the port channel.

You are correct that the BPDU's are hashed just like any other traffic, and will be assigned to a particular member link, but if that member link is removed, a new hash is performed, and it's assigned to another link.

Dan

[Peter Paluch · Accepted solution]

Dan,

The STP "cost" of a portchannel doesn't change when a member link goes 
down

I believe that this is an incorrect statement. The STP cost of a Port-channel interface does change if a member link goes down. In fact, the first thing to look at is the current Port-channel bandwidth value. If N links of capacity C are bundled in a single EtherChannel, Cisco Catalyst switches assign the default bandwidth of N x C to the corresponding Port-channel interface. The STP cost of this interface is subsequently computed from this value. If a member link goes down, the bandwidth of the Port-channel will be automatically updated and thus the STP cost will change as well.

I have verified and confirmed this behavior on 2950, 2960, 3560 and 3560V2 in our lab just now, and I am enclosing a small commented transcript of what does it look like:

Sw2#show cdp n
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
Sw1              Fas 0/23           160          S I      WS-C2950T-Fas 0/23
Sw1              Fas 0/24           160          S I      WS-C2950T-Fas 0/24
Sw2#show ether sum
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        u - unsuitable for bundling
        U - in use      f - failed to allocate aggregator
        d - default port

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Fa0/23(Pd)  Fa0/24(P)

Sw2#show run int fa0/23
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/23
 channel-group 1 mode active
end

Sw2#show run int fa0/24
Building configuration...

Current configuration : 63 bytes
!
interface FastEthernet0/24
 channel-group 1 mode active
end

Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#
00:05:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to down
00:05:06: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to down
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 100000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 19        128.65   P2p
Sw2#
00:05:49: %LINK-3-UPDOWN: Interface FastEthernet0/23, changed state to up
00:05:53: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/23, changed state to up
Sw2#show int po1 | i BW
  MTU 1500 bytes, BW 200000 Kbit, DLY 1000 usec,
Sw2#show span int po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 12        128.65   P2p
Sw2#

Note that originally, the bandwidth of the interface Po1 was set to 200000 Kbit/s (200 Mbps - two FastEthernet ports bundled together), and the resulting STP cost was 12 (running PVST). After I disconnected Fa0/23, the bandwidth decreased automatically to 100000 Kbps and the STP cost was updated to 19 - the common STP cost for a single FastEthernet link. When I returned the link back, the bandwidth was increased back to 200 Mbps and the STP cost was again updated to 12.

Surely, this is not a mechanism of STP alone, rather, this is a cooperation of various mechanisms in the IOS - one of them constantly watches the number of active links in an EtherChannel bundle and updates the default bandwidth on the Port-channel interface, the second - STP - merely reacts to the interface bandwidth modifications.

Best regards,

Peter

[krahmani323 · Accepted solution]

Hello community,

Thanks for the intersting information.

Regarding the very last point about Etherchannel and STP cost, I have seen in this link => http://www.cisco.com/en/US/products/hw/switches/ps700/products_white_paper09186a00801b49a4.shtml#cg6  (check Spanning Tree Port Cost Calculation part) that :

- In CatOS, the spanning-tree port cost for an EtherChannel stays the           same after the port channel member link failure.

- In Cisco IOS Software, the           port cost for the EtherChannel is updated immediately in order to reflect the           new available bandwidth.

Interface	STP Port Cost Short Mode (16-Bit)	STP Port Cost Long Mode (32-Bit)
100-Mbps Ethernet	19	200,000
GE	4	20,000
Two-port Gigabit EtherChannel (GEC)	3	10,000
Three-port GEC	2	6666
Four-port GEC	2	5000
Five-port GEC	2	4000
Six-port GEC	2	3333
Seven-port GEC	2	2857
Eight-port GEC	1	2500
10-GE	2	2000
Two-port 10-Gigabit EtherChannel	1	1000

This discussion was also intersting about Eterchannel STP cost calculation => https://supportforums.cisco.com/message/3107809#3107809

===================================================

An internal formula is used to determine the default cost for an etherchannel which leads to the following default values:

If  you want more granularity and still have the dynamic cost update, you  can change the coding from 16bits to 32bits. In this case you will have a  unique value based on the number of links and the speed of the link.  You need to configure spanning-tree pathcost method long on the swith.

===================================================

Regards.

Karim

[Giuseppe Larosa]

Hello Karim,

you did a very good job in this post clarifying some points that had caused some useful discussion about STP capability or not to update cost of the bundle.

Keep go on in this way

Best Regards

Giuseppe

[chenson]

Agreed. Thank you all for your comments and input. It really helped me understand this.

[krahmani323]

Hi,

Thank you for your message and feedbacks, I appreciate.

And thanks for the nice rating.

Regards.
Karim

[David Williams]

For what it is worth, the Nexus 5010 does not appear to update the  spanning-tree cost when a Port channel member link drops.  The IOS  upstream device does but not the Nexus.  I'm hoping to find a command to  change that behavior but I'm not very optimistic.  I haven't had much luck finding anything that might change that behavior.

So  judging by this post and my testing CATOS does NOT adjust spanning tree  cost with the loss of a member link. IOS DOES adjust spanning tree cost  with the loss of a member link. and NX-OS goes backwards and does NOT  adjust spanning tree cost with the loss of a member link. (At least not  by default)  I'm assuming they figured you would be using vss or vpc and  thus didn't commit the dev resources.  Sure would have been nice.  If  anyone can contradict me please do so!  I really want to be wrong about this. 

[chenson]

Sadly, the more I think about it the more it makes sense. Meaning, I can see both viewpoints.

Viewpoint A - "If I lose a leg from my channel I want STP ro appropriately adjust. These things happen."

Viewpoint B - "If I lose a leg from my channel there is a problem that I need to correct. In the mean time, DON'T adjust STP as I don't want it to flip anything around unintentionally."

I wonder if Veiwpoint B is the Cisco school of thought.....

[danrya]

I was about to type the same thing.

From a "stability" stand point, I wouldn't want an STP convergence on my core due to a single link failure on a port channel.  By updating the cost of the port channel I could pick a different path.  If it's the port channel to the Root I don't think I would want that to happen, since all switches would have to recalculate the tree.

On IOS if you want stability you can manually set the STP cost on the etherchannel, and then it will not cause a spanning tree convergence when a link is lost.  So, in IOS you can have it either way.

[chenson]

Agreed. But a little consistency from Cisco wouldn't hurt either.