Suspended LACP Ports and Their Implication

[visitor68]

Hi

I have a Cisco Nexus 7000 dual homed to a pair of Dell s6000 switches in a VLT (like CIsco's VPC - same crap). There are 8 uplinks altogether - 4 going from the Nexus to S6000-1 and 4 more going from the Nexus to s6000-2. All are supposed to be in a port channel.

The 4 ports on the Nexus going to S6000-1 are UP, UP, port channel is UP, all is good....

The 4 ports on the S6000-1 are also 'up, up' and the port channel interface is up..all is good

Now the problem:

The 4 ports on the Nexus going to S6000-2 are in a suspended state

On the S6000-2, the 4 physical ports connected to those suspended ports are 'up, up,' but the port channel is up, DOWN.

Not sure what to make of this.

What exactly are the implications of a port being in a suspended state? I know it usually means that there is a config inconsistency on the physical port, so its taken out of the bundle. But from the line and protocol perspective, is a suspended port still in a "line up, protocol up" state? I would think it would have to be because the S6000-2 is still seeing all 4 physical ports as being "up, up," even though the port channel interface on the S6000-2 is "up, down," which jives with the suspended ports.

Lastly, if the suspended Nexus ports are indeed 'up, up,' even though they are suspended, then STP must be blocking all those ports, but I am not in front of the switch to verify right now.

Any thoughts?

Thanks

[Steve Fuller]

Hi,

When you say the ports are in a suspended state do you mean that the port is showing with an "s" flag when you run the show port-channel summary command or do you mean suspended due to vPC inconsistency?

If it's simply suspended from the LAG and running as a dynamic LAG i.e., using channel-group <group> mode active, it may be the partner device is not sending LACPDU. Can you post the output of show port-channel summary and show lacp counters from the Nexus?

If it's some vPC issue, can you post the output of the show vpc consistency-parameters interface port-channel <channel_number> command?

Regards

[visitor68]

Hi, I mean exactly what I wrote. Its suspended, period. The port shows as 'spnd' or something like that....I don't own the Cisco and I don't remember exactly which command the tech entered, but that's irrelevant to the question I am asking. Yes, its a dynamic LAG.

I know you're trying to help and I appreciate it, but I am very specific in what I am asking. And I was also specific in describing the topology and it is NOT vPC.

Thanks

REPOSTING:

I have a Cisco Nexus 7000 dual homed to a pair of Dell s6000 switches in a VLT (like CIsco's VPC - same crap). There are 8 uplinks altogether - 4 going from the Nexus to S6000-1 and 4 more going from the Nexus to s6000-2. All are supposed to be in a port channel.

The 4 ports on the Nexus going to S6000-1 are UP, UP, port channel is UP, all is good....

The 4 ports on the S6000-1 are also 'up, up' and the port channel interface is up..all is good

Now the problem:

The 4 ports on the Nexus going to S6000-2 are in a suspended state

On the S6000-2, the 4 physical ports connected to those suspended ports are 'up, up,' but the port channel is up, DOWN.

Not sure what to make of this.

What exactly are the implications of a port being in a suspended state? I know it usually means that there is a config inconsistency on the physical port, so its taken out of the bundle. But from the line and protocol perspective, is a suspended port still in a "line up, protocol up" state? I would think it would have to be because the S6000-2 is still seeing all 4 physical ports as being "up, up," even though the port channel interface on the S6000-2 is "up, down," which jives with the suspended ports.

Lastly, if the suspended Nexus ports are indeed 'up, up,' even though they are suspended, then STP must be blocking all those ports, but I am not in front of the switch to verify right now.

[visitor68]

Folks - any help would be greatly appreciated.

Thank you

[Steve Fuller · Accepted solution]

Thanks for the clarification. When I originally read this I misunderstood and thought the Nexus was running vPC also.

So you asked "What exactly are the implications of a port being in a suspended state?"

If a port is suspended it is still part of the LAG, it's simply not bundled and operating as a member port within the LAG. In this instance it will become what the IEEE 802.1AX standard refers to as an Individual port. From the standard:

"A System may determine that a given link is not able to be aggregated with other links. Such links are referred to as Individual links (as opposed to Aggregateable links)."

You then asked: "But from the line and protocol perspective, is a suspended port still in a "line up, protocol up" state?"

Yes. To demonstrate I’ve forced a member port into a suspended state, which can be seen from the SYSLOG message and when looking at the show interface status and the show interface commands:

2015 Sep  4 09:54:31.466 n5k-2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/22: Ethernet1/22 is suspended

n5k-2# sh int status

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 connected trunk     full    10G     10Gbase-SR
[snip]
Eth1/22       --                 suspnd    trunk     full    1000    SFP-1000BAS

n5k-2# sh int eth1/22
Ethernet1/22 is down (suspended)
 Dedicated Interface
  Belongs to Po999
  Hardware: 1000/10000 Ethernet, address: 547f.ee69.8dfd (bia 547f.ee69.8dfd)
[snip]

While suspended on the Nexus switch we can also see that the neighbour device (a Catalyst switch) shows the port as “up, line protocol is up”:

c2960s-1(config-if)#do sh int gi1/0/26
GigabitEthernet1/0/26 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is ccd5.3997.d61a (bia ccd5.3997.d61a)
  [snip]

And we can see that traffic is still being sent and received on the port.

n5k-2# clear count
n5k-2# 2015 Sep  4 11:41:46.634 n5k-2 %LIBIFMGR-5-ALL_COUNTERS_CLEARED: All interface counters cleared by user
n5k-2# sh int eth1/22
Ethernet1/22 is down (suspended)
 Dedicated Interface
  Belongs to Po999
  Hardware: 1000/10000 Ethernet, address: 547f.ee69.8dfd (bia 547f.ee69.8dfd)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is trunk
  auto-duplex, 1000 Mb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 00:02:40
  Last clearing of "show interface" counters 00:00:18
  30 seconds input rate 7552 bits/sec, 6 packets/sec
  30 seconds output rate 112 bits/sec, 0 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 4.12 Kbps, 4 pps; output rate 1.06 Kbps, 2 pps
  RX
    0 unicast packets  142 multicast packets  0 broadcast packets
    142 input packets  13380 bytes
    0 jumbo packets  0 storm suppression bytes
    0 runts  0 giants  0 CRC  0 no buffer
    0 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    0 unicast packets  1 multicast packets  0 broadcast packets
    1 output packets  147 bytes
    0 jumbo packets
    0 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  0 interface resets

You also asked "Lastly, if the suspended Nexus ports are indeed 'up, up,' even though they are suspended, then STP must be blocking all those ports, but I am not in front of the switch to verify right now."

No. STP is not active on the physical interface. The physical port is still part of a LAG as shown in the “Belongs to Po999” from the output of the show interface command above. STP is operating on the port-channel interface and not on the individual member interfaces.

n5k-2(config-if)# show spanning-tree interf eth1/22
No spanning tree information available for Ethernet1/22

Traffic is passed across the interface, and it is under the control of the LAG to discard those frames.

Where STP does come into play is if the port doesn’t go into a suspended state, but remains operating as an Individual (I) port. On the Nexus switches there is a command lacp suspend-individual (see lacp suspend-individual) within the port-channel interface context that controls what should happen to an "I" port. On the Nexus 7000 switches this is enabled by default and so an I port will become suspended. If this configuration is changed such that we have no lacp suspend-individual configured, then the interface does not get suspended, and spanning tree does run on the port.

n5k-2(config-if)# sh port-ch summ
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/15(P)   Eth1/16(P)
11    Po11(SU)    Eth      LACP      Eth1/1(P)    Eth1/2(P)
999   Po999(SD)   Eth      LACP      Eth1/22(I)

n5k-2(config-if)# show spanning-tree interf eth1/22
Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Desg FWD 20000     128.150  P2p
VLAN0171         Altn BLK 20000     128.150  P2p
VLAN0172         Altn BLK 20000     128.150  P2p
VLAN0173         Altn BLK 20000     128.150  P2p
VLAN0174         Altn BLK 20000     128.150  P2p
VLAN0175         Altn BLK 20000     128.150  P2p
VLAN0178         Altn BLK 20000     128.150  P2p

Hopefully that answers the questions you raised, though without access to the Nexus switches it is going to be difficult to prove exactly what is causing the ports to become suspended. As I mentioned in my original response, in the majority of cases I've seen this behaviour, it is because the Nexus is configured for LACP, but the partner is not sending LACPDU.

Regards

[visitor68]

Steve! Now you're cooking with gas, my friend! :-) Awesome answers and thank you so much for taking the time to test all this in your lab.

Sorry it took me so long to get back to you - been busy the last few days and put this issue on the back burner.

It's all very clear, but for one thing: I am having a problem wrapping my head around the notion of being suspended but still part of a bundle. If such a port is still passing traffic, is still in an 'up, up' state and is not acting individually so as to cause a bridging loop, what then is actually 'suspended'?

Thanks!

[Peter Koltl]

Does the Dell VLT cluster support multichassis LAG in the first place?
 

[visitor68]

of course it does...thats the whole purpose of VLT...its like Cisco's vPC, but a little more robust in that you can run a routing protocol over it

[Steve Fuller]

Hi,

Glad I was finally of some help. I must read the question better next time :)

To your last point:

"I am having a problem wrapping my head around the notion of being suspended but still part of a bundle. If such a port is still passing traffic, is still in an 'up, up' state and is not acting individually so as to cause a bridging loop, what then is actually 'suspended'?"

Unfortunately I've not found anything really solid to answer that. The IEEE 802.1AX 2008 standard doesn't talk about a port or link being suspended or use the term suspended in any form. Therefore we have to conclude this term is a Cisco definition for a port behaving in a certain way.

A port, once configured to be part of a port-channel with the channel-group <group> mode command, means that physical port is part of a Link Aggregation Group (LAG). Whether that port is "Aggregateable" and able to send and receive traffic on behalf of the LAG, or is acting as an Individual port, doesn't change the fact the port is part of the LAG.

In my test I stopped the downstream switch sending LACPDU by using the channel-group <group> mode on command. When I change this to channel-group <group> mode active the port (almost) immediately becomes aggregateable and sending/receiving traffic on behalf of the LAG. For this to happen the port must remain under the control of the LAG such that state changes are monitored and acted upon.

So we know the port remains part of the LAG, but I think we have to accept that Cisco put their own spin on what to do with a port that is not aggregateable.; they offer the option to operate as an Individual port or to "suspend" that port.

When it's operating as an Individual port we can see that spanning tree does what we need and blocks VLANs where necessary. When it's operating in the suspended state, I see this as behaving in a somewhat proprietary manner, and so how the switch internals deal with traffic received on a suspended port, is going to be buried in Cisco code.

Sorry I can't give you a more definitive answer.

Regards