CAT4006\LACP suspends ports when ESX Host is Rebooted EC-5-L3DONTBNDL2

[mewert]

Hello,

We're having an annoying problem with of our recent Nexus 1000V \ vSphere 4.1 implementations. This site is supported by CAT 4006 switches running IOS 12.2(25)EWA14. We have no problem bringing the ESXi Hosts up via the N1KV using LACP, but if there's any sort of interruption to the ESXi Hosts, including ordinary system reboots for maintenance, the CAT 4006 suspends the ports with the following error:

%EC-5-L3DONTBNDL2: Gi2/9 suspended: LACP currently not enabled on the remote port.

And they stay suspended preventing any network communication from the Host, even when ESXi finishes rebooting until we manually take them out of LACP on the Catalyst 4006 switch. This is a classic catch-22 as the ESXi Host can't reconnect to the Nexus and re-establish its LACP bond without being able to first communicate across the physical network.

As we are running the Control and Packet communication VLANs over the Nexus and CAT4006, if an ESX server crashes running one of the supervisor modules we have lost the entire configuration and access to all ESX Hosts when the standby supervisor module didn't come up in time. In these situations all of the other ESX Hosts drop their LACP bonds and the CAT 4006 dutifully suspends all ports.This requires us to to take all ports out of LACP on the CAT in an emergency to restore communication to vSphere. Since we are also running IP Storage over the Nexus, all VMs are lost if we're not very quick about shutting off LACP. It seems that either: LACP is completely dependent on healthy supervisor modules and not just the VEM -OR- even a very very brief (ie. ms) interruption of network communication across all hosts is enough to trigger the CAT 4006 to dump LACP and suspend.

This situtation also requires the network team to always be involved even if we just want to reboot an ESXi Host. If they don't take the ports out of LACP before we reboot, the system won't be able to communicate with the physical network until they do. As you can imagine this is not a great use of resources.

How do we either:

• Prevent IOS on the CAT 4006 from suspending ports when it detects the ESXi Hosts have dropped offline
  

-OR-

• Configure NXOS to keep those ports "alive" or in LACP mode while the ESXi Hosts are disconnected?

We don't want to step back to mac-pinning as LACP provides a noticeable performance improvement on all fronts. However this LACP suspend catch-22 is killing us.

Thanks in advance for help anyone can provide. Relevant excerpts of both IOS and NXOS configs are below. We would be happy to provide any other information that would be useful.

Mark

FYI: the control VLAN is 50, packet VLAN is 51, iSCSI is VLAN 100, VMotion is VLAN 200 and management\VM networking is VLAN 11.

CAT4006 - Base Config:

!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
service compress-config
!
boot-start-marker
boot system flash bootflash:cat4000-i5k91s-mz.122-25.EWA14.bin
boot-end-marker
!
no aaa new-model
vtp domain ''
vtp mode transparent
ip subnet-zero
!
no file verify auto
!
spanning-tree mode rapid-pvst
spanning-tree portfast default
spanning-tree portfast bpduguard default
spanning-tree portfast bpdufilter default

!

!

vlan internal allocation policy ascending

CAT 4006 Port Channel Config:

!
interface Port-channel1
description Nexus 1000V uplinks from ESX01
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 11,50,51,100,200
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk
!

CAT4006 Interface Config (for ESX01 above):

!
interface GigabitEthernet2/9
description ESX01 Intel NIC Port 0
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 11,50,51,100,200
switchport mode trunk
switchport nonegotiate

channel group 1 mode active
speed 1000
spanning-tree portfast trunk
!

!
interface GigabitEthernet2/35
description ESX01 Onboard NIC Port 0
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 11,50,51,100,200
switchport mode trunk
switchport nonegotiate
channel group 1 mode active

speed 1000
spanning-tree portfast trunk
!

!
interface GigabitEthernet4/9
description ESX01 Intel NIC Port 1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 11,50,51,100,200
switchport mode trunk
switchport nonegotiate
channel group 1 mode active

speed 1000
spanning-tree portfast trunk
!

!
interface GigabitEthernet4/35
description ESX01 Onboard NIC Port 1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport trunk allowed vlan 11,50,51,100,200
switchport mode trunk
switchport nonegotiate
channel group 1 mode active

speed 1000
spanning-tree portfast trunk
!

Nexus 1000V Base Config:

version 4.0(4)SV1(3a)

ssh key rsa 2048
ip domain-lookup
ip host NEXUS-VSM-01 192.168.11.XX
kernel core target 0.0.0.0
kernel core limit 1
system default switchport
vem 3
  host vmware id 53d19f64-d663-a017-8922-0030487befa0
vem 4
  host vmware id 53d19f64-d663-a017-8922-003048c0f17e
vem 5
  host vmware id 53d19f64-d663-a017-8922-003048c0ef7a

vrf context management
  ip route 0.0.0.0/0 192.168.11.1
hostname NEXUS-VSM-01
vlan 1
vlan 11
  name Production Network
vlan 50-51,100,200
port-channel load-balance ethernet source-dest-ip-port
vdc NEXUS-VSM-01 id 1
  limit-resource vlan minimum 16 maximum 513
  limit-resource monitor-session minimum 0 maximum 64
  limit-resource vrf minimum 16 maximum 8192
  limit-resource port-channel minimum 0 maximum 256
  limit-resource u4route-mem minimum 32 maximum 80
  limit-resource u6route-mem minimum 16 maximum 48

Nexus Uplink Profile for ESXi Hosts:

port-profile type ethernet ESX-UPLINKS
  vmware port-group
  switchport mode trunk
  switchport trunk native vlan 11
  switchport trunk allowed vlan 11,50-51,100,200
  channel-group auto mode active
  no shutdown
  system vlan 11,50-51,100,200
  state enabled

Nexus interfaces from ESX01:

interface Ethernet3/1
  inherit port-profile ESX-UPLINKS
  mtu 1500

interface Ethernet3/2
  inherit port-profile ESX-UPLINKS
  mtu 1500

interface Ethernet3/3
  inherit port-profile ESX-UPLINKS
  mtu 1500

interface Ethernet3/4
  inherit port-profile ESX-UPLINKS
  mtu 1500

[lwatta]

Mark,

You can try changing the channel-group command on the N1KV to be "passive" instead of "active", but I don't think that will fix the problem.  I would highly recommend you open a case with the Cisco TAC. If you purchased from your licenses from VMware open a case with them and insist on escalation. Otherwise just open a case with the TAC or your network team can open a case as well. There is probably some misconfiguration or bug between lacp on the 4000 and N1KV.

We are working to improve LACP configuration and stability in the next release. There will be some enhancements that make it easier for LACP to be negotiated between the VEM and upstream switch.

louis

[mewert]

Hi Louis,

Thanks for your reply. We've tried all combinations of active and passive on each side of the bond with no luck. The only configuration that doesn't cause the ports to be suspended is: channel-group auto mode on .  Since this doesn't use LACP the Catalyst obviously doesn't suspend any ports for failing to be in LACP mode.

Upon some further investigation and reflection, this has to be an issue with the Nexus 1000V specifically, or it in combination with the particular IOS we are running on the CAT 4006. Rebooting other physical servers or switches connected to this CAT via LACP do NOT trigger it to suspend the ports.

We'll open a case with the TAC.

Question for you in the meantime: is there any benefit to channel-group auto mode on  vs. mac-pinning or CDP?  In this case we're not connecting to two separate switches, but to two different blades in a single CAT, which for all-intents-and-purposes is a single switch. So we're looking for maximum bandwidth and failover without the worry of having two separate access switches, clustered\stacked or not.

Thanks again for your help!

Mark

[lwatta]

Mark,

Most likely you are right and there is kind of issue between the IOS on the CAT and the N1KV NXOS code. I'm sure once you get a TAC case open it will get figured out.

The difference between CDP and Mac-Pinning vs channel-group auto mode active/passive or on. Is that with CDP and MAC-pinning all the load balancing is only on the VEM side. Veth devices get round-robin assigned to one of the uplink ports and stick to that port until something changes. With a real port-channel between the VEM and Switch you have up to 17 available load balancing options.  You are going to get better utilization with a port-channel between the VEM and Switch then you will with Mac-Pinning or CDP.

We recommend that you use LACP port-channels anytime your upstream switch or switches support it.

Hope that answered your question

Louis

[mewert]

Hi Louis,

Thanks for your reply. My question was slightly different. I do understand the limitaitons of MAC pinning. What I was wondering was the difference between using LACP and just etherchannel. Or from the configuration perspective, the difference between:

channel-group auto mode active\passive

and

channel-group auto mode on

Am I correct in understanding that using active or passive simply tells the system to use LACP to negotiate the bond; and to adhere to the 802.3ad standard making it able to connect to non-Cisco switches? And using the second (channel-group auto mode on) uses pure Cisco etherchannel which only supports connectivity with other Cisco switches but otherwise has no performance or failover drawbacks?

I ask because the system works fine if we use channel-group auto mode on. We have no trouble establishing the bonds between the Nexus and the Catalyst; and the Catalyst does not suspend the ports if the ESX Hosts are rebooted since LACP is not used.

So,to summarize: are there any drawbacks to using etherchannel instead of. LACP?  (channel-group auto mode on instead of channel-group auto mode active). And if there are not any drawbacks, why isn't etherchannel the recommended method of connectivity between the Nexus and other Cisco switches instead of LACP?

Thanks again for your help,

Mark

[lwatta]

Mark,

Sorry it has been a very busy week.

Q1

Am I correct in understanding that using active or passive simply tells  the system to use LACP to negotiate the bond; and to adhere to the  802.3ad standard making it able to connect to non-Cisco switches? And using the second (channel-group auto mode on) uses pure Cisco  etherchannel which only supports connectivity with other Cisco switches  but otherwise has no performance or failover drawbacks?

Yes you are correct. Active or Passive attempts LACP and mode on is simple cisco channel group.

Q2

So,to summarize: are there any drawbacks to using etherchannel instead  of. LACP?  (channel-group auto mode on instead of channel-group auto  mode active). And if there are not any drawbacks, why isn't etherchannel  the recommended method of connectivity between the Nexus and other  Cisco switches instead of LACP?

From what I can gather from internal documentation, we prefer a link aggregation protocol because they provide system support for forwarding table arrangement for logical links, traffic distribution mechanism, and failover mechnism (I stole those off a slide :-)) So my understanding is that a static port-channel will still pass traffic on a down link. I have not tested this. I'll give it a try and see. Also LACP insures that the links are configured correctly at both ends. So it ensures consistency. We also prefer to recommend open standards whenever possible.

louis