Nexus 9396 VPC with VMware ESXi 5.5 Standard Switch

[Jason Sinclair]

We just upgraded our core switch to a Nexus 9396, which is connected to dual UCS 6248 FI's and also individually to some Dell R710's with a dual 10Gb Ethernet card. When connecting these standalone servers with an etherchannel VPC, the management IP address goes up and down and some hosts can talk while others can't. If I leave one port disabled, it works fine. However, as soon as both are enabled, connectivity goes down. These ESXi servers are on a standard license, so distributed switches are out of the question. I did enable an eval license and setup a vDS using the enhanced LACP and that works great, it just won't work as a standard etherchannel with a standard virtual switch. The connectivity to the 6248's is also working without any issues running in LACP. Below is the configuration:

Nexus SW01:

interface Ethernet1/5
  description ESX vPC Member
  switchport mode trunk
  channel-group 202
  no shutdown

interface port-channel202
  description ESX
  switchport mode trunk
  vpc 202

Nexus SW02:

interface Ethernet1/5
  description ESX vPC Member
  switchport mode trunk
  channel-group 202
  no shutdown

interface port-channel202
  description ESX
  switchport mode trunk
  vpc 202

ESXi configuration is a standard switch with "Route based on IP hash" and both adapters active. Am I missing something? Is this setup not supported? Any help/advice would be greatly appreciated.

Thanks!

[Reza Sharifi]

Is VPC configured between the 2 9396 switches?

What is the output of "sh vpc"?

HTH

[Jason Sinclair]

Yes. Here's the output from SW02:

vPC domain id                     : 10
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary, operational primary
Number of vPCs configured         : 10
Peer Gateway                      : Enabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Enabled (timeout = 240 seconds)

​

And output from sh vpc 202

vPC status
----------------------------------------------------------------------
id   Port   Status Consistency Reason                     Active vlans
--   ----   ------ ----------- ------                     ------------
202  Po202  up     success     success                    1,10,21-22,
                                                          24,51-52,78
                                                          ,97-101,201
                                                          ,204,244-24
                                                          5,411-413,4 ....

[Reza Sharifi · Accepted solution]

What happens if you don't use Portchannel, just 2 10gig (1/5 on each switch)?

I think the on the ESX host the NICs use HA and can use only one link. Once one fails it uses the other one.  I think, I remember seeing the same issue with ESX hosts a couple of years ago.

Can you test the redundancy with physical ports and no Portchannels?

HTH

[Jason Sinclair]

I didn't try that..I will give it a shot. Do I need to be concerned with port flapping? I'll make the change and let you know.

Thanks!

[Jason Sinclair]

That worked! I did some testing and watched through the ESXi console as I enabled and disabled links. Failover and redundancy worked well. I also didn't see any port flapping messages on the Nexus. I can live with this..these aren't heavy production servers. I appreciate the help!

[Reza Sharifi]

Thanks for the rating and glad to know its working now.

[susim]

Sorry to reply an answered discussion ,for me jason configuration just worked.

" Can you test the redundancy with physical ports and no Portchannels? "

You mean put the vpc direct under the interface like below 

interface Ethernet1/5
  description ESX vPC Member
  switchport mode trunk
  vpc 202
  no shutdown

Thanks

[Jason Sinclair]

I have it running with no portchannels or VPC. vSphere takes care of the redundancy and failover. I do have some hosts using the distributed switches, and for those I setup a true LACP active/active using VPC and that works well.

[susim]

Do i need to do anything  on vmware side ?

Thanks for the information

[Jason Sinclair]

For a standard switch, no. I have mine setup with the defaults of "Route based on originating virtual port ID" and "Link status only" for load balancing and failover detection.

[susim]

Hi,

For some reason it did not work with me well , some guest facing intermittent connectivity .

can you share ur lacp conf for vds . 

[Jason Sinclair]

Here is our standard switch configuration (same on both switches). The standard switch has both adapters as active with default settings.

interface Ethernet1/3
  description ESX1
  switchport mode trunk
  spanning-tree port type edge trunk

Here is the config using VPC and distributed switching (same on both switches):

interface Ethernet1/5
  description ESX30
  switchport mode trunk
  spanning-tree port type edge trunk
  channel-group 202 mode active

interface port-channel202
  description ESX30 vPC
  switchport mode trunk
  spanning-tree port type edge trunk
  vpc 202
 

For distributed switches, you have to configure LACP otherwise it won't work. You can only setup link aggregations through the web client (using vsphere 6, believe it's the same with 5.5). When creating a link aggregation, make sure you select mode active and load balancing source/destination IP address, TCP/UDP port and VLAN.

[susim]

Thanks a million 

[Jason Sinclair]

Anytime. Let me know if you need anything else.