Configure ESXI 6.5

[Faisal.Saoud]

I have a BE6K C220, this server has a vmware esxi 6.5, and need to connect that with two cables as aggregation to trunk ports on the switch, how can I do that, what can I do on the vlan option on the esxi console and must check the two nic boxes? what is the configuration to do that on the web to pass more that vlan on the teaming cables? how to pass cimc, management IP on thses cable? Thanks

[Vaijanath Sonvane · Accepted solution]

Hi,

Here is how I have configured in my environment:

Switch Configuration: 

!
vlan 105
 name IP_PHONES
!
vlan 107
 name UC_APPS
!
interface Port-channel14
 description ### PORT CHANNEL FOR CISCO-BE6K-01 ###
 switchport mode trunk
 switchport trunk allowed vlan all
 switchport nonegotiate
 spanning-tree portfast trunk
!
interface Port-channel15
 description ### PORT CHANNEL FOR CISCO-BE6K-02 ###
 switchport mode trunk
 switchport trunk allowed vlan all
 switchport nonegotiate
 spanning-tree portfast trunk
!
interface GigabitEthernet1/0/17
 description ### CONNECTED TO CISCO-BE6K-01 vmnic0 ###
 switchport mode trunk
 switchport nonegotiate
 channel-group 14 mode on
 spanning-tree portfast trunk
!
interface GigabitEthernet1/0/18
 description ### CONNECTED TO CISCO-BE6K-01 vmnic1 ###
 switchport mode trunk
 switchport nonegotiate
 channel-group 14 mode on
 spanning-tree portfast trunk
!
interface GigabitEthernet1/0/19
 description ### CONNECTED TO CISCO-BE6K-01 CIMC MGMT ###
 switchport access vlan 107
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet2/0/17
 description ### CONNECTED TO CISCO-BE6K-02 vmnic0 ###
 switchport mode trunk
 switchport nonegotiate
 channel-group 15 mode on
 spanning-tree portfast trunk
!
interface GigabitEthernet2/0/18
 description ### CONNECTED TO CISCO-BE6K-02 vmnic1 ###
 switchport mode trunk
 switchport nonegotiate
 channel-group 15 mode on
 spanning-tree portfast trunk
!
interface GigabitEthernet2/0/19
 description ### CONNECTED TO CISCO-BE6K-02 CIMC MGMT ###
 switchport access vlan 107
 switchport mode access
 spanning-tree portfast
!

In this configuration, CIMC port on BE6K is connected separately to the access port of the switch. All UC Applications and ESXi Management data is travelling over same trunk ports. If you have enough ports on your Hardware, you can separate this traffic. 

ESXi Configuration: 

1. In ESXi Administration, go to Networking >> Virtual Switches and update configuration settings for existing vSwitch0 or you can create new vSwitch. In this environment, I am using vSwitch0.

 2. Click Edit settings >> Add uplink and update NIC Teaming settings.

3. Go to Port groups >> add port group. Define port group settings including VLAN ID. Most of the networking settings are inherited from vSwitch that you select. You can create multiple port groups with different VLAN ID's.

4. Associate Virtual Machines with the appropriate Port Group.

5. Test the network connectivity. 

I hope this will help you.

[Anthony Holloway]

"When aggregating appliance interfaces, the switch ports to which they are connected must be configured to use 802.3ad link aggregation"

Source: https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/BE6000/InstallationGuide/12_5/be6k_b_installation-guide-M5-1251/be6k_b_installation-guide-1251_appendix_0110.html#CUCM_RF_C5474FB9_00

I think you need to switch your mode from On to Passive.

Also, you correctly set your ESXi load balancing to IP based, but you didn't show in your switch config where you switch it to match, because the default I think is MAC based. Can you confirm what your load balancing is set to on the switch with this command: show port-channel load-balance

[Vaijanath Sonvane]

Hi @Anthony Holloway,

Here is the output from switch:

SERVER-STACK#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-dst-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address

Thank for pointing out the channel-group mode. EtherChannel and IEEE 802.3ad standards are very similar and accomplish the same goal. There are a few differences between the two, other than EtherChannel is Cisco proprietary and 802.3ad is an open standard.

VMware ESXi supports both EtherChannel and IEEE 802.3ad. Here is the URL and configuration example:

https://kb.vmware.com/s/article/1004048 

[Vaijanath Sonvane]

On the VMware side, ESXi 5.1, 5.5, 6.0 and 6.5 support LACP on vDS only and not on Standard Switch. I am using Standard Switch. Below article lists all the requirements for link aggregation for ESXi. One of the requirement is set link aggregation in static mode ON. 

https://kb.vmware.com/s/article/1001938 

[Anthony Holloway]

I can't seem to find a flaw in your research, so my only conclusion is that cisco has bad documentation. Would you be willing to submit a documentation defect, or shall I?

[Vaijanath Sonvane]

Hi @Anthony Holloway 

Please go ahead and submit the documentation defect.

[Anthony Holloway]

@Vaijanath Sonvane Update for you.  I did open a TAC case, and they did agree with everything we discussed here, however, it's not currently turning into a documentation defect just yet.  I'm still trying to convince them to change the wording and/or the example to match the requirements.  Right now, this one person who has my case is stating that since the example does work with distributed switch, it's "not wrong."  My argument is, that it doesn't state the vDS/Licensing requirements for that example config, and I wonder how many BE customers are doing the vDS+Enterprise license, versus those who aren't.

Anyway, I'll update again, once the case is finalized.  Thanks again for the great conversation and the lesson on LACP.  :)

[Vaijanath Sonvane]

Hi @Anthony Holloway,thanks for the update.

[Anthony Holloway]

Got a defect submitted:

I created a doc defect to request to modify the documentation:

CSCvr85341
NIC teaming on Switch example only supported on vDS
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr85341/?reffering_site=dumpcr

[Vaijanath Sonvane]

Good job @Anthony Holloway 

[Faisal.Saoud]

Hi,

Can I with this configuration add another port group in different Vlan and assign it to VM?

Thanks

[Vaijanath Sonvane]

Yes, you can create multiple port groups with different VLANs. But make sure that the VLAN is allowed on switch trunk side.