Cannot get my 3560G switch to talk to ESX 3.5 servers and the EMC NAS.

[klambert1218]

I have been trying for the past couple of weeks to get my 3560G switch to talk to a couple of ESX 3.5 servers, it's VC, and the EMC NAS. The goal is to implement 2 Etherchannels on the physical switch and NIC teaming on the vSwitch. One port-channel will have 1 VLAN traversing it, and the other port-channel will have 2 VLANs traversing it. VLANs were set up on the vSwitch to mirror the VLANs I have on the 3560G. I want to (have to) use load-balancing. Also, I have read other forums and websites to get some ideas on how to set this up, and each solution is different from the other. The problem seems to reside mostly in the VLAN configuration on the physical switch IMHO, because if I just make the switch a dummy switch, everything can talk without issue. The following information, I already know, not because it is my requirement, but rather pieces that must be present for the whole configuration to succeed.

All interfaces in an Etherchannel must belong to the same VLAN or be configured as trunks. I take this to mean that for the second port-channel, I have to configure the interfaces as trunks, since I need 2 different VLANs to traverse the link.  -I have tried configuring them as both trunks and access ports, and neither seemed to work.

vSwitch must be configured for "Route based on IP hash."

port-channel load-balance should use the src-dst-ip option.

flowcontrol receive desired should be set

system mtu jumbo 9000 should be set

Here is a snippet of my configuration:

port-channel load-balance src-dst-ip

no file verify auto

spanning-tree mode pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

interface Port-channel1     ------Port-channel2 is configured similarly to this one.

switchport trunk encap dot1q

switchport trunk native vlan 2

switchport trunk allowed 3     -----Po2 allows vlans 4,5 instead

switchport mode trunk

!

interface Gigabitethernet0/1     ------gi0/1 - 20 are configured similarly to this one.

switchport trunk encap dot1q

switchport trunk native vlan 2

switchport trunk allowed 3     -------gi0/13 - 14 allow vlan 4 and gi0/15 - 20 allow vlan 5

switchport mode trunk

flowcontrol receive desired

channel-group 1 mode on

!

Interface Gigabitethernet0/21 - 28 are shutdown

!

interface Vlan1

no ip address

shutdown

!

interface Vlan2

ip address x.x.1.2 255.255.255.0

!

interface Vlan3 - 5 are configured identically

no ip address

!

ip default-gateway x.x.1.1

This should be enough to give you an idea on where I am and where I need to be. Any information you can provide would be greatly appreciated!

[Reza Sharifi · Accepted solution]

Hi,

Does you config work without using etherchannel?

On the ESX, I think the native vlan is 0, have you define that?

Also, you are using the 3560 as a layer-2 layer-3 device right?

Reza

[klambert1218]

Hi Reza,

To be perfectly honest, I haven't tried it without Etherchannel because Etherchannel is a must for the system on which it is going to be used. As far as the ESX is concerned, a colleague of mine is responsible for that part of this task, although I'm sure he has it set up the correct way, I will make sure I inquire about that. Currently, the 3560G is functioning as a L2 device. I don't think I need to configure any ports as L3 because inter-VLAN communication is not necessary at this particular moment in time.

[klambert1218]

The native vlan on the ESX is set to vlan 2 also. Still nothing works.

[klambert1218]

I have since made changes on the ESX, like Route based on IP hash. I have also made more changes on the Cisco configuration. Some progress has been made, but not everything is as I need it to be.

[klambert1218]

This question is NOT answered. Still looking for assistance in this matter.

[john.wright]

Klambert

I think I read somewhere just recently that the native vlan has to be a vlan not used. I notice that you have vlan 2 routed.

It is a least worth a try.

[Reza Sharifi]

Hello Klambert,

Can you post your latest switch config?

Also, can you post diagram showing switch connectivity to VM server?

Reza

[klambert1218]

This is a snippet of pertinent information for my switch configuration. Also, I have attached a Visio diagram of how the switch connects to the ESX servers and the SAN (please excuse the crudeness).

port-channel load-balance src-dst-ip

no file verify auto

spanning-tree mode pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

interface Port-channel1     ------Port-channels 2 and3 are configured similarly to this one.

switchport trunk encap dot1q

switchport trunk native vlan 2

switchport trunk allowed 3     -----Po2 allows vlan 4 and Po2 allows 5 instead

switchport mode trunk

switchport nonegotiate

flowcontrol receive desired

!

interface Gigabitethernet0/1     ------gi0/1 - 22 are configured similarly to this one.

switchport trunk encap dot1q

switchport trunk native vlan 2

switchport trunk allowed 3     -------gi0/13 - 16 allow vlan 4 and gi0/17 - 22 allow vlan 5

switchport mode trunk

switchport nonegotiate

flowcontrol receive desired

channel-group 1 mode on

!

Interface Gigabitethernet0/23 - 28 are shutdown

!

interface Vlan1

no ip address

shutdown

!

interface Vlan2

ip address x.x.1.2 255.255.255.0

!

interface Vlan3 - 5 are configured identically

no ip address

!

ip default-gateway x.x.1.1

[aeryilmaz]

Hi Klambert,

Curious if you ever got your issue resolved. I have a similar setup as you - ESX 3.5 servers, NICs are trunked and port-channeled. These ESX Hosts are currently passing traffic on various VLANs, but... I have added a new VLAN and attempting to connect these ESX hosts to a Storage appliance on that new VLAN. I ran a layer 2 trace and confirmed the path looks good from the ESX access switch and my Storage access switch, but they cannot talk to each other.

I can post more details, but curious what your resolution was?

Thanks in advance!