ESXi LAG to Stacked Cisco SG switches Assign all VLANs or not?

[NickDaGeek]

Hi everyone, 

I have inherited an existing 2 host setup and have been troubleshooting the network performance since I arrived. I have noticed that not all the VLANs in use by the VMs have been assigned to the LAG group connecting the ESXi hosts to the SG Stacked switches. The question I have is this:  is that a problem or not?

I am reading that ESXi vSwitches are VLAN agnostic i.e. they don't handle them internally or tag the packets. i.e. it only matters to the Cisco Switch when routing packets from the switch to the ESXi host. In which case I am confused. The VLAN used by our wired and wireless clients is not assigned to the LAG group that feeds the hosts and yet it seems to work most of the time. It is more a throughput performance issue that I am seeing with very intermittent reports of drops.

I am a little out of my depth here, I know how to set up the LAGs and vSwitches as per the existing ones and copying the current setup works when I added the third host not so long ago. What I don't know is if this is optimal or not. I have found various internal network configuration issues on the virtual servers and inside the ESXi configuration that were causing problems so I am a bit suspicious. 

Any pointers to further information or suggestions for network configurations like this gratefully recieved.

[KJK99]

"the VLANs in use by the VMs"

How have you determined that?

"I have noticed that not all the VLANs in use by the VMs have been assigned to the LAG group connecting the ESXi hosts to the SG Stacked switches. The question I have is this:  is that a problem or not?"

It depends on the physical switch, VM applications and ESXi Port Groups. In ESXi, Port Groups are where you can set VLAN IDs. Port Groups are mapped to VM application NICs. If VLAN ID is set to anything from 1 to 4094, the ESXi host will tag outgoing frames with the given VLAN ID. If VLAN ID is set to 0, the ESXi host will send untagged frames so tagging for it needs to be done on the switch. If VLAN ID is set to 4095, it means that VM applications themselves do tagging and the ESXi host will let those tagged frames go through. 

https://knowledge.broadcom.com/external/article/311764/vlan-configuration-on-virtual-switches-p.html

[NickDaGeek]

Hi Kris,

Thanks for your reply. Good question about how I have I determined that. I always get confused with the terminolgy of Port Groups. They do not appear on the Configure Menu nor in the Network Menu of my version of ESXi.

So here goes, inside the Host I have vSwitches to which I have assigned VMkernels whose VLAN ID is 0. I have three VMKernels the standard management one and a pair of VMkernels I have used to assign a pair of custom TCP/IP stacks to the vSwitches one for the VMs and the other for our backup solution. This I believe is doing what it should do, segregate traffic between two different LAG so that the backup network does not impact production.

So if I follow what you are saying my internal (ESXi) network does not tag so I need to do that on the switch.

On the switch side there are roughly thriteen different VLAN IDs which in IP Configuration have (in the IPv4 Interface section) been given a static IP and all are showing in IPv4 Routes.

Virtual Servers on the ESXi hosts need access to all of them bar one (guest WiFi) which the servers never need to touch. However, in the VLAN Managment section under Port VLAN Membership if I select Interface type LAG I can see the LAG Groups only have three of the thirteen VLAN IDs assigned and one of them is untagged (the virtual server VM NIC VLAN ID). The other two Tagged VLANs are the VoIP server and CNC Machine VLAN. The Wired and Wireless PC VLAN ID on which all our domain client computers connect are not included. Neither are the other VLANs all of which are in use on the swtich and which one or more servers will interact with.

The bit I don't understand, and why I am asking, is how is this still working? It is working hand has been for years, albeit with some random hiccups (that is what started me investigating the network.)

So how are the packets from these other devices on other VLANs getting routed to and from the virtual servers if the VLAN they are on is not permitted on the LAG that connects the ESXi host to the Switch Stack. My understanding of VLAN is that a LAG or a Port in Trunk Mode with VLANs assigned to it as Tagged should be rejecting any VLAN not included in the list of assigned VLANs. I am not clear on what happens via an Untagged VLAN assigned to a LAG.

 As you can see I am rusty with regard to network switching and despite doing a bit of reading couldn't find an answer to the question. If this is working how is it working and is it best practice? i.e. can I improve the situation by adding all the required VLANs to the LAGs or is not necessary to add anything other than the Server NIC VLAN (Note: this is only defined in the Switch there are no VLAN IDs specified in either the VM NIC config or in the ESXi Config) The interesting thing is that in ESXi Networking if I look at the IPv4 Routes it can see externally on each Physical NIC connecting them to the switch stack it can only see networks that have been added to the LAG.

Sorry if this seems a bit rambling please ask me for clarification of anything you need.

Thanks

Nick K

[KJK99]

@NickDaGeek 

I have a little trouble following your message. I suspect that you look at Port Groups thinking that they are virtual switches. Anyway, all ESXi networking entities can be easily found by clicking on Networking in the Navigator pane. That opens a large pane to the right with tabs for each networking entity. Also, I'm not an ESXi expert, but I can tell you that there are VMkernel NICs, not just VMkernels. VMkernel NICs are used by the ESXi host itself, not VMs. VMs have their own NICs and they do not use any VMkernel NIC. Both VMkernel and VM NICs are mapped to Port Groups. When you look at a virtual switch, you have Port Groups on the left side and physical adapters on the right side.

I think the best practice is to add only VLANs that are necessary. You may have many VLANs in your network, but VMs may be located in only some of them. VMs can access hosts in the other VLANs by means of routing. It looks like your physical switch does inter-VLAN routing. If there aren't any access issues around those VMs, most likely there is no need to add the other VLANs to those LAGs and I would not add them.

[NickDaGeek]

Hi Kris,
Thanks for the explanation on Port Groups. Makes a lot more sense to me now. So does your comment on the VMKernel NIC. It explains why I am doing what I am doing.
So, to sum up if I have understood you correctly Port Groups are just the names given to the ESXi networks that we assign to the VM NIC in the VM Configuration Menu. VMKernel NICs are not used by VMs they just are the way ESXi connects a VMKernel to the internal side of a vSwitch.
Sorry if my explanation of my setup here was not clear. Please allow me to try again. It was a long and complicated route to get where I am today, and it isn't easy for me to explain as I am not a Cisco or ESXi expert either. The complications of Cisco Static LAG and vSwitch meaning the ESXi routing algorithm needs to be set as IP Route Hash are way over my head.
Here goes.
Out of the box ESXi internally only has one default gateway in a single routing table; the management one. It doesn't matter how many other vSwitches you add using other physical NICs; the routing table only ever contains a single default gateway. If you follow best practice the Management NICs are on their own VLAN away from the Production Network VLAN(s). This is exactly what my predecessor did. I believe that cause issues with Layer 3 switching and inter VLAN routing under heavy loads.
If I have understood the networking theory correctly inside ESXi (without VLAN ID assigned) everything is layer 2 and all is fine, it is all separated. A port group assigned to the VMs fed into a separate vSwitch with separate physical NICs into an external static LAG is on its own. It is only when it hits the CISCO switch that the problems start. The packets in the LAG toward the switch are not VLAN tagged but because they are going to a Layer 3 switch they will be routed toward the default gateway. This will have to be the Management Networks IPv4 VLAN Interface because the ESXi vSwitch only has one default gateway and that is the one on the Management VLAN.
If that is correct, and feel free to correct me if I am wrong, then it explains why I was seeing the switch stack overloading during heavy traffic Backups. It was trying to do inter VLAN routing between the various VLANs all going through the same default gateway and it was killing the ESXi management interface. Ping times went so high the vSphere server was complaining of lost network connectivity to the physical hosts in fact during backups I was seeing lost packets too.
I wanted to split the Management network from the VM and Backup networks and make them use the correct VLAN IPv4 Interface on the switches. I added Custom TCP/IP stacks to the ESXi hosts with the correct default gateways. I found the only way to assign a Custom TCP/IP stack to a vSwitch is to add the TCP/IP stack to a VMKernel and assign that VMKernel to the vSwitch.
In my setup now the VMKernel NIC has a static IP in the network range used by the Port Group assigned to that vSwitch. The VMKernel NIC uses a custom TCP/IP stack and gateway in the VLAN for the Port group. Externally I know that works as I can now ping the VMK NIC IP from outside the ESXi host. For example, the Virtual Machine port group is assigned to its own vSwitch and that has a static IP VMKernel NIC in the correct network range for the VM port group and it's TCP/IP stack is using the default gateway of the VM VLAN's IPv4 Interface on the Switch.
I am hoping that this means that the cisco switches are receiving packets from the VMs down the LAG marked with the default gateway of the IPv4 Interface of the correct VLAN so the Cisco switch is not having to route everything via the management VLAN IPv4 Interface
Does this make sense or am I losing the plot. All I can tell you on that front is that it seems to have worked I do not have the problem during backups anymore.
Thanks again for your helpful information, I hope I have managed to explain my setup in better detail this time.
This leads me back to the original question: if I have correctly routed the network traffic from each port group to the correct Cisco VLAN Interface IP over the LAG then logically it follows that I should add that VLAN to the LAG but from what you say any other VLANs in use outside of the ESXi hosts can and should be handled by the inter VLAN routing in the switch.
Have I understood you correctly Kris?

[KJK99]

@NickDaGeek 

I would say yes, but there isn't really any routing done in ESXi. The ESXi virtual switch does not do any routing. It's just plain VLAN assignment and L2 traffic. I also think that your understanding of the default gateway is incorrect. First of all, it has no function in L2. Second, it is not even used in inter-VLAN routing. For inter-VLAN routing, all routes are defined for each VLAN. If your CISCO switch does inter-VLAN routing, you will find those routes there. They will be described as local, directly connected. 

[NickDaGeek]

Thanks Kris,
Ok that helps clarify things, the way you say it the new custom TCP IP stacks and the VMK I added to the vSwitch have not done anything to the header of packets leaving the ESXi so it has had no effect on the Cisco routing. You may well be right. All I can say is that the problem I was having disappeared after I implemented this. It could of course just be a coincidence.
I did notice something weird when doing a TRACERT from a VM on one physical host to a VM on another physical host though and I cannot explain it.
If I trace between two VMs on two different hosts using the Primary NIC (same IP range) there is no intermediate hop to the external VLAN Interface IP for that IP range and the return is instant.
If I trace between the same VM and the secondary (Backup) NIC IP on the other VM it routes out via the VLAN interface IP as the first hop which is a little delayed by around 4ms
Same if I trace to a domain client pc from a domain server; it routes out via the VLAN Interface IP.
Given what you have said I would have assumed that all traffic would route via the VLAN Interface IP on which the source NIC is located. Clearly, we have an exception and I don’t understand it.
Somehow the VM VLAN which is on the LAG but Untagged never touches the VLAN Interface IP it goes straight between VMs in a single almost instant hop even when going between Physical Hosts which means it must go via LAG out to the Switch. All other IP ranges use inter VLAN routing on the Switch as the first hop. That last bit is weird as there is a separate LAG for backup and it is also the only member and it is also Untagged. I don’t know what this means but mention it because it is weird behaviour.
Can I ask if I understand what you mean by Inter VLAN routing showing on the switch.
All I can see under IP Configuration is in IPv4 Interfaces and IPv4 Routes.
In IPv4 Interfaces they are assigned to a VLAN with a static IP. For example, VLAN 2 192.168.0.1 with a /24 netmask.

Then in IPv4 Routes I can see an un-editable
Destination IP 192.168.0.0 prefix 24
Route type local
Route Owner Directly Connected
Outgoing interface VLAN 2.
As this interface is not editable, I am assuming this was automatically created by the switch when the IPv4 Interface is defined.
Is that what you mean about inter VLAN routing showing on the switch?
I am assuming it is as the Server stack is the only switch on which those interfaces are defined. All other switches, feed by trunks from the server stack, do not have these interfaces defined. I therefore assume that all inter vlan traffic from a peripheral switch must return to the stack switch to route between VLANs.
So, if I understand your correctly, it is not broken, do not try and fix it. I will leave things as they are and look for the answers to my intermittent performance issues elsewhere.
Thanks for your help much appreciated.