Can LLDP frames from VMware dVS or VMs disrupt FCoE?
I had an interesting - and terrible - experience with DELL-Force10 S5000 switches, where hosts FCoE paths were simultaneously going down, disrupting storage connectivity. This was happening on a VMware cluster with 10 DELL R620 hosts, mounting QLogic QLE8262 converged adapters.
These CNAs have 2 ports, each connected to a S5000 switch. Each switch also connects a number of storage devices with native FC ports.
What happened is that, randomly, hosts were losing storage paths on both FCoE adapters, at least for a few minutes. After a while, this issue could disappear on one host and start on another one - in practice bringing the whole cluster down.
The diagnosis made by DELL technical support identified the cause in LLDP frames received by their switches, from VMware host ports, which were conflicting with DCBx frames - used by DCB to negotiate FCoE attributes, therefore bringing down DCB and FCoE paths on the port.
The diagnosis was mainly driven by messages like the following, on the switches log:
%STKUNIT0-M:CP %LLDP-5-LLDP_MULTIPLE_PEER_DETECTED: DCBX operationally disabled due to more than one PEER being present on interface Te 0/52
According to DELL, such LLDP frames could come from VMware dVS, where we had enabled LLDP, but also from VM - they mentioned that Windows 2012 and later by default enables LLDP on network interfaces.
I was really astonished! I could not imagine that enabling LLDP on a dVS could bring a virtual infrastructure to its knees, and even more to hear that this could be caused by a VM - breaking all the assumptions of isolation of a VM in a virtual environment. On the other hand, DELL says that this is the DCBx behavior by design - LLDP frames from more than one source would block it.
I therefore submit this important question to the community. Is anyone aware of this issue? How is DCBx implemented on Nexus switches, and is it also potentially vulnerable to such problems? Could it ever react to regular LLDP frames from a dVS or VM (if LLDP frames generated by VMs can go through a VS - actually I wonder about that...)?
Thanks you very much for your replies and insight!
Cisco launched their solution for hybrid cloud solution for the Microsoft Azure public cloud back in September of 2017. Since that time, Cisco has enjoyed great success with installations around the world from Poland to Australia and many points in ...
NoticeThis is not an official guide, just something I've been testing to help during those "difficult" situations. All works here are my own!GoalsThere are some situations where we need to load an image onto a Nexus switch (or other network devices ...
I was playing a little with some show commands on ACI and i found that on leaves there are VXLAN tunnels also towards the APIC (10.0.0.1 in the figure); i was wondering why there should be these tunnels? APIC is using VLAN 3967 as infrastructure VLAN, and...