Oddest Thing I've Ever Seen - ioFusion Cards cause IP Ping Loss in VMs

[chipskind]

vSphere 6.0 U2 and vSphere 6.0 U3 (problem happened with both versions)

vSphere Distributed Switch and Standard Switch (management)

A handful of Clusters in a Datacenter

ioFusion 1300MP adpaters in two B200 M4 blades - these two blades/ESXi hosts are in their own vSphere Cluster - Standard Switch only

(specific adapter drivers installed and not installed - occurs either way)

When these two specific ESXi hosts are powered up one of our VMs running an application that is Very sensitive to IP losses, starts losing pings leading to the application to start freezing.

Ping losses are noted on various other VMs in the environment but they aren't as sensitive.

None of the VMs affected are in the same Cluster.

No VMs are running on the two ioFusion ESXi hosts.

Since the ONLY difference that we could identify between these two hosts and all the other (Cisco UCS blade related) hosts were the ioFusion cards, I removed the cards and powered up the Hosts.  No Ping loses at all.

I just can't understand why powering up two hosts with ioFusion cards would directly affect a VM in a different Cluster, different VLAN and etc.

Thanks in advance for any comments on this strange issue.

[Kirk J]

Greetings.

You would probably need to take a look at the VMware level stats for the guest VMs losing pings, and the esxi host nic stats (i.e. ethtool -s ).

Would also want to check adapter level dcem-mac stats on the impacted hosts, and general UCSM/FI port counters in general to make sure the blades with the ioFusion cards aren't causing some sort of FCS/CRC storm via the VIC mezz adapters.  Not sure how this would be related unless the ioFusion cards were causing some PCI-E bus issues that was affecting the VIC cards, etc.

I found no record of other similar reports for your combination.

You'll really need to open a TAC case as there are a number of adapter and FI logs that need to be checked, as well as at the esxi level.

Thanks,

Kirk...

[chipskind]

Kirk,

Thank you very much for your reply.

I'll look in to the issue by taking a look at those logs and see if I can figure out anything.

Also, open a TAC as soon as I get a chance.

For now, I'm just going to run those two blades/esxi hosts without the ioFusion cards.

Once again, thanks.

Chris