Not seeing 40g speed between hosts connected on same 9K

[support1@lima.co.uk]

Hi all,

I have several ESX hosts connected to the same 9K switch. They are connected via 40g uplinks but when running iPerf tests I am only seeing a max speeds of 6/8gbps.

Mgmt Vmk0 192.168.82.20 to vmk0 192.168.82.27 = 6.34 Gbits/sec

VSAN Vmk2 192.168.85.74 to vmk2 192.168.85.78 = 8.34 Gbits/sec

We would expect to see up to 40g here as the traffic should stay on the 9K device.. any advice on why this is?

[balaji.bandi]

what kind of card on ESXI side ? try change jumbo Frames. (9K MTU)

[support1@lima.co.uk]

Hi, I will check with the infra team but I expect cards that allow 40gbps speeds.

How do I change to jumbo frames? Is there any service impact from this type of change?

[balaji.bandi]

You can configure per interface

https://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/112080-config-mtu-nexus.html

[Elliot Dierksen]

FWIW, I have 40G interfaces in two FreeNAS hosts and two ESXi hosts. I have to turn off the ESXi firewall to be able to run iperf tests on the ESXi hosts. I think there is something inherent or configured in ESXi that limits the throughput of iperf. I am able to get 39.3G throughput between the FreeNAS hosts, but less when using the ESXi hosts. All four systems are C240 M4's. The one difference is the FreeNAS hosts have Chelsio T580's where the ESXi hosts have Cisco 1387's as the 40 NIC. I highly doubt there is something amiss with the 1387 NIC's, but I haven't bothered trying to load a different OS on my ESXi hosts to test that. I am using jumbo frames on the 40G network which I only use for storage.