Solved: New BE7M (M5) with ESXi 7.0 - Slow datastore upload speeds

kylebrogers · ‎11-06-2021

I'm migrating some VMs from older Cisco C-Series servers running ESXi 5.1 to a pair of new BE7000M (M5) servers running ESXi 7.0. Downloading the VMs to my PC went at normal, expected speeds. The PC is wired and I have verified that everything in this process is connected at 1 Gbps and full duplex. The issues is that when I try to upload the flat VMDK files to either of the new ESXi 7 hosts, the speed flucuates between 20-50 Mbps. For example, I'm currently uploading a 160 GB VMDK, it's been 10 hours and I'm only at 130 GB copied. There are currently no VMs running on these new servers, so there's not exactly a lot of competition for disk access.
Has anyone else run into an issue like this and is there a fix? ESXi version that shipped is 7.0.0 (build 15843807).

kylebrogers · ‎11-08-2021

UPDATE:
I no longer think this is an issue with the write speeds on the VM hosts. I was copying VMDK files to the two hosts simultaneously and getting 20-50 Mbps. Once those copy jobs were completed I did some more testing. If I started a VMDK copy job to only one or the other of the hosts, it copied at 300+ Mbps. As soon as I started copying a VMDK to the other host at the same time, my upload speed would immediately drop by around 90%.
The logical conclusion would be that my issues is that the two copy jobs are competing for disk time on the workstation I'm using to perform the copy job. But when I copied the VMDK files from their old 5.1 systems to the workstation I was able to copy both at the same time and do so at high speeds. Additionally, the workstation is on SSDs, so I'm not going to have the same issues I would have on spinning disks.
So for my next test I started a VMDK copy to one of the hosts' datastore. It was moving at 300+ Mbps. Then I started copying a 5 GB ISO to the second system. It started moving quickly as well and my upload speeds simply bumped up to 400+ Mbps to accomodate the second copy job. So clearly the host was able to handle 2 simlutaneous copy jobs to those same 2 VM host servers. I stopped the ISO upload and tried a VMDK file instead and speeds dropped immediately.

Next I thought that perhaps the ESXi 7 browser-based interface may have issues handling 2 copy jobs simultaneously (even if they are to two completely different hosts). So I started a VMDK copy job in one browser to the first host in Chrome. It was running at 300 Mbps. I kicked off a second VMDK copy job to the second host from Firefox. Immediately my workstation's upload speeds dropped by 90% or so again.

For my next set of migrations, I copied the VMDKs one at a time instead of at the same time and they copied over at appropriate speeds.

I'm not sure what the issue is but it looks like it's not an issues with the storage on the VM hosts. I'm going to keep troubleshooting to see if I can figure out what the actual issue is. Thank you to everyone who tried to help me troubleshoot it as a storage issue.

View solution in original post

Kasun Bandara · ‎11-07-2021

Hi,

uploading is slower than the downloading due to several factors such as, disk write speed, raid level configured, link bandwidth utilization.

Please rate this and mark as solution/answer, if this resolved your issue
Good luck
KB

kylebrogers · ‎11-07-2021

Writing will always be slower than reading, but 12.5 hours to copy 160GB to the data store is way beyond excessive. That's a rate of 28 Mbps (3.5 MBps). There's definitely something wrong with that. If you had a 2 TB VM it would take you just under a week to copy it to the data store.

I've also verified in CIMC that the Write Policy is already set to "WriteBack with Good BBU"

Kirk J · ‎11-07-2021

I guess that begs the questions of:

Is there actually a BBU/Supercap present?
Is the Supercap Healthy, and passed various rounds of 'learning' cycles?
The write through/writeback policy is per VD. The WB/ w good BBU is set on all VDs ?

Also how many disks, what raid config, etc

Kirk...

Steven Tardy · ‎11-08-2021

Check to see if there are discards when vNIC hands off to the OS using ESXi command:

/usr/lib/vmware/vm-support/bin/nicinfo.sh | egrep "^NIC:|rx_no_buf"

If these counters are incrementing while testing then you can increase the vNIC buffers which the OS/driver allocates.

See: https://www.cisco.com/c/dam/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/vic-tuning-wp.pdf

kylebrogers · ‎11-08-2021

UPDATE:
I no longer think this is an issue with the write speeds on the VM hosts. I was copying VMDK files to the two hosts simultaneously and getting 20-50 Mbps. Once those copy jobs were completed I did some more testing. If I started a VMDK copy job to only one or the other of the hosts, it copied at 300+ Mbps. As soon as I started copying a VMDK to the other host at the same time, my upload speed would immediately drop by around 90%.
The logical conclusion would be that my issues is that the two copy jobs are competing for disk time on the workstation I'm using to perform the copy job. But when I copied the VMDK files from their old 5.1 systems to the workstation I was able to copy both at the same time and do so at high speeds. Additionally, the workstation is on SSDs, so I'm not going to have the same issues I would have on spinning disks.
So for my next test I started a VMDK copy to one of the hosts' datastore. It was moving at 300+ Mbps. Then I started copying a 5 GB ISO to the second system. It started moving quickly as well and my upload speeds simply bumped up to 400+ Mbps to accomodate the second copy job. So clearly the host was able to handle 2 simlutaneous copy jobs to those same 2 VM host servers. I stopped the ISO upload and tried a VMDK file instead and speeds dropped immediately.

Next I thought that perhaps the ESXi 7 browser-based interface may have issues handling 2 copy jobs simultaneously (even if they are to two completely different hosts). So I started a VMDK copy job in one browser to the first host in Chrome. It was running at 300 Mbps. I kicked off a second VMDK copy job to the second host from Firefox. Immediately my workstation's upload speeds dropped by 90% or so again.

For my next set of migrations, I copied the VMDKs one at a time instead of at the same time and they copied over at appropriate speeds.

I'm not sure what the issue is but it looks like it's not an issues with the storage on the VM hosts. I'm going to keep troubleshooting to see if I can figure out what the actual issue is. Thank you to everyone who tried to help me troubleshoot it as a storage issue.