Limited traffic throughput within VMM domain

[Nik Noltenius]

Ladies and gentlemen,

just to put the question in perspective: I'm not new to ACI but I'm still delving into new aspects almost on a daily basis.

I have a demo setup with two leafs and a single spine. Two ESXi hosts running a couple of VMs (e.g. the traffic generators I'm going to talk about) are connected to the leafs, one to each.  I recently tried to implement QoS and stumbled upon something completely out of focus. To actually see in bright colours that my QoS settings really work I first setup some test traffic. It struck me right away that I could only get about 2 Gbit/s of traffic across the fabric which is quite strange given the fact that the hosts are UCS C-Series servers with 10 GE interfaces. The following hours I conducted a couple of tests:

- Both traffic generators directly connected back to back (i.e. cable between the UCS) -> approx. 9.2 Gbit/s of goodput -> No issue with the servers
- Traffic generators in separate EPGs both connected to ACI via a VMM domain -> approx. 2 Gbit/s of goodput

- Traffic generators in separate EPGs on the same ESXi host both connected to ACI via a VMM domain (no spine involved, just routing at the leaf) -> approx. 2 Gbit/s of goodput
- Traffic generators in the same EPG connected to ACI via VMM domain -> approx. 2 Gbit/s of goodput
- Traffic generators in the same EPG connected via physical domain and static path bindings -> approx. 9.2 Gbit/s of goodput

Now I got the feeling that there could be something wrong in my configuration of the VMM domain. Is there a default setting for traffic shaping I might have missed or do you have any idea why traffic might be magically limited to 2 Gbit/s? I also suspected the DVS at one point but eventually that's created by the VMM configuration as well so it gets back to that.

Any help would be much appreciated.

Regards,
Nik

[Robert Burns]

Hey Nik,

We're happy to help but need more info.

-Version of Fabric

-Models of Switches (Leafs)

-Version of ESX

-OS of guest VMs

-Traffic Generation method (ie. iperf, wget etc).

-I assume this is a VMware vDS VMM? (not AVS)

I'd like to recreate your test as closely as possible in my lab.

Cheers,

Robert

[Nik Noltenius]

Thanks for the swift reply Robert,

- The fabric is running 2.0(1p)

- Leaf switches are N9K-C93180YC-EX, Spine is a N9K-C9336PQ

- ESX version is 6.0.0

- Guest OS is Ubuntu 14.04.3 LTS

- For traffic generation I use IxChariot from IXIA

- Yes, it's a VMware VMM

If you need anything else just let me know. I'd be happy to help you helping me ;)

[Robert Burns · Accepted solution]

What is the MTU configured on your VM's adapters?

FYI - I ran traffic tests on Windows VMs using iperf on both vSwitch (physical static path) and vDS & AVS VMM domains and didn't notice any significant different between results.  This was using the default MTU of 1500 on the guest adapters.  When I bumped the MTU to 9000, I saw a huge improvement on all accounts.

Robert

[Nik Noltenius]

Hi Robert,

thanks for the effort.

I had - and still have - an MTU of 1500. I did a test with iperf and everything was fine. I got back to IxChariot and now everything there is fine as well...

The only thing that changed in between was that I had to use my servers in a customer setup. So they rebooted a couple of times and now less VMs are running on them.

Unfortunately that does not really explain the issues I had but I think it's safe to say that ACI played no part in them.

Regards,

Nik

[Robert Burns]

Dang, I hate red herrings.  Well keep us posted if you notice anything strange next time!

Cheers,

Robert