Cisco UCS Slow upload speed

[dganta]

Hi Team,

Customer is Using Cisco 6332FI with 5018 chassis and B200M4 blades with the UCSM and the FI using 3.1.1h and the blades using the software 3.1.1.e. Customer has recently done a test using iperf where he has observed that when he is downloading he is getting a speep around 90Mbps where as the upload speed is only 20Mbps. He is using ESXi in these blades. I suggested them to verify whether the NIC teaming is configured for Route based on original virtual port but need to see whether that resolves the issue. However did anyone face this kind of issue what could be the possible solution. Customer is using another UCS chassis with ESXI and for the vlans configured int he 2nd UCS chassis he is getting the same download and upload speec of 90MBps. Also will the performance be impacted if it is a standard switch configured for the Active-active NIC teaming. Please clarify

[Walter Dey]

Which load balancing algorithm is used ?

Route based on originating virtual port ?

be aware that

"Route based on IP hash” does not work with Cisco UCS-B Fabric Interconnects cannot do VPC

[dganta]

Hi Walter,

See the same problem with Route based on Original Virtual port as well, This is a Chassis with VDI we see that between the Chassis and storage which is Flexpod the speed is good however if there is an external traffic trying to reach the UCS chassis, then we see this issue.

[reregala]

Hello,

If ESXi NIC team load balancing was set for 'Route Based on IP Hash', you would see consistent ping drops. This is not supported on any blade/rack server behind Fabric Interconnect.

When running iperf test, we need to verify the end-to-end path of the test. It would be beneficial to first test UCS only without involving any upstream switch and/or gateway.

The troubleshooting methodology should start at layer 2 only involving least amount of devices. Gradually add more path complexity as you verify performance is good.

Test # 1: Two VMs residing on same host, same VLAN.

Test # 2: Two VMs, different host, same VLAN, pinned to same FI.

To verify that the two VMs are pinned to the same FI, grab the MAC address of both. Go to UCS CLI:

# connect nx-os a
# show mac address-table | inc xxxx.xxxx.xxxx

# connect nx-os b
# show mac address-table | inc xxxx.xxxx.xxxx

If you see both each MAC address being learned only on one FI, these two VMs are pinned to the same FI.
The end to end path of this test is Server -> FI -> Server. The traffic will not proceed past FI.

Test # 3: Two VMs, different host, same VLAN, pinned to different FI.

Same mac address verification as Test # 2, however you want to verify they dont learn on same FI.
The end to end path of this test is Server -> FI -> uplinked switch -> peer switch -> FI -> Server.

Let us know what kind of performance you find at layer 2 only.

[Kirk J]

In addition to the other mentioned tests to isolate the traffic, you may want to make sure you aren't hitting some kind of L1 issues.

On the FI's nxos run:

#show int count error  (see if we have any FCS/CRC errors)

If you see any errors on the ports connecting to the IOMs, I'll provide some IOM commands to check those.

On some of the blades in question run the following from the UCSM CLI:

#connect adapter x/y/1  (where x is chassis#, y is blade#)

#connect

#dcem-macstats 0
#dcem-macstats 1
#dcem-macstats 2
#dcem-macstats 3

Look for any counters related to CRCs or drops.

The macstats  might help us see some adapter level errors that might not show up on FI port interface counters.

Additionally, is this setup trying to use Jumbo frames?

If you take one of the NICs out of the vswitches active-active config, and move it to unused/unassigned, does the performance change any?

Thanks,

Kirk...

[dganta]

Hi Kirk,

Thank you for the response.

Yes we are using jumbo frames for all the adapters. Couldnt finf any interface counter errors also I was not able to execute on the adapter dcem-macstats 0 or dcem-macstats 1". Is there an issue with using Jumboframes for all the vnics. The strange thing is it is working well for 1 chassis and we see this problem where there is VDI deployed in the 2nd chassis. Also the difference is ESXi 6.0 is in use for the chassis that is working and ESXi5.5 is in use here

[dganta]

Was able to run dcem-macstats after doing attach-mcp couldnt find any errors though see that the pause frames are relatively a bit high in the range of 14000 in the chassis that is not working however not sure whether this can be of any concern. However see the same pause frames for one of the adapters in another chassis where we do not see this problem