1457 VIC throughput tweaking

[cmturner20]

Can anyone shed light on the burst bytes section on the qos policy? I seem to be bottlenecked at around ~10gig (I have 25gig nics)

https://www.boudewijnplomp.nl/2015/08/my-first-flexpod-part-4/

"The ‘Burst(Bytes)’ value essentially means the NIC speed. If you configure a value of 10240 and link the QoS Policy to a vNIC it will operate as a 10GbE NIC. If you configure it as 1024 it will operate as a 1GbE NIC. UCS even allows you to configure uncommon values like 2048 for 2GbE."

Is this correct? Should my burst bytes value be changed to 25000 for 25 gig VIC? Will it even let me or is this a waste of time?

[Steven Tardy]

How are you testing?
TCP or UDP?
Are you using `iperf`?
`iperf3` includes a "retr" (retransmission) counter to know if you are fighting packet loss or "something else".

I don't have 25Gbps setup in the lab but previously tested 40Gbps in my lab and `iperf` could achieve 39.9Gbps. Without ever changing "Burst(Bytes)".

If you are fighting packet loss, then find/correct that first.

Sometimes increasing RX queues (and/or ring size) prevents packet loss within the destination.

Sometimes increasing TX queues (and/or ring size) can "push" more packets onto the wire.

[cmturner20]

Using iperf with tcp, can try UDP I suppose. I didn't think I needed to but because your doco is very vague I thought I'd investigate this.

I've already increased RX/TX queues to "fight" rx_buffer loss but I'm still digging a bit deeper.

In your doco here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/cli/config/guide/3-0/b_UCSM_CLI_User_Guide_3_0/b_UCSM_CLI_User_Guide_3_0_chapter_010011.pdf

Specifies the expected average rate of traffic. Traffic that falls under
this rate will always conform. The default is line-rate, which equals
Switch-A
/org/qos-policy/egress-policy
# set rate {line-rate | kbps}
burst bytes
Step 6
a value of 10,000,000. The minimum value is 8, and the maximum
value is 40,000,000.

Again, does it need to be set at the"line-rate" of 25 gig or is line-rate equivalent of the known vic speed. Because right now, "line-rate" feels very 10,000,000. 

[Steven Tardy]

The public cisco.com UCSM docs seem to have stripped out the section you are asking about in UCSM 3.1 so you could only find the old UCSM 3.0 docs with it.

The UCSM GUI help docs do include additional details about "Rate field":

The expected average rate of traffic. Traffic that falls under this rate will always conform. 
The default is line-rate, which equals a value of 10,000,000. 
The minimum value is 8, and the maximum value is 40,000,000. 

I found a lab setup with VIC 1475, but was only able to get 9.2 Gbps on Linux VMs on VMware.

Tried tweaking a few settings which typically allow more throughput (larger TCP window, more threads, pin to same FI, more TX ring size), but was unable.

Double checked the SFPs and these were connected with 10Gbps SFPs.

Well. This specific lab setup isn't going to help. I'll keep digging to try to find a 25Gbps end-to-end lab setup.

[cmturner20]

Great thanks Steven. Appreciate the help thus far.