unequal bandwidth distribution for flows

[S891]

hi,

I was doing some flow analysis on our campus LAN and DC network after a user reported problem of slowness on transferring a large file. One time it was fast and next time it was very slow, and this behavior was consistent. 

I used two machines and ran jperf b/w DC  and  LAN network to analyze the problem. I noticed that on jperf single stream was working fine , but when I had multiple streams the distribution was not equal. So for example if I have 3 flows running b/w source and destination, one flow was at 700 mbps and other two flows were around 100 mbps each. I did this test many times and very often it would do the same. Sometimes on even number of flows (2 or 4 flows) it divided equally but odd number of flows (3 or 5 flows) mostly had this issue of unequal load distribution. 

We have N5K in DC and 6500s in core and access. The load balancing methods on Port-channels b/w the switches is default. 

Although the initial user problem was not there in jperf flows but it gave some hints towards flow distribution issues. 

Any suggestions/ thoughts?

[Joseph W. Doherty]

I'm not familiar with load-balancing options on the N5K, but other platforms often offer several load-balancing options.  Sometimes the default load-balancing option is not optimal for your traffic and needs to be changed.

Even with the "best" load-balancing option enabled, there are going to be cases that load-balancing is far from optimal, especially as none, of which I'm aware, dynamically monitor link usage.

Something like PfR (likely not available on your L3 switches) has the capability to dynamically load balance routed flows, but even it will not, by default, split a flow across multiple paths.

Often CEF enabled devices can do packet-by-packet load balancing, across L3 links, but although the load balancing is great, splitting a flow's packets often causes issues due to out-of-sequence delivery.

The only other load balancing option would be to use some kind of hardware mux that can split and recombine packets on both sides of the multiple links.