cisco 2960, 3560, 3750 switch series bidirectional throughput problem

[Balazs Gyorgy]

Hi Cisco Magi,

I have a serious issue here which goes on for a long time and I couldn`t find a solution for it so far.

The problem is affecting at least the 2960, 3560 and3750 series switches. No issues if all the interfaces on the switches are on 1G speed. However if I pick two laptops and one of them is running on 100Mbit and the other one is on gigabit I can`t get a 100Mbit throughput in both dierections at the same time. The traffic in one direction is alright (about 93 Mbit/s) but the other direction is about 7-10Mbit.

As a lab test I tried to isolate the problem with two 2960 switches (IOS - c2960-lanbasek9-mz.122-55.SE6, models: WS-C2960-48TT-L and WS-C2960-24TT-L ). Test topology: Laptop 1 --> 100Mbit switchport on switch 1 --> switch 1 --> gigabit connection --> switch 2 --> gigabit switchport on switch 2 --> Laptop 2

The test shows the same results if I use only one switch, with a 100Mbit and a Gigabit port. Please note that it affects only TCP traffic (tested with iperf and FTP). Nevertheless, If I manually configure the Gigabit port to speed 100 the problem disappears, I have perfect results in both directions.

Tried with and without mls qos, same results. Tried to adjust interface buffers, hold queues; queue-set buffers and thresholds, no success.

Maybe it is just a trivial config change that I`m not aware of, but it is really annoying. Any help/suggestion is appreciated.

Thanks,

Balazs

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

When you have symmetric bandwidth, there should not be any congestion in the switch or switches.

When you have asymmetric bandwidth, there's likely to be congestion from fast to slow; and some packet drops.

Dropping ACK packets isn't the same as dropping "ordinary" data packets.  Your chances are higher, I would suspect, that TCP retransmission is based on timeouts.  These, I recall, besides taking longer to trigger, also I believe put the sender into repetitive slow start transmissions.

Please confirm which of the two asymmetric flows is slower than expected, the one sourced from the gig interface or the one sourced from the 100 Mbps interface.

Oh, if problem is dropped ACKs, unless one of your QoS tests actually treated ACK packets differently, all your combination tests generally might not make much difference.

[Balazs Gyorgy]

Hi Joseph,

OK, so when the traffic flows only in one direction, let`s say from the Gig towards the 100M, the results are fine, I can see a constant 100Mbit traffic (however there is a speed mismatch). It is true in the opposite direction as well. The issue appears in the case when the traffic flows in both direction at the same time.

And when it happens the wrong direction is always from the Gig towards the 100M.

I know there is a congestion when traffic flows between interfaces which have different line rates and packets will be dropped, but for instance I never had such an experience on a router with E1 and Fa/gig interfaces. TCP should handle this and in point of fact most of the time it does.

I can only repeat myself, the switch performs alright, when the traffic is unidirection, but traffic becomes asymetric when it flows in both directions simultaneously. And that is not an expected behaviour, it should be just fine without any fine tuning on Qos settings.

I found a similar bug in previous IOS versions, but according the BUG Toolkit it has been solved. BUG ID:

CSCsc96037

I work in an ISP environment so it kind a hard to say to a customer who bought a full duplex connection that sorry it meant to be full duplex, but it is not. That is the reason I can`t prioritize traffic either, it is just not a realistic situation when you start to apply Qos on TCP ACKs for a simple internet service.

I really appreciate your help so far, any ideas welcome. Referring to my original post this issue affects the mentioned switch series models at least, I had the very same experience previously on a lot of models (with different [and earlier] IOS softwares) coming from those series. It might affect other models form different series as well, I can`t tell that because my tools and my time here are limited to make further tests on other models simply becasue I have no other models to test with.

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Darn, from gig towards 100 Mbps; hmm then it's very curious.  From gig to 100 Mbps, I might expect it to be slower impacted by drops as it congests within the switch dropping from gig rate to 100 Mbps.  Unidirectional, you describe this as fine.

When you add a reverse 100 Mbps flow, the gig flow's ACKs will be interspersed with that flow, as the 100 Mbps flow's ACKs will be interspersed with the gig flow.  From a switch perspective, I can see the switch, again from gig to 100 Mbps, besides dropping some gig flow's packets, also dropping some of the 100 Mbps flow's ACK's, i.e. possible impact to the 100 Mbps flow, but you describe that's not the flow impacted.

The 100 side has 100 Mbps all the way through, i.e. switch shouldn't be an issue (should be zero drops and zero queuing delay).  That leaves the 100 Mbps source host or the gig receiving host somehow delaying or dropping the gig flow's ACKs.  I can easily see the 100 Mbps source host delaying the gig flow's ACKs, but enough to slow the gig flow as you describe?  This is interesting!

I do want to look at your packet capture, but trying to find the time to do it is a problem.  If I do get the time, and find anything, I'll post again.