Upload speed problem on Catalyst 3750

Marian Hercek · ‎02-18-2022

A client is connected to gigabit port on Catalyst 3750E (15.0(2)SE10a).
Catalyst 3750 is uplinked via tengigabit port to Catalyst 6500 (WS-X6704-10GE, 15.1(2)SY15).
Edge router is other vendor than Cisco and is configured to shape client's IP address down to 50 Mbit/s upload and download speed.

The problem is that with QoS configured on 3750, client cannot get more that 30-35 Mbit/s on upload.
Download speed is shaped perfectly arround 50 Mbit/s.
If I unconfigure QoS on 3750 port, upload speed is shaped perfectly around 50 Mbit/s.
By unconfigure QoS on interface I mean applying no mls qos cos 5.

Client's interface on 3750E (speed is auto-negotiated to 1000 Mbit/s full-duplex):
interface GigabitEthernet1/0/3
...

mls qos cos 5
mls qos cos override

...

3750E QoS:
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 70 30
mls qos srr-queue input threshold 1 80 90
mls qos srr-queue input priority-queue 2 bandwidth 30
mls qos srr-queue input cos-map queue 1 threshold 2 3
mls qos srr-queue input cos-map queue 1 threshold 3 6 7
mls qos srr-queue input cos-map queue 2 threshold 1 4
mls qos srr-queue input dscp-map queue 1 threshold 2 24
mls qos srr-queue input dscp-map queue 1 threshold 3 48 56
mls qos srr-queue input dscp-map queue 2 threshold 3 32 40 46
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 40 46
mls qos srr-queue output dscp-map queue 2 threshold 1 16 18 20 22 26 28 30 34
mls qos srr-queue output dscp-map queue 2 threshold 1 36 38
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 56
mls qos srr-queue output dscp-map queue 3 threshold 3 0
mls qos srr-queue output dscp-map queue 4 threshold 1 8
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 100 100
mls qos queue-set output 1 threshold 2 80 90 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 100 100 400
mls qos queue-set output 1 buffers 15 30 35 20
mls qos

6500's QoS on port:
int TenGigabitEthernet12/2

wrr-queue bandwidth 5 25 15 15 5 5 15
wrr-queue queue-limit 5 25 15 15 5 5 15
wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 2 80 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 3 80 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 4 80 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 5 80 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 6 80 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100
wrr-queue random-detect 4
wrr-queue random-detect 5
wrr-queue random-detect 6
wrr-queue random-detect 7
wrr-queue cos-map 1 1 1
wrr-queue cos-map 2 1 0
wrr-queue cos-map 3 1 2
wrr-queue cos-map 4 1 3
wrr-queue cos-map 5 1 6
wrr-queue cos-map 6 1 7
wrr-queue cos-map 7 1 4
mls qos trust cos

I've tried to change values which refers to bandwidth but it didn't improve the upload speed.

It seems that CoS 5 and its mapping to other QoS settings is the problem but I am unable to figure out which parameter is the one responsible for upload speed.

Flavio Miranda · ‎02-18-2022

Hi

If you have shape already on the Edge, why are you applying QoS on the switch? 3750 is a very old device with such a complicated QoS map I´m assuming that it may be out of resource already. To much to analise in every packet.

Marian Hercek · ‎02-18-2022

If you have shape already on the Edge, why are you applying QoS on the switch?
Because of traffic prioritization.

I´m assuming that it may be out of resource already
No, it's not.

Georg Pauwen · ‎02-18-2022

Hello,

what traffic on that port do you need to pritoritize ? Voice traffic ?

Joseph W. Doherty · ‎02-18-2022

You didn't post your 10g uplink interface config. So, don't know what queue-set is being used (probably 1, but often QS 2 is recommended for uplink ports), nor know whether interface PQ is active.

I don't mean to sound harsh, but you have a lot of non-default MLS QoS, so unclear why it's unknown what's happening. Did the person who provided those statements no longer available?

BTW, ASIC drop stats would be likely show show drops for CoS 5, and what queue and threshold they are bumping against.

Marian Hercek · ‎02-19-2022

You didn't post your 10g uplink interface config

Yes, I did. Check 6500's QoS on port.
TenGigabit interface on 3750 has only "mls qos trust cos" configured.

Joseph W. Doherty · ‎02-20-2022

"Yes, I did. Check 6500's QoS on port."

Ah, sorry I was unclear. Thought as you didn't post 3750's interface config, you would have known it wasn't the 6500's port I was asking about.

"Uplink", is 3750, or do you mean both the 6500's and 3750's ports are uplinks, i.e. they are not opposite ends of same link?

Again, what's the story on whoever setup these QoS settings? What are you trying to accomplish (as "Because of traffic prioritization.", to @Flavio Miranda's question doesn't help much). (Understand, it's very easy to make things "worse" with QoS, so "knowing" your QoS goals isn't just idle curiosity.)

BTW, for best help, generally its best to try to provide all the information asked for.

Also BTW, I'm sure it's unintended, but your responses, not to just me, seem a bit argumentative.

I wish you the best of luck resolving your issue.

Marian Hercek · ‎02-23-2022

I did some tests and it turned out that the problem might be tenGig interface at 6500.
I configured exact QoS settings on 3750E which is not uplinked into 6500 and upload was 50 Mb as it should be.
Also if 3750 interface is configured with CoS 3, upload is 50 Mb as it should be.

Test steps:
1. all statistics was cleared at 3750 and 6500
2. speedtest
3. gathered statistics

3750 (gi1/0/3 - tested port, te1/0/2 - uplink to 6500)

sh mls qos interface gi1/0/3 statistics
GigabitEthernet1/0/3 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 99686 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 0 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 279965 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 2 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 0 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 0 0 0 0 0
5 - 7 : 101263 0 0
cos: outgoing
-------------------------------

0 - 4 : 290583 0 0 0 0
5 - 7 : 0 2 8
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 68 31
queue 2: 0 0 290582
queue 3: 0 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

#sh int gi1/0/3
GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is b414.89ce.ba83 (bia b414.89ce.ba83)
Description: TEST
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 7/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:02, output hang never
Last clearing of "show interface" counters 00:04:32
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1731000 bits/sec, 602 packets/sec
5 minute output rate 30596000 bits/sec, 2605 packets/sec
190654 packets input, 73008451 bytes, 0 no buffer
Received 75 broadcasts (69 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 69 multicast, 0 pause input
0 input packets with dribble condition detected
825870 packets output, 1214632612 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

#sh mls qos interface tenGigabitEthernet 1/0/2 statistics
TenGigabitEthernet1/0/2 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 645554 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 0 0 90 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 109 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 21465 0 0 0 0
5 - 9 : 0 0 0 0 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 0
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 151490 0 136 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 5 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 647201 0 0 0 0
5 - 7 : 0 1159 1413
cos: outgoing
-------------------------------

0 - 4 : 22222 0 0 0 0
5 - 7 : 151515 149 68
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 154432
queue 1: 477 564 223
queue 2: 0 0 21522
queue 3: 0 0 0

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 0
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

 sh int te1/0/2
TenGigabitEthernet1/0/2 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is b414.89ce.bab6 (bia b414.89ce.bab6)
Description: UPLINK_TO_6500
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is 10GBase-LR
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:03:17
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 32274000 bits/sec, 2832 packets/sec
5 minute output rate 2012000 bits/sec, 728 packets/sec
664828 packets input, 947369718 bytes, 0 no buffer
Received 4176 broadcasts (2235 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2235 multicast, 0 pause input
0 input packets with dribble condition detected
183283 packets output, 76755040 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

6500 (te12/2 downlink port 3750)

#sh queueing interface tenGigabitEthernet 12/2
Interface TenGigabitEthernet12/2 queueing strategy: Weighted Round-Robin

Port QoS is enabled globally
Queueing on Te12/2: Tx Enabled Rx Enabled

Trust boundary disabled

Trust state: trust COS
Extend trust state: not trusted [COS = 0]
Default COS is 0
Queueing Mode In Tx direction: mode-cos
Transmit queues [type = 1p7q8t]:
Queue Id Scheduling Num of thresholds
-----------------------------------------
01 WRR 08
02 WRR 08
03 WRR 08
04 WRR 08
05 WRR 08
06 WRR 08
07 WRR 08
08 Priority 01

WRR bandwidth ratios: 5[queue 1] 25[queue 2] 15[queue 3] 15[queue 4] 5[queue 5] 5[queue 6] 15[queue 7] 
queue-limit ratios: 5[queue 1] 25[queue 2] 15[queue 3] 15[queue 4] 5[queue 5] 5[queue 6] 15[queue 7] 15[Pri Queue]

queue tail-drop-thresholds
--------------------------
1 70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
2 70[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
3 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
4 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
5 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
6 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
7 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]

queue random-detect-min-thresholds
----------------------------------
1 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
2 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
3 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
4 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
5 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
6 80[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
7 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]

queue random-detect-max-thresholds

1 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
2 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
3 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
4 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
5 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
6 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8] 
7 100[1] 100[2] 100[3] 100[4] 100[5] 100[6] 100[7] 100[8]

WRED disabled queues:

queue thresh cos-map
---------------------------------------
1 1 1 
1 2 
1 3 
1 4 
1 5 
1 6 
1 7 
1 8 
2 1 0 
2 2 
2 3 
2 4 
2 5 
2 6 
2 7 
2 8 
3 1 2 
3 2 
3 3 
3 4 
3 5 
3 6 
3 7 
3 8 
4 1 3 
4 2 
4 3 
4 4 
4 5 
4 6 
4 7 
4 8 
5 1 6 
5 2 
5 3 
5 4 
5 5 
5 6 
5 7 
5 8 
6 1 7 
6 2 
6 3 
6 4 
6 5 
6 6 
6 7 
6 8 
7 1 4 
7 2 
7 3 
7 4 
7 5 
7 6 
7 7 
7 8 
8 1 5

Queueing Mode In Rx direction: mode-cos
Receive queues [type = 1q8t]:
Queue Id Scheduling Num of thresholds
-----------------------------------------
01 WRR 08

WRR bandwidth ratios: 100[queue 1] 
queue-limit ratios: 100[queue 1]

queue tail-drop-thresholds
--------------------------
1 50[1] 50[2] 60[3] 60[4] 80[5] 80[6] 100[7] 100[8]

queue thresh cos-map
---------------------------------------
1 1 0 
1 2 
1 3 1 2 3 4 
1 4 
1 5 6 7 
1 6 
1 7 5 
1 8


Packets dropped on Transmit:
BPDU packets: 0

queue dropped [cos-map]
------------------------------------------------------------

1 0 [1 ]
2 0 [0 ]
3 0 [2 ]
4 0 [3 ]
5 0 [6 ]
6 0 [7 ]
7 0 [4 ]
8 0 [5 ]

Packets dropped on Receive:
BPDU packets: 0

queue dropped [cos-map]
------------------------------------------------------------
1 0 [0 1 2 3 4 6 7 5 ]

#sh int te12/2

TenGigabitEthernet12/2 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is c89c.1dc1.00b9 (bia c89c.1dc1.00b9)
Description: 
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec, 
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-LR
input flow-control is on, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 22:07:59
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 144000 bits/sec, 77 packets/sec
5 minute output rate 632000 bits/sec, 107 packets/sec
6496083 packets input, 1492242753 bytes, 0 no buffer
Received 235546 broadcasts (92860 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
12874030 packets output, 13366217395 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

divanko · ‎12-02-2022

Take a look at this document:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/116089-technote-switches-output-drops-qos-00.html

Check to see if you are seeing output drops due to QOS and adjust the config.

I know for us, our default queue is 3 and we had to make this adjustment:

mls qos queue-set output 1 threshold 3 100 100 100 3200

We had issues when clients were connected to 100mb interfaces and the uplinks were 1Gig .. so somewhat similar issue.

Hope that helps!