Re: 3750X 3 stack having issues with dropped packets

wsalomon75 · ‎07-21-2023

Hello all,

I have a 3 stack of Catalyst 3750Xs that are having issues communicating with a Catalyst 9300. The site is complaining of slowness issues, and after running diagnostics (ping drops 0 packets from 9300 to Core but the 3750X > 9300 drops a lot), I have determined it has to be an issue with the link between the 3750X stack and the 9300. I have done the following:

1. Swapped out SFP and cable on both switches

2. Swapped the port on both switches

3. Changed the global MTU on the 3750X to 1510 and restarted the stack.

Here's the output for a ping to the 9300:

X#ping 10.20.22.250 re 2000 si 1504
Type escape sequence to abort.
Sending 2000, 1504-byte ICMP Echos to 10.20.22.250, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!.!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!.
Success rate is 98 percent (1975/2000), round-trip min/avg/max = 1/6/67 ms

I did 1504 packet size because from what I understand, with encapsulation the packet will be 1504 with the extra 4 coming from the encapsulated frame. I don't know how else to troubleshoot this, except the stack is pretty full and maybe it's just too much traffic coming through a 1G uplink.

Any thoughts?

Flavio Miranda · ‎07-21-2023

Hi @wsalomon75

Check the CPU "show proc cpu history" if CPU is too high, it can be causing the ping fail.

wsalomon75 · ‎07-25-2023

333333333444443333344444333333333344444333333333333333333333
444488888111115555533333888884444400000888887777799999777779
100
90
80
70
60
50
40 ************************* ************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

599469995545544444455444445444444445454444554544455545554455
079806173094286879700648740996555283906256239061711091143972
100 ** * *
90 ** ***
80 ** ***
70 ** #**
60 *# *###
50 *##**###************** ** ******* ***** ******* ******** *
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

655555555555555565555555555555555556556997679999998996999799977969699999
057685536535556607678776867257647696650993669799990899989389912989799699
100 ** ****** ** *** *** * * ***
90 ** ****** ** *** *** * * ***
80 ** ********** *** *** * * ***
70 * *******************************
60 ******* ** **************** *** **************************************
50 ****************************************************#**************#**
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

Joseph W. Doherty · ‎07-21-2023

Many of the Catalyst 3K models are a bit infamous for interface egress drops due to lack of buffer resources and their default settings, especially if QoS is enabled. (Is QoS enabled on your 3750X?)

What you want to do next is check ASIC stats for the port on the 3750X connecting to the 9K.

If there are many port drops, and factory defaults are being used, some configuration tweaking can often drastically reduce egress port drops.

wsalomon75 · ‎07-25-2023

Yes, QoS is enabled on the 3750X. How can I check ASIC stats for the port?

Joseph W. Doherty · ‎07-25-2023

Possibly something like:

show platform port-asic stats drop g#/#/#

wsalomon75 · ‎07-25-2023

Interface Gi3/1/2 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 40
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 4
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 5
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 6
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 7
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0

Joseph W. Doherty · ‎07-25-2023

Ah, so interface drops don't appear to be a problem. (Those stats also agree with the interface drop count.)

So tells (again) why you changed 3750 global MTU to 1510 and are using a ping size of 1504?

Also, what are you pinging from and ping to?

wsalomon75 · ‎07-25-2023

Our current setup at this location is: Catalyst 3750X ---> Catalyst 9300 -------------------------------->Catalyst 9500 at Data Center. I was pinging between the 3750X and the 9300. The 3750X is a 3-stack and almost full. I'm assuming the tech before me added the 9300 in the event there was more room needed. The 9300 to the 9500 doesn't have any issues when I ping between them. No dropped packets. However, when pinging between the 3750X and the 9300, the above is what I see. I changed the MTU to 1510 because I read in another thread that the 3750X adds 4 packets for encapsulation with dot1q enabled (which it is). So I just made it 1510 to clear it and any other frames in the event I was missing something (and I'm testing). I'm pinging 1504 because that's what I assume is going to be the standard packet size.

The original issue this site is having are complaints about slow speeds, and currently their Cisco IP Phones are having issues. Mainly sometimes Person 1 can hear Person 2, but Person 2 can't hear Person 1. It's an issue that happens 80% of the time. So this lead me to think that VoIP packets are dropping, which could also result in the slowness issue as well.

Follow me so far?

Joseph W. Doherty · ‎07-25-2023

For .Q, you normally don't need to adjust MTU. You would then ping using 1500.

If you can, you don't want to ping to/from Cisco network devices as they give low priority to dealing with pings. Ideally, you just want to ping through the Cisco network device.

What I would double check, that you have end-to-end QoS configured to prioritize VoIP traffic. BTW, believe QoS is disabled, by default, on any 3750, and the device defaults, I also recall the device default don't prioritize VoIP (although AutoQoS does).

Your 3750-X has a fully functional dual ring?

wsalomon75 · ‎07-26-2023

I'm not sure what you mean by the 3750X having a fully functional dual ring. If you mean topology, then no. We do not employ dual ring. Here's the output on "show run | i qos"
sho run | i qos
mls qos map policed-dscp 0 10 18 to 8
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 70 30
mls qos srr-queue input threshold 1 80 90
mls qos srr-queue input priority-queue 2 bandwidth 30
mls qos srr-queue input cos-map queue 1 threshold 2 3
mls qos srr-queue input cos-map queue 1 threshold 3 6 7
mls qos srr-queue input cos-map queue 2 threshold 1 4
mls qos srr-queue input dscp-map queue 1 threshold 2 24
mls qos srr-queue input dscp-map queue 1 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue input dscp-map queue 1 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue input dscp-map queue 2 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue input dscp-map queue 2 threshold 3 46 47
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust device cisco-phone
mls qos trust cos
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust device cisco-phone
mls qos trust cos
auto qos voip cisco-phone
mls qos trust device cisco-phone
mls qos trust cos
auto qos voip cisco-phone
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust
mls qos trust cos
auto qos trust

Joseph W. Doherty · ‎07-26-2023

"I'm not sure what you mean by the 3750X having a fully functional dual ring."

You did write the you have a 3 (member) stack, correct?

When you interconnect 3750 with their special cables, you create a stack ring. Ideally, you create two stack rings, which provides both redundancy and twice the throughput capacity.

Unsure why I asked if your ring is set up as (an ideal) dual ring, as unlikely that would be a root problem of your issue.

Joseph W. Doherty · ‎07-26-2023

Hmm, surprise that command shows 8 queues as I recall the 3750 series on has 4 queues (which dovetails with your QoS config statements you recently posted).

Also try:

sh mls qos interface g3/1/2 stat

wsalomon75 · ‎07-26-2023

#sh mls qos interface g3/1/2 stat
GigabitEthernet3/1/2 (All statistics are in packets)

dscp: incoming
-------------------------------

0 - 4 : 12837744 83 1237533 0 13581
5 - 9 : 89 3701 0 0 0
10 - 14 : 6 0 0 0 2
15 - 19 : 0 2193692 0 0 0
20 - 24 : 0 0 0 2 176218
25 - 29 : 0 0 1 0 0
30 - 34 : 0 5 1232 0 0
35 - 39 : 0 0 4 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 1252466 0 545983 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 102321 0 0 0
60 - 64 : 0 0 0 0
dscp: outgoing
-------------------------------

0 - 4 : 20085688 0 0 0 0
5 - 9 : 0 0 0 214608 0
10 - 14 : 0 0 0 0 0
15 - 19 : 0 0 0 0 0
20 - 24 : 0 0 0 0 63771
25 - 29 : 0 0 0 0 0
30 - 34 : 0 0 0 0 0
35 - 39 : 0 0 0 0 0
40 - 44 : 0 0 0 0 0
45 - 49 : 0 6718 0 877605 0
50 - 54 : 0 0 0 0 0
55 - 59 : 0 89 0 0 0
60 - 64 : 0 0 0 0
cos: incoming
-------------------------------

0 - 4 : 71697920 15520 2217418 176208 266
5 - 7 : 3576 367659 0
cos: outgoing
-------------------------------

0 - 4 : 20664189 214608 0 63771 0
5 - 7 : 6718 877605 473992
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 6718
queue 1: 116501 1636388 1351597
queue 2: 0 0 18917842
queue 3: 210391 0 66731

output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 0
queue 1: 0 0 0
queue 2: 0 0 40
queue 3: 0 0 0

Policer: Inprofile: 0 OutofProfile: 0

wsalomon75 · ‎07-26-2023

Can you explain what I'm looking at in this output, Joseph? Also, thank you so much for your help!