05-09-2024 03:28 PM
Hi All
I've inherited a newtwork that includes the C3850 and we've installed the C2960X. These are hooked up via copper ethernet and have a very simple configuration
C3850
interface GigabitEthernet2/0/2
description Uplink to C2960X
switchport mode trunk
end
C2960X
interface GigabitEthernet1/0/22
description Uplink to C3850
switchport mode trunk
end
On the link between C3850 and C2960 I am seeing a lot of dropped packets (Only on the C3850, nothing on the C2960) and when digging deeper, that there are a number of "Excess Defer Frames" on the C3850.
The previous team had a bunch of QOS which we tried to remove as the first step, but I suspect it exists for a reason so will look at putting it back in play.
It used to be, but was removed for diagnosis:
interface GigabitEthernet2/0/2
description Uplink to C2960X
switchport access vlan 77
switchport mode trunk
auto qos trust
spanning-tree portfast
service-policy input AutoQos-4.0-Trust-Cos-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
end
The QOS settings are:
class-map match-any AutoQos-4.0-Output-Multimedia-Conf-Queue
match dscp af41 af42 af43
match cos 4
class-map match-any AutoQos-4.0-Output-Bulk-Data-Queue
match dscp af11 af12 af13
match cos 1
class-map match-any AutoQos-4.0-Output-Priority-Queue
match dscp cs4 cs5 ef
match cos 5
class-map match-any AutoQos-4.0-Output-Multimedia-Strm-Queue
match dscp af31 af32 af33
class-map match-any non-client-nrt-class
class-map match-any AutoQos-4.0-Output-Trans-Data-Queue
match dscp af21 af22 af23
match cos 2
class-map match-any AutoQos-4.0-Output-Scavenger-Queue
match dscp cs1
class-map match-any AutoQos-4.0-Output-Control-Mgmt-Queue
match dscp cs2 cs3 cs6 cs7
match cos 3
!
policy-map port_child_policy
class non-client-nrt-class
bandwidth remaining ratio 10
policy-map AutoQos-4.0-Output-Policy
class AutoQos-4.0-Output-Priority-Queue
priority level 1 percent 30
class AutoQos-4.0-Output-Control-Mgmt-Queue
bandwidth remaining percent 10
queue-limit dscp cs2 percent 80
queue-limit dscp cs3 percent 90
queue-limit dscp cs6 percent 100
queue-limit dscp cs7 percent 100
queue-buffers ratio 10
class AutoQos-4.0-Output-Multimedia-Conf-Queue
bandwidth remaining percent 10
queue-buffers ratio 10
class AutoQos-4.0-Output-Trans-Data-Queue
bandwidth remaining percent 10
queue-buffers ratio 10
class AutoQos-4.0-Output-Bulk-Data-Queue
bandwidth remaining percent 4
queue-buffers ratio 10
class AutoQos-4.0-Output-Scavenger-Queue
bandwidth remaining percent 1
queue-buffers ratio 10
class AutoQos-4.0-Output-Multimedia-Strm-Queue
bandwidth remaining percent 10
queue-buffers ratio 10
class class-default
bandwidth remaining percent 25
queue-buffers ratio 25
policy-map AutoQos-4.0-Trust-Cos-Input-Policy
class class-default
set cos cos table AutoQos-4.0-Trust-Cos-Table
Port status on the C3850
C3850#sh int gi2/0/2 controller
GigabitEthernet2/0/2 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 7486.0b1a.1602 (bia 7486.0b1a.1602)
Description: Uplink to C2960
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 5/255, rxload 4/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:11, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1225557483
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 18542000 bits/sec, 6715 packets/sec
5 minute output rate 23283000 bits/sec, 6995 packets/sec
42573780984 packets input, 22167505633187 bytes, 0 no buffer
Received 791457303 broadcasts (604480626 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 604480626 multicast, 0 pause input
0 input packets with dribble condition detected
45485525989 packets output, 30042550904462 bytes, 0 underruns
1225557483 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet2/0/2 Receive
30042550904462 Total bytes 22167505633187 Total bytes
45033860137 Unicast frames 41782323681 Unicast frames
29990443597399 Unicast bytes 22073848041769 Unicast bytes
320709216 Multicast frames 604480626 Multicast frames
41076870184 Multicast bytes 79508648744 Multicast bytes
130956636 Broadcast frames 186976677 Broadcast frames
11030436879 Broadcast bytes 14148942674 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
1587370217 Minimum size frames 2414907208 Minimum size frames
17062787834 65 to 127 byte frames 19184594389 65 to 127 byte frames
4602431517 128 to 255 byte frames 3999123543 128 to 255 byte frames
3063691377 256 to 511 byte frames 3114913210 256 to 511 byte frames
2241533398 512 to 1023 byte frames 2228186107 512 to 1023 byte frames
6983685553 1024 to 1518 byte frames 4821943045 1024 to 1518 byte frames
9944026093 1519 to 2047 byte frames 6810113482 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
31290328555 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
Reading through this, there is a rabbit hole of information and I'm honestly not sure where to start. (I do know the 3850 is old and probably should also be replaced)
Thanks
S
05-09-2024 10:44 PM
what IOS code running on both the devices.
1. have you tried replacing the physical cable and check?
2. have you tried different port the one having issue may be ?
3. when you clear the counters on the interface how soon the drops grow ?
4. Hope there is no mis match of speed on both the sides ?
5. Try simple configuration removing all the QoS on the trunk interface.
check below guide to troubleshoot :
05-11-2024 02:38 AM
Thanks
Replaced the cable and removed the QOS information and still seeing the output drops. We have a monitoring tool that is showing the drops per min and we are also seeing issues in the real world with things dropping out.
Changed the port on the switch today and will monitor.
05-12-2024 12:25 AM
Changed the port on the switch today and will monitor.
give us the feedback how it go after moving to different port.
Replaced the cable and removed the QOS information and still seeing the output drops. We have a monitoring tool that is showing the drops per min and we are also seeing issues in the real world with things dropping out.
how soon the drops increases after clearing the counters ?
do you see any other ports also increasing the drops ?
05-10-2024 05:33 AM
The Catalyst 3Ks (2Ks too) are a bit infamous for dropping bursty traffic, both due to default settings and lack of buffer RAM. Insufficient information to say if such is your issue, but such isn't uncommon.
The extensive QoS is from auto QoS, but it can be modified. Insufficient information to, again, say that's your case too.
If not already in your config, you might try "qos queue-softmax-multiplier 1200".
05-11-2024 02:42 AM
Thanks. I'll give the cable change a 24hour window to see what it does, but if it still is causing issues will try that next. We also moved the primary device we're having issues with upstream to another switch to minimise the impact.
05-11-2024 02:57 AM
Was still having issues, Tried the command and it's dropped all my connections to the switch
I think it's time to plan for a replacement..
05-11-2024 03:11 AM
qos queue-softmax-multiplier 400 <<- apply this command only to SW face output drop not both SW
MHM
05-11-2024 03:49 AM
That command wouldn't guarantee no drops, or even guarantee a reduction, much depends whether drops are due to transient or substained congestion. It works best for the former. The latter usually requires QoS to better manage congestion and/or more bandwidth.
BTW, was the any reduction in the rate of drops with that change?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide