02-06-2018 07:20 PM - edited 03-08-2019 01:44 PM
Hi all, I'm experiencing an issue with a WS-C3850-24S switch where a small percentage of packets are dropping from the output queues in about half of the trunks. I've done some research and found that it can be caused by an ingress interface that offers much more bandwidth than the egress interface. I'm not entirely sure this is what's causing it to be honest.
Ingress on this switch is a 2 x 10Gb EtherChannel trunk. Firmware version is 03.06.06E .
I've also tried to use some of the fixes described in these forums in regards to changing the queue parameters but have been unsuccessful in making those changes. I would really appreciate it if someone could help me out!
Trunk Interface stats:
reliability 255/255, txload 4/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseBX-10U SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:18, output never, output hang never
Last clearing of "show interface" counters 3w5d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 959239
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 6068000 bits/sec, 2450 packets/sec
30 second output rate 18567000 bits/sec, 1655 packets/sec
147799652 packets input, 78574317037 bytes, 0 no buffer
Received 405149 broadcasts (136421 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 136421 multicast, 0 pause input
0 input packets with dribble condition detected
237563053 packets output, 252406070491 bytes, 0 underruns
959239 output errors, 0 collisions, 0 interface resets
QoS Queue stats:
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
0 0 0 0 1192754861
1 0 0 0 664465155
2 0 0 0 0
3 0 0 0 0
4 0 0 0 0
5 0 0 0 0
6 0 0 0 0
7 0 0 0 0
DATA Port:21 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 0 0 0 0
1 0 0 3619155 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0
AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
PORT TYPE ENQUEUE DROP
--------------------------------------------------
UPLINK PORT-0 N/A 0
UPLINK PORT-1 N/A 0
UPLINK PORT-2 N/A 0
UPLINK PORT-3 N/A 0
NETWORK PORTS 21024980 140441674
RCP PORTS 0 0
CPU PORT 0 0
Note: Queuing stats are in bytes
Solved! Go to Solution.
02-15-2018 07:56 AM
I have seen this at a couple of our customers after they upgrade to these new switches. If you do a wireshark capture if you see 2 or more ARP packets right in a row it could be this bug.
3850 duplicates pass-through ARP packets
Bug ID: CSCur30273
Description
Symptom:
When ARP request/Reply packets enter an access or trunk interface and its L2 switched to an another access or trunk interface in same VLAN the ARP request/Reply packets gets duplicated. This means that when one ARP packet enter the switch two identical ARP packets exit the switch. We have seen IP Packets getting duplicated. This only seems to affect ARP packets.
Conditions:
L2 switching with lanbase license
Workaround:
The issue is only observed with "lanbase" license. Issue is not seen with "ipbase" license.
Enable IPDT
3850-STACK#sh ip device tracking all
Global IP Device Tracking for clients = Disabled >>>>> IPDT is disabled by default on lanbase
-----------------------------------------------------------------------------------------------
IP Address MAC Address Vlan Interface Probe-Timeout State Source
-----------------------------------------------------------------------------------------------
3850-STACK(config)#int range gig1/0/19 , gig2/0/39
3850-STACK(config-if)#ip device tracking maximum ?
<0-65535> Maximum devices (0 means disabled)
3850-STACK(config-if-range)#ip device tracking maximum 20
3850-STACK(config-if-range)#end
3850-STACK#sh ip device tracking all
Global IP Device Tracking for clients = Enabled >>>>> Make sure IPDT is enabled
Global IP Device Tracking Probe Count = 3
Global IP Device Tracking Probe Interval = 30
Global IP Device Tracking Probe Delay Interval = 0
-----------------------------------------------------------------------------------------------
IP Address MAC Address Vlan Interface Probe-Timeout State Source
-----------------------------------------------------------------------------------------------
Total number interfaces enabled: 2
Enabled interfaces:
Gi1/0/19, Gi2/0/39
Further Problem Description:
none
Unicast ARP packets are duplicated
Bug ID: CSCuv78424
Description
Symptom:
3650/3850 duplicates unicast ARP request packets destined for its IP and sends back 2 replies for one ARP request packet sent by the host.
Conditions:
NA
Workaround:
NA
Further Problem Description:
Customer Visible
Was the description about this Bug Helpful?
(0)
Details
Last Modified: Jun 15,2016
Status: Fixed
Severity: 2 Severe
Product: Cisco Catalyst 3850 Series Switches |
Cisco Catalyst 3850 Series Switches
Support Cases:
2
Known Affected Releases: 15.2(3)E |
|
Known Fixed Releases: |
15.2(2)E4
15.2(2)E5
15.2(3)E3
16.1(1.15)
16.1.2
16.2(0.151)
3.6(4)E
3.6(5)E
3.7(3)E
Denali-16.1.2
02-06-2018 07:36 PM
02-06-2018 08:56 PM - edited 02-06-2018 09:19 PM
02-07-2018 12:06 AM
02-07-2018 03:54 PM
02-14-2018 04:15 PM
Thanks Mark and Francesco that makes total sense I thought that was the case.
I haven't applied any QoS in my network yet, it's all best effort.
I'm not using VOIP or video conferencing so it's not absolutely critical at this stage but at some point but I do need to start classifying and marking the more important traffic between some servers and the client workstations.
Would it be essential that I start applying QoS on my traffic or simply change the queue buffers for best effort?
Sorry QoS is not my specialty so I apologise if I sound silly here.
02-14-2018 04:42 PM
Here is the utilisation on the 2 x 10Gb EtherChannel links.
(sh controllers util)
Port Receive Utilization Transmit Utilization
Te1/1/3 0 0
Te1/1/4 0 0
It always looks like this when I'm viewing it so I'm assuming it's just the occasional bursty traffic that's causing the output drops on the egress port. I do have a 30 second load-interval set on the Portchannel interface that manages the links.
Tx and Rx Load always look very low.
TenGigabitEthernet1/1/3 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 547c.6966.ae9f (bia 547c.6966.ae9f)
MTU 1500 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-CX1
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output never, output hang never
Last clearing of "show interface" counters 4w6d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2353000 bits/sec, 602 packets/sec
5 minute output rate 15850000 bits/sec, 1623 packets/sec
3975312442 packets input, 5107283679684 bytes, 0 no buffer
Received 34726967 broadcasts (27553546 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 27553546 multicast, 0 pause input
0 input packets with dribble condition detected
5482773919 packets output, 6821538322629 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
02-15-2018 12:20 AM
02-15-2018 07:28 AM
02-15-2018 07:56 AM
I have seen this at a couple of our customers after they upgrade to these new switches. If you do a wireshark capture if you see 2 or more ARP packets right in a row it could be this bug.
3850 duplicates pass-through ARP packets
Bug ID: CSCur30273
Description
Symptom:
When ARP request/Reply packets enter an access or trunk interface and its L2 switched to an another access or trunk interface in same VLAN the ARP request/Reply packets gets duplicated. This means that when one ARP packet enter the switch two identical ARP packets exit the switch. We have seen IP Packets getting duplicated. This only seems to affect ARP packets.
Conditions:
L2 switching with lanbase license
Workaround:
The issue is only observed with "lanbase" license. Issue is not seen with "ipbase" license.
Enable IPDT
3850-STACK#sh ip device tracking all
Global IP Device Tracking for clients = Disabled >>>>> IPDT is disabled by default on lanbase
-----------------------------------------------------------------------------------------------
IP Address MAC Address Vlan Interface Probe-Timeout State Source
-----------------------------------------------------------------------------------------------
3850-STACK(config)#int range gig1/0/19 , gig2/0/39
3850-STACK(config-if)#ip device tracking maximum ?
<0-65535> Maximum devices (0 means disabled)
3850-STACK(config-if-range)#ip device tracking maximum 20
3850-STACK(config-if-range)#end
3850-STACK#sh ip device tracking all
Global IP Device Tracking for clients = Enabled >>>>> Make sure IPDT is enabled
Global IP Device Tracking Probe Count = 3
Global IP Device Tracking Probe Interval = 30
Global IP Device Tracking Probe Delay Interval = 0
-----------------------------------------------------------------------------------------------
IP Address MAC Address Vlan Interface Probe-Timeout State Source
-----------------------------------------------------------------------------------------------
Total number interfaces enabled: 2
Enabled interfaces:
Gi1/0/19, Gi2/0/39
Further Problem Description:
none
Unicast ARP packets are duplicated
Bug ID: CSCuv78424
Description
Symptom:
3650/3850 duplicates unicast ARP request packets destined for its IP and sends back 2 replies for one ARP request packet sent by the host.
Conditions:
NA
Workaround:
NA
Further Problem Description:
Customer Visible
Was the description about this Bug Helpful?
(0)
Details
Last Modified: Jun 15,2016
Status: Fixed
Severity: 2 Severe
Product: Cisco Catalyst 3850 Series Switches |
Cisco Catalyst 3850 Series Switches
Support Cases:
2
Known Affected Releases: 15.2(3)E |
|
Known Fixed Releases: |
15.2(2)E4
15.2(2)E5
15.2(3)E3
16.1(1.15)
16.1.2
16.2(0.151)
3.6(4)E
3.6(5)E
3.7(3)E
Denali-16.1.2
06-06-2018 07:46 PM
Apologies for the late reply but you were right burleyman it was a bug related issue. I upgraded to a new IOS release of 03.06.08.E.152-2.E8 and all of the output queues now report no packet drops after a 48 hour period. I'm confident that the issue has been resolved.
Thanks to all you awesome people for your help!
08-20-2019 09:29 AM
I have this problem too.
Please, how did you fixed it?
Thank you
Jackie
08-20-2019 10:57 PM
08-21-2019 05:05 AM
Hi Okatu,
Thank you so much for your reply.
I am running 03.07.02E
And still getting the same error. I am worry because I have so many 3850 almost in the entire network.
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.02E RELEASE SOFTWARE (fc1)
Jackie
08-21-2019 07:26 AM
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide