08-01-2013 08:40 AM - edited 03-07-2019 02:42 PM
Hi,
I'm trying to troubleshoot output drops on an uplink interface. Here is the situation:
Some output below:
as2-1-1#show platform port-asic stats miscellaneous gi1/0/23
Port-asic Misc Statistics
=====================================
TxBufferFullDropCount 676470
TxBufferFrameDescBadCrc16DropCountA 0
TxBufferFrameDescBadCrc16DropCountB 0
TxBufferBandwidthDropCount 0
TxQueueBandwidthDropCount 0
TxQueueMissedDropStatistics 0
SneakQueueDropCount 0
RxBufferDropDestIndexCountA 0
RxBufferDropDestIndexCountB 0
LearningQueueOverflowFrames 1
RxNetworkProcessorPacketCount0 0
RxNetworkProcessorPacketCount1 0
as2-1-1#
as2-1-1#show platform port-asic stats drop gi1/0/23
Interface Gi1/0/23 TxQueue Drop Statistics
!!! Output omitted
Weight 0 Frames 648845
Weight 1 Frames 0
Weight 2 Frames 4
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 26293
!!! Output omitted
as2-1-1#
Has anyone overcome this issue without QoS or replacing hardware?
08-01-2013 06:00 PM
Post the following output:
1. sh version;
2. sh interface Gi 1/0/23; and
3. sh controll e Gi 1/0/23
08-01-2013 06:52 PM
Hi Leo,
Thanks for your reply. I've included a zip file containing the outputs you requested and several graphs. There are discontinuities in the graphs because of a memory issue we have with our monitoring server ( unrelated ). I've also include the output of "show platform port-asic stat drop gi1/0/23" in the file "stat_drop.txt"
One thing I notice is that there are drops in multiple queues ( stat_drop.txt) when QoS is disabled. My understanding is that when QoS is disabled globally, there is a single queue ( i.e.e Queue 1 ).
I've done usual checks while output drops were incrementing:
As you'll note from the graphs, the utilization on the link is really low. I've searched through the 111 published bugs on the Cisco bugtracker tool as well and found none relevant to our environment.
This seems to be a widespread issue on the 2960S platform at and before the code level I'm runing ( 12.2(55)SE2 ). Multiple forums suggested upgrading to at least 12.2(58) but others have shown this not to have solved the issue. Other forum post show implementing QoS ( or disabling) it to be a fix, however these things are not currently implemented in our environment and so do not apply.
T.I.A,
Joe
08-01-2013 07:14 PM
H0ly crap! I can't read your file.
Cut and paste the outputs I've requested in the thread.
I would upgrade your IOS to 12.2(55)SE8.
08-02-2013 12:43 PM
Hi Leo,
Sorry for the late reply. Was off today. See below:
#### Show Ver ####
Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE2, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Tue 11-Jan-11 02:23 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x01B00000
ROM: Bootstrap program is Alpha board boot loader
BOOTLDR: C2960S Boot Loader (C2960S-HBOOT-M) Version 12.2(53r)SE, RELEASE SOFTWARE (fc3)
as2-1-1 uptime is 1 week, 3 days, 8 hours, 11 minutes
System returned to ROM by power-on
System restarted at 13:19:33 EDT Mon Jul 22 2013
System image file is "flash:/c2960s-universalk9-mz.122-55.SE2/c2960s-universalk9-mz.122-55.SE2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.
Processor board ID FOC1508X3U0
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
28 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
If you require further assistance please contact us by sending email to
cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.
Processor board ID FOC1508X3U0
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
28 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : B8:BE:BF:30:1C:80
Motherboard assembly number : 73-11910-06
Power supply part number : 341-0328-02
Motherboard serial number : FOC15077RQ0
Power supply serial number : DCA1432M6W8
Model revision number : C0
Motherboard revision number : A0
Model number : WS-C2960S-24TS-L
Daughterboard assembly number : 73-11933-04
Daughterboard serial number : FOC150643E7
System serial number : FOC1508X3U0
Top Assembly Part Number : 800-30954-02
Top Assembly Revision Number : B0
Version ID : V02
CLEI Code Number : COMGG00ARB
Daughterboard revision number : A0
Hardware Board Revision Number : 0x01
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 28 WS-C2960S-24TS-L 12.2(55)SE2 C2960S-UNIVERSALK9-M
Configuration register is 0xF
### Show int ###
as2-1-1#show int gi1/0/23
GigabitEthernet1/0/23 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is b8be.bf30.1c97 (bia b8be.bf30.1c97)
Description: PHY|L2|rhost:cs2-1-1|rport:gi1/1
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:01, output 00:00:03, output hang never
Last clearing of "show interface" counters 06:52:22
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2647
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1659000 bits/sec, 1287 packets/sec
5 minute output rate 2558000 bits/sec, 1198 packets/sec
76224138 packets input, 11348240054 bytes, 0 no buffer
Received 468086 broadcasts (114552 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 114552 multicast, 0 pause input
0 input packets with dribble condition detected
79685680 packets output, 36998897195 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
as2-1-1#
### Show controllers ###
as2-1-1#show controllers ethernet-controller gi1/0/23
Transmit GigabitEthernet1/0/23 Receive
1360350908 Bytes 4218320450 Bytes
1594259627 Unicast frames 2733236679 Unicast frames
371647 Multicast frames 4347119 Multicast frames
564581 Broadcast frames 12786960 Broadcast frames
0 Too old frames 2854689159 Unicast bytes
0 Deferred frames 545250746 Multicast bytes
0 MTU exceeded frames 818380545 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 201288077 Minimum size frames
0 8 collision frames 867680707 65 to 127 byte frames
0 9 collision frames 107819549 128 to 255 byte frames
0 10 collision frames 60471382 256 to 511 byte frames
0 11 collision frames 79485747 512 to 1023 byte frames
0 12 collision frames 1433625296 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
96727731 64 byte frames 0 Valid frames, too small
414347781 127 byte frames
292401594 255 byte frames 0 Too old frames
247938903 511 byte frames 0 Valid oversize frames
201842506 1023 byte frames 0 System FCS error frames
341937340 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
as2-1-1#
### Show platform ... ###
as2-1-1#show platform port-asic stats drop gi1/0/23
Interface Gi1/0/23 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 651608
Weight 1 Frames 0
Weight 2 Frames 4
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 27867
Queue 4
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 5
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 6
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 7
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
as2-1-1#
As mentioned, there is no QoS enabled globally. This is a simple layer 2 switch configuration.
Also attached the graphs.
Thanks again.
Joe
08-02-2013 05:35 PM
As mentioned, there is no QoS enabled globally
Ok, thanks for the output. I see no line errors. I see, however, "output drops".
Disable interface-level QoS and see if output drops still increments.
Can you tell me what's connected to Gi 1/0/23?
08-02-2013 05:42 PM
Leo,
Interface QoS is also disabled:
as2-1-1#show mls qos inter gi1/0/23
GigabitEthernet1/0/23
QoS is disabled. When QoS is enabled, following settings will be applied <<<<<<<<<<< QoS disabled
trust state: not trusted
trust mode: not trusted
trust enabled flag: ena
COS override: dis
default COS: 0
DSCP Mutation Map: Default DSCP Mutation Map
Trust device: none
qos mode: port-based
as2-1-1#
A Cisco 4948 switch is connected on the gi1/0/23 interface (in trunk mode ).
08-02-2013 05:50 PM
Did you disable QoS on the 4948-side too?
08-02-2013 05:56 PM
Good catch,
Inteface QoS is enabled on the other side of the trunk, something I hadn't known about/looked at before:
cs-2-1-1#show qos inter gi1/1
QoS is disabled globally
Port QoS is enabled
Administrative Port Trust State: 'untrusted'
Operational Port Trust State: 'dscp'
Trust device: none
Default DSCP: 0 Default CoS: 0
Appliance trust: none
Tx-Queue Bandwidth ShapeRate Priority QueueSize
(bps) (bps) (packets)
1 250000000 disabled N/A 2336
2 250000000 disabled N/A 2336
3 250000000 disabled normal 2336
4 250000000 disabled N/A 2336
cs-2-1-1#
However, how would enabling QoS on the far end affect Output drops on the near end?
08-02-2013 06:00 PM
However, how would enabling QoS on the far end affect Output drops on the near end?
It's the opposite. If you enable QoS in a LAN, there's a significant chance of getting output drops.
08-02-2013 06:09 PM
Leo Laohoo wrote:
However, how would enabling QoS on the far end affect Output drops on the near end?It's the opposite. If you enable QoS in a LAN, there's a significant chance of getting output drops.
That make sense if QoS is enabled on every switch in the LAN, in this case, only the upstream switch has port QoS enabled. On the downlink switch, there is no QoS and all traffic is treated equally.
I guess my question is, how would having QoS enabled on the upstream switch cause output drops on the downstream switch (e.g. for traffic egressing to the upstream switch )?
07-24-2019 04:36 AM
Hello,
Did the issue finally resolved ?
I am asking because we are facing a similar issue.
Regards,
07-24-2019 05:11 AM
Without knowing the exact specifics of your particulat problem, one of the first thinsg you could usually try is to configure 'qos queue-softmax-multiplier 1200' on your switch...
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide